This is an old revision of the document!
High System Load
General Information
Troubleshooting high system load.
Checklist
- Distro: Enterprise Linux 6.x
Troubleshooting Tools
The following tools are useful when troubleshooting system load.
Typically built in
- uptime
- top
- vmstat
Need to install (if using a minimal install base)
- iotop
- iostat (sysstat package)
Base Repo
yum -y install iotop sysstat
Troubleshooting Steps
- Know how many processors you have. This is essential to determine if load is high.
grep -c processor /proc/cpuinfo
- %Load (decimal) = (Load Average / Number Processors)
- Example: Number of processors = 2, load average seen = 1.50
- 1.50 / 2 = 0.75 or 75% load on the processors
- Check load averages
- Uptime shows the load average for the last 1, 5, and 15 minutes. If it is too high or trending up, time to investigate further.
uptime
- What kind of load
- Use vmstat to determine what kind of system load. “vmstat 1” prints stats every 1 second.
vmstat 1
- Important columns to take note of:
- CPU: “wa” ⇒ Time spent waiting for I/O. If high, something is probably heavily utilizing disk.
- CPU: “id” ⇒ CPU time spent idle. If close to 0, CPU is used heavily.
- CPU: “sy” ⇒ CPU time spent running system/kernel processes. Mail and firewalls are common causes of high system use.
- CPU: “us” ⇒ CPU time spent running user processes. If high, investigate with top.
- IO: “bi” ⇒ blocks received from block device each second. (If high, something is heavily reading a disk)
- IO: “bo” ⇒ blocks sent to a block device each second. (If high, something is heavily writing to disk)
- SWAP: “si” ⇒ Memory swapped in from disk each second.
- SWAP: “so” ⇒ Memory swapped to disk each second.
- If either are high, memory is most likely also very low.
- MEMORY: “free” ⇒ memory free. If this is low, there is probably swapping going on as well.
- Further investigate either high CPU/Memory use or Disk I/O
High CPU
Clues that you should investigate high CPU usage:
- Low CPU “id” (idle)
- High CPU “sy” (system processes)
- High CPU “us” (user processes).
top
While in top:
- Turn on highlighting: 'z'
- Highlight sort column: 'x'
High Memory Use
Clues that you should investigate high memory usage:
- Memory free: Very low
- Swap si: High swapping in from disk
- Swap so: High swapping out to disk
Start top and sort by %mem usage
top -a
While in top:
- Turn on highlighting: 'z'
- Highlight sort column: 'x'
Other memory columns (shift+ '<' or '>' to change sort columns)
- VIRT = virtual memory size used: code, data, and shared libraries plus swapped pages.
- RES = resident size: non-swapped physical memory a process is using
Disk I/O
Clues that you should investigate high Disk I/O:
- High CPU “id” (idle)
- High CPU “wa” (wait)
iostat iotop