General Information
The Linux OOM (Out Of Memory) Killer is a kernel function that will automatically kill off processes if there is memory contention on a system.
Checklist
It does this by calculating how “bad” a process is in order to determine which to kill off.
If invoked, the goal is to kill processes so that:
The function actually does this:
Besides an application/process dying, evidence of oom executing can be find in the kernel log file.
Example:
grep -i "out of memory" /var/log/kernel Jan 15 10:05:32 dbserver01 kernel: Out of memory: Kill process 39018 (mysqld) score 441 or sacrifice child
Normally, when OOM is called, it is because an application has been poorly coded or configured to ask for a ridiculous amount of memory that it doesn't actually need. Then, when the application misbehaves and starts to crash/fork, it eats up a ton of memory very quickly…OOM steps in and starts killing things.
The OS could be configured incorrectly too though.
There are two settings that can be tuned to change the system behavior surrounding memory and swapping. Depending upon how they are adjusted, this can make it less or more likely for OOM to be called.
The settings are both in the virtual file system, /proc:
This setting can allow for applications to reserve (but not use) more total memory than what the system has available. This is similar to memory over commitment in virtual machines.
This setting should stay at the default (0) unless you really know what you are doing and its an edge case.
Possible settings are 0,1,or 2:
Check the setting:
cat /proc/sys/vm/overcommit_memory 0
To change the setting: edit sysctl.conf, add a line, save, and re-read the config:
vim /etc/sysctl.conf vm.overcommit_memory=2 :wq sysctl -p
Swappiness controls how likely the kernel is to move processes out of memory and onto swap disk space.
The setting is from 0 to 100.
Check the setting:
cat /proc/sys/vm/swappiness 60
To change the setting: edit sysctl.conf, add a line, save, and re-read the config:
vim /etc/sysctl.conf vm.swappiness=1 :wq sysctl -p
Starting in kernel 3.5-rc1, the way that swappiness behaves when set to “0” was changed. Red Hat backported this behavior change to RHEL kernel 2.6.32-303.
vm.swappiness=0 meant that the kernel would avoid swapping for as long as it could, but when things got tight, it would swap.
vm.swappiness=0 now means to really…don't swap…until the very last possible moment.
At this point, OOM is typically called before anything gets a chance to swap.
If you are trying to avoid swapping as much as possible, the newer kernels should not be set below 1.
The new vm.swappiness=1 is like the old vm.swappiness=0 behavior.