General Information

Kernel tuning using sysctl.conf

Checklist

• Distro(s): Any

Problem: Unable to access a VM's console, and it shows out of memory errors on boot.

Solution: Reduce the value of “vm.nr_hugepages” to less than 1/2 of total memory in MBs.

/etc/sysctl.conf

vm.nr_hugepages = 1024

1 hugepage = 2 MBs, so the above is equal to 2GB

Explanation: Hugepages that is configured to be greater than or equal to total installed memory causes memory starvation during boot due to no memory available for normal use.

Alternative config location: Hugepages could also be configured at /boot/grub/grub.conf

On some kernels, khugepaged can start running at 100% CPU utilization. This is typically seen on systems that have process/memory intensive processes.

Other symptoms include:

• Normal commands hang (w, uptime, ps)
• Processes showing above 100% CPU (ie sometimes up to 1,300%)

Details

• Known Affected kernels
  • CentOS 6: 2.6.32-431.el6.x86_64
• Reference bug report: https://bugzilla.redhat.com/show_bug.cgi?id=879801

Disable hugepage defragmenting:

echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

Add new cron entry

vim /etc/cron.d/hugepage_defrag
 
# Disable kernel huge page defrag due to bugzilla bug: 879801
@reboot root /bin/echo never > /sys/kernel/mm/transparent_hugepage/defrag

Ultimately, the permanent fix is to update the kernel to a newer version.

TODO: Will add confirmed kernel versions that this bug is fixed in.

Swappiness controls how likely the kernel is to move processes out of memory and onto swap disk space.

The setting is from 0 to 100.

• 0 = Very aggressively avoid swapping for as long as possible
  • High risk of OOM killing from memory and I/O pressure
• 10 = Red Hat recommended for Oracle databases
• 60 = Linux default
• 100 = Aggressively swap from memory to disk

Check the setting:

cat /proc/sys/vm/swappiness
60

To change the setting: edit sysctl.conf, add a line, save, and re-read the config:

vim /etc/sysctl.conf
vm.swappiness=10
:wq
sysctl -p

More details about OOM: https://owlbearconsulting.com/doku.php?id=linux_wiki:oom_killer

Tuning how often cached pages in memory are flushed to disk.

Check current settings:

sysctl -a | grep dirty

Background (async) pagecache flushing:

vm.dirty_background_ratio = 3

• Explanation: Start flushing pages to disk asynchronously when pagecache is equal to 3% of memory.
• Default value = 10

Foreground (sync) pagecache flushing:

vm.dirty_ratio = 15

• Explanation: Sync flush when pagecache is 15% of total memory. This blocks ANY process from using I/O when this happens. If this process takes longer than 120 seconds, kernel panics can happen.
• Default value = 20

Viewing and setting the system wide file descriptors.

View system wide file descriptor information

sysctl fs.file-nr
fs.file-nr = 6144	0	809542

The three numbers returned:

• 6144 = used file descriptors
• 0 = allocated but not used
• 809542 = system max

Change System FD Limits

Edit /etc/sysctl.conf

fs.file-max = 810542

Example above increases system wide max by 1,000.

More details, including per user settings: https://owlbearconsulting.com/doku.php?id=linux_wiki:file_descriptors

Reboot a system after a 10 seconds of kernel panic

kernel.panic = 10