This is an old revision of the document!
Kernel Tuning
General Information
Kernel tuning using sysctl.conf
Checklist
- Any distro, but many examples are done via Red Hat based systems
Hugepages
Problem: Unable to access a VM's console, and it shows out of memory errors on boot.
Solution: Reduce the value of “vm.nr_hugepages” to less than 1/2 of total memory in MBs.
/etc/sysctl.conf
vm.nr_hugepages = 1024
1 hugepage = 2 MBs, so the above is equal to 2GB
Explanation: Hugepages that is configured to be greater than or equal to total installed memory causes memory starvation during boot due to no memory available for normal use.
Alternative config location: Hugepages could also be configured at /boot/grub/grub.conf
HugePage Bug
On some kernels, khugepaged can start running at 100% CPU utilization. This is typically seen on systems that have process/memory intensive processes.
Other symptoms include:
- Normal commands hang (w, uptime, ps)
- Processes showing above 100% CPU (ie sometimes up to 1,300%)
Details
- Known Affected kernels
- CentOS 6: 2.6.32-431.el6.x86_64
- Reference bug report: https://bugzilla.redhat.com/show_bug.cgi?id=879801
Workaround Fix
Disable hugepage defragmenting:
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
Add new cron entry
vim /etc/cron.d/hugepage_defrag # Disable kernel huge page defrag due to bugzilla bug: 879801 @reboot root /bin/echo never | /usr/bin/tee /sys/kernel/mm/transparent_hugepage/defrag
Permanent Fix
Ultimately, the permanent fix is to update the kernel to a newer version.
TODO: Will add confirmed kernel versions that this bug is fixed in.
Swappiness
Swappiness controls how likely the kernel is to move processes out of memory and onto swap disk space.
The setting is from 0 to 100.
- 0 = Very aggressively avoid swapping for as long as possible
- High risk of OOM killing from memory and I/O pressure
- 10 = Red Hat recommended for Oracle databases
- 60 = Linux default
- 100 = Aggressively swap from memory to disk
Check the setting:
cat /proc/sys/vm/swappiness 60
To change the setting: edit sysctl.conf, add a line, save, and re-read the config:
vim /etc/sysctl.conf vm.swappiness=10 :wq sysctl -p
More details about OOM: https://owlbearconsulting.com/doku.php?id=linux_wiki:oom_killer
Page Cache
Tuning how often cached pages in memory are flushed to disk.
Check current settings:
sysctl -a | grep dirty
Background (async) pagecache flushing:
vm.dirty_background_ratio = 3
- Explanation: Start flushing pages to disk asynchronously when pagecache is equal to 3% of memory.
- Default value = 10
Foreground (sync) pagecache flushing:
vm.dirty_ratio = 15
- Explanation: Sync flush when pagecache is 15% of total memory. This blocks ANY process from using I/O when this happens. If this process takes longer than 120 seconds, kernel panics can happen.
- Default value = 20
File Descriptors
Viewing and setting the system wide file descriptors.
View system wide file descriptor information
sysctl fs.file-nr fs.file-nr = 6144 0 809542
The three numbers returned:
- 6144 = used file descriptors
- 0 = allocated but not used
- 809542 = system max
Change System FD Limits
Edit /etc/sysctl.conf
fs.file-max = 810542
Example above increases system wide max by 1,000.
More details, including per user settings: https://owlbearconsulting.com/doku.php?id=linux_wiki:file_descriptors