The mysterious world of shmmax and shmall
Over the Top Tales from the trenches.
Motto: Bringing order to the chaos of every day DBA life.
Dear Diary,
Today I delved into the internal workings of Linux kernel memory settings.
A recalcitrant machine would not allow me to open a second newly created instance due to ORA-27102 “out of memory”.
I determined using free that there was in fact 16 Gigabytes of fresh free RAM available, and this was linux x86_64 from uname -a.
So no jumping through hugepage hoops to get a 32 bit OS to allow a 32 bit db address more than 1.7 Gig.
So what was the story, I wondered…
Do I ring the bat phone on the desk and ask a SA (System Admin) or Unix admin as we call them here, and get the answer in a jiffy, handed on a plate?
No, you can only learn by pushing the envelope, lifting that little extra weight, growing that bit taller to reach the sun using whatever fertiliser required, google and metalink are good starts.
I walked my well worn path to the sage of metalink, payed homage and received enough info (Note:301830.1) to begin the journey to discovering how linux kernel shared memory settings affect Oracle.
Like Soviet central planners there are two important settings. Like most people I knew about shmmax, but it is sly, it is not the maximum amount of memory which can be allocated, it is the maximum size of any shared memory chunk.
Shmmax is how big a bite you want per bite from free memory.
The real godfather, the wizard behind the curtain is shmall. Its value determines the maximum amount of memory that ALL shared memory can take.
Just to make it fun, the actual setting is derived…
the maximum amount of memory = shmall * pagesize
where pagesize = getconf PAGE_SIZE and shmall = cat /proc/sys/kernel/shmall
Making shmall larger than free RAM is a recipe for paging hell and much gnashing of teeth. Oracle recommends half the RAM, we pushed the envelope and chose 75% as 8 gigabytes of free for OS and cache is just wasteful.
Especially given Oracle is already caching hot blocks in its memory.
Happy days and the 2nd and 3rd instance had plenty of room to startup and become managed standbys… a story for another day.
Boiled down to one sentence summary:
shmall determines the total amount of shared memory to be allocated using its value multipled by the OS pagesize.
Have Fun
Paul
September 17th, 2006 at 12:45 pm
Hi Paul,
After reading the diary, I am start to wondering about the settings for our system. We have 2 databases running on the same RHEL3 box. Total RAM is 6G. The current settins are as follows:
kernel.shmall = 2097152
kernel.shmmax = 2147483648
Right now, for both databases, the sga parameters are as follows:
sga_max_size big integer 1536M
sga_target big integer 1536M
I am thinking of doubling the kernel.shmall parameter (from 2097152 to 4194304), since there 2 databases running on the same server. I’d like to hear your thoughts on this.
Thanks,
Hai
September 18th, 2006 at 6:06 am
Hi Hai,
If your pagesize is 4096 that setting for kernel.shmall will allow up to 8G to be addressed in a 64 bit OS running 64 bit Oracle. Given you only have 6G of RAM the setting may be better set lower not higher.
Have Fun
Paul
September 19th, 2006 at 4:52 am
Thanks Paul.
- Hai
July 30th, 2007 at 12:12 pm
[…] pythian.com/blogs/245/the-mysterious-world-of-shmmax-and-shmall […]
July 30th, 2007 at 4:54 pm
Hi Paul,
Just thought that I’d mention that you page size changes
depending of you are using hugepages or not
RHEL 3 - vm.hugetlb_pool
RHEL 4 - vm.nr_hugepages
the default pagesize for Redhat at least is 4K, vs 2048K with hugepages enabled
cat /proc/meminfo |grep Hugepagesize
Thanks
October 13th, 2008 at 9:09 am
#!/bin/bash
cat /proc/sys/kernel/shmall
TotalKbytes=`awk ‘/MemTotal:/ { print $2 }’ /proc/meminfo`
TotalBytes=`expr $TotalKbytes \* 1024`
PageSize=`getconf PAGE_SIZE`
ShmallValue=`expr $TotalBytes / $PageSize`
echo $ShmallValue
echo $ShmallValue > /proc/sys/kernel/shmall
sysctl -p
cat /proc/sys/kernel/shmall
October 30th, 2008 at 12:58 pm
[…] thought that the SHMMAX kernel parameter (see also Paul Moen’s The mysterious world of shmmax and shmall) was limiting as the SGA size was bigger than SHMMAX size. After changing the SHMMAX parameter, […]