Linux HugePages for Oracle on Amazon EC2

Posted in: Technical Track

One of the optimizations available to us when running Oracle on Linux is huge page support. This feature of the Linux kernel enables processes to allocate memory pages of size 2M (instead of 4k). In addition, memory allocated using hugepages is pinned in physical memory. It cannot be swapped out.

It is now common practice to enable huge page support for Oracle databases with large SGAs (one rule of thumb is 8G). Without this feature, the SGA can be, and often is, paged out. Paging out portions of the SGA can result in disastrous consequences from a performance standpoint. There are a variety of load patterns that perform particularly poorly without hugepages. Running with large numbers of processes, sudden increases in processes (connection storms), and highly concurrent access of diverse sets of SGA pages all can bring an Oracle system without hugepages to its knees.

Considering how commonplace use of hugepages is with Oracle, it is surprising that on Amazon EC2, huge page support is not generally available, and that the systems designed expressly for running Oracle cannot use hugepages.

EC2 Virtualization Types

Under the covers, Amazon EC2 uses a hypervisor to (potentially) run many virtual machines on a given physical server. EC2 instances essentially come in two flavours of virtualization: paravirtualization (PVM) and hardware virtualization (HVM). The vast majority of EC2 AMIs use PVM, but for a variety of reasons, only EC2 instances using HVM can allocate hugepages.

Initially, I tried to enable hugepages with PVM. I was able to load a kernel with huge page support, and even enable hugepages and allocate a shared memory segment using hugepages. However upon trying to attach to the shared memory segment, I received a kernel panic. For a simple test case, I used a C program that is included in the Linux kernel distribution which creates a small shared memory segment using hugepages attaches to it: tools/testing/selftests/vm/hugepage-shm.c.

[root@ip-10-28-12-158 ~]# echo 512 >/proc/sys/vm/nr_hugepages
[root@ip-10-28-12-158 ~]# grep Huge /proc/meminfo
HugePages_Total: 512
HugePages_Free: 512
Hugepagesize: 2048 kB

[root@ip-10-28-12-158 ~]# strace ./hugepage-shm
execve(“./hugepage-shm”, [“./hugepage-shm”], [/* 22 vars */]) = 0

shmget(0x2, 268435456, IPC_CREAT|SHM_HUGETLB|0600) = 1146881
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), …}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f469afd1000
write(1, “shmid: 0x118001\n”, 16shmid: 0x118001) = 16
shmat(1146881, 0, 0)………

At this point, the program hangs, and the kernel panics.

However, when I try the same experiment on an identical EC2 instance running with HVM, the test completes successfully. After trying to use hugepages on a variety of operating systems and configurations on EC2, I conclude that for EC2, huge page support is only available on HVM instances.

HVM is available on the following instance classes:

m3.xlarge13.015.00.5012nd Gen Standard
m3.2xlarge26.030.01.0022nd Gen Standard
cc2.8xlarge88.060.52.408Cluster Compute
cr1.8xlarge88.0244.03.508Memory-optimized cluster
hi1.4xlarge35.060.53.104I/O optimized
hs1.8xlarge35.0117.04.604Storage optimized
cg1.4xlarge33.522.52.104Cluster GPU

* Elastic Compute Units: “The equivalent CPU capacity of one 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor”
** US$ per hour on-demand in us-east-1 region

Fine, I’ll use EC2 instances with HVM. What’s the big deal?

Well, at Openworld in 2010, Oracle and Amazon announced that EC2 would allow customers to select Oracle VM as the underlying hypervisor for their Amazon EC2 instances. This made EC2 a platform completely supported by Oracle.

However, when Oracle and Amazon released the OVM-backed instances, they could not be run on any of the above HVM-capable instance classes. The EC2 API responds with:

Client.InvalidParameterCombination: Non-Windows instances with a virtualization type of 'hvm' are currently not supported for this instance type.

This means that if you want to use hugepages on EC2, you cannot use any of the Oracle VM-backed instances. What??? AWS and Oracle went to the trouble of making OVM available to support running Oracle software on EC2, but didn’t bother to enable it on any of the instance classes capable of hugepages, which is recommended by Oracle?

We have all been over and over the question of whether to run Oracle on non-Oracle hypervisors like VMware and Xen. The policy from Oracle is something like “You can run Oracle on non-Oracle hypervisors, but if you call for support and we suspect the problem is with the hypervisor, we reserve the right to ask you to reproduce the problem on bare metal.”

Well for Amazon Xen, the same policy applies. Essentially we are left with the choice: When running Oracle on Amazon EC2, either run efficiently with hugepages on a not-totally-supported hypervisor, or run inefficiently without hugepages on a fully supported Oracle VM-backed instance.

AWS and Oracle take note: As it stands, the OVM offering is not set up in a way that enables real enterprise use of EC2 by Oracle customers. This is a major oversight that needs to be addressed.

Interested in working with Jeremiah? Schedule a tech call.

About the Author

Jeremiah Wilton has over eighteen years of Oracle database administration and systems architecture experience. As’s first database administrator, he helped lead’s database group from the pre-IPO period through the company’s years of exponential growth. He now works for Pythian, a leader in remote database administration for Oracle, Oracle Applications, SQL Server and MySQL. Jeremiah also teaches the Oracle Certificate Program at the University of Washington. At Oracle OpenWorld in 2001, Oracle Education honored Jeremiah as one of the first eight Oracle Certified Masters in the world. Jeremiah is a member of the OakTable, and co-author of the Oak Table’s Oracle Insights. He has presented at numerous conferences and user groups, including Oracle OpenWorld, Collaborate! and UKOUG, and is the author of a variety of technical whitepapers and articles.

2 Comments. Leave new

A totally agree. I’m forced as I grow to either:
1) be stuck with an overpriced pvm instance that does 117G ram, but has 24 2tb disks I don’t need
2) manually install oracle on hvm under opensuse or RHE (which is a big pain, the oracle AMI’s save a ton of time to deploy a new DB) and risk the support situation you mentioned.

I feel I’m forced to take my chances to prototype out a setup of option 2 with a new R3 class instance. I contacted some oracle savvy guys at AWS and they said the issue is the Oracle Enterprise Linux (It appears to be nothing more than a search and replace with from Redhat to Oracle with modified startup scripts to create swaps at first glance) OS itself never got set up to run under HVM types. It appears this is stuck in a ‘no mans land’ between Oracle and AWS on who should fix it. Since they both have public clouds, I think there is less incentive for Oracle to help out AWS in this case if they think they can push you to Oracle’s offering. That’s just my opinion though, I don’t have any proof of that. I’ll try to post our results as we are building a virtual lab to test out Oracle on the HVM instances. It could be that HVM may not (this seems a gray area) fall under the 4 core per socket policy Oracle set up years ago with AWS and because HVM can tie a core directly to a socket unlike pvm, there is less financial incentive if you just license the entire box(2 sockets in most cases) like a VMWare setup as they would make less $ in licenses (2 sockets with 16 cores for Std or Std One == 2 licenses vs 8+ on pvm).

July 7, 2014 8:54 am

Must consider 2 aspects about hugepages… First the hypervisor can use large pages to back all guest memory without having the guest OS knowing. Vmware ESX does that by default (but not Xen). Then you can also enable large pages in the guest OS to further reduce the pagetable lookup overhead. Ideally you have both hypervisor and guest OS using large pages.

On Xen with HVM, you can enable large page in the guest OS (like you saw on EC2), but the hypervisor will not map these as 2MB pages in its own physical page table. For PVM, enabling large pages must be set in the hypervisor and also in the guest configuration (you cant enable that yourself… EC2 must allow it at Xen domain creation).

Also, it makes less and less sense to use PVM mode on modern x86_64 CPUs with Intel VT-x/ AMD-V. PV mode will go extinct.


Leave a Reply

Your email address will not be published. Required fields are marked *