After having discussed about the Memory Overcommitment earlier, it’s time to discuss about the different Memory Management Techniques that an ESXi host implements when it is over committed. As of vSphere 6, ESXi uses 5 memory techniques to ensure host’s memory is used efficiently – Idle Memory Tax, Transparent Page Sharing, Ballooning, Compression and Swapping. Let’s discuss about each one of these in detail.
Idle Memory Tax
VMware Tools inside the guest OS uses its balloon driver to determine the memory blocks that are allocated but idle. These memory blocks can be used for other VMs by Idle Memory Tax. A maximum of up to 75% of allocated memory can be borrowed by IMT to service another VM. This is a per VM setting and can be changed in the advanced settings of the VM.
Transparent Page Sharing
Transparent Page Sharing shares identical memory pages within the VMs and thus decreases the total memory pages consumed, decreasing the total memory footprint on the ESXi host. ESXi calculates hashes of the memory pages to find the identical memory pages, once a match is found, memory pages are mapped to physical memory page reducing the memory footprint on the physical memory. Inter VM TPS is disabled by default from version 6 and updates on 5.0, 5.1, 5.5 also pushes this. More info here.
ESXi uses 4KB memory pages by default, in case of hardware enabled features like Intel Extended Page Tables and AMD Rapid Virtualization Indexing, 2 MB memory pages are used and 4KB hashes are calculated within the 2MB memory pages.
Ballooning works only when VMware Tools are installed on the guest OS, this is due to the fact that, ballooning uses a specific driver called balloon driver (vmmemctl) which is part of VMware Tools. When there is a contention for memory on the ESXi host, hosts communicates to guest OS using VMware Tools, the balloon driver inflates causing a process to start within guest OS and thus gains some free memory from the OS which the balloon driver further handover to ESXi host. This situation does not impact performance because Guest OS is giving away only the free memory but in case if the Guest OS is already under contention, balloon driver causes the memory pages to swap out within OS causing performance impact. Once the ESXi host passes the contention, balloon driver deflates releasing the memory for guest OS. The default maximum balloon size is 65% of the guest memory.
Balloon driver can be disabled by deleting the sched.mem.maxmemctl parameter in the vmx file. How-To? Best practice is to not disable Ballooning.
Memory compression occurs when an ESXi host sees a need for hypervisor swapping. VMkernel attempts to compress the memory and keep them in a compressed memory cache. Memory pages compressed by at least 50% are kept in memory cache and not swapped out to disk. This increases efficiency when memory pages are needed back.
This is the last memory technique an ESXi host implements as this impacts the performance a lot. When there is an absolute contention for memory and no other previously mentioned techniques helped relieve it, swapping happens. Swapping the host memory to the physical disks is absolutely slow and even swapping memory to SSD is also very performance impacting.
All the above mentioned Memory Overcommit Techniques are dependent on the Reservations, Shares and Limits configured on the VMs.
Hope this was informative. Thanks!