vMotion In-Depth

vMotion, the one feature that excited me a lot when i first heard about it. vMotion is still my favourite feature of vSphere, it has come a long way. Many new learners still keep asking about the process of vMotion, so i thought of writing a bit about it.

What is vMotion?

Migration of a powered on VM from one ESXi host to another without any downtime is called as vMotion. Well that is a technical definition, but what exactly are migrated from one host to another? To answer that you should basically understand what actually is needed for a VM to run – Memory, CPU, Network, Storage. Let’s look at the step by step process on how all these components gets migrated from one host to another without downtime.

To help you understand better, let’s consider an example of a VM named ABC being migrated from host ESXi-A to ESXi-B.

Step 1 : vMotion is initiated by Administrator.

Step 2 – PreCopy : The memory contents of the vm ABC are obviously located on the ESXi-A’s physical memory, hence a copy of all the active memory pages of ABC are copied to ESXi-B’s memory. While the copy is being done, the memory pages gets changed due to the obvious reason that the VM is functional. ESXi-A keeps track of all the changes to the memory in a log, this log is known as memory bitmap; memory bitmap doesn’t hold the contents that changed but the adresses of the changes, often referred as dirty memory. Remember, all the copy is done via a VMkernel interface and happens iteratively untill all the contents are copied to ESXi-B.

Step 3 : Now that the ABC’s memory contents are copied to destination, the VM is quiesced and the memory bitmap is transferred to ESXi-B. Quiesce is not suspend, it is just a state where VM is still in memory but is not available for client requests.

Step 4 : ESXi-B now reads the memory addresses in the memory  bitmap and requests the contents in those addresses from ESXi-A.

Step 5 : VM now resumes on the ESXi-B (not to be confused with reboot) and a RARP message is sent by ESXi-B to register it’s MAC against the physical switch port.

Step 6 : ESXi-A now free’s up the memory used by ABC. vCenter Server deletes the VM ABC from ESXi-A.

Great! We now know how a vMotion works,  but for all this magic to happen,we need to understand the requirements of vMotion as well. Here are those…

  • vCenter Server
  • Shared Storage
  • VMkernel port enabled with vMotion; vDS or Standard switch can be used.
  • Identical Virtual Switches and Port Groups on source and destination. If it is vDS, hosts should share the same vDS.
  • Source and Destination host CPU’s must be compatible – same make, family
  • No CD/DVD’s to be connected VM during migration.
  • VM must not be connected to internal only switch.
  • No affinity configured on VM with host.

Relaxation on requirements

  • From vSphere 5.1 vMotion works even without having a shared storage. In this case disk contents are copied first as changes occurring to disks are less compared to changes to memory.
  • vSphere 6 removes the requirement for shared vDS but it is best practice to have them on use same vDS to avoid consistency issues during DRS calculations.
  • vSphere 6 also removes the requirement to have same port group name on the destination host, vMotion wizard now allows us to select a different port group on the destination host.

Other things to know

  • vMotion will only transfer the VM to the destination if it is certain that it can complete the memory copy.
  • If vMotion cannot complete the memory copy it will fail with no impact to the running VM.
  • vMotion can now operate on routed networks. Supports upto 150ms round trip time and cross vCenter migrations.
  • Cross vCenter migrations are supported when both vCenters are atleast on version 6 and must share same PSC
  • vMotion can now use multiple NICs to transfer memory between hosts.
  • The VM’s memory contents are transferred between hosts in clear text and hence it is recommended to use dedicated network for vMotion.
  • High priority migrations do not proceed if sufficent resources are available on destination host. Standard priority migrations proceed but can fail in case of resource unavailability.
  • For RDM connected VM’s, select Same Format as Source during storage vMotion, failing to select this will convert the virtual RDM’s to vmdk and physical RDM’s are unaffected. vmdk’s cannot be converted back to RDM.
  • Storage vMotion renames teh under lying files in the datastore to allign with the name of the VM. If only one disk is being migrated, only the migrated disk is renamed.
  • If source and destination hosts see same storage, then storage vMotion is intelligent enough to use storage network for data transfer instead of VMkernel network.
  • During cross vCenter migration, since the Universal Unique Identifier (UUID) is retained, all the settings like HA, DRS rules, Alarms, Tasks, Events, Shares, Limits and Reservations are also transferred.
  • Since VM performance is written to vCenter DB, during the cross VC migrations, this data is not migrated.
  • The MAC address of a VM is blacklisted on the source vCenter during cross vCenter migrations to prevent conflicts with any new builds.

vMotion Fails

10% — KB

14% — KB1 KB2

82% — KB

90% — KB

Hope this post was informative. Thanks!