How to find the Duplicate VM UUID’s

I recently came across an issue with the VMware SRM where there was a VM having the same UUID in both the primary and recovery site. This VM was a clone from the primary site and was never powered on the recovery site but was rather cloned to template and was used for new builds, over time, everyone forgot about the cloned VM and we recently started seeing issues with the recovery plan. We were unable to test and cleanup the recovery plan. We started seeing the below PANIC message in the SRM logs.

Panic: VERIFY d:/build/ob/bora-3037005/srm/src/recovery/settings/vmRecoverySettingsRepository.cpp:1291

This is a known issue in SRM 6.1 and KB related to this can be found here. The KB does not provide a way to look for the duplicate UUID’s so here is what we used to find the VM UUIDs on both Primary and Recovery vCenter servers.

Run the below code to get the VM’s and their UUIDs in a csv, once you have the csv, its very easy to identify the duplicates. Once you have the VM identified, Shutdown the VM, browse to the VM datastore and backup the vmx file of the VM and edit the value for uuid.bios and re upload to the datastore and power on the VM. Just to make sure it does conflict with the other VMs again, verify the UUID you used is not present in the csv file.

Here are step by step instructions on editing the vmx files Link1 Link2

$vmlist =get-vm | sort name 

forEach ($vm in $vmList) {
       $vmName=$vm | %{(Get-View $_.Id).Name}
       $vmUuid=$vm | %{(Get-View $_.Id).config.UUID}
       echo "$vmName,$vmUuid" | Out-File -Append <path to csv>
}

Here is the sample output from my lab

AD,564dbb42-5c80-879e-889a-5058901c43de
AD01,564d441e-52e1-f65d-8320-41c739f9683e
ADS,564df36b-6d8f-f026-d3b3-56918e66810a
esx01,564d4cc9-767e-42ea-ae45-8dbdbfd55580
esx02,564da15e-9c66-4da2-26ca-912e2a25ed1b
esx03,564df470-444a-6c03-b881-c4b8a5bca030
esxlab1.vcpdcv.lab,564d81df-bfc8-0f68-4a6b-8f26ec0be899
esxlab2.vcpdcv.lab,564d8a89-9cb2-23b1-c307-a039a955c296
esxlab3.vcpdcv.lab,564df488-9e8a-9ef1-3878-c59d15b2a89c

Hope this was informative and useful. Thanks!

Part 17 – Running a Recovery

Finally, we have arrived to a point where we can run the Recovery. SRM migrates all the VMs in the recovery plan to the recovery site. SRM attempts to shutdown the VMs at the Protected Site. Not everyone in the organization are allowed to run recovery as it alters the entire environment in many cases and reversing things is time consuming and needs effort.

#run-recovery, #srm-6-1

Part 16 – Cleanup after testing a Recovery Plan

It is important to clean up after testing the recovery plan; failing to clean up the recovery plan bars us to test or run the recovery plan again. Cleanup does the following things.

• Powers off the recovered VMs
• Replaces recovered VMs with place holder VMs and preservs the configuration information.
• Cleansup the replicated storage snapshots that the recovered VMs used during the test.

#cleanup, #test-recovery-plan

Part 15 – Test a Recovery Plan

Now that we have the recovery plan configured according to our needs, its time we test the recovery plan. When you test a recovery plan, SRM uses the test network to test the VMs in the plan and also no VMs on the Protected Site are powered off unlike when the actual recovery plan is run. Remember that, if a recovery plan requires to suspend or power off any VM on the recovery site, testing the recovery plan also suspends the VMs and no other changes are made to the production environment during the test recovery.

#srm-6-1, #test-recovery-plan, #timeout, #vmware-tools-heartbeat

Part 14 – Configuring a Recovery Plan

Creating a Recovery Plan is a very good thing but how do make it work the way we want it to? Well, every Recovery Plan has to be configured for the fact that, virtual machines depend on other machines in many ways, for example, the DB server has to powered on first for the app server and web server to connect to it and in some cases, we want to run some start-up scripts so that the machine can become available to users. All such things have to be fed to the recovery plan. So let’s see how that is done.

#ip-costomization, #recovery-plan, #recovery-steps, #srm-6-1

Part 13 – Creating a Recovery Plan

We have come a long way in settings up things and working on SRM related stuff. We have almost reached the last steps in enabling a Disaster Recovery Solution for VMs. Now that we have Protection Groups created and verified all the required VMs are being replicated as expected, our next step would be to create a Recovery Plan so that the Protection Groups can be made use of. A Recovery Plan controls every step of the recovery process, including the order in which the VMs have to be powered on, networks to be used, and other stuff related to virtual machines to be protected.

#recovery-plan, #srm-6-1

Part 12 – Creating Protection Group

As discussed in our previous post, Protection Groups are a collection of virtual machines that are protected together. Protection Groups can be created based on the type of replication they have ie whether it is Array Based or vSphere Replication or Storage Policy Based. A Recovery Plan can include Protection Groups from Array Based Replication or vSphere Based Replication but a Storage Policy Based Protection Group cannot be included in the same Recovery Plan as Array or vSphere Replication Protection Group.

#placeholder-virtual-machines, #protection-groups, #srm-6-1

Part 10 – Understanding Inventory Mappings and Other Terms.

Mapping is important for the SRM to work correctly as it specifies how VM resources at Protected Site are to be mapped to the resources at the Recovery Site. During the recovery, SRM uses the resources specified in the mapping at the recovery site to start the virtual machines. Bi-directional protection is also possible in the SRM and for this reverse mapping has to be configured.

#array-based-replication-protection-groups, #consistency-groups, #inventory-mappings, #placeholder-datastores, #placeholder-virtual-machines, #protection-groups, #site-wide, #storage-policy-based-protection-groups, #vsphere-replication-based-protection-groups

Part 9 – Configure and Rescan SRM Array Managers

Let’s get moving with the SRM stuff. The very first thing that needs to be done after pairing the sites and installing SRAs is to configure SRM Array Managers which helps the SRM to discover the replicated devices, compute datastore groups and initiate storage operations. This is a one time operation unless there is a change in the credentials.

#array-manager, #pair-array-manager, #replication-direction, #srm-6-1

Part 8 – Pairing Sites and Installing HP-SRA on SRM 6

Hey there! We now have our storage things set up and its time to work on webclient again. As we are done with all the installations, we now need to pair the sites in the SRM so that the SRM is aware of which site is protected and which is recovery. So let’s begin pairing the Sites.

#hp-sra, #site-pairing, #sra-command-queryinfo-didnt-return-a-response, #srm-6, #storage-replication-adapter