Shutting down a VM Machine cleanly

I've been looking for a way of cloning a Virtual Application Server for disaster recovery, where the machine data files are on a local datastore and attaching the disk again into another datastore which is sat on our LeftHand iSCSI SAN for disaster recovery.  This way any other virtual host could start the host up and after some IP changes on the VM, we could be up and running again in around 15 minutes.

I want to:

• Shut the VM down CLEANLY (not just equivalent of holding the OFF button)
• Clone the disk
• Start the VM up
• All automated (I like my sleep)

This will be done as a WINDOWS scheduled task rather than a CRON job as windows scheduler in 2008 has many more options than CRON in ESXi can offer.  If you just want a simple weekly/monthyly copy CRON is probably the way forward.

The VM Host I am doing this on is not in vCenter, so I can't do any clever stuff that vCenter offers. 

Cloning the machine

Prerequisites:

• The virtual machine must be off
• There must be enough space on the destination machine
• You must be able to get to a SSH shell (Instructions here - will be re-written as v4.1 is much simpler)

I came across this website which had exactly what I wanted in the way of backing up and in the scenario I wanted.

Steps.
1. Logon to SSH
2. Make a new folder for the cloned disk to go to
3. Clone the disk

1. I used PuTTY to connect to the VM Host, but you can choose any SSH Client.

• Enter the IP address and click Enter

• Enter your root username and password.  You are now at a command prompt

2. As I'd already connected added an additional iSCSI disk to the VM I could browse it. (Click here to see how to add an iSCSI disk to VM Host *ADD)

 There is currently nothing on DataStore VM_TESSITURA_A, so I want to create a new folder to hold my cloned disk from RSCB015SV01C

mkdir /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C

I've differed the folder name to what I would usually call the machines by calling it FO_machinename.

3. Starting a disk clone is by using the command vmkfstools.  Effectively we are copying the vmdk file from one place to another

vmkfstools -i /vmfs/volumes/datastore1/RSCB015SV01C/RSCB015SV01C.vmdk /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C/RSCB015SV01C.vmdk

(Note: The above command is all on one line)

This will start the clone.

Destination disk format: VMFS zeroedthick

Cloning disk '/vmfs/volumes/datastore1/RSCB015SV01C/RSCB015SV01C.vmdk'...


Clone: 0% done.
Clone: 1% done.
Clone: 2% done.
etc...
Clone: 99% done.

Clone: 100% done.
 
Excelent!  Right - Arrow up and run the command again to check it works again.  Failed due to the destination already existing.  Not a problem - a simple remove command will do the trick:
 
rm /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C/RSCB015SV01C.vmdk
 
This failed... Why!  Checking the destination I was interested to find the cloned disk was now called machinename-flat.vmdk.  Interesting... I checked the source and sure enough the biggest file is machinename-flat.vmdk (the hard disk I just cloned) and there is also a tiny file called machinename.vmdk (around 500 bytes).  Viewing the datastore through the VM Host, it showed the files as machinename.vmdk
 
The command was changed to:
 
rm /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C/RSCB015SV01C-flat.vmdk
 
and I could then run the disk clone again successfully.
 
While the original VM was still off, on a second VM Host, I attached the storage VM_TESSITURA_A and created a new machine (*Add Link) and attached the cloned disk I just created.  Pressed the Play button and away we go - Windows starts up, adds new drivers & wants a reboot, then is stays on fine.
 
Shutting down the VM

I now needed a way of shutting the VM's cleanly.  I came across this command:
 
vim-cmd vmsvc/power.shutdown
 
which requires the machine number suffix which can be found by running
 
vim-cmd vmsvc/getallvms


Output:

Vmid Name File Guest OS Version Annotation
32 RSCB015SV01C [datastore1] RSCB015SV01C/RSCB015SV01C.vmx windows7Server64Guest vmx-07
80 RSCB015SV02C [datastore1] RSCB015SV02C/RSCB015SV02C.vmx windows7Server64Guest vmx-07
96 RSCB015SV03C [datastore1] RSCB015SV03C/RSCB015SV03C.vmx windows7Server64Guest vmx-07

So, RSCB015SV01C id is 32, so the command is:

vim-cmd vmsvc/power.shutdown 32
 
Yay.  It shutdown cleanly.  Next bit of thinking was start scripting everything together.  I came up with the individual steps:
 
vim-cmd vmsvc/power.on 32

mkdir /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C

rm /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C/RSCB015SV01C-flat.vmdk

 vmkfstools -i /vmfs/volumes/datastore1/RSCB015SV01C/RSCB015SV01C.vmdk /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C/RSCB015SV01C.vmdk


vim-cmd vmsvc/power.on 32
 
, which:

• Shutdown the VM
• Makes a folder on the destination (should always fail)
• Remove the old disk clone
• Reclone the disk
• Start the VM

Excellent!  Quick trip to google to find out how to send commands to a SSH shell via a script and it suggests PLINK part of PuTTY.  I put the above individual steps into the file "backup_RSCB015SV01C.txt" and ran the command from my windows server:
 
plink root@VM_HOST_IP -ssh -pw password -m .\backup_RSCB015SV01C.txt >> output.txt 2>>&1
 
It ran all the steps, just far too quick.  The issue shutdown worked and the remove the old clone worked, but all the other steps failed as the machine was still shutting down when they were run.  Time for a delay of some sort!  Back to Google.
 
After much searching and trying to figure it out myself I stumbled across VMware: virtual machine shutdown -script for backup.  With my very limited knowledge of Linux scripting, it appeared to shutdown multiple machines based on their machine number.  Copy and Paste into a text file and uploading it to the server using the upload functionality in datastore put the file there and I had to do a chmod to allow execution:
 
chmod 700 ./vmshutdown.sh
 
Copy and Paste from the website to a text file produced a character at the end of each line that looked like ^H.  This was removed from each  line using vi editor as it kept erroring out the script.  Next to fail was the Array section.  I couldn't get this working so I adapted the script to something I knew how to do - command line arguments.  The only real changes I made were to remove the array, remove the array loop and the second shutdown statement.  Everything else as-is
 
The final script now looks like this (open in notepad)
 
The final backup_RSCB015SV01C.txt that gets called from windows now looks like:
 
/vmfs/volumes/datastore1/script/vmshutdown.sh 32

mkdir /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C
rm /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C/RSCB015SV01C-flat.vmdk
rm /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C/RSCB015SV01C.vmdk
vmkfstools -i /vmfs/volumes/datastore1/RSCB015SV01C/RSCB015SV01C.vmdk /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C/RSCB015SV01C.vmdk
sleep 60
vim-cmd vmsvc/power.on 32

As I won't be doing this for every VM we have - just a selected few, I'm happy enough to recreate this answer file each time

--Wayne

2011/03/11 addition:
To clone the disk to a thin disk, the switch -d thin needs adding between the source and destination

 vmkfstools -i /vmfs/volumes/datastore1/RSCB015SV01C/RSCB015SV01C.vmdk -d thin /vmfs/volumes/VM_TESSITURA_A/FO_RSCB015SV01C/RSCB015SV01C.vmdk