Steps To Take To Completely Restore XenServer From Backup

This week I had the chance to experience my first XenServer crash. And by crash I mean that suddenly my XenServer became unresponsive and I lost console connection, XenCenter would not connect anymore and I had no way to understand what was really going on until I connected a display to the server itself. It was then that I realized that my XenServer was going thorugh an infinite boot sequence.

Upon further inspection, I noticed that there was a kernel panic right before the screen with the XenServer logo, and the error message was this:

kernel panic-not syncing: VFS: unable to mount root fs on unknown block(0,0)

I did not find any working solution so maybe something got corrupted on the boot USB drive (I actually started noticing some slowdowns when SSH-ing to the machine right before the crash). I also tried installing XenServer again using the installation CD (and the same USB drive as the destination), but the process failed during the initial backup of the existing XenServer data on the USB stick during the backup of the /var directory. So probably something did indeed get corrupt on the boot drive.

I finally tried playing around with BIOS boot settings for a while but I had no success, so I brought things back to how they were before. The only thing left for me to do was to restore XenServer from backup.

Now, understanding backups in XenServer needs some time, as there is not just a single backup to be taken. Instead, there are three different backups you need to take in order to completely restore your machine to a point before the crash:

  • The Host backup: this is a backup of the host configuration and software
  • Backup of your virtual machines: self-explanatory
  • Metadata backup: this includes all information related to the virtual machines, how they relate to the different virtual disks, how the disks are named etc. This is not included in a simple VM backup!

Supposing you do have all of these (if you don’t, have a look at my previous articles on how to backup VM metadata and virtual machines; a host backup can be taken very easily in XenCenter, under Server > Backup), these are the steps to follow to completely restore XenServer after a crash like this.

How to restore XenServer from backup

  1. Install XenServer from the installation CD (downloads page)
  2. Boot into XenServer
  3. Restore the host backup: from XenCenter, go to Server and then select Restore From Backup
  4. Reconnect the previous storage repositories (they won’t attach to the new install automatically): the detailed instructions to do this are available from Citrix themselves
    1. At this point you will see the VM disks but the list of VMs in XenCenter will probably be empty. This is normal as you are missing the VM metadata which you are going to restore later.
  5. Apply all XenServer updates: after a host restore, previously installed updates must be applied again
  6. Restore virtual machine metadata. You have two options here:
    • If you had a metadata VDI on one of your storage repositories before the crash, you can easily restore metadata from the XenServer console (xsconsole):
      1. Select Backup, Restore and Update
      2. Select Restore Virtual Machine Metadata
      3. Enter your username and password
      4. Select the storage repository that is holding the metadata VDI
      5. When the VDI is found, you will be asked if you wish to restore metadata
    • If you are taking VM metadata using my script, the VDI method won’t work, since you are saving metadata in a different location. In this case, you need to use the xe pool-restore-database command:
      1. Run
        xe pool-restore-database file-name=<metadata_backup_file> --force
      2. If you don’t specify the force option, the system will throw a warning and not proceed with the restore
      3. The server will automatically reboot after a metadata restore
      4. After the reboot, you should see the list of your VMs with all the correct disk mappings
  7. Restore miscellaneous files: if you had custom scripts, custom crontab entries and the like, make sure to restore those as well, as apparently they don’t make their way into a host backup.

This completes the list. Overall, I found that if you have all your backups in place, restoring a XenServer instance after a complete crash was not too painful. Hopefully I won’t have to deal with events like this too often, but I will definitely make sure I have a few spare USB drives from now on.

2 Comments

  1. I’m planning to update my XenServer 6.0 to 6.5 SP1 and then to 7.1. I want to backup entire XenServer so that in case of failure I could restore my whole server.

    Is there anyway to Backup entire Sever not just VMs.

Leave a Reply

© 2017 Daniel's TechBlog

Theme by Anders NorénUp ↑

%d bloggers like this: