Backing up a running virtual machine

/!\ This article is work in progress. Please feed back any suggestions regarding the article or the technique.

Download the script

1. Introduction

This article explains how to backup a virtual hard disk to a remote location, even while it is use.

One advantage of running software in a virtual machine is that the entire disk can be backed up in one go, including Operating System, software, configuration files, registry, permissions, data and all. Re-establishing a system after a failure is therefore quicker and more reliable than re-installing software and restoring data.

The method employs LVM to take a snapshot of the guest disk and then uses rsync to update changes to a previous backup on a remote server. If there is a database server on the guest then it is flushed & locked at the point the snapshot is taken. This method came into use around 2006 following wider availability and awareness of virtualization software, processor enhancements, cheaper faster network bandwidth, and cheaper bigger disks.

Requirements of the backup process include

2. Prerequisites

2.1. LVM file system

The LVM (Logical Volume Manager) file system has the ability to create a snapshot of a disk partition which doesn't change while it is being backed up. The snapshot doesn't interfere with the running VM because it is instant.

To minimize the disk space required for the snapshot, store one virtual disk per logical volume. If the guests' disk activity is low and/or you have plenty of disk space then you can store multiple virtual disks per logical volume without the snapshot running out of space.

There are command line and graphical tools available to set up an LVM disk, but documenting them is beyond the scope of this article.

2.2. Virtual machine host

This script was originally written for KVM but there is no direct interaction with the virtualization software itself so it should be able to backup any guest stored on LVM. Similar techniques have been shown to work with other virtualization software such as Xen and VMware.

2.3. Virtual machine guest

All guest operating systems are supported.

If you want to minimize the amount of data transferred to the backup server, then move temporary files and any other files that don't need to be backed up to a second virtual disk.

For KVM, use the -hda and -hdb parameters to specify the disks on the command line:

/usr/bin/qemu-kvm -k en-gb -m 1024 -hda /media/vm01/vm01-hda.raw -hdb /media/archive/vm01-hdb.raw 

Note that the second disk does not need to be in the same logical volume as it does not need to be in the snapshot. If fact, excluding it reduces the space used by the snapshot and reduces the impact on performance of the guest during the backup.

The files that can be excluded from the backup depend on the guest configuration, but here are some suggestions:

On Linux

On Windows

2.4. Guest database access from the host

If the guest is running a database, then a database user account is required to flush and lock the database. It is not advisable to use root as the password must be stored in plain text in the backup script. Furthermore, the user must be granted permission to connect from another machine (the host).

It is advisable to create a dedicated database user account and grant it the minimum necessary permissions. This ensures that if the username and password fall into the wrong hands then they are of limited use.

create user 'backup' identified by 'password-goes-here';
grant lock tables, reload on *.* to 'backup';
flush privileges;

If there is a firewall on the guest, then the appropriate port must be open to allow the host to access the guest database. For mysql, the default port is 3306. The firewall can be configured to restrict access via that port to the host's IP address. The host's firewall typically blocks database access.

2.5. Backup server access from the host

Before the backup can run as a scheduled cron task, ssh access must be configured to connect to the backup server without prompting for a password. Rather than store the password in a file, the backup script uses keys. See the REMOTEKEY definition.

It is recommended that a dedicated account is created on the backup server that has just enough permissions to connect via ssh and access the backup storage. Also, a dedicated key is created for backup use only. Then if the host is compromised, the risk to the backup server is limited.

# Create an unencrypted private key that is used only for backups
ssh-keygen -t dsa -P "" -f ~/.ssh/backup -C "ssh key only for backups"
# Only root can access the key files
chmod 600 ~/.ssh/backup*
# Copy the public key to the backup server
scp ~/.ssh/backup.pub backup@192.168.1.1:
# login to the backup server and add the public key to the authorized keys
ssh 192.168.1.1 -l backup
(enter the password)
cat backup.pub >> .ssh/authorized_keys2
chmod 600 .ssh/authorized_keys2
rm backup.pub
exit
# Now you can login as backup without entering a password
ssh 192.168.1.1 -l backup -i  ~/.ssh/backup

If you are prompted for a password in that last step, then you may find this troubleshooting guide useful: http://sial.org/howto/openssh/publickey-auth/problems/

/!\ A private key granting access to the backup server is now stored in root's .ssh folder. It could be argued that the private key should be encrypted with a passphrase. The key can then be loaded by ssh-add to avoid ssh prompting for it. However, ssh-add needs to be re-run after a reboot. The risk of backups failing after a power outage is worse than the risk to the backup server's security, especially after measures have been taken to limit and mitigate the security risk at both ends of the connection. If you disagree, then use a passphrase to encrypt the key and remember to re-add the key after every reboot of the host.

If you want a better understanding of ssh key authentication, this is a good start: http://www.ibm.com/developerworks/library/l-keyc.html

3. Installation

3.1. Install ShellSql

If the guest is running a database then ShellSQL is used to flush and lock the database while the snapshot is taken. Add the RPMforge repository, which contains !ShellSQL, if you haven't already added it.

yum install shellsql

ShellSQL is used to maintain the connection to the guest's database, and consequently maintain the database lock, while the snapshot is taken. Without it, the lock is lost as soon as the SQL lock command is completed.

3.2. Install the backup script

Download the backup script and save it on the host. Edit the backup script, changing the values defined at the top of the file and the parameters of the function calls at the bottom of the script. Grant 'execute' permissions on the script

chmod +x backup.sh

Login as root and run the script to test it

./backup.sh

3.3. Schedule the script

Schedule the backup script to run at the required time and frequency.

Edit the crontab file via the command-line like so:

crontab -e

To run daily at midnight, the crontab entry would look like this.

#M H D M W
0 0 * * * /root/backup.sh > /dev/null 2>&1

4. Unanswered Questions

5. Options for improving Performance

This backup method is a resource hungry task and we are obliged to optimize it. However, there are so many variables affecting performance that methodical performance testing is the only way to answer these questions. Does anyone know a computing undergraduate looking for a project?

6. To Do

HowTos/BackupKVMGuest (last edited 2009-03-27 17:39:48 by JulianPrice)