Note: This is a development release, only meant for testing purposes. We do not recommend anyone deploy production systems using the content mentioned here.
Standard CentOS-6 does not have the capabilities to run Xen as a virtualisation host due to upstream vendor not shipping Xen in EL6. We are undertaking an effort to try and fix that, with the aim of providing people using Xen on CentOS-5 an upgrade path and also to provide people with an option should they not find KVM to be a suiteable Virtualisation technology in their workloads.
At this time, our efforts are to try and use as much of the content already in the distro, but also to bring in upstream traction from the xen.org project as well as use the new linux kernel content in the Linux-3.4 tree, which is the kernel.org longterm supported stable kernel. Hence the code in this repo is based on :
- Linux Kernel 3.4.x (kernel.org long term stable kernel)
Blktap 2.5 userspace tools (https://github.com/xen-org/blktap)
- Blktap 2.5 drivers in the Centos6 Xen 3.4.x dom0 kernel
- Libvirt 0.10.x with drivers for both xm/xend and xl/libxl toolstacks
Its important that you take on board the fact that when you are done with this setup, you will no longer be running the standard CentOS linux kernel. The kernel we provide has not been processed by our upstream vendor and has not been through their QA process, however we are still confident that it should generally work for most people. Note that even when the 3.4.x kernel seems more recent than the 2.6.32 kernel in EL6, it might still have older versions of some drivers, resulting in different behaviour between the stock el6 kernel and the 3.4 kernel used for dom0, so you might need to update certain device drivers manually.
Note that this is only targetting CentOS-6/x86_64. We are not attemping a Xen-4 build for CentOS-6/i386 or for CentOS-5 at this point.
Known limitations :
- No UEFI support (and thus no Secure Boot). Xen hypervisor 4.2 does have UEFI support, but the Linux 3.4 dom0 kernel doesn't have UEFI support yet.
Known problems :
kernel 3.4 is missing broadcom bnx2 NIC firmware file "bnx2/bnx2-mips-09-6.2.1b.fw" so for example Dell R510 server NICs fail to work out of the box. Note that this is an upstream kernel.org problem. In kernel dmesg you'll see an error like: "bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"". To workaround the problem do: "cd /lib/firmware/bnx2 && ln -s bnx2-mips-09-6.2.1a.fw bnx2-mips-09-6.2.1b.fw". After that "ifdown ethX && ifup ethX" should work.
- libvirt 0.10.x libxl driver does not currently work properly. If you want to use libvirt/virsh/virt-install/virt-manager you need to use the xm/xend toolstack, which has a working libvirt driver, not xl/libxl.
A repository with all the required code is available at : http://dev.centos.org/centos/6/xen-c6/ . To set this up on your machine :
cd /etc/yum.repos.d/ wget http://dev.centos.org/centos/6/xen-c6/xen-c6.repo yum repolist
Firstly, we need to install the newer kernel. To achieve that :
yum install kernel kernel-firmware
Ensure that the 3.4.x kernel and the corresponding kernel-firmware rpms are installed. And check that /etc/grub.conf is setup to boot this new kernel as default.
Second step is to install Xen :
yum install xen
Once that completes, check /boot/ for the presence of xen-4.2.1.gz and edit your /etc/grub.conf to add a stanza like :
title xen root (hd0,0) kernel /xen.gz dom0_mem=256M,max:256M loglvl=all guest_loglvl=all module /vmlinuz-3.4.24-4.el6.centos.alt.x86_64 ro root=/dev/mapper/vg_cs-lv_root rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_LVM_LV=vg_cs/lv_root rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto rd_NO_DM rd_LVM_LV=vg_cs/lv_swap module /initramfs-3.4.24-4.el6.centos.alt.x86_64.img
Ensure that the path to /xen, /vmlinux and /initramfs are correct ( based on the other entries already in /etc/grub.conf ). Also consider tweaking the 256M dom0_mem parameter to match your site requirements. In most cases, a minimal install CentOS-6, with Xen-4 will be fine with 256 Mbytes of RAM for dom0 (see Bug 6893, so dom0_mem=256M,max:512M would get rid of the error message in that bug).
You should now be able to reboot the machine. When the machine comes back, check that the new kernel is running and xen is active by executing :
uname -a ; xl info
Setting up serial console
title xen with serial console root (hd0,0) kernel /xen.gz dom0_mem=1024M,max:1024M loglvl=all guest_loglvl=all com1=115200,8n1 console=com1 module /vmlinuz-3.4.24-4.el6.centos.alt.x86_64 earlyprintk=xen console=hvc0 ro root=/dev/mapper/vg_cs-lv_root rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_LVM_LV=vg_cs/lv_root rd_NO_MD SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto rd_NO_DM rd_LVM_LV=vg_cs/lv_swap module /initramfs-3.4.24-4.el6.centos.alt.x86_64.img
We're setting up Xen hypervisor (xen.gz) to use and log to the serial port, and dom0 Linux kernel (vmlinuz) is configured to log to the serial console through Xen hypervisor (console=hvc0), so both the Xen hypervisor and dom0 Linux kernel console messages appear on the serial console. For more information about setting up Xen serial console with non-standard serial ports etc: http://wiki.xen.org/wiki/Xen_Serial_Console .
If you're experiencing trouble booting up the system please set up the serial console and capture the full boot logs for analysis. Most servers offer serial port in their management processor (iLO/iDRAC/IPMI/BMC), which you can actually use through the network (SOL = Serial Over LAN).
Note: see Bug 6893, so dom0_mem=1024M,max:1536M would get rid of the error message in the bug.
Installing a VM
At this point the libvirt tools do not work with the Xen-4.2 stack, so the distro virt tools might not work. The best way to install a new VM is to bootstrap using the pxe images. This process will work for both paravirt and full / hypervirt environments.
For CentOS-6 these images are located at http://mirror.centos.org/centos/6/os/x86_64/images/pxeboot/ and http://mirror.centos.org/centos/6/os/i686/images/pxeboot/ for 64bit and 32bit respectively. For CentOS-5 these images are located at http://mirror.centos.org/centos/5/os/x86_64/images/xen/ and http://mirror.centos.org/centos/5/os/i386/images/xen/ for 64bit and 32 bit respectively.
- Download the pxe images for the distro and arch you require
- Do any network setup needed ( you will need a bridge setup, refer to the example files in /etc/xen/ for more info )
- Setup storage you are going to use for the VM. eg, if using lvm and the VM is going to have a lv backing store : lvcreate -L 20G -n lv_dom1 /dev/vg_storage
- Create a basic config file that looks something like this
name = "domain-1" memory = "768" kernel = "/tmp/vmlinuz" ramdisk = "/tmp/initrd.img" disk = [ 'tap2:aio:/var/lib/xen/images/disk.0,xvda,w', ] vif = [ 'bridge=virbr0', ] vcpus=1 on_reboot = 'destroy' on_crash = 'destroy'
Note that the kernel and ramdisk lines point to the pxe boot images downloaded in step 2; if you have a kickstart file or want to send some specific parameters to the installer, you can use the 'extra' keyword in there as well.
You should now be able to 'xl create -c <config file>' and start the install process.
Once the install is done, you need to change the kernel line, remove the ramdisk link and add another extra line in the config file to fall back to the pv-grub process. These lines would now look something like this :
kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz" extra = "(hd0,0)/grub/menu.lst"
At this point you should be able to start the VM / DomU with 'xl create <configfile>' and check its running.