This article explains briefly how to move a native CentOS-4 installation into a paravirtualized Xen3 guest. The Xen3 host may be any OS - even NetBSD - but this HowTo assumes it to be a redhatish linux like RHEL5, CentOS-5 or FC6

Contents

Disclaimer
Migration
Notes
Document Roadmap

Disclaimer

This HowTo is experimental and not well tested, it just works for me. So if this breaks your system, don't blame, but mail me: centos-mail at nils dot toedtmann dot net.

Network setup, NFS root or X are beyond the scope of this article, RTFM:

Migration

Get prepared

If CentOS-4.5 is released: upgrade!

CentOS-4.4 is not really prepared for being a DomU. So if you find out that CentOS-4.5 is already released while you read this upgrade your CentOS-4. As you'll see, that makes things easier.

Domain-0

Install a RH based Domain-0 OS (FC6/RHEL5/CentOS-5 with the software group Virtualization) on the hardware containing the CentOS-4 you intend to virtualize. Boot that Dom0 and make sure the service xend is running.

Domain-U alignment

Blockdevice mapping

Let's assume your CentOS-4 installation ist called C4 and is installed on some blockdevices like /dev/mapper/VolGroup00-C4. You need to make some changes to the C4-tree, so mount it somewhere:

[root@dom0 ~]# mount /dev/mapper/VolGroup00-C4 /mnt
[root@dom0 ~]# mount /dev/whatever             /mnt/whereever
[root@dom0 ~]# mount -o bind /proc             /mnt/proc
[root@dom0 ~]# mount -o sys  /sys              /mnt/sys

If your C4 had a separate /boot partition ignore it, you can proceed with an empty /boot directory (you could even remove all kernel and grub rpm packages). Keep it if you plan to dual-boot the C4 natively and virtualized, otherwise abandon it. If you are unsure, copy it's contents:

[root@dom0 ~]# mount  /dev/hda1 /mnt2
[root@dom0 ~]# cp -a  /mnt2/* /mnt/boot
[root@dom0 ~]# umount /mnt2

Map your (old) native C4 blockdevices to (new) virtual SCSI-devices. Example:

mount point	native	virtual
/boot	/dev/hda1	obsolete
/	/dev/mapper/VolGroup00-C4	/dev/sda1
swap	/dev/mapper/VolGroup00-C4_swap	/dev/sda2
/whereever	/dev/whatever	/dev/sda3

Change C4's filesystem table /mnt/etc/fstab accordingly:

...
#obsolete   /boot       ext3    defaults   1 1
/dev/sda1   /           ext3    defaults   1 1
/dev/sda2   swap        swap    defaults   0 0
/dev/sda3   /whereever  ext3    defaults   1 1
...

DomU kernel

Read the notes on kernels below!

CentOS-4.5 only: Should be easy

[root@dom0 ~]# chroot /mnt yum install kernel-xenU

CentOS-4.4 only: fetch one of the experimental DomU kernels for RHEL4 or CentOS4, e.g.

[root@dom0 ~]# wget -P /mnt/tmp/ \
 http://people.redhat.com/~jbaron/rhel4/RPMS.kernel/kernel-xenU-2.6.9-44.EL.i686.rpm
[root@dom0 ~]# chroot /mnt rpm -ivh /tmp/kernel-xenU-2.6.9-44.EL.i686.rpm

In any case you should generate an initial ramdisk - the rpm installation likes to forget xenblk.ko:

[root@dom0 ~]# chroot /mnt mkinitrd --preload xenblk \
               /boot/myinitrd-2.6.9-44.ELxenU.img 2.6.9-44.ELxenU

Bootloader

The grub-alike bootloader PyGrub will reside outside the DomU, it's part of Dom0's Xen3 distribution. It will read DomU's /mnt/boot/grub/menu.lst, so create it according to the DomU kernel you installed (mkdir -p /mnt/boot/grub if /mnt/boot is empty):

default         0
timeout         3
title           CentOS 4 (2.6.9-44.ELxenU)
  kernel        /boot/vmlinuz-2.6.9-44.ELxenU root=/dev/sda1 ro 3
  initrd        /boot/myinitrd-2.6.9-44.ELxenU.img

glibc

CentOS-4.4 on i386/x86_32 only: Replace some libraries with their Xen-friendly versions from Xensource. Find out which libs you have installed and their arch-flavour

[root@dom0 ~]# chroot /mnt rpm -q --qf "%{name} %{arch}\n" \
  glibc{{,-debuginfo}{,-common},-devel,-headers,-profile,-utils} nptl-devel nscd

and replace them. For example

[root@dom0 ~]# wget -P /mnt/tmp/ \
http://xenbits2.xensource.com/glibcs/rhel44/RPMS/i686/glibc-2.3.4-2.25.xs0.4.3.56.i686.rpm
[root@dom0 ~]# wget -P /mnt/tmp/ \
http://xenbits2.xensource.com/glibcs/rhel44/RPMS/i386/glibc-common-2.3.4-2.25.xs0.4.3.56.i386.rpm 
 ...
[root@dom0 ~]# chroot /mnt rpm -Fvh /tmp/*rpm

Warning: Unfortunately, Xensource does not provide a Yum repository. Do if there are CentOS updates for glibc, a yum update will overwirte the Xen friendly versions, resulting in a slower system. Some software (like OpenLDAP, Cyrus-IMAPd) may even break!

Console

(Maybe applicable to CentOS-4.4 only) The Xen-Console will work on the DomU's /dev/console, so you should have a getty on it in /mnt/etc/inittab:

...
# Run gettys in standard runlevels
0:2345:respawn:/sbin/mingetty --noclear console
1:2345:respawn:/sbin/mingetty tty1
...

Networking

The driver for the DomU's ethX will be xennet.ko, so adjust your DomU's /mnt/etc/modprobe.conf:

alias eth0 xennet

(Maybe CentOS-4.4 only) Some DomU-kernels (like the above experimental ones) have a bug in their xennet driver and mess up TCP and UDP checksums. You can avoid running into that by disabling TX checksumming on the ethX devices. Create an executable shell script /mnt/sbin/ifup-local:

ethtool -K ${1} tx off

Finally

You may chroot in your C4 and do some personal custumization (e.g. setup networking in /etc/sysconfig/network{,-scripts/ifcfg-eth*}):

[root@dom0 ~]# chroot /mnt
$ ... do some interesting stuff here ...
$ exit
[root@dom0 ~]#

If your are done, unmount your C4:

[root@dom0 ~]# umount /mnt/{sys,proc,whereever,}

Boot it!

Create a Xen configuration file /etc/xen/C4

name       = 'C4'
bootloader = '/usr/bin/pygrub'
memory     = 128
vif        = [ '' ]
disk       = [ 'phy:mapper/VolGroup00-C4,sda1,w',
               'phy:mapper/VolGroup00-C4_swap,sda2,w',
               'phy:whatever,sda3,w' ]

and boot it:

[root@dom0 ~]# xm create -c C4

If your DomU uses SELinux you have to restore the contexts of the files you replaced or created since the last (native) boot:

[root@C4 ~]# setenforce 0
[root@C4 ~]# restorecon -v -R /etc
[root@C4 ~]# restorecon -v -R /boot
[root@C4 ~]# restorecon -v /sbin/ifup-local
[root@C4 ~]# fixfiles -R kernel-xenU,glibc,glibc-common,... relabel
[root@C4 ~]# reboot

If you don't remember which files have broken context let instead CentOS relabel the hole system:

[root@C4 ~]# touch /.autorelabel
[root@C4 ~]# reboot

Notes

Kernel & hypervisor

Your paravirtualized DomU-Kernel must be able to run on your Xen3 hypervisor. The hypervisor ABI for Xen3 is not really stable yet, but since Xen-3.0.2 there is a backward compatibility mode most hypervisors and kernels know. But there are still three incompatible types of hypervisors, and a paravirtualized DomU-kernel must match this type:

x86_32: 32bit, non PAE
x86_32p: 32bit, PAE
x86_64: 64bit

Look at the first value (xen-3.0-*) in Dom0's /sys/hypervisor/properties/capabilities to find out the type of your Hypervisor/Dom0 combo. The binary 32bit hypervisors and kernels you get from Fedora/RH/CentOS have PAE, those from Cambridge or XenSource have not. (sure? verify!)

Kernel & bootloader

The advantage of using PyGrub is that kernel, LKMs and initrd are completly maintained inside the DomU. Upgrading the kernel just needs a rpm -i or (CentOS-4.5) yum update in DomU.

If you want move the control over kernel selection to Dom0, copy kernel and initrd to Dom0

[root@dom0 ~]# cp /mnt/boot/vmlinuz-2.6.9-44.ELxenU       /boot
[root@dom0 ~]# cp /mnt/boot/myinitrd-2.6.9-44.ELxenU.img  /boot

and replace the bootloader option in the Xen config file /etc/xen/C4 with

kernel  = "/boot/vmlinuz-2.6.9-44.ELxenU"
root    = "/dev/sda1 ro"
extra   = "3"
ramdisk = "/boot/myinitrd-2.6.9-44.ELxenU.img"

But having kernel and initrd in Dom0 but LKMs in DomU is not very upgrade friendly.

If you run a bunch of lean "appliance"-DomUs which need the same set of kernel features, and if Dom0 does all the GNBD/RAID/LVM/DM work (see below, Blockdevices) you should consider building a static kernel with builtin xenblk and xennet: No initrd, no LKMs, kernel for all DomUs is kept in one place.

Libs

(to be done)

Blockdevices

As blockdevices are often layered, you have multiple options how to map them into the DomU. Let's assume there are two harddrives (hda, hdc), two partitions bundled to a Raid1 (hda3+hdc4=md0) which contains the DomU's rootfs on a LVM-volume (VolGroup00-C4).

Map the harddrives - all layering done in DomU:

disk = [ 'phy:hda,sda,w', 'phy:hdc,sdb,w' ]

Map the partitions - partition table parsing in Dom0, RAID and LVM in DomU:

disk = [ 'phy:hda3,sda1,w', 'phy:hdc4,sdb1,w' ]

Map the raid - Raid in Dom0, LVM in DomU:

disk = [ 'phy:md0,sda1,w' ]

Map the logical LVM volume - all layering is done in Dom0:

disk = [ 'phy:mapper/VolGroup00-C4,sda1,w' ]

I prefer the latter, doing all GNBD/LVM/Raid/DM stuff in Domain-0. Keep in mind that a blockdevice mapped to a DomU cannot be used by any other domain!

Networking

If you don't like xend messing up your Dom0's network settings, edit Dom0's /etc/xen/xend-config.sxp, the key options are network-script and vif-script.

I prefer setting up my own bridges using /etc/sysconfig/network-scripts/brX, adjusting /etc/xen/xend-config.sxp to a mixed "routed bridges" setup:

(network-script network-route)
(vif-script vif-bridge)

and putting

vif = [ 'bridge=brX' ]

into xen's DomU-configs. So the guests get bundled via bridges, and the Dom0 routes these bridges to the outside world.

X

There are several options:

X forwarding

You can run X apps via SSH X-forwarding:

[alice@client ~]# ssh -X domU-hostname xterm

If that's enough for you, there is no need for a X or display manager running on the DomU.

Virtual framebuffer

(to be done)

GDM + VNC

(to be done)

GDM + RDP

(to be done)

GDM + remote X

(to be done)

Xvnc

Assume DomU's user alice wants to run a Xvnc server. She should set herself a VNC password

[alice@domU ~]# vncpasswd

DomU's admin configures that VNC server in /etc/sysconfig/vncservers

VNCSERVERS="1:alice"
VNCSERVERARGS[1]="-geometry 800x600"

and starts it

[root@domU ~]# /etc/init.d/vncserver restart

Now Alice can connect from outside to her VNC server:

[alice@client ~]# vncviewer domU-hostname:5901

Document Roadmap

/dev/xvc0
Testing testing testing
Paravirtualization vs HVM mode
Check different kernel/hypervisor combos
Test if SElinux really works
Some elaborated stuff on NPTL/TLS/glibc/db4
X

This is a read-only archived version of wiki.centos.org

Disclaimer

Migration

Get prepared

If CentOS-4.5 is released: upgrade!

Domain-0

Domain-U alignment

Blockdevice mapping

DomU kernel

Bootloader

glibc

Console

Networking

Finally

Boot it!

Notes

Kernel & hypervisor

Kernel & bootloader

Libs

Blockdevices

Networking

X

X forwarding

Virtual framebuffer

GDM + VNC

GDM + RDP

GDM + remote X

Xvnc

Document Roadmap