Xen4 xl CLI Tools CentOS 6 and 7

Xen Basics

Once you have set up a basic install Xen4QuickStart and have bridging ( CentOS6 or CentOS7) (the default bridge name is xenbr0 .. but you can do anything you want), you are ready to do some things with xen. One way is to use the normal VM tools in CentOS via libvirt (see Xen4Libvirt). But the built in libvirt is not extremely xen friendly and it limits what you can do to only the things virsh, virt-manager and virt-install understand. This is not very robust with respect to Xen.

Another option is to use the included xl program that comes with the xen install (xm is no longer supported on Xen4CentOS versions).

This page is a basic example for creating a CentOS-6 and CentOS-7 DomU on a CentOS-6 Dom0 machine using xl.


Use the xl info command to get some information about your install. Here is an example and the result:

[root@c6-xen-dom0 ~]# xl info

host                   : c6-xen-dom0
release                : 4.9.13-22.el6.x86_64
version                : #1 SMP Sun Feb 26 22:18:35 UTC 2017
machine                : x86_64
nr_cpus                : 6
max_cpu_id             : 5
nr_nodes               : 1
cores_per_socket       : 3
threads_per_core       : 2
cpu_mhz                : 3292
hw_caps                : 178bf3ff:2fd3fbff:00000000:00001700:16982203:00000000:01c9bfff:00000000
virt_caps              : hvm
total_memory           : 16340
free_memory            : 15133
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 6
xen_extra              : .3-8.el6
xen_version            : 4.6.3-8.el6
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : Wed Feb 15 02:12:22 2017 -0600 git:35503e1-dirty
xen_commandline        : dom0_mem=1024M,max:1024M cpuinfo com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all
cc_compiler            : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Tue Feb 28 14:18:26 UTC 2017
xend_config_format     : 4

In this example, we will assume a bridge name of xenbr0. All the VMs will use this bridge. Here are the config files being used in this setup on CentOS-6:

[root@c6-xen-dom0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 

[root@c6-xen-dom0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-xenbr0 

Here is what it looks like on the example machine with IPv4 only:

[root@c6-xen-dom0 ~]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr D0:50:99:62:6C:43  
          RX packets:2206 errors:0 dropped:0 overruns:0 frame:0
          TX packets:714 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:208511 (203.6 KiB)  TX bytes:558027 (544.9 KiB)

lo        Link encap:Local Loopback  
          inet addr:  Mask:
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

xenbr0    Link encap:Ethernet  HWaddr D0:50:99:62:6C:43  
          inet addr:  Bcast:  Mask:
          RX packets:1515 errors:0 dropped:0 overruns:0 frame:0
          TX packets:716 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:102780 (100.3 KiB)  TX bytes:558321 (545.2 KiB)

You will also need some storage for your DomU VMs. You can also create image files if you prefer for your disks, but this example will use Logical Volumes (LVs) on LVM.

I personally like to use LVs for my machine drives because you can easily mount them and look at them from the Dom0 machine if there is a problem and it is easy to expand them later if need to. It is also used by default on almost all Linux distributions for the installed file system.

CentOS-6 and CentOS-7 will create a huge LV for /home by default. If you have a large drive it is easy to recover some space there to use LVs for xen drives. You can also use a second disk drive and create a new Physical Volume and a new Volume Group for your LVs. Here is a really good reference for creating some space in your Volume Group for creating blank Logical Volumes (LVM Administration).

For this example, I am using the default VG that was installed by the CentOS installer for CentOS-6, I have some free space to create new logical volumes as shown here:

[root@c6-xen-dom0 ~]# vgdisplay
  --- Volume group ---
  VG Name               vg_c6xendom0
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.82 TiB
  PE Size               4.00 MiB
  Total PE              476806
  Alloc PE / Size       102006 / 398.46 GiB
  Free  PE / Size       374800 / 1.43 TiB
  VG UUID               vqWRX0-SZOy-pxRx-R3oN-O61F-vt1B-xLdgjT

So I have a VG (named vg_c6xendom0) that has 1.43 TB of free space.

I created two 20GB LVs for our installs the lvcreate command:

[root@c6-xen-dom0 ~]# lvcreate -L 20G -n c6-x8664-hvm vg_c6xendom0
  Logical volume "c6-x8664-hvm" created.

[root@c6-xen-dom0 ~]# lvcreate -L 20G -n c7-x8664-hvm vg_c6xendom0
  Logical volume "c7-x8664-hvm" created.

We can see the results with the lvdisplay command:

[root@c6-xen-dom0 ~]# lvdisplay

<other non xen stuff>

  --- Logical volume ---
  LV Path                /dev/vg_c6xendom0/c6-x8664-hvm
  LV Name                c6-x8664-hvm
  VG Name                vg_c6xendom0
  LV UUID                ojMKYF-UvDC-lq7i-C0HO-G1I2-qo5o-QocEHU
  LV Write Access        read/write
  LV Creation host, time c6-xen-dom0, 2017-03-07 23:50:31 -0600
  LV Status              available
  # open                 0
  LV Size                20.00 GiB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
  Block device           253:3
  --- Logical volume ---
  LV Path                /dev/vg_c6xendom0/c7-x8664-hvm
  LV Name                c7-x8664-hvm
  VG Name                vg_c6xendom0
  LV UUID                YJEQVD-t1K9-mM2a-XyKW-2M5G-EreW-1dgIUL
  LV Write Access        read/write
  LV Creation host, time c6-xen-dom0, 2017-03-07 23:54:38 -0600
  LV Status              available
  # open                 0
  LV Size                20.00 GiB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           253:4

We will be installing CentOS from ISOs, so I created a directory on the Dom0 machine called /opt/isos/ .. and downloaded CentOS-6.8-x86_64-minimal.iso and CentOS-7-x86_64-Minimal-1611.iso from CentOS-6 Mirrors and CentOS 7 Mirrors, which are the latest at the time of this article.

Types of VMs in Xen

There are 2 major VM types, PV (paravirtualized) and HVM (fully virtualized). There are also hybrid options. Here is am overview of Xen that discusses these: Xen Overview. In this simple example, we will create an HVM (fully virtualized) install, as that is the easiest type.

Using xl

Here is the documentation for xl from the Xen man pages : xl(1).

We will be creating a CentOS-6 and a CentOS-7 HVM.

First we need config files for the VMs .. one for each. There is an example HVM config file in the xen package called /etc/xen/xlexample.hvm. You can look at the file and at this HVM doc: xl.cfg(5)

Here are our config files .. I created a directory in /etc/xen/ called config.d where I keep config files.

First /etc/xen/config.d/c6-x8664.hvm.cfg

builder = "hvm"
name = "c6-x8664.hvm"
memory = 4096
vcpus = 2
vif = [ 'mac=00:16:3E:29:00:00,bridge=xenbr0' ]
disk = [ 'phy:/dev/vg_c6xendom0/c68-x8664-hvm,xvda,rw', 'file:/opt/isos/CentOS-6.8-x86_64-minimal.iso,xvdb:cdrom,r' ]
boot = "dc"
sdl = 0
vnc = 1
vnclisten  = ""
vncdisplay = 0
vncpasswd  = "supersecret"
videoram = 64

Also /etc/xen/config.d/c7-x8664.hvm.cfg

builder = "hvm"
name = "c7-x8664.hvm"
memory = 4096
vcpus = 2
vif = [ 'mac=00:16:3E:29:00:01,bridge=xenbr0' ]
disk = [ 'phy:/dev/vg_c6xendom0/c73-x8664-hvm,xvda,rw', 'file:/opt/isos/CentOS-7-x86_64-Minimal-1611.iso,xvdb:cdrom,r' ]
boot = "dc"
sdl = 0
vnc = 1
vnclisten  = ""
vncdisplay = 1
vncpasswd  = "supersecret"
videoram = 64

See the above link in xl.cfg(5) for what each option means, but some important points are:

We will just do a normal CentOS install of each version from the ISO.

xl create

The command to start the CentOS-6 VM is:

xl create /etc/xen/config.d/c6-x8664.hvm.cfg

Connect to the 5900 port on and do a normal install. I use this command to connect from a CentOS-7 workstation to do the install:

remote-viewer vnc://

(There are many other VNC clients out there, you can connect from a Windows or Mac machine, etc.)

You can start the CentOS-7 install with this command:

xl create /etc/xen/config.d/c7-x8664.hvm.cfg

You would connect to that install via:

remote-viewer vnc://

Once the installs are complete, you would edit the /etc/xen/config.d/c6-x8664.hvm.cfg (and c7-x8664.hvm.cfg) and change boot to boot="c", to boot from the disk and not the CD.


Using Para-Virtualization (PV) or PV-on-HVM (PVHVM) instead


PV-on-HVM is discussed here in detail (LINK), and actual examples here (LINK). Basically, all you need to do is to add xen_platform_pci=1 to your HVM config files above to use PVHVM.

Para-Virtualization (PV)

Para-Virtualization no longer works by default on the CentOS-6 or CentOS-7 kernels. You could use the Dom0 kernels in your DomU and turn on PV after the install by modifying your config file.

If you would like to try to do it, here are some notes:

1. You can NOT use the default install as the boot partition in Xen PV machines can not be the XFS file system .. the default used by CentOS. Manually setup a /boot partition as ext4 instead in the CentOS installer.

2. Do not use an LVM partition in the CentOS installer for that Boot Partition .. make it a normal partition instead.

3. If you have done those things, you can create a PV configuration file and boot the installed partition in PV Mode. Here is an example for the c7-x8664.hvm DomU above, named /etc/xen/config.d/c7-x8664.pv.cfg

bootloader = "/usr/lib/xen/bin/pygrub"
name = "c7-x8664.pv"
memory = 4096
vcpus = 2
vif = [ 'mac=00:16:3E:29:00:01,bridge=xenbr0' ]
disk = [ 'phy:/dev/vg_c6xendom0/c73-x8664-hvm,xvda,rw' ]
vfb = [ 'type=vnc,vncdisplay=1,vncpasswd=supersecret' ]

If you have followed all the rules (the xen dom0 kernel replacing the CentOS-7 installed kernel, no xfs file system on /boot, no LVM on /boot, etc.) .. then that machine will boot in PV mode. Obviously, since the DISK in this case is shared, you can run the HVM and PV instances at the SAME time. You can run either of them individually though.

To start your PV DomU, use this command:

xl create /etc/xen/config.d/c7-x8664.pv.cfg

HowTos/Xen/Xen4QuickStart/Xen4Cli (last edited 2018-01-06 12:39:14 by JohnnyHughes)