The problem:

The goal is to have a desktop system that has advanced graphics (single desktop, spanning dual DVI/LCD displays at 1600x1200 each) plus the ability to prototype new software/systems via Xen-based virtual machines. The challenge is to be able to run the NVIDIA driver on the main (Dom0) system. When searching the Internet for solutions, 3 main issues come up:

  1. Article is very (years) out of date.
  2. Article skips steps; making it difficult to follow.
  3. Article provides a convoluted/complex solution (e.g., building a custom kernel from scratch).

Most people want a convenient, useful, system and don't wish to deal with solutions such as dual-booting-- or a system that is unstable. In the author's experience, the (proprietary) NVIDIA driver supports this graphics configuration well. Not using the NVIDIA driver gives significantly poorer performance, so this wasn't an option. As of this writing, the NVIDIA driver package does NOT support Xen-enabled kernels. When one goes to install the package, it states this and exits.

Hardware:

Pre-configuration state:

  1. Built base system from CentOS 5.4 x86_64 distribution.
  2. Select Virtualization group during install, so system comes up with a Xen-enabled kernel by default.

  3. This article won't cover the basic system install or working with Xen Virtual Machines.
  4. Please make sure you reserve enough space on /boot and / to allow for multiple (minimum 4) kernels/kernel-devel sets.
  5. It is recommended to run the following steps from a terminal/SSH session from another machine-- makes it easier to cut-n-paste commands.
  6. All of these steps are run as the user "root".

The How-To Cookbook:

Apply all updates:

yum -y update
yum clean all
reboot

Remove outdated kernels:

rpm -qa | grep kernel

Keep only the latest kernel-xen, kernel-xen-devel (this is an example of removing older kernels on the system):

rpm -e kernel-xen-2.6.18-164.el5 kernel-xen-devel-2.6.18-164.el5

So, you should be left with:

[root@localhost ~]# rpm -qa | grep kernel
kernel-xen-devel-2.6.18-164.9.1.el5
kernel-xen-2.6.18-164.9.1.el5
kernel-headers-2.6.18-164.9.1.el5

Install non-Xen kernel and kernel-devel:

yum -y install kernel kernel-devel

So, now we have matching Xen and non-Xen kernels available:

[root@localhost ~]# rpm -qa | grep kernel
kernel-xen-devel-2.6.18-164.9.1.el5
kernel-devel-2.6.18-164.9.1.el5
kernel-2.6.18-164.9.1.el5
kernel-xen-2.6.18-164.9.1.el5
kernel-headers-2.6.18-164.9.1.el5

This method has been successfully used with these two generations of the NVIDIA drivers:

Since we are currently running a Xen kernel, let's build the NVIDIA kernel module first:

[root@localhost ~]# uname -r
2.6.18-164.9.1.el5xen

Bring down the NVIDIA-Linux-x86_64-190.53-pkg2.run package however convenient (from: http://www.nvidia.com).

Move the file to /tmp.

cd /tmp 
chmod +x NVIDIA-Linux-x86_64-190.53-pkg2.run
./NVIDIA-Linux-x86_64-190.53-pkg2.run --extract-only
cd NVIDIA-Linux-x86_64-190.53-pkg2/usr/src/nv
IGNORE_XEN_PRESENCE=y make SYSSRC=/lib/modules/`uname -r`/build module

You will get lots of warnings, but in the end:

NVIDIA: left KBUILD.

Now, let's install the new module into the Xen-kernel library:

mkdir -p /lib/modules/`uname -r`/kernel/drivers/video/nvidia/
cp -i nvidia.ko /lib/modules/`uname -r`/kernel/drivers/video/nvidia/
depmod -a
modprobe nvidia

At this point, we've demonstrated that we can load the nvidia module into the running Xen kernel. None of the rest of the NVIDIA software has been loaded, so things won't work correctly (yet).

This step is optional, but will keep the system from barfing on a non-working X-configuration. This makes the system boot into non-graphical, multi-user mode (of course, you can do this with your favorite text editor instead):

sed -i 's/^id:5:/id:3:/' /etc/inittab

Reboot the system now, but when you see the Grub screen, stop the boot (hit <space>) and select the NON-Xen kernel.

reboot

After the reboot, we should be running the non-Xen kernel:

[root@localhost ~]# uname -r
2.6.18-164.9.1.el5

OK, let's set up the NVIDIA sofware the normal way:

cd /tmp 
./NVIDIA-Linux-x86_64-190.53-pkg2.run
   <Accept the license>
   <Yes- Install NVIDIA's 32-bit compatibility OpenGL libraries>
   <Yes- automatically update your X configuration file>
   <OK- Successfully updated.>

At this point, we have the NVIDIA software fully installed/configured for the NON-Xen environment AND we also have the NVIDIA Xen kernel module built and loaded into the (Xen kernel) module library path.

Let's go for broke (set to auto-start in graphical mode):

sed -i 's/^id:3:/id:5:/' /etc/inittab

Now, on this boot, let it boot the Xen-enabled kernel (should still be the default).

reboot

You should see the NVIDIA splash screen as the system boots. A quick check to see that the driver is loaded:

[root@localhost ~]# dmesg | grep -i nvidia
nvidia: module license 'NVIDIA' taints kernel.
NVRM: loading NVIDIA UNIX x86_64 Kernel Module  190.53  Wed Dec  9 15:29:46 PST 2009

...and that we're running the Xen kernel:

[root@localhost ~]# uname -r
2.6.18-164.9.1.el5xen

Quick check on graphics performance:

glxgears

(With this hardware combination, expect about 15,000 FPS)

For the purposes of getting this running, a 1280 x 1024 DVI-connected flat panel was used with a very simple graphics configuration. Now that everything is installed, one can modify the X-configuration by using:

/usr/bin/nvidia-settings

This whole process must be repeated any time a new kernel is loaded on the system. This whole process takes around 10-15 minutes, once you have done this once or twice.

UPDATE (9 Aug 2012): Moved to different hardware (Dell Optiplex 780 with GeForce 9300 GE graphics card). This requires a newer NVIDIA driver package:

NVIDIA-Linux-x86_64-275.09.07.run

The process changed, but very slightly. Instead of:

cd NVIDIA-Linux-x86_64-190.53-pkg2/usr/src/nv

... the directory changes to:

cd NVIDIA-Linux-x86_64-275.09.07/kernel

This process has NOT been tested yet on CentOS 6, but works for CentOS 5.8 (all updates as of above date).

Page comments/suggestions: CrisRhea

HowTos/Xen/NvidiaWithXen (last edited 2012-08-09 18:47:59 by AlanBartlett)