first look at CentOS 5 Xen

I have just installed a machine running CentOS 5 as a Xen server. I installed a full GUI environment on the dom0 so that GUI tools can be used for managing the virtual servers.

The first problem I had was selecting the "Installation source", it's described in the error message as an "Invalid PV media address" when you get it wrong which caused me a little confusion when installing it at 10PM. Then I had a few problems getting the syntax of a nfs://1.2.3.4:/directory URL correct. But these were trivial annoyances. It was a little annoying that my attempts to use a "file://" URL were rejected, I had hoped that it would just run exportfs to make the NFS export from the local machine (much faster than using an NFS server over the network which is what the current setup will lead people to do).

The first true deficiency I found with the tools is that it provides no way of creating filesystems on block devices. The process of allocating a block device or file from the Xen configuration tool is merely assigning a virtual block device to the Xen image - and only one such virtual block device is permitted. Then the CentOS 5 installation instance that runs under Xen will have to partition the disk (it doesn't support installing directly to an unpartitioned disk) which will make things painful when it comes time to resize the filesystems.

When running Debian Xen servers I do everything manually. A typical Debian Xen instance that I run will have a virtual block device /dev/hda for the root FS, /dev/hdb for swap, and /dev/hdc for /home. Then if I want to resize them I merely stop the Xen instance, run "e2fsck -f" on the filesystem followed by "resize2fs" and the LVM command "lvresize" (in the appropriate order depending on whether I am extending or reducing the filesystem).

Xen also supports creating a virtual partitioned disk. This means I could have /dev/lvm/xenroot, and /dev/lvm/xenswap, and /dev/lvm/xenhome appear in the domU as /dev/hda1, /dev/hda2, and /dev/hda3. This means that I could have a single virtual disk that allows the partitions to be independently resized when the domU in question is not running. I have not tried using this feature as it doesn't suit my usage patterns. But it's interesting and unfortunate that the GUI tools which are part of CentOS don't support it.

When I finally got to run the install process it had a virtual graphics environment (which is good) but unfortunately it suffered badly from the two-mouse-cursor problem with different accellerations used for both cursors so the difference in position of the two cursors varied in different parts of the screen. This was rather surprising as the dom0 had a default GNOME install.

Xen and eth device renaming

Recently I rebooted one of my Debian Xen servers and suddenly all the Ethernet devices which used to be eth0 in the domU's became eth1.

vif = [ '', 'bridge=xenbr1' ]

I used to have the above as the interface definition and for domU's that had only a single interface that worked well (if there is only one interface then it should be eth0). However in a recent etch update this changed, so I had to use ifrename as documented in my previous blog post. It's annoying when things break because a reasonable assumption which previously worked suddenly stops working.

Even if the bug in question (if it is regarded as a bug) is fixed I'll keep using ifrename, it doesn't do any harm.

Xephyr

As part of my work on Xen I've been playing with Xephyr (a replacement for Xnest). My plan is to use Xen instances for running different versions of desktop environments. You can't just ssh -X to a Xen image and run things. One problem is that some programs such as Firefox do strange things to try and ensure that you only have one instance running. Another problem is with security, the X11 security extensions don't seem to do much good. A quick test indicates that a ssh -X session can't copy the window contents of a ssh -Y session, but can copy the contents of all windows run in the KDE environment. So this extension to X (and the matching ssh support) seem to do little good.

One thing I want to do is to have a Xen image for running Firefox with risky extenstions such as Flash and keep it separate from my main desktop for security and managability.

Xephyr :1 -auth ~/.Xauth-Xephyr -reset -terminate -screen 1280x1024

My plan is to use a command such as the above to run the virtual screen. That means to have a screen resolution of 1280x1024, to terminate the X server when the last client exits (both the -reset and the -terminate options are required for this), to be display :1 and listen with TCP (the default), and to use an authority file named ~/.Xauth-Xephyr.

xauth generate :1 .

The first problem is how to generate the auth file, the xauth utility is documented as doing it via the above command. But this really connects to a running X server and copies the auth data from it.

The solution (as pointed out to me by Dr. Brian May) is to be found in the startx script which solves this problem. The way to do it is to use the add :1 . $COOKIE command in xauth to create the auth file used by the X server, and to generate the cookie with the mcookie program.

In ~/.ssh/config:
Host server
SendEnv DISPLAY

In /etc/ssh/sshd_config:
AcceptEnv DISPLAY

The next requirement is to tell the remote machine (which incidentally doesn't need to be a Xen virtual machine, it can be any untrusted host that contains X applications you want to run) which display to use. The first thing to do is to ssh to the machine in question and run the xauth program to add the same cookie as is used for the X server. Then the DISPLAY environment variable can be sent across the link by setting the ~/.ssh/config file at the client end to have the above settings (where server is the name of the host we will connect to via SSH) and in the sshd_config file on the server have the line AcceptEnv DISPLAY to accept the DISPLAY environment variable. It would have been a little easier to configure if I had added the auth entry to the main ~/.Xauthority file and used the command DISPLAY=:1 ssh -X server, this would be the desired configuration when operating over an untrusted network. But when talking to a local Xen instance it gives better performance to not encrypt the X data.

The following script will generate an xauth entry, run a 1280x1024 resolution Xephyr session, and connect to the root account on machine server and run the twm window manager. Xephyr will exit when all X applications end. Note that you probably want to use passwordless authentication on the server as typing a password twice to start the session would be a drag.

#!/bin/sh

COOKIE=`mcookie`
FILE=~/.Xauth-Xephyr
rm -f $FILE
#echo "add 10.1.0.1:1 . $COOKIE" | xauth
ssh root@server "echo \"add 10.1.0.1:1 . $COOKIE\" | xauth"
echo "add :1 . $COOKIE" | xauth -f $FILE
Xephyr :1 -auth $FILE -reset -terminate -screen 1280x1024 $* &
DISPLAY=10.1.0.1:1 ssh root@server twm
wait

Xen shared storage

disk = [ 'phy:/dev/vg/xen1,hda,w', 'phy:/dev/vg/xen1-swap,hdb,w', 'phy:/dev/vg/xen1-drbd,hdc,w', 'phy:/dev/vg/san,hdd,w!' ]

For some work that I am doing I am trying to simulate a cluster that uses fiber channel SAN storage (among other things). The above is the disk line I'm using for one of my cluster nodes, hda and hdb are the root and swap disks for a cluster node, hdc is a DRBD store (DRBD allows a RAID-1 to be run across the cluster nodes via TCP), and hdd is a SAN volume. The important thing to note is the "w!" mode for the device, this means write access is granted even in situations whre Xen thinks it's unwise (IE it's being used by another Xen node or is mounted on the dom0). I've briefly tested this by making a filesystem on /dev/hdd on one node, copying data to it, then umounting it and mounting it on another node to read the data.

There are some filesystems that support having multiple nodes mounting the same device at the same time, these include CXFS, GFS, and probably some others. It would be possible to run one of those filesystems across nodes of a Xen cluster. However that isn't my aim at this time. I merely want to have one active node mount the filesystem while the others are on standby.

One thing that needs to be solved for Xen clusters is fencing. When a node of a cluster is misbehaving it needs to be denied access to the hardware in case it recovers some hours later and starts writing to a device that is now being used by another node. AFAIK the only way of doing this is via the xm destroy command. Probably the only way of doing this is to have a cluster node ssh to the dom0 and then run a setuid program that calls xm destroy.

multiple ethernet devices in Xen

It seems that no-one has documented what needs to be done to correctly run multiple Ethernet devices (with one always being eth0 and the other always being eth1) in a Linux Xen configuration (or if it is documented then google wouldn't find it for me).

vif = [ 'mac=00:16:3e:00:01:01', 'mac=00:16:3e:00:02:01, bridge=xenbr1' ]

Firstly I use a vif line such as the above in the Xen configuration. This means that there is one ethernet device with the hardware address of 00:16:3e:00:01:01 and another with the address of 00:16:3e:00:02:01. I just updated this section, the 00:16:3e prefix has officially been allocated to the Xen project for virtual machines. Therefore on your Xen installation you can do whatever you like with MAC addresses in that range without risk of competing with real hardware. The Xen code uses random MAC addresses in that range if you let it.

I have two bridge devices, xenbr0 and xenbr1. I only need to specify one as Xen can figure the other out.

Now when my domU's boot they assign ethernet device names from the range eth0 to eth8. If there is only one virtual Ethernet device then it is always eth0 and things are easy. But for multiple devices I need to rename the interfaces.

eth0 mac 00:16:3e:00:01:01
eth1 mac 00:16:3e:00:02:01

This is done through the ifrename program (package name ifrename in Debian). I create a file named /etc/iftab with the above contents and then early in the boot process (before the interfaces are brought up) the devices will be renamed.

In the Red Hat model you edit the files such as /etc/sysconfig/networking/devices/ifcfg-eth0 and change the line that starts with HWADDR to cause a device rename on boot.

Update: the original version of this post used MAC addresses with a prefix of 00:00:00, the officially allocated prefix for Xen is 00:16:3e which I now use. Thanks to the person who commented about this.