Xen Desktop Experience

The main guide I am following for this is here http://forums.linuxmint.com/viewtopic.php?f=42&t=112013 but this is obviously just a guide and I will be documenting my findings here for you, and me to look through in the future. When I have perfected(!) this process, I will create a detailed document without all the stumbles that will be included in this notebook.

18 August 2014

Today I did the very basics, I have taken appart the two desktop machines that I have (that are less than 3 years old - I have plenty of ancient machines!) and swapped hardware around. I am waiting for a Nvidia Quadro FX3800 card to be delivered as this will make it possible to achieve graphics passthrough to a virtualised guest OS. While I am waiting for it to be delivered I might use my AMD card as I hear sometimes people get lucky with these too.

The virt box has an SSD for Dom0 and DomU's and a 1TB spinning disk to be used for server storage (where I currently keep all my work, website etc). The machine was powered on, and I installed Ubuntu Server 14.04 64bit with LVM. Not exactly sure what LVM is yet, going to do some reading up on it, but I think it allows a single "parition" to span across multiple disks, and is a much more modern way of partitioning a disk. SSH has been installed and setup on the server, and the server is called "lemon".

Server OS Server Name Server IP
Ubuntu Server 14.04 64Bit lemon (.groupmsl.co.uk) 192.168.2.12 (static)

Nothing else has been done to the server yet. To reduce power consumption I plan to remove the power supply from the CD/DVD burners, a number of system fans (it has 8, and it doesn't need most of them) and turn off the wifi on the motherboard.

19th August 2014

The Quadro card has arrived! And (of course) it has been installed in the machine in question to test if it works. It does. Not alot else happened today though as there is never enough time, but I did get my power meter out and take a few readings. Granted at the moment the machine is just idleing at the Ubuntu server login screen, but it gives a bit of an idea what power usage we're going to be looking at. So with the old GT430 graphics card the machine idled at 66watts. With no graphics card, or the onboard one, it used 44watts, and with the Quadro it used 67watts, which is pretty much the same as with my GT430. This isn't crazy amounts of power, but I'd like to see how much lower I can go. I'm going to look into bios settings at some point to see what I can reduce. I've also unplugged the two (un-needed) CD/DVD writers in the system, and I guess I could turn off the on-board wifi in the BIOS and anything else I'm not going to be using and see if it makes a difference. As it stands the machine consists of a PSU, Motherboard, CPU, 4 sticks DDR3 RAM, SSD, HDD, Quadro, and dedicated network card. I'll put down the detailed specs of the machine later. I will worry about power usage (a lot) later, but I want to see if the machine is actually capable of what I want to achieve first...

Setup Power usage (w)
Nvidia GT430 66.7
No dedicated GPU 44.3
Quadro FX3800 67.4

As soon as I get a chance I'm going to get going with the setup. I'm not sure I've got Ubuntu server installed exactly how I want it, but it works, so I'll keep going and if all is sucessfull I'll start again (to finely document what I did) and make sure I get everything perfect next time.

20th August 2014

Again, didn't get much time today to work on this, but I have at least installed Xen, re-set the static IP, because I got it wrong the first time, and change the boot options in Grub. Below is what I did.

Firstly, I tried to update my apt-get library:

sudo apt-get update

This command didn't actually work because the machine couldn't resolve any of the update servers into IP addresses because it didn't have any nameservers set. I had previously set nameserers in the /etc/resolv.conf file, but it appears something was changing this file on a reboot. I still don't know what was changing the file (probably network-manager), but I did find a solution. It turns out it is actually possible to put nameservers in the /etc/network/interfaces file where I had previously set the static IP. Below is a copy of my final version of this file:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto em1
iface em1 inet static
    address 192.168.2.12
    network 192.168.2.0
    broadcast 192.168.2.255
    netmask 255.255.255.0
    gateway 192.168.2.1
    dns-search groupmsl.co.uk
    dns-nameservers 8.8.8.8 8.8.4.4

After this the update command worked and the software librarys were updated. Then I upgraded all the software to the latest release:

sudo apt-get upgrade

Once the machine was up to date, we were ready to go, and I installed Xen and all the other utils we need:

sudo apt-get install xen-system-amd64 qemu-common qemu-keymaps bridge-utils

The guide I'm following suggests using the Xen xm toolstack, but because I want Windows 8 (and OS X, Linux) as guests, I'm going to try my luck with the xl toolstack. If I have problems I can always switch back later and try again.

sudo sed -i \'s/TOOLSTACK=.*\\+/TOOLSTACK=\"xl\"/\' /etc/default/xen

Next was adding boot options to Grub, so that it's possible to actually boot into Xen. We need to edit the /etc/default/grub file and do the following:

CHANGE: GRUB_DEFAULT=3 - This means grub will default to the third item in the list, which should be your Xen option. Edit to whatever number you need as necessary. COMMENT OUT: # GRUB_HIDDEN_TIMEOUT=0 COMMENT OUT: # GRUB_HIDDEN_TIMEOUT_QUIET=true CHANGE: GRUB_TIMEOUT=10 ADD LINE: GRUB_CMDLINE_XEN=\"iommu=1 dom0_mem=2048M\" - This should be the third GRUB_CMDLINE line in the file, if not, count which number it is, and change the GRUB_DEFAULT to reflect this (see above).

After editing this file, my /etc/default/grub file looked like this:

# If you change this file, run \'update-grub\' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n \'Simple configuration\'

GRUB_DEFAULT=3
#GRUB_HIDDEN_TIMEOUT=0
#GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=\"\"
GRUB_CMDLINE_LINUX=\"\"

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM=\"0x01234567,0xfefefefe,0x89abcdef,0xefefefef\"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo\'
#GRUB_GFXMODE=640x480

# Uncomment if you don\'t want GRUB to pass \"root=UUID=xxx\" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY=\"true\"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE=\"480 440 1\"


GRUB_CMDLINE_XEN=\"iommu=1 dom0_mem=2048M\"

When you've edited this file, it still hasn't taken effect until you run the command below:

sudo update-grub

This is as far as I've got. When I get a longer period of time (longer than 5 minutes!) - hopefully tomorrow, I am going to dive into the BIOS and enable VT-d, and the other things we need for Virtualisation passthrough, and then see if I can get a machine going...

21st and 23rd August 2014

Today was a good day. I got to spend lots of time looking at this project and have made some good progress. Firstly, I went into the BIOS and changed a few settings.

Then the machine was booted into Xen. Once in, I logged into root with sudo -i, then it was time to change the network interfaces again! Xen can bridge the physical network connection of the machine to all the guests, which is what we want it to do. I edited /etc/network/interfaces and when done it looked like the following:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto xenbr0
iface xenbr0 inet static
    address 192.168.2.12
    network 192.168.2.0
    broadcast 192.168.2.255
    netmask 255.255.255.0
    gateway 192.168.2.1
    dns-search groupmsl.co.uk
    dns-nameservers 8.8.8.8 8.8.4.4
    bridge_ports em1
    bridge_stp off

Now the config is in place for the bridge, but the bridge interface doesn't actually exist, so this will have to be created:

brctl addbr xenbr0
service networking restart

Then we check if everything is good:

ifconfig

If you forget what your actual interfaces are called when setting up your bridge, use the ip link show command.

At this point networking is done, and there are just a couple of missing links that need to be created before we can get going with some actual visualization setup:

ln -s /usr/lib /usr/lib64
ln -s /usr/lib/xen-4.3 /usr/lib/xen-default

Now we can see which PCI devices on my computer are VGA devices:

lspci | grep VGA

01:00.0 VGA compatible controller: NVIDIA Corporation GT200GL [Quadro FX 3800] (rev a1)

Now we have the PCI ID of my graphics card, I want to see if it has an onboard sound device...

lspci | grep 01:00.

01:00.0 VGA compatible controller: NVIDIA Corporation GT200GL [Quadro FX 3800] (rev a1)

This means there is no audio device on my graphics card (not surprising given the type of card (Quadro professional graphics, and no HDMI ports)). If you have a second line to the output of the above command, it means you have audio.

Now we need to tell Xen to disable the Dom0 from using this hardware, so another, virtual, machine (DomU) is able to use it:

echo \"xen-pciback passthrough=1 hide=(01:00.0)\" >> /etc/initramfs-tools/modules
update-initramfs -k all -c

Obviously in the above you need to change the PCI ID to the one/ones you found in the section above.

Now we can actually create the config file for the virtual machine. I created a file called /etc/xen/win81.cfg because I am going to be creating a Windows 8.1 guest (just for testing). Below is what I put in this file. It took many changes (and many hours), but only the final version of the file is pasted below:

kernel = \'/usr/lib/xen-default/boot/hvmloader\'
builder=\'hvm\'
memory = 4096
name = \'win81\'
vcpus=2
#pae=1
acpi=1
apic=1
on_xend_stop=\'shutdown\'
vif = [ \'mac=00:16:3e:68:e1:01,bridge=xenbr0\' ]
disk = [ \'phy:/dev/mapper/lemom--vg-win81,hda,w\' , \'file:/media/Home/software/OperatingSystems/Windows/8Professional64/Windows81.iso,hdc:cdrom,r\' ]
#device_model = \'/usr/lib/xen-default/bin/qemu-dm\'
device_model_version = \'qemu-xen-traditional\'
boot=\'dc\'
sdl=0
vnc=1
vncpasswd=\'123\'
stdvga=0
serial=\'pty\'
tsc_mode=0
viridian=1
usb=0
usbdevice=\'tablet\'
gfx_passthru=0
pci=[ \'01:00.0\']
localtime=1
pci_power_mgmt=1

Once the config file was created, it was time to actually create the virtual disk that we have given the DomU in the config file. To see the physical volumes set up for LVM on the system, run the following command:

pvdisplay

--- Physical volume ---
PV Name /dev/sda5
VG Name lemom-vg
PV Size 238.24 GiB / not usable 0
Allocatable yes
PE Size 4.00 MiB
Total PE 60988
Free PE 53837
Allocated PE 7151
PV UUID Hwj9Gw-wDyT-xAZH-iK6R-TL30-5VUG-oxvBDj

From the above, you can see that the partition /dev/sda5 is set up for LVM and has is part of the virtual drive group lemom-vg. This was actually supposed to be "lemon-vg", but it looks like I can't type, and I don't (as of yet) know how to change the name...or if it is possible! Now we know our virtual drive group name, we can create a new virtual drive:

lvcreate -L 50G -n win81 lemom-vg

Logical volume "win81" created

Don't forget to give the drive the same name as you have specified in the config file earlier!!

Now we can check virtual drives with the following:

lvdisplay

--- Logical volume ---
LV Path /dev/lemom-vg/root
LV Name root
VG Name lemom-vg
LV UUID 210rsZ-40Ek-3udY-rSNB-tdgH-nz0h-86c196
LV Write Access read/write
LV Creation host, time lemom, 2014-08-18 21:29:07 +0100
LV Status available
# open 1
LV Size 4.25 GiB
Current LE 1088
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:0

--- Logical volume ---
LV Path /dev/lemom-vg/swap_1
LV Name swap_1
VG Name lemom-vg
LV UUID Vz8z50-mNw2-6H4j-e3zi-NicG-6joI-1KDJxf
LV Write Access read/write
LV Creation host, time lemom, 2014-08-18 21:29:07 +0100
LV Status available
# open 2
LV Size 23.68 GiB
Current LE 6063
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:1

--- Logical volume ---
LV Path /dev/lemom-vg/win81
LV Name win81
VG Name lemom-vg
LV UUID PZRluD-z3iy-hhoH-w7Bw-naUE-S1JQ-jnoV84
LV Write Access read/write
LV Creation host, time lemon, 2014-08-21 19:53:36 +0100
LV Status available
# open 0
LV Size 50.00 GiB
Current LE 12800
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:2

From the above, all looks good, and now all I need to do is get the installation source in place. In this case I mounted an NFS share to /media/Home from my other (old, that I am trying to replace with this project) server. In the win81 config file you can see that I placed the path to the Windows 8.1 iso on my other server. (To be honest, this wasn't my original plan, I stupidly tried to SCP the ISO to the local drive! This was a) Pointless, given it's on my server which is on a fast network and b) silly because I was really stingy with the partition size of the Dom0, and it wouldn't even fit!)

I then rebooted, and checked if the graphics card was in the Xen list of assignable (to a DomU) hardware:

xl pci-assignable-list
0000:01:00.0

This is all good and we are ready to go! I started the VM with xl create /etc/xen/win81.cfg. At first there were a few errors which I had to fix..related to my config file, and then we booted!! When installing and booting, the VM didn't use the passed through graphics card, because it didn't have drivers for it so I had to VNC to the machine to see what was going on (and control it). All of the commands in this guide were actually given from my MacBook Pro over SSH, and this is where I VNC'd into the machine from also. At first VNCing didn't work at all - this was simply because Xen wasn't listening to remote connections. Rather than fixing this (I will do later), I just used SSH to forward port 5900 to port 5901 on my laptop, and VNC'd to localhost:5901 on my laptop. This got me slightly further and I was able to connect, but the connection was almost instantly closed. I tried something other than the built in VNC viewer in OS X, and got the same result. I then did some googling, and it turns out the built in VNC viewer and RealVNC automatically detect the best colour depth and quality for the connection. Normally this is fine, but with Xen (and other Hypervisors apparently) this feature of VNC does not work. You simply have to go into your VNC connection settings (on the client) and SET a colour depth - whatever you set this to seems fine, it's just the auto setting that doesn't work.

Once VNC'd in sucessfully, I saw the Windows 8.1 installation screen!! From here there isn't much else to say - it installed as normal, and once booted into Windows, I installed the Nvidia drivers for my card, rebooted and off I went!!! Currently I still have to VNC to the machine, simply for mouse and keyboard input as I haven't got USB passthrough yet, but that is for another night! Currently feeling very pleased! At some point I want to see if I can have two VM's assigned the same graphics card, as long as only one of them is booted at the same time - we will see!

Resources