Xen on Debian

From wiki

Jump to: navigation, search

Please note - this was dumped from a Moinmoin wiki - i will prettyfy it soon

Contents

[edit]

Introduction

This will explain how to install Xen the VirtualMachine (VM) on Debian Sarge we are installing Xen from the Source tarball and not binary as we want to roll our own kernels. This is how I by-chance did it, it is in no way idiot proof and propably many faults with this - but ohwell, it's just my scratchpad for later in/re-installs. Firstly - make sure you DO NOT USE the default debian kernel, you are better off if you have rolled your own working one before attempting to do Xen. I will be installing Xen 3.0.0 on a machine currently running kernel 2.6.11.12, you must have your config profile handy! Networking wise, I have a machine and another dedicated IP that I can use. Make sure your base machine is NOT bound to that IP address, as Xen will make a bridge and actually use that IP address for your Xen machine.

NOTE: P4 Hyperthreading machines (like mine) are experiencing this bug(not anymore) :P

[edit]

So let's start off...

Make your compile-home: 

fbotha@hydra:~$ mkdir src
fbotha@hydra:~$ cd src
fbotha@hydra:~/src$ wget http://bits.xensource.com/Xen/latest/xen-3.0.0-src.tgz}}}

Extract your kernel source: 

fbotha@hydra:~/src$ tar zxvf xen-3.0.0-src.tgz
...
fbotha@hydra:~/src$ cd xen-3.0.0
fbotha@hydra:~/src/xen-3.0.0$  make world

You will notice that make world will also download kernel 2.6.12, which is the kernel Xen v3 LIKES at the moment. Your Xen machine will be running off this kernel, this is why your current kernel's .config is necessary!

This will also compile the {{{dom0}}} and {{{domU}}} kernels - this is the kernel your ba se machine should run (which will start up your {{{domU}}} virtual servers).

Your {{{domU}}} kernel can be used inbetween all your virtual servers and is basically a stripped down kernel without any Firewall support or device drivers. You CAN compile these in if you want to.

Once compiled we want to change some kernel options, go to the {{{dom0}}} kernel dir: 

fbotha@hydra:~/src/xen-3.0.0$ cd linux-2.6.12-xen0/
fbotha@hydra:~/src/xen-3.0.0/linux-2.6.12-xen0$ make ARCH=xen menuconfig

NOTE: You MUST specify {{{ARCH=xen}}}, otherwise no xen option will be available and the kernel WILL NOT compile.

NOTE2: This is only true in the early Xen 3.0.0 releases (around Jan-EarlyFeb 2006), you can now do "make menuconfig" in the directories in question.

Now choose the kernel options you like.

NOTE: I need to figure out sometime how to use my existing ex-XEN kernel to be happy with the current one - people will just have to choose the right options and compare with their own current .config

NOTE2: Removing DRI support for X, under Devices has helped me with some kernel compilation errors.

General things I like in my xen0 kernel is:

  • Kernel .config support (to have kernel options in /proc)
  • SATA Drivers under SCSI
  • Networking Driver for your Ethernet
  • Iptables support
  • IPV6

Once new config has been saved, you can: 

fbotha@hydra:~/src/xen-3.0.0/linux-2.6.12-xen0$ cd ..

You can ofcourse also change your xenU/domU kernels at this point by going to the linux-2.6.12-xenU directory and doing the same menuconfig as before there (remember the ARCH bit).

So let's compile your kernels: 

fbotha@hydra:~/src/xen-3.0.0$ time make -j2 kernels

Once that's done we can do the install

Become root and do a make install... 

fbotha@hydra:~/src/xen-3.0.0$ su
Password:
hydra:/home/fbotha/src/xen-3.0.0# make install

You will notice that Xen the bastard spams your /boot with all it's kernels and versions, we will worry about that in a bit. Make the Xen services start at boot time, 

update-rc.d xend defaults 20 21
update-rc.d xendomains defaults 21 20

Let's configure your bootloader

You HAVE to basically use GRUB - another bootloader will not work.

Edit your /boot/grub/menu.lst

add: 

title           Debian GNU/Linux, kernel 2.6-xen0
root            (hd0,1)
kernel          /boot/xen-3.0.0.gz console=ttyS0,9600n8 dom0_mem=64000 root=/dev/sda2
module          /boot/vmlinuz-2.6-xen0 root=/dev/sda2 ro console=ttyS0,9600n8
savedefault
boot

Reboot your machine and choose your Xen kernel! 

# reboot

If all goes well you should have a fully working and booted machine.

Now we will make the base-setup of a virtual machine (thereafter you can copy these images to new machines!)

Let's save them all in /usr/local/xen, 

hydra:/home/fbotha# mkdir /usr/local/xen
hydra:/home/fbotha# cd /usr/local/xen
hydra:/usr/local/xen# mkdir images
hydra:/usr/local/xen# cd images/

Let's make a base image of 20gig for a machine: 

hydra:/usr/local/xen/images# dd if=/dev/zero of=vm_base.img bs=1024k count=20000
76226+0 records in
76225+0 records out
39027200 bytes transferred in 2.529881 seconds (15426496 bytes/sec)

And it's swap space of 500Mb: 

hydra:/usr/local/xen/images# dd if=/dev/zero of=vm_base-swap.img bs=1024k count=500
500+0 records in
500+0 records out
524288000 bytes transferred in 20.838443 seconds (25159653 bytes/sec)

Now we format the base disk file with ext3: 

hydra:/usr/local/xen/images# mkfs.ext3 vm_base.img
mke2fs 1.37 (21-Mar-2005)
vm_base.img is not a block special device.
Proceed anyway? (y,n) y

And same for the swap space: 

hydra:/usr/local/xen/images# mkswap vm_base-swap.img
Setting up swapspace version 1, size = 524283 kB
no label, UUID=9b623b12-e2af-45c9-ac62-c18598dde424

Let's make a directory and loopback mount the base filesystem so we can put a OS on it, 

hydra:/usr/local/xen/images# mkdir ../base
hydra:/usr/local/xen/images# mount -o loop vm_base.img ../base/

Make a bootable OS on the base disk,

We use the debootstrap utility to make a bootable OS in the base directory where we mounted the new file(system): 

hydra:/usr/local/xen/images# debootstrap --arch i386 sarge ../base/ ftp://the.earth.li/debian

This takes a while as it ftp's and installs the base OS.

Now we chroot so we ARE the OS that's mounted: 

hydra:/usr/local/xen/images# chroot ../base/
hydra:/#

Run: 

hydra:/#  apt-setup

Choose your local debian mirror Choose that you want security updates, etc.

Edit /etc/apt/sources.list

Replace everything saying 'testing' to 'stable' 

hydra:/# apt-get update


Now we have to setup the locales, you might see lots of errors till you exit the machine: 

hydra:/# apt-get install localeconf

Select en_US ISO-8859-1 or similair for your locale Select the locale as your for the environment. Manage it with debconf Do not change LC variables and Replace your existing locale configs choose again your locale

Now lets do the base config: 

hydra:/# base-config

(ignore locale errors if you encounter it until you reboot)

Configure you timezone Setup your user & passwords Set your hostname And: Finish configuring your base system.

Get rid of your hostname for now: 

hydra:/# rm -f /etc/hostname

Change your /etc/fstab to look like this (IN ALL CONFIGS): 

hydra:/# cat /etc/fstab
/dev/hda1               /               ext3    defaults        1       2
/dev/hda2               none            swap    sw            0       0
/dev/pts                devpts          gid=5,mode=620    0       0
none                    /dev/shm        tmpfs   defaults     0       0

Change your /etc/network/interfaces to look like this: 

auto lo
iface lo inet loopback
        address 127.0.0.1
        netmask 255.0.0.0

Make your /etc/hosts: 

hydra:/# vi /etc/hosts
127.0.0.1       localhost.localdomain   localhost

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Exit the chrooted environment: 

hydra:/# exit
exit
hydra:/usr/local/xen/images#

Copy kernel modules: 

hydra:/usr/local/xen/images# cp -dpR /lib/modules/2.6.12.6-xenU ../base/lib/modules/

Due to a tls issue with linux we move that away: 

hydra:/usr/local/xen/images# mv ../base/lib/tls ../base/lib/tls.disabled

And now cleanly unmount your image: 

hydra:/usr/local/xen/images# umount ../base

Our virtual machine template is now ready, now we can make a machine, lets call it zaphod . {{{ hydra:/usr/local/xen/images# cp vm_base.img zaphod.img ; cp vm_base-swap.img zaphod-swap. img }}} This of course will take a while.

Now we create a startup file for zaphod in /etc/xen/zaphod-config.sxp:

{{{ name ="zaphod" kernel ="/boot/vmlinuz-2.6.12-xenU" root ="/dev/hda1" memory =128 disk = ['file:/usr/local/xen/images/zaphod.img,hda1,w','file:/usr/local/xen/images/zaphod -swap.img,hda2,w']

  1. network

nics=1 dhcp ="off" ip="192.168.0.101" netmask="255.255.255.0" gateway="192.168.0.1" hostname="zaphod.lagged.za.net"

extra="3" }}}

To start zaphod automatically on startup we do: {{{ hydra:/usr/local/xen/images# ln -s /etc/xen/zaphod-config.sxp /etc/xen/auto }}} Now lets start it up: {{{ hydra:/usr/local/xen/images# xm create -c /etc/xen/zaphod-config.sxp }}}

One last thing. on Shutdown we want to save the state of our VM's, just : {{{ hydra:/home/fbotha# mkdir /var/lib/xen/save }}}

Everything should work now - yipee



[edit]

Upgrading to Unstable / New release

Right so I have Xen3.0.0 running but some stupid [http://xenbits.xensource.com/xen-unstab le.hg?cmd=filediff;node=2d31ebf402e1f4c2ae2b335b9474e1b08f64f15c;file=linux-2.6-xen-spars e/drivers/xen/netback/netback.c bug] forces me to upgrade. First install the [http://www. selenic.com/mercurial Mecurial] (Code repository type shit) tools, and let's get some spa ce..

{{{ fbotha@hydra:~/src$ mkdir unstable fbotha@hydra:~/src$ cd unstable fbotha@hydra:~/src/unstable$ hg clone http://xenbits.xensource.com/xen-unstable.hg requesting all changes adding changesets adding manifests adding file changes added 8660 changesets with 76110 changes to 7642 files fbotha@hydra:~/src/unstable$ cd xen-unstable.hg/ }}}


First we do a dry-run, let it download the kernel etc etc and compile it's silly kernels - we will be adapting it to our needs afterwards, you can propably copy kernel source som ewhere at this point but I'm not too bothered about that right now..

{{{ fbotha@hydra:~/src/unstable/xen-unstable.hg$ make world }}}


.. 22minutes later...

Let's copy our *OLD* kernel options (from the last Xen compile) to the current directorie s: {{{ fbotha@hydra:~/src/unstable/xen-unstable.hg$ cp ~fbotha/src/xen-3.0.0/linux-2.6.12-xen0/. config linux-2.6.12-xen0/ fbotha@hydra:~/src/unstable/xen-unstable.hg$ cp ~fbotha/src/xen-3.0.0/linux-2.6.12-xenU/. config linux-2.6.12-xenU/ }}}

Now let's compile the kernels with our own options:

{{{ fbotha@hydra:~/src/unstable/xen-unstable.hg$ make -j2 kernels }}}


.. 8 minutes later ..

We make a backup of our running kernel - JUST IN CASE ...

{{{ fbotha@hydra:~/src/unstable/xen-unstable.hg$ su Password: hydra:/home/fbotha/src/unstable/xen-unstable.hg# cp /boot/xen-3.0.0.gz /boot/old hydra:/home/fbotha/src/unstable/xen-unstable.hg# cp /boot/vmlinuz-2.6-xen0 /boot/vmlinuz- old }}}

Lets install everything:

{{{ hydra:/home/fbotha/src/unstable/xen-unstable.hg# make install }}}


[edit]

Upgrading to Unstable ~2006-02-14th (linux kernel 2.6.16-rc2) release

Looks like this is now the way to go with the new kernel - a few scripts has also been changed.

Basically, grab it from the repository like before, goto the linux-2.6.16* directories, c opy the .config from your PREVIOUS install to the directory in question and execute:

{{{

  1. make world

}}}

It will compile things for now.

{{{

  1. make oldconfig

}}}

You will then ofcourse run through the NEW kernel settings - it might be a good idea to:

{{{

  1. grep -i xen /where/your/old/kernels/.config

}}}

To compare the Xen settings at least.

You can now do the make menuconfig and run through things by hand too from the linux-2.6. 16*xen* directories:

{{{

  1. make menuconfig

}}}

Once done let's compile everything:

{{{

  1. make
  2. cd ../linux-2.6.16-rc2-xenU/ && make
  3. make kernels
  4. make install-kernels
  5. make
  6. make install

}}}

There is no doubt duplication in above commands but it works....


[edit]

Upgrading from one release to the next WITHOUT downtime

(This is a work in progress...)

Lets find out what version you're currently running as there are no propper tools to tell 

you what you are running:

egrep '(^release

Looks like we're using kernel 2.6.16-rc2 at least, so in src/ 

grep $kernel

The only kernel being used in src/ for that is 2006-02-14...

All of this above can ofcourse be avoided if you do not keep MULTIPLE Versions of Xen aro und - but due to all of the above bits of this document - it has been required.

{{{ hydra:/home/fbotha/src# ln -s 2006-02-14-src CURRENT }}}

Now we get the tarball of the new version we want and extract it...

{{{ hydra:/home/fbotha/src# wget http://bits.xensource.com/Xen/latest/xen-3.0.2-src.tgz }}}

Lets do a dry-run kernel setup building a dom0 and domU kernel:

{{{ hydra:/home/fbotha/src# export KERNELS="linux-2.6-xen0 linux-2.6-xenU" hydra:/home/fbotha/src# cd xen-3.0.2-2/ hydra:/home/fbotha/src/xen-3.0.2-2# nice -n 19 make world }}}

Once done, lets copy in the configs of our old kernels

We copy the new-version-given .config so we have a idea of what kernel settings we need I F we do not have it in our old copy of .config

{{{ hydra:/home/fbotha/src/xen-3.0.2-2# cd linux-2.6.16-xen0/ hydra:/home/fbotha/src/xen-3.0.2-2/linux-2.6.16-xen0# cp .config .configDIST hydra:/home/fbotha/src/xen-3.0.2-2/linux-2.6.16-xen0# make oldconfig hydra:/home/fbotha/src/xen-3.0.2-2/linux-2.6.16-xen0# cp ~fbotha/src/CURRENT/xen-unstable .hg/linux-2.6.16-rc2-xen0/.config . hydra:/home/fbotha/src/xen-3.0.2-2/linux-2.6.16-xen0# make oldconfig }}}

If you are unsure about any of the questions that it asks you can just grep the distribut ed .config:

{{{ hydra:/home/fbotha/src/xen-3.0.2-2/linux-2.6.16-xen0# grep XEN_SYSFS .configDIST CONFIG_XEN_SYSFS=y }}}

Now we do the same for xenU:

{{{ hydra:/home/fbotha/src/xen-3.0.2-2/linux-2.6.16-xen0# cd ../linux-2.6.16-xenU/ hydra:/home/fbotha/src/xen-3.0.2-2/linux-2.6.16-xenU# cp .config .configDIST hydra:/home/fbotha/src/xen-3.0.2-2/linux-2.6.16-xenU# cp ~fbotha/src/CURRENT/xen-unstable .hg/linux-2.6.16-rc2-xenU/.config . hydra:/home/fbotha/src/xen-3.0.2-2/linux-2.6.16-xenU# make oldconfig }}}


And now to compile all of this:

  1. make kernels
  2. make install-kernels
  3. make
  4. make install

{{{ hydra:/home/fbotha/src/xen-3.0.2-2/linux-2.6.16-xenU# cd ../ hydra:/home/fbotha/src/xen-3.0.2-2# nice -n 19 time make kernels hydra:/home/fbotha/src/xen-3.0.2-2# mkdir ~fbotha/backup hydra:/home/fbotha/src/xen-3.0.2-2# cp /boot/* ~fbotha/backup/ cp: omitting directory `/boot/grub' hydra:/home/fbotha/src/xen-3.0.2-2# nice -n 19 time make install-kernels hydra:/home/fbotha/src/xen-3.0.2-2# nice -n 19 time make hydra:/home/fbotha/src/xen-3.0.2-2# nice -n 19 time make install }}}

After a make install we need to restart the xend, this is just want connects to dom0 and talks to it via sockets and should not break anything:

{{{ hydra:/etc/init.d# /etc/init.d/xend restart }}}


So now we test our vm's by doing a list:

{{{ hydra:/etc/init.d# xm list Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 64 2 r----- 736082.9 ns1 6 32 1 -b---- 1302.9 speel1 10 128 1 -b---- 481.4 voip 12 128 1 -b---- 4019.4 zaphod 3 256 1 -b---- 115651.6 hydra:/etc/init.d# }}}

All looks fine.

Now we will be playing with speel1 to make sure it boots with its new kernel.... having a 

spare VM about always helps!

{{{ hydra:/etc/init.d# xm shutdown speel1 }}}

Firstly.. lets copy the right modules over.. lets find out what has been written today... .

{{{ hydra:/etc/init.d# date Tue Apr 18 16:03:58 BST 2006 hydra:/etc/init.d# ls -lart /lib/modules/ total 44 drwxr-xr-x 3 500 500 4096 2005-12-13 02:28 2.6.12.6-xen drwxr-xr-x 8 500 500 4096 2005-12-13 02:28 .. drwxr-xr-x 5 root root 4096 2006-01-09 14:21 2.6.8-2-686 drwxr-xr-x 3 root root 4096 2006-01-10 01:50 2.6.11.12 drwxr-xr-x 3 500 500 4096 2006-02-14 12:44 2.6.12.6-xen0 drwxr-xr-x 3 500 500 4096 2006-02-14 12:44 2.6.12.6-xenU drwxr-xr-x 3 root root 4096 2006-02-16 00:48 2.6.16-rc2-xenU drwxr-xr-x 3 root root 4096 2006-03-30 16:47 2.6.16-rc2-xen0 drwxr-xr-x 11 500 500 4096 2006-04-18 15:53 . drwxr-xr-x 3 root root 4096 2006-04-18 15:58 2.6.16-xen0 drwxr-xr-x 3 root root 4096 2006-04-18 15:58 2.6.16-xenU hydra:/etc/init.d# }}}

As you can see /lib/modules/2.6.16-xen0 and xenU is new, so we will copy that in our spee l1-vm image....

{{{ hydra:~# mount -o loop /usr/local/xen/images/speel1.img /usr/local/xen/mount/ hydra:~# cp -dpR /lib/modules/2.6.16-xen* /usr/local/xen/mount/lib/modules/ hydra:~# umount /usr/local/xen/mount/ }}}

Lets check speel1's config to see what kernel it uses: 

grep kernel

kernel ="/boot/vmlinuz-2.6.12-xenU"


Ahh - this version uses 2.6.16, so lets find the /boot files changes today.... 

grep `date +%Y-%m-%d`

-rw-r--r-- 1 root root 3905301 2006-04-18 15:57 xen-syms-3.0.2-2 lrwxrwxrwx 1 root root 14 2006-04-18 15:57 xen.gz -> xen-3.0.2-2.gz lrwxrwxrwx 1 root root 14 2006-04-18 15:57 xen-3.gz -> xen-3.0.2-2.gz lrwxrwxrwx 1 root root 14 2006-04-18 15:57 xen-3.0.gz -> xen-3.0.2-2.gz -rw-r--r-- 1 root root 210973 2006-04-18 15:57 xen-3.0.2-2.gz lrwxrwxrwx 1 root root 19 2006-04-18 15:58 vmlinuz-2.6-xen0 -> vmlinuz-2.6.16-xen0 -rw-r--r-- 1 root root 1395008 2006-04-18 15:58 vmlinuz-2.6.16-xen0 -rw-r--r-- 1 root root 3744224 2006-04-18 15:58 vmlinux-syms-2.6.16-xen0 -rw-rw-r-- 1 root root 699566 2006-04-18 15:58 System.map-2.6.16-xen0 -rw-rw-r-- 1 root root 26747 2006-04-18 15:58 config-2.6.16-xen0 -rw-r--r-- 1 root root 1363733 2006-04-18 15:58 vmlinuz-2.6.16-xenU -rw-r--r-- 1 root root 3658556 2006-04-18 15:58 vmlinux-syms-2.6.16-xenU -rw-rw-r-- 1 root root 26407 2006-04-18 15:58 config-2.6.16-xenU lrwxrwxrwx 1 root root 19 2006-04-18 15:58 vmlinuz-2.6-xenU -> vmlinuz-2.6.16-xenU -rw-rw-r-- 1 root root 685999 2006-04-18 15:58 System.map-2.6.16-xenU drwxr-xr-x 3 500 500 4096 2006-04-18 15:58 . hydra:~#


So /boot/vmlinuz-2.6.16-xenU is a good bet! Edit /etc/xen/speel1.config to use that kernel

Next, have a look at your /boot/grub/menu.lst And make sure the /boot/xen* and /boot/vmlinuz* lines references your new Xen I usually take my backups:

{{{ hydra:~# cp ~fbotha/backup/xen-3.0.0.gz /boot/failsafexen.gz hydra:~# cp ~fbotha/backup/vmlinuz-2.6-xen0 /boot/failsafevmlinuz }}}

And my /boot/grub/menu.lst contains a:

{{{ title Debian GNU/Linux, kernel 2.6-xen0 XEN OLD root (hd0,1) kernel /boot/failsafexen.gz console=ttyS0,9600n8 dom0_mem=64000 root=/dev/sda2 module /boot/failsafevmlinuz root=/dev/sda2 ro console=ttyS0,9600n8 maxcpus=2 ma x_loop=32 savedefault boot }}}

FB-NOTE: My grub boots off sdb so make sure both grub's menu.lst is actually the same in case one dies (:

Lets move the current symlink to the new one and swap stuff around

{{{ hydra:~# rm ~fbotha/src/CURRENT ~fbotha/src/PREVIOUS hydra:~# ln -s ~fbotha/src/2006-02-14-src ~fbotha/src/PREVIOUS hydra:~# ln -s ~fbotha/src/xen-3.0.2-2 ~fbotha/src/CURRENT }}}

It looks like we can ONLY test a new VM kernel if the dom0 is running it, although domU's 

running older kernels seem ok. So, the only way to test this is to actually reboot dom0!
Luckily, the init scripts should save state, so no uptimes should be lost.

See if your speel1 is working correctly (booted up ok?) so we know the new domU kernels w ork and can boot up in case any of your old VM's die over time for whatever reason.


[edit]

3.0.4 Upgrade

{{{ fbotha@hydra:~/src$ wget http://bits.xensource.com/oss-xen/release/3.0.4-1/src.tgz/xen-3. 0.4_1-src.tgz fbotha@hydra:~/src$ tar zxvf xen-3.0.4_1-src.tgz >/dev/null fbotha@hydra:~/src$ cd xen-3.0.4_1-src fbotha@hydra:~/src/xen-3.0.4_1-src$ export KERNELS="linux-2.6-xen0 linux-2.6-xenU" fbotha@hydra:~/src/xen-3.0.4_1-src$ time nice -n 19 make world # about 35min s }}}

On zaphod print out /proc/config.gz as domU On hydra print out /proc/config.gz as dom0

Compare what you would/would not like in your new kernels for each instance below...

{{{ make linux-2.6-xen0-config CONFIGMODE=menuconfig make linux-2.6-xenU-config CONFIGMODE=menuconfig }}}

Compile them with the new settings:

{{{ hydra:/home/fbotha/src/xen-3.0.4_1-src# nice -n 19 time make linux-2.6-xen0-build hydra:/home/fbotha/src/xen-3.0.4_1-src# nice -n 19 time make linux-2.6-xenU-build }}}


Retrieved from "http://www.lagged.za.net/mediawiki/index.php/Xen_on_Debian"