This server will run a CentOS 5 host operating system. It will have Debian guest systems virtualized through Xen. I chose CentOS because it has good install support for software raid and lvm and the Red Hat product it derives from seems to have mature virtualization technology (vt). Debian is running on the existing servers I am migrating from physical to virtual. Debian has excellent long term maintainability and I will use my existing disaster recovery plan for the migration.
I installed CentOS. I chose server-gui and virtualization on install. When I setup LVM I created an LVM that mounts to /xen and left 150+G in the Volume Group but unallocated. I will use the space to additional guest vm partitions.
I downloaded the Debian 3.1 xen virtual machine (vm) package from http://jailtime.org . I chose this since I was able to make it work on another machine I was playing with previously. To keep the system as close to the model vm as possible, I have made some additional links.
- Note: The http://jailtime.org package unpacked to the current directory. It expects to be in /xen/debian so you might as well create the sub-folder and cd there before un-taring.
Setup
I have found the system would kernel panic previously. In part this is because the xenblk module is not loaded. You also need xennet either by including it in the ramdisk or by using modules.conf/modprobe.conf. I chose to include it in the ramdisk.
# uname -r
2.6.18-8.el5xen
# mkinitrd --preload=xenblk --with=xennet /boot/initrd-`uname -r`U.img `uname -r`
# ln -fs /boot/initrd-`uname -r`U.img /boot/initrd-2.6-xenU.img
I linked vmlinuz-2.6-xenU -> vmlinuz-2.6.18-8.el5xen because many of the prebuilt vms expect this to exist.
ln -fs /boot/vmlinuz-`uname -r` /boot/vmlinuz-2.6-xenU
Because the jailtime.org images expect /xen to contain the images, I have linked it to /vserver
ln -s /vserver /xen
Kernel Panic
At this point when I tried to start the vm, it kernel panic’d. This command creates (starts) the vm, and the -c option takes the console you are viewing for the new vm’s console. This lets you view the boot and errors.
xm create -c /xen/debian/debian.3-1.xen3.cfg
The last of the output:
XENBUS: Device with no driver: device/vbd/2049
XENBUS: Device with no driver: device/vbd/2050
XENBUS: Device with no driver: device/vif/0
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
VFS: Cannot open root device "sda1" or unknown-block(0,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
Troubleshooting and the fix
You can mount the image with a loop device and look at the files in the vm image. You cannot do this while the vm is running, and you cannot start the vm while it is mounted like this. Keep this in mind for later; you don’t need to do this right now.
cd /mnt
mkdir vm
mount -o loop /vserver/debian/debian.3-1.img /mnt/vm
I added this to the bottom of /xen/debian/debian.3-1.xen3.cfg
ramdisk = "/boot/initrd-2.6-xenU.img"
Running, almost…
Now I start it, and it boots completely. You can leave the console with Ctrl+]
xm create -c /xen/debian/debian.3-1.xen3.cfg
And the catch? No way to interact.
$ nmap 192.168.0.202
Starting Nmap 4.20 ( http://insecure.org ) at 2007-07-29 23:14 CDT
All 1697 scanned ports on 192.168.0.202 are closed
Nmap finished: 1 IP address (1 host up) scanned in 1.832 seconds
Shutting down
I am shutting down the vm now. I will probably need to mount the image in a loopback and use chroot to add ssh. I will pursue this more tomorrow.
# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 3920 2 r----- 510.5
debian.3-1 5 127 1 r----- 5255.2
# xm shutdown debian.3-1
# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 3920 2 r----- 512.0
Configuration
I need to add ssh for it to be usable as a base system.
I found the root cause of this problem. Skip to the double lines for the fix.
The error trying to start ssh:
Starting OpenBSD Secure Shell server: sshd
PRNG is not seeded
The error is caused by no /dev/urandom being present. Mount the disk image in loop.
nano /etc/init.d/local
and add this
#!/bin/bash
cd /dev
./MAKEDEV mem
./MAKEDEV urandom
/etc/init.d/ssh start
Make it executable and link it to start on boot. This assumes you are in the folder where you mounted to.
chmod +x etc/init.d/local
ln -s etc/iniit.d/local etc/rc3.d/local
Edit etc/init.d/makedev. In the “start)” section, after the “test” statement, add this line. This creates devices and corrects some permissions. Most importantly it lets openssh start, and prevents odd problems.
cd /dev && ./MAKEDEV zero
Change the permission of /tmp
chmod 1777 tmp
As long as you have it mounted, change the ssh setting so you can login as root.
mv etc/ssh/sshd_config{,~} && sed 's/PermitRootLogin no/PermitRootLogin yes/' etc/ssh/sshd_config~ >etc/ssh/sshd_config
Now a few additions to make the deployment easier.
Copy the edited makedev to root/etc. This will become a directory to hold /etc files that should be retained when we rsync /etc
mkdir root/etc/init.d
cp etc/init.d/makedev ~/etc/init.d/
I also put the “flip” script in root since I am using this everytime. vm flip filesystems
Now boot the vm.
Running
The first thing you should do is change the root passwd. The default password of the jailtime.org images is password.
I changed the virtual interface so that it will use the same MAC address everytime.
vif = [ 'bridge=xenbr0,mac=00:16:3e:xx:xx:xx' ]
00:16:3e is the MAC vendor code for Xen. The last 3 bytes should be unique, especially on your network. You can use this to generate 3 unique hex bytes.
dd if=/dev/urandom bs=1 count=3 2>/dev/null | od -tx1 | head -1 | cut -d' ' -f2- | tr -d ' ' | tr '[a-f]' '[A-F]'
I booted the vm and edited /etc/apt/sources.list. I changed “sarge” to “stable” to upgrade to etch. I then ran:
apt-get update
apt-get dist-upgrade
...
Do you want to upgrade glibc now? [Y/n] Y
Do you wish to restart services? [Y/n] Y
Started getting this error:
4gb seg fixup, process dpkg (pid 1292), cs:ip 73:4003ede1
Because this requires changes in /etc and I rsync that directory, I will save the fix for the vms.
I renamed the img and config to debian.3-1 to debian.4-0 to reflect the new version, and changed the config to correctly load the image.
Error during boot
Setting hostname to 'debian_pristine'...hostname: the specified hostname is invalid
I do not plan on fixing this since I will be changing the hostname on deployment.
tar’ing the image as the deployment model.
Automatically starting domains
link to article
If you would like a domain to start automatically when the (dom0) system is started, move the domain configuration to the /etc/xen/auto directory. For instance:
ln -s /xen/debian/debian/cfg /etc/xen/auto/
I will likely try linking to that directory.
LVM
link to article
Create a logical volume of size 4GB named `myvmdisk1′:
# lvcreate -L4096M -n myvmdisk1 vg
You should now see that you have a /dev/vg/myvmdisk1 Make a filesystem, mount it and populate it, e.g.:
# mkfs -t ext3 /dev/vg/myvmdisk1
# mount /dev/vg/myvmdisk1 /mnt
# cp -ax / /mnt
# umount /mnt
Now configure your VM with the following disk configuration:
disk = [ 'phy:vg/myvmdisk1,sda1,w' ]
I am going to name my LVMs after the host and mount point so I can identify them. I use Pooh characters for my servers, so my first will be /dev/VolGroup00/kanga-var and kanga-tmp
kanga vm
Links:
CentOS 5 with Debian 3.1 and 4.0 guests:
http://juanjosec.blogspot.com/2007/06/migrating-xen-installation-from-fc5-to.html
Xen and LVM VBD (Virtual Block Devices)
http://www.linuxtopia.org/online_books/linux_virtualization/xen_3.0_user_guide/linux_virualization_xen_user_44.html
Installing a Xen
DomU on
CentOS 5
http://wiki.centos.org/HowTos/Xen/InstallingCentOSDomU