Negotiation, Negotiation, Negotiation

UK Property Investment news and comments from Mark Harrison of YourPropertyExpert.com

Server virtualisation on Linux – Ubuntu on Ubuntu with KVM

Posted by markharrison on April 20, 2008

It’s a long time since I’ve written an overtly technical post, but after the last 24 hours, I thought I’d learnt enough obscure stuff that I would make notes in case I have to do it again :-)

Over the past few years, the complexity of the various websites I run, particularly the way that www.yourpropertyexpert.com and www.yournegotiationexpert.com interact on the one hand (so that people signing up to one can sign up to the second easily) and www.fabinsurance.com and www.mythinge.com share suppliers so that customers of both sites can get access to a wider range of bids has led to, well, complexity.

Add in the needs for resilience and a development environment, and we’ve ended up with six servers, which for a pair of businesses that only have ten people working for them seemed a bit much.
Hence, last week, I decided to buy a new server to replace several of the old ones.

There were two bits of good news

  • server prices have crashed dramatically in the last few years
  • server virtualisation has meant that we can run several servers on the same piece of hardware

Server virtualisation is a really nice trick – it means that, from a security and resilience point of view, you can make one bit of hardware support several “virtual servers” – even ones running different operating systems, so we can run a Windows server, a Database Server (MySQL on Ubuntu), and a pair of web servers (Apache on Ubuntu) on the same box at the same time. In addition, the way it works means that it we start running out of capacity, we can “move” one of the virtual servers to new hardware very quickly – like, in minutes and with zero downtime!

Such sophistication comes at a price of complexity, and it took me most of the weekend to get things up and running. For the techies who read this blog, here are some gotchas:

Ubuntu is moving over to KVM rather than VMWare as the Virtualisation platform of choice. Because of that, I decided to move over to KVM for this server. KVM operates well as a hypervisor / accelerator for Qemu.

Problem: KVM, at least as far as I can tell, doesn’t full emulate real-mode i386 instructions (or at least, not on the 64-bit Athlon I’m using). Sadly, real-mode is needed during the (guest) installation of Ubuntu LTS server [1] since the Ubuntu install splash screen is a real-mode thing.

Solution: Install Ubuntu guest just in Qemu without KVM hypervisor accelaration, then, once the installation has completed, restart the VM with KVM rather than Qemu

Problem: Installing a guest O/S under Qemu is painfully slow, and I have to do it about six times.

Solution: Install the guest O/S once, then copy the .img file multiple times. The memory space allowed is independent of the .img file, so you can spin up your VMs with different footprints.]

Problem: Copying the .img file means that all of your VMs have the same hostname.

Solution: Edit /etc/hosts  and /etc/hostname on each… much quicker than installing 6 times.

Problem: By default, Qemu runs internal NAT within the host box to give each VM an address in part of the 10.0.0.0 space, which means you can’t route into those servers.

Solution: On the host O/S, replace the reference for eth0 (or whatever) with a bridge network, so that eth0 is now a shared bridge (level 2) and the O/S on the host uses a new interface – br0 – which is simply one of the interfaces that shares the bridge (the others being on the guest O/Ss). You have to run kvm with a parameter  like “-net nic,macaddr=00:16:3e:00:00:0x”, with a different value of x from 0 to 9 for each VM

Problem: If you’ve copied the .img file over, then at install time, Ubuntu has locked down the MAC address that happened to be assigned at that point on each of the guest O/S… hence if you specify the MAC address as a parameter into kvm at run-time, it doesn’t match, and the (virtual) interface on the guest fails to initialise.

Solution: On each guest O/S, go into (on Ubuntu guests at least), /etc/iftab, and edit the MAC address assigned there to match the one that you’re hand-assigning in KVM. I’m sure there should be a better way to do this, but if there is, I can’t find it :-( At this point, it’s worth creating either a shell script, or something else that spins up the right .imgs with the right MAC addresses :-)

[1] To be fair, this problem may go away with the new version of LTS server due for release NEXT WEEK.

About these ads

4 Responses to “Server virtualisation on Linux – Ubuntu on Ubuntu with KVM”

  1. richard said

    I we copy the .img file, to avoid re-install OS, it will create a file with the MAX size even if it is not “full allocated”.
    I am looking for a solution to copy only the allocated size…

  2. I don’t have a good answer, I’m afraid.

    I created the “max size” for each guest straight away, on the basis that it means that I won’t run into subsequent allocation problems.

    Because the things I’m serving are inevitably database-driven, on the database vhost is ever big – the webheads are typically only about 6Gb.

  3. richard said

    Markharrison,

    My goal is to be able to save “guest” servers, in case of “host” disk failure. And “guest” disks are 50M “growable” each.

    I can go for qcow2.img files, or on LVM raw block device. Which ever is better for saving. But I have no clue for copying only the useful sectors.

    And if I go for smaller “guest growable disk”. The day it is too small, how do I get a bigger .img file ? (for LVM raw block device, that would be not too much trouble)

  4. I we copy the .img file, to avoid re-install OS, it will create a file with the MAX size even if it is not “full allocated”.
    I am looking for a solution to copy only the allocated size…

    Not sure about this, but “cp –sparse=always” might help you

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

%d bloggers like this: