It’s a long time since I’ve written an overtly technical post, but after the last 24 hours, I thought I’d learnt enough obscure stuff that I would make notes in case I have to do it again 🙂
Over the past few years, the complexity of the various websites I run, particularly the way that www.yourpropertyexpert.com and www.yournegotiationexpert.com interact on the one hand (so that people signing up to one can sign up to the second easily) and www.fabinsurance.com and www.mythinge.com share suppliers so that customers of both sites can get access to a wider range of bids has led to, well, complexity.
Add in the needs for resilience and a development environment, and we’ve ended up with six servers, which for a pair of businesses that only have ten people working for them seemed a bit much.
Hence, last week, I decided to buy a new server to replace several of the old ones.
There were two bits of good news
- server prices have crashed dramatically in the last few years
- server virtualisation has meant that we can run several servers on the same piece of hardware
Server virtualisation is a really nice trick – it means that, from a security and resilience point of view, you can make one bit of hardware support several “virtual servers” – even ones running different operating systems, so we can run a Windows server, a Database Server (MySQL on Ubuntu), and a pair of web servers (Apache on Ubuntu) on the same box at the same time. In addition, the way it works means that it we start running out of capacity, we can “move” one of the virtual servers to new hardware very quickly – like, in minutes and with zero downtime!
Such sophistication comes at a price of complexity, and it took me most of the weekend to get things up and running. For the techies who read this blog, here are some gotchas:
Ubuntu is moving over to KVM rather than VMWare as the Virtualisation platform of choice. Because of that, I decided to move over to KVM for this server. KVM operates well as a hypervisor / accelerator for Qemu.
Problem: KVM, at least as far as I can tell, doesn’t full emulate real-mode i386 instructions (or at least, not on the 64-bit Athlon I’m using). Sadly, real-mode is needed during the (guest) installation of Ubuntu LTS server [1] since the Ubuntu install splash screen is a real-mode thing.
Solution: Install Ubuntu guest just in Qemu without KVM hypervisor accelaration, then, once the installation has completed, restart the VM with KVM rather than Qemu
Problem: Installing a guest O/S under Qemu is painfully slow, and I have to do it about six times.
Solution: Install the guest O/S once, then copy the .img file multiple times. The memory space allowed is independent of the .img file, so you can spin up your VMs with different footprints.]
Problem: Copying the .img file means that all of your VMs have the same hostname.
Solution: Edit /etc/hosts and /etc/hostname on each… much quicker than installing 6 times.
Problem: By default, Qemu runs internal NAT within the host box to give each VM an address in part of the 10.0.0.0 space, which means you can’t route into those servers.
Solution: On the host O/S, replace the reference for eth0 (or whatever) with a bridge network, so that eth0 is now a shared bridge (level 2) and the O/S on the host uses a new interface – br0 – which is simply one of the interfaces that shares the bridge (the others being on the guest O/Ss). You have to run kvm with a parameter like “-net nic,macaddr=00:16:3e:00:00:0x”, with a different value of x from 0 to 9 for each VM
Problem: If you’ve copied the .img file over, then at install time, Ubuntu has locked down the MAC address that happened to be assigned at that point on each of the guest O/S… hence if you specify the MAC address as a parameter into kvm at run-time, it doesn’t match, and the (virtual) interface on the guest fails to initialise.
Solution: On each guest O/S, go into (on Ubuntu guests at least), /etc/iftab, and edit the MAC address assigned there to match the one that you’re hand-assigning in KVM. I’m sure there should be a better way to do this, but if there is, I can’t find it 😦 At this point, it’s worth creating either a shell script, or something else that spins up the right .imgs with the right MAC addresses 🙂
[1] To be fair, this problem may go away with the new version of LTS server due for release NEXT WEEK.
Negotiation, Negotiation, Negotiation 