Tuesday, November 29, 2011

Virtualization options

Anyone who's heard of virtualization probably has discovered there are many different tools out there to help install and run multiple operating systems on a single computer. While many of these systems are similar, the truth is that there are radically different approaches to virtualization that have developed over the years, with advantages and disadvantages for each method. Here's a quick rundown on popular virtualization approaches and technologies, and reasons why you might want to pick one over another. I've tried to keep the terminology consistent below, although many in the industry will prefer different terms. The most neutral term I could think of for the virtual machines themselves is "guest". The "host" is the operating system that guests run under.

Emulation or "Full virtualization"

Emulation is one of the oldest forms of virtualization and exists in many forms, from systems that emulate every aspect of a computer, including the CPU, to versions that use features of the host CPU to sandbox the hosted operating system and which use software merely to emulate the rest of the hardware. Almost all modern emulators use the latter approach if they can get away with it.

The primary advantage of emulation is that it requires virtually no support within the guest operating system, which is made to believe it's been installed on a normal computer with no special requirements.

The disadvantages of emulation are numerous. Even with CPU support, it's slow and inefficient. As an example, if an application on a guest needs to access the network, the data it sends needs to go through two device drivers (the one for the emulator, and the real one), and a virtual device emulator inbetween. Many CPUs don't support virtualization natively and thus can't be used to run the faster emulators. Emulators typically have very clumsy boundaries in terms of limiting CPU, memory, or disk usage.

Despite the disadvantages, emulators tend to be the most popular choices for virtualization today, largely because of the lack of a need to have the writers of the guest operating system be involved in making their systems work. Less of a problem for free operating systems, where the code can be modified, running multiple proprietary operating systems usually requires emulation unless the vendor cooperates with the virtualization system you want to run.


Para-virtualization is probably my favorite virtualization system, although it comes at some costs. In para-virtualization, the guest operating system is written such that it is aware it doesn't have complete control over the hardware, and instead cooperates with the "real" host operating system.

Implementations of para-virtualization can be dramatically different. One early system, User Mode Linux, and a related project called Cooperative Linux, allowed a Linux-based operating system to run under another operating system such as GNU/Linux or Windows. The UML kernel would simply talk to the underlying operating system and have it do the heavy lifting work.

A more advanced, generic, system, and my choice when it's available, is Xen. Xen is a host operating system that provides the bare minimum for hosted operating systems, handling starting and stopping operating systems, parceling out memory and CPU resources, and telling each instance what hardware it is allowed to access. This operating system is called a hypervisor. Typically, by default, all resources are assigned to a special guest operating system called the "Dom0", and that operating system provides basic networking and other services to the other guests.

Xen is relatively efficient, with guests able to talk directly to the hardware without going through emulation layers; although the counter to that is that it's generally complicated to set up, with the admin needing to have a high degree of knowledge about the hardware if the admin wants to ensure each guest runs efficiently. The "easy" ways to administer a Xen system can result in a very slow system. And like Emulation, Xen has relatively clumsy boundaries to be set when you need to assign resources to specific guests.

The fact each Xen guest is aware it's running in a virtual environment means that you get certain huge improvements. It's easy, for example, to reboot a computer (as in shut it down, cut the power, wait ten seconds, and start it back up) without actually killing the Xen guests, which - beyond seeing the time move forward suddenly - will act as if nothing has happened. More advanced Xen servers make it easy to migrate guests from one physical computer to another without shutting anything down. To be fair, more advanced emulators have the same capabilities, but Xen does it with the full cooperation of the underlying operating system, which in theory, at least, makes the entire process more reliable. The underlying system is expecting an outage, and so doesn't get upset about it.

Para-virtualization with a hypervisor like Xen's is probably the best compromise between application transparency (applications are completely unaware that they're on a virtual platform) and efficiency. However, the requirement for operating system level support means it's hard to run a fully para-virtualized VM system. Xen supports a fall-back mode where emulation is used to run operating systems that do not support its hypervisor, but obviously the moment you use it you lose the advantages of the para-virtualization approach and might as well use something more geared towards emulation.

Operating system level virtualization (or virtual virtualization!)

An extremely common virtualization scenario is where a single computer serves large numbers of guests that all run the same operating system (or operating system kernel.) This comes about because in Enterprises there's usually a deliberate decision not to diversify too far from a standard platform, because amongst nerds like me, there's usually a favorite system we're comfortable with, and because ISPs that offer VPS services usually offer hundreds of customers the same basic systems.

Unix-style operating systems have offered some tools that provide the ability to host different environments running from a common kernel for quite a while, although the concept has only recently become advanced enough for system administrators to take it seriously. The original system, chroot, permitted a tree of processes to see a branch of the core file system as its root file system, and you'd load up that branch with all the files that make up a Unix system. While it worked for some applications, chroot is too crude to work for anything but the simplest applications. Networking, for example, is still shared amongst environments.

The BSD branches of Unix implemented a system called "jails", which took the chroot concept and added all the other aspects of an operating system to ensure that each "jailed" process tree would really have an entire environment that it could play in without ever seeing any evidence it was part of a bigger whole.

It's taken a while for Linux to adopt the same concept. An early version of the concept is OpenVZ, which is used by many VPS providers, and does exactly what you'd expect from the above. A single kernel runs multiple environments, each seeing a subtree of the file system as being their root file system, each seeing their own network devices, and so on. OpenVZ required a patched Linux kernel, and the patches were never integrated into official Linux, and so it had limited support, but it's proven to be very popular nonetheless.

What can OpenVZ run? Well, essentially any operating system that can use the version of the Linux kernel running on the host. The operating system usually requires some small modifications, so that, for example, it doesn't start checking the disk for errors when it starts up, but once up, most applications will never know the difference between the hosted operating system and the main operating system. You can run a combination of different operating systems as long as those systems all support the kernel you're running.

While OpenVZ isn't supported by the Linux developers, the OpenVZ and Linux developers have been cooperating on a very similar project called LXC. LXC is a Linux-friendly version of the same concept, and can run on an unmodified kernel, because all the modifications needed are being integrated into the main Linux tree. LXC uses a Linux technology called "cgroups" that's designed to replace the functionality OpenVZ's kernel modifications added.

It's important to understand that, while reliable, LXC is not considered production quality yet. That doesn't mean you can't use it, or even entrust your important data to it - the very nature of LXC means it's as reliable as the host operating system. However, you need to understand that LXC has certain security holes in it that are going to take time to fix. Those holes will not affect a normal application, even a bug ridden one, but they do mean if the server is public facing, and a hacker is able to gain access to it, that hacker can theoretically gain access to the host computer too.

Operating system virtualization has advantages and disadvantages over other virtualization systems. Like para-virtualization, it has no need for special hardware support, as the host operating system inherently supports the concept natively. The technology is generally much, much, faster and more efficient than the alternatives, as there's no complexity at all between the operating system and the hardware it runs on. And it's much, much, easier to allocate resources, with it being perfectly possible to increase memory and disk space in real time. You can even give all your guests unlimited resources, and start to rein in any that cause problems when the problems start to show up.

Major downsides? There's no easy way to save a guest instance, so all the fancy Xen functionality where you can reboot your computer or move guests between computers without shutting down the guests themselves is simply impossible. The instances are too ingrained in the host operating system for it to be possible to separate them out and save them. And that also means that critical operating system updates, such as updates to the kernel, require you restart the entire system, the host and the guests.

Another downside is that, as yet, no operating system level virtualization platform for Linux is completely, 100%, transparent. One issue, for example, is that because all memory, real and virtual, is shared, it's difficult to give each environment a picture of how much memory is available. This has caused some applications that specifically check for virtual memory (swap memory) to fail because the Linux APIs that report on available memory cannot sanely report what's available in this kind of environment.

So what should you use?
  • If Xen is an option to you, I encourage you to check it out especially if you're trying to run multiple servers.
  • For testing operating systems or running an additional desktop operating system, I recommend an emulator called VirtualBox. It's free, supported by  Oracle (it's a Sun technology), and it's extremely good. It runs under Windows and Linux.
  • Ubuntu users who need to run servers probably need to look into KVM at the moment. There's more native support within Ubuntu for KVM. I don't particularly like the concept, it's very much emulator oriented, but it might suit what you want to do if you can't get Xen to do it.
  • If you have limited hardware, and you're not too concerned about security, LXC is a pretty decent option. If you're concerned about security, I encourage you to look at OpenVZ. For the most part, when LXC is finished, it should be capable of running OpenVZ VMs unchanged, so the only problem you'll run into with OpenVZ is the lack of support. And that whole "Need to reboot all servers from time to time" thing that applies to LXC too.
 Have fun!

Sunday, November 27, 2011

Setting up network bridging for VMs in Ubuntu

I had cause to set up a network bridge in a new Ubuntu system this weekend. For those unsure of the concept, the idea of bridging (in this context) is to connect virtual machines to a network. The name comes from the hardware concept of bridging, where multiple networks are glued together by the use of a "bridge", a hardware device that does all the routing, mostly transparently.

In Linux, there's a "bridge network device" set up to support providing networking to VMs and other more advanced networking concepts.

In the process of configuring my bridge, I made numerous mistakes based upon a hazy idea of what I was setting up. So, if you need to set up bridges for LXC, KVM, or other systems, this is what you need to know and do.

1. What the bridge device is

The bridge device appears as "br0" (like "eth0") on the host machine and it all but replaces eth0 as your primary network device. It handles all routing to and from the external network, and within the machine to the VMs that are using it. How it appears to each VM depends on the virtual machine system you're using - typically, those VMs will think they have a device called "eth0" that's connected to the external network, but the network they're connecting to is the one controlled by the bridge.

Each bridge device (br0, etc) is associated with a "real" device like eth0. If the bridge device sees traffic coming in from the real device addressed (at a low level) to one of the virtual machines or to the host, it routes traffic to that VM or the host. Likewise,  it routes internal traffic internally, and routes traffic from VMs or the host to external addresses via the "real" device. The addressing is via standard Ethernet MAC addresses, not IP addresses - those are higher level addressing concepts, and the bridge device tries to be protocol neutral. The overall effect is one where the network controlled by the bridge is merely an extension of your existing Ethernet network.

2. How you configure it

I'm not going to cover the VM side as that'll depend on your virtual machine manager, but here's the key points for setting up the system. I'm assuming that you DON'T want NAT - virtually none of my readers will want NAT given their external router already handles that for them. The only exception I can think of is if you're experimenting on your work PC and you aren't the network admin and don't want to particularly bother him or her. Anyway, here goes:

Step 1: You need to disable Network Manager and install bridge networking.

Completely. Network Manager isn't really what you want running on a server in the first place. (To be honest, I've never liked the tool, I see why it's there, but it's not particularly reliable in my experience.

Disable it using:

  $ sudo -s
 <your password>
 # /etc/init.d/NetworkManager stop
 # apt-get remove network-manager

sudo -s makes you root until you exit, I'm going to assume you remain root for the remainder of this article. Be careful! Should I ask you to reboot, remember to sudo -s upon logging back in if you need to repeat any of the steps in Step 1 or Step 2.

You'll also need to add the bridge support. You can do this using:

 # apt-get install bridge-utils

Step 2: Set up /etc/network/interfaces and other configuration files

I'm going to make the following assumptions here.
  • You're using a static IP address. For the examples, it's
  • Your router is
  • Your DNS server is
  • Your netmask is which makes your broadcast address 
You'll need to modify the examples accordingly for your situation. IPv6, incidentally, should "just work" if you're using stateless configuration - at least for the bridge and host, your VMs are another matter but that's because of peculiarities in the VM implementations, not the bridge system.

First, edit your /etc/resolv.conf and put in something sensible like:


Second, edit your /etc/network/interfaces file and make it look like this (with the IP addresses changed to suit your network, obviously.)

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto br0
iface br0 inet static
   bridge_ports eth0
   bridge_stp off
   bridge_maxwait 5
   post-up /usr/sbin/brctl setfd br0 0 addif br0 eth0

The biggest thing you'll probably notice about the above is that there's no configuration for eth0. This is intentional. Other than the interface coming up, you don't want it to have anything assigned to it. The bridge device is going to be the one that receives traffic for your host, it'll handle the routing to and from the Ethernet port.

Step 3: Reboot and check

Now, I know you're probably going to ask "Can't I just test it without rebooting?", well, you can, but you will not know for sure that everything's going to work the next time the system goes down. Given your network is all screwy right now anyway, you're not going to lose anything by doing this. So reboot, and upon restart check that your networking is doing what you think it should.

Type the following to verify the network is up and running properly:

 $ ifconfig

This should show, at minimum, the following devices as being "up":

br0 - which should be configured as above
eth0 - which should be up, but have no IPv4 address, and if it has an IPv6 address, it should only be the link-local address (begins with fe80:)
lo - which should be configured as

$ route -n

This should show, at minimum, the following routes:

Destination: Gateway, Genmask, Iface br0
Destination: Gateway, Genmask, Iface br0

$ ping www.google.com

This should work. If it can't find the host, your /etc/resolv.conf is wrong. If the destination host is unreachable, then there's probably a major problem with your configuration so check the numbers and, obviously, make sure your Internet is working fine from another machine on the same network.

Step 4: Configure your VMs

At this point your bridge is working, so you can get to the next messy stage and start configuring your VMs to use it. That's another story, and what you do will depend upon what VM system you're using. Good luck!

Friday, November 25, 2011

Ideapad K1 First Impressions (some updates)

I finally took the plunge and bought an Android tablet this week. It's a Lenovo Ideapad K1, which is essentially a "standard" (ie 10", 1280x800) Honeycomb tablet. As my regular readers know, I'm still unsure about the whole tablet thing, and to be quite honest with you, this device hasn't made me any more positive about the idea.

Lenovo problems

There are four issues with the tablet that are specifically to do with Lenovo's design decisions.

1. Update hell

The first is that the operating system that's pre-installed is horribly out of date, and is bug ridden. To upgrade it, you have to install every single system update that's ever been made. Each update installs, the system is rebooted, and then you repeat the process until you have Honeycomb 3.2. It's horrible, laborious, and completely unnecessary.

2. Unnecessary proprietary dock connector

Popularized by Apple, the proprietary dock connector is a single socket that's essentially a USB socket that's been made wider, uglier, and incompatible with standard USB accessories. Each manufacturer has their own design, and typically the proprietary dock connector is provided instead of, rather than in addition to, a regular USB connector.

It's a stupid, consumer-hostile, system that only exists so that the manufacturer can create, or license, overpriced accessories. You can't use a standard charger or USB cable, should you leave the official versions at home. The K1 has a proprietary dock connector, it has no standard USB connectors. Manufacturers who play these games are demonstrating clear contempt for their customers, and this is always an extremely good reason to avoid them.

3. No USB charging

Want to charge your smartphone? Like most devices that charge using a USB port, all you generally need is the USB cable, so you can plug it in to your computer. This also ensures you have a neat way to keep it around - plug it in, and while you have complete access to your files on your computer, the device is also kept fully charged.

But not the Lenovo K1. The K1 needs to be plugged into the power supply to charge. It will not be charged from your computer's USB port. And, because the same connector is used for USB and charging, this means you cannot charge it and access the files on it via USB at the same time. Well, there's probably an expensive accessory you can buy but...

4. Locked down

Android has a clear hierarchy of accessibility for system tinkerers who want to do advanced things on their devices.

The one that almost all devices support, and thankfully the K1 does too, is the ability to install any application from any source.

But there's a set of other levels too that are generally restricted, to some extent, because if you were to access them on a phone (that is, a device with an almost direct ability to pump out data onto a congested mobile phone network) you could, in theory, do stuff that would affect people other than yourself in a very negative way. These levels are:
  • Root - a level that allows users to install more advanced software
  • Firmware - a level that allows users to install version of Android that didn't come from the manufacturer, for example, CyanogenMod.
The K1 is a tablet. There's no reason for either of the above levels to be disabled on a tablet, any more than you'd expect them on a PC. A tablet is not a mobile phone - at least, this one isn't. But, in their infinite wisdom, Lenovo has, indeed, put in locks to make it harder to access these levels of security.

Which is not to say that people haven't found ways to bypass the above, but right now the process is ugly, and involves exploiting loopholes in the update process. Given Honeycomb is damaged goods - an OS always presented as a prototype version of Android, and whose design is so embarrassing that Google, while making it open source, have gone out of their way to make it difficult to build a real version of, the fact the ability to update the operating system to, say, Icecream Sandwich, without Lenovo's express involvement, has been made difficult is a big red flag.

Honeycomb problems

Honeycomb itself isn't a bad tablet environment. It's fairly open, it looks good, it's easy to use - well, once you get the hang of the buttons. But there are some major issues, and not all of them are going to be fixed in post-Honeycomb operating systems.

1. Poor support

The big argument for using the iPad is that Android doesn't have high quality support in the tablet space, and that's true. I found the following:
  • Most apps don't even bother to consider the tablet form factor. Twitter is a glaring example. It's a horrible app to use on a 1280x800 screen.
  • Many apps don't install at all. (Update: app previously mentioned here, Yahoo Mail, now works, albeit it's a "mobile app made full screen" thing.)
  • Many apps that support the form factor have been restricted from running on entirely compatible hardware. This includes half of Amazon's stuff, Hulu Plus, Qik (the video conferencing system T-Mobile and Sprint are popularizing), and many others. It's usually because of "exclusivity" agreements - but these agreements are often stupid. Qik, for example, depends upon network effects that can only be had via wide availability, so why the hell can it not be installed on any random Honeycomb tablet?
Websites that support a mobile view tend to load in that view, which is pretty ridiculous given the web browser has a larger screen than most Netbooks. And there's no way to tell the browser to pretend to be a non-mobile browser.

2. It's slow.

Honeycomb is chronically slow. Scrolling a web page typically results in chunks being rendered after you've stopped scrolling. Returning to the home screen often takes seconds. Sliding left and right typically causes the screen to move in a jerky fashion. The K1 is a pretty powerful device, it boasts a dual core Tegra, a device that can give most of the better Netbooks a run for their money. The fault has to lie with Honeycomb, either with Google's design or Lenovo's specific implementation.

Update: Just to prove the point, my testing with Grand Theft Auto 3 proved that there's a pretty spiffy CPU under there.  This is a full 3D game, and it's beautifully smooth, running at the K1's full resolution. So why's the web browser so awful? I used use a 233MHz Thinkpad with a 1024x768 screen and a really poor GPU back the early 2000s, and early Firefoxes on that could scroll without problems!

3. Printing

Lenovo has installed some third party support for printing on the K1, but it's not integrated with the operating system, and it's actually a demo - you have to register to make it work. The operating system itself contains no native printing support. Google offers an API called "Google Cloud Printing", supported by some of their apps, which essentially involves a ridiculous system whereby you... I'm not making this up... you run Google Chrome on a Windows PC or Mac on the network with the printer you want to access, go into Settings, select the Cloud Printing option, add a printer from your network, make sure you're online, and then select this printer on your tablet, which uploads the thing that needs to be printed to Google, which in turn sends it back to the web browser on the PC or Mac which then prints the page.

What. The. Hell?

There's nothing in theory to prevent the tablet from running something more like a standard operating system, where printers on your network are accessible, and you just select them and print with them, maybe with some kind of streamlined device driver system, but, well, no, Google wants you to upload anything that needs printing to them, because... because it's a cloud! Woohoo! Cloud computing leveraging synergies for Web 2.0 platforms, yeah!

If the tablet is supposed to be a standalone device that can be used for productivity, it needs a proper system for interfacing with printers. As far as I'm concerned Honeycomb doesn't have this.

4. Google Docs

The productivity side of Honeycomb is supposed to be Google Docs, and I like Google Docs... on a PC. On a tablet the app is almost identical to its mobile phone cousin. You can edit files, but it's a pain, you don't get to see any formatting, and it's not clear to me that any consideration was given to the form factor.

If the tablet is supposed to be the next generation of personal computing devices, you'd have expected Google's own office suite to at least be up to the task.


I hope Icecream Sandwich is better than this. All I can say right now though is that Windows 8 is probably going to take over the entire tablet space. Honeycomb is a prototype, but many of the decisions Google has made (such as Cloud Printing), and the fact even major groups like Twitter, Yahoo, and whoever writes Google Docs, isn't taking the platform seriously, makes me think that this isn't where the industry will ever go.