Linux KVM (kvm-70) on IBM Open Client 2.2

The Linux kernel-based virtual machine (KVM) is a great way for virtualization on computers running Linux. It requires virtualization support by the host processor (most modern x86 CPUs have this) and a kernel module. The kernel module can be built from the KVM sources.

Unfortunately, compiling the module on the IBM Open Client 2.2 distribution doesn’t work out of the box. Instead, a patch is required. The patch is an extended version of this commit to the KVM repository and applies against the KVM-70 release tar ball.

Networking

The KVM networking documentation lists brctl(8) and tunctl(8) as requirements for a bridge between the host and the guest. On the Open Client distribution, the brctl utility is part of the bridge-utils package whereas the tunctl tool is part of uml-utils – on other distributions, that is. However, there is a Fedora Core 9 package available which seems to work.

Before starting the KVM guest, make sure that the tun kernel module is loaded. These are the steps I use to start the guest:

$ sudo modprobe tun
$ MACADDR=`genmac`
$ sudo qemu-system-x86_64 -hda freebsd-7.0.img 
   -net nic,macaddr=$MACADDR -net tap,script=qemu-ifup

Note that the genmac and qemu-ifup scripts are the examples from the KVM documentation.

NAT on the bridge

Because I cannot put the KVM guest on the host network, I need to do NAT on the host. I’ve found this quick tutorial on NAT with iptables. The four steps are:

# echo 1 > /proc/sys/net/ipv4/ip_forward
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
# iptables -A FORWARD -i eth0 -o tap0 -m state 
     --state RELATED,ESTABLISHED -j ACCEPT
# iptables -A FORWARD -i tap0 -o eth0 -j ACCEPT

Also, make sure the tap0 interface has an IP address:

$ sudo ifconfig tap0 192.168.0.1/24

PowerPC Device Tree Compiler

The Device Tree Compiler (DTC) project is hosted at OzLabs. The website seems to be unavailable at the moment, but the git repository at git://ozlabs.org/srv/projects/dtc/dtc.git seems to work.

Cross-building the tools works fine. This is what I did:

$ export CC=ppu-gcc
$ make

This will create the dtc and ftdump tools which can then be copied to the target machine.

Cross-compiling the Linux kernel

I need to cross-compile a PowerPC Linux kernel on an x86 laptop. I’ve found instructions on how to compile (not cross-compile) the Linux kernel at this website. Further, there is a post to a mailing list here which shows how to cros-compile the kernel. The mailing list post mentions a ccontrol file, but I have no clue what that is. Luckily I’ve found this blog post, which seems to be more accurate.

Building a PowerPC Cross Compiler

I need to build my own cross compiler which will run on i386 and produce 64-Bit PowerPC Binaries. I’ve found a pretty neat introduction to building a cross compiler on IBM’s developerWorks site (registration required). The tutorial isn’t a step-by-step guide, but it helped me a lot.

The basic procedure for building a cross-compiler is:

  • Obtain headers specific to the operating system
  • Build binutils for the target platform
  • Build a bootstrap compiler
  • Build a standard C library for the target
  • Build the final cross-compiler, using the C library just built

The developerWorks tutorial doesn’t mention this, but the first three steps can easily be run in parallel. Anyways, before starting, I’ve set these environment variables:

$ export TARGET=powerpc64-unknown-linux-gnu
$ export PREFIX=/opt/crosscompiler
$ export TARGET_PREFIX=$PREFIX/$TARGET
$ export PATH=$PATH:$PREFIX/bin

Obtaining Linux-specific Headers

I’ve followed the developerWorks tutorial on this one: Downloaded and extracting the Linux kernel sources, then copied the relevant files. Here are the commands I ran:

$ wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.25.9.tar.bz2
$ tar xvjf linux-2.6.25.9.tar.bz2
$ cd linux-2.6.25.9
$ make ARCH=powerpc CROSS_COMPILE=powerpc64-linux- menuconfig
(configure options, but tweaking isn't neccessary)
$ mkdir -p $TARGET_PREFIX/include
$ cp -r include/linux $TARGET_PREFIX/include
$ cp -r include/asm-powerpc $TARGET_PRFIX/include/asm
$ cp -r include/asm-generic $TARGET_PREFIX/include

If you read to the end of this post, then you’ll realize that this step wouldn’t be required (for now).

Building GNU binutils

I’m using GNU binutils 2.18, available from the GNU website. These are the steps required to build binutils.

$ wget 
$ tar xjvf
$ ./configure --prefix=$PREFIX --target=$TARGET --disable-nls -v
$ make all
$ make install

While building binutils did take a while, it wasn’t as long as the tutorial makes you believe. On a IBM Thinkpad T60p built around a Centrino Duo CPU running at 2.16 MHz it took less than 10 minutes. Also note the last command ("make install"), which is missing from the developerWorks tutorial.

Building a bootstrap compiler

For my project I need GCC 4.x, the latest version at the time of writing is 4.3.1 which is available from a GNU mirror near you. Downloading and extracting is easy:

$ wget ftp://ftp.gwdg.de/pub/misc/gcc/releases/gcc-4.3.1/gcc-4.3.1.tar.bz2
$ tar xjvf gcc-4.3.1.tar.bz2
$ cd gcc-4.3.1

Here are the steps required to build a bootstrap compiler.

$ ./configure --prefix=$PREFIX --target=$TARGET --without-headers 
  --with-newlib -v
$ make all-gcc
$ make install-gcc

This took longer than building binutils, however it took less than 30 minutes (as opposed to the hours the tutorial talks about).

Building the GNU C Library (glibc)


$ CC=${TARGET}-gcc ./configure --target=$TARGET --prefix=$PREFIX 
  --with-headers=${TARGET_PREFIX}/include

Unfortunately, this command failed with the following error:

(...)
checking whether __attribute__((visibility())) is supported... no
configure: error: compiler support for visibility attribute is required

However, this isn’t important as I won’t need a standard C library for now – I’m building with -ffreestanding and -nostdlib anyways. Therefore I’ve decided that I won’t pursue this futher but may come back later.

Qemu, FreeBSD and coreboot

Since my attempts at getting Qemu running on Mac OS X were unsuccessfull, I’ve decided to go a different route. I’m now trying to build it on FreeBSD again.

Some time ago, I took some notes on how to get Qemu running on FreeBSD and added them to the coreboot wiki. Then some time later, I tried to build Qemu per those instructions, but had to discover that the port had been updated to a newer version of Qemu and no longer works.

So I’ve decided so maintain my own copy of the Qemu sources. The goal is to have a working version of Qemu which can be built on FreeBSD and can run coreboot. The repository is at svn+ssh://phs.subgra.de/usr/svnroot/qemu, a web frontend is available at https://phs.subgra.de/svnweb/index.cgi/qemu. Since the repository is not (yet?) public, here is a tar-ball of the latest version.

Building Qemu for FreeBSD from “my” sources is pretty straight forward. However, it’s not as straight forward as building from Linux or from a FreeBSD port, so here are the full instructions 😉

$ export BSD_MAKE=`which make`
$ ./configure (your options here)
$ gmake
$ gmake install

Have fun.

GCC 3.3 on Mac OS X

I just tried to build Qemu on an Intel Mac. Qemu needs GCC 3.x and won’t compile with GCC 4.x. The configure script for Qemu automatically detects that the GCC 3.x is installed as /usr/bin/gcc-3.3 and tries to use that. The problem is that the compiler is actually a cross-compiler for PowerPC and that it cannot produce x86 binaries. Per this post I found out that using the -arch ppc flag allows the compiler to be used on an Intel machine. Obviously the resulting binary will be a PowerPC binary, but should run under Rosetta on an Intel machine.

Note that I still don’t have a solution for building Qemu on Mac OS X. This post is just a note about one aspect of the whole problem.

Leaky Abstractions

Today I came across an interesting article that sums up why abstractions still require one to know the lower levels. In essence, the author claims that abstractions are good and help us build complex systems, but are a great burden if something fails in the lower levels. I couldn’t agree more, especially when I think of the TianoCore code and the UEFI software model.

The infamous memory hole

Ok, so I’ve always suspected it, i.e. had a theory, but the CPC945 manual (section 7.2) confirms it.

If a machine has 4 GBytes of memory installed, and say, 1 GByte of I/O Memory is mapped at 0x80000000 (2 GBytes) upwards, then the physical memory will still be fully accessible. It will respond to read requests in the region 0x0 thru 0x7fffffff (i.e. 0 thru 2 GBytes – 1) and to the region 0xC0000000 thru 0x140000000 (i.e. 3 GBytes thru 5 GBytes).

This will of course only work if the CPU can make requests in that range, i.e. has a large enough address bus. Hence there is an actual hole that shadows physical memory when installing 4 GBytes in a x86-based System.