Linker Sets

Reminder: When placing something in a dedicated section using __attribute__((section("foobar"))), the GNU toolchain will automatically add a symbol __start_foobar at the beginning and a symbol __stop_foobar at the end of the section.

However, you will need a reference to that symbol in order to prevent the linker from optimizing the symbol away. In other words, you need to declare something like extern void *__start_foobar; and use it.

When using the Microsoft toolchain, the symbols need to be added explicitly. To do that, one can make use of the fact that when the Microsoft linker encounters several sections with a “$_” in their name, it will merge the contents into one final output section. The name of the output section will be the common prefix of the declared sections. The beauty is that the contents are in the order of the section names.

Here’s an example: Supposed you placed something into a section called “foobar$_something”. You can then add a variable __start_foobar into a section “foobar$_a” and a variable __stop_foobar into a section “foobar$_z”. The resulting binary will have one section “foobar” with the contents of variable __start_foobar placed at the beginning, followed the contents of everything in section “foobar$_something” and the contents of the variable __stop_foobar at the end.

Building a PowerPC Cross Compiler

I need to build my own cross compiler which will run on i386 and produce 64-Bit PowerPC Binaries. I’ve found a pretty neat introduction to building a cross compiler on IBM’s developerWorks site (registration required). The tutorial isn’t a step-by-step guide, but it helped me a lot.

The basic procedure for building a cross-compiler is:

  • Obtain headers specific to the operating system
  • Build binutils for the target platform
  • Build a bootstrap compiler
  • Build a standard C library for the target
  • Build the final cross-compiler, using the C library just built

The developerWorks tutorial doesn’t mention this, but the first three steps can easily be run in parallel. Anyways, before starting, I’ve set these environment variables:

$ export TARGET=powerpc64-unknown-linux-gnu
$ export PREFIX=/opt/crosscompiler
$ export TARGET_PREFIX=$PREFIX/$TARGET
$ export PATH=$PATH:$PREFIX/bin

Obtaining Linux-specific Headers

I’ve followed the developerWorks tutorial on this one: Downloaded and extracting the Linux kernel sources, then copied the relevant files. Here are the commands I ran:

$ wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.25.9.tar.bz2
$ tar xvjf linux-2.6.25.9.tar.bz2
$ cd linux-2.6.25.9
$ make ARCH=powerpc CROSS_COMPILE=powerpc64-linux- menuconfig
(configure options, but tweaking isn't neccessary)
$ mkdir -p $TARGET_PREFIX/include
$ cp -r include/linux $TARGET_PREFIX/include
$ cp -r include/asm-powerpc $TARGET_PRFIX/include/asm
$ cp -r include/asm-generic $TARGET_PREFIX/include

If you read to the end of this post, then you’ll realize that this step wouldn’t be required (for now).

Building GNU binutils

I’m using GNU binutils 2.18, available from the GNU website. These are the steps required to build binutils.

$ wget 
$ tar xjvf
$ ./configure --prefix=$PREFIX --target=$TARGET --disable-nls -v
$ make all
$ make install

While building binutils did take a while, it wasn’t as long as the tutorial makes you believe. On a IBM Thinkpad T60p built around a Centrino Duo CPU running at 2.16 MHz it took less than 10 minutes. Also note the last command ("make install"), which is missing from the developerWorks tutorial.

Building a bootstrap compiler

For my project I need GCC 4.x, the latest version at the time of writing is 4.3.1 which is available from a GNU mirror near you. Downloading and extracting is easy:

$ wget ftp://ftp.gwdg.de/pub/misc/gcc/releases/gcc-4.3.1/gcc-4.3.1.tar.bz2
$ tar xjvf gcc-4.3.1.tar.bz2
$ cd gcc-4.3.1

Here are the steps required to build a bootstrap compiler.

$ ./configure --prefix=$PREFIX --target=$TARGET --without-headers 
  --with-newlib -v
$ make all-gcc
$ make install-gcc

This took longer than building binutils, however it took less than 30 minutes (as opposed to the hours the tutorial talks about).

Building the GNU C Library (glibc)


$ CC=${TARGET}-gcc ./configure --target=$TARGET --prefix=$PREFIX 
  --with-headers=${TARGET_PREFIX}/include

Unfortunately, this command failed with the following error:

(...)
checking whether __attribute__((visibility())) is supported... no
configure: error: compiler support for visibility attribute is required

However, this isn’t important as I won’t need a standard C library for now – I’m building with -ffreestanding and -nostdlib anyways. Therefore I’ve decided that I won’t pursue this futher but may come back later.

Embedding binary BLOBs into an ELF file

I needed this yesterday, found a link describing it – and forgot it by today 🙁

For a project I’m working on, I need to embed a file into an ELF executable. The executable then needs to do things with the embedded file, i.e. it has to know where in memory the file resides and how large it is.

So here it goes, largely copied from the link mentioned above.

  • Create an object file from the binary blob:
    $ ld -r -b binary -o example.o example.bin
  • In the sources, declare the symbols:
    extern char _binary_example_bin_start[];
    extern char _binary_example_bin_end[];
    
  • Make sure the object file is linked with the other sources:
    $ gcc -o example example.c example.o
    

Cell SDK 3.0 on IBM’s OpenClient Distribution

Today I was finally able to build a simple HelloWorld-Application with and within the TianoCore build environment. This is good news, as it leads me to the next task: Building the same HelloWorld-Application, but this time for Linux on 64-Bit PowerPC.

Since I do not yet have access to a 64-Bit PowerPC machine running Linux, I’m going to use the Cell SDK 3.0 for now. It can be used from an i386 machine and includes a toolchain as well as the full system simulator. The toolchain includes a cross-compiler that is cabable of producing binaries for the Cell BE’s PPU, which is essentially a 64-Bit PowerPC processor. The system simulator simulates a Linux-installation running on Cell.

I’m still on IBM’s OpenClient Linux distribution, which is apparently based on RHEL 5.1, at least according to /etc/redhat-release

Red Hat Enterprise Linux Client release 5.1 (Tikanga)

This is good on one hand, but made things slightly more complicated on the other hand. But first things first. Here’s what I did to prepare the Cell SDK installation:

  • I went to the developerWorks download page for the Cell SDK 3.0 and downloaded the RHEL 5.1 packages.
  • I had to download the "basic libraries and headers for cross-compiling to Cell Broadband Engine’s PPU", both the 32-Bit version and the 64-Bit version, from the Barcelona Supercomputer Center (BSC). Note that I could have built those RPMs myself, but only if I had a few other required RPMs like e.g. a glibc for PowerPC. Apparently those required RPMs are provided on the RHEL installation CDs, however, I’m on IBM’s OpenClient and thus do not have access to the installation CDs. The good thing is, the Fedora RPMs provided by the BSC turned out to work just fine.
  • For the full system simulator, I had to download the sysroot image from the BSC website.

So that’s it for the preparation part, now to actually installing the SDK.

  • I installed the installer RPM like this:
    # rpm -ivh cell-install-3.0.0-1.0.noarch.rpm

    This installs the installer to /opt/cell.

  • Now I needed to install the cross-compilation libraries and headers:
    # rpm -ivh ppu-sysroot-f7-2.noarch.rpm
    # rpm -ivh ppu-sysroot64-f7-2.noarch.rpm
  • Next I ran the installer as instructed by the installation manual:
    # cd /opt/cell
    # ./cellskd –iso /home/phs/Downloads install

After successfully running the installer, I found a functioning cross-compiler in /opt/cell/toolchain/bin.

For the system simulator, I had to install the sysroot imae RPM like this:

# rpm -ivh sysroot_image-3.0-7.noarch.rpm

Unfortunately, I wasn’t able to make the system simulator work because of a missing dependency on a simulation library.

By the way, there’s also official documentation available here.

Building a Cross Compiler on FreeBSD

I’m currently trying to build a cross compiler (and other required tools) on FreeBSD. The compiler will run on FreeBSD/amd64 and should produce i386 binaries. This wouldn’t be too hard since that task can easily be accomplished by using the FreeBSD source tree. However, I need the toolchain to produce binaries in the PE/COFF format instead of the default ELF format.

Building the toolchain is somewhat tricky, at least I found it to be poorly documented. But maybe I just looked in the wrong places. Building the toolchain requires:

  • Building binutils.
  • Locating some header files.
  • Building the compiler.
  • Building a C Library.
  • Building the compiler again so it uses the C library that was just built.

For my needs, I found that the last two steps weren’t needed. I wrote a script that downloads the sources, extracts the archives and builds the toolchain. Here’s the script in full quote (I really wish there was a way to upload files to this thing):

#/bin/sh

#
# Copyright (c) 2008 Philip Schulz 
#

#
# This script downloads, builds and installs GCC and GNU Binutils that can
# produce x86 binaries in PE/COFF Format. The cross toolchain needs some headers
# that aren't usually present on the host system. However, those headers can be
# obtained from the cygwin sources, that's why a snapshot of the cygwin sources
# is downloaded.
#
# After the script finishes, the tools will be located at
# ${PREFIX}/${TARGET_ARCH}/bin. Some other binaries will be installed in
# ${PREFIX}/bin with their own prefix of ${TARGET_ARCH} but I don't know that
# they are for.
#

# Prefix where the Cross-Tools will live
PREFIX="${PREFIX:-/opt}"

# Target architecture.
TARGET_CPU="${TARGET_CPU:-i386}"
TARGET_ARCH=${TARGET_CPU}-tiano-pe

# Program that can fetch the files.
FETCH_COMMAND="/usr/bin/ftp -V"

# GNU Make
GNU_MAKE=`which gmake`

################################################################################
#
# GCC settings.
#
################################################################################
# What version of GCC will be fetched, built and installed 
GCC_VERSION=gcc-4.2.3
# What mirror to use.
GCC_MIRROR=ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/ftp.gnu.org
# File name of the GCC sources. Should probably not be changed.
GCC_ARCHIVE=$GCC_VERSION.tar.bz2
# Where the GCC Sources can be fetched from. Should probably not be changed.
GCC_URL=$GCC_MIRROR/gcc/$GCC_VERSION/$GCC_ARCHIVE
# Arguments for the GCC configure script. Should probably not be changed.
GCC_CONFIGURE_ARGS="--prefix=${PREFIX} --target=${TARGET_ARCH} "
GCC_CONFIGURE_ARGS+="--with-gnu-as --with-gnu-ld --with-newlib "
GCC_CONFIGURE_ARGS+="--disable-libssp --disable-nls --enable-languages=c "
GCC_CONFIGURE_ARGS+="--program-prefix=${TARGET_ARCH}- "
GCC_CONFIGURE_ARGS+="--program-suffix=-4.2.3 "


################################################################################
#
# Binutils settings.
#
################################################################################
# What version of the GNU binutils will be fetched, build and installed
BINUTILS_VERSION=binutils-2.18
# What mirror to use.
BINUTILS_MIRROR=ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/ftp.gnu.org
# File name of the binutils sources. Should probably not be changed.
BINUTILS_ARCHIVE=$BINUTILS_VERSION.tar.gz
# Where the GCC Sources can be fetched from. Should probably not be changed.
BINUTILS_URL=$BINUTILS_MIRROR/binutils/$BINUTILS_ARCHIVE
# Arguments for the GCC configure script. Should probably not be changed.
BINUTILS_CONFIGURE_ARGS="--prefix=${PREFIX} --target=${TARGET_ARCH} "
BINUTILS_CONFIGURE_ARGS+="--disable-nls "

################################################################################
#
# Cygwin settings.
#
################################################################################
CYGWIN_SNAPSHOT=20080129
CYGWIN_ARCHIVE=cygwin-src-${CYGWIN_SNAPSHOT}.tar.bz2
CYGWIN_MIRROR=http://cygwin.com/
CYGWIN_URL=${CYGWIN_MIRROR}snapshots/${CYGWIN_ARCHIVE}
CYGWIN_DIR=cygwin-snapshot-${CYGWIN_SNAPSHOT}-1

################################################################################
#
# Batch code.
#
################################################################################
#
# Fetches the files.
#
do_fetch() {
        if [ ! ( -f $GCC_ARCHIVE ) ] ; then
                echo "Fetching ${GCC_URL}"
                ${FETCH_COMMAND} ${GCC_URL}
        else
                echo $GCC_ARCHIVE already locally present.
        fi

        if [ ! ( -f $CYGWIN_ARCHIVE ) ] ; then
                echo "Fetching ${CYGWIN_URL}"
                ${FETCH_COMMAND} ${CYGWIN_URL}
        else
                echo $CYGWIN_ARCHIVE already locally present.
        fi

        if [ ! ( -f $BINUTILS_ARCHIVE ) ] ; then
                echo "Fetching ${BINUTILS_URL}"
                ${FETCH_COMMAND} ${BINUTILS_URL}
        else
                echo $BINUTILS_ARCHIVE already locally present.
        fi
}

#
# Extracts the archives.
#
do_extract() {
        # Remove already extracted files first.
        rm -rf ${GCC_VERSION}
        rm -rf ${CYGWIN_DIR}
        rm -rf ${BINUTILS_VERSION}

        # Extract the archives
        if [ -f $GCC_ARCHIVE ] ; then
                echo "Extracting ${GCC_ARCHIVE}"
                tar -jxf ${GCC_ARCHIVE}
        fi

        if [ -f $CYGWIN_ARCHIVE ] ; then
                echo "Extracting ${CYGWIN_ARCHIVE}"
                tar -jxf ${CYGWIN_ARCHIVE}
        fi

        if [ -f $BINUTILS_ARCHIVE ] ; then
                echo "Extracting ${BINUTILS_ARCHIVE}"
                tar -xzf ${BINUTILS_ARCHIVE}
        fi
}


BUILD_DIR_PREFIX=build-

#
# Builds Binutils.
#
do_binutils_build() {
        BUILD_DIR_BINUTILS=${BUILD_DIR_PREFIX}binutils-${TARGET_ARCH}

        # Remove dir if it exists.
        if [ -d $BUILD_DIR_BINUTILS ] ; then
                rm -rf $BUILD_DIR_BINUTILS
        fi

        echo "Building binutils..."

        # Changing directory, so use a sub-shell (?)
        (
                # Create a the build directory.
                mkdir ${BUILD_DIR_BINUTILS} && cd ${BUILD_DIR_BINUTILS};
                # Configure, build and install binutils
                ../${BINUTILS_VERSION}/configure ${BINUTILS_CONFIGURE_ARGS} &&
                ${GNU_MAKE} -j 12 -w all && ${GNU_MAKE} -w install
        )

        # Remove build dir
        rm -rf $BUILD_DIR_BINUTILS

        echo "Binutils Build done."
}

#
# "Builds" cygwin. Actually, it only copies some headers around.
#
do_cygwin_build() {
        HEADERS=${PREFIX}/${TARGET_ARCH}/sys-include

        mkdir -p $HEADERS  &&
        cp -rf ${CYGWIN_DIR}/newlib/libc/include/* $HEADERS &&
        cp -rf ${CYGWIN_DIR}/winsup/cygwin/include/* $HEADERS
}

#
# Builds GCC
#
do_gcc_build() {
        BUILD_DIR_GCC=${BUILD_DIR_PREFIX}gcc-${TARGET_ARCH}

        # Remove dir if it exists.
        if [ -d $BUILD_DIR_GCC ] ; then
                rm -rf $BUILD_DIR_GCC
        fi

        echo "Building GCC..."

        # Changing directory, so use a sub-shell (?)
        (
                # Create a the build directory.
                mkdir ${BUILD_DIR_GCC} && cd ${BUILD_DIR_GCC};
                # Configure, build and install GCC.
                ../${GCC_VERSION}/configure $GCC_CONFIGURE_ARGS &&
                ${GNU_MAKE} -j 12 -w all && ${GNU_MAKE} -w install
        )
        rm -rf $BUILD_DIR_BINUTILS

        echo "GCC Build done."
}

do_fetch
do_extract
do_binutils_build
do_cygwin_build
do_gcc_build

Unfortunately, the gcc binary built by the script, located in /opt/i386-tiano-pe/bin, can’t produce binaries. Invoking the compiler on a source file (“Hello, World!” program) dies with:

$ /opt/i386-tiano-pe/bin/gcc main.c -o main
/opt/lib/gcc/i386-tiano-pe/4.2.3/../../../../i386-tiano-pe/bin/ld: crt0.o: No such file: No such file or directory
collect2: ld returned 1 exit status

I assume this is because I skipped the last two steps in the list at the beginning of this post. However, using the compiler to generate an assembler file (parameter -S) and then running the assembler on that file to produce an object file does indeed produce a PE/COFF object file.

$ cd /opt/i386-tiano-pe/bin
$ ./gcc -S ~/main.c
$ ./as -o main.o main.s
$ file ./main.o
./main.o: MS Windows COFF Intel 80386 object file