TianoCore and coreboot, again

It has been over two years since the last time I worked on TianoCore for coreboot. These days I had some free time to spend and I used it to continue the project. I updated my code to the latest versions of coreboot and TianoCore and got it to work again in QEMU.

Here’s a "screenshot" and a few words on what it’s all about.

The new GenFw Tool

I’ve re-written the GenFw tool part of the TianoCore BaseTools project. The source code can be found here. In order to use the tool, the file Source/C/GenFw/GenFw.c must be replaced with the re-written one. Then, the base tools must be re-built. After that, the EDK2 build process can be started. It will automatically pick up the new tool which will brand an ELF file with an UEFI file type.

Currently, the re-written tool will not compile on Linux. The reason is that Linux lacks implementations of err(3), errx(3), warn(3), etc. library functions which the BSDs have. It should be easy to add some compatibility macros using a combination of fprintf(3), strerror(3) and exit(3). I might add those should the need arise.

Update (Dec 3rd, 2008): I’ve added the compatibility macros for Linux. An updated version of the source code can be downloaded here.

More on TianoCore for coreboot

It’s been a while since I last worked on combining TianoCore and coreboot. Tonight I had some spare time and tried to pursue the project.

The previously mentioned build failure does indeed stem from the fact that the build tools cannot cope with ELF binaries. Especially problematic is the GenFw tool which is supposed to convert the binary file into an UEFI firmware volume file. In order to do that, it parses the header information of the input binary executable file and encodes the type of file (in UEFI terms) in a spare header field. The tool expects to work on PE32 files but the TianoCore developers have added code which converts an ELF image into a PE32 image internally if the tool is pointed at an ELF file. However, this facility is only compiled in if #defined(Linux) is true. Of course, that won’t work on FreeBSD but changing the relevant pre-processor condition allowed me to produce an UEFI firmware volume without any further changes to the code.

However, this shortcut will only work on x86 and only if the target platform is x86, too. The real solution is to avoid the conversion and instead encode the UEFI file type directly into the ELF header. I’ve already done this for my thesis project (*) and back then it seemed that re-writing the GenFw tool was easier than fixing the existing implementation. Well, here’s the next item on the ToDo list…

(*) I used the Java-based tools for the thesis project which means that a different tool with essentially the same functionality was the culprit.

The beginnings of coreboot and TianoCore

In order to create a UEFI payload for coreboot, I’ve started a coreboot platform as part of the TianoCore EDK II. The sources for the platform can be obtained here. Note that the CorebootPkg directory must be placed in the TianoCore $WORKSPACE directory.

To build the package on FreeBSD, a GNU toolchain from vendor sources must be used. This is because the TianoCore tools use some compiler/linker flags unknown to the toolchain included in the FreeBSD base system. The path as well as the names of the toolchain binaries must be adjusted in Conf/tools_def.txt. Because I built the toolchain according to these instructions, the preprocessor will not look in /usr/include for headers which causes errors in the ProcessorBind.h header when it attempts to include stdint.h. This patch can be applied to fix this.

Note that the build process still cannot complete as the tools producing the final Firmware Volume (FV) cannot cope with the ELF binaries produced by the GNU toolchain.

TianoCore and the Python-based Build Process, Part 3

This is part III of my attempts to build the TianoCore EDK II with the Python-based tools. In order to circumvent the error that stopped me in part II, the build process needs to be taught to use GNU make, i.e. gmake, on FreeBSD instead of make, which is BSD make. This can be done by editing the *_ELFGCC_*_MAKE_PATH variable in Conf/tools_def.txt.

The tools_def.txt file is automatically copied from a template part of the BaseTools sources. This patch fixes the template so the changes described above do not have to be applied manually.

At this point, the build process starts and does actually build some modules. However, the UnixPkg cannot be built completely on FreeBSD. This is because the code makes some assumptions only true on Linux, e.g. the presence of the sys/vfs.h header.

TianoCore and the Python-based Build Process, Part 2

So here’s the "sequel" to Part One. This time I’m trying to actually build a Firmware Volume with the Python-based tools.

Prerequisites for build.py

The core of the tools is build.py, a Python script which invokes the tools in order to build a Firmware Volume (FV). On FreeBSD, build.py cannot be run until the following requirements are met:

  • SQLite3 for Python, which can be installed through the databases/py-sqlite3 port.
  • The Python module for ANTLR, a parser generator.
  • Installing the module mentioned above requires EasyInstall, or rather: I don’t know how it can be done otherwise.

Because I could not find a port for EasyInstall, I did this to install the script on FreeBSD:

$ fetch http://peak.telecommunity.com/dist/ez_setup.py
$ chmod +x ez_setup.py
$ ./ez_setup.py

Note that this isn’t the whole truth as the path to the interpreter in the script, i.e. the first line aka "shebang", must be adjusted to /usr/local/bin/python before the script can be executed.

After that, the easy_install command is available and the ANTLR module can be installed by running this:

$ eazy_install 
  http://www.antlr.org/download/Python/antlr_python_runtime-3.0.1-py2.5.egg

Running build.py

In theory, running build.py and thus building a Firmware Volume should be as easy as this:

$ cd path/to/edk2
$ export PYTHONPATH=/path/to/basetools/Source/Python
$ . edksetup.sh BaseTools
$ python $PYTHONPATH/build/build.py

Unfortunately, the last step initially aborted with this error:

build...
 : error 5000: Please execute /home/phs/sandbox/basetools/Bin/FreeBSD-i386:/sbin: 
/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin: 
/home/phs/bin/edksetup.bat to set /home/phs/sandbox/basetools/Bin/Freebsd7 in 
environment variable: PATH!



- Failed -

After some try’n’error time, I think that the above error is caused by user error: I had previously copied the compiled C programs from Source/C/bin to bin/FreeBSD-i386 (paths relative to /path/to/basetools). After removing bin/FreeBSD-i386, I created a link to BinWrappers/PosixLike at the same location:

$ cd /path/to/basetools
$ ln -s BinWrappers/PosixLike Bin/FreeBSD-i386

I then re-ran build.py (see above) and it produced some output that didn’t look like errors:

00:44:09, Sep.21 2008 [FreeBSD-7.1-PRERELEASE-i386-32bit-ELF]

WORKSPACE                = /usr/home/phs/sandbox/edk2
EDK_SOURCE               = /usr/home/phs/sandbox/edk2/EdkCompatibilityPkg
EFI_SOURCE               = /usr/home/phs/sandbox/edk2/EdkCompatibilityPkg
EDK_TOOLS_PATH           = /home/phs/sandbox/basetools

TARGET_ARCH              = IA32
TARGET                   = DEBUG
TOOL_CHAIN_TAG           = ELFGCC

Active Platform          = UnixPkg/UnixPkg.dsc
Flash Image Definition   = UnixPkg/UnixPkg.fdf

Processing meta-data . . . . . . .

Unfortunately, though, right after the dots, an error occured:

build...
UnixPkg/UnixPkg.dsc(...): error 4000: Instance of library class [NetLib] is not found
        in [MdeModulePkg/Universal/Network/ArpDxe/ArpDxe.inf] [IA32]
        consumed by module [MdeModulePkg/Universal/Network/ArpDxe/ArpDxe.inf]
 

- Failed -
00:44:17, Sep.21 2008 [00:08]

Fixing the UnixPkg

The UnixPkg part of the EDK II seems to be broken as the error above indicates a dependency error between modules which is caused by an incorrect platform definition file (*.dsc). Applying this patch fixes the problem.

The patch ensures that all dependencies are met, but the build process still fails with this error:

Processing meta-data . . . . . . . . done!
make: don't know how to make pbuild. Stop


build...
 : error 7000: Failed to execute command
        make pbuild [/usr/home/phs/sandbox/edk2/Build/Unix/DEBUG_ELFGCC/IA32/MdePkg/Library/BaseTimerLibNullTemplate/BaseTimerLibNullTemplate]

Waiting for all build threads exit...
make: don't know how to make pbuild. Stop


build...
 : error 7000: Failed to execute command
        make pbuild [/usr/home/phs/sandbox/edk2/Build/Unix/DEBUG_ELFGCC/IA32/MdePkg/Library/BaseLib/BaseLib]


build...
 : error F002: Failed to build module
        MdePkg/Library/BaseTimerLibNullTemplate/BaseTimerLibNullTemplate.inf [IA32, ELFGCC, DEBUG]

- Failed -
01:01:43, Sep.21 2008 [00:09]

Oh, well, to be continued…

TianoCore and the Python-based Build Process

Now that I can use coreboot and libpayload on FreeBSD, it’s time to try the new Python-based build process for the TianoCore EDK II on FreeBSD.

Prerequisites are:

Note that Subversion access requires a user account at the TianoCore project.

Installing the e2fs-libuuid port

This is trivially easy, just do:

$ cd /usr/ports/misc/e2fs-libuuid
$ sudo make install

That’s all. The headers and libraries are installed under /usr/local.

Building the Base Tools

Compiling the Base Tools, i.e. the Python-based TianoCore build tools, isn’t complicated but doesn’t work out of the box, either. First, these two patches (patch 1, patch 2) must be applied:

$ cd /path/to/basetools
$ patch -p0 < basetools_include.diff
$ patch -p0 < basetools_make.diff

The first patch adjusts some include paths so that /usr/local/include is searched, too, which is required in order to find the uuid/uuid.h header. The second patch replaces invocations of make to use the $(MAKE) variable which holds the name of invoked the make binary. This is required as in FreeBSD (and other BSDs), make is not GNU make, however the latter is required to build the Base Tools. Consequently, when building the project, make sure that gmake is used:

$ gmake

Compiling the EDK II

To be continued...

The TianoCore Contributor’s Agreement

So, I finally found some time to crawl through the TianoCore project’s Contributor’s Agreement. Here’s what I think it means.

  • Preample: So Intel has decided to release some code under what it calls the "BSD license". Personally, I have think the BSD license is something else or maybe even something like this. I don’t think a link to an incomplete license stub is enough, though. But enough of the ranting.

    Just to be clear here, I think it is safe to assume that Intel released their code under the following license (note that it’s just the stub they provide a link to, filled in with some meaningful values):

    Copyright (c) 2006-2008, Intel Corp.
    All rights reserved.
    
    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions are
    met:
    
    * Redistributions of source code must retain the above copyright notice, this
      list of conditions and the following disclaimer.
    
    * Redistributions in binary form must reproduce the above copyright notice,
      this list of conditions and the following disclaimer in the documentation
      and/or other materials provided with the distribution.
    
    * Neither the name of Intel Corp. nor the names of its contributors may be
      used to endorse or promote products derived from this software without
      specific prior written permission.
    
    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
    AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
    LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
    CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
    SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
    INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
    CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
    ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
    POSSIBILITY OF SUCH DAMAGE.
    

    In addition to their own code which they release under the "BSD license", there is some code in the TianoCore tree that is released under other licenses. Specifically the FAT32 code, which is apparently covered by some patents. If other licenses apply, and that’s the key point here, the license is either in the source files themselves or packaged with the source files.

  • Definitions: I’m a "contributor" if I hold the copyright on some code that I contribute to the TianoCore project. If I represent a company, all my employees are considered part of my company and not separate contributors. A "contribution" is anything I sent to the TianoCore project, e.g. via mail, snail mail, telephone, etc. as long as I don’t explicitly mark it as "not a contribution".
  • License for Contributions: So when I decide to contribute something to the TianoCore project, I automatically agree to provide it under a license. That can be found in the contributor’s agreement. The bullet points a) and b) are pretty clear: The permisssion to use and redistribute my contributions, provided that the three conditions laid out in the "BSD license" quoted above are met.

    The next bullet point, c), is somewhat harder to understand. I interpret is as: If I hold a patent on something, and my contribution infringes that patent, I automatically grant a patent license. I grant it to everybody, who wants to exercise his rights I granted him with the copyrigth license mentioned above. However, here’s the catch: That patent license applies only to my unmodified contribution.

    I’m not sure what to think about that. I think, Intel is trying to protect their own patents. So if they release some code to the TianoCore project which is covered by a patent they own, the only grant a minimum patent license. What remains unclear is whether the patent license is still valid, even if I modify their code as permitted by the copyright license they granted.

    The last bullet point, d), is an easy one again. It’s simply the "provided ‘as is’" part in the copyright license cited above.

  • Representations: By signing the agreement, I promise that I created the code myself. If my employer does have any rights, I promise that it explicitly permitted me to contribute my code.
  • Third Party Contributions: If I chose to contribute third party code, I need to explicitly mark it as such. It also must be separate from my own contributions.
  • Miscellaneous. The agreement is in English and translation are not authorative. If I want to take the whole thing to court, I need to to it in Delaware (US).

So what’s the conclusion here? I think Intel is pretty open about releasing their code. However, they are not so open about creating an open source project around their code. What I mean is that there are quite some legal hurdles one has to pass when contributing code to the TianoCore project. In effect, they force the BSD license on any code I contribute and I think that’s OK. On the other hand, however, they prevent me from forking the project by introducing that stupid patent clause since I have no easy way of checking whether a specific piece of code infringes one of their patents.

I really wonder if they only want to protect themselves from getting sued over some code contributed the project from non-Intel employees. Or are they really trying to create an impression of an Open Source project when it’s really not?

Porting TianoCore to a new platform, part 2

So it took me exactly one day to start the TianoCore DXE Core on a new platform. Of course, this doesn’t count the roughly 10 weeks it took me to understand how the TianoCore Codebase works 😉 Also, it took me a fair amount of work to fix one thing or the other.

Anyways, I wanted to take a note that the generic memory test module included in the TianoCore Codebase is nowhere near "quick", despite the fact that it has a mode called QuickMode, when you attempt to throw it at a 8 GByte memory range.