Cross compiling C projects with external dependencies: Click modular router and the Raspberry PI

Introduction

At IBCN – my research group – we rely on Click router for implementing and evaluating various Internet communication protocols. Click is a modular software router in C++ that was developed at the end of the last century by Eddie Kohler and others at MIT (the original research paper is available here). Apart from the IP routing functionality that ships with Click, we’ve added a CoAP implementation that has been widely tested at ETSI Plugtest interoperability events. We also have a working 6LoWPAN implementation that was tested in  at last year’s summer IETF in Berlin. Lately, we’ve also added DTLS integration via CyaSSL.

Usually we run Click directly on x86 machines (e.g. on our laptops), sometimes we run it on Alix system boards via voyage linux. For the latter we compile Click in a voyage Linux VM, which works great as the Alix is still a x86-based system (comes with a AMD Geode CPU). In recent years with the advent of low-cost ARM based computing, we’ve started looking into running our “extended Click router” on these kinds of platforms. Their low cost and power consumption makes them particularly interesting for (Click) routers.

A well-known example is the Raspberry PI that packs an armv6 CPU. One issue is that armv6 cpu emulation via virtual machine software is not as readily available as x86  (although QEMU should be up to the task: see here). Therefor setting up a build VM as we did for the Alix system board, was deemed unfeasible. An alternative could be to compile directly on an actual Raspberry PI; while this would definitely work it would mean very (and I mean very) long compile times. It would also reduce the already limited lifespan of the SD card. In the end, I looked at cross compiling Click for the RPI. Compiling is a tad slower than when I compile for my native platform, but it is still much faster than a RPI. The only downside is that setting up such a cross compiler can be a bit of a hassle. Relying on external libraries that are linked in as dynamic libraries complicates things noticeably. Note that we run Click in userlevel most of the time, therefor using external libraries isn’t a problem for us.

To wrap up the introduction: this post documents some of the problems I encountered while cross compiling Click in userlevel with shared lib dependencies for the RPI. As a cross compilation environment is program agnostic, my findings are broader than cross compiling Click router apply to cross compiling for armv6 based platforms in general (and specifically the RPI). Hopefully, my findings can help with configuring your build system for cross compilation.

Getting started: compiling

As a first step you should get the cross compiler and setup its environment, there are a number of good resources available on this topic. This is a good tutorial: Cross-compiling for the RaspberryPi. As click doesn’t rely on CMake by default I suggest following the tutorial right until after the rsync script.

After the tutorial you should have a working cross compiler and the root fs of your RPI somewhere on your machine. Let’s start with the following script to configure click and get a Makefile. Change RPI_TOOLS and RPI_ROOT to your local setup. Save the script as compile_click_rpi and place it in a subfolder of the click root source directory.

#!/bin/bash

RPI_TOOLS=/home/fvdabeele/GIT/raspberrypi/tools
RPI_ROOT=/home/fvdabeele/Downloads/rpi

# arm-bcm2708hardfp procudes the 'best' binaries for the RPI
export CC=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-gcc
export CXX=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-g++

CPPFLAGS="-g -O0 -W -Wall"
CXXFLAGS="-g -O0 -W -Wall -I${RPI_ROOT}/usr/include -I${RPI_ROOT}/usr/include/arm-linux-gnueabihf"
LDFLAGS="-L${RPI_ROOT}/lib -L${RPI_ROOT}/lib/arm-linux-gnueabihf -L${RPI_ROOT}/usr/lib -L${RPI_ROOT}/usr/lib/arm-linux-gnueabihf"

autoconf
../configure --disable-linuxmodule --enable-local --enable-ip6 --enable-tools=host --enable-dmalloc --enable-tools=host --disable-threads  --disable-test --disable-tcpudp --disable-simple --disable-app --disable-aqm --disable-analysis --host=arm-linux-gnueabi --disable-int64 CXXFLAGS="$CXXFLAGS" LDFLAGS="$LDFLAGS"
echo "Making tools and elementmap"
make elemlist
make tools
make elementmap.xml

echo "Make"
make all -j4

Even prior to compilation configure might already quit with the following error:

checking whether the C compiler works… no
configure: error: in `/home/fvdabeele/GIT/COAPClientGateway/clickDynCOAP/rpi2′:
configure: error: C compiler cannot create executables
See `config.log’ for more details

Checking config.log:

arm-bcm2708hardfp-linux-gnueabi/bin/ld: cannot find /lib/arm-linux-gnueabihf/libc.so.6
arm-bcm2708hardfp-linux-gnueabi/bin/ld: cannot find /lib/arm-linux-gnueabihf/ld-linux-armhf.so.3

My guess is that using a precompiled cross compiler means that some settings are incorrect as they reflect the system on which the cross compiler was built. Although in this specific case removing “-L${RPI_ROOT}/usr/lib/arm-linux-gnueabihf” from LDFLAGS fixes the issue; this isn’t a solution as you’ll get linker errors later on. The quickest work-around I found for this problem is symlinking /lib/arm-linux-gnueabihf to ${RPI_ROOT}/lib/arm-linux-gnueabihf:

sudo ln -s /home/fvdabeele/Downloads/rpi/lib/arm-linux-gnueabihf /lib/arm-linux-gnueabihf

You will want to pass your own enable/disable statements to the configure script. Also note the disable-int64 flag, this is because the RPI doesn’t support 64-bit integers. In Click 2.0.1 however there appears to be a bug with the HAVE_INT64_TYPES define, you will get the following compiler error when enabling the flag:

../../../include/click/bigint.hh:115:2: error: ‘int_multiply’ was not declared in this scope

Fix it by applying the following patch to include/click/integers.hh:

diff --git a/clickDynCOAP/include/click/integers.hh b/clickDynCOAP/include/click/integers.hh
index fec1e59..638d8d0 100644
--- a/clickDynCOAP/include/click/integers.hh
+++ b/clickDynCOAP/include/click/integers.hh
@@ -367,6 +367,7 @@ inline int64_t int_divide(int64_t a, uint32_t b) {
     return a / b;
 # endif
 }
+#endif // florisvda
 
 
 /** @brief Multiply @a a * @a b, placing the low-order bits of the result in @a xlow
@@ -436,6 +437,7 @@ inline int32_t int_divide(int32_t a, uint32_t b, int32_t &quot) {
     return a - quot * b;
 }
 
+#if HAVE_INT64_TYPES // florisvda
 /** @overload */
 inline uint32_t int_divide(uint64_t a, uint32_t b, uint64_t &quot) {
 # if CLICK_LINUXMODULE && BITS_PER_LONG < 64

Linking

With the compiler script above and the patch you should be able to compile Click without too many problems. When you’re using external dependencies however, you’ll run into linker errors. In my case, I’m using libcurl libmemcached and libmysqlclient. Linking fails with:

/home/fvdabeele/GIT/raspberrypi/tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/../lib/gcc/arm-bcm2708hardfp-linux-gnueabi/4.7.1/../../../../arm-bcm2708hardfp-linux-gnueabi/bin/ld: cannot find -lcurl
/home/fvdabeele/GIT/raspberrypi/tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/../lib/gcc/arm-bcm2708hardfp-linux-gnueabi/4.7.1/../../../../arm-bcm2708hardfp-linux-gnueabi/bin/ld: cannot find -lmemcached
/home/fvdabeele/GIT/raspberrypi/tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/../lib/gcc/arm-bcm2708hardfp-linux-gnueabi/4.7.1/../../../../arm-bcm2708hardfp-linux-gnueabi/bin/ld: cannot find -lmysqlclient

On a native RPI with Raspbian PI this problem would be solved by pointing ld to the correct directories via settings in /etc/ld.so.conf and /etc/ld.so.conf.d/*. However, this isn’t an option for the arm toolkit ld as it ignores /etc/ld.so.conf. Instead, we pass along the necessary paths to the linker via -L flags in LDFLAGS. In the original script the LDFLAGS already include the necessary folders:

LDFLAGS="-L${RPI_ROOT}/lib -L${RPI_ROOT}/usr/ -L${RPI_ROOT}/lib/arm-linux-gnueabihf -L${RPI_ROOT}/usr/lib"

However, even if the linker managed to find all shared libraries, some references in these libraries might still be missing. You will see output similair to (truncated):

ld: warning: libidn.so.11, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libssh2.so.1, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: liblber-2.4.so.2, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libldap_r-2.4.so.2, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libgssapi_krb5.so.2, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libssl.so.1.0.0, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libcrypto.so.1.0.0, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: librtmp.so.0, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libz.so.1, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link

libcurl.so: undefined reference to `SSL_CTX_use_certificate_chain_file@OPENSSL_1.0.0'
libcurl.so: undefined reference to `SSL_CTX_set_verify@OPENSSL_1.0.0'
libcurl.so: undefined reference to `ASN1_STRING_type@OPENSSL_1.0.0'
libcurl.so: undefined reference to `CRYPTO_free@OPENSSL_1.0.0'
libcurl.so: undefined reference to `SSL_get_shutdown@OPENSSL_1.0.0'
libcurl.so: undefined reference to `gss_import_name@gssapi_krb5_2_MIT'
libcurl.so: undefined reference to `SSL_get_verify_result@OPENSSL_1.0.0'
libcurl.so: undefined reference to `gss_unwrap@gssapi_krb5_2_MIT'
libcurl.so: undefined reference to `d2i_PKCS12_fp@OPENSSL_1.0.0'
libcurl.so: undefined reference to `EVP_cleanup@OPENSSL_1.0.0'
libcurl.so: undefined reference to `SSL_write@OPENSSL_1.0.0'
libcurl.so: undefined reference to `SSL_peek@OPENSSL_1.0.0'
libcurl.so: undefined reference to `BIO_new@OPENSSL_1.0.0'
libcurl.so: undefined reference to `SSL_set_fd@OPENSSL_1.0.0'
libcurl.so: undefined reference to `idna_to_ascii_lz@LIBIDN_1.0'
libcurl.so: undefined reference to `SSL_get_privatekey@OPENSSL_1.0.0

Luckily the linker gives a hint to solve the problem, update CXXFLAGS and LDFLAGS to include the necessary rpath-links:

CXXFLAGS="-g -O0 -W -Wall -I${RPI_ROOT}/usr/include -I${RPI_ROOT}/usr/include/arm-linux-gnueabihf -Wl,-rpath-link,${RPI_ROOT}/usr/lib/arm-linux-gnueabihf,-rpath-link,${RPI_ROOT}/lib/arm-linux-gnueabihf"
LDFLAGS="-L${RPI_ROOT}/lib -L${RPI_ROOT}/usr/ -L${RPI_ROOT}/lib/arm-linux-gnueabihf -L${RPI_ROOT}/usr/lib -L${RPI_ROOT}/usr/lib/arm-linux-gnueabihf -Wl,-rpath-link,${RPI_ROOT}/usr/lib/arm-linux-gnueabihf,-rpath-link,${RPI_ROOT}/lib/arm-linux-gnueabihf"

Voila, after adding the correct rpath-links linking succeeds and the click binary is available in userlevel/click! Copy it over to you RPI and test to see whether it works (fingers crossed) :). I’ve included the final compiler script in appendix A.

The end.

Appendix A: the final compiler script

#!/bin/bash

RPI_TOOLS=/home/fvdabeele/GIT/raspberrypi/tools
RPI_ROOT=/home/fvdabeele/Downloads/rpi

# arm-bcm2708hardfp procudes the 'best' binaries for the RPI
export CC=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-gcc
export CXX=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-g++

CPPFLAGS="-g -O0 -W -Wall"
CXXFLAGS="-g -O0 -W -Wall -I${RPI_ROOT}/usr/include -I${RPI_ROOT}/usr/include/arm-linux-gnueabihf -Wl,-rpath-link,${RPI_ROOT}/usr/lib/arm-linux-gnueabihf,-rpath-link,${RPI_ROOT}/lib/arm-linux-gnueabihf"
LDFLAGS="-L${RPI_ROOT}/lib -L${RPI_ROOT}/usr/ -L${RPI_ROOT}/lib/arm-linux-gnueabihf -L${RPI_ROOT}/usr/lib -L${RPI_ROOT}/usr/lib/arm-linux-gnueabihf -Wl,-rpath-link,${RPI_ROOT}/usr/lib/arm-linux-gnueabihf,-rpath-link,${RPI_ROOT}/lib/arm-linux-gnueabihf"

autoconf
../configure --disable-linuxmodule --enable-local --enable-ip6 --enable-tools=host --enable-dmalloc --enable-tools=host --disable-threads  --disable-test --disable-tcpudp --disable-simple --disable-app --disable-aqm --disable-analysis --host=arm-linux-gnueabi --disable-int64 CXXFLAGS="$CXXFLAGS" LDFLAGS="$LDFLAGS"
make elemlist
make tools
make elementmap.xml

echo "Make"
make all -j4

Appendix B: cross compiling CyaSSL for the RPI!

CyaSSL is a lightweight alternative to OpenSSL that mainly targets embedded systems. At IBCN we use it mainly for its implementation of the DTLS transport-layer security protocol, as CyaSSL offered DTLS1.2 support whereas OpenSSL did not (this was a year ago). Read more about cyassl at their website. In our case, CyaSSL is actually linked as a static library for handling DTLS communications in Click router.

CyaSSL is an example of a simpler C project that you might want to cross compile. Apart from the C standard library, it does not have any external dependencies like the Click source code that we use. As a result, the compiler script is somewhat shorter (be sure to keep the host flag if you’re calling configure for another project):

#!/bin/bash

RPI_TOOLS=/home/fvdabeele/GIT/raspberrypi/tools
RPI_ROOT=/home/fvdabeele/Downloads/rpi

# arm-bcm2708hardfp procudes the 'best' binaries for the RPI
export CC=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-gcc
export CXX=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-g++

./configure --host=arm-linux-gnueabi --enable-dtls --enable-aesccm --enable-psk --enable-static --enable-debug --enable-ipv6 --enable-ecc

make -j4