Cross compiling C projects with external dependencies: Click modular router and the Raspberry PI

Introduction

At IBCN – my research group – we rely on Click router for implementing and evaluating various Internet communication protocols. Click is a modular software router in C++ that was developed at the end of the last century by Eddie Kohler and others at MIT (the original research paper is available here). Apart from the IP routing functionality that ships with Click, we’ve added a CoAP implementation that has been widely tested at ETSI Plugtest interoperability events. We also have a working 6LoWPAN implementation that was tested in  at last year’s summer IETF in Berlin. Lately, we’ve also added DTLS integration via CyaSSL.

Usually we run Click directly on x86 machines (e.g. on our laptops), sometimes we run it on Alix system boards via voyage linux. For the latter we compile Click in a voyage Linux VM, which works great as the Alix is still a x86-based system (comes with a AMD Geode CPU). In recent years with the advent of low-cost ARM based computing, we’ve started looking into running our “extended Click router” on these kinds of platforms. Their low cost and power consumption makes them particularly interesting for (Click) routers.

A well-known example is the Raspberry PI that packs an armv6 CPU. One issue is that armv6 cpu emulation via virtual machine software is not as readily available as x86  (although QEMU should be up to the task: see here). Therefor setting up a build VM as we did for the Alix system board, was deemed unfeasible. An alternative could be to compile directly on an actual Raspberry PI; while this would definitely work it would mean very (and I mean very) long compile times. It would also reduce the already limited lifespan of the SD card. In the end, I looked at cross compiling Click for the RPI. Compiling is a tad slower than when I compile for my native platform, but it is still much faster than a RPI. The only downside is that setting up such a cross compiler can be a bit of a hassle. Relying on external libraries that are linked in as dynamic libraries complicates things noticeably. Note that we run Click in userlevel most of the time, therefor using external libraries isn’t a problem for us.

To wrap up the introduction: this post documents some of the problems I encountered while cross compiling Click in userlevel with shared lib dependencies for the RPI. As a cross compilation environment is program agnostic, my findings are broader than cross compiling Click router apply to cross compiling for armv6 based platforms in general (and specifically the RPI). Hopefully, my findings can help with configuring your build system for cross compilation.

Getting started: compiling

As a first step you should get the cross compiler and setup its environment, there are a number of good resources available on this topic. This is a good tutorial: Cross-compiling for the RaspberryPi. As click doesn’t rely on CMake by default I suggest following the tutorial right until after the rsync script.

After the tutorial you should have a working cross compiler and the root fs of your RPI somewhere on your machine. Let’s start with the following script to configure click and get a Makefile. Change RPI_TOOLS and RPI_ROOT to your local setup. Save the script as compile_click_rpi and place it in a subfolder of the click root source directory.

#!/bin/bash

RPI_TOOLS=/home/fvdabeele/GIT/raspberrypi/tools
RPI_ROOT=/home/fvdabeele/Downloads/rpi

# arm-bcm2708hardfp procudes the 'best' binaries for the RPI
export CC=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-gcc
export CXX=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-g++

CPPFLAGS="-g -O0 -W -Wall"
CXXFLAGS="-g -O0 -W -Wall -I${RPI_ROOT}/usr/include -I${RPI_ROOT}/usr/include/arm-linux-gnueabihf"
LDFLAGS="-L${RPI_ROOT}/lib -L${RPI_ROOT}/lib/arm-linux-gnueabihf -L${RPI_ROOT}/usr/lib -L${RPI_ROOT}/usr/lib/arm-linux-gnueabihf"

autoconf
../configure --disable-linuxmodule --enable-local --enable-ip6 --enable-tools=host --enable-dmalloc --enable-tools=host --disable-threads  --disable-test --disable-tcpudp --disable-simple --disable-app --disable-aqm --disable-analysis --host=arm-linux-gnueabi --disable-int64 CXXFLAGS="$CXXFLAGS" LDFLAGS="$LDFLAGS"
echo "Making tools and elementmap"
make elemlist
make tools
make elementmap.xml

echo "Make"
make all -j4

Even prior to compilation configure might already quit with the following error:

checking whether the C compiler works… no
configure: error: in `/home/fvdabeele/GIT/COAPClientGateway/clickDynCOAP/rpi2′:
configure: error: C compiler cannot create executables
See `config.log’ for more details

Checking config.log:

arm-bcm2708hardfp-linux-gnueabi/bin/ld: cannot find /lib/arm-linux-gnueabihf/libc.so.6
arm-bcm2708hardfp-linux-gnueabi/bin/ld: cannot find /lib/arm-linux-gnueabihf/ld-linux-armhf.so.3

My guess is that using a precompiled cross compiler means that some settings are incorrect as they reflect the system on which the cross compiler was built. Although in this specific case removing “-L${RPI_ROOT}/usr/lib/arm-linux-gnueabihf” from LDFLAGS fixes the issue; this isn’t a solution as you’ll get linker errors later on. The quickest work-around I found for this problem is symlinking /lib/arm-linux-gnueabihf to ${RPI_ROOT}/lib/arm-linux-gnueabihf:

sudo ln -s /home/fvdabeele/Downloads/rpi/lib/arm-linux-gnueabihf /lib/arm-linux-gnueabihf

You will want to pass your own enable/disable statements to the configure script. Also note the disable-int64 flag, this is because the RPI doesn’t support 64-bit integers. In Click 2.0.1 however there appears to be a bug with the HAVE_INT64_TYPES define, you will get the following compiler error when enabling the flag:

../../../include/click/bigint.hh:115:2: error: ‘int_multiply’ was not declared in this scope

Fix it by applying the following patch to include/click/integers.hh:

diff --git a/clickDynCOAP/include/click/integers.hh b/clickDynCOAP/include/click/integers.hh
index fec1e59..638d8d0 100644
--- a/clickDynCOAP/include/click/integers.hh
+++ b/clickDynCOAP/include/click/integers.hh
@@ -367,6 +367,7 @@ inline int64_t int_divide(int64_t a, uint32_t b) {
     return a / b;
 # endif
 }
+#endif // florisvda
 
 
 /** @brief Multiply @a a * @a b, placing the low-order bits of the result in @a xlow
@@ -436,6 +437,7 @@ inline int32_t int_divide(int32_t a, uint32_t b, int32_t &quot) {
     return a - quot * b;
 }
 
+#if HAVE_INT64_TYPES // florisvda
 /** @overload */
 inline uint32_t int_divide(uint64_t a, uint32_t b, uint64_t &quot) {
 # if CLICK_LINUXMODULE && BITS_PER_LONG < 64

Linking

With the compiler script above and the patch you should be able to compile Click without too many problems. When you’re using external dependencies however, you’ll run into linker errors. In my case, I’m using libcurl libmemcached and libmysqlclient. Linking fails with:

/home/fvdabeele/GIT/raspberrypi/tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/../lib/gcc/arm-bcm2708hardfp-linux-gnueabi/4.7.1/../../../../arm-bcm2708hardfp-linux-gnueabi/bin/ld: cannot find -lcurl
/home/fvdabeele/GIT/raspberrypi/tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/../lib/gcc/arm-bcm2708hardfp-linux-gnueabi/4.7.1/../../../../arm-bcm2708hardfp-linux-gnueabi/bin/ld: cannot find -lmemcached
/home/fvdabeele/GIT/raspberrypi/tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/../lib/gcc/arm-bcm2708hardfp-linux-gnueabi/4.7.1/../../../../arm-bcm2708hardfp-linux-gnueabi/bin/ld: cannot find -lmysqlclient

On a native RPI with Raspbian PI this problem would be solved by pointing ld to the correct directories via settings in /etc/ld.so.conf and /etc/ld.so.conf.d/*. However, this isn’t an option for the arm toolkit ld as it ignores /etc/ld.so.conf. Instead, we pass along the necessary paths to the linker via -L flags in LDFLAGS. In the original script the LDFLAGS already include the necessary folders:

LDFLAGS="-L${RPI_ROOT}/lib -L${RPI_ROOT}/usr/ -L${RPI_ROOT}/lib/arm-linux-gnueabihf -L${RPI_ROOT}/usr/lib"

However, even if the linker managed to find all shared libraries, some references in these libraries might still be missing. You will see output similair to (truncated):

ld: warning: libidn.so.11, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libssh2.so.1, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: liblber-2.4.so.2, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libldap_r-2.4.so.2, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libgssapi_krb5.so.2, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libssl.so.1.0.0, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libcrypto.so.1.0.0, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: librtmp.so.0, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link)
ld: warning: libz.so.1, needed by /home/fvdabeele/Downloads/rpi/usr/lib/arm-linux-gnueabihf/libcurl.so, not found (try using -rpath or -rpath-link

libcurl.so: undefined reference to `SSL_CTX_use_certificate_chain_file@OPENSSL_1.0.0'
libcurl.so: undefined reference to `SSL_CTX_set_verify@OPENSSL_1.0.0'
libcurl.so: undefined reference to `ASN1_STRING_type@OPENSSL_1.0.0'
libcurl.so: undefined reference to `CRYPTO_free@OPENSSL_1.0.0'
libcurl.so: undefined reference to `SSL_get_shutdown@OPENSSL_1.0.0'
libcurl.so: undefined reference to `gss_import_name@gssapi_krb5_2_MIT'
libcurl.so: undefined reference to `SSL_get_verify_result@OPENSSL_1.0.0'
libcurl.so: undefined reference to `gss_unwrap@gssapi_krb5_2_MIT'
libcurl.so: undefined reference to `d2i_PKCS12_fp@OPENSSL_1.0.0'
libcurl.so: undefined reference to `EVP_cleanup@OPENSSL_1.0.0'
libcurl.so: undefined reference to `SSL_write@OPENSSL_1.0.0'
libcurl.so: undefined reference to `SSL_peek@OPENSSL_1.0.0'
libcurl.so: undefined reference to `BIO_new@OPENSSL_1.0.0'
libcurl.so: undefined reference to `SSL_set_fd@OPENSSL_1.0.0'
libcurl.so: undefined reference to `idna_to_ascii_lz@LIBIDN_1.0'
libcurl.so: undefined reference to `SSL_get_privatekey@OPENSSL_1.0.0

Luckily the linker gives a hint to solve the problem, update CXXFLAGS and LDFLAGS to include the necessary rpath-links:

CXXFLAGS="-g -O0 -W -Wall -I${RPI_ROOT}/usr/include -I${RPI_ROOT}/usr/include/arm-linux-gnueabihf -Wl,-rpath-link,${RPI_ROOT}/usr/lib/arm-linux-gnueabihf,-rpath-link,${RPI_ROOT}/lib/arm-linux-gnueabihf"
LDFLAGS="-L${RPI_ROOT}/lib -L${RPI_ROOT}/usr/ -L${RPI_ROOT}/lib/arm-linux-gnueabihf -L${RPI_ROOT}/usr/lib -L${RPI_ROOT}/usr/lib/arm-linux-gnueabihf -Wl,-rpath-link,${RPI_ROOT}/usr/lib/arm-linux-gnueabihf,-rpath-link,${RPI_ROOT}/lib/arm-linux-gnueabihf"

Voila, after adding the correct rpath-links linking succeeds and the click binary is available in userlevel/click! Copy it over to you RPI and test to see whether it works (fingers crossed) :). I’ve included the final compiler script in appendix A.

The end.

Appendix A: the final compiler script

#!/bin/bash

RPI_TOOLS=/home/fvdabeele/GIT/raspberrypi/tools
RPI_ROOT=/home/fvdabeele/Downloads/rpi

# arm-bcm2708hardfp procudes the 'best' binaries for the RPI
export CC=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-gcc
export CXX=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-g++

CPPFLAGS="-g -O0 -W -Wall"
CXXFLAGS="-g -O0 -W -Wall -I${RPI_ROOT}/usr/include -I${RPI_ROOT}/usr/include/arm-linux-gnueabihf -Wl,-rpath-link,${RPI_ROOT}/usr/lib/arm-linux-gnueabihf,-rpath-link,${RPI_ROOT}/lib/arm-linux-gnueabihf"
LDFLAGS="-L${RPI_ROOT}/lib -L${RPI_ROOT}/usr/ -L${RPI_ROOT}/lib/arm-linux-gnueabihf -L${RPI_ROOT}/usr/lib -L${RPI_ROOT}/usr/lib/arm-linux-gnueabihf -Wl,-rpath-link,${RPI_ROOT}/usr/lib/arm-linux-gnueabihf,-rpath-link,${RPI_ROOT}/lib/arm-linux-gnueabihf"

autoconf
../configure --disable-linuxmodule --enable-local --enable-ip6 --enable-tools=host --enable-dmalloc --enable-tools=host --disable-threads  --disable-test --disable-tcpudp --disable-simple --disable-app --disable-aqm --disable-analysis --host=arm-linux-gnueabi --disable-int64 CXXFLAGS="$CXXFLAGS" LDFLAGS="$LDFLAGS"
make elemlist
make tools
make elementmap.xml

echo "Make"
make all -j4

Appendix B: cross compiling CyaSSL for the RPI!

CyaSSL is a lightweight alternative to OpenSSL that mainly targets embedded systems. At IBCN we use it mainly for its implementation of the DTLS transport-layer security protocol, as CyaSSL offered DTLS1.2 support whereas OpenSSL did not (this was a year ago). Read more about cyassl at their website. In our case, CyaSSL is actually linked as a static library for handling DTLS communications in Click router.

CyaSSL is an example of a simpler C project that you might want to cross compile. Apart from the C standard library, it does not have any external dependencies like the Click source code that we use. As a result, the compiler script is somewhat shorter (be sure to keep the host flag if you’re calling configure for another project):

#!/bin/bash

RPI_TOOLS=/home/fvdabeele/GIT/raspberrypi/tools
RPI_ROOT=/home/fvdabeele/Downloads/rpi

# arm-bcm2708hardfp procudes the 'best' binaries for the RPI
export CC=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-gcc
export CXX=${RPI_TOOLS}/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-g++

./configure --host=arm-linux-gnueabi --enable-dtls --enable-aesccm --enable-psk --enable-static --enable-debug --enable-ipv6 --enable-ecc

make -j4

Bibliography available

In the interest of computer science research (ahum) I have put online my bibliography with PDF downloads of my own papers (i.e. where I am the first author).

I would urge any researcher to do the same as there is little to gain by isolating your work behind a pay wall. Apart from pay walls, hosting your own bibliography also makes accepted but unpublished work accessible to others sooner!

Cfr. http://cr.yp.to/bib/online.html

Hello world!

This post marks the beginning of my blog where I plan to write on various topics related to my professional activities as well as my personal interests.

Professionally, I’m currently engaged in a PhD focused on the Internet of Things. I’m employed at the Mobile research group at the department of Information Technology (INTEC) at Ghent University under supervision of prof. Ingrid Moerman. My `academic career’ can be tracked via Google scholar and/or my university’s digital library profile page. As the Internet of Things is a rather broad domain, let me be more specific. Currently I’m focusing on the impact of standardization on the Internet of Things.  I’ll write about the need for standardization in a separate post, as it would lead us to far for the purpose of this introductory post. In the future, you can also expect content about more generic topics related to computer science than just standardization though.

Apart from my academic ponderings, I also plan to write about some of my personal interests. Some of these are quite geeky, while others are more mundane. On the geek-side of the spectrum interests include communication networks/technology, Linux systems (and administration), web development and an occasional computer game. I’ll write about my home network setup in a later post, but for now I’ll say that I’m running a (non-native) dual stack and that this blog is hosted on a Raspberry PI with me at home. One consequence is that this blog is also reachable over IPv6 (try it)! More common interests include cycling (like most Belgians), (in-door) football, the classical piano, (fantasy) novels and hiking. I’ve included a list of my five latest bike rides on Strava on the right-hand side. As far as the piano goes, I’m currently enrolled in my second year at a local music school and am steadily improving (although I haven’t played for some time due to summer break).

Voilà, be sure to check back regularly for more content!