A master guide to Linux cross compiling
High level intro to everything you need to know to get started with cross compiling
Primer
What is cross compiling?
The act of compiling applications to run on a different computer system is referred to as “cross compiling”. This is oppose to “native compiling” where what you compiled is supposed to run on the same system that you compiled it on.
The classic example for cross compiling is when you compile for a low powered embedded ARM device from a powerful x86–64 PC. Cross compiling is preferred here because it is orders of magnitude faster to do so than building natively on the embedded device. This is a MUST have if you are iteratively developing an embedded application.
The catch with cross compiling is that it can be… complicated! Especially for the uninitiated. First you need to build all the dependencies, taking care to avoid contaminating and bricking your build machine with ARM bins. Then you need to massage and manipulate whatever package you are trying to build to get it to cross compile. Some opensource packages were never developed with cross compiling in mind and thus can require a lot of “compiler-error-driven-development”.
The general advice is to prefer a native build if what you are doing is a one time deal. Sure, it might take a few extra hours to build. But on the positive side, you won’t have had to spend that same time frustratingly setting up for a cross build. Seriously, pick your battles!
Terminology
1: Target triple
This is a label used to describe the runtime of an Unix system. It comprises of three values: ISA-Vendor-OperatingSystem. The hardware ISA (instruction set architecture) and vendor values should be pretty self-explanatory. The operating system value on the other hand can be hard to parse. Keep in mind that it is supposed to describe the runtime particulars of a system, so it can contain a plethora of optional information such as the kernel type, the C library of choice and the ABI (application binary interface). The C lib and ABI info are usually combined, thus the operating system label is in the form of: kernel-ClibABI.
e.g. Using gcc -dumpmachine on my laptop I can see that my target triple is x86_64-linux-gnu. Note that here the vendor field is empty, linux-gnu is the OS. Vendor is often irrelevant, thus it usually is marked as unknown , none or omitted entirely. If I use clang, I can see that that my PC reports x86_64-pc-linux-gnu . Here again the vendor is generic and is reported as pc.
Lets take a few target triples examples and break them down in the table below:
| Target Triple | CPU/ISA | Vendor | Kernel | C lib | ABI |
|------------------------------------|----------------|--------|---------|-------|---------|
| x86_64-linux-gnu | x86_64 | - | Linux | GNU | - |
| arm-cortex_a8-poky-linux-gnueabihf | Cortex A8 | Yocto | Linux | GNU | EABI-HF |
| armeb-unknown-linux-musleabi | ARM Big Endian | - | Linux | musl | EABI |
| x86_64-freebsd | x86_64 | - | FreeBSD | - | - |2: Toolchain
A toolchain is a collection of compilers, tools and libraries required for compiling. In addition to the actual compiler, a toolchain can include build tools (like CMake) and commonly used developer libraries such as SQLite. For cross toolchains, a target triple is used to describe the output it will produce.
For details about toolchains checkout my article, Toolchains: A horror story
3: Build vs Host vs Target machines
The key to cross compiling is understanding what these terms means in relation to the different types of things that you will be building.
Build machine: where the code is built
Host machine: where the built code runs
Target machine (only relevant for compiler tools): where the binaries spit out by the built code runs
Again, target triples will be used to describe these systems.
e.g. Lets say I am using a Linux PC (x86_64-linux-gnu) to cross compile a CMake application called “Awesome” to run on a BeagleBone Black SBC (armv7-linux-gnueabihf) using a GCC cross compiler. In this example:
| Component | Type | Build | Host | Target |
|-----------|-------------|--------|--------|--------|
| GCC | Compiler | x86_64 | x86_64 | armv7 |
| CMake | Build tool | x86_64 | x86_64 | N/A |
| Awesome | Application | x86_64 | armv7 | N/A |Your key take away here should be that the build/host/target label changes depending on what you are building. A common n00b mistake is to use the same host/target triple values used to build GCC when building applications that run on your embedded device, like for example Busybox. In this example the host for GCC is x86 but the host for Busybox is ARM. Understanding this fundamental is half the battle of cross compiling!
The first principles of cross compiling
Now that you have a sense of the terminology, the next step is to get a grasp of the fundamental principles.
A. The fundamentals of compiling
Before you tackle cross compiling, it helps to first thoroughly understand native compiling. You must understand how compiling, linking and loading works.
My go to reference for this is the book Advanced C and C++ Compiling.
B. Compile time vs Runtime dependencies
For any given cross compilation task you must be able to distinguish between what is need to compile on a build machine (e.g. C headers) and what is needed to run that an application on a target machine (e.g. shared libs).
C. Build systems
Open source applications use numerous build systems. Some of the big ones are Make, Autotools, CMake, Conan, Ninja and Meson. Each have their pros/cons and each have their own way to configure. Thus getting into a cross compiling task, you need to understand which build system you are dealing with and then do the necessary configuration updates needed to setup for a cross build. Nothing beats “reading the freaking manual” here, but here is my cheat sheet to get you started:
# Make
> export CROSS_COMPILE=arm-cortex_a8-linux-gnueabihf-
> make
> make DESTDIR=$(arm-cortex_a8-linux-gnueabihf-gcc -print-sysroot) install
# Autoconfig
> export CC=arm-cortex_a8-linux-gnueabihf-gcc \
CXX=arm-cortex_a8-linux-gnueabihf-g++
> ./configure --host=arm-cortex_a8-linux-gnueabihf
> make
> make DESTDIR=$(arm-cortex_a8-linux-gnueabihf-gcc -print-sysroot) install
# CMake
> cmake -DCMAKE_C_COMPILER=arm-cortex_a8-linux-gnueabihf-gcc \
-DCMAKE_CXX_COMPILER=arm-cortex_a8-linux-gnueabihf-g++ \
-DCMAKE_INSTALL_PREFIX=$(arm-cortex_a8-linux-gnueabihf-gcc -print-sysroot) ..
> cmake --build .
> cmake --install .D. Backwards compatibility
This is more of a pro tip that can help save time configuring your toolchain and can also help with troubleshooting issues.
The key points here are that the standard library stuff like the Linux kernel headers, GNU libc and libstdc++ are all backwards compatible. This means you can keep reusing old toolchains with relatively old kernel and GCC versions and still have the resultant binaries run on a target running newer versions. The only downside is that you won’t have the benefit of new features and optimizations that could make your applications run faster.
For non standard libraries your mileage may vary. Some libraries like Boost may have breaking changes for every minor version update that they do. While others like OpenSSL might be more stable, breaking compatibility only when jumping to a new major version number.
Ways to cross compile
Now that we have covered the fundamentals, lets jump to the actual how-to
1. With a standalone toolchain
This is the manual, traditional way to cross build.
Step one: Get a toolchain. You can get a precompiled toolchain from your silicon vendor (e.g. for BeagleBone black) or from a project like Linaro. Alternatively you can build one yourself with something like crosstool-ng.
Step two: Install any host tools that will be required (e.g. CMake)
Step three: Cross build and install any libraries that will be required (e.g. OpenSSL, SQLite, etc..). You will need to install the compile time dependencies (i.e. headers and static libs) to the toolchain’s sysroot and install any runtime dependencies (i.e. shared libs) on the target device. Note that you can and should avoid the runtime dependencies by preferring to statically link your dependencies.
Pro tips:
- Use a Docker image for the cross building. It helps to isolate the various host tools and libraries needed so that you don’t accidentally brick your work laptop by installing an ARM build of OpenSSL on your PC. Definitely not a personal experience 😛
- Modern languages like Rust and GO lang have “in-built” support for cross compiling and is generally very easy to setup.
- If you are actively developing a C++/CMake applications, try using Conan. It helps a lot to abstract the cross building process, making the build process less error prone and repeatable. It has the added advantage making onboarding easier.
2. With an Embedded build system: Yocto or Buildroot
I will admit that this is akin to using a bazooka to kill a fly. But then, if you already have a bazooka lying around, it’s not the worst idea in the world.
An embedded build system’s primary job is to build a custom distro or root filesystem for an embedded device. This involves cross compiling numerous applications. So you can leverage the same build infrastructure to do your standalone cross builds as well.
The chances are high that someone has already added support for what you are trying to build in Yocto or buildroot. Which means that the build configuration and dependencies are already available. This can be a life saver for packages with complex dependencies or for packages that need a lot of configuration updates to support cross builds. The obvious con here is that it is a lot harder to setup; especially if you have no prior experience with these build systems.
3. Build on QEMU
Here is a neat trick! You can cheat with an ARM QEMU virtual machine on your developer machine to do “native” builds that will produce binaries that will run on your target embedded device. Take care to configure the QEMU hardware and the native toolchain that you will be using to match with your target device. Also, be aware that the build speed won’ t be great due to the ARM to x86 translation. But otherwise this is a totally viable option, especially for esoteric packages/build systems that has no support for cross compiling.
That’s it folks.. Happy cross compiling!
