Cross Compiling Basics

Learning the terminology and toolchain for cross compiling for ARM on Linux

As a gadget geek, I've always loved small embedded handheld computers. From the palm pilots, gameboys, and pocket pcs of my youth, to the Retroid Pocket, GPD and Steam Decks of today - I've always had a special place in my heart for pocket computing. Recently, I decided to see just exactly what my RG351mp could do if I targeted it natively, rather than running my games through an emulator. This is the story of that ill conceived notion.

References:

Our first step to deploying to our RG351 is to be able to cross-compile a binary on our x86_64 to target the ARM architecture on the device. In this article we will use the incredible musl-cross-make to help us build a cross compile toolchain that we can run on our machine to generate binaries for almost any platform.

Before we get started, there's a few bits of terminology we should make clear:

Cross Compiling Terminology: Build, Host, Target

Normally when we are building binaries, we are building on a particular machine and architecture, for that same machine and architecture. However, when we are cross compiling it becomes very important to distinguish exactly which architecture we are building the tools on, what architecture the tools will run on, and what architecture the tools outputs are for.

These three variables are referred to as:

  • Build The machine building the cross tool
  • Host The machine running the cross tool
  • Target The machine that will run the outputs of the cross tool

Often the build and host targets are the same, this is cross compiling.

Example 1

  • We want to create a compiler toolchain on our linux machine
  • tha runs on our linux machine
  • that produces binaries that run on our RG351 device.
    • BUILD: our linux machine
    • HOST: our linux machine
    • TARGET: our RG351

Example 2

  • We are on a linux machine
  • we want to make a compiler that runs on a raspberry pi
  • it can compile binaries for itself.
    • BUILD: our linux machine
    • HOST: raspberry pi
    • TARGET: raspberry pi

Example 3 And finally, this is a cross compile example known as canadian cross - where all 3 platforms are different:

  • We are on a linux machine
  • we want to make a compiler for a raspberry pi
  • it will produce binaries for our rg351
    • BUILD: our linux machine
    • HOST: raspberry pi
    • TARGET: RG351

Example 1 describes the goal for this article.

Target Triplet

Autoconf has a convention for naming targets called the target triplet which eventually evolved into a convention using 4 values (but still referred to as a triplet because programmers are allergic to change). It typically takes the form cpu-vendor-kernel-os, although parts may be ambiguously omitted so parsing it can take some intuition.

For the RK3326 we will use armv8-a-linux-musleabi

  • it is running an ARMv8-A CPU - this is the 64bit variant of the ARMv8
  • kernel is linux,
  • musleabi is a blob stating that it's linked against musl (a c standard library built on top of the linux API), eabi referring to the embedded application binary interface

Create a user for cross compiling

As someone who wants to retain the existing x86_64 toolchain (for my normal development) I want to make sure that I properly separate the cross compilation toolchain such that it doesn't pollute my standard development environment. The easiest way to do this is to create a new user, put all the cross compiling toolchain executables in an alternate location, and update the new users environment variables to reference the new toolchain.

That way my user remains clean for building traditionally (no cross compiling), When I need to cross compile I sudo su - <user> to take advantage of the cross toolchain.

I named my user clfs (cross linux from scratch, based on one of the guides I was referencing at the time) and after creating both a group clfs and adding the user

sudo groupadd clfs
sudo useradd -s /bin/bash -g clfs -m -k /dev/null clfs

We create a .bashrc file for this user:


#
# CONFIGURATION ###############################################################
# 

# Disable bash's hash function.  This is an optimization to prevent
# bash needing to look up the location of an executable each time it
# is run.  However, as we will be incrementally building new tools
# as we progress - and want those that live in cross-tools to take
# precedence - we disable the hash lookup
set +h

# files and directories should be writable only by owner, but can
# be read and executed by anyone
umask 022

# location of the clfs installation
CLFS=/mnt/clfs

# controls the localization of sertain programs, failure to set this may
# cause some installation steps to fail.
LC_ALL=POSIX

# ensure cross tools comes first, as new tools are built, they should
# take precedence.
export PATH=${CLFS}/cross-tools/bin:/bin:/usr/bin

# make the above variables available to the shell
export CLFS LC_ALL PATH

# CFLAGS must not be set when building cross tools.  since that is all
# this profile will be doing, disable it by default so we don't forget
unset CFLAGS

#
# useful variables for when we are configuring and building cross tools
#

export CLFS_HOST=$(echo ${MACHTYPE} | sed "s/-[^-]*/cross/")
export CLFS_TARGET=aarch64-linux-musleabi
export CLFS_ARCH=aarch64
export CLFS_ARM_ARCH=armv8
export CLFS_FLOAT=hard
export CLFS_FPU=neon

and a .bash_profile

exec env -i HOME=${HOME} TERM=${TERM} PS1='\u:\w\$ ' /bin/bash

Dependencies

The build machine needs a few packages in order to bootstrap:

Bash-5.0
Binutils-2.34
Bzip2-1.0.8
CoreUtils-8.30
DiffUtils-3.7
FindUtils-4.7.0
Flex-2.6.4
Gawk-5.0.1
GCC-9.4.0
Glibc-6
Grep-3.4
Gzip-1.10
libssl-dev-1.1.1
M4-1.4.18
Make-4.2.1
Ncurses-6
Patch-2.7.6
Sed-4.7
Sudo-1.8.31
Tar-1.30
Texinfo-6.7.0

I just installed the default versions provided by my system (ubuntu 20.04)

Building a Cross Compile Toolchain

We will be using musl-cross-make project to help us easily produce a cross-compile toolchain based on:

  • binutils
  • gcc
  • musl for our standard c library

You can get musl-cross-make from this repository: https://github.com/richfelker/musl-cross-make

At the time of this writing, the latest version was 0.9.9, and it is the one we used.

Extract musl-cross-make into the soruces directory, then create a config.mak file. Inside place the following configurations:

TARGET=${CLFS_TARGET}
OUTPUT=${CLFS}/cross-tools/${CLFS_TARGET}

Run make

and then make install

You should now be avle to verify the following executables and their versions:

${CLFS}/cross-tools/${CLFS_TARGET}/bin/${CLFS_TARGET}-ld --version
GNU ld (GNU Binutils) 2.33.1
Copyright (C) 2019 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
${CLFS}/cross-tools/${CLFS_TARGET}/bin/${CLFS_TARGET}-gcc --version
armv8-linux-musleabi-gcc (GCC) 9.2.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

We can now create a tester cpp file tester.cc with the following code

#include <iostream>
int main()
{
    std::cout << "Hello, World\n";
    return 0;
}

We can compile it using our new cross toolchain with the following command:

$CLFS/cross-tools/$CLFS_TARGET/bin/$CLFS_TARGET-g++ --std=gnu++11 --static tester.cc

It will produce an output called a.out. If you attempt to run it, it will fail.

You can examine the header for the executable using the readelf command with the -h option (telling it to read the header)

${CLFS}/cross-tools/${CLFS_TARGET}/bin/${CLFS_TARGET}-readelf -h a.out
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x4006e0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          7744 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         8
  Size of section headers:           64 (bytes)
  Number of section headers:         28
  Section header string table index: 27

We have successfully cross compiled a C++ application for 64-bit arm! We might have to revisit this setup once we start working on deploying to the device, but this quickstart at least shows the basics.

If you do want to run it, you can use qemu-arch64

qemu-aarch64 a.out
Hello, World