Cross Compiling Rust on MacOS to Run as a Unikernel

Most programs on Linux are dynamically linked. So when you are creating a unikernel with OPS out of a linux application OPS goes out and finds all the libraries it's dynamically linked to and throws it onto the disk image. This works well if you are on linux - 99% of everything on linux is dynamically linked.

However, OPS also only loads ELF files. So if you are on a Mac you either need to cross-compile your program as if you were running it on Linux or you use a pre-made 'package' via the ops pkg repository. The latter works well for interpreted languages and vm based languages like the JVM but it also happens to be the fastest/easiest way of running linux programs on a mac that I'm aware of.

In some languages like Go to cross-compile it's a fairly simple:

GOOS=linux go build

However, for many other languages it is not so convenient. Typically when you cross-compile on a different system you'll end up statically linking cause it's a pita otherwise.

One question to ask is before we even get started - do you really need this capability? If you are running production applications on linux that probably implies you have a CI server running from which you can build your artifacts and binaries from. I don't know too many of you all that are writing production server-based apps that are being deployed on a mac.

Also, I personally do not use OPS locally for running applications as it is not necessary. I use it when debugging applications for Nanos or when I'm deploying an application to Google/AWS.

The Problem

Let's highlight the problem by using an example.

First we will create a new rust project:

➜  ~  cargo new bob
     Created binary (application) `bob` package

Then we'll build it and try to run it on a mac:

➜  bob git:(master) ✗ cargo build
   Compiling bob v0.1.0 (/Users/eyberg/bob)
    Finished dev [unoptimized + debuginfo] target(s) in 2.07s
➜  bob git:(master) ✗ ops run target/debug/bob
Only ELF binaries are supported. Is thia a Mach-0 (osx) binary? run 'file target/debug/bob' on it

Hrm... OPS is complaining cause we just built a Mach-O binary. We can verify this ourselves:

➜  bob git:(master) ✗ file target/debug/bob
target/debug/bob: Mach-O 64-bit executable x86_64

Install the Cross-Compiler Toolchain

Ok, so we know OPS doesn't like this file format. However we can fix this. First install your cross-compiler and linker toolchain:

brew tap SergioBenitez/osxct
brew install FiloSottile/musl-cross/musl-cross

Then you want to install the rust target:

rustup target add x86_64-unknown-linux-musl

Note we are specifying linux-musl not linux-gnu - the latter will result in your binary wanting some libraries dynamically linked like libc. That's not necessarily a problem, although it's more frustrating to deal with then just statically linking it.

Dynamically Linking

The majority of you should just skip this section.

If you do go down this route here are some quick tips. I was playing around with various targets and found myself reaching for tools I would normally have on linux. Again - we are dealing with ELFs here not mach-o binaries on a mac so ldd and friends don't exist.

To find out if your elf is statically or dynamically linked and if so what it is linked to, gobjdump will let you know:

brew install binutils
gobjdump -p bob

If you create a dynamically linked binary ops does have the capability of loading it up. You need to point the LD_LIBRARY_PATH though to wherever the libs are. Easiest method is to just create your fs layout in the current directory, specify it in the config.json and then load it up:

LD_LIBRARY_PATH=. ops run -c config.json bob

Statically Linking

Ok, back to statically linking. We will specify in our project root that we are going to use the musl linker. Put this into .cargo/config:

[target.x86_64-unknown-linux-musl]
linker = "x86_64-linux-musl-gcc"

From there we can build with cargo build:

TARGET_CC=x86_64-linux-musl-gcc \
RUSTFLAGS="-C linker=x86_64-linux-musl-gcc" \
cargo build --target=x86_64-unknown-linux-musl

Some of these steps might not be necessary as the rust ecosystem changes quite often but this is what it got to use the right flags for me.

And we can see that it works:

➜  bob git:(master) ✗ ops run target/x86_64-unknown-linux-musl/debug/bob
right before expanding
booting /Users/eyberg/.ops/images/bob.img ...
assigned: 10.0.2.15
Hello, world!
exit status 1

Ok! So that's how you can create a statically linked rust program on a mac and load it up in Nanos - all without touching linux.