Building Elixir Applications as systemd Services

Written by baddev | Published 2021/06/13
Tech Story Tags: elixir | phoenix-elixir | devops | nspawn | linux | debian | containers | ansible

TLDR The story of deploying elixir applications in production is always a tough one. BEAM applications love to live in VM or bare metal without most abstractions. Nspawn is one of the components of Systemd-Nspawn that allows you to run containers (even docker containers) in your system. It's like chroot on steroids. The solution is to use a container with an OS identical to my production VM to deploy applications into production VMs. The only thing left is to trigger build inside of the container as part of the deploy process.via the TL;DR App

The story of deploying elixir applications in production is always a tough one, as BEAM applications love to live in VM or bare metal without most abstractions. If you don't want to use prepared services like gigalixer, deploying elixir applications to your own VMs is never trivial.

Beginning

As a developer and tinkerer I love being on the cutting edge of open source. Arch linux is an OS of choice, a tool of the trade. The projects that I work on always need to be built on Arch. However, more often than not, there will be problems with deploying a project built on a different linux distro or version. Libraries not matching (I am looking at you OpenSSL in Debian), tools not available, and so on.
One solution is to containerize everything into Docker or K8ns, deploy containers, and be done with it.
Since I am a Bad Dev, I ain't got time to learn new tech stack such as docker, and I like doing things the old way - deploying everything into VM directly.
So how do I deploy my projects into Debian or CentOS VM from an Arch machine? - By compiling them first inside a container with an OS identical to my production VM.
What are the options?
  • Virtual Box - lol no
  • Docker or K8s container - no, read above
  • Vagrant - Too heavy. By default, it runs a virtual box.
  • LXD/LXC containers - Maybe. A bunch of configs required though, and permissions set up
  • Systemd-Nspawn - Simple, Light, perfect for Lazy Devs
Systemd
Nspawn is one of the components of systemd that allows you to run containers (even docker containers) in your system. It's like chroot on steroids.
Preparing Nspawn
Our production VMs run Debian Linux, so that's the container we will use. Nspawn comes with systemd, so no need to install anything.
cd /var/lib/machines
debootstrap --include=systemd-container --components=main,universe stable debian https://deb.debian.org/debian
Now let's ensure we have a password set in the container:
cd /var/lib/machines
systemd-nspawn -D ./debian
passwd
logout
Done. Now you have a debian container living in the /var/lib/machines directory you can start it manually with
systemd-nspawn
or
machinectl
commands.
To be able to build our application in the container using deploy tools like ansible, we need couple more things.
Ensure you won't require sudo to start the container. Let's add a line to sudoers:
user-name ALL=NOPASSWD: /bin/machinectl start debian
Where username is your user name and debian is the name of the nspawn container.
Let's create systemd machine file, which will allow us to start the container automatically on system boot (if required), but also will allow us to mount our project directory directly into the container
sudo vi /etc/systemd/nspawn/debian.nspawn
[Network]
VirtualEthernet=yes
Port=23
Private=true

[Files]
Bind=/home/user-name/projects/my-awesome-project:/home/user-name/my-awesome-project

[Exec]
PrivateUsers=false
When we do
machinectl start debian
your container will be booted, with project folder mounted internally, and ready to build our elixir project. Of course, don't forget to install all required libraries inside the container with apt-get.
Bringing this into CI
We've set up our container, now the only thing left is to trigger build inside of the container as part of the deploy process. Let's test if we can run
mix release
command through SSH on our container. Boot container, and test with:
ssh user-name@debian -p 23 'bash -c "cd /home/user-name/projects/my-awesome-project && MIX_ENV=prod mix release --overwrite --force --path ../release"'
If successful you will find a new folder "release" with build artifacts, which you can deploy to your prod server as is.
Now part of ansible pipeline:
- name: Boot NSPAWN debian host
  shell: machinectl start debian

- name: Pause for 2 seconds waiting for machine to boot
  pause:
    seconds: 2

- name: Build App on NSPAWN host
  shell: ssh user-name@debian -p 23 'bash -c "cd /home/user-name/projects/my-awesome-project && MIX_ENV=prod mix release --overwrite --force --path ../release"'

- name: Poweroff NSPAWN debian host
  shell: machinectl poweroff debian
Pause is the only way I found to reliably wait for the container to boot, let me know if there is a better way.
That is how we deploy at pagerevew.io. You can adjust this for any type of CI you have, even simply use make file.

Written by baddev | Not so great developer promoting old ways of doing things
Published by HackerNoon on 2021/06/13