A new Continuous Integration Coordinator written in Elixir & Phoenix
I have been wanting to learn Elixir and Phoenix for a while now, and a couple of months ago I finally took the plunge. I started reading “Programming Elixir 1.3” by Dave Thomas, and “Programming Phoenix” by Chris McCord, et. al. (both from “The Pragmatic Bookshelf” and highly recommended if you want to get started with Elixir and Phoenix).
I followed most of the code guides in the books, but it still felt kind of foreign sometimes. I wanted to build something a bit more complex, and challenging. I also wanted to release it as open source software, so it had to be useful to a wider audience than just me.
I’ve always liked CI systems and the value they provide, but I have yet to find one that offers everything I need. I want it to be highly configurable, easily scalable, easy to install, with a robust executor that can run on multiple operating systems, and to be open source. The CI service that comes closest to this is GitLab CI, but it is tightly coupled with GitLab itself, so I cannot use it with GitHub. I also don’t want to install a whole Source Code Management system just for it’s CI service.
That is where the idea to create a CI system came from. I learned a lot about how CI systems should work from my time at GitLab, so it felt like the right step forward. In fact, I decided to use the GitLab CI Runner as the executor for this project.
The GitLab CI Runner is written in Go and is the piece of code that actually takes care of running the builds, and whatever other action you need in order to execute your pipeline. The greatest thing about it, is that it is basically a “dumb box”, meaning that all you need to do is tell it what to do in the way it expects, and it will do it! It is also incredibly powerful and configurable, it can use Docker to execute your builds, and takes advantage of Docker Machine to let you autoscale the machines it uses for your builds, it can use Kubernetes, run on top of Digital Ocean’s VMs, run Xcode and Windows builds, and a lot more.
Using an already existing project as the executor for my CI system allowed me to concentrate on the coordinator, which is the piece of software that reads your project’s configuration, creates the pipelines, and builds, and sends all this information to the executor in the way it expects it to be. It also collects all the information relayed by the executor, like build trace and other artifacts, and presents the to the user in an easily digestible manner.
Since this is an introduction post, I will not go into details of the implementation of the different parts of AlloyCI. For that I will create several separate posts and will go into as much detail as possible. For now, let me tell you about its features.
(Part 1 of the AlloyCI Dev Diary is live!)
Features
Please keep in mind that this is the first public release of AlloyCI, and it is only at version 0.1.0. A lot of the features will improve over time and a lot more will come as we move towards 1.0. You can check the roadmap to 1.0 here: https://github.com/AlloyCI/alloy_ci/projects/1 and if you’d like to contribute anything, please go ahead.
Basic CI functionality
For version 0.1 the goal was to have at least the very basics of CI functionality. These include the following:
- Parse a configuration file to create and prepare the builds and pipelines
- Support different build stages
- Allow builds and runners to be tagged, thus allowing for fine grained control of which runner picks up which build
- Process and send the build information to the GitLab Runner in the expected format
- Accept information received from the GitLab Runner about the specific build
- Show the output trace of each build and follow it when looking at an active build
- Properly manage the status of each build, and send to the Runner only builds that should be processed
- Report to GitHub the status of each push via the SHA reference of its last commit
- Send notifications to the user about the status of a pipeline via email and/or Slack
- Perform auto-deploys via the
deploystage
As commercially available CI systems go, AlloyCI is dwarfed by the plethora of features other CI services offer, but that’s OK. I wanted to start simple and distill the CI experience to only its most necessary features. Bells and whistles will come later 😉
Extras
Something I did add just for the “nice to have” value are build statistics per project and runner. You can see some nice charts portraying the amount of build in the past week, how many have succeeded or failed, and the total of builds processed.
Project’s Settings
The runners chart will also brake down the amount of builds processed by project, so you can see which runners are picking up which builds.
Runner’s Admin Settings
How can you try it out?
You can install AlloyCI on your own servers using Docker, and Docker Compose. Please follow our documentation to get started.
The easiest way to try AlloyCI out is by going to https://alloy-ci.com, logging in with your GitHub account, and adding a project.
New Project’s View
Before being able to add a project, you will be prompted to install the AlloyCI integration on your account. After doing this, the project you’ll want to add needs to have an .alloy-ci.json configuration file before it can be added. You can read more about how to create this file here. Here is a quick example for Elixir:
{"image": "elixir:latest","services": ["postgres"],"before_script": ["mix deps.get"],"after_script": ["rm secrets"],"stages": ["build","test","deploy"],"mix credo": {"stage": "build","script": ["mix credo"],"tags": ["elixir"]},"mix test": {"stage": "test","script": ["mix test"],"tags": ["postgres", "elixir"]}}
Once the file has been committed to your repository, you are ready to add it to AlloyCI. After the project has been added, and new code has been pushed, you will see the newly created pipeline in the project’s dashboard.
Project’s Dashboard
If you click on the pipeline, it will take you to the its builds, where you can follow their output and overall status.
Build Job’s Output
The test server has limited resources and only has 2 registered runners to process builds, so it will take a while to process your pipeline. Luckily you can register your own runner as a “Project Specific Runner”, which means it will only pick up builds from your own, particular project. Go to your project’s settings for instructions on how to add a specific runner.
Why did I build AlloyCI?
As I mentioned before, I built AlloyCI to learn Elixir and Phoenix. I chose a CI coordinator, because I’m familiar with them, and wanted to add my own take on them. I’m also releasing it as GPLv3 licensed open source code, because I think other people might find it useful. Having a non-trivial codebase to look at is a really great resource to learn a new language. I know my code is far from perfect, but it is a start, and I would love for people to look at it, criticize it, and provide useful feedback, so that I, and others, can learn and benefit from this project.
I will continue to develop AlloyCI for quite some time. I plan to maintain it for the foreseeable future, and I might even consider writing a new runner/executor in Elixir itself, so that the project does not rely on GitLab’s good intentions of keeping the GitLab Runner open source and maintained.
—
If you like the project, please star it on GitHub, and if you think of something the project could use, or have some suggestion or feedback, by all means, please open an issue. I would ❤️ to hear what you think.