Improving the Working Environment for Developers - In 3 Stages Only

… and incidentally establishing DevOps.

Are you working in the Software Development Sector as an App Developer no matter which technology or an IT Project Manager?
Sure thing you have heard some of the following sentences from your customer or your manager.

"The delivered version is weeks old. Can we update it asap?"”

“The App download link is dead.”

“Which Version needs to be built? What version do we need to deliver?”

“Can’t install the app, the certificate is not valid.”

“Anyone with spare time to upload a new build?”

Take a short break and feel your personal emotions those statements are triggering.

What comes to a Developer’s mind getting confronted with those statements regularly?

The exceptional wish of just automating the pains away. We technology enthusiasts like to automate everything everywhere because it’s part of our attitude to simplify our daily tasks — why not fixing those pains by automating the underlying processes.

Wouldn’t that make our lives so much more comfy? In a perfect world I would wake up in the morning. Get my coffee, start developing the new feature, fixing some bugs from yesterdays tested version, grab another coffee… and finish my day by just watching the application to be build and delivered automagically – ready to test. Everyone gets notified about the new version – no need to write another slack message.

Back then we had a Build Server (call it Jenkins) already in place. It was used rarely. In your daily business as a developer you were sometimes confronted with and interrupted by this Jenkins which was introduced for everyone to be able to create application arftifacts without the need to do it on your development machine.

Once in a while a build was triggered manually because it was time again to deliver a new app version to the customer. Many obstacles were lying in this way. Sometimes you were not even assigned to the underlying development project of this application but the only one with a little spare time and luckily a little know-how about this big Jenkins monster. After finding the right Build Job for the project on the server and fixing many build issues, you ended up — usually after many hours of guessing, try and error— with a build artifact.

Relieved seeing the green light flashing after the successful build, your task was to download the right artifact to your machine for the following task.

Your next task was to find some information about the delivery process for this project. Where to upload it? What are the credentials for accessing the uploaded version?

After finding someone somewhere who did know where to upload the artifact and luckily had the right credentials you ended up with an uploaded app and finally sending a notification to the customer.

Happy me! Now: Back to work developing your current assigned project.

Delivery of a version was a matter of luck finding the right way through a jungle of obstacles. This can’t be the final stage. Do we have more possibilities utilising our Build Infrastructure?

We started diving into the documentation of our Repository Manager (GitLab) and checked if we can get an easy integration to our Jenkins Server that triggers a new build if the code was updated. We found out about webhooks and commit triggers and did the setup on both ends.

We did it! With some simple steps we implemented Continuous Integration without knowing it and completed the first checkpoint on our track. We were now part of the CI Hype Train!

The advantages having CI for our project was obvious. After each code push we had a good (or bad) feeling if the application is still building on a sandbox environment and not just on our own local environments. Issues while building were anticipated in a very early stage which led to a less stressful life afterwards.

Fast forward to a couple of months later…, we established Continuous Integration in each new project and for a handful of existing projects and created our first best practices dealing with Jenkins. The stressful days before an application delivery phase reduced. No need anymore to manually trigger a Jenkins build on Friday afternoon and crossing fingers that the green traffic light will show up signalling a successful build. There was already a build artifact lying around on our Jenkins on friday afternoon just waiting to be uploaded to the file server from where it could be shared with the customer.

We asked us if we can get into the next stage of automating the upload to the ftp server and added some lines of code to our Jenkins Build Script which did the job.

Did we just reached the final stage of Continuous Delivery? Yes we did it! (at least we thought, that we did it back then)

The Application was now built after a push to our repo and after a successful build uploaded to the file server. Great!

We enjoyed watching this setup automagically upload one build after another. We introduced a proper versioning so the files on the server did not get overwritten and introduced a manual process of regularly doing a housekeeping on the server and deleting old apps.

But as soon as it was introduced to other projects we saw bad things happen. Jenkins scripts were copied and pasted. Since the beginning this was never challenged and working until now — but now we suddenly had a logic in our script which was generic enough to be shared between jobs… the simple logic of uploading a file.

We started evaluating what are the best practices to share scripts between different jobs and got in contact with the neat way of pulling scripts from a Source Code Repository. Jenkins allows this for each job Pipeline Script. We proceeded with the creation of a git repository for all of our Pipeline Scripts. In the course of starting to create all new job scripts in this repo we also switched from plain bash scripts to groovy scripts and extracted reusable code in a separate utils script which we then imported in the pipeline specific files.

We also searched for a more convenient way than our ftp server to provide App Downloads for iOS, Android and Windows platforms. HockeyApp (now AppCenter) was the solution for that. HockeyApp is a platform that offers an easy way to manage and distribute App binaries to Beta Testers. The switch allowed us to manage different tester groups (internal and external) and sped up the test cycles which noticeable improved the quality of each project.

After the refactoring phase and a lot of feedback from different departments we decided to improve the notification process for new releases. Luckily we were anyways improving our internal communication processes in the same time and introduced slack as our main communication tool. Slack was fitting perfectly in our scenario. We checked for integrations to Jenkins and quickly found a way to enrich our pipeline scripts with slack notifications. In the first step we sent notifications about the releases or build failures to the project slack channel. Which led to interruptions of ongoing discussions there. So we decided to have an additional notifications channel for each project which has a CD Pipeline in place and invited all project members to those channels.

With the addition of notifications we improved the project transparency significantly. Every project member was now aware about the project status at all times.

We managed to find a well fitting solution for each project participant which solved many manual steps with no decrease in the information flow.

We unknowingly did a lot of work in the Development Operations area. Back then when we started, the expression DevOps was not wide spread or at least did not reach us and it took us some time to realise what DevOps means for us. At least we now had a term to describe our tasks which was very handy for the internal communication.
Explaining the term DevOps and the work field rightis really challenging at least towards non-tech people. The definitions reach from “I’m a Text File Engineer” to “I automate the boring stuff….with magic.”
(https://www.reddit.com/r/devops/comments/awx6d7/can_anyone_describe_our_profession_to_nontech/)

As we all know, who stops being better stops being good. So we aim to iteratively improve our processes around our Continuous Delivery Pipelines.

We introduced a DevOps working group including experts of different technology stacks to get more feedback as well as improving the company wide knowledge share about DevOps and the possibilities and advantages. As artifacts we are generating best practices for different technology stacks. The working group is slowly evolving into an internal service provider for all topics regarding improvement of development and release cycles.

We introduced generic job scripts and best practices for different build targets and platforms. This allows the creation of a new Pipeline for a new project in a short time by picking an existing job configuration and changing parameters to fit the new project.

We integrated automated style checks and tests to improve our code quality and to create a common understanding between project members on code quality overall and that sharpened our definition of done. Many discussions were triggered about this topic.

We created scripts to automate the generation of the code documentation for our internal libraries. That itself was a complete topic on its own with many discussions and try and error iterations. I will recap this topic in another post.

We also introduced automatic nightly builds to early anticipate infrastructural failures in our build pipeline, third party integration problems and other errors which did not had their root cause in a breaking code commit. This was necessary for projects which were already rolled-out and had no ongoing development sprints— and therefore were not build regularly.

We analyzed the main pain points during the development phase of a application from the developers perspective and tried to solve as many of them as possible gradually.

We customized some of our existing tools, introduced a couple of best practices and processes and managed to establish a working group over the time of three years.

We learned that there is a term for what we are doing — DevOps. We are still improving what we have implemented in Stage 2 to be able to adapt to new technologies, platforms and processes.

We see DevOps as the main tool to improve the working environment for developers. Reducing disturbance as much as possible by simultaneously improving project transparency and letting the developers focus on the essentials. And additionally getting application quality improvements for free.

The Continuous Delivery Implementation fits perfectly to our agile development approach with regular project reviews and sprints.

Jenkins Build Server — https://jenkins.io
Gitlab Git-repository manager — https://about.gitlab.com
AppCenter (prev. HockeyApp) — https://appcenter.ms