We are very much Atlassian folk where I work. JIRA, HipChat, Confluence, and of course Bitbucket all get a lot of use from us. Atlassian is slowly killing off Bamboo and we do not yet have a CI/CD setup. Since we are not ready to leave the Atlassian ship that means we are using Bitbucket, which leaves us with Bitbucket Pipelines and Jenkins Pipeline as our main contenders.
Bitbucket Pipelines is a CI/CD solution built right into Bitbucket, very similar to GitLab CI/CD.
Jenkins is the longtime ruler of the CI/CD space. It gets some negative opinions at times, but Jenkins Pipeline along with BlueOcean has done wonders to modernize the platform and make it a real contender in the wide world of CI/CD tools nowadays.
Both have adopted the Pipeline as Code mentality popularized by Travis.
Cost
Bitbucket
Bitbucket Pipelines pricing is based on number of users and how many build minutes you want. For example, our team is on the Standard tier of Bitbucket. This gives us 500 minutes per month, at a price of $2 per user. Additional time can be purchased for $10 per month, gaining you an extra 1000 minutes.
For my team of roughly 10 people, we pay $20 per month for 500 minutes. Our builds run anywhere from 3 to 5 minutes on Bitbucket, giving us 150 builds per month on average. If we average this out across all 10 users, that’s only 15 builds per user per month. We would very likely need to purchase another 1000 minutes, putting us up to $30 per month. This puts us at about 450 builds per month, or 45 builds per user per month. This is looking a lot better for us.
For 10 full time developers, you are looking to spend at least $30 per month.
Jenkins
We are an AWS shop, so I will be looking at AWS costs. If we look at the official AWS Jenkins documentation, AWS estimates $89 per month to host a single m4.large EC2 instance backed by 40GB of EBS storage.
The recommended architecture for Jenkins is one master node and at least one agent node. Running builds on the master has been discouraged for some time now. If we just add another m4.large EC2 instance, we add $74 per month. This puts us at $163 per month.
Costs can be adjusted some by playing with what instance type you go with. You can probably run the master on a t2.medium and bring your cost down to $123 per month. If you don’t have any development going on during off-hours, you can also just shut down the master and agent from the hours of 6:00pm to 7:00am. If you’re running two m4.large instances that will save $81 per month, putting you at $82 per month. If you’re running one m4.large and one t2.medium, you will save $59 per month, putting you at $72 per month.
Jenkins has no build time limit since you host the infrastructure yourself. For a small team and a basic Jenkins setup, you are looking at anywhere from $72 to $163 per month.
In the grand scheme, neither of these costs should be an issue. Both are monetarily very cheap.
Management
Bitbucket
Bitbucket requires almost zero management effort. All that is required is clicking a checkbox in your project settings to enable the feature. If anything breaks, Atlassian fixes it. That being said, if Pipelines goes down (and I have seen it drop a few times just the last couple weeks) you are stuck waiting on Atlassian to correct the issue.
Jenkins
Jenkins on the other hand requires some management. Somebody needs to be knowledgeable enough to install it, configure the necessary plugins, and configure the agent(s). With Docker, a lot of this is very simple. You can run the jenkinsci/blueocean Docker image and have a running Jenkins instance in just a couple minutes. It even works with Bitbucket out of the box.
While setup may be fairly simple, it is still work that somebody has to be paid to do. They need at least some familiarity with AWS, Docker, and setting up Linux environments.
If you do not have a dedicated operations team then at least one of your developers will need to be able to troubleshoot and correct any issues that may arise. If a build agent dies, somebody on your team needs to fix it. If builds are mysteriously failing, somebody needs to investigate. If you need to scale out your build agents, someone needs to have the know-how to do this.
A lot of these management tasks are simple enough. Scaling can be as easy as as having an AMI for an agent, spinning up a new EC2 instance, and attaching it to Jenkins. The Jenkins EC2 Plugin can do this whole process automatically as needed. It has the ability to spin up new EC2 instances based on an AMI you provide when demand increases, and then automatically terminate the instance as demand falls.
Bitbucket Pipeline is the clear winner when it comes to management. If just one developer being paid $90k per year spends 10% of their time on Jenkins, that’s $750 a month being put towards CI/CD and not being put towards new features. This cost would likely be offset by the increased agility, and the cost may not even be this high, but it’s hard to justify when there’s a management-free option to choose.
If your organization already has a dedicated operations team or operations person then this person-hour cost may be easier to justify.
Flexibility
Bitbucket
Bitbucket is fairly restricted in what it allows you to do. Atlassian has provided a very solid core set of features to you, but if you feel the need to stray from that path then you are going to have a hard time. I have found Bitbucket Pipelines to work wonderfully for smaller projects that just need a basic build-test-deploy-forget pipeline. If you require custom reporting, say static analysis trends, test results over time, etc then Bitbucket is not going to be very helpful.
Jenkins
Jenkins is incredibly flexible in what it can do. With over 1,000 plugins that can be used, whatever your particular use-case may be there is a good chance that somebody has already solved it. If your particular use-case isn’t solved yet, you can build your own plugin or Shared Library to solve it. Bitbucket Pieplines has no way for you to provide your own functionality like this.
The Jenkinsfile build script can be very simple if you want to use the Jenkins Groovy DSL, or it can be as complex as you want if you go the Scripted Pipeline route. With scripted pipelines, you are provided most of the Groovy language to build your pipeline how you see fit.
Jenkins is the more flexible option here. There is really no limit to what you can accomplish with Jenkins Pipeline.
Builds
Bitbucket
All builds are ran using a Docker image, or many images, of your choice. You can use a different image for each step of your build. Each step runs in a fresh container. This means you need to manage any artifacts that you want to keep between steps. For example, you might not want to compile your code on every step.
Bitbucket does not provide any control over this. You cannot run your builds directly on a VM or on dedicated hardware. The main consequence of this is that Windows builds become hard. In fact, Bitbucket Pipelines do not even support Windows builds right now.
Jenkins
Jenkins follows the Master/Agent architecture. There is a Jenkins Master that coordinates builds across one to many Agents. The Master can also be an Agent, but it is not recommended.
The Agent is really just a piece of software that is installed on some machine that coordinates builds. This can be anything from a physical server sitting in a rack, a VM provisioned in a cloud, or an ephemeral container running on Kubernetes.
A common approach is to provision VMs that contain only Docker and Git and run all builds in Docker containers.
Both Bitbucket and Jenkins support builds running in Docker. This is a good thing as it alleviates a lot of the headaches that CI/CD can bring. Bitbucket forces Docker onto you, while Jenkins lets you choose. If you need to build Windows applications, then Bitbucket Pipelines is not even an option right now.
Notifications
Bitbucket
Bitbucket emails everybody who is part of the team or who is watching a repository on all failed builds. There is a HipChat integration which you can install that will let you notify HipChat channels with build statuses. There is no easy way to notify Slack, Teams, IRC, etc. If you want special notification rules like only email the culprit on failed feature branches and everybody on a failed master branch, you are out of luck.
Jenkins
Jenkins notifications are very customizable. There exists plugins for Slack, Teams, HipChat, IRC, email, and almost anything else you can imagine. The rules for when to send notifications are very customizable. The notification itself can be customized however you want. If you want to send pictures of Chuck Norris on all successful builds, you can do it.
Jenkins lets you hook into the status of a build so you can adjust your notifications based on whether the build failed, passed, was unstable, or is passing after it was previously failing.
Notifications is an area that Bitbucket is really lacking in right now. For now, Jenkins is the winner here.
Conclusion
Bitbucket Pipelines is a very polished but limited experience. It can be a great tool for rapidly getting a small team into the CI/CD world, but if you need more advanced functionality you will quickly hit the limits of the platform.
Jenkins a more involved platform. Setup requires actually installing the Jenkins software, hooking up your Git repository hosting provider to send Webhooks to Jenkins, and provisioning build agents before you can even start a build. After setup you need somebody who can maintain the platform and make sure it stays up and running. This should not be much work, but if you are a small team then you may not be able to spare a person to spend some of their time on Jenkins. With this effort comes the most flexible CI/CD platform out there. You will be very hard pressed to come across a demand that Jenkins cannot meet.
Like what you read? Hit that Clap button and follow Hackernoon and Jared Ready for more great content!