Introduction
With containerization and container orchestration rapidly increasing,
according to the Cloud Native Survey from 2020, the usage of containers in production jumped 300%
the usage of Kubernetes is only going to increase, but there is a difference between using Kubernetes and being able to extend it.
Kubernetes is designed as a platform to build platforms and operators are the ultimate extensibility tools.
The true power of Kubernetes is not just the ability to orchestrate containers, but rather its extensible API and Control Plane. Kubernetes operators and control loops are how the internal machinery of everyone’s favourite container orchestrator works.
In this article, we will look at the operator pattern, learn when it is appropriate to use an operator. Finally, we will explore the operator’s architecture.
Before you continue reading, please note that Kubernetes operators are difficult, take time to get right, and create maintenance challenges. However, if you find a good use case and can commit time and resources to master them, it will unlock additional powerful capabilities.
Kubernetes Architecture
Before we start looking into operators, let’s get a quick refresher on Kubernetes architecture:
Internally Kubernetes architecture uses controllers in closed control loops to ensure cluster health and correctness of the workloads by continuously reconciling the actual state of the cluster with the desired state specified by the user/administrator.
Some of the controllers on the diagram are:
-
controller manager which manages all the Kubernetes native control loops
-
cloud controller which enables cloud providers to plug in their infrastructure elements
As you can see, control loops are at the core of how Kubernetes ensures state reconciliation and self-healing.
Operators Overview
The easiest way to understand operators is to think about them as software counterparts of human operators. Imagine someone performing tasks such as database backups, upgrades, schema changes, etc. There is a lot of complex operational knowledge involved to do all those tasks correctly. What if something fails? Well, even more knowledge and experience is needed to troubleshoot and recover from an indecent.
Now imagine that you have complex software running on your cluster, something like couchbase, apache-spark, or even Prometheus. The knowledge required to properly operationalize all those components is encapsulated in operators.
You might also find yourself on the provider end of the spectrum, and want to provide additional benefits to your customers. Writing and maintaining an operator is a fairly complex task, but if there is a use case, it’s well worth the effort.
Kubernetes operators follow the operator pattern. In simple terms, operator pattern automates operations performed typically by humans, such as:
- installation
- updates and upgrades
- configuration
- backup
- gathering metrics
- troubleshooting
Architecture
Operators build on the Kubernetes architecture and all components are part of the Kubernetes standard building blocks. As you can see on the below-simplified diagram, the controller manager running on a control plane node is responsible for the reconciliation and management of all controllers, built-in and custom alike. A custom controller is just a reconciliation logic running as a program, written in any language, and deployed to a Kubernetes cluster.
The simplest operator consists of:
- a custom controller running in a pod controlled by a Deployment or StatefulSet
- a CRD (Custom Resource Definition) defining schema for a custom resource
- a custom resource that forms an API layer that users can manipulate to interact with the operator
The custom resource follows the same semantics as other Kubernetes resources.
Maturity levels
Operators can be as simple as installing components onto a cluster up to much more complex, like full life cycle management or even performance of configuration fine-tuning live as the application runs.
There are 5 levels of operator maturity:
Operators in practice
Should you start writing operators for everything? Not, so when should you? Here are a few good use cases for operators:
- updating/upgrading stateful workloads
- interacting with external resources outside of a Kubernetes cluster
- chaos engineering
- advanced scheduling, configuration, and failure resiliency
Writing and maintaining an operator is a very complex process, you need:
- deep understanding of Kubernetes internals
- expertise in any programming language (preferably Go)
- strong grasp on concepts like “eventual consistency”, “loose coupling”, “event-driven architecture”
Before looking into creating your own operator, make sure to visit operator hub. There is a chance that the software you are using already has an operator.
Controller vs operator
What is the difference between controllers and operators? Well, there is no technical difference, but rather in semantics.
Controller is a control loop continuously watching and reconciling state of the cluster. Operator is also a controller, but it additionally encapsulates specific domain knowledge.
Helm vs operator
If you look closer at what an operator is doing, it looks a lot like helm, at least from a packaging point of view.
Whenever a question comes to “A vs B”, it is usually a good heuristics to rephrase it into “When is it more appropriate to use A and when B”
Similarly here, helm is great for simple installation and templating, but the operator can do custom logic on the resources which helm simply was not designed for.
Tooling
There is a lot of operator frameworks, SDKs and related tooling, but I recommend looking at kubebuilder, also available in the form of an online book.
Please refer to the last section of this blog “Resources” where you can find more information about available tools.
Demo scenarios
There are a few choices if you would like to experiment with operators. I recommend going through kubebuilder’s quickstart.
- use my repository with a self-contained development environment if you have already a Kubernetes cluster and you would like to try something locally on your machine. Please note that this is a containerized environment, so other than Kubernetes and VS Code, you don’t need anything else.
- follow an excellent Katacoda scenario by @javajon if you don’t want to install anything on your local machine and would rather prefer remote infrastructure.
If you decided to use this repository, there are a few prerequisites:
- VS Code with Remote Development Extensions
- Docker Desktop with Kubernetes enabled. If you have other Kubernetes clusters in your config file, make sure to point to docker-desktop
I’ve tested it with remote cluster and it works too, but doesn’t work with k3s/k3d.
The image will take a while to load as it pulls kubebuilder, Go binaries, and other components, so please patient
Summary
For me, the main takeaway from learning about Kubernetes operators was a much deeper understanding of the Kubernetes architecture, but most importantly the realization that the true power of Kubernetes is in its extensible API and universal control plane.
Whether it’s consuming operators or creating your own, starting with the mindset that Kubernetes is a platform to build platforms, is the path to levelling up your Kubernetes game.
Resources
Here are useful resources for further study.