In this article, I open a series that will consider the approach to the Envoy configuration with Kubernetes CRD. I will tell about the implementation of the controller for Kubernetes, which watch custom resources in the Kubernetes cluster and performs the Envoy configuration based on this data. This approach is used for example in Istio. After reading the article you will also be able to better understand how it works.
The xDS protocol allows Envoy to dynamically configure proxy rules without rebooting the proxy itself and changing its settings. This gives many advantages in modern multi-component and distributed systems. Services can be created and deleted, become temporarily unavailable, or authentication rules can be changed and much more. All this must be done instantly in the production environment. It is important not to change the application code because the infrastructure can change even faster than the application itself.
Custom Resource Definition
So the first thing we’re going to talk about is the Custom Resource Definition. It’s kind of like a table in a database. You first describe the schematic of this table, and then you manipulate the elements: create, update, delete. Within the Kubernetes cluster, we can also track events with these elements and react to those events.
In my example, I will manage an object that should contain behavior information on the circuit breaker pattern
- Connection timeout
- Number of tries before the error is returned
That is, if you receive a request, the envoy will make three attempts to get a response from the backend before returning the error code. This can be useful when we know in advance that the backend does not respond to an average of every tenth request.
Custom Resource Definition
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: sidecars.proxy.company.com
spec:
group: proxy.company.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
appName:
type: string
cb:
type: object
properties:
timeout:
type: integer
tries:
type: integer
scope: Cluster
names:
plural: sidecars
singular: sidecar
kind: Sidecar
shortNames:
- sc
group: proxy.company.com. Group for the resource type. Must contain the domain.name: sidecars.proxy.company.com. Name of a new resource type. Formed by{resource_name}. {domain}names. Here I specify all possible names of the resource type. In singular, plural, etc. These names will be used when working withkubectl.
Note that the schematic description takes place inside the versions block. This is important as the schematic is likely to change. So let’s apply the new configuration.
kubectl apply -f crd.yaml
customresourcedefinition.apiextensions.k8s.io/sidecars.proxy.company.com created
And now I can manage objects (Custom resource or CR) with a new type of proxy resource that I defined.
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
name: service1
spec:
appName: service1
cb:
timeout: 5000
tries: 3
---
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
name: service2
spec:
appName: service2
cb:
timeout: 500
tries: 1
---
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
name: service3
spec:
appName: service3
cb:
timeout: 3000
tries: 3
In this example, I want to have three CR records of Sidecar type with different values. The first thing I did was specify which version of the API I wanted to use. ApiVersion: proxy.company.com/v1. The CRD Declaration is an extension to the Kubernetes API, so you must specify the name of the API and course the version. You can go back and look at the line group: proxy.company.com in CRD. In the spec block, I set the field values I want.
kubectl apply -f services.cr.yaml
sidecar.proxy.company.com/service1 created
sidecar.proxy.company.com/service2 created
sidecar.proxy.company.com/service3 created
Three objects were created. And now you can check that they are what we expect.
kubectl get sidecars -o yaml 1ms
apiVersion: v1
items:
- apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"proxy.company.com/v1","kind":"Sidecar","metadata":{"annotations":{},"name":"service1"},"spec":{"appName":"service1","cb":{"timeout":5000,"tries":3}}}
creationTimestamp: "2022-09-11T12:26:36Z"
generation: 1
name: service1
resourceVersion: "189731"
uid: 67f03ae5-30f6-42a5-af28-75f041c0c41d
spec:
appName: service1
cb:
timeout: 5000
tries: 3
- apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"proxy.company.com/v1","kind":"Sidecar","metadata":{"annotations":{},"name":"service2"},"spec":{"appName":"service2","cb":{"timeout":500,"tries":1}}}
creationTimestamp: "2022-09-11T12:26:36Z"
generation: 1
name: service2
resourceVersion: "189732"
uid: b874770a-257e-4044-bfb8-7ea291cdfbc8
spec:
appName: service2
cb:
timeout: 500
tries: 1
- apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"proxy.company.com/v1","kind":"Sidecar","metadata":{"annotations":{},"name":"service3"},"spec":{"appName":"service3","cb":{"timeout":3000,"tries":3}}}
creationTimestamp: "2022-09-11T12:26:36Z"
generation: 1
name: service3
resourceVersion: "189733"
uid: 934b06fb-6851-4df6-beb5-b7279d0b59b4
spec:
appName: service3
cb:
timeout: 3000
tries: 3
kind: List
metadata:
resourceVersion: ""
selfLink: ""
I can now manage the CR with the usual commands. For example, I can first delete
kubectl delete sidecar service1
sidecar.proxy.company.com "service1" deleted
kubectl get sidecars
NAME AGE
service2 5m4s
service3 5m4s
Then apply the declaration again
kubectl apply -f services.cr.yaml
sidecar.proxy.company.com/service1 created
sidecar.proxy.company.com/service2 unchanged
sidecar.proxy.company.com/service3 unchanged
kubectl get sidecars
NAME AGE
service1 46s
service2 6m49s
service3 6m49s
How Can I Apply it
In the diagram, I showed how the architecture works, which consists of three components.
Sidecar
This is a container in the pod that accepts all requests and proxies to localhost, where the request should already be received directly by the service that runs in production. Sidecar is needed to implement the request processing policy. In this example, it is necessary to implement the pattern of Circuit Breaker, about which I have written above. As a sidecar, I will use Envoy, which can be easily configured via the xDS protocol.
Custom Resources
Kubernetes API. The database in which the settings of sidecars are specified. Yes, you can store this data in any other storage, such as Redis. But why do it when Kubernetes already has distributed storage, based on etcd. In addition, it provides a convenient API for work, as well as the ability to subscribe to resource creation, modification, or removal events.
Controller
This is an application that interacts with Kubernetes API and Sidecar. It may be an application in different programming languages, but in this series, there will be examples on Go, as it is for Go already have well-developed libraries.
Conclusion
In this part, we have looked at the general architecture of the solution, as well as at the Custom Resource Definition. In the next part, I will describe the implementation of the controller. More precisely, the part that can subscribe to events on change custom resources and respond to them.
If you like my articles please encourage me in the Noonies.