In the last article, I explained what CRD is and how it can be useful to solve the problem. In this article, I will show you how you can write a controller that will monitor changes in custom resources. And in the next article, the controller will already begin to respond to these changes and will configure the Envoy.
So, what is a controller? The controller is a program that communicates with Kubernetes through its API. When in the last article, I announced a new type of Custom Resources and announced a new API for it: proxy.company.com/v1.
To interact with the Kubernetes API already, have a lot of ready-made clients in different programming languages, but I will create the examples of golang. Especially since, for golang, there is even an "official" code generator. That’s what I plan to use.
Entities
The first thing I need to do is create entities to which the Sidecar resource data from the API will be mapped. For this, we need two packages:
- k8s.io/apimachinery Provides all necessary abstracts and interfaces
- k8s.io/client-go API client for golang
Let me remind you that in the last article, I announced a new type of Sidecar resource. Now, I will describe the structure for it in golang using the packages listed above in the file types.go.
package v1
import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
type SidecarSpec struct {
AppName string `json:"appName"`
Cb struct {
Timeout int `json:"timeout"`
Tries int `json:"tries"`
}
}
// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type Sidecar struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec SidecarSpec `json:"spec"`
}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type SidecarList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata"`
Items []Sidecar `json:"items"`
}
SidecarSpec matches the description from Custom Resource Definition, recall what it looks like.
spec:
type: object
properties:
appName:
type: string
cb:
type: object
properties:
timeout:
type: integer
tries:
type: integer
Sidecar contains additional fields in which various meta-information must be collected. SidecarList is a structure for a set of Sidecar structures. For cases when a list of sidecars is requested from the API, his fields for meta-information are slightly different.
I will also need the register.go file to register the entity.
package v1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
)
// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{
Group: "proxy.company.com",
Version: "v1",
}
// Kind takes an unqualified kind and returns back a Group qualified GroupKind
func Kind(kind string) schema.GroupKind {
return SchemeGroupVersion.WithKind(kind).GroupKind()
}
// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
return SchemeGroupVersion.WithResource(resource).GroupResource()
}
var (
// SchemeBuilder initializes a scheme builder
SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
// AddToScheme is a global function that registers this API group & version to a scheme
AddToScheme = SchemeBuilder.AddToScheme
)
// Adds the list of known types to Scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
scheme.AddKnownTypes(SchemeGroupVersion,
&Sidecar{},
&SidecarList{},
)
metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
return nil
}
I specified the API name and version.
Group: "proxy.company.com",
Version: "v1",
And added two Sidecar and SidecarList entities to the known data schemes. It is important to place the file according to the name and version of internal/apis/proxy.company.com/v1.
Generator
The main task of the generator - on the basis of the described entities to generate a code, which in essence can make queries to the Kubernetes API and a mapped response to the described data structures. Generator downloaded as standard.
go get k8s.io/code-generator
But since the generator will not import into any of the packages, it will not actually be downloaded to the vendor directory. (I prefer to use this directory to work for more convenient debugging). So, I used a little hack to create a file with a generator import.
package hack
import _ "k8s.io/code-generator"
And now, when doing go mod download, the generator is in vendor and I have the ability to use its bash script generate-groups.sh, which has already solved most of the possible needs of developers. I’ll give you the launch code right away.
SCRIPT_ROOT=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )/.." &> /dev/null && pwd )
bash vendor/k8s.io/code-generator/generate-groups.sh \
"deepcopy,client,informer" \
github.com/antgubarev/xds-controller/internal/generated \
github.com/antgubarev/xds-controller/internal/apis \
proxy.company.com:v1 \
--output-base=$SCRIPT_ROOT/../../.. \
--go-header-file=$SCRIPT_ROOT/hack/boilerplate.go.txt
Argument assignment
- The entities I need to generate. You can also simply specify all, and then all possible will be generated.
- The path by which the generated files will be placed.
- The path where the data scheme from which the generation takes place.
- API name and version. It is important to coincide with the way where the structures are placed, as I wrote above.
There are also two options in the generator.
--output-baseThe root directory into which the generated files will be added. That is to say, the--output-base+github.com/antgubarev/xds-controller/internal/generatedIn my case, this value will beGOPATH/src--go-header-fileThe header file if you want to place any additional information in each generated file.
So, let’s run it and look at the result:
./hack/upd.sh 15ms (master*)
Generating deepcopy funcs
Generating clientset for proxy.company.com:v1 at github.com/antgubarev/xds-controller/internal/generated/clientset
Generating listers for proxy.company.com:v1 at github.com/antgubarev/xds-controller/internal/generated/listers
Generating informers for proxy.company.com:v1 at github.com/antgubarev/xds-controller/internal/generated/informers
3 subdirectories appeared in the internal/generated directory
clientsetClient for API queriesinformersA set of components for tracking changes
listers Informer is needed and in this article, it is of no additional interest
This set will be sufficient for the current task so far.
Controller
Now, I can start tracking changes in Sidecar resources. You can see the full controller code here. I’ll describe it in more detail.
import versionedclientset "github.com/antgubarev/xds-controller/internal/generated/clientset/versioned"
//...
cfg, err := clientcmd.BuildConfigFromFlags("", filepath.Join(homedir.HomeDir(), ".kube", "config"))
if err != nil {
logger.Fatalf("BuildConfigFromFlags: %v", err)
}
_, err = kubernetes.NewForConfig(cfg)
if err != nil {
logger.Fatalf("Error building kubernetes clientset: %s", err.Error())
}
proxyVersionedClient, err := versionedclientset.NewForConfig(cfg)
if err != nil {
logger.Fatalf("Error building example clientset: %s", err.Error())
}
First, I need to create a client. I used the config for kubectl, which is in the home directory, usually, it’s ~/. kube/config. Next, I used the generated code from internal/generated/clientset. I can already create, delete, and edit Sidecar custom resources using this client.
The entire generated code works with the data structures from the internal/apis generator; it has done a lot of work for us.
proxyInformerFactory := proxyinformers.NewSharedInformerFactory(proxyVersionedClient, time.Second*60)
idecarsInformer := proxyInformerFactory.Proxy().V1().Sidecars().Informer()
sidecarsInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: func(obj interface{}) {
sidecar, ok := obj.(*v1.Sidecar)
if !ok {
logger.Errorf("Can't parse added object: %v", obj)
}
logger.Infof("Sidecar %s is added", sidecar.Name)
},
UpdateFunc: func(oldObj interface{}, newObj interface{}) {
switch newObj.(type) {
case *v1.Sidecar:
newSidecar, ok := newObj.(*v1.Sidecar)
if !ok {
logger.Errorf("Can't parse updated object: %v", newSidecar)
}
if oldObj != newObj {
logger.Infof("Sidecar %s is updated", newSidecar.Name)
}
default:
logger.Errorf("update event: unknown object %v", newObj)
}
},
DeleteFunc: func(obj interface{}) {
sidecar, ok := obj.(*v1.Sidecar)
if !ok {
logger.Errorf("Can't parse deleted object: %v", obj)
}
logger.Infof("Sidecar %s is deleted", sidecar.Name)
},
})
Informer is also a client, but it is designed to inform the client about new updates. In the code above, I announced the handler for 3 events
AddFunccreated a new Sidecar resourceUpdateFuncupdated Sidecar range resourceDeleteFuncremoved Sidecar resource
So far, these features do not do anything useful, only log events in the console and later we will definitely see how it works.
stop := make(chan struct{})
defer close(stop)
proxyInformerFactory.Start(stop)
if !cache.WaitForCacheSync(stop, sidecarsInformer.HasSynced) {
logger.Info("Failed to sync cache")
}
<-stop
I called the Start function which creates a go routine to track events. The client stores a cache with the required objects so as not to query them every time in real-time. He also gave me a channel to get the program done properly.
To be sure of the integrity and reliability of this cache in the previous example, the interval of the full update was set to time.Second*60. Note that this procedure generates refresh events for those data that are already in the cache (because they will be updated in the cache).
In order to avoid unnecessary events, I perform a check on the functions UpdateFunc
if oldObj != newObj {
logger.Infof("Sidecar %s is updated", newSidecar.Name)
}
Now, we can test how it works. First, I run the program and wait for readiness (all connections I executed with locally running minikube). I created a manifest with 3 new resources.
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
name: service1
spec:
appName: service1
cb:
timeout: 5000
tries: 3
---
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
name: service2
spec:
appName: service2
cb:
timeout: 500
tries: 1
---
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
name: service3
spec:
appName: service3
cb:
timeout: 3000
tries: 3
Then I applied the manifest (controller is running).
kubectl apply -f services.cr.yaml
sidecar.proxy.company.com/service1 created
sidecar.proxy.company.com/service2 created
sidecar.proxy.company.com/service3 created
Output
INFO[0015] Sidecar service1 is added
INFO[0015] Sidecar service2 is added
INFO[0015] Sidecar service3 is added
That’s right. Three resources were created and triggered the add event for each resource created. Now, I have changed the parameters of the variables in the first two resources to any other values and applied the manifest again.
kubectl apply -f services.cr.yaml
sidecar.proxy.company.com/service1 configured
sidecar.proxy.company.com/service2 configured
sidecar.proxy.company.com/service3 unchanged
Output
INFO[0011] Sidecar service1 is updated
INFO[0011] Sidecar service2 is updated
Everything is working properly again. And now, I will try to remove one of the resources.
kubectl delete sidecar service1
sidecar.proxy.company.com "service1" deleted
Output
INFO[0014] Sidecar service1 is deleted
Conclusion
I showed a way to track the changes that occur with custom resources in Kubernetes. In the next article, I will use these events to configure a sidecar based on Envoy via the xDS protocol. You can see the full example in the repository.
If you like my articles, please encourage me in the Noonies.