Introduction
Kubernetes is an open-source container management system that was introduced by Google. As container usage is proliferating, Kubernetes is playing a pivotal role in the modern technology space.
According to a report published by Forrester, 86% of IT leaders are making containerization a priority. The catch however is that despite the extensibility of Kubernetes, the majority of the enterprises still need to understand how to leverage Kubernetes to its full potential. Kubernetes innately can support workload consolidation for a single cluster only; however, to scale up the performance in several scenarios, a multi-cluster approach is needed. Configuring access to multiple Kubernetes clusters makes the Kubernetes environment highly distributed. The approach also enables a more secured environment for cloud-native applications and facilitates better management of the tasks for departmental and cross-departmental operations within the organization.
If you have distributed workloads across multiple regions, having access to the multiple Kubernetes clusters can be beneficial. Such a setting allows an even distribution of workload across multiple clouds or hybrid clouds with on-premises or within a single or multi-cloud set-up. Configuring access to multiple Kubernetes clusters is a cost-effective technique that makes applications more efficient and functional. However, managing these multiple clusters can be a daunting task.
So, while your organization needs Kubernetes to evenly distribute the workload through apps and better orchestrate your servers in the data center, possibly you need to manage multiple clusters and be on all toes all the time. This guide will help you understand how you can manage access to Kubernetes clusters at scale and how you can streamline, optimize and orchestrate this process with Teleport.
Leveraging multiple Kubernetes clusters can help manage workloads across regions, limiting outrage blast radius; these also help manage compliance issues, hard multitenancy, security, and specialized solutions.
Why Do You Need to Manage Access to Multiple Clusters?
With Kubernetes multi-cluster management, employees can monitor everything happening on all the clusters and can opt for a best practice approach. Kubernetes multi-cluster operations facilitate:
· The creation, up-gradation, and deletion of Kubernetes clusters across multiple environments (such as data centers, private, hybrid, and public clouds, and at the Edge)
· Upgradation of the control plane and compute nodes
· Management of application life cycles across hybrid environments
· Scaling, securing, and up-gradation of clusters, even the provider-independent ones
· Maintenance and up-gradation of multiple nodes
· Searching, finding, and modification of multiple nodes
· Searching, finding, and up-gradation of any Kubernetes resource
· Implementation of Role-Based Access Control (RBAC) over the clusters (For instance, while a database administrator can have access to all clusters, a developer can have access to only the dev cluster.)
· The different resource quotas can also be defined and distributed among clusters
· Several pod budget policies can also be created
· Creation of network and governance policies
· Defining taints and toleration on the clusters
· Scanning the clusters for risks and vulnerabilities
Popular cloud providers such as AWS, Google Cloud, and Microsoft Azure facilitate clients by helping them manage their master node. The master node is responsible for managing the cluster and is responsible for maintaining the state of the cluster. To communicate with master nodes one can use the Kubernetes client tool Kubectl. With provider-managed Kubernetes services, the cloud providers leverage the Kubernetes services managed by them such as Amazon Elastic Kubernetes service, Google Kubernetes Engine, or Azure Kubernetes service. The clients don’t have to worry about the provision or management of the master node. The managed versions are slightly different for various cloud providers. Most of the cloud providers aid dedicated support, pre-configured environments, and hosting.
The top provider-managed Kubernetes services include Google Kubernetes Engine, Amazon Elastic Kubernetes Service, and Azure Kubernetes Service. Some of these services have matured to a point where enterprises can even handle the keys to their clusters. These services facilitate automated deployment, auto-scaling, optimized SLAs, and management of containerized and microservices applications.
Having a Kubernetes cluster for each environment such as in-house dev, test, and prod cluster allow you to run all the application instances within a specific environment. This approach isolates all the environments from each other and is specifically significant for the prod environment. So, even if some misconfiguration occurs in your dev cluster, the prod versions of your app will still be safe from any sort of havoc.
The idea of virtualizing a Kubernetes cluster is similar to that of virtualizing a physical machine. The idea can be a bliss for developers and while the host system can be used for actual computing everything else can be mirrored.
Virtual clusters spin up Kubernetes clusters within existing clusters and sync certain core resources between these two clusters. A host cluster runs the actual virtual cluster pods and needs to be a fully functional Kubernetes cluster. The virtual cluster itself consists of only the core Kubernetes components such as API server, controller manager and etcd. Virtual clusters can be highly beneficial in reducing the cost and efforts of your DevOps team. Instead of creating many small independent clusters, virtual clusters offer the following advantages:
· Less cluster boilerplate (there’s one k3 pod in a shared host cluster against a complete standalone Kubernetes cluster)
· These clusters are easier to manage and helm deploy or delete against the custom terraform scripts
· There is less startup and teardown time
· These clusters are cost-effective and better isolated than namespaces
DevOps teams can use virtual clusters for testing, experimentation, and cloud-native development.
Having many small single-use clusters allows employees to set up separate Kubernetes clusters for every deployment unit. This way Kubernetes can be used as a specialized application runtime for individual application instances offering benefits such as reduced blast radius, isolation, and few users facilitating fewer instances for a breakdown or havoc.
Content Delivery Networks (CDNs) are excellent examples of such types of clusters offering less latency and better Livestream capabilities. To fulfill the modern Livestream and video streaming needs, low latency is vital. This is when edge computing & CDNs become crucial as they bring data storage and computation closer to end-users.
DevOps teams can use virtual clusters for testing, experimentation, and cloud-native development.
Having many small single-use clusters allows employees to set up separate Kubernetes clusters for every deployment unit. This way Kubernetes can be used as a specialized application runtime for individual application instances offering benefits such as reduced blast radius, isolation, and few users facilitating fewer instances for a breakdown or havoc.
Content Delivery Networks (CDNs) are excellent examples of such types of clusters offering less latency and better Livestream capabilities. To fulfill the modern Livestream and video streaming needs, low latency is vital. This is when edge computing & CDNs become crucial as they bring data storage and computation closer to end-users.
Contrary to this, lots of different clusters also exist in form of cluster per application and cluster per environment. With cluster per application, you can have a separate cluster for the instances of a specific application and you can see this as a generalization of a cluster per team since usually, a team develops one or more apps. Such clusters can be customized to an app. Then there are clusters per environment wherein you have a separate cluster for each environment. One can have a dev, test, and prod environment and one can run all the application instances of a specific environment. Such environment facilitates isolation of the prod environment, customized cluster for and environment, and lock down access to prod cluster. Lack of isolation between apps and non-localized app requirements can be the major loopholes.
How Can You Manage Cluster Access at Scale
Configuring and managing multi-cluster Kubernetes deployments can be a daunting task. Many inherent challenges go along with using a multi-cluster Kubernetes architecture. To start with the architecture is complex and this complexity becomes apparent in the form of the following challenges:
a) Security
Security can be one of the vulnerable aspects of the multi-cluster Kubernetes architecture. Every cluster has its own set of roles and security certificates to support that must be managed across data centers and clusters. Having some sort of multi-cluster certificate management system in place would work. Also, the system admins and security personnel will have to be more attentive to role and user creation on a cross-cluster basis. This order is quintessential to support a secure, multi-cluster implementation of Kubernetes.
b) Firewall Configuration
Within a multi-cluster Kubernetes setup, marketers are accessing several API servers. In such a scenario, you just have to spin up a new cluster. This ensures that you have access to the cluster's API server and other desired clusters through the firewall. It’s an added and ongoing work and indeed one can add intelligence to the automation scripts to address the access issues; however, that increases the complexity of the scripts.
c) Deployment
Kubernetes architecture deployments become more complicated in multi-clusters. Within a multi-cluster environment, risks can be significant. One faulty check-in to the source code management (SCM) by a developer can bring down a cluster or where marketers have implemented replication across several clusters.
The basic complications with multi-cluster deployments are all connected to more complexity, majorly in terms of adoption and maintenance. However with this complexity also comes the benefit of versatility and resiliency for large-scale enterprise applications that operate globally.
Multi-cluster architectures are quintessential for the evolution of Kubernetes & offer ways to design applications that can operate across an array of data centers in a versatile yet controlled manner. Also, multi-cluster provides added resiliency for applications running at the enterprise level.
Also, multi-cluster Kubernetes deployments help in optimizing users’ experiences. For maintaining large-scale applications with thousands of users, multi-clusters are core components that help in architecting and optimizing users' experiences. The benefits that come with a multi-cluster environment include regulatory compliance, cluster management, application isolation, increased scalability and availability, and distributed applications.
To take full advantage of Kubernetes architects should take out time to learn details about multi-cluster Kubernetes implementations. You can research tools, techniques, and services that make working with multi-cluster architectures an easier affair.
Segmenting Workloads in Kubernetes
Segmenting workloads in Kubernetes requires you to run different pods or set up different namespaces. For bringing about an optimum level of granularity and the high availability and performance benefits that it can bring – a multi-cluster deployment is quintessential.
Multi-cluster doesn’t essentially have to imply multi-cloud; all the Kubernetes clusters in a multi-cluster deployment can run within the same cloud or within the same local data center if deployed outside the cloud. One of the key benefits that multi-cluster offers is spreading workloads across a wider geographical area and this kind of deployment is common within a multi-cloud architecture.
Also, a multi-cluster deployment hasn't essentially to be managed through a single control plane or interface. Technically, marketers can set up two different clusters and can manage them with the help of totally separate tooling before they can be termed as a multi-cluster enterprise. Multi-cluster deployments can be managed through a single platform. A multi-cluster deployment may or may not have multiple masters and hence it shouldn’t be confused with multi-master or multi-tenant deployments.
How Multi-Cluster Deployments Are Beneficial
As discussed above, deploying multiple Kubernetes clusters can benefit you immensely; however, the four prime benefits are enlisted as under:
a) Lower Latency
By deploying multiple clusters, one can deploy workloads close to different groups of users. For instance, one can also run one cluster in the cloud and another in a co-location center close to your target demographics example. By reducing the geographic distance between your users and your clusters, you can reduce latency.
b) Availability
With multiple clusters, you can improve the availability of your workload and one cluster can be used as a failover or backup environment if another cluster fails. By spreading clusters between different data centers and / clouds, you can avoid the risk that the failure of a single data center and / cloud will bring along, i.e. to disrupt all of your workloads.
c) Scalability
Multiple cluster deployment can prove to be beneficial to scale up your workload as and when required. If everything runs in a single cluster, it gets tougher to determine the specific workloads requiring more resources or replicas even if one doesn’t have sufficient performance data for specific workloads.
Another disadvantage of deploying a single cluster is that one can easily run into “noisy neighbor” issues. For very large clusters, one may face issues when needing to store data greater than 5,000 pods per cluster.
d) Workload Isolation
Configuring access to multiple clusters facilitates maximum possible isolation between workloads. Workloads in separate clusters can’t consume mutual resources or communicate with one other. This approach is specifically useful if you have multiple teams or departments looking forward to deploying workloads on Kubernetes and you don’t wish to be perturbed by the noisy-neighbor or privacy issues. The approach also helps separate a dev/test environment from production or even you’re experimenting with different Kubernetes settings. Having access to multiple Kubernetes clusters doesn't risk configuration change that clouds cause issues for production workloads.
e) Flexibility
One who runs multiple clusters in Kubernetes gains fine-grained control over how each cluster is configured. One can choose a different version of Kubernetes for each cluster, for instance, or choose a different CNI.
The configuration flexibility of multi-cluster deployments is beneficial if one has an application that depends on a certain setup or version of a tool in the stack. This approach is also valuable if one wants to test a new version of Kubernetes in an isolated dev/test cluster before upgrading your production clusters to the new version.
f) Security and Compliance
With multi-cluster Kubernetes, there also follow some security and compliance benefits. Strict isolation between workloads mitigates the risk that a security issue in one pod escalates to impact others.
Segmenting workloads in different clusters enables the strongest isolation. Running multiple clusters facilitates keeping pace with certain compliance rules. For instance, if one needs to keep some workloads on-premises or key data within a specific geographical region due to regulatory requirements, one can deploy a cluster in a location that addresses those requirements while running other clusters elsewhere.
When deciding between single or multi-cluster deployment, scale becomes a vital factor. Within larger organizations, usually, a greater number of applications need to be deployed and there are hence more chances to gain from multiple clusters.
One’s approach to dev/test also becomes vital. For those who want to run their development applications in the same cluster as production, multiple clusters aren't much significant. On the contrary, multiple clusters can be a good strategy if one is planning to isolate dev/test from production.
The dispersion of your users is something that needs to be thought about. Those who have users spread over a large geographical area, need multiple clusters to reduce latency and improve performance.
By leveraging Kubernetes management tooling, one can efficiently manage multiple clusters. A multi-cluster deployment can, however, create more hassle than it is worth. Those who can deploy and manage multiple clusters easily can reap the maximum benefits of a multi-cluster architecture without having management complexity undercut the value that Kubernetes facilitates.
Approaches to Multi-Cluster Kubernetes Deployments
There are several ways to set up and manage multi-cluster deployment:
- Setting Up Multiple Cluster Manually
The simplest way to set up multiple Kubernetes clusters is the so-called DIY approach. This method requires optimum effort but provides you with maximum flexibility over where clusters run and how they’re managed. Clusters can be set up virtually in any cloud or private data center, and you can manage them using platforms that support multi-cluster management such as Teleport.
- Using a Multi-Cluster Distribution
A Kubernetes distribution is designed for multi-cluster support. Most of the major distributions now do this:
Anthos: Anthos is Google’s Kubernetes-based hybrid cloud platform that can manage clusters running on multiple clouds as well as on-premises. The management layer is provided by GKE, Google’s Kubernetes distribution.
EKS Anywhere: Amazon's recently announced EKS Anywhere platform is endowed with features to allow marketers to deploy and manage clusters both in the AWS cloud and on-premises. EKS Anywhere is also likely to support clusters in other public clouds; however, that's still a concept.
Tanzu: Tanzu is VMware’s Kubernetes platform that supports multiple clusters running in any public cloud or on-premises, as long as the clusters conform to CNCF standards.
How To Configure Access to Multiple Clusters
After your clusters, users, and contexts are defined in one or multiple configuration files. Also, you can quickly switch between clusters by using the following command:
kubectl config use-context
The file that is used to configure access to a cluster is sometimes also called a kubeconfig file. This usually is a generic way of referring to configuration files. It doesn’t signify that a file named kubeconfig exists.
You should only use kubeconfig files from trusted sources. Usually, a specifically-crafted kubeconfig file can result in malicious code execution or file exposure. Those using kubeconfig files from untrusted sources must first inspect it carefully as much as they would like to inspect a shell script.
Pre-Requisites
Before one begins, one needs to have a Kubernetes cluster, and the kubectl command-line tool to be configured for users to communicate with their cluster. The people who don’t yet have a Kubernetes cluster can create one by using minikube or can use one of the following Kubernetes playgrounds:
a) Kataconda
b) Play with Kubernetes
To check whether kubectl is installed, one needs to run kubectl version –client. The kubectl version should be within one minor version of your cluster’s API server.
Define clusters, users, and contexts
Suppose you have two clusters, one for development of work and one for scratch work. In the development cluster, your frontend developers work in a namespace called frontend, and your storage developers work in a namespace called storage. In your scratch cluster, developers work in the default namespace, or they create auxiliary namespaces as they find useful. While access to the scratch cluster requires authentication by certificate, access to the scratch cluster requires authentication by username and password.
One can first start with creating a directory named config-exercise. Within the config-exercise directory, one can create a file named config-demo using the content below:
apiVersion: v1
kind: Config
preferences: {}
clusters:
- cluster:
name: development
- cluster:
name: scratch
users:
- name: developer
- name: experimenter
contexts:
- context:
name: dev-frontend
- context:
name: dev-storage
- context:
name: exp-scratch
Within configuration file clusters, users and contexts are described. Your config-demo file has the framework to illustrate two clusters, two users, and three contexts.
Once you go to the config-exercise directory, you can enter these commands to add cluster details to your configuration file.
kubectl config --kubeconfig=config-demo set-cluster development --server=https://1.2.3.4 --certificate-authority=fake-ca-file
kubectl config --kubeconfig=config-demo set-cluster scratch --server=https://5.6.7.8 --insecure-skip-tls-verify
Also, the user details can be added to your configuration file:
kubectl config --kubeconfig=config-demo set-credentials developer --client-certificate=fake-cert-file --client-key=fake-key-seefile
kubectl config --kubeconfig=config-demo set-credentials experimenter --username=exp --password=some-password
Note:
- To delete a user you can run kubectl --kubeconfig=config-demo config unset users.<name>
- To remove a cluster, you can run kubectl --kubeconfig=config-demo config unset clusters.<name>
- To remove a context, you can run kubectl --kubeconfig=config-demo config unset contexts.<name>
Add context details to your configuration file:
kubectl config --kubeconfig=config-demo set-context dev-frontend --cluster=development --namespace=frontend --user=developer
kubectl config --kubeconfig=config-demo set-context dev-storage --cluster=development --namespace=storage --user=developer
kubectl config --kubeconfig=config-demo set-context exp-scratch --cluster=scratch --namespace=default --user=experimenter
Open your config-demo file to see the added details. Alternatively, you can use the config view command instead of the config-demo file.
kubectl config --kubeconfig=config-demo view
The output displays the two clusters, two users, and three contexts:
apiVersion: v1
clusters:
- cluster:
certificate-authority: fake-ca-file
server: https://1.2.3.4
name: development
- cluster:
insecure-skip-tls-verify: true
server: https://5.6.7.8
name: scratch
contexts:
- context:
cluster: development
namespace: frontend
user: developer
name: dev-frontend
- context:
cluster: development
namespace: storage
user: developer
name: dev-storage
- context:
cluster: scratch
namespace: default
user: experimenter
name: exp-scratch
current-context: ""
kind: Config
preferences: {}
users:
- name: developer
user:
client-certificate: fake-cert-file
client-key: fake-key-file
- name: experimenter
user:
password: some-password
username: exp
The fake-ca-file, fake-cert-file, and fake-key-file described above constitute the placeholders for the pathnames of the certificate files. You must change these to actual pathnames of certificate files in your environment.
Sometimes you may wish to use Base64-encoded data embedded here instead of separate certificate files; in that case, you need to add the suffix -data to the keys, for example, certificate-authority-data, client-certificate-data, client-key-data.
Every cluster is a triple (cluster, user, namespace). For instance, the dev-frontend context illustrates, “Use the credentials of the developer uses to access the frontend namespace of the development cluster."
See the current context:
kubectl config --kubeconfig=config-demo use-context dev-frontend
The output demonstrates configuration information aligned with the dev –frontend context:
apiVersion: v1
clusters:
- cluster:
certificate-authority: fake-ca-file
server: https://1.2.3.4
name: development
contexts:
- context:
cluster: development
namespace: frontend
user: developer
name: dev-frontend
current-context: dev-frontend
kind: Config
preferences: {}
users:
- name: developer
user:
client-certificate: fake-cert-file
client-key: fake-key-file
Now, what if you want to work for a while in the scratch cluster? Change the current context to exp-scratch:
kubectl config --kubeconfig=config-demo use-context exp-scratch
Now any kubectl command that you give will apply to the default namespace of the scratch cluster. Also, the command will use the credentials of the user listed in the exp-scratch context.
You can also view the configuration associated with the new current context, exp-scratch.
kubectl config --kubeconfig=config-demo view –minify
Finally, if one needs to work for a while in the storage namespace of the development cluster, one can change the current content to dev-storage:
kubectl config --kubeconfig=config-demo use-context dev-storage
You can also view configuration associated with the new current context, dev-storage:
kubectl config --kubeconfig=config-demo view –minify
Creating a second configuration file
In their config-exercise directory, you can create a file named config-demo-2 with the following content:
apiVersion: v1
kind: Config
preferences: {}
contexts:
- context:
cluster: development
namespace: ramp
user: developer
name: dev-ramp-up
The preceding config file defines a new context named dev-ramp-up.
Setting Up the KUBECONFIG environment Variable
You need to see if you have an environment variable named KUBECONFIG. If so, save the current value of your KUBECONFIG environment variable, so that you can restore it later. For example:
Linux
export KUBECONFIG_SAVED=$KUBECONFIG
Windows PowerShell
$Env:KUBECONFIG_SAVED=$ENV:KUBECONFIG
$Env:KUBECONFIG=("config-demo;config-demo-2")
In your config-exercise directory, you can enter the following command:
kubectl config view
The output shows merged information from all the files listed in your KUBECONFIG environment variable. The merged information contains the dev-ramp-up context from the config-demo-2 file and the three contexts from the config-demo file.
contexts:
- context:
cluster: development
namespace: frontend
user: developer
name: dev-frontend
- context:
cluster: development
namespace: ramp
user: developer
name: dev-ramp-up
- context:
cluster: development
namespace: storage
user: developer
name: dev-storage
- context:
cluster: scratch
namespace: default
user: experimenter
name: exp-scratch
Exploring the $HOME/.kube directory
Those who already have a cluster can use kubectl to interact with the cluster. Then you’ll have a file named config in the $HOME/.kube directory.
Go to $HOME/.kube, and see what files are available. Typically, there’s a file named config. There might also be other configuration files in this directory. Briefly familiarize yourself with the contents of these files.
Append $HOME/.kube/cofig to your KUBECONFIG environment variable
If you have $HOME/ .kube/config file, and it’s not already listed in your KUBECONFIG environment variable, append it to your KUBECONFIG environment variable now. For example:
Linux
export KUBECONFIG=$KUBECONFIG:$HOME/.kube/config
Windows Powershell
$Env:KUBECONFIG="$Env:KUBECONFIG;$HOME\.kube\config"
Linux
export KUBECONFIG=$KUBECONFIG_SAVED
Windows PowerShell
export KUBECONFIG=$KUBECONFIG_SAVED
Return your KUBECONFIG environment variable to its original value. For example:
Linux
export KUBECONFIG=$KUBECONFIG_SAVED
Windows PowerShell
export KUBECONFIG=$KUBECONFIG_SAVED
Multi-cluster Kubernetes offers innumerable benefits, especially for teams that need to operate at a large scale or that require strict isolation between their workloads. To derive the maximum efforts out of a multi-cluster approach, it's critical to select a management platform that allows you to manage multiple clusters efficiently.
Teleport is an excellent example that provides access to Kubernetes clusters across all environments. The platform facilitates you to accomplish compliance requirements and has complete visibility into access and behavior. For DevOps environments, Teleport easily allows anyone to secure his Kubernetes clusters by leveraging security best practices. One can implement the industry best practices for Kubernetes access with minimum configuration. Teleport also allows easy enforcement of the multi-factor authentication (MFA), role-based access control (RBAC), Single sign-on (SSO) using identity-based short-lived X.509 certificates.
Access requests move away from admin accounts with just-in-time Kubernetes privilege escalation that’s enabled for administrative tasks. The access requests can be approved via Slack or other supported plugins. Teleport facilitates TLS routing wherein certificate-based protocol shrinks the network attack surface area of all your Kubernetes clusters to a single TCP/IO port and reduces operational overhead. The logged-in users are also allowed to implement multi-factor authorization of privileged operations.
The trusted clusters of Teleport have been designed to facilitate users to connect to compute infrastructure located behind firewalls without any open TCP ports. Some of the examples in ways they are used include:
a) Managed service providers (MSP) remotely managed the infrastructure of their clients.
b) Device manufacturers remotely maintain computing appliances deployed on-premises.
c) Large cloud software vendors manage multiple data centers using a common proxy.
Conclusion
By now you must have understood that Kubernetes multi-cluster deployment provides innumerable benefits in terms of performance (lower latency, availability, workload scalability), workload isolation, flexibility, and security.
The prime advantage of using Teleport to manage distribution to multiple clusters is that users are not tied to specific types of cluster configurations or tooling. Teleport facilitates users to configure each cluster differently (using different CNIs, for example) and can manage them centrally.
Teleport also provides multi-version support and is helpful for users looking to deploy different versions of Kubernetes in each cluster. This way users get the flexibility to run whichever version or versions of Kubernetes they need. This can be helpful if you want to test one version of Kubernetes in one cluster while confining it to a tried-and-tested version for your production clusters. Teleport allows you to manage all these clusters and versions centrally while allowing granular control over versioning.
A platform like Teleport is essential to get the maximum out of your multi-cluster approach as it allows you to manage multiple clusters efficiently and that’s the very essence of multi-cluster Kubernetes. The approach can be exceptionally beneficial for teams that need to operate at a large scale or that require strict isolation between their workloads.