Migrating from DC/OS to Kubernetes: The Challenges You May Face

Enterprises in the IT world forever look for the most adaptable and effective approach to build up tools for hosting the best services. As a result, many organizations are shifting away from VM’s to Containers.

Organizations might migrate to container platforms for different reasons — costs, scalability, operational expense, or the ability to quickly test their new application. So, one container-based model may not always be the best solution for all their needs, which is why many models of Container orchestration and management tools have developed, such as Kubernetes, Mesos (DC/OS), ECS, Swarm, and Nomad.

This blog will only discuss DC/OS and Kubernetes and what challenges you might face when migrating between these container platforms. We will also suggest some best practices that will help you mark the difference between a successful and non-successful migration between DC/OS and Kubernetes.

DC/OS (Data Center Operating System) was introduced as an open-source project by Mesosphere in 2016 to simplify the container orchestration platform Mesos. Mesos was initially written as a research project at Berkeley and was later adopted by Twitter to answer Google’s Borg (Kubernetes).

DC/OS comes with various frameworks and application stacks that provide resource allocation features. Each framework consists of a scheduler and an executor. Marathon is the framework, which provides users with an easy way to manage container orchestration.

It is also a great option for large-scale deployments; and for enterprises that need support for different classes of application technology, such as Java, containers, and databases; and for stateful applications.

Kubernetes is an open-source container orchestration platform that was released in 2014 by Google. Kubernetes automates the deployment, scaling, and management of containerized applications.

Kubernetes has been built mainly for container orchestration. It’s based on many years of experience managing Linux containers at Google. Kubernetes provides massive scalability and can be Deployed at scale very easily. Kubernetes is also backed by enterprise offerings such as Google (GKE) and RedHat (OpenShift) compared to DC/OS, which is only backed and distributed by Mesosphere.

Kubernetes can be classified as Container Tools, while DC/OS is grouped with a cluster management tool. Kubernetes can run on top of DC/OS and schedule containers with it. It is more of an orchestration framework compared to DC/OS, which is an Operating system that runs containerized workloads.

When migrating from DC/OS to Kubernetes. Implementation for highly available clusters will differ and requires manpower with a different set of skills for additional configuration, management, and tighter version control.

In DC/OS. Containers can be scheduled for HA without constraints using node placement. Each container can have a unique node given that the number of slave nodes should be equal to the number of containers. High availability for DC/OS and Marathon is supported using Zookeeper, which conducts the election of DC/OS clusters and maintains HA state during downtime and failures.

In Kubernetes, Deployments allow pods to be distributed among nodes to provide HA, by tolerating infrastructure or application failures. Load-balanced services detect unhealthy pods and remove them, which promotes clusters that have high uptime and performance. Multiple master nodes and worker nodes can be load balanced for HA from kubectl.

DC/OS networking model allows mapping container ports to host ports, which are a limited resource. The container will not automatically acquire an IP until a third party network tool such as calico is integrated. Multiple containers can also not share the same network namespace.

Compared to Kubernetes, which allows any pod to communicate with any service or other pods. Although It requires separate networks to operate, both having connectivity from inside the cluster. Kubernetes also make use of network addressing and routing depending on the topology of the cluster. This means you need to consider the complexity of clusters when you are migrating workloads from DC/OS, as it can affect service routing.

It is also important to know during migration what the network implementations of your cloud vendor are. They cater to a large volume of customers with a wide range of use cases across a large geographic span, which means they’ve implemented different solutions to optimize routing and delivery to reduce overhead.

Kubernetes is used by several small scale companies and developers and is also delivered as a service by platforms such as Red Hat, OpenShift, and Microsoft Azure compared to Mesos, which is only implemented by large organizations such as Twitter, Apple, and Yelp. Mesos network implementation can different as its core focus is on the reliability of data for large organizations. Determining how your staff will manage all of these network needs when transitioning from DC/OS to Kubernetes is challenging.

Load balancers sit in-line and play a crucial role in routing customer traffic to containerized applications. When migrating in bulk from DC/OS to Kubernetes. Configuring Load balancers becomes tricky as the infrastructure implemented by both platforms are different.

In DC/OS, Host ports can be mapped to multiple container ports, which can be reached through an acting load balancer, known as Mesos-DNS, compared to Kubernetes, where Pods are exposed through a Service, which can be a load balancer.

Kubernetes is ideal for clustering world beginners; it provides a quick and easy way to start experimenting with cluster oriented development. It offers a very high portability level since it is being supported by a lot of various providers such as Microsoft, IBM, VMWare. Kubernetes can be easily recommended for companies ready to migrate and want to effectively manage their containerized apps.

DC/OS, on the other hand, is suited for large systems and is designed for maximum reliability. DC/OS is a very stable platform, but overly complex for small-scale systems. That’s why It is usually recommended to companies that use multi-cloud and multi-region clusters such as Uber or Twitter.DC/OS is built primarily for big data, so if you have existing workloads such as Hadoop, Kafka, etc. It becomes challenging to migrate to another container orchestration platform because of the massive scalability it provides for these workloads

Kubernetes can scale up to 5000 nodes with no more than 150000 total pods and 300000 total containers. Compared to Mesosphere tier architecture (with Marathon), which has been scaled to 10,000 nodes.by organizations, such as Apple, Bloomberg, Netflix, etc.

(Source: Mesosphere blog)

Lack of single-vendor control in Kubernetes can complicate an enterprise migration decision. Kubernetes has a Community that includes many vendors such as Red Hat, Google, and many more. (Source: CNCF) compared to DC/OS, which is its sole vendor. Single vendor control may allow for more accountability with bug resolution and better collaboration with feature development, which can be a challenge for enterprises migrating to Kubernetes, which has a pool of different vendors.

Instead of scheduling containers for application deployment in DC/OS, Kubernetes schedules custom structures called Pods. A Pod simulates a group of co-located containers which can have clones across multiple nodes. Kubernetes schedule pods which are distributed among worker nodes. As compared to DC/OS, where applications run as tasks scheduled by Marathon distributed among slave nodes.

Enterprises who are migrating workloads from DC/OS to Kubernetes must realize that they might be getting savings or benefits from the new pod architecture of Kubernetes, but there is a learning curve which wants teams to acquire the necessary skills and learn to deploy applications properly using Kubernetes.

Migrating data between DC/OS and Kubernetes can be risky, time-consuming, and bring up issues. There are a lot of variables that factor into the data migration such as data type, how it is managed and stored, storage backend type, backups, and transfers

Kubernetes provides abstractions for individual storage backends (e.g., NFS, AWS EBS, Ceph, Flocker) compared to DC/OS, which can only make use of external storage, such as Amazon EBS. Also, A DC/OS Container using Marathon can only use persistent volumes, which are local to the node where they are created. Hence the container is always required to run on the same node.

Enterprise looking forward to migrating from DC/OS to Kubernetes has to understand that it comes with new abstractions and concepts; ensuring that the teams acquire these practices can be challenging for organizations.

DC/OS was intended for scale and speed: If those are your goals, it’s hard to beat the Mesos environment. In any case, Kubernetes has built a system of its own that is a lot more extensive and more dynamic than Mesos’. Kubernetes comes up short on the demonstrated scale and execution of DC/OS, yet its wide expansiveness of help implies the pace of progression in the Kubernetes space is unrivaled.