The cloud led to a fundamental change in the way enterprises procure and provision IT resources. Kubernetes has changed the way IT teams consume those resources. It has also allowed enterprises to innovate faster, improve developer efficiency, efficiency and portability and unlock scale.
But before enterprises can take advantage of the benefits that come with well-architected enterprise Kubernetes environments, they first have to set up and configure these environments and figure out how to manage and operate them.
That is no easy task. There are two ways to go about setting up enterprise Kubernetes environments: the hard way and the “comparatively” easy way.
Kelsey Hightower has outlined the hard way in exquisite detail here. That however is only recommended for learning outcomes and not for enterprise environments.
The "comparatively" easy way to go about setting up and configuring enterprise Kubernetes environments is to use one of a number of managed Kubernetes distributions.
Kubernetes distributions are prebuilt software platforms that make it easier to deploy, manage and operate fully fledged cloud native Kubernetes environments.
Most Kubernetes distributions provide a baseline feature set that makes it easier to install, configure and upgrade Kubernetes environments. Some allow enterprises to cast the net wider and integrate other cloud native tools to monitor, observe and secure Kubernetes environments.
In this article, we will outline the features that CIOs, CTOs and ITDMs need to consider when evaluating enterprise Kubernetes distributions. In an upcoming article we will compare the top 5 Kubernetes distributions on the CNCF cloud native landscape and compare them based on the feature set outlined in this blog post.
So let's get started.
Ease of setup, installation, cluster creation and configuration is the very first feature set CIOs and ITDMs need to evaluate managed Kubernetes distributions on.
CIOs should evaluate and compare the workflow and UI for spinning up and configuring Kubernetes clusters as well as managing and operating them. This evaluation should extend to the workflow for deploying pods, deployments, services and applications.
Some distributions also provide cluster and asset inventory dashboards that provide a central repository for all assets, infrastructure and containerised workloads across cloud, on-premise and hybrid environments. These are the ones that should be preferred.
One key element that Kubernetes distributions need to be evaluated on is high availability and disaster recovery. Both are essential to enterprise environments and are also important aspects of Kubernetes cluster configuration that are not native to Kubernetes.
The distribution chosen should support provisioning of Kubernetes clusters in a highly available configuration. Highly available cluster configurations are ones with multiple Kubernetes master and etcd nodes. 3 is the minimum requirement for a cluster configuration to be highly available. For etcd, Kubernetes recommends operating a static 5-member cluster.
The distribution chosen should also support etcd backups, snapshots and restore. Since etcd stores cluster state the ability to create backups and snapshots and restore them are essential to disaster recovery.
Hybrid and multi-cloud support is another aspect that CIOs and ITDMs need to evaluate managed Kubernetes distributions on. This is especially true of enterprises that need to maintain an on-premise footprint because of regulatory or compliance requirements.
The ability to deploy and manage clusters across any of the major cloud providers and bare-metal servers will allow enterprise workloads to be more portable as well as help them avoid vendor lock-in.
For on-premises deployments, the choice of a distribution will depend on the virtualisation solution being used, however, support for any of the most popular one’s like VMware Vsphere, Openstack, Microsoft Hyper-V or KVM is a good start.
Day-2 operations kick in once clusters have been deployed and configured. Day-2 operations on Kubernetes usually include activities like observability, monitoring, maintenance and troubleshooting. Kubernetes distributions under consideration should provide a broad feature set for all three aspects of day-2 operations.
CIOs and ITDMs should start off by comparing Kubernetes distributions based on stated SLA, uptime and availability numbers. They can then dig deeper to evaluate distributions based on degree of support for monitoring of the Kubernetes control plane, cluster nodes, pods, applications and other Kubernetes components.
Monitoring of cluster health, automated alerts, troubleshooting and level of support provided are also important features to consider.
With the emphasis placed on making systems observable, the degree of integration with popular metrics, tracing and logging tools should also figure in the decision to choose a managed Kubernetes distribution.
Seamless, automated upgrades and rollbacks should be another baseline requirement. Kubernetes clusters deployed using the chosen managed Kubernetes distribution should be easy to upgrade without any downtime as well as easy to rollback in case of any issues.
Identity and access management are important aspects of enterprise Kubernetes environments. Kubernetes provides a native RBAC feature that allows operators to regulate access to resources.
The managed Kubernetes distribution chosen should support and build on the native RBAC resource to provide access control and resource isolation for multi-cluster environments. Support for identity management and SSO tooling like Okta, Active Directory, Auth0 or other SAML tools should also be a baseline requirement.
The Kubernetes networking model has been designed to be less-complex overall. There are a couple of reasons for this: to allow the easy migration of workloads that were previously hosted in VMs to containers and to make it highly configurable and pluggable.
This low-complexity design also allows the Kubernetes networking model to be implemented in a number of ways. The Kubernetes distribution chosen should either have a native SDN solution that can meet the requirements of enterprise workloads or integrate with one of the more popular CNI based networking implementations like Flannel, Calico, Weavenet or OVN etc.
Storage is a hard nut to crack in the context of Kubernetes. One reason for this is the fact that containers were initially designed to host stateless applications.
To support enterprise grade stateful applications however, Kubernetes introduces a number of native abstractions that make it easier to provision and manage storage for containerised applications. Persistent volumes (PVs) is one such storage abstraction that has a life cycle independent of pods or applications.
To support PVs enterprises need to ensure that the Kubernetes distribution chosen integrates a storage solution already in use internally or supports one that has a feature set compatible with their workloads. Some of the more famous Kubernetes storage solutions include OpenEBS, Potworx, StorageOS and Rook.
Want to learn more? Download the complete CIOs Guide to Kubernetes:
Fan of all things cloud, containers and micro-services!
A review of the best practices, processes and cultural paradigms that are recommended by the FinOps foundation. These best practices and processes are instrumental in developing and operating a successful FinOps practice that views the cloud as a driver of innovation and business value while at the same time improving transparency and accountability.
April 12, 2021
4 min read
Part 3 of the Ultimate guide to cloud FinOps blog series, which outlines core FinOps principles, and provides an in-depth review of each one.
April 6, 2021
4 min read
Part 2 of the Ultimate guide to cloud FinOps blog series, which takes a deep dive into FinOps domains and roles, reviews the main responsibilities of those domains and identifies the current organizational roles that are candidates for inclusion in FinOps teams.
March 22, 2021
4 min read