[Kubernetes, Kubernetes in Production, Production Readiness Checklist, Kubernetes Best Practices]

Kubernetes in Production: Readiness Checklist and Best Practices for Resource Management

Deploying Kubernetes in production is no easy task. Kubernetes environments need to hold up on the rough seas of production from an availability, security, scalability, resilience, monitoring and resource management perspective. In this instalment of the Kubernetes Production Readiness and Best Practices Checklist series, we dive into the topic of Resource Management.

Hasham Haider

Hasham Haider

February 27, 2019

9 minute read

 

This is the second instalment in our blog series outlining a best practices checklist for Kubernetes in Production. In the first instalment, we looked into best practices for ensuring highly available Kubernetes deployments, both on the infrastructure as well as the application layer.

In this post, we will branch off into Production readiness best practices for resource management. But first a quick recap of what we mean by production ready Kubernetes; We defined production readiness for Kubernetes in terms of high availability, security, resource management, scalability and monitoring.

A production-ready Kubernetes deployment is one in which DevOps and Kubernetes administrators have implemented best practices for all of the above-mentioned aspects of their environment. With this checklist, DevOps and Kubernetes administrators can ensure they are following best practices and that their deployments are ready for production.  

We have already covered high availability best practices for Kubernetes in production in a previous blog post. You can download the complete production readiness and best practices checklist here.

Resource Management

Kubernetes has three types of resources; CPU and memory together referred to as compute resources and ephemeral storage. DevOps can apply both soft limits (resource requests) and hard limits (resource limits) to the amount of these resources that can be consumed by containers.

That, however, is just the start. Kubernetes resources can be managed on many different levels of abstraction. Kubernetes also provides many other knobs to manage resource consumption in production. Below we go through an extensive checklist of some of these to ensure efficient resource consumption for Kubernetes workloads in production.

Configured Resource Requests and Limits for Containers?

Resource requests and limits help you manage resource consumption by individual containers. Resource requests are a soft limit on the amount of resources that can be consumed by individual containers. Limits are the maximum amount of resources that can be consumed.

Resource requests and limits can be set for CPU, memory and ephemeral storage resources. Setting resource requests and limits is a Kubernetes best practice and will help avoid containers getting throttled due to lack of resources or going berserk and hogging resources.

To check whether all containers inside a pod have resource requests and limits defined use the following command 

kubectl describe pod -n <namespace_name> <pod_name>

This will display a list of all containers with the corresponding limits and requests for both CPU and memory resources.

Download the complete Checklist:

Banner with book cover and download button

Specified Resource Requests and Limits for Local Ephemeral Storage?

Local ephemeral storage is a new type of resource introduced in Kubernetes 1.8. Containers use ephemeral storage for local storage. If you have configured local ephemeral storage check to see that you have set requests and limits for each container for this resource type.

Here is how you can check whether requests and limits have been defined for local ephemeral storage for all containers:

kubectl describe pod -n <namespace_name> <pod_name>

Created Separate Namespaces for your Teams?

Kubernetes namespaces are virtual partitions of your Kubernetes clusters. It is recommended best practice to create separate namespaces for individual teams, projects or customers. Examples include Dev, production, frontend etc. You can also create separate namespaces based on custom application or organizational requirements. Here is how you can display a list of all namespaces:

kubectl get namespaces

Or

kubectl get namespaces --show-labels

You can also display a list of all the pods running inside a namespace with:

kubectl get pods --all-namespaces

Configured Default Resource Requests and Limits for Namespaces?

Default requests and limits specify the default values for memory and CPU resources for all containers inside a namespace. In situations where resource request and limit values are not specifically defined for a container created inside a namespace with default values, that container will automatically inherit the default values. Configuring default values on a namespace level is a best practice to ensure that all containers created inside that namespace get assigned both request and limit values.

Here is how you check whether a namespace has been assigned default resource requests and limits:

kubectl describe namespace <namespace_name>

Configured Limit Ranges for Namespaces?

Limit ranges also work on the namespace level and allow us to specify the minimum and maximum CPU and memory resources that can be consumed by individual containers inside a namespace.

Whenever a container is created inside a namespace with limit ranges, it has to have a resource request value that is equal to or higher than the minimum value we defined in the limit range. The container also has to have both CPU and memory limits that are equal to lower than the max value defined in the limit range.

Check whether limit ranges have been configured:

kubectl describe namespace <namespace_name>

Specified Resource Quotas for Namespaces?

Resource quotas also work on the namespace level and provide another layer of control over cluster resource usage.

Resource Quotas limit the total amount of CPU, memory and storage resources that can be consumed by all containers running in a namespace.

Consumption of storage resources by persistent volume claims can also be limited based on individual storage class. Kubernetes administrators can define storage classes based on quality of service levels or backup policies.

Check whether resource quotas have been configured:

kubectl describe namespace <namespace_name>

Configured Pod and API Quotas for Namespaces?

Pod quotas allow you to restrict the total number of pods that can run inside a namespace. API quotas let you set limits for other API objects like PersistentVolumeClaims, Services and ReplicaSets.

Pod and API quotas are a good way to manage resource usage on a namespace level.

To check whether quotas have been configured:

kubectl describe namespace <namespace_name>

Ensured Resource Availability for Etcd?

Typically, Etcd clusters have a pretty large resource footprint. Therefore, it is best practice to run these clusters on dedicated hardware to ensure they have access to enough resources. Resource starvation can lead to the cluster becoming unstable, which will, in turn, mean that no new pods can be scheduled.

Here is an etcd resource guide  based on the number of nodes in the cluster and the clients being served. Typical etcd clusters need 2-4 CPU cores, 8GB of memory, 50 sequential IOPS and a 1 Gbe network connection to run smoothly. For larger etcd clusters, check out this handy guide.

Configured Etcd Snapshot Memory Usage?

Etcd snapshots are backups of the etcd cluster which can be used for cluster disaster recovery. The --snapshot-count flag determines the number of changes that need to happen to etcd before a snapshot is taken. Higher --snapshot-count will hold a higher number of entries until the next snapshot, which can lead to higher memory usage. The default value for --snapshot-count in etcd v3.2 is 100,000.

Make sure to configure this number based on your unique cluster requirements.

You can do this using:

etcd --snapshot-count=X

Attached Labels to Kubernetes Objects?

Labels allow Kubernetes objects to be queried and operated upon in bulk. They can also be used to identify and organize Kubernetes objects into groups. As such defining labels should figure right at the top of any Kubernetes best practices list. Here is a list of recommended Kubernetes labels that should be defined for every deployment.

Check whether pods have been labelled

kubectl get pods --show-labels 

Limited the Number of Pods that can Run on a Node?

You can also control the number of pods that can be scheduled on a node using the --max-pods flag in Kubelet.

This will help avoid scenarios where rogue or misconfigured jobs create pods in such large numbers as to overwhelm system pods.

Reserved Compute Resources for System Daemons?

Another best practice is to reserve resources for system daemons that are needed by both the OS and Kubernetes itself to run. All three resource types CPU, memory and ephemeral storage resources can be reserved for system daemons. Once reserved these resources are deducted from node capacity and are exposed as node allocable resources. Below are kubelet flags that can be used to reserve resources for system daemons:

--kube-reserved: allows you to reserve resources for Kubernetes system daemons like the kubelet, container runtime and node problem detector.

--system-reserved: allows you to reserve resources for OS system daemons like sshd and udev. 

Configured API Request Processing for API Server?

To manage CPU and memory consumption by the API server, make sure to configure the maximum number of requests that can be processed by the API server in parallel.

This can be done using the --max-requests-inflight and --max-mutating-requests-inflight flags.

Processing a lot of API requests in parallel can be very CPU intensive for the API server and can also lead to OOM (out of memory) events.

Configured out of Resource Handling?

Make sure you configure out of resource handling to prevent unused images and dead pods and containers taking up too much unnecessary space on the node.

Out of resource handling specifies Kubelet behaviour when the node starts to run low on resources. In such cases, the Kubelet will first try to reclaim resources by deleting dead pods (and their containers) and unused images. If it cannot reclaim sufficient resources, it will then start evicting pods.

You can influence when the Kubelet kicks into action by configuring eviction thresholds for eviction signals. Thresholds can be configured for nodefs.available, nodefs.inodesfree, imagefs.available and imagefs.inodesfree eviction signals in the pod spec.

Following are some examples:

  • nodefs.available<10%
  • nodefs.inodesFree<5%
  • imagefs.available<15%
  • imagefs.inodesFree<20%

Doing this will ensure that unused images and dead containers and pods do not take up unnecessary disk space.

You should also consider specifying a threshold for memory.available signal. This will ensure that Kubelet kicks into action when free memory on the node falls below your desired level.

Another best practice is to pass the --eviction-minimum-reclaim to Kubelet. This will ensure that the Kubelet does not pop up and down the eviction threshold by reclaiming a small amount of resources. Once an eviction threshold is triggered the Kubelet will evict pods till the minimum threshold is reached.

Using Recommended Settings for Persistent Volumes?

Persistent Volumes (PVs) represent a piece of storage in a Kubernetes cluster. PVs are similar to regular Kubernetes volumes with one difference; they have a lifecycle that is independent of any specific pod in the cluster. Kubernetes volumes, on the other hand, do not persist data across pod restarts.

Persistent Volume Claims (PVCs) are requests for storage resources by a user. PVCs consume PV resources in the same way that a pod consumes node resources.

When creating PV’s Kubernetes documentation recommends the following best practices:

  • Always include Persistent Volume Claims in the config
  • Never include PVs in the config
  • Always create a default storage class
  • Give the user the option of providing a storage class name

Enabled Log Rotation?

If you have node-level logging enabled, make sure you also enable log rotation to avoid logs consuming all available storage on the node. Enable the logrotate tool for clusters deployed by the kube-up.sh script on GCP or Docker’s log-opt for all other environments.

Prevented Kubelet from Setting or Modifying Label Keys?

If you are using labels and label selectors to target pods to specific nodes for security or regulatory purposes, make sure you also choose label keys that cannot be modified by the Kubelet. This will ensure that compromised nodes cannot use their Kubelet credentials to label their node object and schedule pods.

Make sure you use the Node authorizer, enable the NodeRestriction admission plugin and always prefix node labels with node-restriction.Kubernetes.io/ as well as add the same prefix to label selectors.

Download the Complete Checklist with Checks, Recipes and Best Practices for Resource Management, Security, Scalability and Monitoring for Production-Ready Kubernetes

Banner with book cover and download button

Hasham Haider

Author

Hasham Haider

Fan of all things cloud, containers and micro-services!

Want to Dig Deeper and Understand How Different Teams or Applications are Driving Your Costs?

Request a quick 20 minute demo to see how you can seamlessly allocate Kubernetes costs while saving up to 30% on infrastructure costs using Replex.

Schedule a Meeting