How to limit Kubernetes cluster resources usage

You are in charge of a Kubernetes cluster, and you want to limit the CPU and the memory resources usage.

You can the following methods to limit resources usage:

1. ResourceQuota - limit total resources usage in a specific namespace
2. LimitRange - limit resources usage for any pod in a specific namespace

ResourceQuota Example 

Lets create a new namespace named managed1.

kubectl create namespace managed1

Configure a ResourceQuota to limit the resources allow per the entire namespace:

kubectl --namespace=managed1 create -f quota.yaml

Where the quota.yaml is:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: quota-limit
spec:
  hard:
    memory: "200Mi"

Create a deployment. In this example, we use an NGINX deployment.

kubectl --namespace=managed1 create -f nginx.yaml

Where the nginx.yaml deployment is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          limits:
            memory: "100Mi"
            cpu: "200m"

Notice that we have specified resources limits of 100 MB per pod.

Checking the deployment status:

kubectl --namespace=managed1 get deployments

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   2/3     2            2           26s

We can see that only 2 out of the required 3 replicas are started.

We can find the cause when running the command:

kubectl --namespace=managed1 describe replicasets nginx-deployment-7d84d68fbf

And see the error:

Error creating: 
pods "nginx-deployment-7d84d68fbf-tzg58" is forbidden: exceeded quota: 
quota-limit, requested: memory=100Mi, used: memory=200Mi, limited: memory=200Mi

LimitRange Example 

Lets create a new namespace named managed1.

kubectl create namespace managed1

Configure a LimitRange to limit the resources allow per pod:

kubectl --namespace=managed1 create -f limit.yaml

Where the limit.yaml is:

apiVersion: v1
kind: LimitRange
metadata:
  name: limit-pod
spec:
  limits:
  - max:
      cpu: "100m"
      memory: "200Mi"
    type: Pod

Create a deployment. In this example, we use an NGINX deployment.

kubectl --namespace=managed1 create -f nginx.yaml

Where the nginx.yaml deployment is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          requests:
            memory: "100Mi"
            cpu: "200m"

Notice that we have specified resources requests that are over the LimitRange, but we did not specify resource limits.

We can see that no pods were created for the deployment,
We can find the cause when running the command:

kubectl --namespace=managed1 describe replicasets nginx-deployment-656df65f95

And see the error:

Error creating: pods "nginx-deployment-656df65f95-p2qng" is forbidden: 
[maximum cpu usage per Pod is 100m.  
No limit is specified, maximum memory usage per Pod is 200Mi.]

If we update the NGINX deployment, and add resources limits:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          requests:
            memory: "100Mi"
            cpu: "200m"
          limits:
            memory: "100Mi"
            cpu: "200m"

We get another error in the kubectl command:

Error creating: pods "nginx-deployment-9b6fc86cd-qp9gc" is forbidden: 
maximum cpu usage per Pod is 100m, but limit is 200m

Further details

For more details, see the kubernetes documentation: ResourceQuota, LimitRange

What is this KISS blog?

This blog is a KISS technical blog!

In many technical sites, I've found complicated long explanations that I've eventually used only 1% of them. While, in the long term, it might be great to be aware of all of the consequences of each action that we do, we usually need to do something quick, and move on.

Only later, and only for ~20% of the items, we later need to investigate more to find more about the item.

I've decided to publish some post about items I'm handling.

I will try using the following guideline:

1. Provide minimal information about the issue I'm addressing.
2. Give a TLDR summary of the steps to handle it.
3. Provide additional links and information for the ones who really need it. This is intended to be used by the ones who find the post as one of their 20% of interest.

I hope you will find this blog beneficial.