How to limit Kubernetes cluster resources usage

You are in charge of a Kubernetes cluster, and you want to limit the CPU and the memory resources usage.

You can the following methods to limit resources usage:

1. ResourceQuota - limit total resources usage in a specific namespace
2. LimitRange - limit resources usage for any pod in a specific namespace

ResourceQuota Example 

Lets create a new namespace named managed1.

kubectl create namespace managed1

Configure a ResourceQuota to limit the resources allow per the entire namespace:

kubectl --namespace=managed1 create -f quota.yaml

Where the quota.yaml is:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: quota-limit
spec:
  hard:
    memory: "200Mi"

Create a deployment. In this example, we use an NGINX deployment.

kubectl --namespace=managed1 create -f nginx.yaml

Where the nginx.yaml deployment is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          limits:
            memory: "100Mi"
            cpu: "200m"

Notice that we have specified resources limits of 100 MB per pod.

Checking the deployment status:

kubectl --namespace=managed1 get deployments

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   2/3     2            2           26s

We can see that only 2 out of the required 3 replicas are started.

We can find the cause when running the command:

kubectl --namespace=managed1 describe replicasets nginx-deployment-7d84d68fbf

And see the error:

Error creating: 
pods "nginx-deployment-7d84d68fbf-tzg58" is forbidden: exceeded quota: 
quota-limit, requested: memory=100Mi, used: memory=200Mi, limited: memory=200Mi

LimitRange Example 

Lets create a new namespace named managed1.

kubectl create namespace managed1

Configure a LimitRange to limit the resources allow per pod:

kubectl --namespace=managed1 create -f limit.yaml

Where the limit.yaml is:

apiVersion: v1
kind: LimitRange
metadata:
  name: limit-pod
spec:
  limits:
  - max:
      cpu: "100m"
      memory: "200Mi"
    type: Pod

Create a deployment. In this example, we use an NGINX deployment.

kubectl --namespace=managed1 create -f nginx.yaml

Where the nginx.yaml deployment is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          requests:
            memory: "100Mi"
            cpu: "200m"

Notice that we have specified resources requests that are over the LimitRange, but we did not specify resource limits.

We can see that no pods were created for the deployment,
We can find the cause when running the command:

kubectl --namespace=managed1 describe replicasets nginx-deployment-656df65f95

And see the error:

Error creating: pods "nginx-deployment-656df65f95-p2qng" is forbidden: 
[maximum cpu usage per Pod is 100m.  
No limit is specified, maximum memory usage per Pod is 200Mi.]

If we update the NGINX deployment, and add resources limits:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        resources:
          requests:
            memory: "100Mi"
            cpu: "200m"
          limits:
            memory: "100Mi"
            cpu: "200m"

We get another error in the kubectl command:

Error creating: pods "nginx-deployment-9b6fc86cd-qp9gc" is forbidden: 
maximum cpu usage per Pod is 100m, but limit is 200m

Further details

For more details, see the kubernetes documentation: ResourceQuota, LimitRange