Dynamically Allocate and Delete PersistentVolumeClaim for a CronJob

 

In this post we will review the steps to dynamically create and delete a PersistentVolumeClaim for a CronJob.

A CronJob might require a large amount of temporary storage, and we don't want to keep the PersistenceVolumeClaim active while the job is not running, since the cost might be high. For example, assumeing we have a CronJob running once in a week for 3 hours, and required 1TB disk space for calculations. The cost of leaving such a disk active for an entire week is very high, hence we should dynamically allocate and remove the disk.

Kubernetes does not supply out of the box mechanism to handle this, hence we can do it ourselves. We handle this by 3 CronJobs:

1. The allocate CronJob which create the PVC

2. The actual computation CronJob

3. The cleanup CronJob

The Allocate CronJob

This CronJob creates the PVC before the computation job is starting. It should use a schedule that run it just a few minutes before the computation job. The following are the kubernetes entities to run the allocate CronJob. Notice that we must also provide permissions for handling the PVC.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: allocate-role
rules:
  - apiGroups: [ "" ]
    resources: [ "persistentvolumes" ]
    verbs: ["create","list","delete","get","patch","watch"]

  - apiGroups: [ "" ]
    resources: [ "persistentvolumeclaims" ]
    verbs: ["create","list","delete","get","patch","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: allocate-role-binding
subjects:
  - kind: ServiceAccount
    name: allocate-service-account
    namespace: default
roleRef:
  kind: ClusterRole
  name: allocate-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: allocate-service-account
  namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: allocate-config
data:
  pvc.yaml: |-
    
    
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: my-pvc
      labels:
        type: local
    spec:
      storageClassName: "gp2"
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 1000Gi

---
kind: CronJob
metadata:
  name: allocate-cronjob
spec:
  schedule: "0 0 * * *"
  startingDeadlineSeconds: 36000
  concurrencyPolicy: Replace
  timeZone: "Etc/UTC"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: allocate-service-account
          restartPolicy: Never
          containers:
            - name: cleanup
              image: repo/allocate/dev:latest
              imagePullPolicy: IfNotPresent
              env:
                - name: PVC_NAME
                  value: my-pvc
                - name: NAMESPACE
                  value: default              
              volumeMounts:
                - name: config
                  mountPath: /config
          volumes:
            - name: config
              configMap:
                name: allocate-config

The allocate image is a simple script that runs kubectl to create the PVC:

#!/usr/bin/env bash
set -e
set -x

echo "prepare starting"

kubectl delete pvc ${PV_NAME} --namespace ${NAMESPACE} --ignore-not-found=true
kubectl apply -f /config/pvc.yaml

echo "prepare done"

The Cleanup CronJob

The Cleanup CronJob runs after the computation job completion and deletes the PVC. This includes the following kubernetes entities:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cleanup-role
rules:
  - apiGroups: [ "" ]
    resources: [ "persistentvolumeclaims" ]
    verbs: ["create","list","delete","get","patch","watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cleanup-role-binding
subjects:
  - kind: ServiceAccount
    name: cleanup-service-account
    namespace: default
roleRef:
  kind: ClusterRole
  name: cleanup-role
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cleanup-service-account
  namespace: default

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cleanup-cronjob
spec:
  schedule: "0 4 * * *"
  startingDeadlineSeconds: 36000
  concurrencyPolicy: Replace
  timeZone: "Etc/UTC"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: cleanup-service-account
          restartPolicy: Never
          containers:
            - name: cleanup
              image: repo/cleanup/dev:latest
              imagePullPolicy: IfNotPresent
              env:
                - name: PV_NAME
                  value: attackrepo-pv-0
                - name: NAMESPACE
                  value: default

The cleaner image runs the following script:

#!/usr/bin/env bash
set -e
set -x

echo "cleanup starting"

kubectl delete pvc ${PV_NAME} --namespace ${NAMESPACE} --ignore-not-found=true


echo "cleanup done"