In this post we will review the steps to dynamically create and delete a PersistentVolumeClaim for a CronJob.
A CronJob might require a large amount of temporary storage, and we don't want to keep the PersistenceVolumeClaim active while the job is not running, since the cost might be high. For example, assumeing we have a CronJob running once in a week for 3 hours, and required 1TB disk space for calculations. The cost of leaving such a disk active for an entire week is very high, hence we should dynamically allocate and remove the disk.
Kubernetes does not supply out of the box mechanism to handle this, hence we can do it ourselves. We handle this by 3 CronJobs:
1. The allocate CronJob which create the PVC
2. The actual computation CronJob
3. The cleanup CronJob
The Allocate CronJob
This CronJob creates the PVC before the computation job is starting. It should use a schedule that run it just a few minutes before the computation job. The following are the kubernetes entities to run the allocate CronJob. Notice that we must also provide permissions for handling the PVC.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: allocate-role
rules:
- apiGroups: [ "" ]
resources: [ "persistentvolumes" ]
verbs: ["create","list","delete","get","patch","watch"]
- apiGroups: [ "" ]
resources: [ "persistentvolumeclaims" ]
verbs: ["create","list","delete","get","patch","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: allocate-role-binding
subjects:
- kind: ServiceAccount
name: allocate-service-account
namespace: default
roleRef:
kind: ClusterRole
name: allocate-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: allocate-service-account
namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:
name: allocate-config
data:
pvc.yaml: |-
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
labels:
type: local
spec:
storageClassName: "gp2"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1000Gi
---
kind: CronJob
metadata:
name: allocate-cronjob
spec:
schedule: "0 0 * * *"
startingDeadlineSeconds: 36000
concurrencyPolicy: Replace
timeZone: "Etc/UTC"
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
serviceAccountName: allocate-service-account
restartPolicy: Never
containers:
- name: cleanup
image: repo/allocate/dev:latest
imagePullPolicy: IfNotPresent
env:
- name: PVC_NAME
value: my-pvc
- name: NAMESPACE
value: default
volumeMounts:
- name: config
mountPath: /config
volumes:
- name: config
configMap:
name: allocate-config
The allocate image is a simple script that runs kubectl to create the PVC:
#!/usr/bin/env bash
set -e
set -x
echo "prepare starting"
kubectl delete pvc ${PV_NAME} --namespace ${NAMESPACE} --ignore-not-found=true
kubectl apply -f /config/pvc.yaml
echo "prepare done"
The Cleanup CronJob
The Cleanup CronJob runs after the computation job completion and deletes the PVC. This includes the following kubernetes entities:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cleanup-role
rules:
- apiGroups: [ "" ]
resources: [ "persistentvolumeclaims" ]
verbs: ["create","list","delete","get","patch","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cleanup-role-binding
subjects:
- kind: ServiceAccount
name: cleanup-service-account
namespace: default
roleRef:
kind: ClusterRole
name: cleanup-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cleanup-service-account
namespace: default
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: cleanup-cronjob
spec:
schedule: "0 4 * * *"
startingDeadlineSeconds: 36000
concurrencyPolicy: Replace
timeZone: "Etc/UTC"
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
serviceAccountName: cleanup-service-account
restartPolicy: Never
containers:
- name: cleanup
image: repo/cleanup/dev:latest
imagePullPolicy: IfNotPresent
env:
- name: PV_NAME
value: attackrepo-pv-0
- name: NAMESPACE
value: default
The cleaner image runs the following script:
#!/usr/bin/env bash
set -e
set -x
echo "cleanup starting"
kubectl delete pvc ${PV_NAME} --namespace ${NAMESPACE} --ignore-not-found=true
echo "cleanup done"
