GCP Jobs

 

In this post we will review usage of GCP jobs. GCP jobs can run containers to perform tasks. The nice thing about it is that we can also send parameters to the jobs, hence create an army of workers.

The Details

To use GCP job, we would probably want start with pushing the container image to GCR (GCP container registry). To push, use the following commands:

docker tag my-snapshot-local/my-image/latest gcr.io/my-project/my-repo/my-image/latest
docker push gcr.io/my-project/my-repo/my-image/latest
docker rmi gcr.io/my-project/my-repo/my-image/latest

Now we can run a job that uses this image:

gcloud beta batch jobs submit my-job --location us-central1 --config job.json

where the job.json is the configuration of the job:

{
  "name": "projects/my-project/locations/us-central1/jobs/my-job",
  "taskGroups": [
    {
      "taskCount": "1",
      "parallelism": "1",
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "gcr.io/my-project/my-repo/my-image/latest",
              "entrypoint": "/my-entrypoint.sh",
              "commands": [
                "my-argument"
              ]
            }
          }
        ],
        "computeResource": {
          "cpuMilli": "4000",
          "memoryMib": "8192"
        }
      }
    }
  ],
  "allocationPolicy": {
    "instances": [
      {
        "policy": {
          "provisioningModel": "STANDARD"
        }
      }
    ]
  },
  "logsPolicy": {
    "destination": "CLOUD_LOGGING"
  }
}

Notice that the job configuration includes both the entrypoint and the arguments for the job. This allows us to submit the job with different arguments on each run, and create many containers working for us. For example, if we want to run a stress test for our site, this is a great method to create heavy load from different containers. Not only that, we can also create the load from different locations.

Final Note

We have presented how to use GCP jobs. In the project I've used the GCP jobs, I've automated submitting of a bulk of jobs, and used sed to automatically replace the arguments, for example:

jobs=4

for ((i=0;i<${jobs};i++));
do
  cp job.json job_temp.json
  sed -i "s/___JOB_ID___/${i}/g" job_temp.json
  sed -i "s/___JOB_INDEX___/${i}/g" job_temp.json
  sed -i "s/___JOBS___/${jobs}/g" job_temp.json
  gcloud beta batch jobs submit job${i} --location us-central1 --config job_temp.json
  rm job_temp.json
done

This shows the power of GCP jobs - with a very small amount of effort, we can get great results.