Managing Dependencies in Cloud Build

In this post we will review a method of managing dependencies in Google Cloud Build (GCB).

For the basics of GCB, check the post Using Google Cloud Build.

While GCB provide a great method of building artifacts and docker images, it does not handle dependencies. This might be added in a later stage, as GCB is still in its beta phase. For now, we need to handle this ourselves.

Let review an example of how dependencies management is achieved. Let's assume that we have two triggers: trigger-1 and trigger-2 that we want to run in parallel. Then we want trigger-3 to start only after the first two triggers were successfully completed.

To make this work, we will create a control plane trigger: trigger-control.

This trigger will lunch trigger-1 and trigger-2, wait for their completion, and the start trigger-3. 

Notice that the trigger control is initialized by a push the the source repository, while all of the other triggers are manually started by the control trigger.

The trigger is using the predefined builder image of the gcloud image.

The control trigger configuration is:

name: trigger-control

triggerTemplate:
  branchName: .*
  projectId: my-project
  repoName: my-source-repo

build:
  timeout: 3600s
  steps:
    - id: main
      name: gcr.io/cloud-builders/gcloud
      entrypoint: bash
      timeout: 3600s
      args:
        - /workspace/trigger_control.sh
        - ${BRANCH_NAME}

And the trigger shell code is:

trigger_control.sh

#!/usr/bin/env bash

branchName=$1
runIds=""

function run_trigger() {
  name=$1
  runId=$(gcloud beta builds triggers run image-${name} --branch=${branchName} | grep "name:" | grep builds | cut -d:  -f2)
  runId=$(echo ${runId} | xargs)

  if [[ "${runIds}" == "" ]]; then
    runIds="${runId}"
  else
    runIds="${runIds},${runId}"
  fi
}

function wait_for_build() {
    buildId=$1
    image=$(gcloud beta builds list --format yaml --filter name=${buildId} | grep "TRIGGER_NAME" | cut -d: -f2)
    image=$(echo ${image} | xargs)
    status="WORKING"
    while [ "${status}" == "WORKING" ]
    do
      status=$(gcloud beta builds list --format yaml --filter name=${buildId} | grep ^status | cut -d: -f2)
      status=$(echo ${status} | xargs)
      if [[ "${status}" == "WORKING" ]]; then
        sleep 5
      fi
    done

  if [[ "${status}" != "SUCCESS" ]]; then
    echo "build for ${image} failed, status is ${status}"
    exit 1
  fi
}

function wait_for_builds(){
  IFS=',' read -r -a array <<< "${runIds}"
  for element in "${array[@]}"
  do
      wait_for_build ${element}
  done
}

run_trigger trigger-1
run_trigger trigger-2
wait_for_builds

gcloud beta builds triggers run trigger-3 --branch=${branchName}

In this example, the launch of trigger-3 is in background, and the control trigger does not wait for it. We can change this to wait for trigger-3 and hence the control trigger would be success only if all triggers are success.

Final Note

We have reviewed a method to manage dependencies and parallel running of triggers using cloud build. This base could be easily used for complex dependencies management in the GCB.

Notice that you can also use the trigger's "waitFor" configuration to manage parallel steps, but the problem is that the parallel steps run on the same server, hence you will need to use a much powerful and expensive server to run your builds in parallel. This method is much faster and much simpler.