Parallel Images Build Using Bash Job Pool

 

In this post we will review how to use parallel bash processes to increase build images process.

TL;DR

Use shellutils to run parallel tasks as part of the build main script.

The Local Build Process

In many projects we have ability to locally build all the project's docker images on the local development machine. This can be used both to validate that we did not break the build when changing a major common functionality, and for locally running kubernetes cluster with the fresh docker images.

The build usually started by a central build script which then sequentially launches each of the images build script. Hence if we have 20 images, each image build script running for one minute, we have to wait for 20 minutes. One way to handle this is to run the central build and then go fo a coffee break :)

An alternative is to run the images build in parallel, using a process pool. We can use the shellutils helper functions to wrap the images scripts parallelism.

Parallel Central Build

An example for such central build is the following:

#!/usr/bin/env bash

rm -rf ./build_tmp
mkdir ./build_tmp

. ./job_pool.sh

echo "Building docker images in parallel"
job_pool_init 10 1
while read name ; do
  job_pool_run ./images/${name}/build.sh ./build_tmp/${name}.out
done < ./images_to_build
job_pool_shutdown

if [[ "${job_pool_nerrors}" != "0" ]]; then
  echo ""
  echo "****************************************"
  echo "            Failure summary             "
  echo "****************************************"

  cat ./build_tmp/*

  echo "****************************************"
  echo "   Total num of errors is: ${job_pool_nerrors}   "
  echo "****************************************"
  exit 1
fi
rm -rf ./build_tmp

To use this, first download the job_pool.sh from the shellutils, and save it in the same folder as the central build script. The central build script loads the job_pool.sh, and creates a pool of 10 workers.

Then for each image listed in the images_to_build file runs the image build script in images/IMAGE_NAME/build.sh. Finally the script reports success or failures in case of a problem.

Final Note

We have demonstrated usage of parallel shell scripts run for a faster local build. This is a real improvement that instead of waiting 20 minutes for the build, we ended up waiting only 2-3 minutes.

This bash job pool can be used for any other tasks that we manage by bash scripts, such as pushing docker images to images repository, generating load, and more.