CI/CD in a Shell

 

Recently I had to create a CI/CD for a new project whose source repository was in bitbucket. There are standard methods to handle this, using triggers from bitbucket, AWS CodeBuild, and AWS CodePipeline. However, I had only read permissions from the bitbucket and hence was limited in my ability to use the standard tools. I've decided to create CI/CD in a bash and surprisingly I've found it exteremly simple as well as lower cost and faster that the standard tools. I am aware of the downside of using scripts for such processes, such as lake of visibility, redundancy, and standards, but still the result was so good I think startup project should definitely consider it.

Listed below are the shell based CI/CD components.

The Poll Script

The poll script is running on a t3a.nano EC2 whose price is ~3$/month.

It polls the bitbucket repository every 5 minutes, and once a change on the deployment related branch is located, it starts the builder EC2 VM, and runs the build and deploy script.

#!/bin/bash

set -eE

instanceId=""
publicIp=""
intervalSeconds=300

cleanup() {
    if [ -n "${instanceId}" ]; then
        echo "Stopping instance: ${instanceId}"
        if ! aws ec2 stop-instances --instance-ids "${instanceId}"; then
          echo "Warning: Failed to stop instance ${instanceId}. Will retry on next run."
        else
          echo "Instance stopped successfully."
        fi
        instanceId=""
    fi
}

restart_script() {
    echo "Command '$BASH_COMMAND' failed with exit code $?"
    cleanup
    echo "Restarting soon..."
    sleep ${intervalSeconds}
    exec "$0" "$@"
}

trap 'restart_script "$@"' ERR


runBuild(){
  trap cleanup RETURN

  instanceId=$(aws ec2 describe-instances \
      --filters "Name=tag:Name,Values=my-builder-vm" \
      --query "Reservations[*].Instances[*].InstanceId" \
      --output text)

  echo "Starting instance: ${instanceId}"
  aws ec2 start-instances --instance-ids ${instanceId}

  echo "Waiting for instance to be in 'running' state..."
  aws ec2 wait instance-running --instance-ids ${instanceId}

  publicIp=$(aws ec2 describe-instances \
    --instance-ids ${instanceId} \
    --query "Reservations[0].Instances[0].PublicIpAddress" \
    --output text)

  echo "Running build remote"
  ssh -o StrictHostKeyChecking=no ec2-user@${publicIp} /home/ec2-user/build/my-repo/deploy/aws/production/deploy.sh

  cleanup
  echo "Build done"
}

checkOnce(){
  echo "Check run time: $(date)"
  commitFilePath=/home/ec2-user/build/last_commit.txt
  latestCommit=$(git ls-remote git@bitbucket.org:my-project/my-repo.git my-deploy-branch | awk '{print $1}')
  echo "Latest commit: ${latestCommit}"

  lastCommit=$(cat ${commitFilePath} 2>/dev/null || echo "")
  echo "Last deployed: ${lastCommit}"

  if [ "${latestCommit}" != "${lastCommit}" ]; then
      echo "New commit detected, starting build"
      runBuild
      echo "${latestCommit}" > ${commitFilePath}
      echo "last commit updated"
  else
      echo "No new commits"
  fi
}

while true; do
    checkOnce
    sleep ${intervalSeconds}
done

To make this script part of the poller VM instance startup, use the following:

sudo cat <<EOF > /etc/systemd/system/poll.service
[Unit]
Description=Poll Script Startup
After=network.target

[Service]
Type=simple
ExecStart=/home/ec2-user/build/poll.sh
Restart=on-failure
User=ec2-user
WorkingDirectory=/home/ec2-user/build
StandardOutput=append:/home/ec2-user/build/output.txt
StandardError=append:/home/ec2-user/build/output.txt

[Install]
WantedBy=multi-user.target
EOF


sudo systemctl daemon-reload
sudo systemctl enable poll.service   # auto-start on boot
sudo systemctl start poll.service    # start immediately

The Build Script - Step 1

The build script is running on c6i.4xlarge EC2 whose price is ~500$/month, but I don't care since this EC2 instance is running only during the deployment itself, so the price is very low here as well.

The script runs on the repository itself, which I've manually cloned once after the EC2 creation. It only pulls the latest version and runs another "step 2" script to handle the build. The goal is to be able to accept changes into "step 2" script as part of the git pull.

#!/bin/bash
set -e

cd /home/ec2-user/build/my-repo
git checkout my-deploy-branch
git pull

./deploy_step2.sh

The Build Script - Step 2

The "step 2" script does the actual work: 

1. Increments the build number
2. Builds the docker images
3. Login to the ECR
4. Push the images to ECR
5. Push a new tag to the GIT
6. uses `helm upgrade` to upgrade the production deployment.

Notice that the EC2 uses a role that enables it to access the ECR and the EKS without user and password, for example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:CompleteLayerUpload",
        "ecr:UploadLayerPart",
        "ecr:InitiateLayerUpload",
        "ecr:PutImage"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "eks:DescribeCluster"
      ],
      "Resource": "*"
    }
  ]
}

The script is:

#!/bin/bash
set -e

export AWS_ACCOUNT=123456789012
export AWS_REGION=us-east-1
export AWS_DEFAULT_REGION=${AWS_REGION}
export EKS_CLUSTER_NAME=my-eks

rootFolder=/home/ec2-user/build
buildVersionFile=${rootFolder}/build_number.txt

if [[ -f "${buildVersionFile}" ]]; then
    lastBuildNumber=$(cat "${buildVersionFile}")
else
    lastBuildNumber=1000
fi
newBuildNumber=$((lastBuildNumber + 1))
echo "${newBuildNumber}" > ${buildVersionFile}
echo "Build number updated to: ${newBuildNumber}"

./build_my_images.sh

aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com
RemoteTag=deploy-${newBuildNumber} ./push_images_to_ec2.sh

newTag=deploy-${newBuildNumber}
git tag ${newTag}
git push origin ${newTag}

DEPLOY_VERSION=":${newTag}" ./helm_deploy.sh 

echo "I did it again!"

Final Note

This build system is super fast. Why? Because it uses local cache for the docker images. This means we do not require a docker proxy to cache the images, which also makes it cheap.

To sum: don't use this for a big project, but you can use it for startups for sure.