Recently I had to create a CI/CD for a new project whose source repository was in bitbucket. There are standard methods to handle this, using triggers from bitbucket, AWS CodeBuild, and AWS CodePipeline. However, I had only read permissions from the bitbucket and hence was limited in my ability to use the standard tools. I've decided to create CI/CD in a bash and surprisingly I've found it exteremly simple as well as lower cost and faster that the standard tools. I am aware of the downside of using scripts for such processes, such as lake of visibility, redundancy, and standards, but still the result was so good I think startup project should definitely consider it.
Listed below are the shell based CI/CD components.
The Poll Script
#!/bin/bash
set -eE
instanceId=""
publicIp=""
intervalSeconds=300
cleanup() {
if [ -n "${instanceId}" ]; then
echo "Stopping instance: ${instanceId}"
if ! aws ec2 stop-instances --instance-ids "${instanceId}"; then
echo "Warning: Failed to stop instance ${instanceId}. Will retry on next run."
else
echo "Instance stopped successfully."
fi
instanceId=""
fi
}
restart_script() {
echo "Command '$BASH_COMMAND' failed with exit code $?"
cleanup
echo "Restarting soon..."
sleep ${intervalSeconds}
exec "$0" "$@"
}
trap 'restart_script "$@"' ERR
runBuild(){
trap cleanup RETURN
instanceId=$(aws ec2 describe-instances \
--filters "Name=tag:Name,Values=my-builder-vm" \
--query "Reservations[*].Instances[*].InstanceId" \
--output text)
echo "Starting instance: ${instanceId}"
aws ec2 start-instances --instance-ids ${instanceId}
echo "Waiting for instance to be in 'running' state..."
aws ec2 wait instance-running --instance-ids ${instanceId}
publicIp=$(aws ec2 describe-instances \
--instance-ids ${instanceId} \
--query "Reservations[0].Instances[0].PublicIpAddress" \
--output text)
echo "Running build remote"
ssh -o StrictHostKeyChecking=no ec2-user@${publicIp} /home/ec2-user/build/my-repo/deploy/aws/production/deploy.sh
cleanup
echo "Build done"
}
checkOnce(){
echo "Check run time: $(date)"
commitFilePath=/home/ec2-user/build/last_commit.txt
latestCommit=$(git ls-remote git@bitbucket.org:my-project/my-repo.git my-deploy-branch | awk '{print $1}')
echo "Latest commit: ${latestCommit}"
lastCommit=$(cat ${commitFilePath} 2>/dev/null || echo "")
echo "Last deployed: ${lastCommit}"
if [ "${latestCommit}" != "${lastCommit}" ]; then
echo "New commit detected, starting build"
runBuild
echo "${latestCommit}" > ${commitFilePath}
echo "last commit updated"
else
echo "No new commits"
fi
}
while true; do
checkOnce
sleep ${intervalSeconds}
done
To make this script part of the poller VM instance startup, use the following:
sudo cat <<EOF > /etc/systemd/system/poll.service
[Unit]
Description=Poll Script Startup
After=network.target
[Service]
Type=simple
ExecStart=/home/ec2-user/build/poll.sh
Restart=on-failure
User=ec2-user
WorkingDirectory=/home/ec2-user/build
StandardOutput=append:/home/ec2-user/build/output.txt
StandardError=append:/home/ec2-user/build/output.txt
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable poll.service # auto-start on boot
sudo systemctl start poll.service # start immediately
The Build Script - Step 1
The build script is running on c6i.4xlarge EC2 whose price is ~500$/month, but I don't care since this EC2 instance is running only during the deployment itself, so the price is very low here as well.
The script runs on the repository itself, which I've manually cloned once after the EC2 creation. It only pulls the latest version and runs another "step 2" script to handle the build. The goal is to be able to accept changes into "step 2" script as part of the git pull.
#!/bin/bash
set -e
cd /home/ec2-user/build/my-repo
git checkout my-deploy-branch
git pull
./deploy_step2.sh
The Build Script - Step 2
The "step 2" script does the actual work:
- Increments the build number
- Builds the docker images
- Login to the ECR
- Push the images to ECR
- Push a new tag to the GIT
- uses `helm upgrade` to upgrade the production deployment.
Notice that the EC2 uses a role that enables it to access the ECR and the EKS without user and password, for example:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:UploadLayerPart",
"ecr:InitiateLayerUpload",
"ecr:PutImage"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"eks:DescribeCluster"
],
"Resource": "*"
}
]
}
The script is:
#!/bin/bash
set -e
export AWS_ACCOUNT=123456789012
export AWS_REGION=us-east-1
export AWS_DEFAULT_REGION=${AWS_REGION}
export EKS_CLUSTER_NAME=my-eks
rootFolder=/home/ec2-user/build
buildVersionFile=${rootFolder}/build_number.txt
if [[ -f "${buildVersionFile}" ]]; then
lastBuildNumber=$(cat "${buildVersionFile}")
else
lastBuildNumber=1000
fi
newBuildNumber=$((lastBuildNumber + 1))
echo "${newBuildNumber}" > ${buildVersionFile}
echo "Build number updated to: ${newBuildNumber}"
./build_my_images.sh
aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com
RemoteTag=deploy-${newBuildNumber} ./push_images_to_ec2.sh
newTag=deploy-${newBuildNumber}
git tag ${newTag}
git push origin ${newTag}
DEPLOY_VERSION=":${newTag}" ./helm_deploy.sh
echo "I did it again!"
