GO Libraries as Multi Repo project

 

In this post we review the steps to create GO libraries as a multi repo project. Originally we have used a mono repo for our projects. Mono repo is good and easy solution for a product that need to evolve fast, keeps changing in terms of services and architecture. However for a more mature product with a large code base, and specifically for multiple products using a shared code this causes code duplication and redundant work.

These are the steps I've made to externalize the common libraries from a project into a common repo using by multiple products.

Libraries Code Updates

Create a new repo (bitbucket in this case) and move the libraries code to it. The libraries were already setup as GO modules, so this was pretty easy. 

I've added all the libraries under the go-lib folder, for example

~GIT-ROOT/go-lib/library1

The important thing to notice is using GO standard for the package name, for example the package name

com.my-company.project1.library1

is updated to 

bitbucket.org/my-company/common/go-lib/library1

In case of cross-library usage we can use replace in the go.mod. The replace command affects only when building the actual module and not when another project that uses the library is compiled. For example:

replace bitbucket.org/my-company/common/go-lib/library2 => ../library2

Libraries CI/CD

I wanted to automated the entire versioning of the libraries, so I called a script as part of the Jenkinsfile that loops over the list of libraries:

#!/usr/bin/env bash

set -e
cd "$(dirname "$0")"
buildVersion=$1
cloudCredentials=$2
fullVersion=v1.0.${buildVersion}
mapfile -t modules < modules.txt
tags=()
for module in "${modules[@]}"; do
  tag="${module}/${fullVersion}"
  git tag "${tag}"
  tags+=("${tag}")
done
git push https://"${cloudCredentials}"@bitbucket.org/my-company/common "${tags[@]}"

This script creates a tag according to the go standard, for example:

go-lib/library1/v1.0.1234

Consumer Project

The bitbucket repo is a private repo, and the consumer project CI/CD must use credentials to access the libraries. On the other hand local build are using private key to access the bitbucket. This was a bit tricky, but eventually I've updated the docker build to dynamically decide on the method to access the private bitbucket repo.

This is the result dockerfile related section:

#================
# Stage1: compile
#================
ARG dockerCacheRepo
FROM ${dockerCacheRepo}golang:1.26.4 AS go-compiler
ARG BITBUCKET_USE_SSH=true
ARG BB_USER
ARG BB_TOKEN
ENV GOPATH=/go
ENV GOBIN=/go/bin
ENV GOPRIVATE=bitbucket.org/my-company/*

...

# Configure Bitbucket authentication
RUN if [ "$BITBUCKET_USE_SSH" = "true" ]; then \
        echo "Using SSH authentication for Bitbucket" && \
        git config --global url."git@bitbucket.org:".insteadOf "https://bitbucket.org/" && \
        mkdir -p ~/.ssh && \
        ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts; \
    else \
        echo "Using HTTPS authentication for Bitbucket" && \
        git config --global credential.helper store && \
        echo "https://${BB_USER}:${BB_TOKEN}@bitbucket.org" > ~/.git-credentials; \
    fi
# Download private dependencies
RUN --mount=type=ssh \
    go mod download

...

RUN go build -o /output/___PROJECT___

The ENV GOPRIVATE instructs GO to use credentials to access libraries with a specified prefix, and we have a BITBUCKET_USE_SSH flag to decide whether to use credentials or SSH.

Local Development

One last issue is that we still need IDE development to recognize the common repo, and hence each developer need to setup this on the GO configuration:

go env -w GOPRIVATE=bitbucket.org/my-company/*
git config --global url."git@bitbucket.org:".insteadOf https://bitbucket.org/

Final Note

Once all this is done we gain a smaller git repo for each project, and a central update of the common libraries. in addition each project directly controls the each library version in it go.mod, for example:

require bitbucket.org/my-company/common/go-lib/project1 v1.0.19

This gains a great flexability.

I've also added the script to update all dependencies using GO CLI:

go get -u ./...

and alternatively upgrade only the libraries using:

    go get \
      bitbucket.org/my-company/common/go-lib/library1@latest \
      bitbucket.org/my-company/common/go-lib/library2@latest