NATS cluster setup in a kubernetes environment

In this post we will review the steps to "nationalize" a NATS cluster installation as part of our own kubernetes solution.

Using a 3rd-party technology as part of a kubernetes solution usually requires that this technology is installed as part of the solution itself, using a single helm chart for the entire solution. This means we need to create our own docker images to wrap the 3rd-party docker images, and integrate the 3rd-party helm chart into our solution helm charts. Let's review the steps to accomplish this for a NATS cluster.

Docker Images

First, start by wrapping up the NATS docker images. The core NATS server uses 2 images. In addition we will wrap the nats-box images which supplies CLI utilities.

The required docker files are:

NATS reloader Dockerfile:

ARG dockerCacheRepo
FROM ${dockerCacheRepo}natsio/nats-server-config-reloader:0.14.0

NATS container Dockerfile:

ARG dockerCacheRepo
FROM ${dockerCacheRepo}nats:2.10.5-alpine

NATS box Dockerfile:

ARG dockerCacheRepo
FROM ${dockerCacheRepo}natsio/nats-box:0.14.1

Notice that we use a build argument: dockerCacheRepo, which should be sent to the docker build command using the argument:

--build-arg dockerCacheRepo=https://my.docker.proxy.com

This enables us to use a docker proxy server and avoid the docker hub download limit.

The docker images should be tagged using our own solution naming, for example:

my-project-repo.com/solution1/nats-container:0.0.1

Helm Chart

To create templates for the NATS deployment, we using the official NATS chart:

helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm template -f ./helm-values.yaml nats nats/nats --output-dir ./target

and the helm-values.yaml file is:

fullnameOverride: solution1-nats
reloader:
  image:
    repository: my-project-repo.com/solution1/nats-reloader/dev
    tag: latest
    pullPolicy: IfNotPresent
container:
  image:
    repository: my-project-repo.com/solution1/nats-container/dev
    tag: latest
    pullPolicy: IfNotPresent
natsBox:
  container:
    image:
      pullPolicy: IfNotPresent
      repository: my-project-repo.com/solution1/nats-box/dev
      tag: latest
config:
  cluster:
    enabled: true
    port: 6222
    replicas: 2
  nats:
    port: 4222
  monitor:
    port: 8222

 

This creates multiple files under the target folder. This files are the templates that should be integrated into our solution, probably as one of the sub charts. An example of one of these files containing the NATS deployment is below.

---
# Source: nats/templates/nats-box/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: nats-box
    app.kubernetes.io/instance: nats
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nats
    app.kubernetes.io/version: 2.10.5
    helm.sh/chart: nats-1.1.5
  name: solution1-nats-box
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: nats-box
      app.kubernetes.io/instance: nats
      app.kubernetes.io/name: nats
  template:
    metadata:
      labels:
        app.kubernetes.io/component: nats-box
        app.kubernetes.io/instance: nats
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: nats
        app.kubernetes.io/version: 2.10.5
        helm.sh/chart: nats-1.1.5
    spec:
      containers:
      - args:
        - trap true INT TERM; sleep infinity & wait
        command:
        - sh
        - -ec
        - |
          work_dir="$(pwd)"
          mkdir -p "$XDG_CONFIG_HOME/nats"
          cd "$XDG_CONFIG_HOME/nats"
          if ! [ -s context ]; then
            ln -s /etc/nats-contexts context
          fi
          if ! [ -f context.txt ]; then
            echo -n "default" > context.txt
          fi
          cd "$work_dir"
          exec sh -ec "$0"
        image: my-project-repo.com/solution1/nats-box/dev:latest
        imagePullPolicy: IfNotPresent
        name: nats-box
        volumeMounts:
        - mountPath: /etc/nats-contexts
          name: contexts
      enableServiceLinks: false
      volumes:
      - name: contexts
        secret:
          secretName: solution1-nats-box-contexts