Deploy redis cluster on kubernetes

Notice: See an update for this issue in the next article: redis survivability in kubernetes

Redis is an in-memory data store that can be used as cache and message broker.
It is very useful in a kubernetes environment where multiple replicas of one deployment might need to save/pass information that will later be used by replicas of another deployment.

To deploy redis on kubernetes, I've used a simple working implementation.
This article presents the steps I've made:

• Create a docker image to wrap the redis, and add some files
• Create a kubernetes statefulset
• Create a kubernetes service

The Docker Image 

A docker image is required to wrap the original redis docker images with some extensions.

Notice that you can do this even without creating your own image, but instead by updating the changes using the kubernetes deployment. However, I've found it more clear to create a dedicated image.

Dockerfile:

FROM redis:5.0.6
COPY files /
ENTRYPOINT /entrypoint.sh

The 'files' folder contains two files:

redis.conf:

• configuring the instance as part of cluster enabled 
• specifying path of the redis configuration files

port 6379
cluster-enabled yes
cluster-require-full-coverage yes
cluster-node-timeout 5000
appendonly yes
cluster-config-file /data/HOSTNAME/nodes.conf

entrypoint.sh:

• The entrypoint script replaces the IP in the nodes.conf. This is required to allow redis to identify itself in case of a pod restart. The script itself was originated from this issue.
• Once the IP address is handled, start the redis server

#!/usr/bin/env bash
set -e
HOST_FOLDER=/data/${HOSTNAME}
CLUSTER_CONFIG="${HOST_FOLDER}/nodes.conf"
mkdir -p ${HOST_FOLDER}

if [[ -f ${CLUSTER_CONFIG} ]]; then
  if [ -z "${POD_IP}" ]; then
    echo "Unable to determine Pod IP address!"
    exit 1
  fi
  echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
  sed -i.bak -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${CLUSTER_CONFIG}
fi

sed -i "s/HOSTNAME/${HOSTNAME}/g" /redis.conf
exec /usr/local/bin/redis-server /redis.conf

The Kubernetes StatefulSet

A kubernetes StatefulSet (unlike deployment) is required to ensure that the pod host name remains the same between restarts. This allows the entrypoint.sh script (mentioned above) to replace the IP based on the (Pod) host name.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-statefulset
spec:
  serviceName: redis-service
  podManagementPolicy: "Parallel"
  replicas: 6
  selector:
    matchLabels:
      configid: redis-container
  template:
    metadata:
      labels:
        configid: redis-container        
    spec:
      containers:
      - name: redis
        image: my-registry/my-redis:latest
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: redis-data
          mountPath: /data
          readOnly: false
        livenessProbe:
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
          initialDelaySeconds: 5
          periodSeconds: 10
          exec:
            command:
              - /usr/local/bin/redis-cli
              - ping
        readinessProbe:
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 1
          initialDelaySeconds: 5
          periodSeconds: 10
          exec:
            command:
              - /usr/local/bin/redis-cli
              - ping
      volumes:
      - name: redis-data
        hostPath:
          path: /opt/redis

In this case 6 replicas are used, so we will have 3 masters and 3 slaves.
We also include liveness and readiness probes. These only check that the redis instance is alive, but do not check the cluster health (out of scope for this article).
The image specified as my-registry/my-redis should point to the docker image, which was created at the previous step.

The Kubernetes Service

The service exposes 2 ports:

• 6379 is used by clients connecting to the redis cluster
• 16379 is used by the redis instances for the cluster management

apiVersion: v1
kind: Service
metadata:
  name: redis-service
  labels:    
spec:
  selector:
    configid: redis-container
  type: NodePort
  ports:
      - port: 6379
        targetPort: 6379
        name: clients
        protocol: TCP
        nodePort: 30002
      - port: 16379
        targetPort: 16379
        name: gossip
        protocol: TCP

Redis Cluster Initialization

The last step is to tell redis to create cluster.
This can be done, for example, as part of a helm post install script.
I've used a javascript to wait for all of the redis instances to start, and then configure the cluster using kubectl exec on the first redis pod.

const {exec} = require('child_process')

const redisPods = 6


async function kubectl(args) {
  return await new Promise((resolve, reject) => {
    const commandLine = `kubectl ${args}`
    exec(commandLine, (err, stdout, stderr) => {
      if (err) {
        reject(err)
        return
      }
      resolve(stdout)
    })
  })
}

async function getRedisPods() {
  const args = `get pods -l configid=redis-container -o jsonpath='{range.items[*]}{.status.podIP} '`
  const stdout = await kubectl(args)
  return stdout.trim().split(' ')
}

async function executeClusterCreate(pods) {
  let redisNodes = ''
  pods.forEach(p => {
    redisNodes += ` ${p}:6379`
  })

  const command = 'exec redis-statefulset-0 ' +
    '-- redis-cli --cluster create --cluster-replicas 1 --cluster-yes ' +
    redisNodes

  const createResult = await kubectl(command)
  if (createResult.includes('[OK] All 16384 slots covered')) {
    return true
  }
  return false
}

async function configureRedisCluster() {
  while (true) {
    const pods = await getRedisPods()

    if (pods.length !== redisPods) {
      continue
    }

    if (!await executeClusterCreate(pods)) {
      console.warn(`create cluster failed, retrying in a moment`)
      await timeUtil.sleep(1000)
      continue
    }

    return
  }
}

configureRedisCluster()

If you run this script as part of a helm post install hook job, you will need to add permissions for the job to get and list pods, and to execute on pods.

Summary

After all these steps, we have a running redis cluster, BUT the redis cluster is not transparent to it users. A user addressing one redis instance might get a MOVED redirection (see Moved Redirection at redis cluster specification documentation), which means that there is another instance which handles the related information, and it is up to the client to address the relevant instance.

To make this actually transparent, use a redis client library that handles theses issues.
I have used the ioredis javascript library, which handles the MOVED redirection.