Deploy a PostgreSQL Cluster on Kubernetes

 

In this post we will review the steps to deploy a PostgreSQL cluster on a kubernetes cluster. 

PostgreSQL is a veteran DBMS existing for ~30 years(!!). As such it is less a kubernetes player and does not automatically adjust the cluster like other technologies such as NATS and ClickHouse. Hence to maintain the cluster we require first to deploy an operator. There are several operators available, and in this post we use the CloudNativePG.

Cloud Native PG Operator

To deploy the CloudNativePG we can use the simple command:

kubectl apply -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.29/releases/cnpg-1.29.0.yaml

However, in case of need to include this as part of an umbrella helm chart, we should download the file and split it to multiple templates. The CRDs should be included in a different folder, for example:

my-umbrella-helm/charts/cnpg/crds

The CRDs folder is a special helm folder enabling deployment of the CRDs before any additional templates are rendered.

Other template can be deployed in the standard templates folder, for example:

my-umbrella-helm/charts/cnpg/templates

We would usually update the templates in this folder to match our own helm deployment standards such as entities names, images location, labels, enable/disable flags. Notice that CloudNativePG requires the label:

app.kubernetes.io/name: cloudnative-pg

So we must include this label when customizing the labels.

The PostgreSQL cluster

Once the operator is up and running, all we need to do is to render a cluster CRD, for example:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgresql
spec:
  instances: 3

  imageName: my-repo/my-postgresql/dev:18.3
  imagePullPolicy: Always
  storage:
    size: 10Gi

  postgresql:
    parameters:
      max_connections: "200"

  bootstrap:
    initdb:
      database: my-db
      owner: admin
      secret:
        name: postgresql-secret

We usually update the image name to be downloaded from our own repo. Notice the CloudNativePG blocks usage of "latest" version for the PostgreSQL cluster, so even on the local kind deployment we must use a specific version.

In the cluster resource we specify several important settings: the default database name, the default owner role of the database, and a secret with the password for that user.

Final Note

We've reviewed steps to setup a PostgreSQL cluster using the CloudNativePG operator. While not as easy to setup as other DBMS, PostgreSQL is considered one of the best relational DBMS with a good support for updates.

MCP Client in GO and Stub

 

In the following post we present creation of MCP Client in GO including API, implementation, and stub. This method of providing APIs is critical for a successful testing which should not access any external entity. Note that in this case, we also provide a factory API, implementation, and stub further enabling the code to create MCP client even when it is running in the tests scope.

MCP Client

The client API:

type McpToolInfo struct {
  ToolName        string
  ToolDescription string
  InputSchema     string
  OutputSchema    string
}

type McpClientApi interface {
  ListTools() []*McpToolInfo
  CloseSession()
}

The implementation:

// static - to use connection pool for all servers
var staticWebClient = web.ProduceClientImpl(core.Config.McpConnectionTimeout)

type McpClientImpl struct {
  session *mcp.ClientSession
  context context.Context
}

func ProduceMcpClientImpl(
  mcpServerUrl string,
) *McpClientImpl {
  m := &McpClientImpl{
   context: context.Background(),
  }

  client := mcp.NewClient(
   &mcp.Implementation{
    Name:    "my-go-client",
    Version: "1.0.0",
   },
   nil,
  )

  transport := &mcp.StreamableClientTransport{
   Endpoint:   mcpServerUrl,
   HTTPClient: staticWebClient.UsedClient(),
  }

  session, err := client.Connect(m.context, transport, nil)
  kiterr.RaiseIfError(err)

  m.session = session
  return m
}

func (m *McpClientImpl) ListTools() []*McpToolInfo {
  parameters := mcp.ListToolsParams{}
  tools, err := m.session.ListTools(m.context, &parameters)
  kiterr.RaiseIfError(err)

  var result []*McpToolInfo
  for _, tool := range tools.Tools {
   info := McpToolInfo{
    ToolName:        tool.Name,
    ToolDescription: tool.Description,
    InputSchema:     fmt.Sprintf("%v", tool.InputSchema),
    OutputSchema:    fmt.Sprintf("%v", tool.OutputSchema),
   }
   result = append(result, &info)
  }

  return result
}

func (m *McpClientImpl) CloseSession() {
  err := m.session.Close()
  m.session = nil
  kiterr.RaiseIfError(err)
}

And the stub:

type McpClientStub struct {
}

func ProduceMcpClientStub(
  mcpServerUrl string,
) *McpClientStub {
  return &McpClientStub{}
}

func (m *McpClientStub) ListTools() []*McpToolInfo {
  return []*McpToolInfo{
   {
    ToolName:        "tool-1",
    ToolDescription: "the best tool",
    InputSchema:     "my-schema-input",
    OutputSchema:    "my-schema-output",
   },
  }
}

func (m *McpClientStub) CloseSession() {
}

MCP Client Factory

The factory API:

import "radware.com/mcpp/commons/mcp/client"

type McpClientFactoryApi interface {
  ProduceMcpClient(
   mcpServerUrl string,
  ) client.McpClientApi
}

The factory implementation:

type McpClientFactoryImpl struct {
}

func ProduceMcpClientFactoryImpl() *McpClientFactoryImpl {
  return &McpClientFactoryImpl{}
}

func (f *McpClientFactoryImpl) ProduceMcpClient(
  mcpServerUrl string,
) client.McpClientApi {
  return client.ProduceMcpClientImpl(mcpServerUrl)
}

And the stub:

type McpClientFactoryStub struct {
}

func ProduceMcpClientFactoryStub() *McpClientFactoryStub {
  return &McpClientFactoryStub{}
}

func (f *McpClientFactoryStub) ProduceMcpClient(
  mcpServerUrl string,
) client.McpClientApi {
  return client.ProduceMcpClientStub(mcpServerUrl)
}

ClickHouse Cluster Agnostic DDL

 

In this post we present an method to run DDL for table creation and table drop regardless the fact whether we use a clickhouse cluster or a standalone clickhouse.

Clickhouse standalone and clickhouse cluster differ on the statement to create and drop a table.

Below are examples

Table Creation for Standalone

CREATE TABLE IF NOT EXISTS events
(
    id UInt64,
    user_id UInt32,
    event_type String,
    created_at DateTime
)
ENGINE = MergeTree() 
ORDER BY (id)

Table Creation for Cluster

Here we need to create both the local table that we exist on each of the cluster nodes, and the distributed table that enables fetching from any cluster node.

CREATE TABLE IF NOT EXISTS events_LOCAL on CLUSTER my_cluster
(
    id UInt64,
    user_id UInt32,
    event_type String,
    created_at DateTime
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/{table}', '{replica}')
ORDER BY (id)

;

CREATE TABLE IF NOT EXISTS events on CLUSTER my_cluster
(
    id UInt64,
    user_id UInt32,
    event_type String,
    created_at DateTime
)
ENGINE = Distributed(
    my_cluster,
    default,
    events_LOCAL,
    1
)

Table Drop for Standalone

DROP TABLE IF EXISTS events

Table Drop for Cluster

Notice that not only we need to drop the local and distributed tables, but we also need to remove the replica from clickhouse keeper.

DROP TABLE IF EXISTS events on cluster my_cluster
;
DROP TABLE IF EXISTS events_LOCAL on cluster my_cluster
;
SYSTEM DROP REPLICA 'clickhouseserver-statefulset-0'
FROM ZKPATH '/clickhouse/tables/01/events_LOCAL'
;
SYSTEM DROP REPLICA 'clickhouseserver-statefulset-1'
FROM ZKPATH '/clickhouse/tables/01/events_LOCAL'

Abstract DDL Wrapper

To simplify all these actions, we create a wrapper code for the DDL.

func GetLocalTableName(
  tableName string,
) string {
  if clickhouse.Config.ClickhouseConnectionToCluster {
   return tableName + "_LOCAL"
  }
  return tableName
}

func convertToClusterEngine(
  engineAndMore string,
) string {
  re := regexp.MustCompile(`(\w*MergeTree)\s*\(([^)]*)\)`)

  located := re.FindStringSubmatch(engineAndMore)
  if located == nil {
   kiterr.RaiseError("unable to locate engine in %v", engineAndMore)
  }

  engine := located[1]
  parameters := located[2]

  if parameters != "" {
   parameters = ", " + parameters
  }

  replacement := fmt.Sprintf(
   "Replicated%s('/clickhouse/tables/{shard}/{table}', '{replica}'%s)",
   engine,
   parameters,
  )

  return re.ReplaceAllString(engineAndMore, replacement)
}

func CreateTable(
  clickHouse clickhouse.ClickhouseApi,
  tableName string,
  columns string,
  engineAndMore string,
) {
  engineAndMore = strings.TrimSpace(engineAndMore)

  if clickhouse.Config.ClickhouseConnectionToCluster {
   local := fmt.Sprintf(
    `
CREATE TABLE IF NOT EXISTS %v on CLUSTER %v
%v
ENGINE = %v
`,
    GetLocalTableName(tableName),
    clickhouse.Config.ClickhouseClusterName,
    columns,
    convertToClusterEngine(engineAndMore),
   )
   clickHouse.ExecuteStatement(local)

   distributed := fmt.Sprintf(
    `
CREATE TABLE IF NOT EXISTS %v on CLUSTER %v
%v
ENGINE = Distributed(
    %v,
    default,
    %v,
    1
)
`,
    tableName,
    clickhouse.Config.ClickhouseClusterName,
    columns,
    clickhouse.Config.ClickhouseClusterName,
    GetLocalTableName(tableName),
   )
   clickHouse.ExecuteStatement(distributed)

  } else {
   statement := fmt.Sprintf(
    `
CREATE TABLE IF NOT EXISTS %v
%v
ENGINE = %v
`,
    tableName,
    columns,
    engineAndMore,
   )
   clickHouse.ExecuteStatement(statement)
  }
}

func DropTable(
  clickHouse clickhouse.ClickhouseApi,
  tableName string,
) {
  if clickhouse.Config.ClickhouseConnectionToCluster {
   distributed := fmt.Sprintf(
    `DROP TABLE IF EXISTS %v on cluster %v`,
    tableName,
    clickhouse.Config.ClickhouseClusterName,
   )
   clickHouse.ExecuteStatementOnAllNodes(distributed)

   local := fmt.Sprintf(
    `DROP TABLE IF EXISTS %v on cluster %v`,
    GetLocalTableName(tableName),
    clickhouse.Config.ClickhouseClusterName,
   )
   clickHouse.ExecuteStatementOnAllNodes(local)

   dropReplica(clickHouse, tableName, 0)
   dropReplica(clickHouse, tableName, 1)

  } else {
   clickHouse.ExecuteStatement(`DROP TABLE IF EXISTS ` + tableName)
  }
}

func dropReplica(
  clickHouse clickhouse.ClickhouseApi,
  tableName string,
  nodeIndex int,
) {
  err := kiterr.RunSafe(func() {
   keeper := fmt.Sprintf(
    `SYSTEM DROP REPLICA 'democlickhouseserver-statefulset-%v' FROM ZKPATH '/clickhouse/tables/01/%v'`,
    nodeIndex,
    GetLocalTableName(tableName),
   )
   clickHouse.ExecuteStatementOnAllNodes(keeper)
  })
  //ignoring error
  kiterr.LogWarnIfError(err)
}

and an example for calling this code is:

  tableName := "events"

  columns := `
(
    id UInt64,
    user_id UInt32,
    event_type String,
    created_at DateTime
)
`

  engine := "MergeTree() ORDER BY (id)"

  CreateTable(
   clickhouseApi,
   tableName,
   columns,
   engine,
  )

  DropTable(
   clickhouseApi,
   tableName,
  )

Final Note

We've shown the DDL differences for create and drop tables between clickhouse standalone and clickhouse cluster.

We've also shown how to create a code that is agnostic to these difference.

Edit

Another non DDL issue is about delete rows which also looks different. This can be handle in an agnostic method similar to the functions we've presented above.

func DeleteFromTable(
  clickHouse clickhouse.ClickhouseApi,
  tableName string,
  whereClause string,
  args ...any,
) {
  if clickhouse.Config.ClickhouseConnectionToCluster {
   local := fmt.Sprintf(`
ALTER TABLE %v
ON CLUSTER %v
DELETE %v
`,
    GetLocalTableName(tableName),
    clickhouse.Config.ClickhouseClusterName,
    whereClause,
   )
   clickHouse.ExecuteStatement(local, args...)
  } else {
   local := GetDeleteStatementStandalone(tableName, whereClause)
   clickHouse.ExecuteStatement(local, args...)
  }
}

func GetDeleteStatementStandalone(
  tableName string,
  whereClause string,
) string {
  return fmt.Sprintf(`
ALTER TABLE %v
DELETE %v
`,
   tableName,
   whereClause,
  )
}

And an example for call is:

tableName := "events"

whereClause := "WHERE user_id = ?"

DeleteFromTable(
  clickhouseApi,
  tableName,
  whereClause,
  12345, 
)

Deploy ClickHouse Cluster on Kubernetes without Operator

 

In a recent post we've seen how to deploy ClickHouse standalone on kubernetes without operator. In this post we will further show deployment on kubernetes without operator, but this time we will use clickhouse cluster.

To install a cluster, we need to deploy 2 deployments: clickhouse keeper that is used to maintain a quorum, and the clickhouse server which includes the actual cluster nodes. We will use 3 replicas for the keeper, and 2 replicas for the server.

ClickHouse Keeper

The keeper includes the entities below.

The service

apiVersion: v1
kind: Service
metadata:
  name: clickhousekeeper-service
spec:
  selector:
    configid: clickhousekeeper-container
  clusterIP: None
  ports:
    - name: client
      port: 9181
    - name: raft
      port: 9444

The config

apiVersion: v1
kind: ConfigMap
metadata:
  name: clickhousekeeper-config
data:
  keeper_config.xml: |-
    <clickhouse>
    
        <logger>
            <level>information</level>
            <console>true</console>
        </logger>
    
        <listen_host>0.0.0.0</listen_host>
    
        <keeper_server>
            <tcp_port>9181</tcp_port>
    
            <server_id>REPLACE_ME_SERVER_ID</server_id>
    
            <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
            <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
    
            <raft_configuration>
                <server>
                    <id>1</id>
                    <hostname>clickhousekeeper-statefulset-0.clickhousekeeper-service.default.svc.cluster.local</hostname>
                    <port>9444</port>
                </server>
                <server>
                    <id>2</id>
                    <hostname>clickhousekeeper-statefulset-1.clickhousekeeper-service.default.svc.cluster.local</hostname>
                    <port>9444</port>
                </server>
            </raft_configuration>
        </keeper_server>
    </clickhouse>

The statefuleset

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: clickhousekeeper-statefulset
spec:
  serviceName: clickhousekeeper-service
  podManagementPolicy: Parallel
  replicas: 3
  selector:
    matchLabels:
      configid: clickhousekeeper-container
  template:
    metadata:
      labels:
        configid: clickhousekeeper-container        
    spec:
      terminationGracePeriodSeconds: 10
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchExpressions:
                    - key: configid
                      operator: In
                      values:
                        - clickhousekeeper-container

      initContainers:
        - name: config
          image: clickhouse/clickhouse-keeper:latest
          imagePullPolicy: Always
          command:
            - bash
            - -c
            - |
              set -e
              POD_ORDINAL=$(echo $HOSTNAME | awk -F'-' '{print $NF}')
              cp /config-origin/keeper_config.xml /etc/clickhouse-keeper/keeper_config.xml
              sed -i "s/REPLACE_ME_SERVER_ID/$((POD_ORDINAL+1))/" /etc/clickhouse-keeper/keeper_config.xml
              cat /etc/clickhouse-keeper/keeper_config.xml
          volumeMounts:
            - name: clickhousekeeper-config
              mountPath: /config-origin
            - name: clickhousekeeper-runtime-config
              mountPath: /etc/clickhouse-keeper
      containers:
        - name: clickhousekeeper
          image: clickhouse/clickhouse-keeper:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 9181
              name: client
            - containerPort: 9444
              name: raft
          volumeMounts:
            - name: pvc
              mountPath: /var/lib/clickhouse
            - name: clickhousekeeper-runtime-config
              mountPath: /etc/clickhouse-keeper

      volumes:
        - name: clickhousekeeper-config
          configMap:
            name: clickhousekeeper-config
        - name: clickhousekeeper-runtime-config
          emptyDir: {}

  volumeClaimTemplates:
    - metadata:
        name: pvc
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "gp2"
        resources:
          requests:
            storage: 10Gi

Notice the config map is saved to a side folder, and updated with the related ID by the pod name.

ClickHouse Server

The server includes the entities below.

The service

apiVersion: v1
kind: Service
metadata:
  name: clickhouseserver-service
spec:
  selector:
    configid: clickhouseserver-container
  clusterIP: None
  ports:
    - name: tcp
      port: 9000
    - name: http
      port: 8123

The config

apiVersion: v1
kind: ConfigMap
metadata:
  name: clickhouseserver-config
data:
  custom.xml: |-
    <clickhouse>
        <logger>
            <level>information</level>
            <console>true</console>
        </logger>
    
        <listen_host>0.0.0.0</listen_host>
    
        <keeper_server>
            <tcp_port>9181</tcp_port>
        </keeper_server>
    
        <zookeeper>
            <node>
                <host>clickhousekeeper-statefulset-0.clickhousekeeper-service.default.svc.cluster.local</host>
                <port>9181</port>
            </node>
            <node>
                <host>clickhousekeeper-statefulset-1.clickhousekeeper-service.default.svc.cluster.local</host>
                <port>9181</port>
            </node>
            <node>
                <host>clickhousekeeper-statefulset-2.clickhousekeeper-service.default.svc.cluster.local</host>
                <port>9181</port>
            </node>
        </zookeeper>
    
        <remote_servers>
            <llmp_clickhouse_cluster>
                <shard>
                    <replica>
                        <host>clickhouseserver-statefulset-0.clickhouseserver-service.default.svc.cluster.local</host>
                        <port>9000</port>
                    </replica>
                    <replica>
                        <host>clickhouseserver-statefulset-1.clickhouseserver-service.default.svc.cluster.local</host>
                        <port>9000</port>
                    </replica>
                </shard>
            </llmp_clickhouse_cluster>
        </remote_servers>
    </clickhouse>
  users.xml: |-
    <clickhouse>
        <users>
            <default>
                <password></password>
                <networks>
                    <ip>::/0</ip>
                </networks>
            </default>
        </users>
    </clickhouse>
  macros.xml: |-
    <clickhouse>
        <macros>
        </macros>
    </clickhouse>

The statefulset

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: clickhouseserver-statefulset
spec:
  serviceName: clickhouseserver-service
  podManagementPolicy: Parallel
  replicas: 2
  selector:
    matchLabels:
      configid: clickhouseserver-container
  template:
    spec:
      terminationGracePeriodSeconds: 10
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchExpressions:
                    - key: configid
                      operator: In
                      values:
                        - clickhouseserver-container

      initContainers:
        - name: config
          image: lclickhouse/clickhouse-server:latest
          imagePullPolicy: Always
          command:
            - bash
            - -c
            - |
              set -e
              cp /config-origin/custom.xml /etc/clickhouse-server/config.d/custom.xml
              cp /config-origin/macros.xml /etc/clickhouse-server/config.d/macros.xml
              sed -i "s/REPLACE_ME_HOST_NAME/${HOSTNAME}/" /etc/clickhouse-server/config.d/macros.xml
          volumeMounts:
            - name: clickhouseserver-config
              mountPath: /config-origin
            - name: clickhouseserver-runtime-config
              mountPath: /etc/clickhouse-server/config.d
      containers:
        - name: clickhouseserver
          image: clickhouse/clickhouse-server:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 9181
              name: client
            - containerPort: 9444
              name: raft
          volumeMounts:
            - name: pvc
              mountPath: /var/lib/clickhouse
            - name: clickhouseserver-runtime-config
              mountPath: /etc/clickhouse-server/config.d
            - name: clickhouseserver-config
              mountPath: /etc/clickhouse-server/users.d/users.xml
              subPath: users.xml

      volumes:
        - name: clickhouseserver-config
          configMap:
            name: clickhouseserver-config
        - name: clickhouseserver-runtime-config
          emptyDir: {}

  volumeClaimTemplates:
    - metadata:
        name: pvc
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "gp2"
        resources:
          requests:
            storage: 10Gi

Here again we update the configuration using an init container.

Final Note

Using clickhouse operator is much simpler than the manual mode, but in some cases specific adaptations are required that block us from using the operator. In this post we've displayed the full requirements for a minimal manual deployment of a clickhouse cluster.

Analyzing JSON and Text using Bash

In this post we will review simple methods to quickly analyze JSON and text data using bash CLIs and pipelines. While these methods might lack some of the abilities that we can find in more complex GO/JavaScript/python code, they have a great advantage of quickly grepping a piece of information we're looking for.

For this post we will use an input file containing list of HTTP requests, for example:

JSON Parsing

This first thing we want to do it to use the `jq` command to get this as a parsed JSON.

cat data.json | jq

Next, lets get only the source IPs from the file:

cat data.json | jq  '.[].SourceIp'

Count and Sort

Let's get unique count of requests per IP sorted by count.

cat data.json | jq '.[].SourceIp' | sort | uniq -c | sort -n

Text Parsing

Let's get the methods usage in the HTTP request.

cat data.json | jq '.[].HttpRequest' 

Now split the text by the first space and get only the first part, then remove the first character (the quotes). Then use the previous method to count per method.

cat data.json | jq '.[].HttpRequest' | awk -F' ' '{print $1}' | cut -c2-1000 | sort | uniq -c | sort -n

We can do the same to get only the HTTP path.

cat data.json | jq '.[].HttpRequest' | awk -F' ' '{print $2}' | sort | uniq -c | sort -n

Grep Area

What if we want to get transactions only for a specific IP?

We use the grep -A, -B, -C flags.

-A = get also one line after the located text

-B = get also one line before the located text

-C = get also one line before and after the located text

cat data.json | jq  | grep -B1 46.117.105.66

Now we analyze the transactions for this IP like we've done before:

cat data.json | jq  | grep -B1 46.117.105.66 | grep HttpRequest | awk -F ' ' '{print $3}' | sort | uniq -c | sort -n

Final Note

In this post we're had a taste of the power of bash piping.

I highly recommend experimenting with these CLIs since you can do in a few seconds things that would otherwise take you much longer.

Add Grafana Alert for Restarting Pod

 

In this post we will configure an alert in grafana for restarting pods.

This relates to grafana version 11.5.1. Other version might have a different syntax.

Follow the next steps:

• Open grafana GUI
  
  
• Click on Alerting, Alert rules
  
  
• Click on New alert rule
  
  
• Enter rule name: restarting pods
  
  
• Select prometheus as the data source
  
  
• Make sure you have kube-state-metrics installed on the kubernetes cluster. In case it is not, install using:
  
  

  helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics

  helm repo update

  helm install kube-state-metrics kube-state-metrics/kube-state-metrics \

    --namespace kube-system \

    --create-namespace

  
  
• Use the following PromQL query: sum by (namespace, pod) ( increase(kube_pod_container_status_restarts_total[5m]) ) > 1
• Leave the threshold as: A > 0
• Under Configure no data and error handling, set the Alert state if no data or all values are null to Normal
• In the email notification message, use the following summary:
  
  Pod {{ $labels.pod }} restarted
  
  
• In the email description use the following text:
  
  

  Pod {{ $labels.pod }} in namespace {{ $labels.namespace }}

  restarted more than once in the last 5 minutes.

  Current value: {{ $values.A.Value }}
  
  
• Make sure to configure email provider such as SendGrid to enable sending emails from grafana

Avoiding Huge Docker Image For LLM Model

 

In this post we will review a method to reduce docker image size for an LLM inference image.

In general using LLM in a docker container tend to create images whose size might be over 20G.

Downloaded docker images from ECR or from NEXUS might be very long. In addition, saving images in these technologies might present a challenge. 

To avoid this we use the following:

1. We extract the high disk space usage folders out from the docker image, and save it into S3. Then we create a new slim image without these folders.

2. Upon deployment, we run an init container to download the file from S3, and extract it to an emptyDir volume.

Step 1: Extract folders and create slim image

The following script runs the original huge image build.

Then it extracts the huge folders ExportFolder1, ExportFolder2 into the OutputFile. This file should be loaded in AWS S3.

Next the script creates a new slim image without the folders ExportFolder1, ExportFolder2.

#!/usr/bin/env bash
set -e

cd "$(dirname "$0")"

DockerRegistry=my-registry.my-company
ProjectVersion=latest
BuildNumber=1234

OutputFile="${OutputFile:-llm-image-${BuildNumber}.tar.gz}"
SlimImageName="${DockerRegistry}/llm-image-slim${ProjectVersion}"

ExportFolder1="root/.cache"
ExportFolder2="usr/local/lib/python3.12"

TempFolder="$(mktemp -d)"
ImageFileSystem="${TempFolder}/rootfs"
TempContainer=llm-image-tmp

date
echo "build full image"
./build.sh


date
echo "extracting container to ${ImageFileSystem}"
mkdir -p "${ImageFileSystem}"
docker rm -f ${TempContainer}
docker create --name ${TempContainer} llm-image/dev:latest
docker export ${TempContainer} | tar -x -C "${ImageFileSystem}"
docker rm -f ${TempContainer}

date
echo "compressing folders to ${OutputFile}"
tar -czf "${OutputFile}" -C "${ImageFileSystem}"  "${ExportFolder1}"  "${ExportFolder2}"

date
echo "creating slim image ${SlimImageName}"
rm -rf "${ImageFileSystem}/${ExportFolder1}" "${ImageFileSystem}/${ExportFolder2}"
tar -C "${ImageFileSystem}" -c . | docker import - "${SlimImageName}"
docker push ${SlimImageName}

date
echo "cleaning up"
rm -rf "${TempFolder}"

date
echo "Done"

Step 2: Deployment

To handle the S3 download we create an extractor image.

Dockerfile:

FROM amazon/aws-cli


COPY files /

ENTRYPOINT ["/entrypoint.sh"]

Downloader script: entrypoint.sh

#!/usr/bin/env bash
set -e

localFileName="/models-local.tar.gz"

cd /local-storage

echo "Downloading ${MODELS_S3_PATH}"
aws s3 cp ${MODELS_S3_PATH} ${localFileName}

echo "Extracting ${localFileName}"
tar -xzf ${localFileName}

echo "Cleaning up..."
rm ${localFileName}

In the kubernetes deployment file, we add an init container.

initContainers:
  - name: extract
    image: modelsextract
    env:
      - name: MODELS_S3_PATH
        value: {{ .Values.modelsS3Path | quote }}
    volumeMounts:
      - mountPath: /local-storage
        name: local-storage

An emptyDir volume and mount for the exported folders:

volumes:
  - name: local-storage
    emptyDir:
      sizeLimit: 100Gi

volumeMounts:
  - mountPath: /root/.cache
    subPath: root/.cache
    name: local-storage
  - mountPath: /usr/local/lib/python3.12
    subPath: usr/local/lib/python3.12
    name: local-storage

Also, make sure to use the slim image instead of the huge image in the deployment container.

Final Note

We have reviewed creation of a slim docker image that later downloads the extra data from AWS S3. To save cost and gain speed, the actual download should be done in the same region as the deployment, so make sure to copy it the a relevant regional AWS S3 bucket.