In a recent post we've seen how to deploy ClickHouse standalone on kubernetes without operator. In this post we will further show deployment on kubernetes without operator, but this time we will use clickhouse cluster.
To install a cluster, we need to deploy 2 deployments: clickhouse keeper that is used to maintain a quorum, and the clickhouse server which includes the actual cluster nodes. We will use 3 replicas for the keeper, and 2 replicas for the server.
ClickHouse Keeper
The keeper includes the entities below.
The service
apiVersion: v1
kind: Service
metadata:
name: clickhousekeeper-service
spec:
selector:
configid: clickhousekeeper-container
clusterIP: None
ports:
- name: client
port: 9181
- name: raft
port: 9444
The config
apiVersion: v1
kind: ConfigMap
metadata:
name: clickhousekeeper-config
data:
keeper_config.xml: |-
<clickhouse>
<logger>
<level>information</level>
<console>true</console>
</logger>
<listen_host>0.0.0.0</listen_host>
<keeper_server>
<tcp_port>9181</tcp_port>
<server_id>REPLACE_ME_SERVER_ID</server_id>
<log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
<snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
<raft_configuration>
<server>
<id>1</id>
<hostname>clickhousekeeper-statefulset-0.clickhousekeeper-service.default.svc.cluster.local</hostname>
<port>9444</port>
</server>
<server>
<id>2</id>
<hostname>clickhousekeeper-statefulset-1.clickhousekeeper-service.default.svc.cluster.local</hostname>
<port>9444</port>
</server>
</raft_configuration>
</keeper_server>
</clickhouse>
The statefuleset
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: clickhousekeeper-statefulset
spec:
serviceName: clickhousekeeper-service
podManagementPolicy: Parallel
replicas: 3
selector:
matchLabels:
configid: clickhousekeeper-container
template:
metadata:
labels:
configid: clickhousekeeper-container
spec:
terminationGracePeriodSeconds: 10
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchExpressions:
- key: configid
operator: In
values:
- clickhousekeeper-container
initContainers:
- name: config
image: clickhouse/clickhouse-keeper:latest
imagePullPolicy: Always
command:
- bash
- -c
- |
set -e
POD_ORDINAL=$(echo $HOSTNAME | awk -F'-' '{print $NF}')
cp /config-origin/keeper_config.xml /etc/clickhouse-keeper/keeper_config.xml
sed -i "s/REPLACE_ME_SERVER_ID/$((POD_ORDINAL+1))/" /etc/clickhouse-keeper/keeper_config.xml
cat /etc/clickhouse-keeper/keeper_config.xml
volumeMounts:
- name: clickhousekeeper-config
mountPath: /config-origin
- name: clickhousekeeper-runtime-config
mountPath: /etc/clickhouse-keeper
containers:
- name: clickhousekeeper
image: clickhouse/clickhouse-keeper:latest
imagePullPolicy: Always
ports:
- containerPort: 9181
name: client
- containerPort: 9444
name: raft
volumeMounts:
- name: pvc
mountPath: /var/lib/clickhouse
- name: clickhousekeeper-runtime-config
mountPath: /etc/clickhouse-keeper
volumes:
- name: clickhousekeeper-config
configMap:
name: clickhousekeeper-config
- name: clickhousekeeper-runtime-config
emptyDir: {}
volumeClaimTemplates:
- metadata:
name: pvc
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
Notice the config map is saved to a side folder, and updated with the related ID by the pod name.
ClickHouse Server
The server includes the entities below.
The service
apiVersion: v1
kind: Service
metadata:
name: clickhouseserver-service
spec:
selector:
configid: clickhouseserver-container
clusterIP: None
ports:
- name: tcp
port: 9000
- name: http
port: 8123
The config
apiVersion: v1
kind: ConfigMap
metadata:
name: clickhouseserver-config
data:
custom.xml: |-
<clickhouse>
<logger>
<level>information</level>
<console>true</console>
</logger>
<listen_host>0.0.0.0</listen_host>
<keeper_server>
<tcp_port>9181</tcp_port>
</keeper_server>
<zookeeper>
<node>
<host>clickhousekeeper-statefulset-0.clickhousekeeper-service.default.svc.cluster.local</host>
<port>9181</port>
</node>
<node>
<host>clickhousekeeper-statefulset-1.clickhousekeeper-service.default.svc.cluster.local</host>
<port>9181</port>
</node>
<node>
<host>clickhousekeeper-statefulset-2.clickhousekeeper-service.default.svc.cluster.local</host>
<port>9181</port>
</node>
</zookeeper>
<remote_servers>
<llmp_clickhouse_cluster>
<shard>
<replica>
<host>clickhouseserver-statefulset-0.clickhouseserver-service.default.svc.cluster.local</host>
<port>9000</port>
</replica>
<replica>
<host>clickhouseserver-statefulset-1.clickhouseserver-service.default.svc.cluster.local</host>
<port>9000</port>
</replica>
</shard>
</llmp_clickhouse_cluster>
</remote_servers>
</clickhouse>
users.xml: |-
<clickhouse>
<users>
<default>
<password></password>
<networks>
<ip>::/0</ip>
</networks>
</default>
</users>
</clickhouse>
macros.xml: |-
<clickhouse>
<macros>
</macros>
</clickhouse>
The statefulset
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: clickhouseserver-statefulset
spec:
serviceName: clickhouseserver-service
podManagementPolicy: Parallel
replicas: 2
selector:
matchLabels:
configid: clickhouseserver-container
template:
spec:
terminationGracePeriodSeconds: 10
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchExpressions:
- key: configid
operator: In
values:
- clickhouseserver-container
initContainers:
- name: config
image: lclickhouse/clickhouse-server:latest
imagePullPolicy: Always
command:
- bash
- -c
- |
set -e
cp /config-origin/custom.xml /etc/clickhouse-server/config.d/custom.xml
cp /config-origin/macros.xml /etc/clickhouse-server/config.d/macros.xml
sed -i "s/REPLACE_ME_HOST_NAME/${HOSTNAME}/" /etc/clickhouse-server/config.d/macros.xml
volumeMounts:
- name: clickhouseserver-config
mountPath: /config-origin
- name: clickhouseserver-runtime-config
mountPath: /etc/clickhouse-server/config.d
containers:
- name: clickhouseserver
image: clickhouse/clickhouse-server:latest
imagePullPolicy: Always
ports:
- containerPort: 9181
name: client
- containerPort: 9444
name: raft
volumeMounts:
- name: pvc
mountPath: /var/lib/clickhouse
- name: clickhouseserver-runtime-config
mountPath: /etc/clickhouse-server/config.d
- name: clickhouseserver-config
mountPath: /etc/clickhouse-server/users.d/users.xml
subPath: users.xml
volumes:
- name: clickhouseserver-config
configMap:
name: clickhouseserver-config
- name: clickhouseserver-runtime-config
emptyDir: {}
volumeClaimTemplates:
- metadata:
name: pvc
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
Here again we update the configuration using an init container.
Final Note
Using clickhouse operator is much simpler than the manual mode, but in some cases specific adaptations are required that block us from using the operator. In this post we've displayed the full requirements for a minimal manual deployment of a clickhouse cluster.
No comments:
Post a Comment