Configure TLS for ElasticSearch in a Kubernetes Deployment

In this post I will review the steps to configure TLS for existing ElasticSearch, Kibana,FileBeat in a kubernetes deployment. All of these steps are performed automatically as part of a helm chart deployment. The only input for a pre-install helm hook are the ElasticSearch credentials.

Some of the items in this post are derived from the formal ElasticSearch helm chart.

The following updates were done:

• Pre-install hook-  Create TLS certificates, and create credentials secret
• ElasticSearch statefulset - Add environment variables to enable TLS
• Kibana deployment - Update kibana.yml to use the TLS
• FileBeat daemonset - Update filebeat.yaml to use the TLS

These steps are described below.

Pre-Install Hook

The first change is to create a job, that is run as a pre-install helm hook. The job runs a script that receives as input the required ElasticSearch cerdentials, and creates a kubernetes secret. The script also generates a Certificate Authority (CA), and self signs a key for the ElasticSearch server. The CA will be used by the ElasticSearch server, as well as by others: Kibana and FileBeat.

#!/usr/bin/env bash

credentialsSecretName=elastic-credentials
mkdir /certificates
# This is the name of the kubernetes service for elastic search
master=elasticsearch-rest-service

echo "===> Create CA"
elasticsearch-certutil ca \
  --out /certificates/elastic-stack-ca.p12 \
  --pass ''

echo "===> Create certificate"
elasticsearch-certutil cert \
  --name ${master} \
  --dns ${master} \
  --ca /certificates/elastic-stack-ca.p12 \
  --pass '' \
  --ca-pass '' \
  --out /certificates/elastic-certificates.p12

echo "===> Convert certificate"
openssl pkcs12 -nodes -passin pass:'' -in /certificates/elastic-certificates.p12 -out /certificates/elastic-certificate.pem
openssl x509 -outform der -in /certificates/elastic-certificate.pem -out /certificates/elastic-certificate.crt

echo "===> Extract CA chain"
openssl pkcs12 -passin pass:'' -in /certificates/elastic-certificates.p12 -cacerts -nokeys -out /certificates/elastic-ca-chain.pem

echo "===> Create CA secret"
kubectl create secret generic elastic-certificates --from-file=/certificates/elastic-certificates.p12

echo "===> Create CA chain secret"
kubectl create secret generic elastic-ca-chain --from-file=/certificates/elastic-ca-chain.pem

echo "===> Create certificate pem secret"
kubectl create secret generic elastic-certificate-pem --from-file=/certificates/elastic-certificate.pem

echo "===> Create certificate crt secret"
kubectl create secret generic elastic-certificate-crt --from-file=/certificates/elastic-certificate.crt

echo "===> Create credentials secret"
kubectl create secret generic ${credentialsSecretName} --from-literal=password=${ELASTICSEARCH_PASSWORD} --from-literal=username=${ELASTICSEARCH_USER}

ElasticSearch

The ElasticSearch statefulset should be configured to use TLS, so we add the certificates secret as volume:

volumes:
  - name: elastic-certificates
    secret:
      secretName: elastic-certificates

and map the volume to the ElasticSearch container:

volumeMounts:
  - name: elastic-certificates
    mountPath: /usr/share/elasticsearch/config/certs

Next, we add the TLS enabling environment variables:

- name: xpack.security.enabled
  value: "true"
- name: xpack.security.transport.ssl.enabled
  value: "true"
- name: xpack.security.transport.ssl.verification_mode
  value: "certificate"
- name: xpack.security.transport.ssl.keystore.path
  value: "/usr/share/elasticsearch/config/certs/elastic-certificates.p12"
- name: xpack.security.transport.ssl.truststore.path
  value: "/usr/share/elasticsearch/config/certs/elastic-certificates.p12"
- name: xpack.security.http.ssl.enabled
  value: "true"
- name: xpack.security.http.ssl.truststore.path
  value: "/usr/share/elasticsearch/config/certs/elastic-certificates.p12"
- name: xpack.security.http.ssl.keystore.path
  value: "/usr/share/elasticsearch/config/certs/elastic-certificates.p12"

And the credentials environment variables:

- name: ELASTIC_PASSWORD
  valueFrom:
    secretKeyRef:
      name: elastic-credentials
      key: password
- name: ELASTIC_USERNAME
  valueFrom:
    secretKeyRef:
      name: elastic-credentials
      key: username

Kibana

For the Kibana deployment, we should add the CA secret volume

volumes:
  - name: elastic-ca-chain
    secret:
      secretName: elastic-ca-chain

and mount it to the Kibana container

volumeMounts:
  - name: elastic-ca-chain
    mountPath: /ssl-certificates/elastic-ca-chain.pem
    subPath: elastic-ca-chain.pem

Then add environment variables for the ElasticSearch credentials

- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      name: elastic-credentials
      key: username
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      name: elastic-credentials
      key: password

and update kibana.yml with the TLS configuration

elasticsearch.hosts: ["https://elasticsearch-rest-service"]
elasticsearch.username: "elastic"
elasticsearch.password: "elastic"
elasticsearch.ssl.certificateAuthorities: [ "/ssl-certificates/elastic-ca-chain.pem" ]
xpack.monitoring.elasticsearch.ssl.verificationMode: "certificate"
server.host: "0.0.0.0"

FileBeat

In the FileBeat daemonset we should add the CA secret volume

volumes:
  - name: elastic-ca-chain
    secret:
      secretName: elastic-ca-chain

and mount it to the Kibana container

volumeMounts:
  - name: elastic-ca-chain
    mountPath: /ssl-certificates/elastic-ca-chain.pem
    subPath: elastic-ca-chain.pem

Then add environment variables for the ElasticSearch credentials

- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      name: elastic-credentials
      key: username
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      name: elastic-credentials
      key: password

And update filebeat.yaml with the TLS configuration

output.elasticsearch:
  hosts: ['https://elasticsearch-rest-service']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}
  ssl:
    certificate_authorities: ["/ssl-certificates/elastic-ca-chain.pem"]