Collecting Kubernetes Logs using LOKI

 

In this post we review the step to deploy a kubernetes logs collection system using LOKI, Promtail, and grafana.

• Grafana is a front-end GUI enabling view of the logs.
  I've already covered grafana deployment on kubernetes in this post.
  
  
• LOKI is a logs storage and log query component that works like a charm in kubernetes environment
  
  
• Promtail is responsible for sending the pods logs to LOKI.

Let's review the deployment steps.

Part of this is based on the LOKI deployment guide.

LOKI AWS entities

Create LOKI logs storage buckets:

aws s3api create-bucket --bucket  agentic-loki-chunks --region us-east-1  
aws s3api create-bucket --bucket  agentic-loki-ruler  --region us-east-1 

Create a permission policy for LOKI

rm -f loki-s3-policy.json

cat <<EOF > loki-s3-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LokiStorage",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::agentic-loki-chunks",
                "arn:aws:s3:::agentic-loki-chunks/*",
                "arn:aws:s3:::agentic-loki-ruler",
                "arn:aws:s3:::agentic-loki-ruler/*"
            ]
        }
    ]
}
EOF


aws iam create-policy --policy-name LokiS3AccessPolicy --policy-document file://loki-s3-policy.json
rm loki-s3-policy.json

Create LOKI trust policy to enable it to use the role in the kubernetes cluster

rm -f trust-policy.json

cat << EOF > trust-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::662909476770:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/873FB195FF4FAEC482E18822F7D4CBF9"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/873FB195FF4FAEC482E18822F7D4CBF9:sub": "system:serviceaccount:loki:loki",
                    "oidc.eks.us-east-1.amazonaws.com/id/873FB195FF4FAEC482E18822F7D4CBF9:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
EOF

Create role and attach the policies

aws iam create-role --role-name LokiServiceAccountRole --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --role-name LokiServiceAccountRole --policy-arn arn:aws:iam::662909476770:policy/LokiS3AccessPolicy

LOKI Deployment

We use helm to deploy LOKI on the kubernetes cluster.

kubectl create namespace loki

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

rm -f values.yaml

cat << EOF > values.yaml

loki:
   schemaConfig:
     configs:
       - from: "2024-04-01"
         store: tsdb
         object_store: s3
         schema: v13
         index:
           prefix: loki_index_
           period: 24h
   storage_config:
     aws:
       region: us-east-1  
       bucketnames: agentic-loki-chunks
       s3forcepathstyle: false
   ingester:
       chunk_encoding: snappy
   pattern_ingester:
       enabled: true
   limits_config:
     allow_structured_metadata: true
     volume_enabled: true
     retention_period: 672h # 28 days retention
   compactor:
     retention_enabled: true 
     delete_request_store: s3
   ruler:
    enable_api: true
    storage:
      type: s3
      s3:
        region: us-east-1
        bucketnames: agentic-loki-ruler
        s3forcepathstyle: false
      alertmanager_url: http://prom:9093 # The URL of the Alertmanager to send alerts (Prometheus, Mimir, etc.)

   querier:
      max_concurrent: 4

   storage:
      type: s3
      bucketNames:
        chunks: "agentic-loki-chunks" 
        ruler: "agentic-loki-ruler" 
      s3:
        region: us-east-1

serviceAccount:
 create: true
 annotations:
   "eks.amazonaws.com/role-arn": "arn:aws:iam::662909476770:role/LokiServiceAccountRole" 

deploymentMode: Distributed

ingester:
 replicas: 2
 zoneAwareReplication:
  enabled: false

querier:
 replicas: 2
 maxUnavailable: 1

queryFrontend:
 replicas: 2
 maxUnavailable: 1

queryScheduler:
 replicas: 2

distributor:
 replicas: 2
 maxUnavailable: 1
compactor:
 replicas: 1

indexGateway:
 replicas: 2
 maxUnavailable: 1

ruler:
 replicas: 1
 maxUnavailable: 1


gateway:
 service:
   type: ClusterIP
 basicAuth: 
     enabled: false

lokiCanary:
  extraArgs: []
  extraEnv: []

minio:
 enabled: false

backend:
 replicas: 0
read:
 replicas: 0
write:
 replicas: 0

singleBinary:
 replicas: 0

EOF

helm upgrade -i --values values.yaml loki grafana/loki -n loki 
rm values.yaml

Promtail

We use helm to deploy Promtail

rm -f values.yaml

cat << EOF > values.yaml

extraVolumes:
  - name: positions
    emptyDir: {}

extraVolumeMounts:
  - name: positions
    mountPath: /promtail/positions

config:
  clients:
    - url: http://loki-distributor.loki.svc.cluster.local:3100/loki/api/v1/push
      tenant_id: test

  positions:
    filename: /promtail/positions/positions.yaml

  scrape_configs:
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: pod


EOF

helm upgrade --install promtail grafana/promtail \
  --namespace promtail --create-namespace \
  -f values.yaml

rm values.yaml

Grafana

To view logs in grafana,  Add LOKI as datasource under Connections, Data sources:

http://loki-gateway.loki.svc.cluster.local

and add a customer header:

X-Scope-OrgID = test

Then add a visualization:

• New dashboard
• Settings, variables, add
• Show on dashboard: "Label and Value"
• Query type: "Label Values"
• Label: "container"
• Save
• Add visualization
• select on the top right "Logs"
• select LOKI as data source
• add filter container=${container}

Final Note

In this post we have reviewed the steps to collect kubernetes pods logs using Promtail, LOKI, and grafana. We have used the LOKI Simple Scalable Mode, but for a large kubernetes the LOKI Microservice mode is preferred.