Collecting Kubernetes Logs using LOKI

 

In this post we review the step to deploy a kubernetes logs collection system using LOKI, Promtail, and grafana.

• Grafana is a front-end GUI enabling view of the logs.
  I've already covered grafana deployment on kubernetes in this post.
  
  
• LOKI is a logs storage and log query component that works like a charm in kubernetes environment
  
  
• Promtail is responsible for sending the pods logs to LOKI.

Let's review the deployment steps.

Part of this is based on the LOKI deployment guide.

LOKI AWS entities

Create LOKI logs storage buckets:

aws s3api create-bucket --bucket  agentic-loki-chunks --region us-east-1  
aws s3api create-bucket --bucket  agentic-loki-ruler  --region us-east-1 

Create a permission policy for LOKI

rm -f loki-s3-policy.json

cat <<EOF > loki-s3-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LokiStorage",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::agentic-loki-chunks",
                "arn:aws:s3:::agentic-loki-chunks/*",
                "arn:aws:s3:::agentic-loki-ruler",
                "arn:aws:s3:::agentic-loki-ruler/*"
            ]
        }
    ]
}
EOF


aws iam create-policy --policy-name LokiS3AccessPolicy --policy-document file://loki-s3-policy.json
rm loki-s3-policy.json

Create LOKI trust policy to enable it to use the role in the kubernetes cluster

rm -f trust-policy.json

cat << EOF > trust-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::662909476770:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/873FB195FF4FAEC482E18822F7D4CBF9"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/873FB195FF4FAEC482E18822F7D4CBF9:sub": "system:serviceaccount:loki:loki",
                    "oidc.eks.us-east-1.amazonaws.com/id/873FB195FF4FAEC482E18822F7D4CBF9:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
EOF

Create role and attach the policies

aws iam create-role --role-name LokiServiceAccountRole --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --role-name LokiServiceAccountRole --policy-arn arn:aws:iam::662909476770:policy/LokiS3AccessPolicy

LOKI Deployment

We use helm to deploy LOKI on the kubernetes cluster.

kubectl create namespace loki

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

rm -f values.yaml

cat << EOF > values.yaml

loki:
   schemaConfig:
     configs:
       - from: "2024-04-01"
         store: tsdb
         object_store: s3
         schema: v13
         index:
           prefix: loki_index_
           period: 24h
   storage_config:
     aws:
       region: us-east-1  
       bucketnames: agentic-loki-chunks
       s3forcepathstyle: false
   ingester:
       chunk_encoding: snappy
   pattern_ingester:
       enabled: true
   limits_config:
     allow_structured_metadata: true
     volume_enabled: true
     retention_period: 672h # 28 days retention
   compactor:
     retention_enabled: true 
     delete_request_store: s3
   ruler:
    enable_api: true
    storage:
      type: s3
      s3:
        region: us-east-1
        bucketnames: agentic-loki-ruler
        s3forcepathstyle: false
      alertmanager_url: http://prom:9093 # The URL of the Alertmanager to send alerts (Prometheus, Mimir, etc.)

   querier:
      max_concurrent: 4

   storage:
      type: s3
      bucketNames:
        chunks: "agentic-loki-chunks" 
        ruler: "agentic-loki-ruler" 
      s3:
        region: us-east-1

serviceAccount:
 create: true
 annotations:
   "eks.amazonaws.com/role-arn": "arn:aws:iam::662909476770:role/LokiServiceAccountRole" 

deploymentMode: Distributed

ingester:
 replicas: 2
 zoneAwareReplication:
  enabled: false

querier:
 replicas: 2
 maxUnavailable: 1

queryFrontend:
 replicas: 2
 maxUnavailable: 1

queryScheduler:
 replicas: 2

distributor:
 replicas: 2
 maxUnavailable: 1
compactor:
 replicas: 1

indexGateway:
 replicas: 2
 maxUnavailable: 1

ruler:
 replicas: 1
 maxUnavailable: 1


gateway:
 service:
   type: ClusterIP
 basicAuth: 
     enabled: false

lokiCanary:
  extraArgs: []
  extraEnv: []

minio:
 enabled: false

backend:
 replicas: 0
read:
 replicas: 0
write:
 replicas: 0

singleBinary:
 replicas: 0

EOF

helm upgrade -i --values values.yaml loki grafana/loki -n loki 
rm values.yaml

Promtail

We use helm to deploy Promtail

rm -f values.yaml

cat << EOF > values.yaml

extraVolumes:
  - name: positions
    emptyDir: {}

extraVolumeMounts:
  - name: positions
    mountPath: /promtail/positions

config:
  clients:
    - url: http://loki-distributor.loki.svc.cluster.local:3100/loki/api/v1/push
      tenant_id: test

  positions:
    filename: /promtail/positions/positions.yaml

  scrape_configs:
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: pod


EOF

helm upgrade --install promtail grafana/promtail \
  --namespace promtail --create-namespace \
  -f values.yaml

rm values.yaml

Grafana

To view logs in grafana,  Add LOKI as datasource under Connections, Data sources:

http://loki-gateway.loki.svc.cluster.local

and add a customer header:

X-Scope-OrgID = test

Then add a visualization:

• New dashboard
• Settings, variables, add
• Show on dashboard: "Label and Value"
• Query type: "Label Values"
• Label: "container"
• Save
• Add visualization
• select on the top right "Logs"
• select LOKI as data source
• add filter container=${container}

Final Note

In this post we have reviewed the steps to collect kubernetes pods logs using Promtail, LOKI, and grafana. We have used the LOKI Simple Scalable Mode, but for a large kubernetes the LOKI Microservice mode is preferred.

Subscribing To Microsoft Copilot Events

 

In this post we will review how to subscribe to Microsoft Copilot events. 

Notice that the Microsoft Copilot uses a totally different mechanism than the Microsoft Copilot Studio agents, see the Microsoft External Threat Detection post.

Create Encryption Key

We start by create an a-symmetric encryption key:

openssl genrsa -out private.key 2048
openssl req -new -x509 -key private.key -out publicCert.cer -days 365
base64 publicCert.cer > publicCertBase64.txt
awk 'NF {printf "%s", $0}' publicCertBase64.txt > cert_clean.txt

Create App Registration

Use Microsoft Entra to create a new App Registration with  permissions:  AiEnterpriseInteraction.Read.All. Notice this permission is under the "Microsoft graph” section.

After adding the permissions to the AppRegistration, click the Grant Admin Consent button for the permissions.

We also add client secret allowing to use the AppRegistration in a script. As far as I could see, there is not GUI available for this, and we must use a script.

Create a Service

To supply a subscribe endpoint that Microsoft would send the messages to, create a public available service using a valid TLS certificate. For example, the endpoint can be:

https://my-site.com/interactions

Notice this endpoint should accept both GET and POST requests.

A very simple example of such endpoint is below.

func (e *Executor) Execute(p web.Parser) interface{} {
    log.Info("interactions starting")
    validation := p.QueryParam("validationToken")
    log.Info("token: %v", validation)

    data, err := p.GetBodyAsBytes()
    if err != nil {
       kiterr.RaiseIfError(err)
    }
    log.Info("body: %v", string(data))

    p.SetHeader("Content-Type", "text/plain")
    p.WriteStreamingResponse([]byte(validation))

    return nil
}

Call the Subscribe API

Use the following to subscribe to events:

#!/bin/bash

TENANT_ID="12345678-1234-1234-1234-123456789012"
CLIENT_ID="12345678-1234-1234-1234-123456789012"
CLIENT_SECRET="abcdefghijklmnopqrstuvwxyz1234567890abcd"

request_token(){
  SCOPE=graph.microsoft.com
  curl -s -X POST "https://login.microsoftonline.com/$TENANT_ID/oauth2/v2.0/token" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "client_id=$CLIENT_ID&scope=https%3A%2F%2F${SCOPE}%2F.default&client_secret=$CLIENT_SECRET&grant_type=client_credentials" \
    | jq -r '.access_token'
}

request_subscription(){
  curl -H "Authorization: Bearer $ACCESS_TOKEN" \
       -H "Content-Type: application/json" \
       -d '
{
  "changeType": "created,deleted,updated",
  "notificationUrl": "https://my-site.com/interactions",
  "resource": "/copilot/interactionHistory/getAllEnterpriseInteractions",
  "includeResourceData": true,
  "encryptionCertificate": "LS0tLS1CRUdJTiBDRjhgjkhgkjhgkjhJKJHKJLHLKJHLKJHKJLlkjhlkjhlkjhlkjhlkjhkojghlkjhlkjhlkjhlkjhlkjKJLHLKJHLKJHLKJH8769869876IUGHKLJHKJLYH876Y87H78YH87BN87HKJBJKHGKJLGKJLHlkjhlkjhkljhkjhlkjhlhjkEdVeElUQWZCZ05WQkFvTQpHRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MFpEQWVGdzB5TlRFeE1qY3hNakkzTURoYUZ3MHlOakV4Ck1qY3hNakkzTURoYU1FVXhDekFKQmdOVkJBWVRBa2xNTVJNd0VRWURWUVFJREFwVGIyMWxMVk4wWVhSbE1TRXcKSHdZRFZRUUtEQmhKYm5SbGNtNWxkQ0JYYVdSbmFYUnpJRkIwZVNCTWRHUXdnZ0VpTUEwR0NTcUdTSWIzRFFFQgpBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRRFB4Ny8wVzc4N0NLUUh0dHMyVDBoL25LZ0o1ejArb1ZHeFFzcFhSWnlnCnBuanpETkdqUjBtWGFVU2RTZ2JWNW05MDMrNnhqbS9LbHpuTlltOTdoUjJNcnBFSXd1OVVYaWhxU1FTS1ZVcTkKbDk0OVEzME5PK29lT0Z4K3huOC9ycGFMVmpxUzIzR3VUV09Ka3p2aktPeXVnV1BRN3FBazgrdjQ3NjdVUkVvYQpJV2l3aXBIVW4rajBMOTVDTEtFOUZQUXdLMkUzNnZrdWNzd1krSGh5bm45N1piSGszVUM3NXd1QlYwTWVyT0o2CjVQdTFYQUVPZ2JnSFFVUEhuVkViT05MdkNwSUl1MHZlZDZFZmRQbVlzTk1IK2xHSlBOZnFOemRYSEZYSXE4VWMKbHdjbDlPRllUb0dMSEdHWTJiRWpzNWxFUjN1OWtLNFlvc1llUFc2ZmJ3NHhBZ01CQUFHalV6QlJNQjBHQTFVZApEZ1FXQkJUbW45UTBBcmFtVFNTK0phbWtIbzR3eVVVSDd6QWZCZ05WSFNNRUdEQVdnQlRtbjlRMEFyYW1UU1MrCkphbWtIbzR3eVVVSDd6QVBCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRREwKRVh2cnUxb0NKNXlERVc2Njc3RlRuQWt5bitheWJqQXBaVmRiRi9vMXZyZWZKWHVBVzdnZ09WZjBrT2xCN2U0WgoyQW0rUnU1bmNiRXdBN0o0L2N0WWlLdVByLzA4U0NjTnp6ZGp6RG9qem5wL1ZadnRiYXo5NGlVOE52YmRyWXBkCkVnb1o1RVk3YzZpQW9JNDlGK2ZNOGZLR3FrL09oVDA0dUNuWk1SUFpFR0lob1dBR1J0ODg1R1VXcVNEdzJDYVAKT3F6eU5WeS8vMFpWQm40dTBER3VjQjVLVkp0Smh0MUNrRTlzeXJGV3IrSTFxTkltMkZoN3pyR1diSWRPL2gvMgpIOEFKY0xEM3QvdzNuZGUrdWl3dnFMbTVhUTcwS0k4Q2ZoZk5Mam9WcmUxTFMwK1ZxRjNlOEl6cXFtSEFQLytJCjk0aDFsOEMreVU5MHFxa3E4OFE5Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
  "encryptionCertificateId": "my-id",
  "expirationDateTime": "2025-11-27T14:00:00.0000000Z",
  "clientState": "my-state"
}
       ' \
       "https://graph.microsoft.com/v1.0/subscriptions"
}

ACCESS_TOKEN=$(request_token_graph)
request_subscription

Notice the subscription is limit to up to 2 days in the future, and should be renew to continue to get events.

The encryption certification is the content of the cert_clean.txt that we've created.

Decryption of the Messages

A simple bash to handle the encrypted parts of the messages is below.

dataKey_base64=$(jq -r '.value[].encryptedContent.dataKey' event/event.json)
encrypted_data_base64=$(jq -r '.value[].encryptedContent.data' event/event.json)
dataSignature_base64=$(jq -r '.value[].encryptedContent.dataSignature' event/event.json)

# Decode the base64-encoded symmetric key
echo "$dataKey_base64" | base64 --decode > encrypted_key.bin

# Decrypt the symmetric key using your RSA private key with OAEP padding
openssl pkeyutl -decrypt -inkey key/private.key -pkeyopt rsa_padding_mode:oaep -in encrypted_key.bin -out symmetric_key.bin

# Extract first 16 bytes of symmetric key as IV (hex)
iv=$(xxd -p -l 16 symmetric_key.bin)

# Decode encrypted data
echo "$encrypted_data_base64" | base64 --decode > encrypted_data.bin

# Decrypt using AES-CBC with PKCS7 padding
openssl enc -aes-256-cbc -d -in encrypted_data.bin -out decrypted_data.json \
    -K $(xxd -p -c 256 symmetric_key.bin) \
    -iv "$iv"

Final Note

As expected for a Microsoft API this is a complicated method to get data. 

Why is double encryption of the messages required? We are already using TLS.

Why can't we subscribe forever?

Anyway, eventually it works and can be used to store the agents interactions. 

Have fun.