Full Blog TOC

Full Blog Table Of Content with Keywords Available HERE

Monday, May 11, 2026

Run Lambda from AWS Bedrock Agent


 


In this post we will review the step required to create an AWS bderock agent that run a lambda function. I am creating this post since AWS documentation is not sufficient to achieve this goal. I guess most software engineers today assume that you will use repeatadly use LLM such as ChatGPT to solve their no-so-good product and documentation issues.


Create an Agent

Start with agent creation, you can follow the steps in this post.


Create Action Group

Notice:
Make sure to select a model that you have permissions for. For example, in case you choose one of Anthropic models, you will probably fail with permissions since you did not purchased this model. And yes, this error will be just a general error forcing you to look for the actual failure reason yourself.


Next we need to create an action group: 

  • Edit the agent, and under the action group section click add new action group
  • Select "Define with API schemas" as the action group type
  • Select "Quick create a new Lambda function" in the action group invocation
  • Select "Define via in-line schema editor" in the action group schema

An example schema is the following:

openapi: 3.0.0
info:
  title: Person Info API
  version: 1.0.0
  description: API to get the person info
paths:
  /get_person_info:
    get:
      summary: Gets the person info
      description: Gets the person info
      operationId: get_person_info
      responses:
        '200':
          description: Gets the person info
          content:
            'application/json':
              schema:
                type: object
                properties:
                  info:
                    type: string
                    description: The person info



Create the Lambda

After clicking save, we can click on the button "View" next to the "Select Lambda function" combo. This opens a new tab where we can edit the lambda code. An example working lambda code is:


import logging
from typing import Dict, Any
from http import HTTPStatus
import json

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event: Dict[str, Any], context: Any) -> Dict[str, Any]:

try:
action_group = event['actionGroup']
api_path = event['apiPath']
http_method = event['httpMethod']
message_version = event.get('messageVersion', '1.0')

# Your business logic output
result = {
"info": "this Person ate the entire cake"
}

response = {
"messageVersion": message_version,
"response": {
"actionGroup": action_group,
"apiPath": api_path,
"httpMethod": http_method,
"httpStatusCode": 200,
"responseBody": {
"application/json": {
"body": json.dumps(result)
}
}
}
}

logger.info("Response: %s", response)
return response

except Exception as e:
logger.error("Error: %s", str(e))

return {
"messageVersion": "1.0",
"response": {
"actionGroup": event.get("actionGroup", ""),
"apiPath": event.get("apiPath", ""),
"httpMethod": event.get("httpMethod", ""),
"httpStatusCode": 500,
"responseBody": {
"application/json": {
"body": json.dumps({"error": str(e)})
}
}
}
}



Notice:
Do not use the lambda code example that you get, it contains bugs.

Using the Lambda


Using the lambda is not straight forward. You need to perform additional 3 steps:

1. Deploy
After each update the the lambda code, click on Deploy.
Then, on the top of the screen, click on Actions, Publish new version.

2. Permissions
You need to add permissions to the agent role. Notice the permission includes the deployed version, so you need to rerurn this after each deployment. An example for permissions update is:

aws lambda add-permission   \ 
  --function-name get_person_info-3urcv   \
  --statement-id allow-bedrock-invoke-version-1 \
  --action lambda:InvokeFunction   \
  --principal bedrock.amazonaws.com   \
  --source-arn arn:aws:bedrock:us-east-1:123460611234:agent/*   \
  --qualifier 2   \
  --region us-east-1



3. Configure the agent
Edit the agent, then edit the action group, and replace the used lambda version under "Select Lambda function". Then save AND prepare the agent to use the updated configuration.

Final Note

Fun, it was not, but eventually it is working. We can now chat with the agent from the AWS console and using code like the one in the agent creation post.

Sunday, May 3, 2026

Deploy a PostgreSQL Cluster on Kubernetes


 


In this post we will review the steps to deploy a PostgreSQL cluster on a kubernetes cluster. 

PostgreSQL is a veteran DBMS existing for ~30 years(!!). As such it is less a kubernetes player and does not automatically adjust the cluster like other technologies such as NATS and ClickHouse. Hence to maintain the cluster we require first to deploy an operator. There are several operators available, and in this post we use the CloudNativePG.


Cloud Native PG Operator

To deploy the CloudNativePG we can use the simple command:

kubectl apply -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.29/releases/cnpg-1.29.0.yaml

However, in case of need to include this as part of an umbrella helm chart, we should download the file and split it to multiple templates. The CRDs should be included in a different folder, for example:

my-umbrella-helm/charts/cnpg/crds

The CRDs folder is a special helm folder enabling deployment of the CRDs before any additional templates are rendered.

Other template can be deployed in the standard templates folder, for example:

my-umbrella-helm/charts/cnpg/templates

We would usually update the templates in this folder to match our own helm deployment standards such as entities names, images location, labels, enable/disable flags. Notice that CloudNativePG requires the label:

app.kubernetes.io/name: cloudnative-pg

So we must include this label when customizing the labels.

The PostgreSQL cluster

Once the operator is up and running, all we need to do is to render a cluster CRD, for example:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgresql
spec:
instances: 3

imageName: my-repo/my-postgresql/dev:18.3
imagePullPolicy: Always
storage:
size: 10Gi

postgresql:
parameters:
max_connections: "200"

bootstrap:
initdb:
database: my-db
owner: admin
secret:
name: postgresql-secret


We usually update the image name to be downloaded from our own repo. Notice the CloudNativePG blocks usage of "latest" version for the PostgreSQL cluster, so even on the local kind deployment we must use a specific version.

In the cluster resource we specify several important settings: the default database name, the default owner role of the database, and a secret with the password for that user.

Final Note

We've reviewed steps to setup a PostgreSQL cluster using the CloudNativePG operator. While not as easy to setup as other DBMS, PostgreSQL is considered one of the best relational DBMS with a good support for updates.



Sunday, April 12, 2026

MCP Client in GO and Stub


 


In the following post we present creation of MCP Client in GO including API, implementation, and stub. This method of providing APIs is critical for a successful testing which should not access any external entity. Note that in this case, we also provide a factory API, implementation, and stub further enabling the code to create MCP client even when it is running in the tests scope.


MCP Client

The client API:

type McpToolInfo struct {
ToolName string
ToolDescription string
InputSchema string
OutputSchema string
}

type McpClientApi interface {
ListTools() []*McpToolInfo
CloseSession()
}

The implementation:

// static - to use connection pool for all servers
var staticWebClient = web.ProduceClientImpl(core.Config.McpConnectionTimeout)

type McpClientImpl struct {
session *mcp.ClientSession
context context.Context
}

func ProduceMcpClientImpl(
mcpServerUrl string,
) *McpClientImpl {
m := &McpClientImpl{
context: context.Background(),
}

client := mcp.NewClient(
&mcp.Implementation{
Name: "my-go-client",
Version: "1.0.0",
},
nil,
)

transport := &mcp.StreamableClientTransport{
Endpoint: mcpServerUrl,
HTTPClient: staticWebClient.UsedClient(),
}

session, err := client.Connect(m.context, transport, nil)
kiterr.RaiseIfError(err)

m.session = session
return m
}

func (m *McpClientImpl) ListTools() []*McpToolInfo {
parameters := mcp.ListToolsParams{}
tools, err := m.session.ListTools(m.context, &parameters)
kiterr.RaiseIfError(err)

var result []*McpToolInfo
for _, tool := range tools.Tools {
info := McpToolInfo{
ToolName: tool.Name,
ToolDescription: tool.Description,
InputSchema: fmt.Sprintf("%v", tool.InputSchema),
OutputSchema: fmt.Sprintf("%v", tool.OutputSchema),
}
result = append(result, &info)
}

return result
}

func (m *McpClientImpl) CloseSession() {
err := m.session.Close()
m.session = nil
kiterr.RaiseIfError(err)
}

And the stub:

type McpClientStub struct {
}

func ProduceMcpClientStub(
mcpServerUrl string,
) *McpClientStub {
return &McpClientStub{}
}

func (m *McpClientStub) ListTools() []*McpToolInfo {
return []*McpToolInfo{
{
ToolName: "tool-1",
ToolDescription: "the best tool",
InputSchema: "my-schema-input",
OutputSchema: "my-schema-output",
},
}
}

func (m *McpClientStub) CloseSession() {
}

MCP Client Factory

The factory API:

import "radware.com/mcpp/commons/mcp/client"

type McpClientFactoryApi interface {
ProduceMcpClient(
mcpServerUrl string,
) client.McpClientApi
}

The factory implementation:

type McpClientFactoryImpl struct {
}

func ProduceMcpClientFactoryImpl() *McpClientFactoryImpl {
return &McpClientFactoryImpl{}
}

func (f *McpClientFactoryImpl) ProduceMcpClient(
mcpServerUrl string,
) client.McpClientApi {
return client.ProduceMcpClientImpl(mcpServerUrl)
}

And the stub:

type McpClientFactoryStub struct {
}

func ProduceMcpClientFactoryStub() *McpClientFactoryStub {
return &McpClientFactoryStub{}
}

func (f *McpClientFactoryStub) ProduceMcpClient(
mcpServerUrl string,
) client.McpClientApi {
return client.ProduceMcpClientStub(mcpServerUrl)
}


Monday, March 23, 2026

ClickHouse Cluster Agnostic DDL


 

In this post we present an method to run DDL for table creation and table drop regardless the fact whether we use a clickhouse cluster or a standalone clickhouse.

Clickhouse standalone and clickhouse cluster differ on the statement to create and drop a table.

Below are examples


Table Creation for Standalone

CREATE TABLE IF NOT EXISTS events
(
id UInt64,
user_id UInt32,
event_type String,
created_at DateTime
)
ENGINE = MergeTree()
ORDER BY (id)


Table Creation for Cluster

Here we need to create both the local table that we exist on each of the cluster nodes, and the distributed table that enables fetching from any cluster node.

CREATE TABLE IF NOT EXISTS events_LOCAL on CLUSTER my_cluster
(
id UInt64,
user_id UInt32,
event_type String,
created_at DateTime
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/{table}', '{replica}')
ORDER BY (id)

;

CREATE TABLE IF NOT EXISTS events on CLUSTER my_cluster
(
id UInt64,
user_id UInt32,
event_type String,
created_at DateTime
)
ENGINE = Distributed(
my_cluster,
default,
events_LOCAL,
1
)


Table Drop for Standalone

DROP TABLE IF EXISTS events


Table Drop for Cluster

Notice that not only we need to drop the local and distributed tables, but we also need to remove the replica from clickhouse keeper.


DROP TABLE IF EXISTS events on cluster my_cluster
;
DROP TABLE IF EXISTS events_LOCAL on cluster my_cluster
;
SYSTEM DROP REPLICA 'clickhouseserver-statefulset-0'
FROM ZKPATH '/clickhouse/tables/01/events_LOCAL'
;
SYSTEM DROP REPLICA 'clickhouseserver-statefulset-1'
FROM ZKPATH '/clickhouse/tables/01/events_LOCAL'


Abstract DDL Wrapper

To simplify all these actions, we create a wrapper code for the DDL.


func GetLocalTableName(
tableName string,
) string {
if clickhouse.Config.ClickhouseConnectionToCluster {
return tableName + "_LOCAL"
}
return tableName
}

func convertToClusterEngine(
engineAndMore string,
) string {
re := regexp.MustCompile(`(\w*MergeTree)\s*\(([^)]*)\)`)

located := re.FindStringSubmatch(engineAndMore)
if located == nil {
kiterr.RaiseError("unable to locate engine in %v", engineAndMore)
}

engine := located[1]
parameters := located[2]

if parameters != "" {
parameters = ", " + parameters
}

replacement := fmt.Sprintf(
"Replicated%s('/clickhouse/tables/{shard}/{table}', '{replica}'%s)",
engine,
parameters,
)

return re.ReplaceAllString(engineAndMore, replacement)
}

func CreateTable(
clickHouse clickhouse.ClickhouseApi,
tableName string,
columns string,
engineAndMore string,
) {
engineAndMore = strings.TrimSpace(engineAndMore)

if clickhouse.Config.ClickhouseConnectionToCluster {
local := fmt.Sprintf(
`
CREATE TABLE IF NOT EXISTS %v on CLUSTER %v
%v
ENGINE = %v
`,
GetLocalTableName(tableName),
clickhouse.Config.ClickhouseClusterName,
columns,
convertToClusterEngine(engineAndMore),
)
clickHouse.ExecuteStatement(local)

distributed := fmt.Sprintf(
`
CREATE TABLE IF NOT EXISTS %v on CLUSTER %v
%v
ENGINE = Distributed(
%v,
default,
%v,
1
)
`,
tableName,
clickhouse.Config.ClickhouseClusterName,
columns,
clickhouse.Config.ClickhouseClusterName,
GetLocalTableName(tableName),
)
clickHouse.ExecuteStatement(distributed)

} else {
statement := fmt.Sprintf(
`
CREATE TABLE IF NOT EXISTS %v
%v
ENGINE = %v
`,
tableName,
columns,
engineAndMore,
)
clickHouse.ExecuteStatement(statement)
}
}

func DropTable(
clickHouse clickhouse.ClickhouseApi,
tableName string,
) {
if clickhouse.Config.ClickhouseConnectionToCluster {
distributed := fmt.Sprintf(
`DROP TABLE IF EXISTS %v on cluster %v`,
tableName,
clickhouse.Config.ClickhouseClusterName,
)
clickHouse.ExecuteStatementOnAllNodes(distributed)

local := fmt.Sprintf(
`DROP TABLE IF EXISTS %v on cluster %v`,
GetLocalTableName(tableName),
clickhouse.Config.ClickhouseClusterName,
)
clickHouse.ExecuteStatementOnAllNodes(local)

dropReplica(clickHouse, tableName, 0)
dropReplica(clickHouse, tableName, 1)

} else {
clickHouse.ExecuteStatement(`DROP TABLE IF EXISTS ` + tableName)
}
}

func dropReplica(
clickHouse clickhouse.ClickhouseApi,
tableName string,
nodeIndex int,
) {
err := kiterr.RunSafe(func() {
keeper := fmt.Sprintf(
`SYSTEM DROP REPLICA 'democlickhouseserver-statefulset-%v' FROM ZKPATH '/clickhouse/tables/01/%v'`,
nodeIndex,
GetLocalTableName(tableName),
)
clickHouse.ExecuteStatementOnAllNodes(keeper)
})
//ignoring error
kiterr.LogWarnIfError(err)
}

and an example for calling this code is:


  tableName := "events"

columns := `
(
id UInt64,
user_id UInt32,
event_type String,
created_at DateTime
)
`

engine := "MergeTree() ORDER BY (id)"

CreateTable(
clickhouseApi,
tableName,
columns,
engine,
)

DropTable(
clickhouseApi,
tableName,
)


Final Note

We've shown the DDL differences for create and drop tables between clickhouse standalone and clickhouse cluster.
We've also shown how to create a code that is agnostic to these difference.



Edit

Another non DDL issue is about delete rows which also looks different. This can be handle in an agnostic method similar to the functions we've presented above.



func DeleteFromTable(
clickHouse clickhouse.ClickhouseApi,
tableName string,
whereClause string,
args ...any,
) {
if clickhouse.Config.ClickhouseConnectionToCluster {
local := fmt.Sprintf(`
ALTER TABLE %v
ON CLUSTER %v
DELETE %v
`,
GetLocalTableName(tableName),
clickhouse.Config.ClickhouseClusterName,
whereClause,
)
clickHouse.ExecuteStatement(local, args...)
} else {
local := GetDeleteStatementStandalone(tableName, whereClause)
clickHouse.ExecuteStatement(local, args...)
}
}

func GetDeleteStatementStandalone(
tableName string,
whereClause string,
) string {
return fmt.Sprintf(`
ALTER TABLE %v
DELETE %v
`,
tableName,
whereClause,
)
}

And an example for call is:


tableName := "events"

whereClause := "WHERE user_id = ?"

DeleteFromTable(
clickhouseApi,
tableName,
whereClause,
12345,
)

Monday, March 16, 2026

Deploy ClickHouse Cluster on Kubernetes without Operator


 


In a recent post we've seen how to deploy ClickHouse standalone on kubernetes without operator. In this post we will further show deployment on kubernetes without operator, but this time we will use clickhouse cluster.

To install a cluster, we need to deploy 2 deployments: clickhouse keeper that is used to maintain a quorum, and the clickhouse server which includes the actual cluster nodes. We will use 3 replicas for the keeper, and 2 replicas for the server.


ClickHouse Keeper

The keeper includes the entities below.


The service

apiVersion: v1
kind: Service
metadata:
name: clickhousekeeper-service
spec:
selector:
configid: clickhousekeeper-container
clusterIP: None
ports:
- name: client
port: 9181
- name: raft
port: 9444


The config

apiVersion: v1
kind: ConfigMap
metadata:
name: clickhousekeeper-config
data:
keeper_config.xml: |-
<clickhouse>

<logger>
<level>information</level>
<console>true</console>
</logger>

<listen_host>0.0.0.0</listen_host>

<keeper_server>
<tcp_port>9181</tcp_port>

<server_id>REPLACE_ME_SERVER_ID</server_id>

<log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
<snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>

<raft_configuration>
<server>
<id>1</id>
<hostname>clickhousekeeper-statefulset-0.clickhousekeeper-service.default.svc.cluster.local</hostname>
<port>9444</port>
</server>
<server>
<id>2</id>
<hostname>clickhousekeeper-statefulset-1.clickhousekeeper-service.default.svc.cluster.local</hostname>
<port>9444</port>
</server>
</raft_configuration>
</keeper_server>
</clickhouse>


The statefuleset

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: clickhousekeeper-statefulset
spec:
serviceName: clickhousekeeper-service
podManagementPolicy: Parallel
replicas: 3
selector:
matchLabels:
configid: clickhousekeeper-container
template:
metadata:
labels:
configid: clickhousekeeper-container
spec:
terminationGracePeriodSeconds: 10
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchExpressions:
- key: configid
operator: In
values:
- clickhousekeeper-container

initContainers:
- name: config
image: clickhouse/clickhouse-keeper:latest
imagePullPolicy: Always
command:
- bash
- -c
- |
set -e
POD_ORDINAL=$(echo $HOSTNAME | awk -F'-' '{print $NF}')
cp /config-origin/keeper_config.xml /etc/clickhouse-keeper/keeper_config.xml
sed -i "s/REPLACE_ME_SERVER_ID/$((POD_ORDINAL+1))/" /etc/clickhouse-keeper/keeper_config.xml
cat /etc/clickhouse-keeper/keeper_config.xml
volumeMounts:
- name: clickhousekeeper-config
mountPath: /config-origin
- name: clickhousekeeper-runtime-config
mountPath: /etc/clickhouse-keeper
containers:
- name: clickhousekeeper
image: clickhouse/clickhouse-keeper:latest
imagePullPolicy: Always
ports:
- containerPort: 9181
name: client
- containerPort: 9444
name: raft
volumeMounts:
- name: pvc
mountPath: /var/lib/clickhouse
- name: clickhousekeeper-runtime-config
mountPath: /etc/clickhouse-keeper

volumes:
- name: clickhousekeeper-config
configMap:
name: clickhousekeeper-config
- name: clickhousekeeper-runtime-config
emptyDir: {}

volumeClaimTemplates:
- metadata:
name: pvc
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "gp2"
resources:
requests:
storage: 10Gi


Notice the config map is saved to a side folder, and updated with the related ID by the pod name.


ClickHouse Server

The server includes the entities below.


The service

apiVersion: v1
kind: Service
metadata:
name: clickhouseserver-service
spec:
selector:
configid: clickhouseserver-container
clusterIP: None
ports:
- name: tcp
port: 9000
- name: http
port: 8123


The config

apiVersion: v1
kind: ConfigMap
metadata:
name: clickhouseserver-config
data:
custom.xml: |-
<clickhouse>
<logger>
<level>information</level>
<console>true</console>
</logger>

<listen_host>0.0.0.0</listen_host>

<keeper_server>
<tcp_port>9181</tcp_port>
</keeper_server>

<zookeeper>
<node>
<host>clickhousekeeper-statefulset-0.clickhousekeeper-service.default.svc.cluster.local</host>
<port>9181</port>
</node>
<node>
<host>clickhousekeeper-statefulset-1.clickhousekeeper-service.default.svc.cluster.local</host>
<port>9181</port>
</node>
<node>
<host>clickhousekeeper-statefulset-2.clickhousekeeper-service.default.svc.cluster.local</host>
<port>9181</port>
</node>
</zookeeper>

<remote_servers>
<llmp_clickhouse_cluster>
<shard>
<replica>
<host>clickhouseserver-statefulset-0.clickhouseserver-service.default.svc.cluster.local</host>
<port>9000</port>
</replica>
<replica>
<host>clickhouseserver-statefulset-1.clickhouseserver-service.default.svc.cluster.local</host>
<port>9000</port>
</replica>
</shard>
</llmp_clickhouse_cluster>
</remote_servers>
</clickhouse>
users.xml: |-
<clickhouse>
<users>
<default>
<password></password>
<networks>
<ip>::/0</ip>
</networks>
</default>
</users>
</clickhouse>
macros.xml: |-
<clickhouse>
<macros>
</macros>
</clickhouse>


The statefulset

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: clickhouseserver-statefulset
spec:
serviceName: clickhouseserver-service
podManagementPolicy: Parallel
replicas: 2
selector:
matchLabels:
configid: clickhouseserver-container
template:
spec:
terminationGracePeriodSeconds: 10
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchExpressions:
- key: configid
operator: In
values:
- clickhouseserver-container

initContainers:
- name: config
image: lclickhouse/clickhouse-server:latest
imagePullPolicy: Always
command:
- bash
- -c
- |
set -e
cp /config-origin/custom.xml /etc/clickhouse-server/config.d/custom.xml
cp /config-origin/macros.xml /etc/clickhouse-server/config.d/macros.xml
sed -i "s/REPLACE_ME_HOST_NAME/${HOSTNAME}/" /etc/clickhouse-server/config.d/macros.xml
volumeMounts:
- name: clickhouseserver-config
mountPath: /config-origin
- name: clickhouseserver-runtime-config
mountPath: /etc/clickhouse-server/config.d
containers:
- name: clickhouseserver
image: clickhouse/clickhouse-server:latest
imagePullPolicy: Always
ports:
- containerPort: 9181
name: client
- containerPort: 9444
name: raft
volumeMounts:
- name: pvc
mountPath: /var/lib/clickhouse
- name: clickhouseserver-runtime-config
mountPath: /etc/clickhouse-server/config.d
- name: clickhouseserver-config
mountPath: /etc/clickhouse-server/users.d/users.xml
subPath: users.xml

volumes:
- name: clickhouseserver-config
configMap:
name: clickhouseserver-config
- name: clickhouseserver-runtime-config
emptyDir: {}

volumeClaimTemplates:
- metadata:
name: pvc
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "gp2"
resources:
requests:
storage: 10Gi



Here again we update the configuration using an init container.


Final Note

Using clickhouse operator is much simpler than the manual mode, but in some cases specific adaptations are required that block us from using the operator. In this post we've displayed the full requirements for a minimal manual deployment of a clickhouse cluster.



Monday, March 2, 2026

Analyzing JSON and Text using Bash




In this post we will review simple methods to quickly analyze JSON and text data using bash CLIs and pipelines. While these methods might lack some of the abilities that we can find in more complex GO/JavaScript/python code, they have a great advantage of quickly grepping a piece of information we're looking for.

For this post we will use an input file containing list of HTTP requests, for example:












JSON Parsing

This first thing we want to do it to use the `jq` command to get this as a parsed JSON.

cat data.json | jq











Next, lets get only the source IPs from the file:

cat data.json | jq  '.[].SourceIp'








Count and Sort

Let's get unique count of requests per IP sorted by count.


cat data.json | jq '.[].SourceIp' | sort | uniq -c | sort -n




























Text Parsing

Let's get the methods usage in the HTTP request.

cat data.json | jq '.[].HttpRequest' 

















Now split the text by the first space and get only the first part, then remove the first character (the quotes). Then use the previous method to count per method.

cat data.json | jq '.[].HttpRequest' | awk -F' ' '{print $1}' | cut -c2-1000 | sort | uniq -c | sort -n







We can do the same to get only the HTTP path.

cat data.json | jq '.[].HttpRequest' | awk -F' ' '{print $2}' | sort | uniq -c | sort -n












Grep Area

What if we want to get transactions only for a specific IP?

We use the grep -A, -B, -C flags.

-A = get also one line after the located text
-B = get also one line before the located text
-C = get also one line before and after the located text


cat data.json | jq  | grep -B1 46.117.105.66

















Now we analyze the transactions for this IP like we've done before:

cat data.json | jq  | grep -B1 46.117.105.66 | grep HttpRequest | awk -F ' ' '{print $3}' | sort | uniq -c | sort -n










Final Note


In this post we're had a taste of the power of bash piping.
I highly recommend experimenting with these CLIs since you can do in a few seconds things that would otherwise take you much longer.


Monday, February 23, 2026

Add Grafana Alert for Restarting Pod


 


In this post we will configure an alert in grafana for restarting pods.

This relates to grafana version 11.5.1. Other version might have a different syntax.

Follow the next steps:

  • Open grafana GUI

  • Click on Alerting, Alert rules

  • Click on New alert rule

  • Enter rule name: restarting pods

  • Select prometheus as the data source

  • Make sure you have kube-state-metrics installed on the kubernetes cluster. In case it is not, install using:

    helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics

    helm repo update

    helm install kube-state-metrics kube-state-metrics/kube-state-metrics \

      --namespace kube-system \

      --create-namespace


  • Use the following PromQL query: sum by (namespace, pod) ( increase(kube_pod_container_status_restarts_total[5m]) ) > 1
  • Leave the threshold as: A > 0
  • Under Configure no data and error handling, set the Alert state if no data or all values are null to Normal
  • In the email notification message, use the following summary:

    Pod {{ $labels.pod }} restarted

  • In the email description use the following text:

    Pod {{ $labels.pod }} in namespace {{ $labels.namespace }}
    restarted more than once in the last 5 minutes.
    Current value: {{ $values.A.Value }}

  • Make sure to configure email provider such as SendGrid to enable sending emails from grafana