Full Blog TOC

Full Blog Table Of Content with Keywords Available HERE

Wednesday, January 1, 2025

Setting Up a Publicly Accessible VM with Docker, Nginx, and SSL on GCP


In this post we review the step to setup a publicly accessible web site.

The web site is based on a docker container running the famous juice-shop in a GCP based VM. 

We use Let's Encrypt to produce a valid SSL certificate for the site.


All the steps below are using "demo" prefix for the entities. Make sure to use your own suitable prefix instead.


GCP Steps

Add VPC

Login to the GCP console and open the GCP VPC network service.

Create a new VPC network named demo-vpc.
  • use IPv4
  • add a subnet 
  • add Firewall rules to allow TCP ports 22 (SSH), 80 (HTTP), 443(HTTPS)

Add VM

Open the GCP compute engine service.
Add new VM named demo-vm.
Stop the VM, and wait for the stopping to complete.
Edit the VM, and update the VM network interfaces to use the demo-vpc.
Start the VM.


Open the GCP VPC network service, and select IP addresses.
Reserve new external static IP named demo-static-ip, and assign it to the VM.

Add DNS

Open the GCP cloud domains service.
Add new domain registration named demo.com, and complete the verification process.


Open the GCP networking service, and select cloud DNS.
Click on the demo-com zone.
Add a standard A-record www.demo.com, and set the IP to the value of demo-static-ip.

Site Steps

Create Docker Compose

Open the GCP compute engine service.
Click on the demo-vm, and connect using SSH button.
Install docker on the machine, and enable non-root access.
Create docker-compose.yaml file:


version: '3'

services:
juiceshop:
image: bkimminich/juice-shop
container_name: juiceshop
environment:
- NODE_ENV=production
ports:
- "3000:3000"
restart: always

nginx:
image: nginx:latest
container_name: nginx
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
- ./data/certbot/conf:/etc/letsencrypt
- ./data/certbot/www:/var/www/certbot
ports:
- "80:80"
- "443:443"
depends_on:
- juiceshop
restart: always

certbot:
image: certbot/certbot
container_name: certbot
volumes:
- ./data/certbot/conf:/etc/letsencrypt
- ./data/certbot/www:/var/www/certbot
entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
restart: always


Create NGINX


Create nginx.conf file:



events {
use epoll;
worker_connections 128;
}

error_log /var/log/nginx.log info;

http {


server {
listen 80;
server_name www.demo.com;

location / {
proxy_pass http://juiceshop:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}

location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
}

server {
listen 443;
server_name www.demo.com;
#replace with this block later
#listen 443 ssl;
#server_name www.demo.com;
#ssl_certificate /etc/letsencrypt/live/www.demo.com/fullchain.pem;
#ssl_certificate_key /etc/letsencrypt/live/www.demo.com/privkey.pem;

location / {
proxy_pass http://juiceshop:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
}


Notice that the nginx.conf SSL section is commented out. We will revive it after issuing a valid certificate.

Create SSL Certification 


Start the containers:
docker compose up -d

Initiate a certificate request:
docker-compose exec certbot certbot certonly --webroot --webroot-path=/var/www/certbot -d www.demo.com --email your-email@demo.com --agree-tos --non-interactive


Update the nginx.conf, and revive the commented SSL section.
Restart the containers:
docker compose restart


Final Note

I have created this post since that is a common practice required for many engineer, but the information is scattered over the sites with many mistakes. 

When following these steps, make sure to validate that each step has successfully completed. For example - check connection to the public IP from a client machine, check DNS resolution, etc.


Service Extensions


I've recently worked on a project with GCP service extensions. The service extensions has 2 modes. 


Service Extensions Plugins

The first mode is service extensions plugins that are WASM based plugins that are run within the load balancer context. This is useful for stateless validation and updates to the request and  response.


The plugins can be implemented using C++ and Rust, which is ok as long as you have a small picece of code. This mode is very similar to the AWS lambda

The advantage is we do not need to manage compute resources, it is done as part of the load balancer itself, which is great.

The disadvantage is that we can have only a short small stateless piece of code. Forget about complex logic here.


Service Extensions Callouts

The second mode us service extensions callouts which is a deployment running out of the load balancer context. This callouts backend can be based for example on (VM) instances groups, and on a kubernetes cluster deployment.



The advatage is that we get a much flexible implementation choices, and we can manage state and complex logic.

The disadvantage is that we need to manage the compute resources. This is not just managing a single deployment, but managing a deploying in each region the load balancer is working.

For example if we have a global load balancer, and the callouts backend is a kubernetes based solution, we need to have a deployment in each of the load balancer regions, and since GCP supports 40 regions, we need 40(!) kubernetes clusters.


Service Extenstion Callouts Example

Following the GCP documents, we can create a simple working example of the service extension callouts for an internal regional load balancer. By the way, it is obvious why the example is for a regional load balancer, and not for a global load balancer that would require 40 deployments.


The creation is based on gcloud CLI. We start by creating a load balancer over a VMs instance group backend.

#VPC
gcloud compute networks create lb-network --subnet-mode=custom

gcloud compute networks subnets create backend-subnet \
--network=lb-network \
--range=10.1.2.0/24 \
--region=us-west1

gcloud compute networks subnets create europe-subnet \
--network=lb-network \
--range=10.3.4.0/24 \
--region=europe-west1

gcloud compute networks subnets create proxy-only-subnet \
--purpose=REGIONAL_MANAGED_PROXY \
--role=ACTIVE \
--region=us-west1 \
--network=lb-network \
--range=10.129.0.0/23


# Firewall Rules

gcloud compute firewall-rules create fw-allow-ssh \
--network=lb-network \
--action=allow \
--direction=ingress \
--target-tags=allow-ssh \
--rules=tcp:22

gcloud compute firewall-rules create fw-allow-health-check \
--network=lb-network \
--action=allow \
--direction=ingress \
--source-ranges=130.211.0.0/22,35.191.0.0/16 \
--target-tags=load-balanced-backend \
--rules=tcp

gcloud compute firewall-rules create fw-allow-proxies \
--network=lb-network \
--action=allow \
--direction=ingress \
--source-ranges=10.129.0.0/23 \
--target-tags=load-balanced-backend \
--rules=tcp:80,tcp:443,tcp:8080

# IP for the load balancer

gcloud compute addresses create l7-ilb-ip-address \
--region=us-west1 \
--subnet=backend-subnet

gcloud compute addresses describe l7-ilb-ip-address \
--region=us-west1


# managed vm instance group
gcloud compute instance-templates create l7-ilb-backend-template \
--region=us-west1 \
--network=lb-network \
--subnet=backend-subnet \
--tags=allow-ssh,load-balanced-backend \
--image-family=debian-12 \
--image-project=debian-cloud \
--metadata=startup-script='#! /bin/bash
apt-get update
apt-get install apache2 -y
a2ensite default-ssl
a2enmod ssl
vm_hostname="$(curl -H "Metadata-Flavor:Google" \
http://metadata.google.internal/computeMetadata/v1/instance/name)"
echo "Page served from: $vm_hostname" | \
tee /var/www/html/index.html
systemctl restart apache2'

gcloud compute instance-groups managed create l7-ilb-backend-example \
--zone=us-west1-a \
--size=2 \
--template=l7-ilb-backend-template


# load balancer
gcloud compute health-checks create http l7-ilb-basic-check \
--region=us-west1 \
--use-serving-port

gcloud compute backend-services create l7-ilb-backend-service \
--load-balancing-scheme=INTERNAL_MANAGED \
--protocol=HTTP \
--health-checks=l7-ilb-basic-check \
--health-checks-region=us-west1 \
--region=us-west1

gcloud compute backend-services add-backend l7-ilb-backend-service \
--balancing-mode=UTILIZATION \
--instance-group=l7-ilb-backend-example \
--instance-group-zone=us-west1-a \
--region=us-west1

gcloud compute url-maps create l7-ilb-map \
--default-service=l7-ilb-backend-service \
--region=us-west1


gcloud compute target-http-proxies create l7-ilb-proxy \
--url-map=l7-ilb-map \
--url-map-region=us-west1 \
--region=us-west1


gcloud compute forwarding-rules create l7-ilb-forwarding-rule \
--load-balancing-scheme=INTERNAL_MANAGED \
--network=lb-network \
--subnet=backend-subnet \
--address=l7-ilb-ip-address \
--ports=80 \
--region=us-west1 \
--target-http-proxy=l7-ilb-proxy \
--target-http-proxy-region=us-west1


Next we create a test VM that is used the send requests to the internal load balancer.

gcloud compute instances create l7-ilb-client-us-west1-a \
--image-family=debian-12 \
--image-project=debian-cloud \
--network=lb-network \
--subnet=backend-subnet \
--zone=us-west1-a \
--tags=allow-ssh


gcloud compute addresses describe l7-ilb-ip-address \
--region=us-west1

gcloud compute ssh l7-ilb-client-us-west1-a \
--zone=us-west1-a

# RUN FROM THE TEST VM
#curl -D - -H "host: example.com" http://IP OF LOAD BALANCER


Last we create a callout backend with a rule to handle hostname example.com.


# create callout backend

gcloud compute instances create-with-container callouts-vm \
--container-image=us-docker.pkg.dev/service-extensions/ext-proc/service-callout-basic-example-python:latest \
--network=lb-network \
--subnet=backend-subnet \
--zone=us-west1-a \
--tags=allow-ssh,load-balanced-backend

gcloud compute instance-groups unmanaged create callouts-ig \
--zone=us-west1-a

gcloud compute instance-groups unmanaged set-named-ports callouts-ig \
--named-ports=http:80,grpc:443 \
--zone=us-west1-a

gcloud compute instance-groups unmanaged add-instances callouts-ig \
--zone=us-west1-a \
--instances=callouts-vm


# update the load balancer

gcloud compute health-checks create http callouts-hc \
--region=us-west1 \
--port=80

gcloud compute backend-services create l7-ilb-callout-service \
--load-balancing-scheme=INTERNAL_MANAGED \
--protocol=HTTP2 \
--port-name=grpc \
--health-checks=callouts-hc \
--health-checks-region=us-west1 \
--region=us-west1

gcloud compute backend-services add-backend l7-ilb-callout-service \
--balancing-mode=UTILIZATION \
--instance-group=callouts-ig \
--instance-group-zone=us-west1-a \
--region=us-west1

```


# Traffic extension

```bash

cat >traffic.yaml <<EOF
name: traffic-ext
forwardingRules:
- https://www.googleapis.com/compute/v1/projects/radware-cto/regions/us-west1/forwardingRules/l7-ilb-forwarding-rule
loadBalancingScheme: INTERNAL_MANAGED
metadata: {"fr": "{forwarding_rule_id}", "key2": {"key3":"value"}}
extensionChains:
- name: "chain1"
matchCondition:
celExpression: 'request.host == "example.com"'
extensions:
- name: 'ext11'
authority: ext11.com
service: https://www.googleapis.com/compute/v1/projects/radware-cto/regions/us-west1/backendServices/l7-ilb-callout-service
failOpen: false
timeout: 0.1s
supportedEvents:
- RESPONSE_HEADERS
EOF


gcloud service-extensions lb-traffic-extensions import traffic-ext \
--source=traffic.yaml \
--location=us-west1


The callout backend is based on the example in https://github.com/GoogleCloudPlatform/service-extensions. I would recommend starting with this Go or python based example as a first step.









Tuesday, December 10, 2024

JWT Attacks




In this post we review 3 methods of JWT attacks. 

Why should we know about these attacks? TO better understand how to protected and test our servers and applications.



JSON Web Tokens are an open, industry standard RFC 7519 method for representing claims securely between two parties.

From https://jwt.io/

 


The JWT includes 3 sections:



1. The header, which contains the algorithm used for signature.

2. The claims - a JSON of specifications about the logged-in user.

3. A signature of #1 + #2


Let's review 3 types of authorization attacks based on this.


1. Omit The Authorization Header

The simplest attack: an attack tries accessing the API without authorization header. This can work if for some reason the programmer had forgotten to add authorization validation on the API implementation.

To check this, we use an existing valid request, remove the Authorization header, and resent it.

2. Self Signed JWT

JWT section #3 is the JWT signature which ensures the JWT was created by an authorized entity. In most servers framework validation of the signature can be disabled for debugging purposes. This can be done globally for the server and specificall for an API. crutial

To check this, we use an existing valid request, decode the JWT token from the Authorization header, and sign it with a random secret.

3. Use The "None" Algorithm

The application server and API implementation must restrict the signing algorithm to the only one the server is using. Failure to do this due to debuggin purposes of due to a configuration problem would enable the "None" attack.

To check this, we use an existing valid request, decode the JWT token from the Authorization header, and sign without a secret while using the None algorithm.


Final Words

We're reviewed 3 types of JWT attacks. It is crucial to add validation as part of any deployment and upgrade that our entire APIs schema is not exposed to any of these attacks.








Wednesday, December 4, 2024

Graphical Game in Go



In this post we review how to create a graphical application in Go.

The graphics is base on Ebitengine - a great game framework. For this post we create a bouncing balls animation. To use this library, first install the dependencies.

The main simply runs the game:

package main

import (
"balls/game"
"github.com/hajimehoshi/ebiten/v2"
"log"
)

func main() {
g := game.ProduceGame()
ebiten.SetWindowResizingMode(ebiten.WindowResizingModeEnabled)
ebiten.SetWindowTitle("Balls")

if err := ebiten.RunGame(g); err != nil {
log.Fatal(err)
}
}


The game class is the interaction with the ebittengine framework. It implements the Layout, Update, and Draw methods.


package game

import (
"balls/balls"
"balls/circle"
"balls/point"
"balls/rectangle"
"balls/velocity"
"github.com/hajimehoshi/ebiten/v2"
"image/color"
"math/rand/v2"
)

type Game struct {
balls []*balls.Ball
size *point.Point
bounds []*rectangle.Rectangle
}

func ProduceGame() *Game {
g := Game{
size: point.ProducePoint(1000, 1000),
}

wall := rectangle.ProduceRectangle(
&color.RGBA{
R: 40,
G: 40,
B: 40,
A: 0,
},
point.ProducePoint(0, 0),
g.size,
)
innerBox := rectangle.ProduceRectangle(
g.randomColor(),
point.ProducePoint(300, 300),
point.ProducePoint(600, 600),
)

g.bounds = []*rectangle.Rectangle{
wall,
innerBox,
}

for range 30 {
g.addBall()
}

return &g
}

func (g *Game) randomShort() uint8 {
return uint8(rand.Uint() % 255)
}

func (g *Game) randomColor() color.Color {
return &color.RGBA{
R: g.randomShort(),
G: g.randomShort(),
B: g.randomShort(),
A: 0,
}
}

func (g *Game) addBall() {
radiusLimit := rand.Float32() * g.size.GetX() / 10
radius := radiusLimit
center := point.RandomPoint(g.size.AddAxis(-2 * radius))
center = center.AddAxis(radius)

ballCircle := circle.ProduceCircle(center, radius)
newBall := balls.ProduceBall(
ballCircle,
g.randomColor(),
velocity.ProduceRandomVelocity(4+radiusLimit-radius),
)
g.balls = append(g.balls, newBall)
}

func (g *Game) Update() error {
for _, ball := range g.balls {
ball.Update(g.bounds)
}
return nil
}

func (g *Game) Draw(screen *ebiten.Image) {
for _, bound := range g.bounds {
bound.Draw(screen)
}
for _, ball := range g.balls {
ball.Draw(screen)
}
}

func (g *Game) Layout(_, _ int) (screenWidth, screenHeight int) {
return g.size.GetXTruncated(), g.size.GetYTruncated()
}


We have basic point class:

package point

import (
"fmt"
"math"
"math/rand/v2"
)

type Point struct {
x float32
y float32
}

func ProducePoint(
x float32,
y float32,
) *Point {
return &Point{
x: x,
y: y,
}
}

func RandomPoint(
limits *Point,
) *Point {
x := rand.Float32() * limits.GetX()
y := rand.Float32() * limits.GetY()
return ProducePoint(x, y)
}

func (p *Point) GetX() float32 {
return p.x
}

func (p *Point) GetY() float32 {
return p.y
}

func (p *Point) GetXTruncated() int {
return int(p.x)
}

func (p *Point) GetYTruncated() int {
return int(p.y)
}

func (p *Point) AddPoint(
other *Point,
) *Point {
return ProducePoint(p.x+other.x, p.y+other.y)
}

func (p *Point) AddAxis(
value float32,
) *Point {
return ProducePoint(p.x+value, p.y+value)
}

func (p *Point) DivideAxis(
value float32,
) *Point {
return ProducePoint(p.x/value, p.y/value)
}

func (p *Point) MultiplyAxis(
value float32,
) *Point {
return ProducePoint(p.x*value, p.y*value)
}

func (p *Point) SubtractPoint(
other *Point,
) *Point {
return ProducePoint(p.x-other.x, p.y-other.y)
}

func (p *Point) String() string {
return fmt.Sprintf("(%.2f, %.2f)", p.x, p.y)
}

func (p *Point) Distance(
other *Point,
) float32 {
delta := p.SubtractPoint(other)
return float32(math.Sqrt(float64(delta.x*delta.x + delta.y*delta.y)))
}


And a circle class:


package circle

import (
"balls/point"
"fmt"
)

type Circle struct {
center *point.Point
radius float32
}

func ProduceCircle(
center *point.Point,
radius float32,
) *Circle {
return &Circle{
center: center,
radius: radius,
}
}

func (c *Circle) GetCenter() *point.Point {
return c.center
}

func (c *Circle) GetRadius() float32 {
return c.radius
}

func (c *Circle) String() string {
return fmt.Sprintf("center %v, radius %v", c.center, c.radius)
}


The line class is more complex as it need to find intersection points:


package line

import (
"balls/circle"
"balls/kitmath"
"balls/point"
"fmt"
"math"
)

type Line struct {
start *point.Point
end *point.Point
}

func ProduceLine(
start *point.Point,
end *point.Point,
) *Line {
return &Line{
start: start,
end: end,
}
}

func (l *Line) Extend(
distance float32,
) *Line {
delta := l.end.SubtractPoint(l.start)
magnitude := math.Sqrt(float64(delta.GetX()*delta.GetX() + delta.GetY()*delta.GetY()))
normalized := delta.DivideAxis(float32(magnitude))
extendedVector := normalized.MultiplyAxis(distance)

newEnd := point.ProducePoint(
l.end.GetX()+extendedVector.GetX(),
l.end.GetY()+extendedVector.GetY(),
)

return ProduceLine(
l.start,
newEnd,
)
}

func (l *Line) GetStart() *point.Point {
return l.start

}

func (l *Line) IsVertical() bool {
return l.start.GetX() == l.end.GetX()
}

func (l *Line) FindIntersectionLine(
other *Line,
) *point.Point {
if l.IsVertical() || other.IsVertical() {
return l.findIntersectionAtLeastOneVertical(other)
}
return l.findIntersectionNonVertical(other)
}

func (l *Line) findIntersectionAtLeastOneVertical(
other *Line,
) *point.Point {
if l.IsVertical() && other.IsVertical() {
// if we have multiple crossing point - ignore
return nil
}

if l.IsVertical() {
return l.findIntersectionOnlyIVertical(other)
}
return other.findIntersectionOnlyIVertical(l)
}

func (l *Line) findIntersectionNonVertical(
other *Line,
) *point.Point {
mySlope := l.getEquationSlope()
otherSlope := other.getEquationSlope()
if kitmath.Float32Equal(mySlope, otherSlope) {
// if we have multiple crossing point - ignore
return nil
}

myB := l.getEquationB()
otherB := other.getEquationB()

x := (otherB - myB) / (mySlope - otherSlope)
y := mySlope*x + myB

intersection := point.ProducePoint(x, y)
if l.containsPoint(intersection) && other.containsPoint(intersection) {
return intersection
}

return nil
}

func (l *Line) containsY(
y float32,
) bool {
minY := min(l.start.GetY(), l.end.GetY())
maxY := max(l.start.GetY(), l.end.GetY())
return minY <= y && y <= maxY
}

func (l *Line) containsX(
x float32,
) bool {
minX := min(l.start.GetX(), l.end.GetX())
maxX := max(l.start.GetX(), l.end.GetX())
return minX <= x && x <= maxX
}

func (l *Line) getEquationSlope() float32 {
delta := l.start.SubtractPoint(l.end)
return delta.GetY() / delta.GetX()
}

func (l *Line) getEquationB() float32 {
slope := l.getEquationSlope()
return l.start.GetY() - slope*l.start.GetX()
}

func (l *Line) findIntersectionOnlyIVertical(
other *Line,
) *point.Point {
x := l.start.GetX()

if !other.containsX(x) {
return nil
}

y := other.getEquationSlope()*x + other.getEquationB()
if !l.containsY(y) {
return nil
}

return point.ProducePoint(x, y)
}

func (l *Line) containsPoint(
location *point.Point,
) bool {
return l.containsX(location.GetX()) && l.containsY(location.GetY())
}

func (l *Line) FindIntersectionCircle(
circle *circle.Circle,
) []*point.Point {
var limitedIntersections []*point.Point
intersections := l.findIntersectionCircleUnlimited(circle)
for _, intersection := range intersections {
if !l.containsPoint(intersection) {
continue
}
limitedIntersections = append(limitedIntersections, intersection)
}

return limitedIntersections
}

func (l *Line) findIntersectionCircleUnlimited(
circle *circle.Circle,
) []*point.Point {
if l.IsVertical() {
return l.findIntersectionCircleVertical(circle)
}
return l.findIntersectionCircleHorizontal(circle)
}

func (l *Line) findIntersectionCircleVertical(
circle *circle.Circle,
) []*point.Point {
c := l.start.GetX()
x0 := circle.GetCenter().GetX()
y0 := circle.GetCenter().GetY()
r := circle.GetRadius()

inner := r*r - (c-x0)*(c-x0)
if inner < 0 {
return nil
}

if inner == 0 {
return []*point.Point{
point.ProducePoint(c, y0),
}
}

innerSqrt := float32(math.Sqrt(float64(inner)))
return []*point.Point{
point.ProducePoint(c, y0+innerSqrt),
point.ProducePoint(c, y0-innerSqrt),
}
}

func (l *Line) findIntersectionCircleHorizontal(
circle *circle.Circle,
) []*point.Point {
c := l.start.GetY()
x0 := circle.GetCenter().GetX()
y0 := circle.GetCenter().GetY()
r := circle.GetRadius()

inner := r*r - (c-y0)*(c-y0)
if inner < 0 {
return nil
}

if inner == 0 {
return []*point.Point{
point.ProducePoint(x0, c),
}
}

innerSqrt := float32(math.Sqrt(float64(inner)))
return []*point.Point{
point.ProducePoint(x0+innerSqrt, c),
point.ProducePoint(x0-innerSqrt, c),
}
}

func (l *Line) String() string {
return fmt.Sprintf("start %v end %v", l.start, l.end)
}


The rectangle class is used to represent both the screen edges, and obstacles.


package rectangle

import (
"balls/circle"
"balls/line"
"balls/point"
"github.com/hajimehoshi/ebiten/v2"
"github.com/hajimehoshi/ebiten/v2/vector"
"image/color"
)

type Rectangle struct {
color color.Color
topLeft *point.Point
bottomRight *point.Point
}

func ProduceRectangle(
color color.Color,
topLeft *point.Point,
bottomRight *point.Point,
) *Rectangle {
return &Rectangle{
color: color,
topLeft: topLeft,
bottomRight: bottomRight,
}
}

func (r *Rectangle) getLines() []*line.Line {
topRight := point.ProducePoint(r.bottomRight.GetX(), r.topLeft.GetY())
bottomLeft := point.ProducePoint(r.topLeft.GetX(), r.bottomRight.GetY())

return []*line.Line{
line.ProduceLine(r.topLeft, topRight),
line.ProduceLine(topRight, r.bottomRight),
line.ProduceLine(r.bottomRight, bottomLeft),
line.ProduceLine(bottomLeft, r.topLeft),
}
}

func (r *Rectangle) FindIntersectionCircle(
circle *circle.Circle,
) (*point.Point, *line.Line) {
var minDistance float32
var intersectionLine *line.Line
var intersectionPoint *point.Point
for _, lineBound := range r.getLines() {
boundIntersections := lineBound.FindIntersectionCircle(circle)
for _, boundIntersection := range boundIntersections {
distance := circle.GetCenter().Distance(boundIntersection)
if intersectionLine == nil || distance < minDistance {
minDistance = distance
intersectionLine = lineBound
intersectionPoint = boundIntersection
}
}
}
return intersectionPoint, intersectionLine
}

func (r *Rectangle) FindIntersectionLine(
moveLine *line.Line,
) (*point.Point, *line.Line) {
var minDistance float32
var intersectionLine *line.Line
var intersectionPoint *point.Point
for _, lineBound := range r.getLines() {
boundIntersection := lineBound.FindIntersectionLine(moveLine)
if boundIntersection == nil {
continue
}
distance := moveLine.GetStart().Distance(boundIntersection)
if intersectionLine == nil || distance < minDistance {
minDistance = distance
intersectionLine = lineBound
intersectionPoint = boundIntersection
}
}
return intersectionPoint, intersectionLine
}

func (r *Rectangle) Draw(screen *ebiten.Image) {
delta := r.bottomRight.SubtractPoint(r.topLeft)
vector.DrawFilledRect(
screen,
r.topLeft.GetX(),
r.topLeft.GetY(),
delta.GetX(),
delta.GetY(),
r.color,
false,
)
}



The velocity class is:


package velocity

import (
"balls/line"
"balls/point"
"fmt"
"math"
"math/rand/v2"
)

type Velocity struct {
speed float32
angleRadian float32
}

func ProduceVelocity(
speed float32,
angle float32,
) *Velocity {
return &Velocity{
speed: speed,
angleRadian: angle,
}
}
func ProduceRandomVelocity(
speedLimit float32,
) *Velocity {
speed := rand.Float32() * speedLimit
angle := rand.Float32() * 2 * math.Pi
return ProduceVelocity(speed, angle)
}

func (v *Velocity) Apply(
location *point.Point,
) *point.Point {
x := location.GetX() + v.speed*float32(math.Cos(float64(v.angleRadian)))
y := location.GetY() + v.speed*float32(math.Sin(float64(v.angleRadian)))
return point.ProducePoint(x, y)
}

func (v *Velocity) isLeft() bool {
return v.angleRadian > math.Pi/2 && v.angleRadian < 1.5*math.Pi
}

func (v *Velocity) isRight() bool {
return v.angleRadian < math.Pi/2 || v.angleRadian > 1.5*math.Pi
}

func (v *Velocity) Bounce(
line *line.Line,
) *Velocity {
if line.IsVertical() {
return v.bounceVertical()
}
return v.bounceHorizontal()
}

func (v *Velocity) bounceVertical() *Velocity {
if v.angleRadian < math.Pi {
return ProduceVelocity(v.speed, math.Pi-v.angleRadian)
}
return ProduceVelocity(v.speed, 3*math.Pi-v.angleRadian)
}

func (v *Velocity) bounceHorizontal() *Velocity {
return ProduceVelocity(v.speed, 2*math.Pi-v.angleRadian)
}

func (v *Velocity) String() string {
return fmt.Sprintf("speed %.2f angle %.2f", v.speed, v.angleRadian)
}


And the ball class represents a ball with circle an velocity. It handles it own movement.


package balls

import (
"balls/circle"
"balls/line"
"balls/rectangle"
"balls/velocity"
"fmt"
"github.com/hajimehoshi/ebiten/v2"
"github.com/hajimehoshi/ebiten/v2/vector"
"image/color"
)

type Ball struct {
circle *circle.Circle
color color.Color
velocity *velocity.Velocity
}

func ProduceBall(
circle *circle.Circle,
color color.Color,
velocity *velocity.Velocity,
) *Ball {
return &Ball{
circle: circle,
color: color,
velocity: velocity,
}
}

func (b *Ball) Update(
bounds []*rectangle.Rectangle,
) {
newCenter := b.velocity.Apply(b.circle.GetCenter())
newCircle := circle.ProduceCircle(newCenter, b.circle.GetRadius())

for _, bound := range bounds {
currentIntersection, _ := bound.FindIntersectionCircle(b.circle)
if currentIntersection != nil {
// while we are intersecting, we do not bounce
b.circle = circle.ProduceCircle(newCenter, b.circle.GetRadius())
return
}
}

var minDistance *float32
var bouncedLine *line.Line
for _, bound := range bounds {
intersectionPoint, intersectionLine := bound.FindIntersectionCircle(newCircle)
if intersectionPoint == nil {
continue
}

distance := b.circle.GetCenter().Distance(intersectionPoint)
if minDistance == nil || distance < *minDistance {
minDistance = &distance
bouncedLine = intersectionLine
}
}

if minDistance != nil {
b.velocity = b.velocity.Bounce(bouncedLine)
newCenter = b.velocity.Apply(b.circle.GetCenter())
}

b.circle = circle.ProduceCircle(newCenter, b.circle.GetRadius())
}

func (b *Ball) Draw(screen *ebiten.Image) {
vector.DrawFilledCircle(
screen,
b.circle.GetCenter().GetX(),
b.circle.GetCenter().GetY(),
b.circle.GetRadius(),
b.color,
false,
)
}

func (b *Ball) String() string {
return fmt.Sprintf("%v velocity %v color %v", b.circle, b.velocity, b.color)
}


We also have a math helper class:

package kitmath

var grace = float32(0.001)

func Float32Equal(
f1 float32,
f2 float32,
) bool {
delta := f1 - f2
return -grace <= delta && delta <= grace
}



Wednesday, November 27, 2024

Redpanda Connect Introduction

 



Redpanda Connect, previously known as Benthos, is a streaming pipeline that reads and write messages from/to many connectors. It enables transforming the messages using built-in processors. The goal of this framework is to enable us connect and convert data streams without writing a proprietary code, but only using a configuration file with minimal dedicated processing code. 


To install use the following:

curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
unzip rpk-linux-amd64.zip
sudo mv rpk /usr/local/bin/
rm rpk-linux-amd64.zip


Create a file named connect.yaml:


input:
stdin: {}

pipeline:
processors:
- mapping: root = content().uppercase()

output:
stdout: {}


And run it:

rpk connect run connect.yaml


Now any input to the STDIN is copied to the STDOUT.


While Redpanda connect can manage string messages, most f its abilities are built toward JSON messages handling. For example, consider the following connect.yaml file:


input:
generate:
interval: 1s
count: 0
mapping: |
let first_name = fake("first_name")
let last_name = fake("last_name")

root.id = counter()
root.name = ($first_name + " " + $last_name)
root.timestamp = now()

pipeline:
processors:
- sleep:
duration: 100ms
- group_by:
- check: this.id % 2 == 0
processors:
- mapping: |
root.original_doc = this
root.encoded = this.name.hash("sha256").encode("base64")

output:
stdout: {}


It will generate the following output:

{"id":1,"name":"Ivah Mohr","timestamp":"2024-11-27T16:36:37.542410543+02:00"}
{"encoded":"oaaBKF/7oz0N6j6VlSZs14u8FD2dAwPSBAoJvIIMpWI=","original_doc":{"id":2,"name":"Darrion Miller","timestamp":"2024-11-27T16:36:38.541429372+02:00"}}
{"id":3,"name":"Maddison Paucek","timestamp":"2024-11-27T16:36:39.542047832+02:00"}
{"encoded":"+e6EyBEILJYStdV+DH6cohEVvfW04VTo1q2YLXp4ft8=","original_doc":{"id":4,"name":"Eleazar Sporer","timestamp":"2024-11-27T16:36:40.542410537+02:00"}}
{"id":5,"name":"Junius Renner","timestamp":"2024-11-27T16:36:41.542298415+02:00"}
{"encoded":"A4o97ySPf9yWFMXcutPBiI6a6Fd19ofqBmK/7s84ZZ4=","original_doc":{"id":6,"name":"Carlie Osinski","timestamp":"2024-11-27T16:36:42.542277315+02:00"}}
{"id":7,"name":"Raphaelle Reichel","timestamp":"2024-11-27T16:36:43.542382662+02:00"}
{"encoded":"U+pLsYjRaWH87nUFdZmRLJwvGrIfUOCYeUrazKGoEHA=","original_doc":{"id":8,"name":"Elmira Douglas","timestamp":"2024-11-27T16:36:44.542245761+02:00"}}
{"id":9,"name":"Ambrose Hudson","timestamp":"2024-11-27T16:36:45.542307678+02:00"}
{"encoded":"J2NxrIaC1cvHtqIduSOx85TlyFLQrT48QaYw8iR9To0=","original_doc":{"id":10,"name":"Jedidiah Veum","timestamp":"2024-11-27T16:36:46.542276779+02:00"}}



Final Words

Yeah, that's cool: You save time coding and fixing bugs. This is nice if you need a simple fast and reliable pipeline.
However, in real life you will usually require more the the builtin abilities, and then you need to extend Redpanda Connect with your own code in GO. Is it better than just writing all in GO? I think it depends on how much bugs do you expect from the programmer.