Full Blog TOC

Full Blog Table Of Content with Keywords Available HERE

Thursday, May 21, 2020

Local Outlier Factor - A simple GO implementation



In this post we will review the LOF: Local Outlier Factor algorithm.
For full source of the example, see my GO sources at https://github.com/alonana/lof
An example of using the LOF is available in this post.

The LOF is based on several terms:
  • reachability distance
  • LRD
  • LOF
To implement LOF, we'll first implement K-Nearest-Neighbors detection.
Once we have the list of k nearest neighbors for each point, we can run the LOF algorithm.

The LOF provides score for each point.
A LOF score for an outlier would be higher 1.
How high?
It depends on you data, but the more clear outlier, the higher the score would be.


The LOF calculation is based on the following pseudo code:

func LOF(p) {
  sum=0
  foreach neighbor in neighbors(point) {
     sum += LRD(neighbor)
  }
  return sum / (k * LRD(p))
}

The LRD calculation is based on the following pseudo code:

func LRD(p) {
  sum=0
  foreach neighbor in neighbors(point) {
     sum += reachabilityDistance(p,neighbor)
  }
  return k / sum
}


The reachability distance calculation is based on the following pseudo code:

func reachabilityDistance(a, b) {
  return max(Distance(a,b), kDistance(b))
}

The k-distance is based on the following pseudo code:

func kDistance(p) {
  highest=0
  foreach neighbor in neighbors(point) {
     highest = max(highest, Distance(p,neighbor))
  }
  return highest
}

Again, notice that the "neighbors" used in the pseudo code, returns only the k-nearest points, and not all the neighbors.



Monday, May 18, 2020

How to cleanup Google Cloud Container Registry Images





Using Google Cloud Container Registry is pretty easy. 

But as time go by, you get more and more images accumulating in the container registry.
To delete the images, you can manually delete each image, which is OK for a single image removal, but frustrating if you want to remove multiple images.

I have created a short script to remove images recursively from a specific container registry folder.


#!/usr/bin/env bash
set -e

REPO=gcr.io/MY_PROJECT/MY_GCR_FOLDER


deleteTag() {
NAME=$1
HASH=$2
echo "delete hash ${NAME} ${HASH}"
gcloud container images delete -q --force-delete-tags ${NAME}@${HASH}
}

deleteTags() {
NAME=$1
echo "scan tags for ${NAME}"
gcloud container images list-tags ${NAME} --format='get(digest)' | while read line; do deleteTag $NAME $line; done
}

deleteImages() {
NAME=$1
echo "scan images for ${NAME}"
gcloud container images list --repository=${NAME} | grep -v ^NAME | while read line; do deleteImages $line; done
deleteTags ${NAME}
}

deleteImages ${REPO}



To run the script, supply the following as an argument: gcr.io/MY_PROJECT/MY_GCR_FOLDER.
The script will recursively delete the folder and files under this folder.





Update: keep the latest version


To keep the latest version for each image, change the following functions:


deleteTag() {
LATEST=$1
NAME=$2
HASH=$3
TAG=$4

if [[ "${LATEST}" == "${TAG}" || "latest" == "${TAG}" ]]; then
echo "skip delete of ${NAME} ${TAG}"
return 0
fi

echo "delete hash ${NAME} ${TAG} ${HASH}"
gcloud container images delete -q --force-delete-tags ${NAME}@${HASH}
}

deleteTags() {
NAME=$1
echo "scan tags for ${NAME}"
LATEST=$(gcloud container images list-tags ${NAME} --format='get(tags)' | sort -n | tail -1)
echo "keeping latest: ${LATEST}"
gcloud container images list-tags ${NAME} --format='get(digest,tags)' | while read line; do deleteTag $LATEST $NAME $line; done
}






Thursday, May 14, 2020

Access Google Cloud BigQuery from GO



In this post we will review how to access Google Cloud BigQuery from a GOlang Application.

"Serverless, highly scalable, and cost-effective cloud data warehouse designed for business agility."

In simple words, BigQuery enables use to save huge amount of data in a relational DBMS, and access it using plain SQL language, enriched with some of BigQuery proprietary functions.


Access Key


Now that we have data in BigQuery, we'll probably want to process it.
To access the BigQuery, we first need to create an access key.

To create an access key, login to Google Cloud Platform console, and select IAM and Admin, Service Accounts. Then select the account, and using the menu, select Create key, and export to a JSON format.







Let's save the file in path key.json.

Notice:
The selected service account should be granted with permissions to access the BigQuery.


The GO Application


Now, we can use the key.json file to access BigQuery.


package main

import (
"cloud.google.com/go/bigquery"
"context"
"fmt"
"google.golang.org/api/iterator"
"os"
)

func main() {
projectId := "YOUR_GCP_PROJECT_NAME"

_ = os.Setenv("GOOGLE_APPLICATION_CREDENTIALS", "key.json")
bigQueryClient, err := bigquery.NewClient(context.Background(), projectId)
if err != nil {
panic(err)
}

sql := "select col1,col2 from YOUR_DATASET_NAME.YOUR_TABLE_NAME limit 100"
query := bigQueryClient.Query(sql)
result, err := query.Read(context.Background())
if err != nil {
panic(err)
}

for {
var row []bigquery.Value
err := result.Next(&row)
if err == iterator.Done {
return
}
if err != nil {
panic(err)
}

stringColumn := row[2].(string)
intColumn := row[3].(int64)

fmt.Println(stringColumn, intColumn)
}
}


Notice that the path to the key.json is supplied as an environment variable.
In addition, we need to specify our project ID (even though it is already specified in the key.json), and the SQL text that we want to run.


And that's it, very simple.




Tuesday, May 5, 2020

Cloud Migration: Move Your Application to Google Kubernetes Engine



Lately I had to move an application running on a local kubernetes cluster to run on Google Kubernetes Engine(GKE), which is a part of the Google Cloud Platform (GCP). 


In this post I will review the steps done to get things working:

  1. Install Google Cloud SDK
  2. Create a new kubernetes cluster on GKE
  3. Enable a local kubectl to access the kubernetes cluster on GKE
  4. Upload the images to Google Cloud container registry
  5. Adjust the kubernetes templates to use GKE's persistence disks


1. Install Google Cloud SDK


The first step is to install the gcloud CLI, which is the Google cloud SDK.
Google cloud SDK is required to create, and update the various GCP entities, for example: login to GCP, create a kubernetes cluster, configure docker to connect to the GCP registry, and much more.

Specifically, for Ubuntu, follow the instructions of: Install Google Cloud SDK using apt-get.
The summary of these instructions is below.

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get install apt-transport-https ca-certificates gnupg
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt-get update && sudo apt-get install google-cloud-sdk
gcloud init


2. Create a new kubernetes cluster on GKE


To create a new kubernetes cluster, I've simply used the GCP web console.
  • Login to the GCP web console using your personal user or your work related user.
  • Click the Menu on the top left, and select: Kubernetes Engine, Clusters.
  • Click on Create Cluster, and change any settings you need (I've used all of the defaults)
Wait several minutes, and your kubernetes cluster is ready.

Later I've found that the cluster was configured to use 3 machines, that each has a single CPU.
For most applications, this is not enough, so I've updated the cluster to use 3 machines of 8 CPUs, using the gcloud SDK:

gcloud container node-pools create MY_NEW_POOL --cluster=MY_K8S_CLUSTER --num-nodes=3 --machine-type=n1-standard-8


3. Enable a local kubectl to access the kubernetes cluster on GKE


Now, our kubernetes cluster is ready, but how can we use the kubectl CLI to access it?

The first method which is simple, but less convenient in my option, is to use the kubectl from the GCP web console.
  • Login to the GCP web console using your personal user or your work related user.
  • Click the Menu on the top left, and select: Kubernetes Engine, Clusters.
  • On the clusters table, click on the connect button on the right side of your cluster.
  • That's it, you have a SSH session and a configured kubectl ready for your use

The second method requires some more steps, but in the long run, is easier to use. It is based on the Configuring cluster access for kubectl guide.
First, enable kubernetes API:
  • Login to the GCP web console using your personal user or your work related user.
  • Click the Menu on the top left, and select: API & services -> Enable API & services -> kubernetes API 

Next, update the local kubectl configuration (at ~/.kube/config) using the gcloud CLI:


gcloud container clusters get-credentials MY_K8S_CLUSTER


As a side note, when working with multiple kubernetes cluster, you should be aware of the kubectl contexts.

kubectl context   =  kubernetes Cluster   +   kubernetes Namespace   +   kubernetes User

Use the following commands to list, view, and update the current kubectl context:

kubectl config get-contexts                          # display list of contexts 
kubectl config current-context                       # display the current-context
kubectl config use-context my-cluster-name           # set the default context to my-cluster-name


4. Upload the images to Google Cloud container registry


OK, your cluster is up and running, and you can access it. But how can you access your images?

If you already have a public accessible container registry, great! You can skip this step.

Otherwise, you can use GCP container registry.

First, enable the container registry API:
  • Login to the GCP web console using your personal user or your work related user.
  • Click the Menu on the top left, and select: API & services -> Enable API & services -> container registry API 
Next, login to the machine where the docker images resides, and run the following:

gcloud auth login
gcloud auth configure-docker

This enables your local docker to access the GCP container registry.

Finally, to upload a docker image, tag it using the GCP prefix, and push it:

IMAGE_FULL_ID=MY_IMAGES_FOLDER/MY_IMAGE_NAME
GCR_TAG=gcr.io/MY_GCP_PROJECT_NAME/${IMAGE_FULL_ID}
docker tag MY_LOCAL_REGISTRY_SERVER:MY_LOCAL_REGISTRY_PORT/${IMAGE_FULL_ID} ${GCR_TAG}
docker push ${GCR_TAG}


5. Adjust the kubernetes templates to use GKE's persistence disks


In case the application has persistence volume claims, you should update it to use GCP's persistence instead,
This is done by dropping the storage class name from the persistence volume claims.

For example, remove the red line here:

volumeClaimTemplates:
  - metadata:
      name: persist-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "hostPath"
      resources:
       
requests:
         
storage: "1g"

Summary


GCP is a great platform, allowing quick implementation and deployment of applications.
In this post we have reviewed move of a single application to GKE.
Once the application is located in GKE, it can also easily use additional GCP services, such as BigQuery, Pub/Sub, and AI.



Thursday, April 30, 2020

How to Create a Maven Plugin




In a recent project I've had to create a maven plugin.
The maven plugin goal was to analyze a file, and set a maven property based on the file content.
In this post, we will review the step to create such a maven plugin, and to use it.


Creating the Maven Plugin


To create the maven plugin, we create a new maven project with 2 files:

  • pom.xml - configuring the maven plugin build
  • MyPlugin.java - doing the actual plugin work

The pom.xml is pretty simple, we use maven-plugin-plugin (a very creative name...) to build a new plugin.


<project>
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.demo</groupId>
    <artifactId>analyze-file</artifactId>
    <packaging>maven-plugin</packaging>



    <dependencies>
        <dependency>
            <groupId>org.apache.maven</groupId>
            <artifactId>maven-plugin-api</artifactId>
            <version>3.6.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.maven.plugin-tools</groupId>
            <artifactId>maven-plugin-annotations</artifactId>
            <version>3.6.0</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.maven</groupId>
            <artifactId>maven-project</artifactId>
            <version>2.2.1</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-plugin-plugin</artifactId>
                <version>3.6.0</version>
                <configuration>
                    <skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
                </configuration>
                <executions>
                    <execution>
                        <id>mojo-descriptor</id>
                        <goals>
                            <goal>descriptor</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>


The MyPlugin.java does the actual work.
It analyzes a file, and sets a maven property based on the analysis.
Both the input file name, and the output property are configurable in the plugin run. We will later see this while presenting the usage of the plugin.
We will use the @Parameter annotation to specify a configurable parameter.


package com.dfc.maven.plugin.parsejar;


import org.apache.maven.plugin.AbstractMojo;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.maven.plugins.annotations.Component;
import org.apache.maven.plugins.annotations.Mojo;
import org.apache.maven.plugins.annotations.Parameter;
import org.apache.maven.project.MavenProject;

import java.io.File;

@Mojo(name = "parse")
public class MyPlugin extends AbstractMojo {

    @Component
    private MavenProject project;

    @Parameter
    private String inputFile;

    @Parameter
    private String toProperty;

    public void execute() throws MojoExecutionException {
        try {
            doWork();
        } catch (Throwable e) {
            e.printStackTrace();
            throw new MojoExecutionException("analyze " + inputFile + " failed", e);
        }
    }

    private void doWork() throws Exception {
        File file = new File(inputFile);
        getLog().info("parsing file: " + file.getAbsolutePath());

        // ... analyze the file, and set the property value
        String value = ...;

        project.getProperties().put(toProperty, value);
    }
}


Using the Maven Plugin


To use the plugin, we add it as part of another maven project's pom.xml, for example:


<build>
    <plugins>
        <plugin>
            <artifactId>analyze-file</artifactId>
            <groupId>com.demo</groupId>
            <executions>
                <execution>
                    <goals>
                        <goal>parse</goal>
                    </goals>
                    <configuration>
                        <inputFile>${basedir}/my-input-file.txt</inputFile>
                        <toProperty>analyze-output</toProperty>
                    </configuration>
                </execution>
            </executions>
        </plugin>


The configurable properties are listed as part of the configuration element of the plugin usage.


Summary


In this post we have create a configurable maven plugin, which can be reused among other maven project. By creation of our own maven plugin, we can extend the maven build to match our specialized requirements.

Friday, April 24, 2020

Use AWS S3 to Update Server Files


This post is about a nice trick that a colleague of mine had found.

We have a server running on a production data center, and so the access to this server is through several security layers. We run some POC test code on this server, and had to update the application binary several times. As this was a POC stage project, we had no CI/CD process to automate deployment of the binary, and we've had to upload the code manually. But uploading the code was extremely slow and complicated, due to the multiple security layers.

The solution we've finally used was to upload the application binary to a AWS S3 bucket, and use a python code to download it on the production server.

The first step is to create an AWS S3 bucket.
In his case I've create a bucket named binary-bucket.



Then, upload the binary to this bucket.



To retrieve the application binary on the production server, we' have use a short python script.
Notice that due to the security tiers, we could not directly access the AWS S3, so we've used a proxy instead.

import boto3
from botocore.config import Config

BUCKET_NAME = 'binary-bucket'
BINARY_FILE = 'app.bin'
PROXY = '10.1.1.1:443'

s3 = boto3.client('s3', config=Config(proxies={'https': PROXY}))
obj = s3.get_object(Bucket=BUCKET_NAME, Key=BINARY_FILE)
data = obj['Body'].read()
with open(BINARY_FILE, 'wb') as code:
    code.write(data)

The last thing to do, is to use an AWS access key for the AWS authentication.
This is done by creating of a configuration file inthe home folder: ~/.aws/config


[default]
aws_access_key_id=A12DEHAA72B4PAAATJRA
aws_secret_access_key=TelYU6M3Dfg/ssl3PWJAPSwXg/rJw9kD44Rdd7sq

and that's it. The binary is downloaded within seconds.


Update: Upload to AWS S3

After a week of work, I've found that another useful task is to retrieve files back from the production server. So I've added an upload script.

import boto3
from botocore.config import Config

BUCKET_NAME = 'binary-bucket'
UPLOAD_FILE = 'app.data'
PROXY = '10.1.1.1:443'

s3 = boto3.client('s3', config=Config(proxies={'https': PROXY}))
s3.upload_file(UPLOAD_FILE, BUCKET_NAME, UPLOAD_FILE)


and we're done.

Thursday, April 16, 2020

Capture HTTP Transactions into HAR file

This week I've created a utility to capture HTTP (clear text) transactions into a HAR file.
See more details in the repository: https://github.com/alonana/httshark

This is useful in case you want to view transactions that are sent to a none secured site.
Another common usage is after an SSL terminator, as illustrated below: