Full Blog TOC

Full Blog Table Of Content with Keywords Available HERE

Wednesday, March 31, 2021

Capture XMLHttpRequest calls


 


In this post we will review how to capture the JavaScript XMLHttpRequest API calls. This is required if you want to modify all the requests sent by the XMLHttpRequest object in the HTML page (e.g. add headers), or just log something from the requests.


The XMLHttpRequest includes open method, and a send method. We start by keeping references to the original implementations:


const origOpen = window.XMLHttpRequest.prototype.open
const origSend = window.XMLHttpRequest.prototype.send


Next, we update the open implementation, to keep the url that is being accessed:


XMLHttpRequest.prototype.open = function (method, url, async, user, password) {
this._url = url
origOpen.apply(this, arguments)
}


Once the send method is called, we can log the URL and the body. We can also add headers in this step.


XMLHttpRequest.prototype.send = function () {
if (this.onreadystatechange) {
this._onreadystatechange = this.onreadystatechange
}

this.onreadystatechange = onReadyStateChangeReplacement

console.log(`sending url ${this._url} body:`, arguments[0])
this.setRequestHeader('my-header', 'my-value')

actualSend()

async function actualSend() {
origSend.apply(this, arguments)
}

function onReadyStateChangeReplacement() {
if (this._onreadystatechange) {
return this._onreadystatechange.apply(this, arguments)
}
}
}


Notice the special handling for the onreadystatechange, in goal to use the correct "this" object.


Once this is done, any use of XMLHttpObject in the page will print to console log the url and the body, and also will add a custom header to the request.







Tuesday, March 23, 2021

Create AWS WebACL for CloudFront using AWS SDK v2

 


 
In this post we will use AWS GO API V2 to create a WebACL, and then we will associate it with a CloudFront distribution.

Before starting, make sure to setup credentials and region as specified in this post.


We will use the AWS SDK version 2, so the imports should be:



import (
"context"
"fmt"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/cloudfront"
"github.com/aws/aws-sdk-go-v2/service/wafv2"
"github.com/aws/aws-sdk-go-v2/service/wafv2/types"
)

const aclName = "my-webacl"

type Acl struct {
}



Start by setting a connection to AWS:



awsConfig, err := config.LoadDefaultConfig(context.Background(), config.WithRegion("us-east-1"))
if err != nil {
panic(err)
}



Next, create the WebACL. We create an example of WebACL rule to block any request with query string that have a specific suffix.



func (a *Acl) createWebAcl(wafClient *wafv2.Client) string {
statement := types.Statement{
ByteMatchStatement: &types.ByteMatchStatement{
FieldToMatch: &types.FieldToMatch{
QueryString: &types.QueryString{},
},
PositionalConstraint: "ENDS_WITH",
SearchString: []byte("/3"),
TextTransformations: []types.TextTransformation{
{
Priority: 1,
Type: "NONE",
},
},
},
}

rule := types.Rule{
Name: aws.String("rule-1"),
Priority: 1,
Action: &types.RuleAction{
Block: &types.BlockAction{},
},
Statement: &statement,
VisibilityConfig: &types.VisibilityConfig{
CloudWatchMetricsEnabled: false,
SampledRequestsEnabled: false,
MetricName: aws.String("my-rule-metric"),
},
}

aclInput := wafv2.CreateWebACLInput{
DefaultAction: &types.DefaultAction{
Block: &types.BlockAction{},
},
Name: aws.String(aclName),
Rules: []types.Rule{rule},
Scope: "CLOUDFRONT",
VisibilityConfig: &types.VisibilityConfig{
CloudWatchMetricsEnabled: false,
SampledRequestsEnabled: false,
MetricName: aws.String("my-rule-metric"),
},
}

acl, err := wafClient.CreateWebACL(context.Background(), &aclInput)
if err != nil {
panic(err)
}

fmt.Printf("acl output %+v\n", *acl)

return * acl.Summary.ARN
}



Next we can update the CloudFront distribution to use this WebACL.

Notice that you cannot use the WebACL API to associate the WebACL to the CloudFront distribution (why? ask AWS team. I guess they did not want to make it easy).



func (a *Acl) updateDistribution(awsConfig *aws.Config, aclId string) {
getConfigInput := cloudfront.GetDistributionConfigInput{
Id: aws.String("E1Y55CUPVONMHF"),
}
cloudFrontClient := cloudfront.NewFromConfig(*awsConfig)

distributionConfigOutput, err := cloudFrontClient.GetDistributionConfig(context.Background(), &getConfigInput)
if err != nil {
panic(err)
}

fmt.Printf("distribution acl: %v\n", *distributionConfigOutput.DistributionConfig.WebACLId)
distributionInput := cloudfront.UpdateDistributionInput{
DistributionConfig: distributionConfigOutput.DistributionConfig,
Id: aws.String("E1Y55CUPVONMHF"),
IfMatch: distributionConfigOutput.ETag,
}
distributionInput.DistributionConfig.WebACLId = aws.String(aclId)

_, err = cloudFrontClient.UpdateDistribution(context.Background(), &distributionInput)
if err != nil {
panic(err)
}
}



That's it, we have a WebACL assigned to our CloudFront distribution.



Final Notes


I've added this post, since there are just no good examples of how to do this.

I hope you will find this useful.

Monday, March 22, 2021

Create AWS WebACL for CloudFront


 



NOTICE:
New AWS SDK was published.
You better use it instead of the one specified in the post.
For more details, see here.




In this post we will use AWS Go API to create a WebACL, and then we will associate it with a CloudFront distribution.

Before starting, make sure to setup credentials and region as specified in this post.


We will start by creating an example of WebACL rule to block any request with query string that have a specific suffix.


awsSession, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
})
wafClient := wafv2.New(awsSession)
if err != nil {
panic(err)
}

statement := wafv2.Statement{
ByteMatchStatement: &wafv2.ByteMatchStatement{
FieldToMatch: &wafv2.FieldToMatch{
QueryString: &wafv2.QueryString{},
},
PositionalConstraint: aws.String("ENDS_WITH"),
SearchString: []byte("/3"),
TextTransformations: []*wafv2.TextTransformation{
{
Priority: aws.Int64(1),
Type: aws.String("NONE"),
},
},
},
}

rule := wafv2.Rule{
Name: aws.String("rule-1"),
Priority: aws.Int64(1),
Action: &wafv2.RuleAction{
Block: &wafv2.BlockAction{},
},
Statement: &statement,
VisibilityConfig: &wafv2.VisibilityConfig{
CloudWatchMetricsEnabled: aws.Bool(false),
SampledRequestsEnabled: aws.Bool(false),
MetricName: aws.String("my-rule-metric"),
},
}

aclInput := wafv2.CreateWebACLInput{
DefaultAction: &wafv2.DefaultAction{
Block: &wafv2.BlockAction{},
},
Name: aws.String("my-webacl"),
Rules: []*wafv2.Rule{&rule},
Scope: aws.String("CLOUDFRONT"),
VisibilityConfig: &wafv2.VisibilityConfig{
CloudWatchMetricsEnabled: aws.Bool(false),
SampledRequestsEnabled: aws.Bool(false),
MetricName: aws.String("my-rule-metric"),
},
}

acl, err := wafClient.CreateWebACL(&aclInput)
if err != nil {
panic(err)
}


Next we can update the CloudFront distribution to use this WebACL.

Notice that you cannot use the WebACL API to associate the WebACL to the CloudFront distribution (why? ask AWS team. I guess they did not want to make it easy).


getConfigInput := cloudfront.GetDistributionConfigInput{
Id: aws.String("E1Y55CUPVONMHF"),
}
cloudFrontClient := cloudfront.New(awsSession)

distributionConfigOutput, err := cloudFrontClient.GetDistributionConfig(&getConfigInput)
if err != nil {
panic(err)
}

distributionInput := cloudfront.UpdateDistributionInput{
DistributionConfig: distributionConfigOutput.DistributionConfig,
Id: aws.String("E1Y55CUPVONMHF"),
IfMatch: distributionConfigOutput.ETag,
}
distributionInput.DistributionConfig.WebACLId = acl.Summary.Id

_, err = cloudFrontClient.UpdateDistribution(&distributionInput)
if err != nil {
panic(err)
}



That's it, we have a WebACL assigned to our CloudFront distribution.



Final Notes


I've added this post, since there are just no good examples of how to do this.

I hope you will find this useful.






Tuesday, March 16, 2021

Using Ingress in AWS EKS


 

In this post we will review the steps required to use an ingress an a kubernetes cluster deployed on AWS EKS. The official document for the procedure is very long and frustrating. However I believe that AWS team would make it easier in the future. Until this is done, you can use the steps I've listed below.

To support ingress, AWS actually deploys an application load balancer to handle the ingress traffic dispatching. This is automated by deploying a load balancer controller as a pod in your existing EKS cluster. So, to use an ingress in EKS you need to:

  1. Deploy a load balancer controller
  2. Deploy ingress


Load Balancer Controller


I have created a short script to handle the load balancer contoller creation. The script can be run several times in case it fails, and you fix something, and want to rerun it.

Before running the script, start by authenticating to the AWS, and setting the configuration. You can use the env.sh file in this post.


Make sure to update the environment variables at the top of this script before running it.



#!/bin/bash
set -e

# Update these before starting
export CdnClusterName=my-eks-cluster
export CdnAwsAccount=123456789123
export CdnAwsRegion=us-east-1
export CdnVpcId=vpc-12345678912345678


echo "create IAM OIDC provider"
eksctl utils associate-iam-oidc-provider --cluster ${CdnClusterName} --approve

echo "check if IAM policy exists"
policyExists=$(aws iam list-policies|grep AWSLoadBalancerControllerIAMPolicy|wc -l)
if [[ "${policyExists}" = "0" ]]; then

echo "download IAM policy for the LoadBalancerController"
curl -s -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.1.3/docs/install/iam_policy.json

echo "create IAM policy"
aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://iam_policy.json

rm -f iam_policy.json
fi

echo "create IAM service account"
eksctl create iamserviceaccount \
--cluster=${CdnClusterName} \
--namespace=kube-system \
--name=aws-load-balancer-controller \
--attach-policy-arn=arn:aws:iam::${CdnAwsAccount}:policy/AWSLoadBalancerControllerIAMPolicy \
--override-existing-serviceaccounts \
--approve

echo "install LoadBalancerController CRDs helm chart"
kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"
helm repo add eks https://aws.github.io/eks-charts

echo "install LoadBalancerController helm chart"
helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \
--set clusterName=${CdnClusterName} \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller \
--set region=${CdnAwsRegion} \
--set vpcId=${CdnVpcId} \
-n kube-system

kubectl get deployment -n kube-system aws-load-balancer-controller

echo "Done"



If the script is success you will be able to see the deployed load balancer controller which is created by the script.



Ingress


Once the load balancer controller is working, we can deploy an ingress. The following is an example of an ingress. Notice the annotations that enable this ingress to be accessed from the internet.



apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
spec:
rules:
- http:
paths:
- path: /my-prefix/*
backend:
serviceName: my-service-1
servicePort: 80
- host: my.host.com
http:
paths:
- backend:
serviceName: my-service-2
servicePort: 80



Once deployed, you can check AWS console, EC2,  Load Balancers, and see the created load balancer. The public IP for the load balancer is displayed in the load balancer's properties, and you can use it to access the kubernetes.

Final Note


We have presented a simple way of using ingress in AWS EKS. Feel free to use/modify the script per your requirements. It is not bullet-proof, but it dramatically save your time when in need of ingress creation.


Update:

See this for fixing permission issue.


Wednesday, March 10, 2021

Push images to AWS ECR




In this post we will review how to push docker images to AWS Container Registry (aka ECR). The instructions here are intended to be used as part of a CLI to automate the push.


Start by authenticating to the AWS, and setting the configuration. You can use the env.sh file in this post.


Next authenticate to the docker:


aws ecr get-login-password --region REGION | docker login --username AWS --password-stdin USERID.dkr.ecr.REGION.amazonaws.com


Replace the USERID and the REGION with the relevant values for your deployment, for example:


aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 234567786.dkr.ecr.us-east-1.amazonaws.com


AWS ECR does not support sub folders within the registry, and so we need to configure an ECR per image.  This can be done using the following script:


repoName=my-image/dev
aws ecr describe-repositories --repository-names ${repoName} >/dev/null 2>&1
status=$?
if [[ ! "${status}" -eq 0 ]]; then
aws ecr create-repository --repository-name ${repoName}
fi


The last step is to tag the image and to push the image to the ECR.


imageName=my-image/dev:latest
EcrTag=USERID.dkr.ecr.REGION.amazonaws.com/${imageName}
LocalTag=MY-LOCAL-REGISTRY/${imageName}
docker tag ${LocalTag} ${EcrTag}
docker push ${EcrTag}
docker rmi ${EcrTag}



Final Note


In this post we have reviewed the steps to automate push of docker image to AWS ECR. The ECR can be latest used by a EKS as a registry container. To accomplish this, make sure to add permission to the EKS account to access the ECR.


Wednesday, March 3, 2021

Create Kubernetes Cluster on AWS EKS



 

In this post we will review the steps to create a kubernetes cluster on AWS EKS service.


First we need to configure AWS CLI credentails and zone. I like to use a script to handle these configurations:


env.sh

export AWS_SHARED_CREDENTIALS_FILE=${PWD}/credentials
export AWS_CONFIG_FILE=${PWD}/config
export AWS_PAGER=""


Where the credentials file contains your AWS access key, and the config file contains the region configuration.


credentials

[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX


config

[default]
region=us-east-1
output=json



AWS EKS cluster is partly managed by the eksctl CLI, which creates several AWS cloud formation templates that configure the kubernetes cluster entities on AWS.

To install eksctl, use the following:


curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin



Next create a YAML file to configure the cluster:


cluster.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
name: my-eks-cluster
region: us-east-1

availabilityZones:
- us-east-1a
- us-east-1b

nodeGroups:
- name: ng-1
instanceType: m5.large
desiredCapacity: 2
ssh:
publicKeyPath: ./id_rsa.pub
availabilityZones:
- us-east-1a



Notice that the file points to an existing SSH public key, which will be used for the EC2 instances that are part of the cluster.


Also note that even we have used 2 availability zones for the cluster control plane, we have configured its nodes in a single availability zone, which means higher communication but less availability.


See also the schema of the YAML file here, and some examples here.

Next, we can install the cluster using the following command:


eksctl create cluster -f ./cluster.yaml



The installation lasts ~20 minutes (go have a coffee...), and it can be tracked in the AWS cloud formation GUI. Once complete, it configures the ~/.kube/config file, so we can connect to the cluster, and run, for example:



$ kubectl get nodes

NAME                             STATUS   ROLES    AGE   VERSION

ip-192-168-1-114.ec2.internal    Ready    <none>   10m   v1.18.9-eks-d1db3c

ip-192-168-12-214.ec2.internal   Ready    <none>   10m   v1.18.9-eks-d1db3c



We can also create the ~/.kube/config on another machine using the command:


aws eks update-kubeconfig --name cdn-middle-1 --region us-east-1



Final Note


I must admit, that compared with Google's GCP kubernetes cloud services, AWS EKS looks very poor, and including minimal support for cluster maintenance.