Analyzing JSON and Text using Bash

In this post we will review simple methods to quickly analyze JSON and text data using bash CLIs and pipelines. While these methods might lack some of the abilities that we can find in more complex GO/JavaScript/python code, they have a great advantage of quickly grepping a piece of information we're looking for.

For this post we will use an input file containing list of HTTP requests, for example:

JSON Parsing

This first thing we want to do it to use the `jq` command to get this as a parsed JSON.

cat data.json | jq

Next, lets get only the source IPs from the file:

cat data.json | jq  '.[].SourceIp'

Count and Sort

Let's get unique count of requests per IP sorted by count.

cat data.json | jq '.[].SourceIp' | sort | uniq -c | sort -n

Text Parsing

Let's get the methods usage in the HTTP request.

cat data.json | jq '.[].HttpRequest' 

Now split the text by the first space and get only the first part, then remove the first character (the quotes). Then use the previous method to count per method.

cat data.json | jq '.[].HttpRequest' | awk -F' ' '{print $1}' | cut -c2-1000 | sort | uniq -c | sort -n

We can do the same to get only the HTTP path.

cat data.json | jq '.[].HttpRequest' | awk -F' ' '{print $2}' | sort | uniq -c | sort -n

Grep Area

What if we want to get transactions only for a specific IP?

We use the grep -A, -B, -C flags.

-A = get also one line after the located text

-B = get also one line before the located text

-C = get also one line before and after the located text

cat data.json | jq  | grep -B1 46.117.105.66

Now we analyze the transactions for this IP like we've done before:

cat data.json | jq  | grep -B1 46.117.105.66 | grep HttpRequest | awk -F ' ' '{print $3}' | sort | uniq -c | sort -n

Final Note

In this post we're had a taste of the power of bash piping.

I highly recommend experimenting with these CLIs since you can do in a few seconds things that would otherwise take you much longer.