run KISS: NGINX Performance Tuning

The Stress Test

Last week I've run some stress tests for a product running on kubernetes.
We've have several nodes in the kubernetes, and the general product design was as follows.

The product is using NGINX as a reverse proxy to multiple micro services.
The micro services are using the REDIS as the persistence layer

Using a stress client application, which simulates multiple browsers, I've started sending transactions through the NGINX server.

In previous stress tests I've run, one of the micro services is prominent at its CPU/memory consumption, and hence the tuning focus is directed to the specific micro service.

However, in this case, I've received a unusual result. The first findings were:

The transaction per second rate was much lower than the expected.
All of the components CPU and memory consumption was low.

This was a mysterious issue. What's holding the transactions back?

In a desperate action, I've tried doubling the replicas for all of the deployments in the kubernetes cluster: The NGINX, the REDIS, and each of the micro services had now doubled replica count.

This did not change the transactions per second rate at all!

After investigation, I've located the culprit.

It was NGINX.

Well, actually, NGINX was doing exactly what it was configured to do.

The point is that to get real high performing NGINX, it must be carefully configured.

In this post, I will review the changes that I've made to NGINX configuration, and how to make sure the configuration changes are indeed working as expected.

NGINX and the Ephemeral Ports

Each TCP connection to a TCP listener creates an entry in the ports table.

For example, upon startup of the NGINX listener, running netstat -a would show the following entries:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN

Once a browser had connect to port 8080 of the NGINX listener, the following is added (again, this is shown using netstat -a):

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN     
tcp        0      0 192.168.100.11:8080     192.168.100.35:51416    ESTABLISHED

However, once the client disconnects, it leave the connection in a TIME_WAIT state.

The goal of this state is to ignore future packets appearing on the same combination of the quadrille:

SRC host&port, DST host&port.

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN     
tcp        0      0 192.168.100.11:8080     192.168.100.35:51416    TIME_WAIT

The TIME_WAIT state is kept for 2 minutes (depends on the OS configuration), and the the entry is removed.

Creation of a new entry in the ports table has several implications:

The ports table is limited in its size. The actual limit depends on the OS configuration, but it is an order of 30K entries. In case reaching to the limit, new connection would be rejected by the server.
Creating a new socket has an high footprint cost. A rough estimation is 10ms, but it varies depending on the packet round-trip time, and on the server resources.

How to Investigate NGINX Pod Port Table

When connecting to the NGINX, using the command:

kubectl exec -it NGINX_POD_NAME bash

And then trying to run the netstat command, you will find that netstat is not installed.
To install it, create your own docker image, based on the NGINX image, and install the netstat.

FROM nginx
RUN apt-get update
RUN apt-get install -y net-tools

And the you can connect, and run netstat in the NGINX pod.

Configuring NGINX

To prevent the TIME_WAIT connections, we need to configure NGINX to use keep-alive, that is, to ask NGINX to reuse connections, and not to close them whenever possible.

Notice that this should be configured both for the client connecting with incoming connections, as well as for the outgoing connections to the micro-services.

First, for the client side connection, add the following configuration in nginx.conf:

http {
    ...
    keepalive_requests 1000000;
    keepalive_timeout 300s;

Notice that most clients would probably have only few connections, so the keepalive_requests limit seems extremely high. The reason for the high limit is to enable the stress client to reuse the sam connection as much as possible, while simulating multiple browsers clients.

Next, for the outgoing proxy side to the micro-services, add the following configuration in nginx.conf:

http {
  
  ...
  
  upstream micro-service-1 {
    server micro-service-1.example.com;
    keepalive 1000;
    keepalive_requests 1000000;
    keepalive_timeout 300s;
  }

  ...
  
  server {

  ...
  
    location /my-redirect/micro-service-1 {
      proxy_pass http://micro-service-1;

      proxy_set_header      Connection "";
      proxy_ignore_headers    "Cache-Control" "Expires";
      proxy_buffers         32 4m;
      proxy_busy_buffers_size   25m;
      proxy_buffer_size       512k;
      client_max_body_size    10m;
      client_body_buffer_size   4m;
      proxy_connect_timeout     300;
      proxy_read_timeout      300;
      proxy_send_timeout      300;
      proxy_intercept_errors    off;
      proxy_http_version      1.1;
    }

  ...

Note the following:

We configure the micro-service as an upstream entry, where we can configure it to use keepalive
We configure the relevant URL location to redirect to the upstream entry, and to use HTTP 1.1. In addition, we add some tuning configuration for buffers sizes and timeouts.

Special Case - Auth Request

NGINX include a support for an auth request. This means that upon a client request to access a specific URL, NGINX would first forward the request to another micro service, to approve the access. However, as specified in this bug, NGINX would disable the keep-alive configuration for the auth request.

To overcome this, use auth request that is sent by NGINX JavaScript. See an example, in the following configuration in nginx.conf:

http {
  
  load_module modules/ngx_http_js_module.so;
  load_module modules/ngx_stream_js_module.so;
  js_include /etc/nginx/auth.js;
  
  ...
  
  upstream authenticate-micro-service-1 {
   server authenticate-micro-service-1.example.com;
   keepalive 1000;
   keepalive_requests 1000000;
   keepalive_timeout 300s;
  }

  ...
  
  server {

   ...

   location /authenticate-micro-service-1 {
    proxy_pass http://authenticate-micro-service-1;
    proxy_http_version  1.1;
   }

   location /javascript-auth {
    internal;
    js_content validate;
    proxy_http_version  1.1;
   }

   location /secured-access {
    auth_request   /javascript-auth;
    proxy_http_version  1.1;
   }

   ...

Where is auth.js is the following:

function validate(r){

    function done(res){
        var validateResponse = JSON.parse(res.responseBody);
        if validateResponse.ok {
            r.return(200);
        } else {
            r.return(500);
        }
    }

    r.subrequest("/authenticate-micro-service-1", r.variables.args, done);
}

Final Notes

After the NGINX configuration changes, things started moving, the transactions per seconds rate increase, and I was able to start tuning the microservices that were now getting high transactions rate, but this is another story, for another post.

Liked this post? Leave a comment.

Monday, March 30, 2020

NGINX Performance Tuning