Monitoring NGINX on Kubernetes using Prometheus

 

In this post we will review how to monitor NGINX server on kubernetes using Prometheus and Grafana.

In general, Prometheus monitors any kubernetes deployment that specified the set of relevant Prometheus annotations (see below). Once prometheus marks a pod for scraping, it will send requests to the pod, and update the results in its time series database.

The problem is that NGINX and Prometheus speak different languages. 

NGINX uses the stub status page, which upon access returns the following output:

root@nginx-deployment-64bf95d447-gs65z:/# curl 127.0.0.1:8080/stub_status
Active connections: 2 
server accepts handled requests
 2132 2132 100501 
Reading: 0 Writing: 1 Waiting: 1 

while the Prometheus expects it own format:

root@nginx-deployment-64bf95d447-gs65z:/# curl 127.0.0.1:9113/metrics
# HELP nginx_connections_accepted Accepted client connections
# TYPE nginx_connections_accepted counter
nginx_connections_accepted 2174
# HELP nginx_connections_active Active client connections
# TYPE nginx_connections_active gauge
nginx_connections_active 1

To enable Prometheus scraping for NGINX, we we an exporter, that provides the NGINX statistics in the Prometheus format. We will create a deployment that includes 2 containers: The NGINX container, and the exporter container. The exporter samples NGINX to get the updated statistics, and returns it to the Prometheus in its expected format.

Let's start with the NGINX configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |-
    user  nginx;
    worker_processes  10;
    
    error_log /dev/stdout warn;
    pid       /var/run/nginx.pid;
    
    events {
      worker_connections  10240;
    }
    
    http {
        include         /etc/nginx/mime.types;
        default_type    application/octet-stream;

        server {
            listen       8080;
            server_name  localhost;
    
            location /stub_status {
                allow 127.0.0.1;
                deny all;
                stub_status on;
            }
    
            location / {
                return 200 'NGINX is alive';
            }
        }
    }
    

We have added to the NGINX configuration file the /stub_status location, which exposes the NGINX statistics. In addition we have blocked all IPs to access to this location, expect for the localhost IP.

Next, we create the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      configid: nginx
  template:
    metadata:
      labels:
        configid: nginx
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "9113"
    spec:
      containers:
        - name: nginx
          image: nginx
          volumeMounts:
            - name: nginx-config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
        - name: exporter
          image: nginx/nginx-prometheus-exporter:0.8.0
          args:
            - -nginx.scrape-uri=http://127.0.0.1:8080/stub_status
      volumes:
        - name: nginx-config
          configMap:
            name: nginx-config

Notice some of the used deployment configuration:

• Using the annotations, we mark the deployment to be scraped by Prometheus, but we specify the port 9113, which is not the NGINX port, but the exporter port.
  
  
• We include 2 containers: NGINX and exporter.
  
  
• The exporter is configured to access the stub_status page that the NGINX exposes

Now we can visualize this in Grafana.

Final Notes

We have reviewed NGINX monitoring using Prometheus scraping and an exporter. Additional metrics are available from the exporter. Use the Prometheus GUI to display all of them, and display additional metrics, such as the amount of the active connections, NGINX aliveness, and more.