NATS Monitoring Using Prometheus/Grafana

In a recent post we've setup a NATS cluster in kubernetes. In this post, we will review the steps to monitor NATS using Prometheus and Grafana.

The prometheus counters for NATS monitoring are provided using an exporter. This is implemented as a sidecar - an additional container in each pod which pulls the monitoring data from the local pod NATS container, and exposes an HTTP endpoint for prometheus.

The Exporter

To implement an exporter we create a minimal GO program to run the NATS exporter. The exporter uses localhost address, as it is part of the same pod.

package main

import (
  "github.com/nats-io/prometheus-nats-exporter/exporter"
)

func main() {
  opts := exporter.GetDefaultExporterOptions()
  opts.ListenAddress = "0.0.0.0"
  opts.ListenPort = 8080
  opts.GetVarz = true
  opts.NATSServerURL = "http://localhost:8222"

  natsExporter := exporter.NewExporter(opts)
  err := natsExporter.Start()
  if err != nil {
   panic(err)
  }
  natsExporter.WaitUntilDone()
}

The Sidecar

To use a sidecar, we add an additional container after the 2 existing ones.

The Scraping

To enable scraping of the pod by prometheus, we add the related annotations to the template section of the NATS statefulset.

template:
  metadata:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/path: "/metrics"
      prometheus.io/port: "8080"

The Grafana Dashboard

Last, we import a pre-made example dashboard from here.

Notice that the dashboard displays absolute values, while in real life you will probably want to change it to use rate, for example, change this:

gnatsd_varz_in_bytes[1m]

to:

rate(gnatsd_varz_in_bytes[1m])