In this post we'll review adding a simple counter metric to a GO application.
See also the related posts:
Let's assume we have a GO application providing two services over HTTP: /foo and /bar.
For this example, the services implementation is sleep for a random time, and return a string in the response.
package main
import (
"log"
"math/rand"
"net/http"
"time"
)
func main() {
http.HandleFunc("/foo", fooHandler)
http.HandleFunc("/bar", barHandler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
func fooHandler(w http.ResponseWriter, _ *http.Request) {
randomTime := time.Duration(int(rand.Float32() * 1000))
time.Sleep(time.Millisecond * randomTime)
_, _ = w.Write([]byte("foo is done"))
}
func barHandler(w http.ResponseWriter, _ *http.Request) {
randomTime := time.Duration(int(rand.Float32() * 1000))
time.Sleep(time.Millisecond * randomTime)
_, _ = w.Write([]byte("bar is done"))
}
First we want to integrate the /metrics URL with the prometheus handler.
We add the following code:
package main
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
func main() {
http.Handle("/metrics", promhttp.Handler())
Once this code is added, we immediately get some defaults GO related counters in the /metrics URL call:
$ curl localhost:8080/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 7
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.2"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 639848
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 639848
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 3836
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 112
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.436808e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 639848
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.5093632e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.589248e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 2260
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 6.5093632e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.668288e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 0
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 2372
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 13888
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 37400
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 49152
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.473924e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.034244e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 425984
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 425984
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.1649288e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 7
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.42
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 9
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 8.392704e+06
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.59184909075e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.112522752e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
Next, we want to add our own metrics.
Let's add counters for the amount of each service invocation.
To implement this, we add a vector of counters containing an entry per each handler, and we wrap the handler execution with a counter update code.
package main
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/client_golang/prometheus/promhttp"
"log"
"math/rand"
"net/http"
"time"
)
var counters *prometheus.CounterVec
func main() {
counters = promauto.NewCounterVec(
prometheus.CounterOpts{
Name: "invocations",
Help: "counters for each HTTP service",
},
[]string{"handler"},
)
http.Handle("/metrics", promhttp.Handler())
addHandlerFunc("/foo", fooHandler)
addHandlerFunc("/bar", barHandler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
func addHandlerFunc(pattern string, handler func(http.ResponseWriter, *http.Request)) {
http.HandleFunc(pattern, func(w http.ResponseWriter, r *http.Request) {
labels := make(map[string]string)
labels["handler"] = pattern
counters.With(labels).Inc()
handler(w, r)
})
}
Now we get the custom counters in the /metrics HTTP response:
$ curl localhost:8080/metrics | grep invocations
# HELP invocations counters for each HTTP service
# TYPE invocations counter
invocations{handler="/bar"} 2
invocations{handler="/foo"} 1
Great!
Our application is now reporting the custom metrics.
Let run it in a kubernetes, that already has a prometheus installed.
We need to specify the prometheus annotations, to indicate that we want prometheus to scrape the pod:
- prometheus.io/scrape: "true"
- prometheus.io/path: "/metrics"
- prometheus.io/port: "8080"
We also configure the deployment to run 2 replicas of our application.
apiVersion: apps/v1
kind: Deployment
metadata:
name: prom-deployment
spec:
replicas: 2
selector:
matchLabels:
configid: prom-container
template:
metadata:
labels:
configid: prom-container
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "8080"
spec:
containers:
- name: prom
image: myregistry:5000/prom/dev:latest
imagePullPolicy: IfNotPresent
Let's view the counters in prometheus GUI.
We search metric by the name invocations, and we find 4 entries, as we have 2 replicas, each running 2 handlers.
Using the metrics collected by the prometheus, we can easily add grafana graphs to view the statistics overtime.
For example, we can view the counters by pod:
and we can view the counters by handler:
Final Notes
In this post we have reviewed how to create an HTTP service based on a GO application, and report metric for prometheus.
We have also see how can we view the reported metrics in prometheus, and create graphs in grafana.
No comments:
Post a Comment