Downsides of Investigate a GO Memory Leak using PPROF

 

In this post we will review the limitations of using GO PPROF. A long time ago, I have listed Monitoring a GO application using pprof, which explains the basic steps to investigate a GO process. However when we encounter a memory leak, pprof might be misleading.

The Garbage Collector Issue

The first issue is that GO is a managed language: memory allocation and release is managed by the GO runtime, specifically by the garbage collector. So for any object we use in the process, GO keeps pointers and counters to manage it. This is reason that when we see in pprof heap analysis that the GO is using 1 GB of memory, the actual process memory might be double or even more.

The case might worsen in case we use many small objects. For example if we have a single string object of size 1M, the actual memory used will be very close to 1M, since the GO garbage collector should manage a single object. However if we have a JSON object or struct on 1M with many small elements and values, the GO garbage collector will keep pointers and counters for each of the elements, causing much higher memory usage. This is one of the reasons that in case possible, long term memory data is better to be stored as string or bytes and only converted to objects when required.

The Memory Allocation Issue

The second issue is what does PPROF heap supply? It provides a great graphical representation of memory allocations locations. This is different than memory references, and might be not useful. For example:

A GO process receives messages from NATS, deserialize them to structs using JSON unmarhsaling, process these structs and save some of the processed strings in baselines in the memory. Let's assume we have a bug in the baselines management that causes a leak. Where would the leak appear in pprof? In the memory allocation point, hence it will be shown in the deserializing code and not in the baseline management code. This is an important issue to understand.

Actually the GO garbage collector does have the referencing information, but pprof does not provide a way to get it, which is a shame. In case of need, we can create a full heap dump and analyze it using other external tools, which is not easy.

Final Note

We have presented two limitations of using GO pprof for analysis of a memory leak. While these are real issues that we face in a production application, pprof is still the best tool available. In future versions I hope GO maintainers would consider addressing these issues.

local-path-provisioner

 

The local-path-provisioner is a persistent volume provisioner dedicated for a kind based kubernetes cluster. Using it both simplifies helm charts and provides a better compatibility with EKS gp2 and gp3 storage clasess. In this post we will see how to use it.

Working Without a Provisioner

In case we don't have a provisioner we need to manually handle the persistent volume creation as part of the helm chart. This can be done using a new dedicated storage class for our kind development cluster. For example:

{{- if eq .Values.data.storageClass "development-storage" }}
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-1
  labels:
    type: local
spec:
  storageClassName: development-storage-class
  capacity:
    storage: 10G
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/hostpath/data"
{{- end }}

Then we need to specify this storage class as part of the persistent volume claim:

  volumeClaimTemplates:
    - metadata:
        name: my-pvc
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "development-storage-class"
        resources:
          requests:
            storage: 10G

Using The Local-Path-Provisioner

To use the local-path-provisioner we install it:

kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml

And then all we need it to specify local-path as the storage class, for example:

  volumeClaimTemplates:
    - metadata:
        name: my-pvc
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "local-path"
        resources:
          requests:
            storage: 10G

We not only gain this simpler helm chart, but also permissions respectful provisioner. This simplifies life for pods that use modified security context such as grafana and clickhouse:

      securityContext:
        runAsUser: 50001
        runAsGroup: 50001
        fsGroup: 50001
        fsGroupChangePolicy: OnRootMismatch

Final Note

Actually, I guess I've got used to using the old method without the local-path-provisioner that I took it as is. Now that I know the local-path-provisioner, I think to myself, well why the complexity, I should have known it earlier.