Downsides of Investigate a GO Memory Leak using PPROF

 

In this post we will review the limitations of using GO PPROF. A long time ago, I have listed Monitoring a GO application using pprof, which explains the basic steps to investigate a GO process. However when we encounter a memory leak, pprof might be misleading.

The Garbage Collector Issue

The first issue is that GO is a managed language: memory allocation and release is managed by the GO runtime, specifically by the garbage collector. So for any object we use in the process, GO keeps pointers and counters to manage it. This is reason that when we see in pprof heap analysis that the GO is using 1 GB of memory, the actual process memory might be double or even more.

The case might worsen in case we use many small objects. For example if we have a single string object of size 1M, the actual memory used will be very close to 1M, since the GO garbage collector should manage a single object. However if we have a JSON object or struct on 1M with many small elements and values, the GO garbage collector will keep pointers and counters for each of the elements, causing much higher memory usage. This is one of the reasons that in case possible, long term memory data is better to be stored as string or bytes and only converted to objects when required.

The Memory Allocation Issue

The second issue is what does PPROF heap supply? It provides a great graphical representation of memory allocations locations. This is different than memory references, and might be not useful. For example:

A GO process receives messages from NATS, deserialize them to structs using JSON unmarhsaling, process these structs and save some of the processed strings in baselines in the memory. Let's assume we have a bug in the baselines management that causes a leak. Where would the leak appear in pprof? In the memory allocation point, hence it will be shown in the deserializing code and not in the baseline management code. This is an important issue to understand.

Actually the GO garbage collector does have the referencing information, but pprof does not provide a way to get it, which is a shame. In case of need, we can create a full heap dump and analyze it using other external tools, which is not easy.

Final Note

We have presented two limitations of using GO pprof for analysis of a memory leak. While these are real issues that we face in a production application, pprof is still the best tool available. In future versions I hope GO maintainers would consider addressing these issues.