GOMEMLIMIT - Mandatory To Use

 

In this post we will review the usage of the Go environment variable GOMEMLIMIT, and explain how to use it.

First, let's understand the core behavior of the garbage collection on a Go process runtime. By default, the garbage collection runs whenever the process allocated memory doubles. 

See for example, the following memory chart:

This chart displays 2 event of garbage collection, each occurring once the process memory doubles: 

• The memory reaches 200M, grabage collection cleaning 20M, and the process memory drops to 180M
• The memory reaches 360M, grabage collection cleaning 60M, and the process memory drops to 300M

This might look suitable on first sight, but it might cause the process to fail on out of memory error when running on kubernetes.  For example:

• The process memory limit is 3G RAM
• The garbage collection last run had finished with 2G RAM
• The next planned garbage collection run is at 4G RAM
• The process terminates a 3G RAM with out of memory error.

One way to solve this is to use the GOGC environment variable which can configure when to run the garbage collector. The default is GOGC=100, which mean to run the garbage collector when the memory rises by 100%. We could set it to GOGC=10, but then the garbage collector would run (and delay the process performance) even when the memory consumption is low.

Here is where the GOMEMLIMIT environment variable comes to help. It sets up a soft limit for a process runtime. This means that the garbage collection is run more often when the process memory is above this threshold. Notice that the GOMEMLIMIT does not include some OS related memory, so we would usually set it to a smaller value than the actual memory limit.

How do we actually use this?

We should set the GOMEMLIMIT to a value close the the actual limit of memory by kubernetes. For example, if the kubernetes pod memory limit is 5G, we will set the GOMEMLIMIT to ~4G.