Monday, September 25, 2023

Go End The Loop Bug

 


Finally, it is solved!

The Go maintainers have finally took the correct actions to fix the loop variable bug. This bug is very annoying, and I have personally made it several times.

Let's examine a simple example of this bug.


func main() {
var waitGroup sync.WaitGroup
for i := 0; i < 10; i++ {
waitGroup.Add(1)
go func() {
defer waitGroup.Done()
fmt.Printf("%v\n", i)
}()
}
waitGroup.Wait()
}


We would expect this simple loop to print the numbers zero to nine, but actually the result is unexpected:


10
10
10
4
10
10
10
10
10
10


Why? This is due to the fact that the scope of the loop variable `i` is the loop itself, so it is created only once, and referred by the iterations. To avoid this, we used to "clone" the loop variable, for example:


func main() {
var waitGroup sync.WaitGroup
for i := 0; i < 10; i++ {
waitGroup.Add(1)
clonedI := i
go func() {
defer waitGroup.Done()
fmt.Printf("%v\n", clonedI)
}()
}
waitGroup.Wait()
}


and the output:


8
9
2
3
0
5
4
1
7
6


So this works, but you always need to remember this. In the upcoming Go 1.22 version, the scope of the loop variable is changed into the iteration, so there is no longer need of the clone trick. We can even force it into Go version 1.21 using the environment variable: 

GOEXPERIMENT=loopvar


Good move GO! (better late than never)






Sunday, September 17, 2023

Using mirrord JetBrains plugin for development within Kubernetes


 


I this post we will review how to use the mirrord plugin within JetBrains IDE.

Mirrord is a new open source tool enabling developer to run processes on the local development machine as if the process is running on a remote kubernetes cluster. It actually works by connecting to a pod, and capturing network and IO requests on the pod, mirroring these to the local development machine. See the architecture flow from the mirrord sire:



The mirrord plugin for JetBrains IDEs simplifies development to a whole new level. Let have a quick walk-though of the steps to use it. 


For this example, I have a deployment and a service named "guibackend" on the remote kubernetes cluster, and I want to run it locally, but I need to use other services to run it. The redis service is part of the kubernetes cluster, so I cannot access it from my local development machine. Let use mirrord to run it as if it is located on the remote kubernetes cluster.


Install the mirrord plugin


In the GoLand IDE, double click the Shift key, type "Plugins", and open the plugins window:


Click on Marketplace, search for mirrord plugin, and click the Install button:


That's all, mirrord is installed.


Using the mirrord plugin

We now have a new button on the toolbar to enable/disable run and debug through mirrord:



Next we can easily run the service locally on the machine as if it was remotely installed on the kubernetes cluster. When mirrord is enabled, running the program displays a popup to select the target pod/deployment:



Once we select the guibackend deployment, we now run on local development machine, but we get the IO and network of the remote pod. Pretty amazing!

This mirrord can be configured to match our requirement, for example, I want always to select the guibackend deployment without the selection popup, and I want not just to mirror the network requests, but instead I want to steal them, hence I use the following configuration:




For complete configuration options, see here.


Final Note

The mirrord is doing a real great job at simplify development and debugging processes on a kubernetes cluster environment. I find it very easy to use, and I highly recommend using it!



Sunday, September 3, 2023

GOMEMLIMIT - Mandatory To Use

 



In this post we will review the usage of the Go environment variable GOMEMLIMIT, and explain how to use it.


First, let's understand the core behavior of the garbage collection on a Go process runtime. By default, the garbage collection runs whenever the process allocated memory doubles. 

See for example, the following memory chart:




This chart displays 2 event of garbage collection, each occurring once the process memory doubles: 

  • The memory reaches 200M, grabage collection cleaning 20M, and the process memory drops to 180M
  • The memory reaches 360M, grabage collection cleaning 60M, and the process memory drops to 300M

This might look suitable on first sight, but it might cause the process to fail on out of memory error when running on kubernetes.  For example:

  • The process memory limit is 3G RAM
  • The garbage collection last run had finished with 2G RAM
  • The next planned garbage collection run is at 4G RAM
  • The process terminates a 3G RAM with out of memory error.
One way to solve this is to use the GOGC environment variable which can configure when to run the garbage collector. The default is GOGC=100, which mean to run the garbage collector when the memory rises by 100%. We could set it to GOGC=10, but then the garbage collector would run (and delay the process performance) even when the memory consumption is low.

Here is where the GOMEMLIMIT environment variable comes to help. It sets up a soft limit for a process runtime. This means that the garbage collection is run more often when the process memory is above this threshold. Notice that the GOMEMLIMIT does not include some OS related memory, so we would usually set it to a smaller value than the actual memory limit.


How do we actually use this?

We should set the GOMEMLIMIT to a value close the the actual limit of memory by kubernetes. For example, if the kubernetes pod memory limit is 5G, we will set the GOMEMLIMIT to ~4G.