4.4 Golang Memory Leaks

This is a historical learning note and may contain outdated or incomplete understanding.

1. Memory Leaks

Memory leaks.md (the original link is no longer valid)

2. Golang Memory Leaks

2.1. Root Object References

var cache = map[interface{}]interface{}{}

func keepalloc() {
  for i := 0; i < 10000; i++ {
    m := make([]byte, 1<<10)
    cache[i] = m
  }
}

2.2. Goroutine Leaks

Goroutine Leaks: Principles, Scenarios, Detection, and Prevention - SegmentFault

If you start a goroutine but it does not exit as expected, that goroutine will not exit until the program ends. This is a goroutine leak.
When a goroutine leak occurs:
- The goroutine’s stack (generally starting from about 2 KB of memory) remains occupied and cannot be released.
- Space allocated on the heap by functions inside the goroutine also cannot be reclaimed by the garbage collector. A program continuously creates new goroutines:

func keepalloc2() {
  for i := 0; i < 100000; i++ {
    go func() {
      select {}
    }()
  }
}

2.3. Memory Leaks Not Managed by the Runtime

For example, memory allocated by cgo is not managed by the Golang runtime.

3. How to Analyze Memory Leaks

Use the top command to see whether the Golang program is using a large amount of memory.
Use pprof for analysis.
1. Use go tool pprof http://xxx/debug/pprof/heap?debug=1 to inspect runtime.MemStat.
  - Check HeapSys+StackSys or Sys. If it is far smaller than the RSS shown by top, then the consumed memory is not managed by the Go runtime, and the problem may be in cgo.
    - For cgo, in order not to block goroutines, cgo processing uses separate threads. This is not managed by the runtime; use ps -mp PID to inspect the thread count. If the count is very large, the threads may have been created by cgo.
    - Use valgrind or bcc to analyze C-program memory leaks: Quickly Locating Memory Leaks in Go Programs - MySpace
  - If HeapInuse is far smaller than the RSS shown by top, it may be caused by the MADV_FREE mode introduced in Go 1.12. HeapIdle-HeapReleased tells you how much memory has not been returned to the OS.
2. Check the number of goroutines and analyze whether goroutines are blocked and causing leaks.
  1. Use go tool pprof http://xxx/debug/pprof/goroutine?debug=1 to inspect the goroutine count.
  2. Use go tool pprof http://xxx/debug/pprof/goroutine?debug=2 to inspect goroutine details and see whether there are locks or blocked channel sends.
3. Use go tool pprof http://xxx/debug/pprof/heap to inspect heap-object usage and find the largest consumers.

4. Memory Leak Examples

4.1. Top RSS > Golang Heap Size

After load testing, the RSS shown by top did not decrease. top showed that the Golang process had high RSS usage, while Golang pprof analysis did not find any memory object with unusually high usage. Investigation showed that this was caused by the MADV_FREE mode introduced in Go 1.12: to improve allocation efficiency, after the Golang runtime reclaimed objects it did not return the memory to the operating system.

4.2. Analyzing Memory Dropping to -1

Monitoring showed that the Golang program’s memory dropped to -1 every two hours. I initially thought it was a panic, but the panic logs looked normal. Golang pprof did not show any high-memory objects. Because the program referenced a cgo library, I suspected that the C program had core-dumped and caused a panic that was not captured. After enabling core dumps, there still was nothing. Eventually I suspected the monitoring itself. On another property-monitoring page, the memory was normal and did not drop to -1, so the conclusion was that it was a monitoring problem.

4.3. cgo Memory Leak

The program runs periodically: it retrieves a streamer’s cover image over HTTP, calls ImageMagick to adjust the pixels, and then uploads it again over HTTP. The HTTP client only configured the TCP connection-establishment timeout, not the HTTP request timeout.

DefaultCli = &http.Client{
		Transport: &http.Transport{
			DialContext: (&net.Dialer{
				Timeout:   2 * time.Second,
				KeepAlive: 30 * time.Second,
			}).DialContext,
	}}

The channel was created without a buffer. When the operation times out, the function returns. At that point ch no longer has a consumer, so the goroutine remains blocked forever.

func RebuildImage() {
	var wg sync.WaitGroup
	wg.Add(3)

	// Takes time 1
	go func() {
		// do sth
		defer wg.Done()
	} ()

	// Takes time 2
	go func() {
		// do sth
		defer wg.Done()
	} ()

	// Takes time 3
	go func() {
		// do sth
		defer wg.Done()
	} ()

	ch := make(chan struct{})

	go func () {
		wg.Wait()
		ch <- struct{}{}
	}()

	// Receive completion or timeout
	select {
	case <- ch:
		return
	case <- time.After(time.Second * 10):
		return
	}
}

The service uses a cgo library for image processing, which occupies a large amount of memory while processing. For some reason, threads became blocked or were not released, causing the service’s thread count to grow rapidly and eventually leading to a Golang memory leak.