From aeb5c246a68d940e60beadfdf42119bb06dbd738 Mon Sep 17 00:00:00 2001 From: Christian Schwarz Date: Sun, 8 Sep 2019 09:47:31 +0200 Subject: [PATCH] Created Debugging (markdown) --- Debugging.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 Debugging.md diff --git a/Debugging.md b/Debugging.md new file mode 100644 index 0000000..54a34f0 --- /dev/null +++ b/Debugging.md @@ -0,0 +1,17 @@ +## Memory Leaks & Goroutine Leaks + +A good example for a goroutine & memory leak caused by not draining a channel is +[3bfe0c16d0233cac66a01a6f89959c34ef01c663](https://github.com/zrepl/zrepl/commit/3bfe0c16d0233cac66a01a6f89959c34ef01c663d) ("rpc/dataconn/stream: fix goroutine leaks & transitive buffer leaks") + +1. Run zrepl with autostarted pprof server: `ZREPL_DAEMON_AUTOSTART_PPROF_SERVER=:12345 zrepl daemon` + (We will assume the address is 192.168.124.252) +2. `watch 'curl 192.168.124.252:22345/metrics | grep -v "#" | grep memstats'`
+ => `go_memstats_heap_inuse_bytes` or `go_memstats_heap_alloc_bytes` should be rising +3. `watch curl http://192.168.124.252:12345/debug/pprof/goroutine?debug=1` + + ``` + 9 @ 0x469db0 0x43d5f4 0x43d5ca 0x43d355 0xc767b0 0x498f51 + # 0xc767af github.com/zrepl/zrepl/rpc/dataconn/stream.doWriteStream.func1+0x2ff /mnt/zrepl /rpc/dataconn/stream/stream.go:92 + ``` + + => 9 kept increasing to higher numbers over time => look at `doWriteStream` impl and fix goroutine leak (see commit)