From f8541dfeefe1f248535254d4bd99ceb4d462f05b Mon Sep 17 00:00:00 2001 From: Christian Schwarz Date: Fri, 13 Sep 2019 15:06:45 +0200 Subject: [PATCH] Updated Debugging (markdown) --- Debugging.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/Debugging.md b/Debugging.md index 54a34f0..880cd13 100644 --- a/Debugging.md +++ b/Debugging.md @@ -3,11 +3,14 @@ A good example for a goroutine & memory leak caused by not draining a channel is [3bfe0c16d0233cac66a01a6f89959c34ef01c663](https://github.com/zrepl/zrepl/commit/3bfe0c16d0233cac66a01a6f89959c34ef01c663d) ("rpc/dataconn/stream: fix goroutine leaks & transitive buffer leaks") -1. Run zrepl with autostarted pprof server: `ZREPL_DAEMON_AUTOSTART_PPROF_SERVER=:12345 zrepl daemon` - (We will assume the address is 192.168.124.252) -2. `watch 'curl 192.168.124.252:22345/metrics | grep -v "#" | grep memstats'`
+1. Run zrepl with autostarted pprof server on port `:12345` and prometheus endpoint on `:22345` + * configure prometheus endpoint in [config]( https://zrepl.github.io/configuration/monitoring.html?highlight=prometheus#prometheus-grafana) + * `zrepl pprof on :12345` if the daemon is already running (for example when we want to capture a rare deadlock which would be resolved by restarting the daemon) + * or `ZREPL_DAEMON_AUTOSTART_PPROF_SERVER=:12345 zrepl daemon` if restarting the daemon is a good idea + +2. `watch 'curl localhost:22345/metrics | grep -v "#" | grep memstats'`
=> `go_memstats_heap_inuse_bytes` or `go_memstats_heap_alloc_bytes` should be rising -3. `watch curl http://192.168.124.252:12345/debug/pprof/goroutine?debug=1` +3. `watch curl http://localhost:12345/debug/pprof/goroutine?debug=1` ``` 9 @ 0x469db0 0x43d5f4 0x43d5ca 0x43d355 0xc767b0 0x498f51