extern/zrepl
1
0
Fork 0
mirror of https://github.com/zrepl/zrepl.git synced 2024-11-22 08:23:50 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
904c1512a3
zrepl/zfs/zfscmd/zfscmd-logging-scraper
History
Christian Schwarz 10a14a8c50 [#307] add package trace, integrate it with logging, and adopt it throughout zrepl 
package trace:

- introduce the concept of tasks and spans, tracked as linked list within ctx
    - see package-level docs for an overview of the concepts
    - **main feature 1**: unique stack of task and span IDs
        - makes it easy to follow a series of log entries in concurrent code
    - **main feature 2**: ability to produce a chrome://tracing-compatible trace file
        - either via an env variable or a `zrepl pprof` subcommand
        - this is not a CPU profile, we already have go pprof for that
        - but it is very useful to visually inspect where the
          replication / snapshotter / pruner spends its time
          ( fixes #307 )

usage in package daemon/logging:

- goal: every log entry should have a trace field with the ID stack from package trace

- make `logging.GetLogger(ctx, Subsys)` the authoritative `logger.Logger` factory function
    - the context carries a linked list of injected fields which
      `logging.GetLogger` adds to the logger it returns
    - `logging.GetLogger` also uses package `trace` to get the
      task-and-span-stack and injects it into the returned logger's fields
2020-05-19 11:30:02 +02:00
..
analysis.ipynb zfs: introduce pkg zfs/zfscmd for command logging, status, prometheus metrics 2020-04-05 20:47:25 +02:00
README.md zfs: introduce pkg zfs/zfscmd for command logging, status, prometheus metrics 2020-04-05 20:47:25 +02:00
scrape_graylog_csv.bash zfs: introduce pkg zfs/zfscmd for command logging, status, prometheus metrics 2020-04-05 20:47:25 +02:00
zfscmd_logging_scraper.go [#307] add package trace, integrate it with logging, and adopt it throughout zrepl 2020-05-19 11:30:02 +02:00
zfscmd_logging_scrapter_test.go [#307] add package trace, integrate it with logging, and adopt it throughout zrepl 2020-05-19 11:30:02 +02:00

README.md

The tool in this package (go run . -h) scrapes log lines produces by the github.com/zrepl/zrepl/zfs/zfscmd package into a stream of JSON objects.

The analysis.ipynb then runs some basic analysis on the collected log output.

Deps for the scrape_graylog_csv.bash script

pip install --upgrade git+https://github.com/lk-jeffpeck/csvfilter.git@ec433f14330fbbf5d41f56febfeedac22868a949