mirror of
https://github.com/openziti/zrok.git
synced 2024-11-07 16:54:23 +01:00
97 KiB
97 KiB
Testing the Limits
Consider the following zrok controller configuration stanza, describing the limits we'll be using for this testing scenario:
limits:
environments: -1
shares: -1
bandwidth:
per_account:
period: 5m
warning:
rx: -1
tx: -1
total: -1
limit:
rx: -1
tx: -1
total: -1
per_environment:
period: 5m
warning:
rx: -1
tx: -1
total: -1
limit:
rx: -1
tx: -1
total: -1
per_share:
period: 5m
warning:
rx: -1
tx: -1
total: 1048576
limit:
rx: -1
tx: -1
total: 2097152
enforcing: true
cycle: 1m
Any limit values set to -1 are "unlimited". In this case, we're only enforcing a transfer limit on for shares. This limits configuration will send a warning when a share has transferred more than 1 megabyte in a 5 minute period, and will temporarily deactivate the share when it has transferred more than 2 megabytes in a 5 minute period.
We're going to use the zrok test loop public framework to create a number of public shares and generate traffic. Here are the parameters we'll be using:
$ zrok test loop public -l 7 -i 10000 --min-pacing-ms 100 --max-pacing-ms 1500
This configuration will create 7 shares. Each share will perform 10,000 iterations. The delay between iterations will be randomly generated with a floor of 100ms and a ceiling of 1500ms.
Let's look at the zrok controller log for this run:
First, our zrok test loop public command will create the 7 shares:
[ 2.047] INFO zrok/controller.(*shareHandler).Handle: added frontend selection 'public' with ziti identity 'rBayMvm7UI' for share '0evcupz5k410'
[ 2.081] INFO zrok/controller.(*shareHandler).Handle: added frontend selection 'public' with ziti identity 'rBayMvm7UI' for share '8k6dnu7x7ag0'
[ 2.082] INFO zrok/controller/zrokEdgeSdk.CreateConfig: created config '19cyxfHo32R6fhVsYHZ84g' for environment 'd.wJYlpt9'
[ 2.083] INFO zrok/controller.(*shareHandler).Handle: added frontend selection 'public' with ziti identity 'rBayMvm7UI' for share '53z6mz4re7tu'
[ 2.086] INFO zrok/controller/zrokEdgeSdk.CreateShareService: created share '0evcupz5k410' (with ziti id '3WHJGqUdxkDtPYLgEL5V3q') for environment 'd.wJYlpt9'
[ 2.090] INFO zrok/controller.(*shareHandler).Handle: added frontend selection 'public' with ziti identity 'rBayMvm7UI' for share '7u9szn30ikh0'
[ 2.090] INFO zrok/controller.(*shareHandler).Handle: added frontend selection 'public' with ziti identity 'rBayMvm7UI' for share 'dh3f3jj7zhig'
[ 2.091] INFO zrok/controller.(*shareHandler).Handle: added frontend selection 'public' with ziti identity 'rBayMvm7UI' for share 'tr7vpyrzvmh0'
[ 2.096] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyBind: created bind service policy '4V8FsgCt63ySkG2pFWG5fz' for service '3WHJGqUdxkDtPYLgEL5V3q' for identity 'd.wJYlpt9'
[ 2.097] INFO zrok/controller/zrokEdgeSdk.CreateConfig: created config '5nG9jM8VNl0uBFcRRt3AvI' for environment 'd.wJYlpt9'
[ 2.098] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '74f2gUotsC7DteqpsWrxp0' for service '3WHJGqUdxkDtPYLgEL5V3q' for identities '[rBayMvm7UI]'
[ 2.099] INFO zrok/controller/zrokEdgeSdk.CreateShareService: created share '8k6dnu7x7ag0' (with ziti id '2J0I9dPe2JGnY1GwjmM6n7') for environment 'd.wJYlpt9'
[ 2.100] INFO zrok/controller/zrokEdgeSdk.CreateShareServiceEdgeRouterPolicy: created service edge router policy '2AqCUMqNtarmglOfhvnkI' for service '3WHJGqUdxkDtPYLgEL5V3q' for environment 'd.wJYlpt9'
[ 2.100] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyBind: created bind service policy '4vT5eEPahgWEVdAuKN91Sd' for service '2J0I9dPe2JGnY1GwjmM6n7' for identity 'd.wJYlpt9'
[ 2.104] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '5UHCkXZabFHeWYHmF01Zoc' for service '2J0I9dPe2JGnY1GwjmM6n7' for identities '[rBayMvm7UI]'
[ 2.106] INFO zrok/controller.(*shareHandler).Handle: recorded share '0evcupz5k410' with id '503' for 'michael@quigley.com'
[ 2.106] INFO zrok/controller/zrokEdgeSdk.CreateConfig: created config '6U3XDGnBjtONN5H6pUze12' for environment 'd.wJYlpt9'
[ 2.108] INFO zrok/controller/zrokEdgeSdk.CreateShareServiceEdgeRouterPolicy: created service edge router policy '2RIKOBMOckfbI2xMSLAKxC' for service '2J0I9dPe2JGnY1GwjmM6n7' for environment 'd.wJYlpt9'
[ 2.109] INFO zrok/controller/zrokEdgeSdk.CreateShareService: created share '53z6mz4re7tu' (with ziti id '2NiotGOyBHBEbFZwbTFJ2u') for environment 'd.wJYlpt9'
[ 2.109] INFO zrok/controller/zrokEdgeSdk.CreateConfig: created config '1FnBhnGNXDe58dwTpbFc1x' for environment 'd.wJYlpt9'
[ 2.109] INFO zrok/controller.(*shareHandler).Handle: recorded share '8k6dnu7x7ag0' with id '504' for 'michael@quigley.com'
[ 2.112] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyBind: created bind service policy 'RRfDaA5kjCqUBVC9LvN1H' for service '2NiotGOyBHBEbFZwbTFJ2u' for identity 'd.wJYlpt9'
[ 2.112] INFO zrok/controller/zrokEdgeSdk.CreateConfig: created config '2gid15nP0GIUVuaFQ15GWV' for environment 'd.wJYlpt9'
[ 2.115] INFO zrok/controller/zrokEdgeSdk.CreateShareService: created share '7u9szn30ikh0' (with ziti id '6FzYnK0RFJmT0rDSP1bzVE') for environment 'd.wJYlpt9'
[ 2.115] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '1oo3LuBKxduKAs1wsKndtW' for service '2NiotGOyBHBEbFZwbTFJ2u' for identities '[rBayMvm7UI]'
[ 2.117] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyBind: created bind service policy '1mabRt9jefSe52CJh6FmhB' for service '6FzYnK0RFJmT0rDSP1bzVE' for identity 'd.wJYlpt9'
[ 2.117] INFO zrok/controller/zrokEdgeSdk.CreateShareServiceEdgeRouterPolicy: created service edge router policy '2CM03d1cNpG4rma38BLzCQ' for service '2NiotGOyBHBEbFZwbTFJ2u' for environment 'd.wJYlpt9'
[ 2.118] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '3dBtc3v2G70aqqDSqujQOy' for service '6FzYnK0RFJmT0rDSP1bzVE' for identities '[rBayMvm7UI]'
[ 2.119] INFO zrok/controller.(*shareHandler).Handle: recorded share '53z6mz4re7tu' with id '505' for 'michael@quigley.com'
[ 2.121] INFO zrok/controller/zrokEdgeSdk.CreateShareServiceEdgeRouterPolicy: created service edge router policy '3xAG26zA9yska3LeZQUJ3N' for service '6FzYnK0RFJmT0rDSP1bzVE' for environment 'd.wJYlpt9'
[ 2.122] INFO zrok/controller.(*shareHandler).Handle: added frontend selection 'public' with ziti identity 'rBayMvm7UI' for share 's0uzz1p7xjrr'
[ 2.124] INFO zrok/controller.(*shareHandler).Handle: recorded share '7u9szn30ikh0' with id '506' for 'michael@quigley.com'
[ 2.128] INFO zrok/controller/zrokEdgeSdk.CreateShareService: created share 'tr7vpyrzvmh0' (with ziti id '7jyiTZ0z2ediD5hZbxu7KH') for environment 'd.wJYlpt9'
[ 2.130] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyBind: created bind service policy '6RwWEoIsb8gBVKJfZP3ur3' for service '7jyiTZ0z2ediD5hZbxu7KH' for identity 'd.wJYlpt9'
[ 2.131] INFO zrok/controller/zrokEdgeSdk.CreateConfig: created config '76iBDASRcxOmGtdwjVHo26' for environment 'd.wJYlpt9'
[ 2.132] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '1cURGP202D8n6fzpzWhcgK' for service '7jyiTZ0z2ediD5hZbxu7KH' for identities '[rBayMvm7UI]'
[ 2.138] INFO zrok/controller/zrokEdgeSdk.CreateShareService: created share 'dh3f3jj7zhig' (with ziti id 'nyKOLlxUWWbCzD7h9Jhjq') for environment 'd.wJYlpt9'
[ 2.139] INFO zrok/controller/zrokEdgeSdk.CreateShareServiceEdgeRouterPolicy: created service edge router policy '2nMZaiChQAPpFnblNn1ljP' for service '7jyiTZ0z2ediD5hZbxu7KH' for environment 'd.wJYlpt9'
[ 2.142] INFO zrok/controller.(*shareHandler).Handle: recorded share 'tr7vpyrzvmh0' with id '507' for 'michael@quigley.com'
[ 2.143] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyBind: created bind service policy '1xF4ky6cDJm63tzlNTqoLC' for service 'nyKOLlxUWWbCzD7h9Jhjq' for identity 'd.wJYlpt9'
[ 2.143] INFO zrok/controller/zrokEdgeSdk.CreateConfig: created config '4AN4sOtdQv99uHmFn3erx4' for environment 'd.wJYlpt9'
[ 2.145] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '7GerqbN5lVfcOng91J2J6I' for service 'nyKOLlxUWWbCzD7h9Jhjq' for identities '[rBayMvm7UI]'
[ 2.145] INFO zrok/controller/zrokEdgeSdk.CreateShareService: created share 's0uzz1p7xjrr' (with ziti id 'KtK5E46HR93YIBrrwUlIN') for environment 'd.wJYlpt9'
[ 2.147] INFO zrok/controller/zrokEdgeSdk.CreateShareServiceEdgeRouterPolicy: created service edge router policy '2ZnnIXSTQ3Zscha1kykqQr' for service 'nyKOLlxUWWbCzD7h9Jhjq' for environment 'd.wJYlpt9'
[ 2.149] INFO zrok/controller.(*shareHandler).Handle: recorded share 'dh3f3jj7zhig' with id '508' for 'michael@quigley.com'
[ 2.155] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyBind: created bind service policy '6oohOQFEo75yl9vnIbyzdj' for service 'KtK5E46HR93YIBrrwUlIN' for identity 'd.wJYlpt9'
[ 2.156] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '7eB3ubrntSHxkeHBCGJcOY' for service 'KtK5E46HR93YIBrrwUlIN' for identities '[rBayMvm7UI]'
[ 2.157] INFO zrok/controller/zrokEdgeSdk.CreateShareServiceEdgeRouterPolicy: created service edge router policy '2CGCz8dcquNvZC0ZUwDZ5F' for service 'KtK5E46HR93YIBrrwUlIN' for environment 'd.wJYlpt9'
[ 2.159] INFO zrok/controller.(*shareHandler).Handle: recorded share 's0uzz1p7xjrr' with id '509' for 'michael@quigley.com'
Next, we observe metrics being reported from OpenZiti into the zrok metrics infrastructure for each of the 7 shares:
[ 10.183] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 32.4 kB, tx: 32.6 kB} frontend {rx: 32.6 kB, tx: 32.4 kB}
[ 10.192] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 22.5 kB, tx: 22.8 kB} frontend {rx: 22.8 kB, tx: 22.5 kB}
[ 10.196] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 15.1 kB, tx: 15.3 kB} frontend {rx: 15.3 kB, tx: 15.1 kB}
[ 15.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 53.0 kB, tx: 53.4 kB} frontend {rx: 53.4 kB, tx: 53.0 kB}
[ 15.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 50.3 kB, tx: 50.6 kB} frontend {rx: 50.6 kB, tx: 50.3 kB}
[ 15.170] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 46.2 kB, tx: 46.6 kB} frontend {rx: 46.6 kB, tx: 46.2 kB}
[ 15.172] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 26.4 kB, tx: 26.8 kB} frontend {rx: 26.8 kB, tx: 26.4 kB}
[ 20.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 26.9 kB, tx: 27.1 kB} frontend {rx: 27.1 kB, tx: 26.9 kB}
[ 20.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 26.0 kB, tx: 26.2 kB} frontend {rx: 26.2 kB, tx: 26.0 kB}
[ 20.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 67.1 kB, tx: 67.6 kB} frontend {rx: 67.6 kB, tx: 67.1 kB}
[ 25.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 38.1 kB, tx: 38.4 kB} frontend {rx: 38.4 kB, tx: 38.1 kB}
[ 25.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 26.3 kB, tx: 26.7 kB} frontend {rx: 26.7 kB, tx: 26.3 kB}
[ 25.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 18.2 kB, tx: 18.4 kB} frontend {rx: 18.4 kB, tx: 18.2 kB}
[ 25.171] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 22.6 kB, tx: 23.0 kB} frontend {rx: 23.0 kB, tx: 22.6 kB}
[ 30.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 45.1 kB, tx: 45.4 kB} frontend {rx: 45.4 kB, tx: 45.1 kB}
[ 30.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 44.0 kB, tx: 44.3 kB} frontend {rx: 44.3 kB, tx: 44.0 kB}
[ 30.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 65.1 kB, tx: 65.5 kB} frontend {rx: 65.5 kB, tx: 65.1 kB}
[ 35.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 35.9 kB, tx: 36.1 kB} frontend {rx: 36.1 kB, tx: 35.9 kB}
[ 35.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 36.4 kB, tx: 36.9 kB} frontend {rx: 36.9 kB, tx: 36.4 kB}
[ 35.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 28.9 kB, tx: 29.3 kB} frontend {rx: 29.3 kB, tx: 28.9 kB}
[ 35.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 40.9 kB, tx: 41.2 kB} frontend {rx: 41.2 kB, tx: 40.9 kB}
[ 40.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 31.0 kB, tx: 31.3 kB} frontend {rx: 31.3 kB, tx: 31.0 kB}
[ 40.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 28.5 kB, tx: 28.8 kB} frontend {rx: 28.8 kB, tx: 28.5 kB}
[ 40.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 40.4 kB, tx: 40.8 kB} frontend {rx: 40.8 kB, tx: 40.4 kB}
[ 45.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 60.1 kB, tx: 60.4 kB} frontend {rx: 60.4 kB, tx: 60.1 kB}
[ 45.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 64.8 kB, tx: 65.2 kB} frontend {rx: 65.2 kB, tx: 64.8 kB}
[ 45.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 39.2 kB, tx: 39.5 kB} frontend {rx: 39.5 kB, tx: 39.2 kB}
[ 45.170] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 23.9 kB, tx: 24.1 kB} frontend {rx: 24.1 kB, tx: 23.9 kB}
[ 50.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 23.0 kB, tx: 23.2 kB} frontend {rx: 23.2 kB, tx: 23.0 kB}
[ 50.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 41.4 kB, tx: 41.8 kB} frontend {rx: 41.8 kB, tx: 41.4 kB}
[ 50.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 50.8 kB, tx: 51.2 kB} frontend {rx: 51.2 kB, tx: 50.8 kB}
[ 55.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 29.2 kB, tx: 29.5 kB} frontend {rx: 29.5 kB, tx: 29.2 kB}
[ 55.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 27.8 kB, tx: 28.0 kB} frontend {rx: 28.0 kB, tx: 27.8 kB}
[ 55.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 21.7 kB, tx: 21.9 kB} frontend {rx: 21.9 kB, tx: 21.7 kB}
[ 55.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 30.0 kB, tx: 30.3 kB} frontend {rx: 30.3 kB, tx: 30.0 kB}
[ 60.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 43.4 kB, tx: 43.7 kB} frontend {rx: 43.7 kB, tx: 43.4 kB}
[ 60.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 44.7 kB, tx: 44.9 kB} frontend {rx: 44.9 kB, tx: 44.7 kB}
[ 60.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 30.6 kB, tx: 30.8 kB} frontend {rx: 30.8 kB, tx: 30.6 kB}
[ 65.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 41.9 kB, tx: 42.2 kB} frontend {rx: 42.2 kB, tx: 41.9 kB}
[ 65.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 38.1 kB, tx: 38.4 kB} frontend {rx: 38.4 kB, tx: 38.1 kB}
[ 65.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 42.8 kB, tx: 43.3 kB} frontend {rx: 43.3 kB, tx: 42.8 kB}
[ 65.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 28.9 kB, tx: 29.2 kB} frontend {rx: 29.2 kB, tx: 28.9 kB}
[ 70.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 43.6 kB, tx: 43.9 kB} frontend {rx: 43.9 kB, tx: 43.6 kB}
[ 70.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 30.3 kB, tx: 30.7 kB} frontend {rx: 30.7 kB, tx: 30.3 kB}
[ 70.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 58.9 kB, tx: 59.5 kB} frontend {rx: 59.5 kB, tx: 58.9 kB}
[ 75.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 32.5 kB, tx: 32.7 kB} frontend {rx: 32.7 kB, tx: 32.5 kB}
[ 75.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 31.7 kB, tx: 32.2 kB} frontend {rx: 32.2 kB, tx: 31.7 kB}
[ 75.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 42.2 kB, tx: 42.6 kB} frontend {rx: 42.6 kB, tx: 42.2 kB}
[ 75.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 61.7 kB, tx: 62.0 kB} frontend {rx: 62.0 kB, tx: 61.7 kB}
[ 80.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 48.3 kB, tx: 48.7 kB} frontend {rx: 48.7 kB, tx: 48.3 kB}
[ 80.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 54.5 kB, tx: 55.2 kB} frontend {rx: 55.2 kB, tx: 54.5 kB}
[ 80.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 21.3 kB, tx: 21.5 kB} frontend {rx: 21.5 kB, tx: 21.3 kB}
[ 85.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 47.7 kB, tx: 48.1 kB} frontend {rx: 48.1 kB, tx: 47.7 kB}
[ 85.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 27.0 kB, tx: 27.4 kB} frontend {rx: 27.4 kB, tx: 27.0 kB}
[ 85.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 72.9 kB, tx: 73.4 kB} frontend {rx: 73.4 kB, tx: 72.9 kB}
[ 85.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 78.6 kB, tx: 79.1 kB} frontend {rx: 79.1 kB, tx: 78.6 kB}
[ 90.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 58.7 kB, tx: 59.1 kB} frontend {rx: 59.1 kB, tx: 58.7 kB}
[ 90.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 48.9 kB, tx: 49.3 kB} frontend {rx: 49.3 kB, tx: 48.9 kB}
[ 90.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 63.4 kB, tx: 63.7 kB} frontend {rx: 63.7 kB, tx: 63.4 kB}
[ 95.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 59.0 kB, tx: 59.4 kB} frontend {rx: 59.4 kB, tx: 59.0 kB}
[ 95.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 65.9 kB, tx: 66.2 kB} frontend {rx: 66.2 kB, tx: 65.9 kB}
[ 95.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 48.9 kB, tx: 49.3 kB} frontend {rx: 49.3 kB, tx: 48.9 kB}
[ 95.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 27.5 kB, tx: 27.8 kB} frontend {rx: 27.8 kB, tx: 27.5 kB}
[ 100.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 56.3 kB, tx: 56.8 kB} frontend {rx: 56.8 kB, tx: 56.3 kB}
[ 100.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 25.8 kB, tx: 26.2 kB} frontend {rx: 26.2 kB, tx: 25.8 kB}
[ 100.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 33.9 kB, tx: 34.2 kB} frontend {rx: 34.2 kB, tx: 33.9 kB}
[ 105.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 30.7 kB, tx: 31.0 kB} frontend {rx: 31.0 kB, tx: 30.7 kB}
[ 105.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 64.6 kB, tx: 64.9 kB} frontend {rx: 64.9 kB, tx: 64.6 kB}
[ 105.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 49.0 kB, tx: 49.3 kB} frontend {rx: 49.3 kB, tx: 49.0 kB}
[ 105.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 36.2 kB, tx: 36.6 kB} frontend {rx: 36.6 kB, tx: 36.2 kB}
Our first share receives a bandwidth warning, after transferring more than 1 megabyte:
[ 105.189] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning 'tr7vpyrzvmh0'
[ 106.192] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 110.162] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 30.6 kB, tx: 30.9 kB} frontend {rx: 30.9 kB, tx: 30.6 kB}
[ 110.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 42.9 kB, tx: 43.3 kB} frontend {rx: 43.3 kB, tx: 42.9 kB}
[ 110.170] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 31.5 kB, tx: 31.7 kB} frontend {rx: 31.7 kB, tx: 31.5 kB}
[ 115.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 45.3 kB, tx: 45.7 kB} frontend {rx: 45.7 kB, tx: 45.3 kB}
[ 115.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 36.4 kB, tx: 36.8 kB} frontend {rx: 36.8 kB, tx: 36.4 kB}
[ 115.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 46.5 kB, tx: 46.9 kB} frontend {rx: 46.9 kB, tx: 46.5 kB}
[ 115.170] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 63.9 kB, tx: 64.4 kB} frontend {rx: 64.4 kB, tx: 63.9 kB}
More shares start receiving bandwidth warnings:
[ 115.230] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning 'dh3f3jj7zhig'
[ 116.575] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 120.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 58.4 kB, tx: 58.8 kB} frontend {rx: 58.8 kB, tx: 58.4 kB}
[ 120.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 30.5 kB, tx: 30.8 kB} frontend {rx: 30.8 kB, tx: 30.5 kB}
[ 120.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 29.9 kB, tx: 30.2 kB} frontend {rx: 30.2 kB, tx: 29.9 kB}
[ 120.180] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning '53z6mz4re7tu'
[ 122.733] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 125.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 47.0 kB, tx: 47.3 kB} frontend {rx: 47.3 kB, tx: 47.0 kB}
[ 125.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 55.5 kB, tx: 56.0 kB} frontend {rx: 56.0 kB, tx: 55.5 kB}
[ 125.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 49.6 kB, tx: 49.9 kB} frontend {rx: 49.9 kB, tx: 49.6 kB}
[ 125.170] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 17.6 kB, tx: 17.8 kB} frontend {rx: 17.8 kB, tx: 17.6 kB}
[ 125.211] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning 's0uzz1p7xjrr'
[ 126.117] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 130.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 38.8 kB, tx: 39.0 kB} frontend {rx: 39.0 kB, tx: 38.8 kB}
[ 130.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 76.6 kB, tx: 76.9 kB} frontend {rx: 76.9 kB, tx: 76.6 kB}
[ 130.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 50.1 kB, tx: 50.5 kB} frontend {rx: 50.5 kB, tx: 50.1 kB}
[ 130.178] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning '0evcupz5k410'
[ 130.921] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 135.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 32.8 kB, tx: 33.2 kB} frontend {rx: 33.2 kB, tx: 32.8 kB}
[ 135.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 34.7 kB, tx: 35.0 kB} frontend {rx: 35.0 kB, tx: 34.7 kB}
[ 135.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 38.9 kB, tx: 39.2 kB} frontend {rx: 39.2 kB, tx: 38.9 kB}
[ 135.170] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 51.4 kB, tx: 51.8 kB} frontend {rx: 51.8 kB, tx: 51.4 kB}
[ 140.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 52.8 kB, tx: 53.2 kB} frontend {rx: 53.2 kB, tx: 52.8 kB}
[ 140.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 33.1 kB, tx: 33.4 kB} frontend {rx: 33.4 kB, tx: 33.1 kB}
[ 140.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 35.6 kB, tx: 36.0 kB} frontend {rx: 36.0 kB, tx: 35.6 kB}
[ 145.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 27.5 kB, tx: 27.8 kB} frontend {rx: 27.8 kB, tx: 27.5 kB}
[ 145.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 24.6 kB, tx: 25.1 kB} frontend {rx: 25.1 kB, tx: 24.6 kB}
[ 145.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 30.1 kB, tx: 30.5 kB} frontend {rx: 30.5 kB, tx: 30.1 kB}
[ 145.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 24.7 kB, tx: 25.1 kB} frontend {rx: 25.1 kB, tx: 24.7 kB}
[ 150.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 72.0 kB, tx: 72.4 kB} frontend {rx: 72.4 kB, tx: 72.0 kB}
[ 150.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 31.8 kB, tx: 32.1 kB} frontend {rx: 32.1 kB, tx: 31.8 kB}
[ 150.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 43.7 kB, tx: 43.9 kB} frontend {rx: 43.9 kB, tx: 43.7 kB}
[ 155.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 49.4 kB, tx: 49.8 kB} frontend {rx: 49.8 kB, tx: 49.4 kB}
[ 155.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 46.4 kB, tx: 46.6 kB} frontend {rx: 46.6 kB, tx: 46.4 kB}
[ 155.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 50.7 kB, tx: 51.0 kB} frontend {rx: 51.0 kB, tx: 50.7 kB}
[ 155.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 58.5 kB, tx: 58.9 kB} frontend {rx: 58.9 kB, tx: 58.5 kB}
[ 160.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 43.0 kB, tx: 43.3 kB} frontend {rx: 43.3 kB, tx: 43.0 kB}
[ 160.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 66.0 kB, tx: 66.4 kB} frontend {rx: 66.4 kB, tx: 66.0 kB}
[ 160.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 31.5 kB, tx: 31.9 kB} frontend {rx: 31.9 kB, tx: 31.5 kB}
[ 165.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 60.0 kB, tx: 60.3 kB} frontend {rx: 60.3 kB, tx: 60.0 kB}
[ 165.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 47.3 kB, tx: 47.6 kB} frontend {rx: 47.6 kB, tx: 47.3 kB}
[ 165.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 41.1 kB, tx: 41.3 kB} frontend {rx: 41.3 kB, tx: 41.1 kB}
[ 165.170] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 37.2 kB, tx: 37.5 kB} frontend {rx: 37.5 kB, tx: 37.2 kB}
[ 165.216] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning '8k6dnu7x7ag0'
[ 165.930] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 170.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 43.1 kB, tx: 43.5 kB} frontend {rx: 43.5 kB, tx: 43.1 kB}
[ 170.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 45.4 kB, tx: 45.8 kB} frontend {rx: 45.8 kB, tx: 45.4 kB}
[ 170.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 58.0 kB, tx: 58.3 kB} frontend {rx: 58.3 kB, tx: 58.0 kB}
[ 175.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 63.5 kB, tx: 63.9 kB} frontend {rx: 63.9 kB, tx: 63.5 kB}
[ 175.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 45.0 kB, tx: 45.3 kB} frontend {rx: 45.3 kB, tx: 45.0 kB}
[ 175.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 35.0 kB, tx: 35.2 kB} frontend {rx: 35.2 kB, tx: 35.0 kB}
[ 175.171] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 22.8 kB, tx: 23.2 kB} frontend {rx: 23.2 kB, tx: 22.8 kB}
[ 180.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 42.3 kB, tx: 42.6 kB} frontend {rx: 42.6 kB, tx: 42.3 kB}
[ 180.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 39.6 kB, tx: 40.1 kB} frontend {rx: 40.1 kB, tx: 39.6 kB}
[ 180.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 53.1 kB, tx: 53.4 kB} frontend {rx: 53.4 kB, tx: 53.1 kB}
[ 185.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 74.1 kB, tx: 74.6 kB} frontend {rx: 74.6 kB, tx: 74.1 kB}
[ 185.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 71.0 kB, tx: 71.4 kB} frontend {rx: 71.4 kB, tx: 71.0 kB}
[ 185.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 78.8 kB, tx: 79.2 kB} frontend {rx: 79.2 kB, tx: 78.8 kB}
[ 185.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 37.8 kB, tx: 38.2 kB} frontend {rx: 38.2 kB, tx: 37.8 kB}
[ 185.213] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning '7u9szn30ikh0'
[ 186.862] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 190.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 43.3 kB, tx: 43.8 kB} frontend {rx: 43.8 kB, tx: 43.3 kB}
[ 190.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 39.6 kB, tx: 39.9 kB} frontend {rx: 39.9 kB, tx: 39.6 kB}
[ 190.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 38.6 kB, tx: 38.9 kB} frontend {rx: 38.9 kB, tx: 38.6 kB}
[ 195.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 44.0 kB, tx: 44.4 kB} frontend {rx: 44.4 kB, tx: 44.0 kB}
[ 195.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 45.2 kB, tx: 45.5 kB} frontend {rx: 45.5 kB, tx: 45.2 kB}
[ 195.170] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 67.1 kB, tx: 67.5 kB} frontend {rx: 67.5 kB, tx: 67.1 kB}
[ 195.172] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 44.4 kB, tx: 44.8 kB} frontend {rx: 44.8 kB, tx: 44.4 kB}
[ 200.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 23.7 kB, tx: 23.9 kB} frontend {rx: 23.9 kB, tx: 23.7 kB}
[ 200.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 46.7 kB, tx: 47.1 kB} frontend {rx: 47.1 kB, tx: 46.7 kB}
[ 200.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 48.7 kB, tx: 49.1 kB} frontend {rx: 49.1 kB, tx: 48.7 kB}
[ 205.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 47.8 kB, tx: 48.1 kB} frontend {rx: 48.1 kB, tx: 47.8 kB}
[ 205.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 57.2 kB, tx: 57.6 kB} frontend {rx: 57.6 kB, tx: 57.2 kB}
[ 205.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 47.7 kB, tx: 47.9 kB} frontend {rx: 47.9 kB, tx: 47.7 kB}
[ 205.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 38.9 kB, tx: 39.3 kB} frontend {rx: 39.3 kB, tx: 38.9 kB}
[ 210.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 37.5 kB, tx: 37.8 kB} frontend {rx: 37.8 kB, tx: 37.5 kB}
[ 210.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 15.3 kB, tx: 15.5 kB} frontend {rx: 15.5 kB, tx: 15.3 kB}
[ 210.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 41.3 kB, tx: 41.5 kB} frontend {rx: 41.5 kB, tx: 41.3 kB}
[ 215.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 28.0 kB, tx: 28.4 kB} frontend {rx: 28.4 kB, tx: 28.0 kB}
[ 215.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 42.5 kB, tx: 42.8 kB} frontend {rx: 42.8 kB, tx: 42.5 kB}
[ 215.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 47.6 kB, tx: 48.0 kB} frontend {rx: 48.0 kB, tx: 47.6 kB}
[ 215.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 43.4 kB, tx: 43.8 kB} frontend {rx: 43.8 kB, tx: 43.4 kB}
[ 220.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 19.4 kB, tx: 19.7 kB} frontend {rx: 19.7 kB, tx: 19.4 kB}
[ 220.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 37.1 kB, tx: 37.4 kB} frontend {rx: 37.4 kB, tx: 37.1 kB}
[ 220.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 69.1 kB, tx: 69.5 kB} frontend {rx: 69.5 kB, tx: 69.1 kB}
Our first share crosses the 2 megabyte boundary and the system limits its ability to transfer additional data by removing its dial service policy:
[ 220.195] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting 'dh3f3jj7zhig'
[ 220.211] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '7GerqbN5lVfcOng91J2J6I' for environment 'd.wJYlpt9'
[ 220.211] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for 'dh3f3jj7zhig'
[ 225.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 45.1 kB, tx: 45.5 kB} frontend {rx: 45.5 kB, tx: 45.1 kB}
[ 225.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 26.9 kB, tx: 27.3 kB} frontend {rx: 27.3 kB, tx: 26.9 kB}
[ 225.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: fNXXHVSuw backend {rx: 40.9 kB, tx: 41.0 kB} frontend {rx: 41.0 kB, tx: 40.9 kB}
[ 225.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 28.8 kB, tx: 29.1 kB} frontend {rx: 29.1 kB, tx: 28.8 kB}
[ 230.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 55.4 kB, tx: 55.8 kB} frontend {rx: 55.8 kB, tx: 55.4 kB}
[ 230.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 44.8 kB, tx: 45.2 kB} frontend {rx: 45.2 kB, tx: 44.8 kB}
[ 230.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 52.2 kB, tx: 52.5 kB} frontend {rx: 52.5 kB, tx: 52.2 kB}
[ 235.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 53.2 kB, tx: 53.6 kB} frontend {rx: 53.6 kB, tx: 53.2 kB}
[ 235.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 43.1 kB, tx: 43.4 kB} frontend {rx: 43.4 kB, tx: 43.1 kB}
[ 235.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: ONzzjVS0w backend {rx: 45.9 kB, tx: 46.2 kB} frontend {rx: 46.2 kB, tx: 45.9 kB}
[ 240.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 50.6 kB, tx: 51.0 kB} frontend {rx: 51.0 kB, tx: 50.6 kB}
[ 240.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: cNzzH4i0w backend {rx: 49.7 kB, tx: 50.0 kB} frontend {rx: 50.0 kB, tx: 49.7 kB}
[ 240.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 37.3 kB, tx: 37.6 kB} frontend {rx: 37.6 kB, tx: 37.3 kB}
More shares become limited and are prevented from transferring data. Notice the metrics output reducing in the logs. As more shares become limited, we're naturally seeing less data transfer occurring on the OpenZiti network:
[ 240.188] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting '0evcupz5k410'
[ 240.203] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '74f2gUotsC7DteqpsWrxp0' for environment 'd.wJYlpt9'
[ 240.203] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for '0evcupz5k410'
[ 245.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 28.2 kB, tx: 28.5 kB} frontend {rx: 28.5 kB, tx: 28.2 kB}
[ 245.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: dev-h4iuwD backend {rx: 47.3 kB, tx: 47.9 kB} frontend {rx: 47.9 kB, tx: 47.3 kB}
[ 245.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 48.2 kB, tx: 48.5 kB} frontend {rx: 48.5 kB, tx: 48.2 kB}
[ 245.194] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting '53z6mz4re7tu'
[ 245.196] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '1oo3LuBKxduKAs1wsKndtW' for environment 'd.wJYlpt9'
[ 245.197] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for '53z6mz4re7tu'
[ 250.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: dev-h4iuwD backend {rx: 33.2 kB, tx: 33.5 kB} frontend {rx: 33.5 kB, tx: 33.2 kB}
[ 250.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 48.2 kB, tx: 48.4 kB} frontend {rx: 48.4 kB, tx: 48.2 kB}
[ 250.191] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting 's0uzz1p7xjrr'
[ 250.194] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '7eB3ubrntSHxkeHBCGJcOY' for environment 'd.wJYlpt9'
[ 250.194] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for 's0uzz1p7xjrr'
[ 255.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 35.8 kB, tx: 36.0 kB} frontend {rx: 36.0 kB, tx: 35.8 kB}
[ 255.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: RZzXHVSuw backend {rx: 50.4 kB, tx: 50.6 kB} frontend {rx: 50.6 kB, tx: 50.4 kB}
[ 255.179] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting 'tr7vpyrzvmh0'
[ 255.182] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '1cURGP202D8n6fzpzWhcgK' for environment 'd.wJYlpt9'
[ 255.182] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for 'tr7vpyrzvmh0'
[ 260.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 37.3 kB, tx: 47.7 kB} frontend {rx: 47.7 kB, tx: 37.3 kB}
[ 260.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: MZXXjVi0w backend {rx: 69.2 kB, tx: 69.7 kB} frontend {rx: 69.7 kB, tx: 69.2 kB}
[ 265.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 59.7 kB, tx: 60.1 kB} frontend {rx: 60.1 kB, tx: 59.7 kB}
[ 270.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 46.9 kB, tx: 47.2 kB} frontend {rx: 47.2 kB, tx: 46.9 kB}
[ 275.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 32.9 kB, tx: 33.2 kB} frontend {rx: 33.2 kB, tx: 32.9 kB}
[ 280.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 43.2 kB, tx: 43.7 kB} frontend {rx: 43.7 kB, tx: 43.2 kB}
[ 285.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 50.0 kB, tx: 50.4 kB} frontend {rx: 50.4 kB, tx: 50.0 kB}
[ 290.162] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: SNXXjViuwU backend {rx: 51.8 kB, tx: 52.3 kB} frontend {rx: 52.3 kB, tx: 51.8 kB}
By this point, we're seeing very little traffic on the OpenZiti network:
[ 290.176] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting '8k6dnu7x7ag0'
[ 290.190] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '5UHCkXZabFHeWYHmF01Zoc' for environment 'd.wJYlpt9'
[ 290.191] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for '8k6dnu7x7ag0'
[ 295.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: iNzXj4S0r backend {rx: 44.0 kB, tx: 44.4 kB} frontend {rx: 44.4 kB, tx: 44.0 kB}
[ 295.178] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting '7u9szn30ikh0'
[ 295.181] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '3dBtc3v2G70aqqDSqujQOy' for environment 'd.wJYlpt9'
[ 295.181] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for '7u9szn30ikh0'
Notice the timestamps on the log messages. There have been no metrics messages for 60 seconds.
The limits agent runs a periodic process to look for limited resources to re-enable. It produces messages like this when there are no resources to re-enable:
[ 355.183] INFO zrok/controller/limits.(*Agent).relax: relaxing
[ 355.188] INFO zrok/controller/limits.(*Agent).relax: share 'dh3f3jj7zhig' still over limit
[ 355.192] INFO zrok/controller/limits.(*Agent).relax: share '0evcupz5k410' still over limit
[ 355.196] INFO zrok/controller/limits.(*Agent).relax: share '53z6mz4re7tu' still over limit
[ 355.199] INFO zrok/controller/limits.(*Agent).relax: share 's0uzz1p7xjrr' still over limit
[ 355.203] INFO zrok/controller/limits.(*Agent).relax: share 'tr7vpyrzvmh0' still over limit
[ 355.207] INFO zrok/controller/limits.(*Agent).relax: share '8k6dnu7x7ag0' still over limit
[ 355.220] INFO zrok/controller/limits.(*Agent).relax: share '7u9szn30ikh0' still over limit
[ 415.223] INFO zrok/controller/limits.(*Agent).relax: relaxing
[ 415.228] INFO zrok/controller/limits.(*Agent).relax: share 'dh3f3jj7zhig' still over limit
[ 415.232] INFO zrok/controller/limits.(*Agent).relax: share '0evcupz5k410' still over limit
[ 415.236] INFO zrok/controller/limits.(*Agent).relax: share '53z6mz4re7tu' still over limit
[ 415.240] INFO zrok/controller/limits.(*Agent).relax: share 's0uzz1p7xjrr' still over limit
[ 415.245] INFO zrok/controller/limits.(*Agent).relax: share 'tr7vpyrzvmh0' still over limit
[ 415.250] INFO zrok/controller/limits.(*Agent).relax: share '8k6dnu7x7ag0' still over limit
[ 415.253] INFO zrok/controller/limits.(*Agent).relax: share '7u9szn30ikh0' still over limit
Enough time has finally passed that the agent is able to remove the restrictions on some of the services:
[ 475.255] INFO zrok/controller/limits.(*Agent).relax: relaxing
[ 475.260] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing 'dh3f3jj7zhig'
[ 475.274] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '3LQG2ptwUxIuWtRzTLAqAc' for service 'nyKOLlxUWWbCzD7h9Jhjq' for identities '[rBayMvm7UI]'
[ 475.274] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for 'dh3f3jj7zhig'
[ 475.279] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing '0evcupz5k410'
[ 475.281] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '4BPqQhFsGGmoBsqFDIWlWA' for service '3WHJGqUdxkDtPYLgEL5V3q' for identities '[rBayMvm7UI]'
[ 475.281] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for '0evcupz5k410'
[ 475.285] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing '53z6mz4re7tu'
[ 475.287] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '64Kz6F7CluxH1drfyMkzDx' for service '2NiotGOyBHBEbFZwbTFJ2u' for identities '[rBayMvm7UI]'
[ 475.287] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for '53z6mz4re7tu'
[ 475.292] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing 's0uzz1p7xjrr'
[ 475.295] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '6MZ8i9sqvom96P70P24FJQ' for service 'KtK5E46HR93YIBrrwUlIN' for identities '[rBayMvm7UI]'
[ 475.295] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for 's0uzz1p7xjrr'
[ 475.299] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing 'tr7vpyrzvmh0'
[ 475.301] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '1kfuMP2APitf3qC2tsOC1b' for service '7jyiTZ0z2ediD5hZbxu7KH' for identities '[rBayMvm7UI]'
[ 475.301] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for 'tr7vpyrzvmh0'
[ 475.305] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing '8k6dnu7x7ag0'
[ 475.308] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '12jWOvjIIuvYRW9vXfkRKw' for service '2J0I9dPe2JGnY1GwjmM6n7' for identities '[rBayMvm7UI]'
[ 475.308] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for '8k6dnu7x7ag0'
[ 475.313] INFO zrok/controller/limits.(*Agent).relax: share '7u9szn30ikh0' still over limit
And notice that we're now starting to see traffic on those shares again:
[ 485.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 59.7 kB, tx: 60.0 kB} frontend {rx: 60.0 kB, tx: 59.7 kB}
[ 485.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 50.1 kB, tx: 50.4 kB} frontend {rx: 50.4 kB, tx: 50.1 kB}
[ 485.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 80.3 kB, tx: 80.7 kB} frontend {rx: 80.7 kB, tx: 80.3 kB}
[ 485.200] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning '8k6dnu7x7ag0'
[ 486.095] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 490.162] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 40.6 kB, tx: 40.9 kB} frontend {rx: 40.9 kB, tx: 40.6 kB}
[ 490.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 44.5 kB, tx: 45.0 kB} frontend {rx: 45.0 kB, tx: 44.5 kB}
[ 490.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 60.7 kB, tx: 61.1 kB} frontend {rx: 61.1 kB, tx: 60.7 kB}
[ 495.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 45.3 kB, tx: 45.6 kB} frontend {rx: 45.6 kB, tx: 45.3 kB}
[ 495.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 42.0 kB, tx: 42.4 kB} frontend {rx: 42.4 kB, tx: 42.0 kB}
[ 495.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 33.3 kB, tx: 33.8 kB} frontend {rx: 33.8 kB, tx: 33.3 kB}
[ 500.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 28.2 kB, tx: 28.5 kB} frontend {rx: 28.5 kB, tx: 28.2 kB}
[ 500.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 40.0 kB, tx: 40.3 kB} frontend {rx: 40.3 kB, tx: 40.0 kB}
[ 500.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 53.6 kB, tx: 54.0 kB} frontend {rx: 54.0 kB, tx: 53.6 kB}
[ 505.201] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 38.4 kB, tx: 38.6 kB} frontend {rx: 38.6 kB, tx: 38.4 kB}
[ 505.208] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 33.8 kB, tx: 34.2 kB} frontend {rx: 34.2 kB, tx: 33.8 kB}
[ 505.210] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 39.7 kB, tx: 40.0 kB} frontend {rx: 40.0 kB, tx: 39.7 kB}
[ 510.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 74.0 kB, tx: 74.5 kB} frontend {rx: 74.5 kB, tx: 74.0 kB}
[ 510.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 51.5 kB, tx: 51.8 kB} frontend {rx: 51.8 kB, tx: 51.5 kB}
[ 510.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 33.5 kB, tx: 33.9 kB} frontend {rx: 33.9 kB, tx: 33.5 kB}
[ 515.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 62.7 kB, tx: 63.0 kB} frontend {rx: 63.0 kB, tx: 62.7 kB}
[ 515.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 32.5 kB, tx: 32.9 kB} frontend {rx: 32.9 kB, tx: 32.5 kB}
[ 515.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 47.4 kB, tx: 47.7 kB} frontend {rx: 47.7 kB, tx: 47.4 kB}
[ 520.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 48.2 kB, tx: 48.5 kB} frontend {rx: 48.5 kB, tx: 48.2 kB}
[ 520.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 45.8 kB, tx: 46.1 kB} frontend {rx: 46.1 kB, tx: 45.8 kB}
[ 520.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 34.2 kB, tx: 34.4 kB} frontend {rx: 34.4 kB, tx: 34.2 kB}
[ 525.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 35.0 kB, tx: 35.4 kB} frontend {rx: 35.4 kB, tx: 35.0 kB}
[ 525.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 40.1 kB, tx: 40.4 kB} frontend {rx: 40.4 kB, tx: 40.1 kB}
[ 525.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 24.7 kB, tx: 25.0 kB} frontend {rx: 25.0 kB, tx: 24.7 kB}
[ 530.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 57.3 kB, tx: 57.9 kB} frontend {rx: 57.9 kB, tx: 57.3 kB}
[ 530.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 51.4 kB, tx: 51.7 kB} frontend {rx: 51.7 kB, tx: 51.4 kB}
[ 530.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 40.6 kB, tx: 41.0 kB} frontend {rx: 41.0 kB, tx: 40.6 kB}
[ 535.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 41.5 kB, tx: 41.9 kB} frontend {rx: 41.9 kB, tx: 41.5 kB}
[ 535.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 61.4 kB, tx: 61.9 kB} frontend {rx: 61.9 kB, tx: 61.4 kB}
[ 535.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 33.3 kB, tx: 33.6 kB} frontend {rx: 33.6 kB, tx: 33.3 kB}
[ 540.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 37.2 kB, tx: 37.5 kB} frontend {rx: 37.5 kB, tx: 37.2 kB}
[ 540.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 53.8 kB, tx: 54.3 kB} frontend {rx: 54.3 kB, tx: 53.8 kB}
[ 540.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 75.8 kB, tx: 76.4 kB} frontend {rx: 76.4 kB, tx: 75.8 kB}
[ 545.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 69.9 kB, tx: 70.2 kB} frontend {rx: 70.2 kB, tx: 69.9 kB}
[ 545.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 24.9 kB, tx: 25.2 kB} frontend {rx: 25.2 kB, tx: 24.9 kB}
[ 545.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 39.4 kB, tx: 39.6 kB} frontend {rx: 39.6 kB, tx: 39.4 kB}
[ 550.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 64.0 kB, tx: 64.3 kB} frontend {rx: 64.3 kB, tx: 64.0 kB}
[ 550.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 39.2 kB, tx: 39.6 kB} frontend {rx: 39.6 kB, tx: 39.2 kB}
[ 550.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 47.6 kB, tx: 47.9 kB} frontend {rx: 47.9 kB, tx: 47.6 kB}
[ 555.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 54.2 kB, tx: 54.8 kB} frontend {rx: 54.8 kB, tx: 54.2 kB}
[ 555.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 44.5 kB, tx: 44.8 kB} frontend {rx: 44.8 kB, tx: 44.5 kB}
[ 555.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 66.3 kB, tx: 66.7 kB} frontend {rx: 66.7 kB, tx: 66.3 kB}
[ 560.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 18.9 kB, tx: 19.2 kB} frontend {rx: 19.2 kB, tx: 18.9 kB}
[ 560.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 51.5 kB, tx: 51.8 kB} frontend {rx: 51.8 kB, tx: 51.5 kB}
[ 560.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 36.3 kB, tx: 36.7 kB} frontend {rx: 36.7 kB, tx: 36.3 kB}
[ 565.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 70.7 kB, tx: 71.0 kB} frontend {rx: 71.0 kB, tx: 70.7 kB}
[ 565.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 58.7 kB, tx: 59.1 kB} frontend {rx: 59.1 kB, tx: 58.7 kB}
[ 565.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 33.8 kB, tx: 34.0 kB} frontend {rx: 34.0 kB, tx: 33.8 kB}
[ 570.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 35.4 kB, tx: 35.7 kB} frontend {rx: 35.7 kB, tx: 35.4 kB}
[ 570.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 47.2 kB, tx: 47.6 kB} frontend {rx: 47.6 kB, tx: 47.2 kB}
[ 570.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 75.5 kB, tx: 75.8 kB} frontend {rx: 75.8 kB, tx: 75.5 kB}
[ 575.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 34.3 kB, tx: 34.6 kB} frontend {rx: 34.6 kB, tx: 34.3 kB}
[ 575.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 36.2 kB, tx: 36.6 kB} frontend {rx: 36.6 kB, tx: 36.2 kB}
[ 575.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 53.9 kB, tx: 54.2 kB} frontend {rx: 54.2 kB, tx: 53.9 kB}
[ 575.178] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning '53z6mz4re7tu'
[ 575.953] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 580.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 50.3 kB, tx: 50.7 kB} frontend {rx: 50.7 kB, tx: 50.3 kB}
[ 580.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 55.7 kB, tx: 56.1 kB} frontend {rx: 56.1 kB, tx: 55.7 kB}
[ 580.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 41.0 kB, tx: 41.3 kB} frontend {rx: 41.3 kB, tx: 41.0 kB}
[ 585.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 32.3 kB, tx: 32.6 kB} frontend {rx: 32.6 kB, tx: 32.3 kB}
[ 585.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 18.5 kB, tx: 18.8 kB} frontend {rx: 18.8 kB, tx: 18.5 kB}
[ 585.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 43.2 kB, tx: 43.6 kB} frontend {rx: 43.6 kB, tx: 43.2 kB}
[ 590.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 32.8 kB, tx: 33.0 kB} frontend {rx: 33.0 kB, tx: 32.8 kB}
[ 590.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 63.4 kB, tx: 63.7 kB} frontend {rx: 63.7 kB, tx: 63.4 kB}
[ 590.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 18.1 kB, tx: 18.3 kB} frontend {rx: 18.3 kB, tx: 18.1 kB}
[ 590.208] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning 'tr7vpyrzvmh0'
[ 591.168] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 595.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 34.1 kB, tx: 34.5 kB} frontend {rx: 34.5 kB, tx: 34.1 kB}
[ 595.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 46.0 kB, tx: 46.3 kB} frontend {rx: 46.3 kB, tx: 46.0 kB}
[ 595.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 49.1 kB, tx: 49.4 kB} frontend {rx: 49.4 kB, tx: 49.1 kB}
[ 600.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 34.0 kB, tx: 34.3 kB} frontend {rx: 34.3 kB, tx: 34.0 kB}
[ 600.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 46.6 kB, tx: 47.1 kB} frontend {rx: 47.1 kB, tx: 46.6 kB}
[ 600.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 23.2 kB, tx: 23.5 kB} frontend {rx: 23.5 kB, tx: 23.2 kB}
[ 600.189] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning 's0uzz1p7xjrr'
[ 600.949] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 605.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 61.5 kB, tx: 61.8 kB} frontend {rx: 61.8 kB, tx: 61.5 kB}
[ 605.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 38.3 kB, tx: 38.7 kB} frontend {rx: 38.7 kB, tx: 38.3 kB}
[ 605.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 35.1 kB, tx: 35.5 kB} frontend {rx: 35.5 kB, tx: 35.1 kB}
[ 610.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 37.7 kB, tx: 38.1 kB} frontend {rx: 38.1 kB, tx: 37.7 kB}
[ 610.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 50.4 kB, tx: 50.7 kB} frontend {rx: 50.7 kB, tx: 50.4 kB}
[ 610.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 27.9 kB, tx: 28.2 kB} frontend {rx: 28.2 kB, tx: 27.9 kB}
[ 615.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 25.7 kB, tx: 26.0 kB} frontend {rx: 26.0 kB, tx: 25.7 kB}
[ 615.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 32.2 kB, tx: 32.5 kB} frontend {rx: 32.5 kB, tx: 32.2 kB}
[ 615.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 47.2 kB, tx: 47.6 kB} frontend {rx: 47.6 kB, tx: 47.2 kB}
[ 620.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 35.7 kB, tx: 36.2 kB} frontend {rx: 36.2 kB, tx: 35.7 kB}
[ 620.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 60.6 kB, tx: 60.9 kB} frontend {rx: 60.9 kB, tx: 60.6 kB}
[ 620.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 51.8 kB, tx: 52.3 kB} frontend {rx: 52.3 kB, tx: 51.8 kB}
[ 620.178] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning 'dh3f3jj7zhig'
[ 620.929] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 625.162] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 41.6 kB, tx: 42.0 kB} frontend {rx: 42.0 kB, tx: 41.6 kB}
[ 625.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 46.4 kB, tx: 46.7 kB} frontend {rx: 46.7 kB, tx: 46.4 kB}
[ 625.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 48.4 kB, tx: 48.7 kB} frontend {rx: 48.7 kB, tx: 48.4 kB}
[ 630.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 29.3 kB, tx: 29.5 kB} frontend {rx: 29.5 kB, tx: 29.3 kB}
[ 630.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 49.7 kB, tx: 50.2 kB} frontend {rx: 50.2 kB, tx: 49.7 kB}
[ 630.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 47.7 kB, tx: 48.0 kB} frontend {rx: 48.0 kB, tx: 47.7 kB}
[ 635.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 69.2 kB, tx: 69.6 kB} frontend {rx: 69.6 kB, tx: 69.2 kB}
[ 635.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 45.8 kB, tx: 46.2 kB} frontend {rx: 46.2 kB, tx: 45.8 kB}
[ 635.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 38.6 kB, tx: 39.1 kB} frontend {rx: 39.1 kB, tx: 38.6 kB}
[ 640.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 41.7 kB, tx: 42.0 kB} frontend {rx: 42.0 kB, tx: 41.7 kB}
[ 640.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 27.5 kB, tx: 28.0 kB} frontend {rx: 28.0 kB, tx: 27.5 kB}
[ 640.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 28.4 kB, tx: 28.7 kB} frontend {rx: 28.7 kB, tx: 28.4 kB}
[ 645.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 39.8 kB, tx: 40.0 kB} frontend {rx: 40.0 kB, tx: 39.8 kB}
[ 645.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 52.2 kB, tx: 52.5 kB} frontend {rx: 52.5 kB, tx: 52.2 kB}
[ 645.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 39.3 kB, tx: 39.6 kB} frontend {rx: 39.6 kB, tx: 39.3 kB}
[ 645.300] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning '0evcupz5k410'
[ 647.031] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 650.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 36.3 kB, tx: 36.7 kB} frontend {rx: 36.7 kB, tx: 36.3 kB}
[ 650.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 50.7 kB, tx: 51.0 kB} frontend {rx: 51.0 kB, tx: 50.7 kB}
[ 650.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 48.5 kB, tx: 48.8 kB} frontend {rx: 48.8 kB, tx: 48.5 kB}
[ 655.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 24.6 kB, tx: 24.9 kB} frontend {rx: 24.9 kB, tx: 24.6 kB}
[ 655.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 45.6 kB, tx: 46.0 kB} frontend {rx: 46.0 kB, tx: 45.6 kB}
[ 655.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 51.8 kB, tx: 52.1 kB} frontend {rx: 52.1 kB, tx: 51.8 kB}
[ 655.284] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting '53z6mz4re7tu'
[ 655.299] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '64Kz6F7CluxH1drfyMkzDx' for environment 'd.wJYlpt9'
[ 655.299] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for '53z6mz4re7tu'
[ 660.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 70.9 kB, tx: 71.4 kB} frontend {rx: 71.4 kB, tx: 70.9 kB}
[ 660.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: CjNBpViur backend {rx: 49.0 kB, tx: 49.5 kB} frontend {rx: 49.5 kB, tx: 49.0 kB}
[ 660.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 36.2 kB, tx: 36.5 kB} frontend {rx: 36.5 kB, tx: 36.2 kB}
[ 665.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 48.3 kB, tx: 48.7 kB} frontend {rx: 48.7 kB, tx: 48.3 kB}
[ 665.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 42.1 kB, tx: 42.4 kB} frontend {rx: 42.4 kB, tx: 42.1 kB}
[ 665.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 47.4 kB, tx: 47.7 kB} frontend {rx: 47.7 kB, tx: 47.4 kB}
[ 670.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 40.2 kB, tx: 40.6 kB} frontend {rx: 40.6 kB, tx: 40.2 kB}
[ 670.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 62.1 kB, tx: 62.4 kB} frontend {rx: 62.4 kB, tx: 62.1 kB}
[ 675.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 13.8 kB, tx: 14.1 kB} frontend {rx: 14.1 kB, tx: 13.8 kB}
[ 675.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 36.6 kB, tx: 36.8 kB} frontend {rx: 36.8 kB, tx: 36.6 kB}
[ 675.168] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 35.3 kB, tx: 35.6 kB} frontend {rx: 35.6 kB, tx: 35.3 kB}
[ 680.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 55.3 kB, tx: 55.8 kB} frontend {rx: 55.8 kB, tx: 55.3 kB}
[ 680.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 46.6 kB, tx: 46.9 kB} frontend {rx: 46.9 kB, tx: 46.6 kB}
[ 685.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 56.2 kB, tx: 56.5 kB} frontend {rx: 56.5 kB, tx: 56.2 kB}
[ 685.172] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 42.4 kB, tx: 42.8 kB} frontend {rx: 42.8 kB, tx: 42.4 kB}
[ 685.175] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 46.8 kB, tx: 47.1 kB} frontend {rx: 47.1 kB, tx: 46.8 kB}
[ 690.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 72.4 kB, tx: 72.8 kB} frontend {rx: 72.8 kB, tx: 72.4 kB}
[ 690.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 58.5 kB, tx: 58.7 kB} frontend {rx: 58.7 kB, tx: 58.5 kB}
[ 695.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 21.1 kB, tx: 21.4 kB} frontend {rx: 21.4 kB, tx: 21.1 kB}
[ 695.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 41.5 kB, tx: 41.8 kB} frontend {rx: 41.8 kB, tx: 41.5 kB}
[ 695.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 49.5 kB, tx: 49.8 kB} frontend {rx: 49.8 kB, tx: 49.5 kB}
[ 700.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 28.6 kB, tx: 28.9 kB} frontend {rx: 28.9 kB, tx: 28.6 kB}
[ 700.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 58.6 kB, tx: 59.0 kB} frontend {rx: 59.0 kB, tx: 58.6 kB}
[ 700.193] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting 's0uzz1p7xjrr'
[ 700.208] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '6MZ8i9sqvom96P70P24FJQ' for environment 'd.wJYlpt9'
[ 700.208] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for 's0uzz1p7xjrr'
[ 705.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 40.1 kB, tx: 40.6 kB} frontend {rx: 40.6 kB, tx: 40.1 kB}
[ 705.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 55.7 kB, tx: 56.1 kB} frontend {rx: 56.1 kB, tx: 55.7 kB}
[ 705.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: OJyBpVS0w backend {rx: 46.0 kB, tx: 46.2 kB} frontend {rx: 46.2 kB, tx: 46.0 kB}
[ 710.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 20.4 kB, tx: 20.6 kB} frontend {rx: 20.6 kB, tx: 20.4 kB}
[ 710.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 74.0 kB, tx: 74.4 kB} frontend {rx: 74.4 kB, tx: 74.0 kB}
[ 710.178] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting 'tr7vpyrzvmh0'
[ 710.192] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '1kfuMP2APitf3qC2tsOC1b' for environment 'd.wJYlpt9'
[ 710.192] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for 'tr7vpyrzvmh0'
[ 715.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 59.7 kB, tx: 60.2 kB} frontend {rx: 60.2 kB, tx: 59.7 kB}
[ 715.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: LdZLhVSuw backend {rx: 67.5 kB, tx: 67.8 kB} frontend {rx: 67.8 kB, tx: 67.5 kB}
[ 720.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 43.7 kB, tx: 44.1 kB} frontend {rx: 44.1 kB, tx: 43.7 kB}
[ 725.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 36.4 kB, tx: 36.6 kB} frontend {rx: 36.6 kB, tx: 36.4 kB}
[ 725.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 72.2 kB, tx: 72.7 kB} frontend {rx: 72.7 kB, tx: 72.2 kB}
[ 730.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 41.0 kB, tx: 41.4 kB} frontend {rx: 41.4 kB, tx: 41.0 kB}
[ 735.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 31.3 kB, tx: 31.6 kB} frontend {rx: 31.6 kB, tx: 31.3 kB}
[ 735.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 39.1 kB, tx: 39.4 kB} frontend {rx: 39.4 kB, tx: 39.1 kB}
[ 740.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: dh3f3jj7zhig, circuit: 5TNBhVS0r backend {rx: 30.7 kB, tx: 31.0 kB} frontend {rx: 31.0 kB, tx: 30.7 kB}
[ 740.177] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting 'dh3f3jj7zhig'
[ 740.192] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '3LQG2ptwUxIuWtRzTLAqAc' for environment 'd.wJYlpt9'
[ 740.192] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for 'dh3f3jj7zhig'
[ 745.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 38.2 kB, tx: 38.5 kB} frontend {rx: 38.5 kB, tx: 38.2 kB}
[ 745.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: qwlLhVS0w backend {rx: 42.3 kB, tx: 42.7 kB} frontend {rx: 42.7 kB, tx: 42.3 kB}
[ 745.192] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting '8k6dnu7x7ag0'
[ 745.195] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '12jWOvjIIuvYRW9vXfkRKw' for environment 'd.wJYlpt9'
[ 745.195] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for '8k6dnu7x7ag0'
[ 750.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 36.4 kB, tx: 36.7 kB} frontend {rx: 36.7 kB, tx: 36.4 kB}
[ 760.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: fMNBpVi0w backend {rx: 57.5 kB, tx: 58.0 kB} frontend {rx: 58.0 kB, tx: 57.5 kB}
[ 760.178] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting '0evcupz5k410'
[ 760.194] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '4BPqQhFsGGmoBsqFDIWlWA' for environment 'd.wJYlpt9'
[ 760.194] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for '0evcupz5k410'
[ 820.195] INFO zrok/controller/limits.(*Agent).relax: relaxing
[ 820.200] ERROR zrok/controller/limits.(*Agent).checkShareLimit: expected 2 results; got '0' (from(bucket: "zrok")|> range(start: -5m0s)|> filter(fn: (r) => r["_measurement"] == "xfer")|> filter(fn: (r) => r["_field"] == "rx" or r["_field"] == "tx")|> filter(fn: (r) => r["namespace"] == "backend")|> filter(fn: (r) => r["share"] == "7u9szn30ikh0")|> sum())
[ 820.201] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing '7u9szn30ikh0'
[ 820.215] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '4yz1WSGg04BeARMuVkmxf7' for service '6FzYnK0RFJmT0rDSP1bzVE' for identities '[rBayMvm7UI]'
[ 820.215] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for '7u9szn30ikh0'
[ 820.219] INFO zrok/controller/limits.(*Agent).relax: share '53z6mz4re7tu' still over limit
[ 820.223] INFO zrok/controller/limits.(*Agent).relax: share 's0uzz1p7xjrr' still over limit
[ 820.227] INFO zrok/controller/limits.(*Agent).relax: share 'tr7vpyrzvmh0' still over limit
[ 820.231] INFO zrok/controller/limits.(*Agent).relax: share 'dh3f3jj7zhig' still over limit
[ 820.236] INFO zrok/controller/limits.(*Agent).relax: share '8k6dnu7x7ag0' still over limit
[ 820.240] INFO zrok/controller/limits.(*Agent).relax: share '0evcupz5k410' still over limit
[ 830.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 61.6 kB, tx: 61.9 kB} frontend {rx: 61.9 kB, tx: 61.6 kB}
[ 840.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 61.4 kB, tx: 61.7 kB} frontend {rx: 61.7 kB, tx: 61.4 kB}
[ 850.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 30.4 kB, tx: 30.7 kB} frontend {rx: 30.7 kB, tx: 30.4 kB}
[ 860.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 52.6 kB, tx: 53.0 kB} frontend {rx: 53.0 kB, tx: 52.6 kB}
[ 870.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 42.2 kB, tx: 42.6 kB} frontend {rx: 42.6 kB, tx: 42.2 kB}
[ 880.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 27.1 kB, tx: 27.4 kB} frontend {rx: 27.4 kB, tx: 27.1 kB}
[ 890.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 49.9 kB, tx: 50.3 kB} frontend {rx: 50.3 kB, tx: 49.9 kB}
[ 900.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 51.5 kB, tx: 51.8 kB} frontend {rx: 51.8 kB, tx: 51.5 kB}
[ 910.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 44.0 kB, tx: 44.5 kB} frontend {rx: 44.5 kB, tx: 44.0 kB}
[ 920.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 43.5 kB, tx: 43.8 kB} frontend {rx: 43.8 kB, tx: 43.5 kB}
[ 930.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 61.3 kB, tx: 61.7 kB} frontend {rx: 61.7 kB, tx: 61.3 kB}
[ 930.177] INFO zrok/controller/limits.(*shareWarningAction).HandleShare: warning '7u9szn30ikh0'
[ 931.057] INFO zrok/controller/limits.sendLimitWarningEmail: limit warning email sent to 'michael@quigley.com'
[ 940.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 30.2 kB, tx: 30.5 kB} frontend {rx: 30.5 kB, tx: 30.2 kB}
[ 950.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 56.2 kB, tx: 56.6 kB} frontend {rx: 56.6 kB, tx: 56.2 kB}
[ 960.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 73.1 kB, tx: 73.6 kB} frontend {rx: 73.6 kB, tx: 73.1 kB}
[ 970.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 35.1 kB, tx: 35.4 kB} frontend {rx: 35.4 kB, tx: 35.1 kB}
[ 980.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 63.6 kB, tx: 64.0 kB} frontend {rx: 64.0 kB, tx: 63.6 kB}
[ 990.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 46.6 kB, tx: 47.0 kB} frontend {rx: 47.0 kB, tx: 46.6 kB}
[1000.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 36.8 kB, tx: 37.3 kB} frontend {rx: 37.3 kB, tx: 36.8 kB}
[1010.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 24.5 kB, tx: 24.9 kB} frontend {rx: 24.9 kB, tx: 24.5 kB}
[1020.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 47.3 kB, tx: 47.7 kB} frontend {rx: 47.7 kB, tx: 47.3 kB}
[1030.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 29.6 kB, tx: 29.9 kB} frontend {rx: 29.9 kB, tx: 29.6 kB}
[1040.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 48.7 kB, tx: 49.1 kB} frontend {rx: 49.1 kB, tx: 48.7 kB}
[1050.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 7u9szn30ikh0, circuit: ESgSh4i0r backend {rx: 41.8 kB, tx: 42.0 kB} frontend {rx: 42.0 kB, tx: 41.8 kB}
[1050.284] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: limiting '7u9szn30ikh0'
[1050.300] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '4yz1WSGg04BeARMuVkmxf7' for environment 'd.wJYlpt9'
[1050.300] INFO zrok/controller/limits.(*shareLimitAction).HandleShare: removed dial service policy for '7u9szn30ikh0'
[1110.301] INFO zrok/controller/limits.(*Agent).relax: relaxing
[1110.307] ERROR zrok/controller/limits.(*Agent).checkShareLimit: expected 2 results; got '0' (from(bucket: "zrok")|> range(start: -5m0s)|> filter(fn: (r) => r["_measurement"] == "xfer")|> filter(fn: (r) => r["_field"] == "rx" or r["_field"] == "tx")|> filter(fn: (r) => r["namespace"] == "backend")|> filter(fn: (r) => r["share"] == "53z6mz4re7tu")|> sum())
[1110.307] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing '53z6mz4re7tu'
[1110.321] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy 'WxOiC60VDWvHHlbtcaJ6D' for service '2NiotGOyBHBEbFZwbTFJ2u' for identities '[rBayMvm7UI]'
[1110.321] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for '53z6mz4re7tu'
[1110.325] ERROR zrok/controller/limits.(*Agent).checkShareLimit: expected 2 results; got '0' (from(bucket: "zrok")|> range(start: -5m0s)|> filter(fn: (r) => r["_measurement"] == "xfer")|> filter(fn: (r) => r["_field"] == "rx" or r["_field"] == "tx")|> filter(fn: (r) => r["namespace"] == "backend")|> filter(fn: (r) => r["share"] == "s0uzz1p7xjrr")|> sum())
[1110.325] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing 's0uzz1p7xjrr'
[1110.327] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '2ubWYvKo2EOnrn1U4MQ4Cu' for service 'KtK5E46HR93YIBrrwUlIN' for identities '[rBayMvm7UI]'
[1110.327] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for 's0uzz1p7xjrr'
[1110.331] ERROR zrok/controller/limits.(*Agent).checkShareLimit: expected 2 results; got '0' (from(bucket: "zrok")|> range(start: -5m0s)|> filter(fn: (r) => r["_measurement"] == "xfer")|> filter(fn: (r) => r["_field"] == "rx" or r["_field"] == "tx")|> filter(fn: (r) => r["namespace"] == "backend")|> filter(fn: (r) => r["share"] == "tr7vpyrzvmh0")|> sum())
[1110.331] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing 'tr7vpyrzvmh0'
[1110.343] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '1Q2DMHZ9AFsBA8D2SNzC4l' for service '7jyiTZ0z2ediD5hZbxu7KH' for identities '[rBayMvm7UI]'
[1110.343] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for 'tr7vpyrzvmh0'
[1110.348] ERROR zrok/controller/limits.(*Agent).checkShareLimit: expected 2 results; got '0' (from(bucket: "zrok")|> range(start: -5m0s)|> filter(fn: (r) => r["_measurement"] == "xfer")|> filter(fn: (r) => r["_field"] == "rx" or r["_field"] == "tx")|> filter(fn: (r) => r["namespace"] == "backend")|> filter(fn: (r) => r["share"] == "dh3f3jj7zhig")|> sum())
[1110.349] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing 'dh3f3jj7zhig'
[1110.351] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy 'BrG9wKvUsajfhPkVfz44g' for service 'nyKOLlxUWWbCzD7h9Jhjq' for identities '[rBayMvm7UI]'
[1110.351] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for 'dh3f3jj7zhig'
[1110.356] ERROR zrok/controller/limits.(*Agent).checkShareLimit: expected 2 results; got '0' (from(bucket: "zrok")|> range(start: -5m0s)|> filter(fn: (r) => r["_measurement"] == "xfer")|> filter(fn: (r) => r["_field"] == "rx" or r["_field"] == "tx")|> filter(fn: (r) => r["namespace"] == "backend")|> filter(fn: (r) => r["share"] == "8k6dnu7x7ag0")|> sum())
[1110.356] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing '8k6dnu7x7ag0'
[1110.364] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy '1kbYWDgPbtk0JYjIPsRGOC' for service '2J0I9dPe2JGnY1GwjmM6n7' for identities '[rBayMvm7UI]'
[1110.364] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for '8k6dnu7x7ag0'
[1110.372] ERROR zrok/controller/limits.(*Agent).checkShareLimit: expected 2 results; got '0' (from(bucket: "zrok")|> range(start: -5m0s)|> filter(fn: (r) => r["_measurement"] == "xfer")|> filter(fn: (r) => r["_field"] == "rx" or r["_field"] == "tx")|> filter(fn: (r) => r["namespace"] == "backend")|> filter(fn: (r) => r["share"] == "0evcupz5k410")|> sum())
[1110.372] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: relaxing '0evcupz5k410'
[1110.374] INFO zrok/controller/zrokEdgeSdk.CreateServicePolicyDial: created dial service policy 'E30643mY9M6vU6bQSQHa9' for service '3WHJGqUdxkDtPYLgEL5V3q' for identities '[rBayMvm7UI]'
[1110.374] INFO zrok/controller/limits.(*shareRelaxAction).HandleShare: added dial service policy for '0evcupz5k410'
[1110.378] INFO zrok/controller/limits.(*Agent).relax: share '7u9szn30ikh0' still over limit
[1115.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: gaSGp4i0r backend {rx: 3.4 kB, tx: 3.4 kB} frontend {rx: 3.4 kB, tx: 3.4 kB}
[1120.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: gaSGp4i0r backend {rx: 26.5 kB, tx: 26.7 kB} frontend {rx: 26.7 kB, tx: 26.5 kB}
[1120.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: nESTh4iur backend {rx: 65.6 kB, tx: 66.1 kB} frontend {rx: 66.1 kB, tx: 65.6 kB}
[1120.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: GGiTp4S0w backend {rx: 53.5 kB, tx: 54.0 kB} frontend {rx: 54.0 kB, tx: 53.5 kB}
[1125.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: gaSGp4i0r backend {rx: 43.7 kB, tx: 44.1 kB} frontend {rx: 44.1 kB, tx: 43.7 kB}
[1125.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: fSiGh4iur backend {rx: 17.8 kB, tx: 18.0 kB} frontend {rx: 18.0 kB, tx: 17.8 kB}
[1125.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 0evcupz5k410, circuit: k4SGhVSuw backend {rx: 51.9 kB, tx: 52.3 kB} frontend {rx: 52.3 kB, tx: 51.9 kB}
[1130.163] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: gaSGp4i0r backend {rx: 50.9 kB, tx: 51.2 kB} frontend {rx: 51.2 kB, tx: 50.9 kB}
[1130.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: GGiTp4S0w backend {rx: 48.6 kB, tx: 49.0 kB} frontend {rx: 49.0 kB, tx: 48.6 kB}
[1130.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: fSiGh4iur backend {rx: 37.8 kB, tx: 38.0 kB} frontend {rx: 38.0 kB, tx: 37.8 kB}
[1135.164] INFO zrok/controller/metrics.(*influxWriter).Handle: share: tr7vpyrzvmh0, circuit: gaSGp4i0r backend {rx: 57.6 kB, tx: 58.1 kB} frontend {rx: 58.1 kB, tx: 57.6 kB}
[1135.166] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: fSiGh4iur backend {rx: 43.6 kB, tx: 44.1 kB} frontend {rx: 44.1 kB, tx: 43.6 kB}
[1135.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: GGiTp4S0w backend {rx: 51.0 kB, tx: 51.4 kB} frontend {rx: 51.4 kB, tx: 51.0 kB}
[1140.165] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 8k6dnu7x7ag0, circuit: nESTh4iur backend {rx: 28.4 kB, tx: 28.6 kB} frontend {rx: 28.6 kB, tx: 28.4 kB}
[1140.167] INFO zrok/controller/metrics.(*influxWriter).Handle: share: 53z6mz4re7tu, circuit: fSiGh4iur backend {rx: 31.6 kB, tx: 32.0 kB} frontend {rx: 32.0 kB, tx: 31.6 kB}
[1140.169] INFO zrok/controller/metrics.(*influxWriter).Handle: share: s0uzz1p7xjrr, circuit: GGiTp4S0w backend {rx: 23.8 kB, tx: 24.2 kB} frontend {rx: 24.2 kB, tx: 23.8 kB}
[1141.514] INFO zrok/controller/zrokEdgeSdk.DeleteServiceEdgeRouterPolicy: deleted service edge router policy '2RIKOBMOckfbI2xMSLAKxC' for environment 'd.wJYlpt9'
[1141.517] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '1kbYWDgPbtk0JYjIPsRGOC' for environment 'd.wJYlpt9'
[1141.519] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '4vT5eEPahgWEVdAuKN91Sd' for environment 'd.wJYlpt9'
[1141.521] INFO zrok/controller/zrokEdgeSdk.DeleteConfig: deleted config '5nG9jM8VNl0uBFcRRt3AvI' for 'd.wJYlpt9'
[1141.522] INFO zrok/controller/zrokEdgeSdk.DeleteService: deleted service '2J0I9dPe2JGnY1GwjmM6n7' for environment 'd.wJYlpt9'
[1141.599] INFO zrok/controller/zrokEdgeSdk.DeleteServiceEdgeRouterPolicy: deleted service edge router policy '2CM03d1cNpG4rma38BLzCQ' for environment 'd.wJYlpt9'
[1141.602] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy 'WxOiC60VDWvHHlbtcaJ6D' for environment 'd.wJYlpt9'
[1141.635] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy 'RRfDaA5kjCqUBVC9LvN1H' for environment 'd.wJYlpt9'
[1141.639] INFO zrok/controller/zrokEdgeSdk.DeleteConfig: deleted config '6U3XDGnBjtONN5H6pUze12' for 'd.wJYlpt9'
[1141.645] INFO zrok/controller/zrokEdgeSdk.DeleteService: deleted service '2NiotGOyBHBEbFZwbTFJ2u' for environment 'd.wJYlpt9'
[1141.701] INFO zrok/controller/zrokEdgeSdk.DeleteServiceEdgeRouterPolicy: deleted service edge router policy '2ZnnIXSTQ3Zscha1kykqQr' for environment 'd.wJYlpt9'
[1141.704] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy 'BrG9wKvUsajfhPkVfz44g' for environment 'd.wJYlpt9'
[1141.706] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '1xF4ky6cDJm63tzlNTqoLC' for environment 'd.wJYlpt9'
[1141.707] INFO zrok/controller/zrokEdgeSdk.DeleteConfig: deleted config '76iBDASRcxOmGtdwjVHo26' for 'd.wJYlpt9'
[1141.708] INFO zrok/controller/zrokEdgeSdk.DeleteService: deleted service 'nyKOLlxUWWbCzD7h9Jhjq' for environment 'd.wJYlpt9'
[1141.926] INFO zrok/controller/zrokEdgeSdk.DeleteServiceEdgeRouterPolicy: deleted service edge router policy '3xAG26zA9yska3LeZQUJ3N' for environment 'd.wJYlpt9'
[1141.927] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: did not find a service policy
[1141.929] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '1mabRt9jefSe52CJh6FmhB' for environment 'd.wJYlpt9'
[1141.931] INFO zrok/controller/zrokEdgeSdk.DeleteConfig: deleted config '2gid15nP0GIUVuaFQ15GWV' for 'd.wJYlpt9'
[1141.932] INFO zrok/controller/zrokEdgeSdk.DeleteService: deleted service '6FzYnK0RFJmT0rDSP1bzVE' for environment 'd.wJYlpt9'
[1142.053] INFO zrok/controller/zrokEdgeSdk.DeleteServiceEdgeRouterPolicy: deleted service edge router policy '2nMZaiChQAPpFnblNn1ljP' for environment 'd.wJYlpt9'
[1142.056] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '1Q2DMHZ9AFsBA8D2SNzC4l' for environment 'd.wJYlpt9'
[1142.058] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '6RwWEoIsb8gBVKJfZP3ur3' for environment 'd.wJYlpt9'
[1142.064] INFO zrok/controller/zrokEdgeSdk.DeleteConfig: deleted config '1FnBhnGNXDe58dwTpbFc1x' for 'd.wJYlpt9'
[1142.066] INFO zrok/controller/zrokEdgeSdk.DeleteService: deleted service '7jyiTZ0z2ediD5hZbxu7KH' for environment 'd.wJYlpt9'
[1142.320] INFO zrok/controller/zrokEdgeSdk.DeleteServiceEdgeRouterPolicy: deleted service edge router policy '2AqCUMqNtarmglOfhvnkI' for environment 'd.wJYlpt9'
[1142.324] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy 'E30643mY9M6vU6bQSQHa9' for environment 'd.wJYlpt9'
[1142.326] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '4V8FsgCt63ySkG2pFWG5fz' for environment 'd.wJYlpt9'
[1142.329] INFO zrok/controller/zrokEdgeSdk.DeleteConfig: deleted config '19cyxfHo32R6fhVsYHZ84g' for 'd.wJYlpt9'
[1142.330] INFO zrok/controller/zrokEdgeSdk.DeleteService: deleted service '3WHJGqUdxkDtPYLgEL5V3q' for environment 'd.wJYlpt9'
[1142.701] INFO zrok/controller/zrokEdgeSdk.DeleteServiceEdgeRouterPolicy: deleted service edge router policy '2CGCz8dcquNvZC0ZUwDZ5F' for environment 'd.wJYlpt9'
[1142.704] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '2ubWYvKo2EOnrn1U4MQ4Cu' for environment 'd.wJYlpt9'
[1142.708] INFO zrok/controller/zrokEdgeSdk.DeleteServicePolicy: deleted service policy '6oohOQFEo75yl9vnIbyzdj' for environment 'd.wJYlpt9'
[1142.709] INFO zrok/controller/zrokEdgeSdk.DeleteConfig: deleted config '4AN4sOtdQv99uHmFn3erx4' for 'd.wJYlpt9'
[1142.710] INFO zrok/controller/zrokEdgeSdk.DeleteService: deleted service 'KtK5E46HR93YIBrrwUlIN' for environment 'd.wJYlpt9'