Before this change it wasn't possible to see where transfers were
going from and to in core/stats and core/transferred.
When use in rclone mount in particular this made interpreting the
stats very hard.
Before this change StatsInfo.ResetCounters() and stopAverageLoop()
(when called from time.AfterFunc) could race on StatsInfo.average.
This was because the deferred stopAverageLoop accessed
StatsInfo.average without locking.
For some reason this only ever happened on macOS. This caused the CI
to fail on macOS thus causing the macOS builds not to appear.
This commit fixes the problem with a bit of extra locking.
It also renames all StatsInfo methods that should be called without
the lock to start with an initial underscore as this is the convention
we use elsewhere.
Fixes#7567
Before this fix we were not counting transferred files nor transferred
bytes for server side moves/copies.
If the server side move/copy has been marked as a transfer and not a
checker then this accounts transferred files and transferred bytes.
The transferred bytes are not accounted to the network though so this
should not affect the network stats.
Currently, the average transfer speed will stop calculating 1 minute
after the last queued transfer completes. This causes the average to
stop calculating when checking is slow and the transfer queue becomes
empty.
This change will require all checks to complete before stopping the
average speed calculation.
Before this change we showed both server side moves and server side
copies as bytes transferred.
This made a nice easy to use stats display, but also caused confusion
for users who saw unrealistic transfer times. It also caused a problem
with --max-transfer and chunker which renames each chunk after
uploading which was counted as a transfer byte.
This patch instead accounts the server side move and copy statistics
as a seperate lines in the stats display which will only appear if
there are any server side moves / copies. This is also output in the
rc.
This gives users something to look at when transfers are running which
was the point of the original change but it now means that transfer
bytes represents data transfers through this rclone instance only.
Fixes#7183
The SIGUSR2 signal handler for bandwidth limits currently only starts
if rclone is started at a time when a bandwidth limit applies. This
means that if rclone starts _outside_ such a time, i.e. with no
bandwidth limits, then enters a time where bandwidth limits do apply,
it will not be possible to use SIGUSR2 to toggle it.
This fixes that by always starting the signal handler, but only
toggling the limiter if there is a bandwidth limit configured.
In 04aa6969a4 we updated the displayed speed to be a rolling
average in core/stats and the progress output but we didn't update the
Prometheus metrics.
This patch updates the Prometheus metrics too.
Fixes#7053
Before this change, all types of checkers showed "checking" after the
file name despite the fact that not all of them were checking.
After this change, they can show
- checking
- deleting
- hashing
- importing
- listing
- merging
- moving
- renaming
See: https://forum.rclone.org/t/what-is-rclone-checking-during-a-purge/35931/
No need to report hours, minutes, and even seconds when the
ETA is several years, e.g. "292y24w3d23h47m16s". Now only
reports the 3 most significant units, sacrificing precision,
e.g. "292y24w3d", "24w3d23h", "3d23h47m", "23h47m16s".
Fixes#6381
Integer overflow would lead to ETA such as "-255y7w4h11m22s966ms",
as reported in #6381. Now the value will be clipped at the maximum
"292y24w3d23h47m16s", and it will be shown as infinity.
This was caused by nested calls to NewTransfer/Done.
This fixes the problem by only incrementing transfers if the remote is
present in the transferMap which means we only increment it once.
Whenever transfer.Account() is called, a new goroutine acc.averageLoop()
is started. This goroutine exits only when the channel acc.exit is closed.
acc.exit is closed when acc.Done() is called, which happens during tr.Done().
However, if tr.Reset is called during a copy low level retry, it replaces
the tr.acc, without calling acc.Done(), which results in the goroutine
mentioned above never exiting.
This commit calls acc.Done() during a tr.Reset()
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.
This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
Before this fix, on Windows, the --bwlimit would max out at 2.5Gbps
even when set to 10 Gbps.
This turned out to be because of the maximum token bucket size.
This fix scales up the token bucket size linearly above a bwlimit of
2Gbps.
Fixes#5507
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
This patch adds the missing stats to the output of core/stats
- totalChecks
- totalTransfers
- totalBytes
- eta
This now includes enough information to rebuild the normal stats
output from rclone including percentage completions and ETAs.
Fixes#5116
Before this change the core bandwidth limit was limited to upload or
download value if the other value was off.
This fix only applies a core bandwidth limit when both values are set.
This change decreases the edge limiter burst size which dramatically
increases the smoothness of the bandwidth limiting.
The core bandwidth limiter remains with a large burst so it isn't
affected by double rate limiting on the edge limiters.
See: #4395
See: https://forum.rclone.org/t/bwlimit-is-not-really-smooth/20947
This change uses the bwlimit code to apply limits to the receive and
transmit data functions in the HTTP Transport.
This means that all HTTP transactions will have limiting applied -
this includes listings for example.
For HTTP based transorts this makes the limiting in Accounting
redundant and possibly counter productive
