Currently rclone check supports matching two file trees by sizes and hashes.
This change adds support for SUM files produced by GNU utilities like sha1sum.
Fixes#1005
Note: checksum by default checks, hashsum by default prints sums.
New flag is named "--checkfile" but carries hash name.
Summary of introduced command forms:
```
rclone check sums.sha1 remote:path --checkfile sha1
rclone checksum sha1 sums.sha1 remote:path
rclone hashsum sha1 remote:path --checkfile sums.sha1
rclone sha1sum remote:path --checkfile sums.sha1
rclone md5sum remote:path --checkfile sums.md5
```
There was no easy way to automatically test the end-to-end functionality
of commands, flags, environment variables etc.
The need for end-to-end testing was highlighted by the issues fixed
in #5341. There was no automated test to continually verify current
behaviour, nor a framework to quickly test the correctness of the fixes.
This change adds an end-to-end testing framework in the cmdtest folder.
It has some simple examples in func TestCmdTest in cmdtest_test.go. The
tests should be readable by anybody familiar with rclone and look like
this:
// Test the rclone version command with debug logging (-vv)
out, err = rclone("version", "-vv")
if assert.NoError(t, err) {
assert.Contains(t, out, "rclone v")
assert.Contains(t, out, "os/version:")
assert.Contains(t, out, " DEBUG : ")
}
The end-to-end tests are executed just like the Go unit tests, that is:
go test ./cmdtest -v
The change also contains a thorough test of environment variables in
environment_test.go.
Thanks to @ncw for encouragement and introduction to the TestMain trick.
Some environment variables didn’t behave like their corresponding
command line flags. The affected flags were --stats, --log-level,
--separator, --multi-tread-streams, --rc-addr, --rc-user and --rc-pass.
Example:
RCLONE_STATS='10s'
rclone check remote: remote: --progress
# Expected: rclone check remote: remote: --progress –-stats=10s
# Actual: rclone check remote: remote: --progress
Remote specific options set by environment variables was overruled by
less specific backend options set by environment variables. Example:
RCLONE_DRIVE_USE_TRASH='false'
RCLONE_CONFIG_MYDRIVE_USE_TRASH='true'
rclone deletefile myDrive:my-test-file
# Expected: my-test-file is recoverable in the trash folder
# Actual: my-test-file is permanently deleted (not recoverable)
Backend specific options set by environment variables was overruled by
general backend options set by environment variables. Example:
RCLONE_SKIP_LINKS='true'
RCLONE_LOCAL_SKIP_LINKS='false'
rclone lsd local:
# Expected result: Warnings when symlinks are skipped
# Actual result: No warnings when symlinks are skipped
# That is RCLONE_SKIP_LINKS takes precedence
The above issues have been fixed.
The debug logging (-vv) has been enhanced to show when flags are set by
environment variables.
The documentation has been enhanced with details on the precedence of
configuration options.
See pull request #5341 for more information.
- Unify all hash names as lowercase alphanumerics without punctuation.
- Legacy names continue to work but disappear from docs, they can be depreciated or dropped later.
- Make rclone hashsum print supported hash list in case of wrong spelling.
- Update documentation.
Fixes#5071Fixes#4841
