Commit Graph

7020 Commits

Author SHA1 Message Date
d05cf6aba8 Add partev to contributors 2021-07-07 16:59:51 +01:00
76a3fef24d Add Xuanchen Wu to contributors 2021-07-07 16:59:51 +01:00
b40d9bd4c4 cmd: add hashSUM file support (#5352)
Currently rclone check supports matching two file trees by sizes and hashes.
This change adds support for SUM files produced by GNU utilities like sha1sum.

Fixes #1005 

Note: checksum by default checks, hashsum by default prints sums.
New flag is named "--checkfile" but carries hash name.
Summary of introduced command forms:

```
rclone check sums.sha1 remote:path --checkfile sha1
rclone checksum sha1 sums.sha1 remote:path             
rclone hashsum sha1 remote:path --checkfile sums.sha1
rclone sha1sum remote:path --checkfile sums.sha1
rclone md5sum remote:path --checkfile sums.md5
```
2021-07-07 18:34:16 +03:00
4680c0776d backend/local: skip entries removed concurrently with List() (#5297)
This change fixes the bug described below:
if a file is removed while the local backend List() runs,
the call will flag an accounting error.
The bug manifests itself if local backend is the Sync target
due to intrinsic concurrency.
The odds to hit this bug depend on --checkers and --transfers.
Chunker over local backend is affected even more because
updating a composite object with a smaller size content
translates into removing chunks on the underlying file system
and involves a number of List() calls.
2021-07-07 16:50:19 +03:00
fb305b5976 fichier: check that we actually got a download token and retry if we didn't 2021-07-06 14:58:50 +02:00
5e91b93e59 cmdtest: end-to-end test for commands, flags and environment variables
There was no easy way to automatically test the end-to-end functionality
of commands, flags, environment variables etc.

The need for end-to-end testing was highlighted by the issues fixed
in #5341. There was no automated test to continually verify current
behaviour, nor a framework to quickly test the correctness of the fixes.

This change adds an end-to-end testing framework in the cmdtest folder.
It has some simple examples in func TestCmdTest in cmdtest_test.go. The
tests should be readable by anybody familiar with rclone and look like
this:

    // Test the rclone version command with debug logging (-vv)
    out, err = rclone("version", "-vv")
    if assert.NoError(t, err) {
        assert.Contains(t, out, "rclone v")
        assert.Contains(t, out, "os/version:")
        assert.Contains(t, out, " DEBUG : ")
    }

The end-to-end tests are executed just like the Go unit tests, that is:

    go test ./cmdtest -v

The change also contains a thorough test of environment variables in
environment_test.go.

Thanks to @ncw for encouragement and introduction to the TestMain trick.
2021-07-05 16:38:20 +01:00
58c99427b3 config: fixed issues with flags/options set by environment vars.
Some environment variables didn’t behave like their corresponding
command line flags. The affected flags were --stats, --log-level,
--separator, --multi-tread-streams, --rc-addr, --rc-user and --rc-pass.
Example:

    RCLONE_STATS='10s'
    rclone check remote: remote: --progress
    # Expected: rclone check remote: remote: --progress –-stats=10s
    # Actual: rclone check remote: remote: --progress

Remote specific options set by environment variables was overruled by
less specific backend options set by environment variables. Example:

    RCLONE_DRIVE_USE_TRASH='false'
    RCLONE_CONFIG_MYDRIVE_USE_TRASH='true'
    rclone deletefile myDrive:my-test-file
    # Expected: my-test-file is recoverable in the trash folder
    # Actual: my-test-file is permanently deleted (not recoverable)

Backend specific options set by environment variables was overruled by
general backend options set by environment variables. Example:

    RCLONE_SKIP_LINKS='true'
    RCLONE_LOCAL_SKIP_LINKS='false'
    rclone lsd local:
    # Expected result: Warnings when symlinks are skipped
    # Actual result: No warnings when symlinks are skipped
    # That is RCLONE_SKIP_LINKS takes precedence

The above issues have been fixed.

The debug logging (-vv) has been enhanced to show when flags are set by
environment variables.

The documentation has been enhanced with details on the precedence of
configuration options.

See pull request #5341 for more information.
2021-07-05 16:38:20 +01:00
fee0abf513 docs: add note about use of user and logname environment variables for current username 2021-07-05 16:31:16 +01:00
40024990b7 fs/operations: Don't update timestamps of files in --compare-dest 2021-07-05 16:29:44 +01:00
04aa6969a4 accounting: calculate rolling average speed 2021-07-05 16:27:33 +01:00
d2050523de accounting: fix startTime of statsGroups.sum 2021-07-05 16:27:33 +01:00
1cc6dd349e Add google search widget to rclone.org 2021-07-05 16:21:36 +01:00
721bae11c3 docs: ease contribution for beginners in Go, Git and GitHub
Improved/added steps to:
 * Install Git with basic setup
 * Use both SSH and HTTPS for the git origin
 * Install Go and verify the GOPATH
 * Update the forked master
 * Find a popular editor for Go
2021-07-05 16:03:53 +01:00
b439199578 azureblob: Fix typo in Azure Blob help
Change the command to create RBAC file to the correct one
`az ad sp create-for-rbac`
Add the link to the command documentation
https://docs.microsoft.com/en-us/cli/azure/ad/sp?view=azure-cli-latest#az_ad_sp_create_for_rbac
2021-07-05 15:58:41 +01:00
0bfd6f793b docs: replace OSX with macOS 2021-07-05 14:51:00 +01:00
76ea716abf ftp: make upload error 250 indicate success
Some servers seem to send return code 250 to indicate successful
upload - previously rclone was treating this as an error.

See: https://forum.rclone.org/t/transfer-on-mega-in-ftp-mode-is-not-working/24642/
2021-07-05 10:35:02 +01:00
e635f4c0be fs: make --dump imply -vv (#5418) 2021-06-23 00:32:26 +08:00
0cb973f127 onedrive: Make link return direct download link (#5417)
Co-authored-by: Cnly <minecnly@gmail.com>
2021-06-22 21:25:08 +08:00
96ace599a8 fs: fix logging level mentioned in docs of Logf 2021-06-21 23:30:26 +08:00
80bccacd83 fs: split overgrown fs.go (#5405)
Nothing is added or removed and no package is renamed by this change.
Just rearrange definitions between source files in the fs directory.

New source files:
- types.go      Filesystem types and interfaces
- features.go   Features and optional interfaces
- registry.go   Filesystem registry and backend options
- newfs.go      NewFs and its helpers
- configmap.go  Getters and Setters for ConfigMap
- pacer.go      Pacer with logging and calculator
The final fs.go contains what is left.

Also rename options.go to open_options.go
to dissociate from registry options.
2021-06-14 14:42:49 +03:00
3349b055f5 fichier: fix move of files in the same directory
See: https://forum.rclone.org/t/1fichier-rclone-does-not-allow-to-rename-files-and-folders-when-you-mount-a-1fichier-disk-drive/24726/24
2021-06-11 14:21:23 +01:00
bef0c23e00 fichier: make error messages report text from the API
See: https://forum.rclone.org/t/1fichier-rclone-does-not-allow-to-rename-files-and-folders-when-you-mount-a-1fichier-disk-drive/24726/24
2021-06-11 14:21:23 +01:00
84201ed891 zoho: improve wording for region - fixes #5377 2021-06-11 14:21:23 +01:00
04608428bf Add Florian Penzkofer to contributors 2021-06-11 14:21:23 +01:00
6aaa06d7be Add darrenrhs to contributors 2021-06-11 14:21:23 +01:00
e53bad5353 Add Reid Buzby to contributors 2021-06-11 14:21:23 +01:00
f5397246eb Add Chris Lu to contributors 2021-06-11 14:21:23 +01:00
b8b73f2656 Add database64128 to contributors 2021-06-11 14:21:23 +01:00
96b67ce0ec Add Tyson Moore to contributors 2021-06-11 14:21:23 +01:00
e2beeffd76 Add Tom to contributors 2021-06-11 14:21:23 +01:00
30b949642d Add acsfer to contributors 2021-06-11 14:21:23 +01:00
92b3518c78 fichier: support downloading password protected files and folders 2021-06-10 19:00:26 +02:00
062919e08c deprecate cache backend (#5382) 2021-06-10 19:52:55 +03:00
654f5309b0 docs: drive: include requirement to publish app in step-by-step - fixes #5393 2021-06-10 17:00:52 +01:00
318fa4472b docs: fix incorrect syntax in config update example 2021-06-10 08:59:18 +02:00
5104e24153 docs: fix incorrect token type for yandex
https://forum.rclone.org/t/yandex-documentation/24445/2
2021-06-09 13:04:55 +02:00
9d87a5192d docs: fix code section formatting in filtering docs
Fixes #5387
2021-06-08 18:53:18 +02:00
29f967dba3 make commanddocs for v1.56 (#5383) 2021-06-08 18:57:04 +03:00
1f846c18d4 s3: Add SeaweedFS 2021-06-08 09:59:57 +01:00
41f561bf26 jottacloud: fix invalid url in output from link command
Fixes #5370
2021-05-31 10:40:21 +02:00
df60e6323c 🧹 GCS: Clean up time format constants 2021-05-28 14:44:50 +01:00
58006a925a 📑 GCS: Update docs on mtime
- Mention the new modification time behavior and the modify window issue.
- Unify markdown format.
- ref rclone/rclone#5331
2021-05-28 14:44:50 +01:00
ee2fac1855 🕰️ GCS: Compatible with gsutil's mtime metadata
- Write `goog-reserved-file-mtime` in addition to `mtime`.
- Fallback to `goog-reserved-file-mtime` if `mtime` doesn't exist.
- ref rclone/rclone#5331
2021-05-28 14:44:50 +01:00
2188fe38e5 docs: add caveat about DSCP on Windows 2021-05-28 13:43:38 +01:00
b5f8f0973b fshttp: implement graceful DSCP error handling 2021-05-28 13:43:38 +01:00
85b8ba9469 fshttp: rework address parsing for DSCP (fixes #5293) 2021-05-28 13:43:38 +01:00
Tom
04a1f673f0 serve sftp: add --stdio flag to serve via stdio - fixes #5311 2021-05-28 13:40:32 +01:00
0574ebf44a vfs: do not print notice about missing poll-interval support when set to 0
Fixes #5359
2021-05-28 13:09:15 +02:00
22e86ce335 vfs: fix that umask option cannot be set as environment variable (#5351)
Fixes #5350
2021-05-22 20:48:02 +02:00
c9fce20249 tardigrade: add warning about too many open files - Fixes #5310 2021-05-21 20:04:57 +01:00