Commit Graph

1132 Commits

Author SHA1 Message Date
Nick Craig-Wood
122a47fba6 accounting: Allow transfers to be canceled with context #3257
This makes all transfers cancelable even if the backend doesn't
support context as all transfers are done using the Accounting
framework.
2020-07-28 16:41:17 +01:00
Nick Craig-Wood
421585dd72 accounting: add context to Account and propagate changes #3257
This is preparation for getting the Accounting to check the context,
buf first we need to get it in place. Since this is one of those
changes that makes lots of noise, this is in a seperate commit.
2020-07-28 16:41:17 +01:00
Nick Craig-Wood
26001d520a fs: add --bwlimit-file flag to limit speeds of individual file transfers 2020-07-28 11:46:24 +01:00
Nick Craig-Wood
62f0bbb598 dedupe: Make it obey the --size-only flag for duplicate detection #4321 2020-07-28 11:40:37 +01:00
Nick Craig-Wood
8f42532b6d sync: add --track-renames-strategy leaf
See: https://forum.rclone.org/t/how-to-minimize-bandwith-w-r-t-renames-during-sync/16928/22
2020-07-28 11:34:27 +01:00
Chaitanya Bankanhal
8913679d88
accounting: Fix elapsed time not show actual time since beginning
This fixes the elapsed time display in the statistics output in the rc and in the log messages.
2020-07-26 11:59:50 +01:00
Nick Craig-Wood
aa93b39d9b build: fix tests on go1.15
go1.15 introduced a stricter policy for what you can convert with
`string()` and now `go vet` warns if you try to do `string(int)`.

See: https://github.com/golang/go/issues/32479
2020-07-25 18:51:28 +01:00
Nick Craig-Wood
744828a4de rc: allow JSON parameters to simplify command line usage
If the parameter being passed is an object then it can be passed as a
JSON string rather than using the `--json` flag which simplifies the
command line.

rclone rc operations/list fs=/tmp remote=test opt='{"showHash": true}'

Rather than

rclone rc operations/list --json '{"fs": "/tmp", "remote": "test", "opt": {"showHash": true}}'
2020-07-22 18:40:52 +01:00
Nick Craig-Wood
ff84351655 operations: factor Check and related functions into its own files 2020-07-21 22:08:13 +01:00
Nick Craig-Wood
8b6f2bbb4b check,cryptcheck: add reporting of filenames for same/missing/changed #3264
See: https://forum.rclone.org/t/rclone-check-v-doesnt-show-once-per-minute-update-counts/17402
2020-07-21 22:08:13 +01:00
Nick Craig-Wood
811b30d116 sync: fix deadlock with --track-renames-strategy modtime - fixes #4427
Before this change we could exit the popRenameMap function with the
lock held.

This fixes the problem by defer-ring the unlock.

See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992
2020-07-17 17:09:58 +01:00
Nick Craig-Wood
bcd362fcd5 accounting: fix documentation for speed/speedAvg
Fix the documentation for the very confusingly names speed and
speedAvg stats items.

See: https://github.com/rclone/rclone-webui-react/issues/99
2020-07-17 15:35:15 +01:00
Max Sum
bfa5715017 fs/accounting: sort transfers by start time 2020-07-07 15:42:06 +01:00
Max Sum
e2183ad661 fs/accounting: use transferMap instead of stringSet 2020-07-07 15:42:06 +01:00
Nick Craig-Wood
8d5bc7f28b fs/cache: fix moveto/copyto remote:file remote:file2
Before this change, if the cache was given a source `remote:file` it
stored `remote:` with the error `fs.ErrorIsFile` attached. This meant
that if it `remote:` was subsequently looked up it would return the
`fs.ErrorIsFile` error.

This broke `moveto remote:file remote:file2` as moveto would lookup
`remote:` from the second argument and erroneously get the
`fs.ErrorIsFile` error.

This likely broke other commands too.

This was broken in

4c9836035 fs/cache: Add Pin and Unpin and canonicalised lookup

Which was released in v1.52.0

The fix is to make a new cache entry for `remote:` with no error
attached in the case that the original call returned `fs.ErrorIsFile`.
2020-07-02 10:55:36 +01:00
Nick Craig-Wood
99c293a403 log: fix --use-json-log going to stderr not --log-file on Windows - fixes #4367 2020-07-01 20:47:37 +01:00
Nick Craig-Wood
05ddef117a accounting: add HasBuffer method to Account 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
ae8bbc63da cache: export Canonicalize method for external use 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
79f5d940cf fs: add Fingerprint to detect changes in an object 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
c820576329 fs: define SlowModTime and SlowHash features in the relevant backends 2020-06-30 12:01:36 +01:00
Chaitanya Bankanhal
a2dd23efd3 rc: Add tests for operations/uploadfile
rc: Go import file rc_test.go
2020-06-25 11:38:24 +01:00
Chaitanya Bankanhal
fa43d02874 rc: Add operations/uploadfile to upload a file through rc using encoding multipart/form-data 2020-06-25 11:38:24 +01:00
Chaitanya
d0de39ebcd rc: add NeedsRequest to call. 2020-06-25 11:38:24 +01:00
Nick Craig-Wood
3e82771413 Start v1.52.2-DEV development 2020-06-24 14:35:12 +01:00
Nick Craig-Wood
ce394426b0 check: fix successful retries with --download counting errors
See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017
2020-06-20 14:52:04 +01:00
Nick Craig-Wood
40611fc4fc check: retry downloads if they fail when using the --download flag
See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017/7
2020-06-18 16:16:19 +01:00
Nick Craig-Wood
7c4ba9fcb2 check: Fix misleading message which printed errors instead of differences
See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017/7
2020-06-18 16:16:19 +01:00
Nick Craig-Wood
3efdf5e095 operations: implement --refresh-times flag to set modtimes on hashless backends 2020-06-17 10:48:13 +01:00
Nick Craig-Wood
d174b97af7 errors: add WSAECONNREFUSED and more to the list of retriable Windows errors
This adds the missing WSAECONNREFUSED error to the list of errors we
can retry under Windows.

> Connection refused.  No connection could be made because the target
> computer actively refused it.

It also adds any relevant errors I could see in the error code list.

See: https://forum.rclone.org/t/failing-to-upload-large-file-to-b2/17085
2020-06-17 10:46:22 +01:00
jtagcat
d6a9017298
fs: fix formatting of errInvalidCharacters error message
errInvalidCharacters: 'and space .' -> 'and space.' Remove a space between the word, space, and period.

Co-authored-by: jtagcat <gitlab@c7.ee>
2020-06-16 15:08:09 +01:00
jtagcat
2d88d24881 config: reject remote names starting with a dash. (#4261) 2020-06-16 15:00:34 +01:00
Nick Craig-Wood
a3f6fe5287 dedupe: fix logging to be easier to understand #4321 2020-06-16 11:41:56 +01:00
Nick Craig-Wood
b58bb03e95 test: Don't run unreliable tests on CI #4171 2020-06-15 21:34:37 +01:00
Gary Kim
763944f673
rcd: fix incorrect prometheus metrics - fixes #4341
This was caused by using the stats group from the context passed in by the rcd
rather than the global stats group.

Signed-off-by: Gary Kim <gary@garykim.dev>
2020-06-14 10:09:24 +01:00
Nick Craig-Wood
fb06427c69 sync: fix --track-renames-strategy modtime
Before this change `--track-renames-strategy` was broken. The hashing
method it used could declare times that were very close together to be
different.

The time hash was discarded and instead we check the modification time
window on every hash match.

Provided that the user doesn't use `--track-renames-strategy` on a
huge number of identically sized files this will perform just fine.

See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992/5
2020-06-12 15:38:35 +01:00
Nick Craig-Wood
d55053098f check: make check do --checkers files concurrently - fixes #4318 2020-06-10 17:20:54 +01:00
Nick Craig-Wood
63cf0b1cdd check: make check command obey --dry-run/-i/--interactive - fixes #4325 2020-06-10 17:20:54 +01:00
Nick Craig-Wood
a1382a03aa Start v1.52.1-DEV development 2020-06-10 16:49:55 +01:00
Nick Craig-Wood
b23cf58a41 operations: Add skip all, do all, quit operations to --interactive - fixes #3886
This also adds SkipDestructive into all the remaing places --dry-run
was used and adds documentation.
2020-06-10 12:33:53 +01:00
fishbullet
ba5eb230fb operations: interactive mode -i/--interactive for destructive operations #3886 2020-06-10 12:33:53 +01:00
Nick Craig-Wood
b5c654a100 lib/structs: factor reflection based structure manipulation into a library 2020-06-10 12:28:48 +01:00
Nick Craig-Wood
3c14a893fb asyncreader: Make StopBuffer as well as Abandon and fix confusion in callers 2020-06-09 18:05:12 +01:00
Roman Kredentser
55ad1354b6 link: Add --expire and --unlink flags
This adds expire and unlink fields to the PublicLink interface.

This fixes up the affected backends and removes unlink parameters
where they are present.
2020-06-05 14:51:05 +01:00
Nick Craig-Wood
b62d08d136 config: set RCLONE_CONFIG_DIR for use in config files and subprocesses
See: https://forum.rclone.org/t/relative-path-in-rclone-config-service-account-json/16693
2020-06-03 17:39:08 +01:00
Nick Craig-Wood
1cceadaf7c Start v1.52.0-DEV development 2020-05-27 18:36:32 +01:00
Nick Craig-Wood
6882aeff97 Version v1.52.0 2020-05-27 17:31:10 +01:00
Martin Michlmayr
041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Nick Craig-Wood
518d39815c sync,copy,move: add --check-first to do all checking before starting transfers
See: https://forum.rclone.org/t/rclone-sync-doing-transfer-and-checking-in-paralel/16352/
2020-05-22 17:50:07 +01:00
Nick Craig-Wood
147f97d1f7 sync: allow --max-backlog to be -ve meaning as large as possible 2020-05-22 17:50:07 +01:00
Martin Michlmayr
4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00