190 Commits

Author	SHA1	Message	Date
lostheli	c8cfa43ccc	Add a download flag to hashsum and related commands to force rclone to download and hash files locally ... This commit modifies the operations.hashSum function by adding an alternate code path. This code path is triggered by passing downloadFlag = True. When activated, rclone will download files from the remote and hash them locally. downloadFlag = False preserves the existing behavior of using the remote to retrieve the hash. This commit modifies HashLister to support the new hashSum method as well as consolidating the roles of HashLister, HashListerBase64, Md5sum, and Sha1sum. The printing of hashes from the function defined in HashLister has been revised to work with --progress. There are light changes to operations.syncFprintf and cmd.startProgress for this. The unit test operations_test.TestHashSums is modified to support this change and test the download functionality. The command functions hashsum, md5sum, sha1sum, and dbhashsum are modified to support this change. A download flag has been added and an output-file flag has been added. The output-file flag writes hashes to a file instead of stdout to avoid the need to redirect stdout.	2020-12-27 15:40:44 +00:00
Nick Craig-Wood	86014cebd7	dedupe: add --dedupe-mode list to just list dupes, changing nothing	2020-12-02 16:52:12 +00:00
Nick Craig-Wood	507f861c67	dedupe: add --by-hash to dedupe on hash not file name - fixes #1674	2020-12-02 16:52:12 +00:00
Nick Craig-Wood	c22d04aa30	filter: deglobalise to put filter config into the context #4685	2020-11-27 17:28:42 +00:00
Nick Craig-Wood	9d574c0d63	fshttp: read config from ctx not passed in ConfigInfo #4685	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	1fb6ad700f	accounting: add context.Context #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	e3fe31f7cb	fs: add context.Context to fs.GetModifyWindow #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d69b96a94c	test: Add context to mockfs.NewFs #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Anagh Kumar Baranwal	30c8b1b84f	fs: Fix nil pointer on copy & move operations directly to remote ... Fix the copy and move operations that broke in 127f0fc when copying directly to a remote without a specific destination. Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>	2020-11-09 12:36:10 +00:00
Nick Craig-Wood	af55a74bd2	stats: add counter for deleted directories - fixes #4676	2020-11-03 11:47:00 +00:00
Nick Craig-Wood	d00c126cef	operations: fix --cutof-mode hard not cutting off immediately ... This failure was noted on the integration tests server. The fix was to be more careful about which error message was emitted with which --cutoff-mode	2020-11-02 17:13:19 +00:00
Adam Plánský	127f0fc64c	operations: move and copy log name of the destination object in verbose ... If the object is moved or copied rclone in verbose mode prints name of the destination object into the info log.	2020-10-30 15:31:54 +00:00
Josh Soref	e4a87f772f	docs: spelling: e.g. ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	bbe7eb35f1	docs: spelling: server-side ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Nick Craig-Wood	4e2b5389d7	check: make the error count match up in the log message ... Before this change we counted the final summary error as an error, producing confusing log messages like: Failed to check with 54 errors: last error was: 53 differences found This change marks the summary error as already being counted, so the error message becomes: Failed to check with 53 errors: last error was: 53 differences found This change also returns a listing failure in preference to a summary error. See: https://forum.rclone.org/t/slow-checksum-validation/19763/22	2020-10-15 12:57:48 +01:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Nick Craig-Wood	20b00db390	operations: fix spurious "--checksum is in use but the source and destination have no hashes in common" ... Before this change rclone would emit the message --checksum is in use but the source and destination have no hashes in common; falling back to --size-only When the source or destination hash was missing as well as when the source and destination had no hashes in common. This first case is very confusing for users when the source and destination do have a hash in common. This change fixes that and makes sure the error message is not emitted on missing hashes even when there is a hash in common. See: https://forum.rclone.org/t/source-and-destination-have-no-hashes-in-common-for-unencrypted-drive-to-local-sync/19531	2020-10-05 16:07:05 +01:00
Nick Craig-Wood	db4bbf9521	operations: fix use of --suffix without --backup-dir ... As part of the original work adding this feature it was overlooked that this didn't actually work for full rclone copy/sync. This commit fixes the problem and adds a test to make sure it stays working. See: https://forum.rclone.org/t/suffix-not-working-on-folder-upload-via-ssh-sftp/19526	2020-10-04 16:40:20 +01:00
Nick Craig-Wood	2b7994e739	operations: add logs when need to upload files to set mod times #1505	2020-09-29 17:04:29 +01:00
Nick Craig-Wood	75de30cfa8	sync: fix --cutoff-mode soft & cautious so it doesn't end the transfer early ... Before ths fix --cutoff-mode soft and cautious would emit a Fatal error which stopped the sync immediately. This fix introduces a new error which is checked in the sync error processing which stops the sync gracefully. Fixes #4576	2020-09-09 12:53:21 +01:00
Chaitanya Bankanhal	61c7ea4085	rc: fix rc/uploadfile only working for root of the fs	2020-08-10 17:09:46 +01:00
Nick Craig-Wood	a2afa9aadd	fs: Add directory to optional Purge interface - fixes #1891 ... - add a directory to the optional Purge interface - fix up all the backends - add an additional integration test to test for the feature - use the new feature in operations.Purge Many of the backends had been prepared in advance for this so the change was trivial for them.	2020-07-31 17:43:17 +01:00
Nick Craig-Wood	421585dd72	accounting: add context to Account and propagate changes #3257 ... This is preparation for getting the Accounting to check the context, buf first we need to get it in place. Since this is one of those changes that makes lots of noise, this is in a seperate commit.	2020-07-28 16:41:17 +01:00
Nick Craig-Wood	62f0bbb598	dedupe: Make it obey the --size-only flag for duplicate detection #4321	2020-07-28 11:40:37 +01:00
Nick Craig-Wood	ff84351655	operations: factor Check and related functions into its own files	2020-07-21 22:08:13 +01:00
Nick Craig-Wood	8b6f2bbb4b	check,cryptcheck: add reporting of filenames for same/missing/changed #3264 ... See: https://forum.rclone.org/t/rclone-check-v-doesnt-show-once-per-minute-update-counts/17402	2020-07-21 22:08:13 +01:00
Chaitanya Bankanhal	a2dd23efd3	rc: Add tests for operations/uploadfile ... rc: Go import file rc_test.go	2020-06-25 11:38:24 +01:00
Chaitanya Bankanhal	fa43d02874	rc: Add operations/uploadfile to upload a file through rc using encoding multipart/form-data	2020-06-25 11:38:24 +01:00
Chaitanya	d0de39ebcd	rc: add NeedsRequest to call.	2020-06-25 11:38:24 +01:00
Nick Craig-Wood	ce394426b0	check: fix successful retries with --download counting errors ... See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017	2020-06-20 14:52:04 +01:00
Nick Craig-Wood	40611fc4fc	check: retry downloads if they fail when using the --download flag ... See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017/7	2020-06-18 16:16:19 +01:00
Nick Craig-Wood	7c4ba9fcb2	check: Fix misleading message which printed errors instead of differences ... See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017/7	2020-06-18 16:16:19 +01:00
Nick Craig-Wood	3efdf5e095	operations: implement --refresh-times flag to set modtimes on hashless backends	2020-06-17 10:48:13 +01:00
Nick Craig-Wood	a3f6fe5287	dedupe: fix logging to be easier to understand #4321	2020-06-16 11:41:56 +01:00
Nick Craig-Wood	d55053098f	check: make check do --checkers files concurrently - fixes #4318	2020-06-10 17:20:54 +01:00
Nick Craig-Wood	63cf0b1cdd	check: make check command obey --dry-run/-i/--interactive - fixes #4325	2020-06-10 17:20:54 +01:00
Nick Craig-Wood	b23cf58a41	operations: Add skip all, do all, quit operations to --interactive - fixes #3886 ... This also adds SkipDestructive into all the remaing places --dry-run was used and adds documentation.	2020-06-10 12:33:53 +01:00
fishbullet	ba5eb230fb	operations: interactive mode -i/--interactive for destructive operations #3886	2020-06-10 12:33:53 +01:00
Roman Kredentser	55ad1354b6	link: Add --expire and --unlink flags ... This adds expire and unlink fields to the PublicLink interface. This fixes up the affected backends and removes unlink parameters where they are present.	2020-06-05 14:51:05 +01:00
Martin Michlmayr	041b201abd	doc: fix typos throughout docs and code	2020-05-25 11:23:58 +01:00
Martin Michlmayr	4aee962233	doc: fix typos throughout docs and code	2020-05-20 15:54:51 +01:00
Nick Craig-Wood	f37af9afec	lsjson: Add --hash-type parameter and use it in lsf to speed up hashing ... Before this change if you specified --hash MD5 in rclone lsf it would calculate all the hashes and just return the MD5 hash which was very slow on the local backend. Likewise specifying --hash on rclone lsjson was equally slow. This change introduces the --hash-type flag (and corresponding internal parameter) so that the hashes required can be selected in lsjson. This is used internally in lsf when the --hash parameter is selected to speed up the hashing by only hashing with the one hash specified. Fixes #4181	2020-04-29 16:09:45 +01:00
Nick Craig-Wood	d80fdad6da	rc: implement backend/command for running backend commands remotely	2020-04-29 10:10:57 +01:00
Nick Craig-Wood	1dced3b3c4	rcat: add support for --header-upload #59	2020-04-23 11:34:31 +01:00
Nick Craig-Wood	087bf1d584	cat: add support for --header-download #59	2020-04-23 11:34:24 +01:00
Nick Craig-Wood	f5455d865b	accounting: check for max transfer in WriteTo ... Before this change the max transfer tests were failing for remotes which were using WriterTo.	2020-04-23 11:13:13 +01:00
Tim Gallant	9bf3d3da4c	fs: add UploadHeaders, DownloadHeaders to Update/Put/Open options	2020-04-23 11:07:21 +01:00
Nick Craig-Wood	44b1a591a8	crypt: get rid of the unused Cipher interface as it obfuscated the code	2020-04-22 11:33:48 +01:00