1197 Commits

Author	SHA1	Message	Date
Maciej Zimnoch	70e8b11805	accounting: fix data race in Transferred() ... startedTransfers is accessed by multiple threads, and it wasn't protected by the mutex call in Transferred() func. Fixes #4799	2020-11-26 22:54:38 +00:00
Nick Craig-Wood	9d574c0d63	fshttp: read config from ctx not passed in ConfigInfo #4685	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Ankur Gupta	33ea55efed	fs: parseduration: fixed tests to use UTC time	2020-11-23 17:11:56 +00:00
buengese	2812816142	hash: add MultiHasher.Sum() to retrieve a single specific hash	2020-11-13 02:31:59 +01:00
Nick Craig-Wood	f7efce594b	config: add context.Context #3257 #4685 ... This add config to the Config callback in the backends and the related config functions.	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	1fb6ad700f	accounting: add context.Context #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	e3fe31f7cb	fs: add context.Context to fs.GetModifyWindow #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	8b96933e58	fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d69b96a94c	test: Add context to mockfs.NewFs #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Anagh Kumar Baranwal	30c8b1b84f	fs: Fix nil pointer on copy & move operations directly to remote ... Fix the copy and move operations that broke in 127f0fc when copying directly to a remote without a specific destination. Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>	2020-11-09 12:36:10 +00:00
Nick Craig-Wood	af55a74bd2	stats: add counter for deleted directories - fixes #4676	2020-11-03 11:47:00 +00:00
Nick Craig-Wood	d00c126cef	operations: fix --cutof-mode hard not cutting off immediately ... This failure was noted on the integration tests server. The fix was to be more careful about which error message was emitted with which --cutoff-mode	2020-11-02 17:13:19 +00:00
Adam Plánský	127f0fc64c	operations: move and copy log name of the destination object in verbose ... If the object is moved or copied rclone in verbose mode prints name of the destination object into the info log.	2020-10-30 15:31:54 +00:00
Josh Soref	3e1cb8302a	docs: spelling: etc. ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	e4a87f772f	docs: spelling: e.g. ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	bbe7eb35f1	docs: spelling: server-side ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Nick Craig-Wood	4e2b5389d7	check: make the error count match up in the log message ... Before this change we counted the final summary error as an error, producing confusing log messages like: Failed to check with 54 errors: last error was: 53 differences found This change marks the summary error as already being counted, so the error message becomes: Failed to check with 53 errors: last error was: 53 differences found This change also returns a listing failure in preference to a summary error. See: https://forum.rclone.org/t/slow-checksum-validation/19763/22	2020-10-15 12:57:48 +01:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Nick Craig-Wood	51a230d7fd	fs: Implement UnWrapObjectInfo for getting original object out of src objects	2020-10-14 15:20:06 +01:00
Nick Craig-Wood	72083c65ad	cmd: make backend env vars show in help as the defaults for backend flags ... Before this change RCLONE_DRIVE_CHUNK_SIZE=111M rclone help flags | grep drive-chunk-size Would show the default value, not the setting of RCLONE_DRIVE_CHUNK_SIZE as the non backend flags do. This change makes it work as expected by setting the default of the option to the environment variable. Fixes #4659	2020-10-13 15:43:58 +01:00
LaSombra	aac2406e19	cmd: add --progress-terminal-title to print ETA to terminal title ... Adds a flag, --progress-terminal-title, that when used with --progress, will print the string `ETA: %s` to the terminal title. This also adds WriteTerminalTitle to lib/terminal	2020-10-06 16:34:26 +01:00
Hekmon	c58023a9ba	enhance systemd integration ... * log level identification * manual activation with flag * automatic systemd launch detection	2020-10-06 16:21:30 +01:00
Nick Craig-Wood	20b00db390	operations: fix spurious "--checksum is in use but the source and destination have no hashes in common" ... Before this change rclone would emit the message --checksum is in use but the source and destination have no hashes in common; falling back to --size-only When the source or destination hash was missing as well as when the source and destination had no hashes in common. This first case is very confusing for users when the source and destination do have a hash in common. This change fixes that and makes sure the error message is not emitted on missing hashes even when there is a hash in common. See: https://forum.rclone.org/t/source-and-destination-have-no-hashes-in-common-for-unencrypted-drive-to-local-sync/19531	2020-10-05 16:07:05 +01:00
Nick Craig-Wood	db4bbf9521	operations: fix use of --suffix without --backup-dir ... As part of the original work adding this feature it was overlooked that this didn't actually work for full rclone copy/sync. This commit fixes the problem and adds a test to make sure it stays working. See: https://forum.rclone.org/t/suffix-not-working-on-folder-upload-via-ssh-sftp/19526	2020-10-04 16:40:20 +01:00
Nick Craig-Wood	2b7994e739	operations: add logs when need to upload files to set mod times #1505	2020-09-29 17:04:29 +01:00
Nick Craig-Wood	433b73a5a8	accounting: stabilize display order of transfers on Windows ... Before this change we sorted transfers in the stats list solely on time started. However if --check-first was in use then lots of transfers could be started in the same millisecond. Because Windows time resolution is only 1mS this caused the entries to sort equal and bounce around in the list. This change fixes the sort so that if the time is equal it uses the name which should stabilize the order. Fixes #4599	2020-09-24 19:10:37 +01:00
Nick Craig-Wood	5c49096e11	acounting: fix incorrect speed and transferTime in core/stats ... Before this change the code which summed up the existing transfers over all the stats groups forgot to add the old transfer time and old transfers in. This meant that the speed and elapsedTime got increasingly inaccurate over time due to the transfers being culled from the list but their time not being accounted for. This change adds the old transfers into the sum which fixes the problem. This was only a problem over the rc. Fixes #4569	2020-09-15 12:01:18 +01:00
Nick Craig-Wood	75de30cfa8	sync: fix --cutoff-mode soft & cautious so it doesn't end the transfer early ... Before ths fix --cutoff-mode soft and cautious would emit a Fatal error which stopped the sync immediately. This fix introduces a new error which is checked in the sync error processing which stops the sync gracefully. Fixes #4576	2020-09-09 12:53:21 +01:00
Nick Craig-Wood	05bea46c3e	accounting: remove new line from end of --stats-one-line display	2020-09-08 16:10:52 +01:00
Chaitanya Bankanhal	c8a719ae0d	webui: Prompt user for updating webui if an update is available	2020-09-07 16:45:00 +01:00
Nick Craig-Wood	344bce7e2a	docs: fix formatting of rc docs page ... See: https://forum.rclone.org/t/rclone-1-53-release/18880/24	2020-09-03 11:53:24 +01:00
Nick Craig-Wood	41ecb586c4	Start v1.54.0-DEV development	2020-09-02 17:52:58 +01:00
Nick Craig-Wood	70c8566cb8	fs: Pin created backends until parents are finalized ... This attempts to solve the backend lifecycle problem by - Pinning backends mentioned on the command line into the cache indefinitely - Unpinning backends when the containing structure (VFS, wrapping backend) is destroyed See: https://forum.rclone.org/t/rclone-rc-backend-command-not-working-as-expected/18834	2020-09-01 18:21:03 +01:00
Nick Craig-Wood	3affc2e066	fspath: make JoinRootPath convert backslashes to slashes on Windows ... The function is used for contructing remotes which may have backslashes in on Windows.	2020-09-01 18:21:03 +01:00
Nick Craig-Wood	c31defbbd3	fs: add debug to show when a backend is being created ... See: https://forum.rclone.org/t/rclone-rc-backend-command-not-working-as-expected/18834/	2020-08-31 14:51:06 +01:00
Nick Craig-Wood	75d54d720c	version: replace internal code with github.com/coreos/go-semver ... We were already importing go-semver so it makes sense to remove the duplicated semver parsing code and just use go-semver	2020-08-31 13:55:04 +01:00
Nick Craig-Wood	cc0421cb9e	rc/webgui: skip AddPlugin and RemovePlugin tests if download fails	2020-08-31 13:45:06 +01:00
Nick Craig-Wood	9c01ac9894	rc/webgui: improve error handling on web fetches	2020-08-31 13:45:06 +01:00
Chaitanya Bankanhal	20300d1f61	plugins: Change failing plugin test to new repo rclone/rclone-test-plugin	2020-08-31 13:45:06 +01:00
Chaitanya Bankanhal	6231beefc5	webui: Fix broken webui because of plugins redirection	2020-08-31 13:45:06 +01:00
Aaron Gokaslan	bf685f600e	webgui: fixes previously unhandled error in JSON marshall in fs/rc/webgui/plugins.go:writeToFile	2020-08-30 17:15:03 +01:00
Leo Luan	c665201b85	vfs: support synchronous cache space recovery upon ENOSPC ... This patch provides the support of synchronous cache space recovery to allow read threads to recover from ENOSPC errors when cache space can be recovered from cache items that are not in use or safe to be reset/emptied . The patch complements the existing cache cleaning process in two ways. Firstly, the existing cache cleaning process is time-driven that runs periodically. The cache space can run out while the cache cleaner thread is still waiting for its next scheduled run. The io threads encountering ENOSPC return an internal error to the applications in this case even when cache space can be recovered to avoid this error. This patch addresses this problem by having the read threads kick the cache cleaner thread in this condition to recover cache space preventing unnecessary ENOSPC errors from being seen by the applications. Secondly, this patch enhances the cache cleaner to support cache item reset. Currently the cache purge process removes cache items that are not in use. This may not be sufficient when the total size of the working set exceeds the cache directory's capacity. Like in the current code, this patch starts the purge process by removing cache files that are not in use. Cache items whose access times are older than vfs-cache-max-age are removed first. After that, other not-in-use items are removed in LRU order until vfs-cache-max-size is reached. If the vfs-cache-max-size (the quota) is still not reached at this time, this patch adds a cache reset step to reset/empty cache files that are still in use but not dirtied. This enables application processes to continue without seeing an error even when the working set depletes the cache space as long as there is not a large write working set hoarding the entire cache space. By design this patch does not add ENOSPC error recovery for write IOs. Rclone does not empty a write cache item until the file data is written back to the backend upon close. Allowing more cache space to be consumed by dirty cache items when the cache space is already running low would increase the risk of exhausting the cache space in a way that the vfs mount becomes unreadable.	2020-08-25 21:12:06 +01:00
Chaitanya Bankanhal	d6996e3347	plugins: Add url query params to regex for referrer path	2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal	dffcc99373	plugins: Create availablePlugins config file if it does not exist.	2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal	09b79679cd	plugins: restructure and add tests for pluginsctl/* calls	2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal	cf68e61f40	Add redirection for plugin urls	2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal	22674d1146	plugins: Add reverse proxy pluginsHandler for serving plugins	2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal	f9ee0dc3f2	plugins: allow installation and use of plugins and test plugins with rclone-webui	2020-08-24 10:56:04 +01:00