226 Commits

Author	SHA1	Message	Date
Nick Craig-Wood	d9959b0271	drive: pass context on to drive SDK - this will help with cancellation	2021-03-16 16:17:22 +00:00
Nick Craig-Wood	f2c0f82fc6	backends: Add context checking to remaining backends #4504 ... This is a follow up to 4013bc4a4c which missed some backends. It adds a ctx parameter to shouldRetry and checks it.	2021-03-16 16:17:22 +00:00
Nick Craig-Wood	4fbb50422c	drive: don't stop server side copy if couldn't read description ... Before this change we would error out of a server side copy if we couldn't read the description for a google doc. This patch just logs an ERROR message and carries on. See: https://forum.rclone.org/t/rclone-google-drive-to-google-drive-migration-for-multiple-users/19024/3	2021-03-15 19:22:06 +00:00
Saksham Khanna	4d8ef7bca7	cmd/dedupe: make largest directory primary to minimize data moved (#3648) ... This change makes dedupe recursively count elements in same-named directories and make the largest one primary. This allows to minimize the amount of data moved (or at least the amount of API calls) when dedupe merges them. It also adds a new fs.Object interface `ParentIDer` with function `ParentID` and implements it for the drive and opendrive backends. This function returns parent directory ID for objects on filesystems that allow same-named dirs. We use it to correctly count sizes of same-named directories. Fixes #2568 Co-authored-by: Ivan Andreev <ivandeex@gmail.com>	2021-03-11 20:40:29 +03:00
Nick Craig-Wood	e32f08f37b	drive: refer to Shared Drives instead of Team Drives	2021-02-07 12:30:21 +00:00
Nick Craig-Wood	bcac8fdc83	Use http.NewRequestWithContext where possible after go1.13 minimum version	2021-02-03 17:41:27 +00:00
Nick Craig-Wood	f66928a846	drive: fix copyid command with a bare filename: can't use empty string as a path ... Before this change, running rclone backend copyid drive: ID file.txt Failed with the error command "copyid" failed: failed copying "ID" "file.txt": can't use empty string as a path This fixes the problem.	2021-02-01 15:17:25 +00:00
Pau Rodriguez-Estivill	f0bf9cfda1	drive: add xdg office icons to xdg desktop files	2021-01-28 17:12:48 +00:00
Nick Craig-Wood	0be69018b8	drive: log that emptying the trash can take some time - fixes #4915	2021-01-19 18:09:36 +00:00
Nick Craig-Wood	250f8d9371	drive: allow shortcut resolution and creation to be retried ... This was an oversight in the original code - these operations should always have been retriable.	2020-12-02 15:28:38 +00:00
Nick Craig-Wood	9d574c0d63	fshttp: read config from ctx not passed in ConfigInfo #4685	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
buengese	636fb5344a	drive: implement CleanUp workaround for team drives - fixes #2418	2020-11-13 03:30:28 +01:00
Nick Craig-Wood	f7efce594b	config: add context.Context #3257 #4685 ... This add config to the Config callback in the backends and the related config functions.	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	8b96933e58	fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Josh Soref	3e1cb8302a	docs: spelling: etc. ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	e4a87f772f	docs: spelling: e.g. ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	bbe7eb35f1	docs: spelling: server-side ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Anagh Kumar Baranwal	5b09599a23	drive: Added flag --drive-stop-on-download-limit to stop transfers when the download limit is exceeded ... Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>	2020-10-14 15:51:01 +01:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Nick Craig-Wood	e5190f14ce	drive: implement "rclone backend copyid" command for copying files by ID ... This allows files to be copied by ID from google drive. These can be copied to any rclone remote and if the remote is a google drive then server side copy will be attempted. Fixes #3625	2020-09-25 17:53:51 +01:00
Tim Gallant	c3884aafd9	drive: adds special oauth help test - fixes #4555	2020-09-07 12:48:46 +01:00
themylogin	57c10babfe	drive: Remove --drive-alternate-export in favor of exportLinks ... Google engineer confirms that the new official API should works properly: https://issuetracker.google.com/issues/36761333#comment8	2020-09-02 12:16:25 +01:00
Nick Craig-Wood	068cfdaa00	drive: fix "panic: send on closed channel" when recycling dir entries ... In this commit: cbf3d43561 drive: fix missing items when listing using --fast-list / ListR We introduced a bug where under specific circumstances it could cause a "panic: send on closed channel". This was caused by: - rclone engaging the workaround from the commit above - one of the listing routines returning an error - this caused the `in` channel to be closed to stop the readers - however the workaround was recycling stuff into the `in` channel at the time - hence the panic on closed channel This fix factors out the sending to the `in` channel into `sendJob` and calls this both from the master go routine and the list runners. `sendJob` detects the `in` channel being closed properly and also deals correctly with contention on the `in` channel. Fixes #4511	2020-08-31 11:41:15 +01:00
Jay McEntire	45afe97e8e	drive: Added --drive-starred-only to only show starred files - fixes #3928	2020-08-21 17:30:41 +01:00
Nick Craig-Wood	2bcc66c805	drive: fix duplication of Google docs on server side copy #4517 ... Before this change, rclone was looking for the file without the extension to see if it existed which meant that it never did. This change checks the destination file exists firsts, before removing the extension.	2020-08-20 20:19:33 +01:00
Nick Craig-Wood	b5ba077a2f	drive: work around drive bug which didn't set modtime of copied docs ... Google drive appears to no longer be copying the modification time of google docs. Setting the mod time immediately after the copy doesn't work either, so this patch copies the object, waits for 1 second and then sets the modtime. Fixes #4517	2020-08-20 20:19:33 +01:00
Tim Gallant	30eb094f28	oauthutil: adds SharedOptions for OAuth backends ... 1. adds SharedOptions data structure to oauthutil 2. adds config.ConfigToken option to oauthutil.SharedOptions 3. updates the backends that have oauth functionality Fixes #2849	2020-08-07 16:32:01 +01:00
Nick Craig-Wood	8e7eb37456	drive: implement backend command untrash ... rclone backend untrash drive:directory This was based on: https://gitlab.com/B4dM4n/drive-untrash See: https://forum.rclone.org/t/rclone-teamdrive-undelete/18278/3	2020-08-07 11:10:37 +01:00
Nick Craig-Wood	f50ab981f7	drive: stop using root_folder_id as a cache #4419 ... Previous to this change rclone cached the looked up root_folder_id in the root_folder_id config variable. This has caused a lot of confusion and a few attempts at workarounds and ultimately was a mistake. This reverts rclone attempting to cache anything in root_folder_id and returns that variable to be entirely user modified. It gives a little hint in the debug that rclone could be sped up slightly by setting it, but it is up to the user to think about whether that would be OK or not. Google drive root '': root_folder_id = "XXX" - save this in the config to speed up startup It does not change root_folder_id itself, leaving this to the user. See: https://forum.rclone.org/t/rclone-gdrive-no-longer-returning-anything/17215	2020-08-02 11:47:07 +01:00
Nick Craig-Wood	a2afa9aadd	fs: Add directory to optional Purge interface - fixes #1891 ... - add a directory to the optional Purge interface - fix up all the backends - add an additional integration test to test for the feature - use the new feature in operations.Purge Many of the backends had been prepared in advance for this so the change was trivial for them.	2020-07-31 17:43:17 +01:00
Nick Craig-Wood	0bab9903ee	drive: factor creation of the Fs so it can be re-used in team drive listing	2020-07-28 16:24:00 +01:00
Nick Craig-Wood	700deb0a81	drive: add rclone backend drives to list shared drives (teamdrives) ... See: https://forum.rclone.org/t/google-drive-remotes-team-drive-list-commend/17595	2020-07-28 16:24:00 +01:00
Nick Craig-Wood	101f82c6b3	drive: drop "Disabling ListR" messages down to debug ... This was causing unecessary anguish for users since these messages are harmless and really only interesting for debugging. See: https://forum.rclone.org/t/rclone-gdrive-error/18098	2020-07-25 16:50:55 +01:00
Nick Craig-Wood	e6fdc3a932	drive: make dangling shortcuts appear in listings ... Previous to this a dangling shortcut would error the directory listing. This patch makes dangling shortcuts appear as 0 sized objects in the directory listing so they can be deleted. These objects can't be read though.	2020-07-02 22:12:44 +01:00
Nick Craig-Wood	61e4b4db42	drive: Allow the use of --drive-impersonate with the root_folder_id "appDataFolder" ... In this commit 5c5ad6220 drive: fix --drive-impersonate with cached root_folder_id We disabled the use of root_folder_id with --drive-impersonate to fix a problem with a cached root_folder_id giving the wrong results. This, alas, broke one users setup with a root_folder_id of appDataFolder. Since this is identifiable and definitely couldn't have been cached, we can safely skip this check in this case. See: https://forum.rclone.org/t/rclone-gdrive-no-longer-returning-anything/17215/10	2020-06-25 21:43:11 +01:00
Nick Craig-Wood	2121c0fa23	dircache: factor DirMove code out of backends into dircache ... Before this change there was lots of duplicated code in all the dircache using backends to support DirMove. This change factors this code into the dircache library.	2020-06-25 09:41:36 +01:00
Nick Craig-Wood	a8652e2252	dircache: simplify interface, fix corner cases and apply to backends ... Dircache was changed to: - Remove special cases for the root directory - Remove Fatal errors - Call FindRoot on behalf of the user wherever possible - Bring up to modern Go standards Backends were changed to: - Remove calls to FindRoot - Change calls to FindRootAndPath to FindPath - Don't make special cases for the root This fixes several corner cases, for example removing a non existent directory if FindRoot hasn't been called.	2020-06-25 09:41:36 +01:00
Nick Craig-Wood	81151523af	drive: fix shortcut tests	2020-06-24 15:52:02 +01:00
Nick Craig-Wood	5c5ad62208	drive: fix --drive-impersonate with cached root_folder_id ... Before this fix rclone v1.51 and 1.52 would incorrectly use the cached root_folder_id when the --drive-impersonate flag was in use. This meant that rclone could be looking up the wrong directory ID with unpredictable results - usually all files apparently being missing. This fix makes rclone look up the root_folder_id always when using --drive-impersonate. It does this by clearing the root_folder_id and making a NOTICE message that it is ignoring the cached value. It also stops rclone caching the root_folder_id when using --drive-impersonate. See: https://forum.rclone.org/t/rclone-gdrive-no-longer-returning-anything/17215	2020-06-20 15:01:37 +01:00
Nick Craig-Wood	5e6f4ab281	drive: fix creating a directory inside a shortcut ... See: https://forum.rclone.org/t/cant-create-new-directory-on-google-drive-remote/17208	2020-06-17 11:32:28 +01:00
Nick Craig-Wood	848c5b78e1	drive: fix not being able to delete a directory with a trashed shortcut ... When we resolve the shortcut we now propagate the trashed status of the shortcut into the resolved item which fixes the issue.	2020-06-12 15:10:35 +01:00
Roman Kredentser	55ad1354b6	link: Add --expire and --unlink flags ... This adds expire and unlink fields to the PublicLink interface. This fixes up the affected backends and removes unlink parameters where they are present.	2020-06-05 14:51:05 +01:00
Nick Craig-Wood	973e3d6a7b	backends: make sure backends expand ~ and environment vars in file names they use ... See: https://forum.rclone.org/t/relative-path-in-rclone-config-service-account-json/16693	2020-06-03 17:39:08 +01:00
Nick Craig-Wood	cbf3d43561	drive: fix missing items when listing using --fast-list / ListR ... This is caused by a bug in Google drive where, in some circumstances querying for "(A in parents) or (B in parents)" returns nothing whereas querying for "A in parents" and "B in parents" separately works fine. This has been reported here: https://issuetracker.google.com/issues/149522397 This workaround detects this condition by seeing if a listing for more than one directory at once returns nothing. If it does then it retries each one individually. This can potentially have a false positive if the user has multiple empty directories which are queried at once. The consequence of this will be that ListR is disabled for a while until the directories are found to be actually empty in which case it will be re-enabled. Fixes #3114 and Fixes #4289	2020-05-31 11:44:15 +01:00
Nick Craig-Wood	49ba4eeb86	oauthutil: tidy interface to Config to add Options struct ... The interface was getting so that a new function was needed for every Config variant. Adding an Options struct fixes this.	2020-05-26 11:27:01 +01:00
Martin Michlmayr	041b201abd	doc: fix typos throughout docs and code	2020-05-25 11:23:58 +01:00
Martin Michlmayr	4aee962233	doc: fix typos throughout docs and code	2020-05-20 15:54:51 +01:00
Nick Craig-Wood	cf5d0f5c1f	Revert "drive: server side copy docs use default description if empty" ... This reverts commit 9e4b68a364. This does not work as intended - it only changes docs files and to make it change drive files would take an extra roundtrip. I think the sematics of server side copy are now correct - additional features should be added with a new flag. See #4230	2020-05-19 16:48:02 +01:00