17 Commits

Author	SHA1	Message	Date
Ivan Andreev	a2545066e2	drive: constrain list by filter #5023 ... Google Drive API allows for clauses like "modifiedTime > '2012-06-04T12:00:00'" in the query param, so the filter flags --max-age and --min-age can be applied directly at the directory listing phase rather than in a filter. This is extremely helpful when we want to do an incremental backup of a remote drive with many files but the number of recently changed file is small. Co-authored-by: fotile96 <fotile96@users.noreply.github.com>	2021-10-07 22:11:22 +03:00
Nick Craig-Wood	c22d04aa30	filter: deglobalise to put filter config into the context #4685	2020-11-27 17:28:42 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Ankur Gupta	75a6c49f87	Fix error counter - fixes #3650 ... For few commands, RClone counts a error multiple times. This was fixed by creating a new error type which keeps a flag to remember if the error has already been counted or not. The CountError function now wraps the original error eith the above new error type and returns it.	2019-11-18 14:13:02 +00:00
Nick Craig-Wood	af05e290cf	Fix --files-from without --no-traverse doing a recursive scan ... In a28239f005 we made --files-from obey --no-traverse. In the process this caused --files-from without --no-traverse to do a complete recursive scan unecessarily. This was only noticeable in users of fs/march, so sync/copy/move/etc not in ls/lsf/etc. This fix makes sure that we use conventional directory listings in fs/march unless `--files-from` and `--no-traverse` is set or `--fast-list` is active. Fixes #3619	2019-10-15 19:51:01 +01:00
Nick Craig-Wood	6c38bddf3e	walk: fix listing with filters listing whole remote ... Prior to this fix, a request such as rclone lsf -R --include "/dir/**" remote: Would use ListR which is very inefficient as it lists the whole remote for one directory. This changes it to use recursive walking if the filters imply any directory filtering. So `--include *.jpg` and `--exclude *.jpg` will still use ListR wheras `--include "/dir/**` will not.	2019-08-08 14:15:50 +01:00
Nick Craig-Wood	57d5de6fba	build: fix up package paths after repo move ... git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g' goimports -w `find . -name \*.go`	2019-07-28 18:47:38 +01:00
Nick Craig-Wood	bc70bff125	fs/dirtree: factor DirTree out of fs/walk and add tests	2019-07-02 15:26:55 +01:00
Aleksandar Jankovic	f78cd1e043	Add context propagation to rclone ... - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.	2019-06-19 11:59:46 +01:00
Nick Craig-Wood	a28239f005	filter: Make --files-from traverse as before unless --no-traverse is set ... In c5ac96e9e7 we made --files-from only read the objects specified and don't scan directories. This caused problems with Google drive (very very slow) and B2 (excessive API consumption) so it was decided to make the old behaviour (traversing the directories) the default with --files-from and use the existing --no-traverse flag (which has exactly the right semantics) to enable the new non scanning behaviour. See: https://forum.rclone.org/t/using-files-from-with-drive-hammers-the-api/8726 Fixes #3102 Fixes #3095	2019-04-12 17:16:49 +01:00
Nick Craig-Wood	efd720b533	walk: Implement walk.ListR which will use ListR if at all possible ... It otherwise has the nearly the same interface as walk.Walk which it will fall back to if it can't use ListR. Using walk.ListR will speed up file system operations by default and use much less memory and start immediately compared to if --fast-list had been supplied.	2019-03-16 17:41:12 +00:00
Nick Craig-Wood	bdc638530e	walk: make NewDirTree always use ListR #2946 ... This fixes vfs/refresh with recurse=true needing the --fast-list flag	2019-02-04 10:37:27 +00:00
Nick Craig-Wood	c5ac96e9e7	Make --files-from only read the objects specified and don't scan directories ... Before this change using --files-from would scan all the directories that the files could possibly be in causing rclone to do more work that was necessary. After this change, rclone constructs an in memory tree using the --fast-list mechanism but from all of the files in the --files-from list and without scanning any directories. Any objects that are not found in the --files-from list are ignored silently. This mechanism is used for sync/copy/move (march) and all of the listing commands ls/lsf/md5sum/etc (walk).	2018-10-20 18:13:31 +01:00
Nick Craig-Wood	512f4b4487	Update error checking on fmt.Fprint* after errcheck update ... Now we need to check or ignore errors on fmt.Fprint* explicitly - previously errcheck just ignored them for us.	2018-05-22 09:41:13 +01:00
Nick Craig-Wood	e0eb666dbf	fs/walk: fix new golint warning about unused variables in range	2018-03-02 17:01:58 +00:00
Nick Craig-Wood	11da2a6c9b	Break the fs package up into smaller parts. ... The purpose of this is to make it easier to maintain and eventually to allow the rclone backends to be re-used in other projects without having to use the rclone configuration system. The new code layout is documented in CONTRIBUTING.	2018-01-15 17:51:14 +00:00