24 Commits

Author	SHA1	Message	Date
Kyle Reynolds	7835991147	fs: add more detailed logging for file includes/excludes ... This makes a DEBUG log to show why files were included or excluded. Fixes #7463	2024-01-22 16:46:26 +00:00
Nick Craig-Wood	bd787e8f45	filter: Fix incorrect filtering with UseFilter context flag and wrapping backends ... In this commit 8d1fff9a82 local: obey file filters in listing to fix errors on excluded files We started using filters in the local backend so the user could short circuit troublesome files/directories at a low level. However this caused a number of integration tests to fail. This turned out to be in backends wrapping the local backend. For example the combine backend test failed because it changes the paths passed to the local backend so they no longer match the paths in the current filter. To fix this, a new feature flag `FilterAware` was added and the UseFilter context flag is only passed to backends which support it. As the wrapping backends don't support the flag, this fixes the problems in the integration tests. In future the wrapping backends could modify the active filters to match the path modifications and then they could set the FilterAware flag. See #6376	2022-09-05 16:19:50 +01:00
Nick Craig-Wood	6fd9e3d717	build: reformat comments to pass go1.19 vet ... See: https://go.dev/doc/go1.19#go-doc	2022-08-05 16:35:41 +01:00
albertony	f4f0e444bf	filter: allow multiple --exclude-if-present flags - fixes #6219	2022-06-08 17:11:52 +02:00
Nick Craig-Wood	da404dc0f2	sync,copy: Fix --fast-list --create-empty-src-dirs and --exclude ... Before this change, if --fast-list was in use while doing a sync or copy with --create-empty-src-dirs and --exclude excluded all the files from the directory (but not the directory), then the directory would not be created. This is also visible with `rclone tree` which uses the same tree building approach as `rclone sync --fast-list` where the directories would go missing from the tree view. This was caused by not adding the parents of excluded files to the directory tree. See: https://forum.rclone.org/t/create-empty-src-dirs-issue-with-b2/30856	2022-05-23 10:10:55 +01:00
Nick Craig-Wood	e43b5ce5e5	Remove github.com/pkg/errors and replace with std library version ... This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.	2021-11-07 11:53:30 +00:00
Atílio Antônio	c08d48a50d	docs: improve grammar and fix typos (#5361) ... This alters some comments in source files, but is interested mainly in documentation files and help messages.	2021-11-04 12:50:43 +01:00
Ivan Andreev	a2545066e2	drive: constrain list by filter #5023 ... Google Drive API allows for clauses like "modifiedTime > '2012-06-04T12:00:00'" in the query param, so the filter flags --max-age and --min-age can be applied directly at the directory listing phase rather than in a filter. This is extremely helpful when we want to do an incremental backup of a remote drive with many files but the number of recently changed file is small. Co-authored-by: fotile96 <fotile96@users.noreply.github.com>	2021-10-07 22:11:22 +03:00
Nick Craig-Wood	c22d04aa30	filter: deglobalise to put filter config into the context #4685	2020-11-27 17:28:42 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Ankur Gupta	75a6c49f87	Fix error counter - fixes #3650 ... For few commands, RClone counts a error multiple times. This was fixed by creating a new error type which keeps a flag to remember if the error has already been counted or not. The CountError function now wraps the original error eith the above new error type and returns it.	2019-11-18 14:13:02 +00:00
Nick Craig-Wood	af05e290cf	Fix --files-from without --no-traverse doing a recursive scan ... In a28239f005 we made --files-from obey --no-traverse. In the process this caused --files-from without --no-traverse to do a complete recursive scan unecessarily. This was only noticeable in users of fs/march, so sync/copy/move/etc not in ls/lsf/etc. This fix makes sure that we use conventional directory listings in fs/march unless `--files-from` and `--no-traverse` is set or `--fast-list` is active. Fixes #3619	2019-10-15 19:51:01 +01:00
Nick Craig-Wood	6c38bddf3e	walk: fix listing with filters listing whole remote ... Prior to this fix, a request such as rclone lsf -R --include "/dir/**" remote: Would use ListR which is very inefficient as it lists the whole remote for one directory. This changes it to use recursive walking if the filters imply any directory filtering. So `--include *.jpg` and `--exclude *.jpg` will still use ListR wheras `--include "/dir/**` will not.	2019-08-08 14:15:50 +01:00
Nick Craig-Wood	57d5de6fba	build: fix up package paths after repo move ... git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g' goimports -w `find . -name \*.go`	2019-07-28 18:47:38 +01:00
Nick Craig-Wood	bc70bff125	fs/dirtree: factor DirTree out of fs/walk and add tests	2019-07-02 15:26:55 +01:00
Aleksandar Jankovic	f78cd1e043	Add context propagation to rclone ... - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.	2019-06-19 11:59:46 +01:00
Nick Craig-Wood	a28239f005	filter: Make --files-from traverse as before unless --no-traverse is set ... In c5ac96e9e7 we made --files-from only read the objects specified and don't scan directories. This caused problems with Google drive (very very slow) and B2 (excessive API consumption) so it was decided to make the old behaviour (traversing the directories) the default with --files-from and use the existing --no-traverse flag (which has exactly the right semantics) to enable the new non scanning behaviour. See: https://forum.rclone.org/t/using-files-from-with-drive-hammers-the-api/8726 Fixes #3102 Fixes #3095	2019-04-12 17:16:49 +01:00
Nick Craig-Wood	efd720b533	walk: Implement walk.ListR which will use ListR if at all possible ... It otherwise has the nearly the same interface as walk.Walk which it will fall back to if it can't use ListR. Using walk.ListR will speed up file system operations by default and use much less memory and start immediately compared to if --fast-list had been supplied.	2019-03-16 17:41:12 +00:00
Nick Craig-Wood	bdc638530e	walk: make NewDirTree always use ListR #2946 ... This fixes vfs/refresh with recurse=true needing the --fast-list flag	2019-02-04 10:37:27 +00:00
Nick Craig-Wood	c5ac96e9e7	Make --files-from only read the objects specified and don't scan directories ... Before this change using --files-from would scan all the directories that the files could possibly be in causing rclone to do more work that was necessary. After this change, rclone constructs an in memory tree using the --fast-list mechanism but from all of the files in the --files-from list and without scanning any directories. Any objects that are not found in the --files-from list are ignored silently. This mechanism is used for sync/copy/move (march) and all of the listing commands ls/lsf/md5sum/etc (walk).	2018-10-20 18:13:31 +01:00
Nick Craig-Wood	512f4b4487	Update error checking on fmt.Fprint* after errcheck update ... Now we need to check or ignore errors on fmt.Fprint* explicitly - previously errcheck just ignored them for us.	2018-05-22 09:41:13 +01:00
Nick Craig-Wood	e0eb666dbf	fs/walk: fix new golint warning about unused variables in range	2018-03-02 17:01:58 +00:00
Nick Craig-Wood	11da2a6c9b	Break the fs package up into smaller parts. ... The purpose of this is to make it easier to maintain and eventually to allow the rclone backends to be re-used in other projects without having to use the rclone configuration system. The new code layout is documented in CONTRIBUTING.	2018-01-15 17:51:14 +00:00