extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-11-25 18:04:55 +01:00
Code Issues Packages Projects Releases Wiki Activity
5,692 Commits 177 Branches 114 Tags 317 MiB
a2545066e2
Commit Graph

20 Commits

Author SHA1 Message Date
Ivan Andreev
a2545066e2 drive: constrain list by filter #5023 
Google Drive API allows for clauses like "modifiedTime > '2012-06-04T12:00:00'"
in the query param, so the filter flags --max-age and --min-age can be applied
directly at the directory listing phase rather than in a filter.
This is extremely helpful when we want to do an incremental backup of a remote
drive with many files but the number of recently changed file is small.

Co-authored-by: fotile96 <fotile96@users.noreply.github.com>
 2021-10-07 22:11:22 +03:00
Nick Craig-Wood
c22d04aa30 filter: deglobalise to put filter config into the context #4685 2020-11-27 17:28:42 +00:00
Nick Craig-Wood
2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Nick Craig-Wood
d69b96a94c test: Add context to mockfs.NewFs #3257 #4685 2020-11-09 18:05:54 +00:00
Josh Soref
d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Nick Craig-Wood
aa93b39d9b build: fix tests on go1.15 
go1.15 introduced a stricter policy for what you can convert with
`string()` and now `go vet` warns if you try to do `string(int)`.

See: https://github.com/golang/go/issues/32479
 2020-07-25 18:51:28 +01:00
Ankur Gupta
75a6c49f87 Fix error counter - fixes #3650 
For few commands, RClone counts a error multiple times. This was fixed by
creating a new error type which keeps a flag to remember if the error has
already been counted or not. The CountError function now wraps the original
error eith the above new error type and returns it.
 2019-11-18 14:13:02 +00:00
Nick Craig-Wood
af05e290cf Fix --files-from without --no-traverse doing a recursive scan 
In a28239f005 we made --files-from obey --no-traverse.  In the
process this caused --files-from without --no-traverse to do a
complete recursive scan unecessarily.

This was only noticeable in users of fs/march, so sync/copy/move/etc
not in ls/lsf/etc.

This fix makes sure that we use conventional directory listings in
fs/march unless `--files-from` and `--no-traverse` is set or
`--fast-list` is active.

Fixes #3619
 2019-10-15 19:51:01 +01:00
Nick Craig-Wood
6c38bddf3e walk: fix listing with filters listing whole remote 
Prior to this fix, a request such as

    rclone lsf -R --include "/dir/**" remote:

Would use ListR which is very inefficient as it lists the whole remote
for one directory.

This changes it to use recursive walking if the filters imply any
directory filtering.  So `--include *.jpg` and `--exclude *.jpg` will
still use ListR wheras `--include "/dir/**` will not.
 2019-08-08 14:15:50 +01:00
Nick Craig-Wood
57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Nick Craig-Wood
bc70bff125 fs/dirtree: factor DirTree out of fs/walk and add tests 2019-07-02 15:26:55 +01:00
Aleksandar Jankovic
f78cd1e043 Add context propagation to rclone 
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
 2019-06-19 11:59:46 +01:00
Fabian Möller
48c09608ea
fix spelling 2019-04-30 14:12:18 +02:00
Nick Craig-Wood
a28239f005 filter: Make --files-from traverse as before unless --no-traverse is set 
In c5ac96e9e7 we made --files-from only read the objects specified and
don't scan directories.

This caused problems with Google drive (very very slow) and B2
(excessive API consumption) so it was decided to make the old
behaviour (traversing the directories) the default with --files-from
and use the existing --no-traverse flag (which has exactly the right
semantics) to enable the new non scanning behaviour.

See: https://forum.rclone.org/t/using-files-from-with-drive-hammers-the-api/8726

Fixes #3102 Fixes #3095
 2019-04-12 17:16:49 +01:00
Nick Craig-Wood
efd720b533 walk: Implement walk.ListR which will use ListR if at all possible 
It otherwise has the nearly the same interface as walk.Walk which it
will fall back to if it can't use ListR.

Using walk.ListR will speed up file system operations by default and
use much less memory and start immediately compared to if --fast-list
had been supplied.
 2019-03-16 17:41:12 +00:00
Nick Craig-Wood
bdc638530e walk: make NewDirTree always use ListR #2946 
This fixes vfs/refresh with recurse=true needing the --fast-list flag
 2019-02-04 10:37:27 +00:00
Nick Craig-Wood
c5ac96e9e7 Make --files-from only read the objects specified and don't scan directories 
Before this change using --files-from would scan all the directories
that the files could possibly be in causing rclone to do more work
that was necessary.

After this change, rclone constructs an in memory tree using the
--fast-list mechanism but from all of the files in the --files-from
list and without scanning any directories.

Any objects that are not found in the --files-from list are ignored
silently.

This mechanism is used for sync/copy/move (march) and all of the
listing commands ls/lsf/md5sum/etc (walk).
 2018-10-20 18:13:31 +01:00
Nick Craig-Wood
512f4b4487 Update error checking on fmt.Fprint* after errcheck update 
Now we need to check or ignore errors on fmt.Fprint* explicitly -
previously errcheck just ignored them for us.
 2018-05-22 09:41:13 +01:00
Nick Craig-Wood
e0eb666dbf fs/walk: fix new golint warning about unused variables in range 2018-03-02 17:01:58 +00:00
Nick Craig-Wood
11da2a6c9b Break the fs package up into smaller parts. 
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
 2018-01-15 17:51:14 +00:00