extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2025-07-22 14:34:21 +02:00
Code Issues Packages Projects Releases Wiki Activity
4,845 Commits 189 Branches 122 Tags
00a5086ff2ecdb07d328cfe82ba87fdafe4850f3
Commit Graph

56 Commits

Author SHA1 Message Date
Josh Soref
d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Nick Craig-Wood
70c8566cb8 fs: Pin created backends until parents are finalized 
This attempts to solve the backend lifecycle problem by

- Pinning backends mentioned on the command line into the cache
  indefinitely

- Unpinning backends when the containing structure (VFS, wrapping
  backend) is destroyed

See: https://forum.rclone.org/t/rclone-rc-backend-command-not-working-as-expected/18834
 2020-09-01 18:21:03 +01:00
Nick Craig-Wood
47d08ac1f1 vfs: recommend --vfs-cache-modes writes on backends which can't stream 2020-08-18 17:33:27 +01:00
Nick Craig-Wood
15402e46c9 vfs: Add recovered items on cache reload to directory listings 
Before this change, if we restarted an upload after a restart then the
file would get uploaded but never added to the directory listings.

This change makes sure we add virtual items to the directory cache
when reloading the cache so that they show up properly.
 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
2f66355f20 vfs: re-use existing VFS if possible 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
7781ea8d59 vfs: add an optional fs parameter to vfs rc methods 
Before this change we initialized the rc for a single VFS. However
rclone can have multiple VFSes in use now so this is no longer
adequate.

This change adds an optional fs parameter to all the VFS methods to
disambiguate VFSes when there is more than one in use.

It also adds a method vfs/list to show all the active VFSes.

This adds outline tests for the rc commands which didn't have tests
before.
 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
e4e53a2e61 vfs: add --vfs-writeback option to delay writes back to cloud storage 
This is enabled by default and can be disabled with --vfs-writeback 0
 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
cfcdc85b26 vfs: Add SetSys() methods to Node to allow caching stuff on a node 2020-05-04 11:38:07 +01:00
Nick Craig-Wood
70db13e6e8 vfs: pin the Fs in use in the Fs cache 
This means we can reliably look up the Fs from the cache when using
`backend/command`.
 2020-05-01 17:11:45 +01:00
Nick Craig-Wood
eed9c5738d vfs: factor the vfs cache into its own package 2020-04-20 10:42:33 +01:00
Nick Craig-Wood
0961763082 vfs: add utility methods to match os package 2020-04-19 15:40:55 +01:00
Nick Craig-Wood
07908f3f54 vfs: bring open_tests.go generator back into line with the generated tests 
In

54deb01f00 vfs: Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC

The generated file open_test.go was edited directly without editing
the generator.

This commit brings the generator make_open_tests.go back into line
with that edit. It also makes it so `go generate` can be used to
regenerate the tests.
 2020-04-19 15:18:49 +01:00
Nick Craig-Wood
19db0df639 vfs: stop reading Dir members from outside dir.go 2020-04-19 15:16:43 +01:00
Nick Craig-Wood
da41db4712 vfs,mount,cmount: report 1PB free for unknown disk sizes 
Factor the logic into the VFS layer so we don't have to duplicate it
into mount and cmount.

See: https://forum.rclone.org/t/rclone-mount-question/15454/
 2020-04-11 13:31:10 +01:00
Yves G
b1b5e09081 vfs: make df output more consistent on a rclone mount. 
When 2 values are known among vfs:{free,used,total}, compute the 3rd
 2020-03-01 08:54:07 +00:00
Nick Craig-Wood
393b94bb70 vfs: add --vfs-read-wait and --vfs-write-wait flags 
--vfs-read-wait duration    Time to wait for in-sequence read before seeking. (default 5ms)
    --vfs-write-wait duration   Time to wait for in-sequence write before giving error. (default 1s)

See: https://forum.rclone.org/t/constantly-high-iowait-add-log/14156
 2020-02-27 16:12:33 +00:00
Nick Craig-Wood
76f5e273d2 vfs: stop change notify polling clearing so much of the directory cache 
Before this change, change notify polls would clear the directory
cache recursively. So uploading a file to the root would clear the
entire directory cache.

After this change we just invalidate the directory cache of the parent
directory of the item and if the item was a directory we invalidate it
too.
 2019-10-21 20:51:11 +01:00
Brett Dutro
7d0d7e66ca vfs: move writeback of dirty data out of close() method into its own method (FlushWrites) and remove close() call from Flush() 
If a file handle is duplicated with dup() and the duplicate handle is
flushed, rclone will go ahead and close the file, making the original
file handle stale. This change removes the close() call from Flush() and
replaces it with FlushWrites() so that the file only gets closed when
Release() is called. The new FlushWrites method takes care of actually
writing the file back to the underlying storage.

Fixes #3381
 2019-10-09 10:07:29 +01:00
Ivan Andreev
1c4e33d4ad vfs: add flag --vfs-case-insensitive for windows/macOS mounts 
rclone mount when run on Windows & macOS will now default to `--vfs-case-insensitive`.
This means that
 2019-09-04 21:30:48 +01:00
Nick Craig-Wood
a7eec91d69 vfs: add Fs() method to return underlying fs.Fs 2019-08-06 11:43:42 +01:00
Nick Craig-Wood
57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Aleksandar Jankovic
f78cd1e043 Add context propagation to rclone 
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
 2019-06-19 11:59:46 +01:00
Nick Craig-Wood
a43ed567ee vfs: implement --vfs-cache-max-size to limit the total size of the cache 2019-02-02 12:30:10 +00:00
Nick Craig-Wood
554ee0d963 vfs: add --dir-perms and --file-perms flags - fixes #2897 
This allows files to be shown with the execute bit which allows
binaries to be run under Windows and Linux.
 2019-01-08 17:29:38 +00:00
Fabian Möller
1d14972e41 vfs: reduce directory cache cleared by poll-interval 
Reduce the number of nodes purged from the dir-cache when ForgetPath is
called. This is done by only forgetting the cache of the received path
and invalidating the parent folder cache by resetting *Dir.read.

The parent will read the listing on the next access and reuse the
dir-cache of entries in *Dir.items.
 2018-10-02 10:21:14 +01:00
Fabian Möller
1eec59e091 fs: update ChangeNotifier interface 
This introduces a channel to the ChangeNotify function, which can be
used to update the poll-interval and cleanly exit the polling function.
 2018-09-25 14:01:13 +02:00
Fabian Möller
7983b6bdca vfs: enable vfs-read-chunk-size by default 2018-07-29 18:17:05 +01:00
Fabian Möller
98bf65c43b vfs: fix ChangeNotify for new or changed folders 
Fixes #2251
 2018-05-05 12:54:03 +01:00
Fabian Möller
bd3ad1ac3e vfs: add option to read source files in chunks 2018-05-05 12:49:42 +01:00
Nick Craig-Wood
2b855751fc vfs,mount,cmount: use About to return the correct disk total/used/free 
Disks total, used, free now shows correctly for mount and cmount (eg
`df` for Unix or in the Windows explorer).
 2018-04-18 18:27:34 +01:00
Nick Craig-Wood
2b7957cc74 vfs: Only make the VFS cache if --vfs-cache-mode > Off 
This stops the cache cleaner running unnecessarily and saves
resources.

This also helps with issue #2227 which was caused by a second mount
deleting objects in the first mounts cache.
 2018-04-16 17:06:41 +01:00
Nick Craig-Wood
80588a5a6b Replace "golang.org/x/net/context" with "context" for go1.7+ #2154 2018-04-07 11:42:08 +01:00
Nick Craig-Wood
2db0c4dd95 vfs: add remote control for directory cache flushing 2018-03-17 10:34:02 +00:00
Remus Bunduc
70f07fd3ac fs: add ChangeNotify and backend support for it (#2094) 
* fs: rename DirChangeNotify to ChangeNotify

* cache: switch to ChangeNotify

* ChangeNotify: keep order of notifications
 2018-03-08 22:03:34 +02:00
Nick Craig-Wood
54deb01f00 vfs: Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC 
Before this change Open("name", os.O_RDONLY|os.O_TRUNC) would have
truncated the file.  This is what Linux does, but is counterintuitive.
POSIX states this is undefined, so return an error in this case
instead.  This preserves the invariant O_RDONLY => file is not
changed.
 2018-02-26 17:04:27 +00:00
Nick Craig-Wood
04a0a7406b vfs: downgrade "poll-interval is not supported" message to Info 
...to save confusion as it isn't very important
 2018-02-09 07:57:50 +00:00
Nick Craig-Wood
a80d8a21dc vfs: add flags parameter to Dir.Create 2018-01-19 13:18:40 +00:00
Nick Craig-Wood
11da2a6c9b Break the fs package up into smaller parts. 
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
 2018-01-15 17:51:14 +00:00
Nick Craig-Wood
b4083b4371 vfs: rename Fsync to Sync and implement Sync on Node and Handle 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
9738f8532b vfs: Add FlushDirCache method 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
a5b034a992 vfs: add WaitForWriters to wait until all writers have finished 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
c560017934 vfs: add Path method to Node and use it to stop reading nil DirEntry 
All DirEntry calls now have been checked for nil or converted to use Path.
 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
e7f2935333 vfs: decode flags in Open/OpenFile for debug 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
7f20e1d7f3 vfs: add read write files and caching #711 
This adds new flags to mount, cmount, serve *

    --cache-max-age duration         Max age of objects in the cache. (default 1h0m0s)
    --cache-mode string              Cache mode off|minimal|writes|full (default "off")
    --cache-poll-interval duration   Interval to poll the cache for stale objects. (default 1m0s)
 2017-11-20 12:36:50 +00:00
Nick Craig-Wood
3fb1e96988 vfs: factor Open logic from Dir.Create into vfs.OpenFile 2017-11-20 12:00:32 +00:00
Fabian Möller
5acb6f47e7 mountlib: log when poll-interval is ineffective 
Notify the user in case poll-interval is used on a unsupported remote
 2017-11-19 20:08:14 +00:00
Nick Craig-Wood
2083ac6e2a vfs: add ECLOSED and tidy errors 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
22ee839d05 cmount,vfs: unify Read and Write handles and File and Dir where possible 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
5634659ea3 mount,vfs: unify Read and Write handles in preparation for ReadWrite handles 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
e18122e88b vfs: add tests and subsequent fixes 
* Tests for VFS layer
  * Small fixes found during testing
  * Fix Close, Flush and Release behaviour for ReadFileHandle and WriteFileHandle
  * Fix nil object bugs on File
 2017-11-04 10:24:10 +00:00