extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2025-02-02 03:29:51 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,546 Commits 170 Branches 115 Tags 360 MiB
8cfe42d09f
Commit Graph

84 Commits

Author SHA1 Message Date
izouxv
0275d3edf2
vfs: close the change notify channel on Shutdown 2025-01-17 15:38:09 +00:00
Dimitri Papadopoulos
b1d4de69c2 docs: fix typos found by codespell in docs and code comments 2025-01-16 10:39:01 +01:00
Nick Craig-Wood
a5abe4b8b3 vfs: add symlink support to VFS 
This is somewhat limited in that it only resolves symlinks when files
are opened. This will work fine for the intended use in rclone mount,
but is inadequate for the other servers probably.
 2024-12-13 12:43:20 +00:00
Filipe Azevedo
126f00882b vfs: Add VFS --links command line switch 
This will be used to enable links support for the various mount engines
in a follow up commit.
 2024-12-13 12:43:20 +00:00
Nick Craig-Wood
44c3f5e1e8 vfs: add vfs.WriteFile to match os.WriteFile 2024-12-13 12:43:20 +00:00
Ilias Ozgur Can Leonard
2446c4928d vfs: with --vfs-used-is-size value is calculated and then thrown away - fixes #8220 2024-12-04 22:57:41 +00:00
Nick Craig-Wood
a28287e96d vfs: convert vfs options to new style 
This also
- move in use options (Opt) from vfsflags to vfscommon
- change os.FileMode to vfscommon.FileMode in parameters
- rework vfscommon.FileMode and add tests
 2024-07-15 11:09:54 +01:00
Nick Craig-Wood
fc1d8dafd5 vfs: convert time.Duration option to fs.Duration 2024-07-15 11:09:54 +01:00
albertony
e739ee2c27 docs: ensure empty line between text and a following heading 2024-04-05 21:39:44 +02:00
Nick Craig-Wood
831d1df67f docs: factor large docs into separate .md files to make them easier to maintain. 
We then use the go embed command to embed them back into the binary.
 2023-11-13 16:27:09 +00:00
Saleh Dindar
25f59b2918 vfs: Add go-billy dependency and make sure vfs.Handle implements billy.File 
billy defines a common file system interface that is used in multiple go packages.
vfs.Handle implements billy.File mostly, only two methods needed to be added to
make it compliant.

An interface check is added as well.

This is a preliminary work for adding serve nfs command.
 2023-10-06 14:08:20 +01:00
Beyond Meat
3337fe31c7 vfs: add --vfs-refresh flag to read all the directories on start 
Refreshes the directory listing recursively at VFS start time.
 2023-10-06 13:11:09 +01:00
Roberto Ricci
91b8152321 vfs: use atomic types 2023-08-22 12:52:13 +01:00
Nick Craig-Wood
fc798d800c vfs: fix backends being Shutdown too early when startup takes a long time 
Before this change if the VFS took more than 5 to initialise (which
can happen if there is a lot of files or a lot of files which need
uploading) the backend was dropped out of the cache before the VFS was
fully created.

This was noticeable in the dropbox backend where the batcher Shutdown
too soon and prevented further uploads.

This fixes the problem by Pinning backends before the VFS cache is
created.

https://forum.rclone.org/t/if-more-than-251-elements-in-the-que-to-upload-fails-with-batcher-is-shutting-down/38076/2
 2023-05-18 16:16:12 +01:00
Nick Craig-Wood
9a9ef040e3 vfs: fix reload: failed to add virtual dir entry: file does not exist 
This error happened on a restart of the VFS with files to upload into
a new directory on a bucket based backend. Rclone was assuming that
directories created before the restart would still exist, but this is
a bad assumption for bucket based backends which don't really have
directories.

This change creates the pretend directory and thus the directory cache
if the parent directory does not exist when adding a virtual on a
backend which can't have empty directories.

See: https://forum.rclone.org/t/that-pesky-failed-to-reload-error-message/34527
 2023-04-13 18:00:26 +01:00
Nick Craig-Wood
ca403dc90e vfs: add MkdirAll function to make a directory and all beneath 2023-04-13 18:00:22 +01:00
Dimitri Papadopoulos
9ee1b21ec2 vfs: fix typos found by codespell 2023-03-25 09:33:34 +01:00
albertony
5d6b8141ec Replace deprecated ioutil 
As of Go 1.16, the same functionality is now provided by package io or
package os, and those implementations should be preferred in new code.
 2022-11-07 11:41:47 +00:00
albertony
555def2da7 build: add package comments to silence revive linter 2022-08-28 13:43:51 +02:00
Claudio Maradonna
62bcc84f6f vfs: add --vfs-disk-space-total-size option to manually set the total disk space 
Now you can specify --vfs-disk-space-total-size to set the total disk
space (default to -1)

fixes #3270
 2022-07-08 17:26:54 +01:00
albertony
7b8c974dec staticcheck: ineffective break statement 2022-07-04 11:24:59 +02:00
Nick Craig-Wood
626a416ff8 vfs: factor out the VFS option initialization for re-use #3259 2022-06-16 16:48:09 +01:00
Nick Craig-Wood
d252816706 vfs: add vfs/stats remote control to show statistics - fixes #5816 2021-11-23 18:00:21 +00:00
Nick Craig-Wood
a98e3ea6f1 build: replace the deprecated golint linter with revive 
This fixes up a small number of new lint items also
 2021-10-15 12:51:31 +01:00
albertony
0574ebf44a vfs: do not print notice about missing poll-interval support when set to 0 
Fixes #5359
 2021-05-28 13:09:15 +02:00
albertony
2925e1384c Use binary prefixes for size and rate units 
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
 2021-04-27 02:25:52 +03:00
tYYGH
c0cf54067a
vfs: --vfs-used-is-size to report used space using recursive scan (#4043) 
Some backends, most notably S3, do not report the amount of bytes used.
This patch introduces a new flag that allows instead of relying on the
backend, use recursive scan similar to `rclone size` to compute the total
used space. However, this is ineffective and should be used as a last resort.

Co-authored-by: Yves G <theYinYeti@yalis.fr>
 2021-02-17 23:36:13 +03:00
Josh Soref
e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Nick Craig-Wood
70c8566cb8 fs: Pin created backends until parents are finalized 
This attempts to solve the backend lifecycle problem by

- Pinning backends mentioned on the command line into the cache
  indefinitely

- Unpinning backends when the containing structure (VFS, wrapping
  backend) is destroyed

See: https://forum.rclone.org/t/rclone-rc-backend-command-not-working-as-expected/18834
 2020-09-01 18:21:03 +01:00
Nick Craig-Wood
47d08ac1f1 vfs: recommend --vfs-cache-modes writes on backends which can't stream 2020-08-18 17:33:27 +01:00
Nick Craig-Wood
15402e46c9 vfs: Add recovered items on cache reload to directory listings 
Before this change, if we restarted an upload after a restart then the
file would get uploaded but never added to the directory listings.

This change makes sure we add virtual items to the directory cache
when reloading the cache so that they show up properly.
 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
2f66355f20 vfs: re-use existing VFS if possible 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
7781ea8d59 vfs: add an optional fs parameter to vfs rc methods 
Before this change we initialized the rc for a single VFS. However
rclone can have multiple VFSes in use now so this is no longer
adequate.

This change adds an optional fs parameter to all the VFS methods to
disambiguate VFSes when there is more than one in use.

It also adds a method vfs/list to show all the active VFSes.

This adds outline tests for the rc commands which didn't have tests
before.
 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
e4e53a2e61 vfs: add --vfs-writeback option to delay writes back to cloud storage 
This is enabled by default and can be disabled with --vfs-writeback 0
 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
cfcdc85b26 vfs: Add SetSys() methods to Node to allow caching stuff on a node 2020-05-04 11:38:07 +01:00
Nick Craig-Wood
70db13e6e8 vfs: pin the Fs in use in the Fs cache 
This means we can reliably look up the Fs from the cache when using
`backend/command`.
 2020-05-01 17:11:45 +01:00
Nick Craig-Wood
eed9c5738d vfs: factor the vfs cache into its own package 2020-04-20 10:42:33 +01:00
Nick Craig-Wood
0961763082 vfs: add utility methods to match os package 2020-04-19 15:40:55 +01:00
Nick Craig-Wood
07908f3f54 vfs: bring open_tests.go generator back into line with the generated tests 
In

54deb01f00 vfs: Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC

The generated file open_test.go was edited directly without editing
the generator.

This commit brings the generator make_open_tests.go back into line
with that edit. It also makes it so `go generate` can be used to
regenerate the tests.
 2020-04-19 15:18:49 +01:00
Nick Craig-Wood
19db0df639 vfs: stop reading Dir members from outside dir.go 2020-04-19 15:16:43 +01:00
Nick Craig-Wood
da41db4712 vfs,mount,cmount: report 1PB free for unknown disk sizes 
Factor the logic into the VFS layer so we don't have to duplicate it
into mount and cmount.

See: https://forum.rclone.org/t/rclone-mount-question/15454/
 2020-04-11 13:31:10 +01:00
Yves G
b1b5e09081 vfs: make df output more consistent on a rclone mount. 
When 2 values are known among vfs:{free,used,total}, compute the 3rd
 2020-03-01 08:54:07 +00:00
Nick Craig-Wood
393b94bb70 vfs: add --vfs-read-wait and --vfs-write-wait flags 
--vfs-read-wait duration    Time to wait for in-sequence read before seeking. (default 5ms)
    --vfs-write-wait duration   Time to wait for in-sequence write before giving error. (default 1s)

See: https://forum.rclone.org/t/constantly-high-iowait-add-log/14156
 2020-02-27 16:12:33 +00:00
Nick Craig-Wood
76f5e273d2 vfs: stop change notify polling clearing so much of the directory cache 
Before this change, change notify polls would clear the directory
cache recursively. So uploading a file to the root would clear the
entire directory cache.

After this change we just invalidate the directory cache of the parent
directory of the item and if the item was a directory we invalidate it
too.
 2019-10-21 20:51:11 +01:00
Brett Dutro
7d0d7e66ca vfs: move writeback of dirty data out of close() method into its own method (FlushWrites) and remove close() call from Flush() 
If a file handle is duplicated with dup() and the duplicate handle is
flushed, rclone will go ahead and close the file, making the original
file handle stale. This change removes the close() call from Flush() and
replaces it with FlushWrites() so that the file only gets closed when
Release() is called. The new FlushWrites method takes care of actually
writing the file back to the underlying storage.

Fixes #3381
 2019-10-09 10:07:29 +01:00
Ivan Andreev
1c4e33d4ad vfs: add flag --vfs-case-insensitive for windows/macOS mounts 
rclone mount when run on Windows & macOS will now default to `--vfs-case-insensitive`.
This means that
 2019-09-04 21:30:48 +01:00
Nick Craig-Wood
a7eec91d69 vfs: add Fs() method to return underlying fs.Fs 2019-08-06 11:43:42 +01:00
Nick Craig-Wood
57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Aleksandar Jankovic
f78cd1e043 Add context propagation to rclone 
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
 2019-06-19 11:59:46 +01:00