Commit Graph

132 Commits

Author SHA1 Message Date
Nick Craig-Wood
d260238f99 cmount: use ReaddirPlus on Windows to improve directory listing performance
Before this change Windows would read a directory then immedately stat
every item in the directory.

After this change we return the stat information along with the
directory which stops so many callbacks.
2020-05-04 11:38:07 +01:00
Nick Craig-Wood
3f654dac37 mount: map more rclone errors into file systems errors
This improves the error reporting, in particular for
fs.ErrorPermissionDenied which was being reported as an IO error.
2020-04-21 16:31:43 +01:00
Nick Craig-Wood
eed9c5738d vfs: factor the vfs cache into its own package 2020-04-20 10:42:33 +01:00
Nick Craig-Wood
fd39cbc193 vfstests: move functional tests from mountlib and make them work with VFS
The tests are now run for the mount commands and for the plain VFS.

This makes the tests much easier to debug when running with a VFS than
through a mount.
2020-04-20 10:42:33 +01:00
Nick Craig-Wood
da41db4712 vfs,mount,cmount: report 1PB free for unknown disk sizes
Factor the logic into the VFS layer so we don't have to duplicate it
into mount and cmount.

See: https://forum.rclone.org/t/rclone-mount-question/15454/
2020-04-11 13:31:10 +01:00
Nick Craig-Wood
ec1271818f mount2: hide mount2 command for the moment 2020-02-11 14:28:13 +00:00
Nick Craig-Wood
0b7f959433 cmount: when setting dates discard out of range dates
It appears that sometimes Windows/WinFSP/cgofuse sends dates which are
the epoch to rclone.  These dates appear as 1601-01-01 00:00:00 plus
or minus the timezone.

These dates aren't being sent from rclone.

This patch filters dates out before 1601-01-02 so rclone does not
attempt to set them.

See: https://forum.rclone.org/t/bug-corruption-of-modtime-via-vfs-layer/12204
See: https://forum.rclone.org/t/io-error-googleapi-error-403-insufficient-permission-insufficientpermissions/11372
See: https://github.com/billziss-gh/cgofuse/issues/35
2020-01-18 11:13:35 +00:00
Nick Craig-Wood
c21a4fee58 mount,cmount: make sure we call unmount when exiting 2019-11-11 22:08:52 +00:00
Brett Dutro
378a3f4133 mount: replace use of WriteAt with Write for cache mode >= writes and O_APPEND
os.File.WriteAt returns an error if a file was opened with O_APPEND.
This replaces it with os.File.Write if the file was opened with
O_APPEND.
2019-10-26 17:27:52 +01:00
Nick Craig-Wood
59026c4761 mount, cmount: don't pass huge filenames (>4k) to FUSE as it can't cope 2019-10-21 20:51:11 +01:00
Nick Craig-Wood
0baafb158f mount: allow files of unkown size to be read properly
Before this change, files of unknown size (eg Google Docs) would
appear in file listings with 0 size and would only allow 0 bytes to be
read.

This change sets the direct_io flag in the FUSE return which bypasses
the cache for these files.  This means that they can be read properly.

This is compatible with some, but not all applications.
2019-09-14 13:22:33 +01:00
Nick Craig-Wood
ffa1dac10b build: apply gofmt from go1.13 to change case of number literals 2019-09-05 13:59:06 +01:00
Nick Craig-Wood
57d5de6fba build: fix up package paths after repo move
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
2019-07-28 18:47:38 +01:00
Fabian Möller
48c09608ea
fix spelling 2019-04-30 14:12:18 +02:00
Nick Craig-Wood
84e21ade6b cmount: fix on Linux - only apply volname for Windows and macOS 2018-11-22 20:41:05 +00:00
Nick Craig-Wood
b3704597f3 cmount: make --volname work for Windows - fixes #2679 2018-11-12 16:32:02 +00:00
Nick Craig-Wood
174ca22936 mount,cmount: clip the number of blocks to 2^32-1 on macOS
OSX FUSE only supports 32 bit number of blocks which means that block
counts have been wrapping.  This causes f_bavail to be 0 which in turn
causes problems with programs like borg backup.

Fixes #2356
2018-08-26 17:32:59 +01:00
Nick Craig-Wood
77806494c8 mount,cmount: adapt to sdnotify API change 2018-08-20 12:34:29 +01:00
Nick Craig-Wood
473e3c3eb8 mount/cmount: implement --daemon-timeout flag for OSXFUSE
By default the timeout is 60s which isn't long enough for long
transactions.  The symptoms are rclone just quitting for no reason.
Supplying the --daemon-timeout flag fixes this causing the kernel to
wait longer for rclone.
2018-07-19 13:26:51 +01:00
Nick Craig-Wood
9d8d7ae1f0 mount,cmount: make --noappledouble --noapplexattr and change defaults #2287
Before this change we would unconditionally set the OSXFUSE options
noappledouble and noapplexattr.

However the noapplexattr options caused problems with copies in the
Finder.

Now the default for noapplexattr is false so we don't add the option
by default and the user can override the defaults using the
--noappledouble and --noapplexattr flags.
2018-05-07 20:37:09 +01:00
Nick Craig-Wood
6ce32e4661 mount,cmount: Add --volname flag and remove special chars from it #2287
Before this change rclone would set the volume name from the
remote:path normally.  However this has `:` and `/` in which make it
difficult to use in macOS.

Now rclone will remove the special characters and replace them with
spaces.  It also allows the volume name to be set with the --volname
flag.
2018-05-07 20:37:09 +01:00
Nick Craig-Wood
cb5bd47e61 build: fix errors spotted by ineffassign linter
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
2018-05-05 17:32:41 +01:00
Nick Craig-Wood
2b855751fc vfs,mount,cmount: use About to return the correct disk total/used/free
Disks total, used, free now shows correctly for mount and cmount (eg
`df` for Unix or in the Windows explorer).
2018-04-18 18:27:34 +01:00
Nick Craig-Wood
06e3fa3aba mounttest: reduce duplicated code and improve test output #2154
The written out list of tests was replaced with a nested test for
mount and cmount. The tests for each VFS cache mode were also replaced
with nested tests which makes the output and the code much cleaner.
2018-04-08 15:04:14 +01:00
Nick Craig-Wood
4f6f07c074 cmount: fix error handling for Open/OpenDir 2018-03-21 19:44:30 +00:00
Nick Craig-Wood
34c45a7c04 mount, cmount: remove addition of O_CREATE to flags on file open #2141
Previously this was adding it in to all file opens which was causing
inefficiencies under Windows where it stats the file using
open/fstat/close.

This change will make stat operations run much quicker under Windows
as they won't have to open the underlying file

This problem was introduced in61b6159a05336bd7ba105766de2d2ff171f7fb81
where we added O_CREATE to all file opens and creates.
2018-03-15 20:48:56 +00:00
Nick Craig-Wood
7713acf23d mount: skip failing test TestFileModTimeWithOpenWriters on Windows 2018-03-09 23:15:38 +00:00
Nick Craig-Wood
fc32fee4ad mount, cmount: add --attr-timeout to control attribute caching in kernel
This flag allows the attribute caching in the kernel to be controlled.
The default is 0s - no caching - which is recommended for filesystems
which can change outside the control of the kernel.

Previously this was at the default meaning it was 60s for mount and 1s
for cmount.  This showed strange effects when files changed on the
remote not via the kernel.  For instance Caddy would serve corrupted
files for a while when serving from an rclone mount when a file
changed on the remote.
2018-03-04 11:20:22 +00:00
Nick Craig-Wood
54deb01f00 vfs: Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC
Before this change Open("name", os.O_RDONLY|os.O_TRUNC) would have
truncated the file.  This is what Linux does, but is counterintuitive.
POSIX states this is undefined, so return an error in this case
instead.  This preserves the invariant O_RDONLY => file is not
changed.
2018-02-26 17:04:27 +00:00
Nick Craig-Wood
a80d8a21dc vfs: add flags parameter to Dir.Create 2018-01-19 13:18:40 +00:00
Nick Craig-Wood
11da2a6c9b Break the fs package up into smaller parts.
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
2018-01-15 17:51:14 +00:00
Nick Craig-Wood
2b666187a6 cmount: disable tests on windows + race detector
These either hang or produce incorrect results for reasons I haven't
worked out yet.
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
eb3415db50 cmount: enable more tests for Windows 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
61b6159a05 mount, cmount: add O_CREATE to Open calls since fuse doesn't seem to supply it 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
c560017934 vfs: add Path method to Node and use it to stop reading nil DirEntry
All DirEntry calls now have been checked for nil or converted to use Path.
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
7c3584f4e6 mountlib: wait for mountpoint to disappear under Windows 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
dec21ccf63 vfs, cmount: make truncate work properly in the presence or otherwise of open files 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
94adf4f43b cmount: translate FUSE open flags into OS flags
On Windows the fuse.O_* flags do not have the same values as the
os.O_* flags so translate between the two representations.  They are
mostly the same which is why this hasn't caused a problem before.
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
f5f8c0c438 cmount: make Truncate call the correct Handle or Node method 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
60cdcf784c cmount: use -o atomic_o_trunc to make sure O_TRUNC is supplied to Open() 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
57a5c67729 mounttest: run the tests for all 4 VFS cache modes 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
8951875c21 vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
3fb1e96988 vfs: factor Open logic from Dir.Create into vfs.OpenFile 2017-11-20 12:00:32 +00:00
Fabian Möller
d8b3bf014d mount: use sdnotify to signal systemd the mount is ready
When the NOTIFY_SOCKET environment variable is set notify systemd after
the mount is ready.
2017-11-20 11:03:10 +00:00
Nick Craig-Wood
2083ac6e2a vfs: add ECLOSED and tidy errors 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
22ee839d05 cmount,vfs: unify Read and Write handles and File and Dir where possible 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
e18122e88b vfs: add tests and subsequent fixes
* Tests for VFS layer
  * Small fixes found during testing
  * Fix Close, Flush and Release behaviour for ReadFileHandle and WriteFileHandle
  * Fix nil object bugs on File
2017-11-04 10:24:10 +00:00
Nick Craig-Wood
8184ec4b70 vfs: add EPERM to errors 2017-11-03 13:00:00 +00:00
Nick Craig-Wood
190367d917 vfs: factor duplicated Open code into vfs from mount/cmount 2017-11-03 13:00:00 +00:00
Nick Craig-Wood
7e065440fb vfs: rename Lookup to Stat to be more in keeping with os 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
e8883e9fdb vfs: factor flags into vfsflags and remove global variables 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
1a8f824bad vfs: use os package errors where possible 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
c1aaff220d Factor new vfs module out of cmd/mountlib
This is an OS style file system abstraction with directory caching
used in mount, cmount, serve webdav and serve http.
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
ca19fd2d7e mountlib: Make read/write file handles support more standard interfaces
Including Read, ReadAt, Seek, Close for read handles and Write,
WriteAt, Close for read handles.
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
2fac74b517 mountlib: store only Node in *Dir removing DirEntry struct 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
0bb84efe75 mountlib: Rename Remove to RemoveName 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
750690503e mountlib: make Node satisfy os.FileInfo interface 2017-11-03 12:59:58 +00:00
Nick Craig-Wood
dcce65b2b3 mount/cmount: factor duplicated code into mountlib 2017-06-19 14:36:51 +01:00
Stefan Breunig
9782c264e9 hand dirCacheTime through again 2017-06-01 09:02:22 +01:00
Stefan Breunig
a2e3af0523 poll for Google Drive changes when mounted 2017-05-28 17:54:52 +01:00
Nick Craig-Wood
8f54dc06a2 Use build tags to control when and where cmount is built 2017-05-19 17:08:04 +01:00
Nick Craig-Wood
ecedcd0e7f cmount: stop failing tests on Windows 2017-05-15 17:40:44 +01:00
Nick Craig-Wood
b259f8b752 cmount, mount, mountlib: make --read-only reject modify operations
Normally mount/cmount use `-o ro` to get the kernel to mark the fs as
read only.  However this is ignored by WinFsp, so in addition if
`--read-only` is in effect then return EROFS ("Read only File System")
from all methods which attempt to modify something.
2017-05-15 16:56:47 +01:00
Nick Craig-Wood
204a19e67f cmount: Wait for mountpoint to appear on Windows before declaring mounted 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
e6ffe3464c cmount: check for filesystem blowing up before Init is called 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
0384364c3e cmount: pass --FileSystemName under windows 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
763facfd78 cmount: implement --fuse-flag to pass commands to fuse library directly
Useful for `--fuse-flag -h` to see exactly which options the library supports.
2017-05-15 16:56:46 +01:00
Nick Craig-Wood
bc88f1dafa cmount: fix openFile leak 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
0c055a1215 cmount: Statfs: reduce max size of volume for Windows 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
938d7951ab cmount: allow extra options to pass to fuse with -o 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
b4466bd9b1 Add -o uid=-1 -o gid=-1 for Windows/WinFsp 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
31f76aa464 cmount: implement no-ops for Fsync, Chmod, Chown, Access, Fsyncdir and stop using fuse.FileSystemBase 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
c887c164dc cmount: add function tracing 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
9c3048580a cmount: fix code quality warnings 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
bc9856b570 Forward port 930ff266f2 to cmount branch
compare checksums on upload/download via FUSE
2017-05-15 16:56:44 +01:00
Nick Craig-Wood
855071cc19 cmount: name the command mount under windows and cmount under linux 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
b179540e80 cmount: fix Getattr to work on directories 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
7b47a1e842 cmount: set the correct values for uid, gid under Windows 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
bcd87009e2 Fix docs typo 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
caf85737c3 cmount: fix Windows compile (thanks Bill Zissimopoulos) 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
e1516e0159 Forward port 58a82cd578 into cmount branch
allow the fuse directory cached to be cleaned manually
2017-05-15 16:56:43 +01:00
Nick Craig-Wood
ee1111e4c9 cmount: a new mount option based on cgofuse.
This with the aid of WinFSP should work on Windows.

Unfinished bits
  * 1 test doesn't pass
  * docs
  * build
2017-05-15 16:56:43 +01:00