extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-11-23 00:43:49 +01:00
Code Issues Packages Projects Releases Wiki Activity
3,582 Commits 177 Branches 114 Tags 315 MiB
59dba1de88
Commit Graph

523 Commits

Author SHA1 Message Date
Nick Craig-Wood
e42edc8e8c copy, move: Copy single files directly, don't use --files-from work-around 
Before this change rclone would inefficiently and confusingly read all
the files in the source directory when copy or moving a single file.
This caused confusion for the users to see log messages about files
which weren't part of the sync.

After the change the copy and move commands use the new infrastructure
made for the copyto and moveto command for single file copy and move.
 2018-05-07 20:39:52 +01:00
Nick Craig-Wood
291954baba cmd: make names of argument parsing functions more consistent 2018-05-07 20:39:52 +01:00
Nick Craig-Wood
9d8d7ae1f0 mount,cmount: make --noappledouble --noapplexattr and change defaults #2287 
Before this change we would unconditionally set the OSXFUSE options
noappledouble and noapplexattr.

However the noapplexattr options caused problems with copies in the
Finder.

Now the default for noapplexattr is false so we don't add the option
by default and the user can override the defaults using the
--noappledouble and --noapplexattr flags.
 2018-05-07 20:37:09 +01:00
Nick Craig-Wood
6ce32e4661 mount,cmount: Add --volname flag and remove special chars from it #2287 
Before this change rclone would set the volume name from the
remote:path normally.  However this has `:` and `/` in which make it
difficult to use in macOS.

Now rclone will remove the special characters and replace them with
spaces.  It also allows the volume name to be set with the --volname
flag.
 2018-05-07 20:37:09 +01:00
Nick Craig-Wood
1755ffd1f3 mount: make Get/List/Set/Remove xattr return ENOSYS #2287 
By default bazil fuse will return ENOTSUPP for these.  However if we
return ENOSYS then OSXFUSE (at least) will never call them again
saving round trips though fuse.
 2018-05-07 20:37:09 +01:00
Nick Craig-Wood
e80ae4e09c build: remove unused struct fields spotted by structcheck 2018-05-05 17:32:41 +01:00
Nick Craig-Wood
1320e84bc2 build: remove unused code spotted by the deadcode linter 2018-05-05 17:32:41 +01:00
Nick Craig-Wood
cb5bd47e61 build: fix errors spotted by ineffassign linter 
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
 2018-05-05 17:32:41 +01:00
Nick Craig-Wood
f1a43eca4d mount: make --daemon work for macOS without CGO 2018-05-05 16:23:47 +01:00
Nick Craig-Wood
21383877df cmd: make exit code 8 for --max-transfer exceeded 2018-05-05 12:58:28 +01:00
Fabian Möller
bd3ad1ac3e vfs: add option to read source files in chunks 2018-05-05 12:49:42 +01:00
NoLooseEnds
a5f1811892 cmd: Fixed a typo – minimum 2018-05-01 09:42:21 +01:00
Nick Craig-Wood
d8e88f10cd rc: take note of the --rc-addr flag too as per the docs - fixes #2184 2018-04-26 17:00:44 +01:00
Richard Yang
a81ec00a8c dedupe: Add dedupe largest functionality - fixes #2269 2018-04-26 16:21:07 +01:00
Nick Craig-Wood
96a62d55a2 lsd: Add -R flag and fix and update docs for all ls commands 2018-04-26 08:55:03 +01:00
Nick Craig-Wood
9fbc40c5b9 fstests: List missing dir must return ErrorDirNotFound for non bucket based remotes 
List or ListR of an non existent directory must return
ErrorDirNotFound for non bucket based remotes.  For bucket based
remotes it may return ErrorDirNotFound or it may return no error and
no entries.
 2018-04-20 23:22:46 +01:00
Nick Craig-Wood
c51d97c752 hashsum: make generic tool for any hash to produce md5sum like output 2018-04-20 21:02:37 +01:00
Nick Craig-Wood
1f255a8567 Add a mega.nz remote #163 
Not supported yet:
  * Hash
  * ModTime
  * Server Side Copy

Otherwise fully functional and passing all the tests.
 2018-04-18 21:09:54 +01:00
Nick Craig-Wood
9948b39dba about: don't attempt retries 2018-04-18 21:09:54 +01:00
Nick Craig-Wood
2b855751fc vfs,mount,cmount: use About to return the correct disk total/used/free 
Disks total, used, free now shows correctly for mount and cmount (eg
`df` for Unix or in the Windows explorer).
 2018-04-18 18:27:34 +01:00
Nick Craig-Wood
1ac6dacf0f about: complete other providers and re-work internals 
* Implement about for:
    * local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
  * Implement `--json` and `---full` flag for `rclone about`
  * change About interface to return a Usage structure
  * Remove operations.About as it is too thin an interface
  * Implement Integration test

Relates to #1138 and #1564
 2018-04-17 21:53:27 +01:00
a-roussos
94e277d759 about: add new command 'about' to get quota info from a remote 
Implemented for drive only.

Relates to #1138 and #1564.
 2018-04-17 21:50:14 +01:00
Nick Craig-Wood
2b7957cc74 vfs: Only make the VFS cache if --vfs-cache-mode > Off 
This stops the cache cleaner running unnecessarily and saves
resources.

This also helps with issue #2227 which was caused by a second mount
deleting objects in the first mounts cache.
 2018-04-16 17:06:41 +01:00
Matthew Holt
9e4cd55477 size: Add --json flag 2018-04-13 13:38:06 +01:00
Nick Craig-Wood
dcf8334673 fs: add --dump goroutines and --dump openfiles 
These are developer flags useful for tracking down resource leaks.
 2018-04-11 20:55:58 +01:00
Nick Craig-Wood
06e3fa3aba mounttest: reduce duplicated code and improve test output #2154 
The written out list of tests was replaced with a nested test for
mount and cmount. The tests for each VFS cache mode were also replaced
with nested tests which makes the output and the code much cleaner.
 2018-04-08 15:04:14 +01:00
Nick Craig-Wood
be54fd8f70 Remove builds conditional on go1.7 since that is now guaranteed #2154 
Old fallback code was deleted and the go1.7 style code inlined where
appropriate.
 2018-04-07 11:42:55 +01:00
Nick Craig-Wood
e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood
80588a5a6b Replace "golang.org/x/net/context" with "context" for go1.7+ #2154 2018-04-07 11:42:08 +01:00
Matt Holt
e13f65b953 serve restic: Print actual listener address 2018-04-04 14:56:26 +01:00
Alexander Neumann
5b8977a053 serve restic: Disallow overwriting files in append-only mode - Fixes #2195 
* Disallow overwriting files in append-only mode
* Add tests for append-only mode
 2018-04-04 14:49:13 +01:00
Steve Kriss
2c2bb0f750 cmd/serve/restic: add append-only mode 2018-03-30 19:54:52 +01:00
Stefan
a8267d1628
link: allow creating public link to files and folders - closes #1562 2018-03-29 09:10:19 +02:00
Keith Goldfarb
6c10312c75 ncdu: added a "refresh" key - for #2174 
Added Control+L key to refresh screen. Not sure if this is the
best choice, but it appears to be somewhat common.
 2018-03-26 21:02:39 +01:00
Nick Craig-Wood
98a924602f mount, cmount: set --attr-timeout default to 1s - fixes #2157 
This  works around these 3 problems:

  * rclone using too much memory #2157
  * rclone not serving files to samba
    * https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112
  * excessive time listing directories #2095
 2018-03-23 22:42:51 +00:00
Nick Craig-Wood
4f6f07c074 cmount: fix error handling for Open/OpenDir 2018-03-21 19:44:30 +00:00
Nick Craig-Wood
1afac32d80 serve restic: script for running integration test against all remotes 2018-03-18 19:15:39 +00:00
Nick Craig-Wood
26fbd00b4f serve restic: don't buffer the JSON output in memory for the list command 2018-03-18 16:26:58 +00:00
Nick Craig-Wood
1313b529ff serve restic: use ListR (--fast-list) if available 
For Restic's use case, --fast-list will use less transactions and
calling ListR directly means we can avoid the usual memory overhead.
 2018-03-18 16:22:05 +00:00
Nick Craig-Wood
82e835d6fc serve restic: make it easy to run integration tests against any remote 
Just `cd cmd/serve/restic` then `go test -v -remote TestRemote:`
 2018-03-18 14:23:56 +00:00
Nick Craig-Wood
fa867a9a4c serve restic: implement accounting for uploads and downloads 
This means the bandwidth stats will be correct and the bandwidth
throttling will work correctly.  This was forgotten as a previous
iteration of the code was using the higher level operations.Rcat which
took care of this.
 2018-03-18 14:19:43 +00:00
Stefan
86892467d9
config: load config file only on first access (closes #1659, closes #2096) (#2147) 2018-03-17 12:36:30 +01:00
Nick Craig-Wood
34c45a7c04 mount, cmount: remove addition of O_CREATE to flags on file open #2141 
Previously this was adding it in to all file opens which was causing
inefficiencies under Windows where it stats the file using
open/fstat/close.

This change will make stat operations run much quicker under Windows
as they won't have to open the underlying file

This problem was introduced in61b6159a05336bd7ba105766de2d2ff171f7fb81
where we added O_CREATE to all file opens and creates.
 2018-03-15 20:48:56 +00:00
Alexander Neumann
84776c4e43 serve/restic: Remove log message on Close 2018-03-14 21:50:33 +00:00
Nick Craig-Wood
c1a3e363a6 mount: return ENOSYS rather than EIO on attempted link 
This fixes FileZilla accessing an rclone mount served over sftp.

See: https://forum.rclone.org/t/moving-files-on-rclone-mount-with-filezilla/5029
 2018-03-14 21:10:20 +00:00
Nick Craig-Wood
7ccc6080b0 serve restic: add more info to GET request error 2018-03-14 21:09:47 +00:00
remusb
97b48cf988 rc: add support for Go 1.6 2018-03-14 22:58:20 +02:00
Nick Craig-Wood
86e5a35491 Implement Remote Control for rclone #2111 
This implements a remote control protocol activated with the --rc flag
and a new command `rclone rc` to use that interface.

Still to do
  * docs - need finishing
  * tests
 2018-03-14 22:58:20 +02:00
Nick Craig-Wood
8bb2854fe4 httplib: allow the flags to be prefixed when instantiating a server 2018-03-14 22:58:20 +02:00
Nick Craig-Wood
4e90ad04d5 serve restic: only accept v2 API requests for list 2018-03-11 17:35:01 +00:00
Alexander Neumann
fa003e89b6 serve restic: When listing return empty list instead of 'null' 2018-03-11 14:48:46 +00:00
Alexander Neumann
5114b11d6f serve restic: add http2 server on stdin/stdout 2018-03-11 14:48:46 +00:00
Alexander Neumann
f832433fa5 serve restic: Return empty list for non-existing dirs 2018-03-11 14:48:43 +00:00
Nick Craig-Wood
d073efdc6c serve restic: serves a remote in restic REST API format 2018-03-11 14:43:03 +00:00
Nick Craig-Wood
9e48748182 httplib: Note that authentication is a good idea for non localhost 2018-03-11 14:38:54 +00:00
Nick Craig-Wood
66c69fe620 mount: wait longer for consistency after rm in tests 2018-03-09 23:15:38 +00:00
Nick Craig-Wood
7713acf23d mount: skip failing test TestFileModTimeWithOpenWriters on Windows 2018-03-09 23:15:38 +00:00
Nick Craig-Wood
473a388f6d mount: disable failing test TestWriteFileDoubleClose on OSX 2018-03-09 23:15:37 +00:00
Nick Craig-Wood
09c14af6d1 cmd: Fix go routines at exit message to make it less confusing 2018-03-09 17:15:48 +00:00
Jakub Tasiemski
acae10cd6f lsjson: add --encrypted to show encrypted name #1765 2018-03-09 08:44:02 +00:00
Remus Bunduc
70f07fd3ac
fs: add ChangeNotify and backend support for it (#2094) 
* fs: rename DirChangeNotify to ChangeNotify

* cache: switch to ChangeNotify

* ChangeNotify: keep order of notifications
 2018-03-08 22:03:34 +02:00
Nick Craig-Wood
fc32fee4ad mount, cmount: add --attr-timeout to control attribute caching in kernel 
This flag allows the attribute caching in the kernel to be controlled.
The default is 0s - no caching - which is recommended for filesystems
which can change outside the control of the kernel.

Previously this was at the default meaning it was 60s for mount and 1s
for cmount.  This showed strange effects when files changed on the
remote not via the kernel.  For instance Caddy would serve corrupted
files for a while when serving from an rclone mount when a file
changed on the remote.
 2018-03-04 11:20:22 +00:00
ishuah
ebfeec9fb4 mount: run rclone mount in the background - fixes #723 2018-03-04 14:06:07 +03:00
Nick Craig-Wood
54deb01f00 vfs: Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC 
Before this change Open("name", os.O_RDONLY|os.O_TRUNC) would have
truncated the file.  This is what Linux does, but is counterintuitive.
POSIX states this is undefined, so return an error in this case
instead.  This preserves the invariant O_RDONLY => file is not
changed.
 2018-02-26 17:04:27 +00:00
Fabian Möller
00adf40f9f
cryptdecode: use Cipher instead of NewFs (#2087) 
* crypt: extract NewCipher out of NewFs
* cryptdecode: make use of crypt.NewCipher

Fixes #2075
 2018-02-25 12:57:14 +01:00
Nick Craig-Wood
334bf49d30 httplib: add Close() method to shut the server down and use it in tests 2018-02-19 15:45:24 +00:00
Nick Craig-Wood
d8f78a7266 serve http/webdav: update docs on SSL 2018-02-19 14:08:17 +00:00
Nick Craig-Wood
358c1fbac9 serve http/webdav: support SSL/TLS 2018-02-16 18:28:10 +00:00
Nick Craig-Wood
cc9d7156e4 serve http/webdav: add --user --pass authentication #1802 2018-02-16 18:28:10 +00:00
Nick Craig-Wood
221a8a9c5d serve http/webdav: add --htpasswd option for authentication #1802 2018-02-16 18:28:10 +00:00
Nick Craig-Wood
5530662ccc serve http/webdav: factor common http server creation to httplib 2018-02-16 17:48:20 +00:00
Nick Craig-Wood
0f787e43b0 mount: link the nssm service manager for mount under Windows 2018-02-12 11:29:32 +00:00
Nick Craig-Wood
3a7bb7b2df mount: update docs showing --vfs-cache-mode to work around limitations 2018-02-12 11:29:32 +00:00
Jakub Tasiemski
1018e9bb27 cmd: rewrite touch tests #1934 2018-02-02 13:46:56 +00:00
Fabian Möller
3c7a755631 lsjson: explain the Path value in the docs 2018-01-31 20:06:01 +00:00
Nick Craig-Wood
c277a4096c mount: don't set modtime twice #2021 2018-01-29 20:49:13 +00:00
Fabian Möller
1f5e23aedb scripts: make absolute paths consistent 
Change absolute binary paths in scripts to /usr/bin/env or make them
relative.
This allows the scripts to be used on linux distributions
like NixOS, where binaries are not located in /usr/ or /bin/.
 2018-01-26 09:39:05 +00:00
Nick Craig-Wood
fa500e6d21 lib/atexit: factor from cmd so it can be used by backend/cache #1946 2018-01-25 10:33:00 +00:00
Jody Frankowski
5e73acd40a Clean up mount.go and vfs/help.go docs 
* Title cleanups
* Typos
* `rclone mount vs rclone sync/copy` update with `File Caching`
 2018-01-20 18:14:20 +00:00
Nick Craig-Wood
a80d8a21dc vfs: add flags parameter to Dir.Create 2018-01-19 13:18:40 +00:00
Nick Craig-Wood
517bdc719b vfs: make specialized file Open functions private 2018-01-19 11:46:01 +00:00
Nick Craig-Wood
5ad226ab54 fs: Add dir option to fs.Purge #1891 
Purge optional interface needs fixing too.
 2018-01-19 11:45:50 +00:00
Nick Craig-Wood
97c414f025 config/hash: rename more symbols after factoring into own package 2018-01-18 20:27:52 +00:00
Nick Craig-Wood
71722b5b95 config: factor Obscure and Reveal into its own package 2018-01-18 20:19:55 +00:00
Nick Craig-Wood
821be5ebed ncdu: add link to asciinema demo of it in action 2018-01-18 14:22:43 +00:00
Ernest Borowski
5cce74d630 flags: remove --no-traverse flag because it is obsolete - fixes #1813 
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
 2018-01-18 11:00:25 +00:00
Nick Craig-Wood
11da2a6c9b Break the fs package up into smaller parts. 
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
 2018-01-15 17:51:14 +00:00
Nick Craig-Wood
60afda007b Move dircache, oauthutil, rest and pacer modules into lib 2018-01-12 17:07:38 +00:00
Nick Craig-Wood
b8b620f5c2 Move all backends into backend directory 2018-01-12 17:07:38 +00:00
ishuah
0a7731cf0d cryptdecode: added option to return encrypted file names. Fixes #1923 2018-01-11 19:22:40 +03:00
Nick Craig-Wood
0eba37d8f3 lsf: add --files-only and --dirs-only flags 2018-01-06 18:04:24 +00:00
Nick Craig-Wood
c74c3b37da lsf: add option to print hashes 2018-01-06 17:53:37 +00:00
Nick Craig-Wood
ed20fa5ee7 ls* commands: update docs and add defaults into options for lsf 2018-01-06 17:00:20 +00:00
Nick Craig-Wood
54a9fdf421 ls2: remove in favour of lsf 2018-01-06 14:41:36 +00:00
Jakub Tasiemski
0d041602cf cmd: new command lsf 2018-01-06 14:39:31 +00:00
Nick Craig-Wood
65618afd8c serve/http: fix serving files with : in - fixes #1939 2018-01-05 17:25:05 +00:00
Nick Craig-Wood
2a01fa9fa0 moveto,copyto: clarify error message if source doesn't exist - fixes #1022 2017-12-15 11:37:31 +00:00
lewapm
9c242edc10 rmdirs: add --leave-root flag - fixes #1874 2017-12-13 10:23:54 +00:00
Nick Craig-Wood
2cf808c825 ncdu: fix crashes on empty directories - fixes #1910 
Up arrow or right arrow in an empty directory would crash ncdu
 2017-12-12 13:54:15 +00:00
remusb
b48b537325
cache: plex integration, refactor chunk storage and worker retries (#1899) 2017-12-09 23:54:26 +02:00
Laurence
e767c9ac9f Fix typo in dbhashsum description 2017-12-03 10:24:33 +00:00
Nick Craig-Wood
6d3df6f172 cmount: make tests more reliable on Windows 2017-11-28 20:39:24 +00:00
ishuah
aab8051f50 move: add --delete-empty-src-dirs flag - fixes #1854 2017-11-28 11:38:19 +03:00
Nick Craig-Wood
1248beb0b2 cachestats: Fix nil pointer if not a cache remote - fixes #1855 
Also don't retry or show stats
 2017-11-24 10:22:23 +00:00
Nick Craig-Wood
dc1c679c65 mount: support truncate properly 2017-11-20 19:42:35 +00:00
Nick Craig-Wood
af50f31f7d mounttest: wait for Release after every Read to stop using in use files under Windows 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
2b666187a6 cmount: disable tests on windows + race detector 
These either hang or produce incorrect results for reasons I haven't
worked out yet.
 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
00b46a8b96 mounttest: wait for files to disappear from the directory listing 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
b21f227bd3 mounttest: fix crash when FUSE not present 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
e98e550021 mounttest: wait for all background Close/Release after writing a file 
The filesystem does a certain amount of things asynchronously waiting
for the file to be released after writing it means everything should
be in a consistent state.
 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
b4083b4371 vfs: rename Fsync to Sync and implement Sync on Node and Handle 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
eb3415db50 cmount: enable more tests for Windows 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
9fbd8a6419 mounttest: fixes for running under Windows 
* don't mount and unmount between cache runs - WinFSP doesn't suport it
  * use OS paths for opening things
 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
61b6159a05 mount, cmount: add O_CREATE to Open calls since fuse doesn't seem to supply it 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
c560017934 vfs: add Path method to Node and use it to stop reading nil DirEntry 
All DirEntry calls now have been checked for nil or converted to use Path.
 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
7c3584f4e6 mountlib: wait for mountpoint to disappear under Windows 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
981cfb1bec mounttest: retry directory listings to account for slow updates on Windows 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
dec21ccf63 vfs, cmount: make truncate work properly in the presence or otherwise of open files 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
94adf4f43b cmount: translate FUSE open flags into OS flags 
On Windows the fuse.O_* flags do not have the same values as the
os.O_* flags so translate between the two representations.  They are
mostly the same which is why this hasn't caused a problem before.
 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
f5f8c0c438 cmount: make Truncate call the correct Handle or Node method 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
60cdcf784c cmount: use -o atomic_o_trunc to make sure O_TRUNC is supplied to Open() 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
57a5c67729 mounttest: run the tests for all 4 VFS cache modes 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
d7908c06c9 mountlib: ensure we don't open files with read and write intent 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
8951875c21 vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
05a1e1532b vfs,mount,cmount,serve: Add documentation for vfs caching modes 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
3fb1e96988 vfs: factor Open logic from Dir.Create into vfs.OpenFile 2017-11-20 12:00:32 +00:00
Jakub Tasiemski
3cf7c61aa0 Add touch command - fixes #1594 2017-11-20 11:16:05 +00:00
Fabian Möller
d8b3bf014d mount: use sdnotify to signal systemd the mount is ready 
When the NOTIFY_SOCKET environment variable is set notify systemd after
the mount is ready.
 2017-11-20 11:03:10 +00:00
ishuah
31d2fb4e11 mount: Fix mount breaking on Windows - fixes #1827 2017-11-16 15:20:53 +03:00
Remus Bunduc
016abf825e cache: first version 2017-11-15 15:23:21 +00:00
ishuah
aec2265be0 rclone: implement exit codes - #1136 2017-11-15 17:48:37 +03:00
Substantiel
2423fa40e2 config: add password sub command for setting obscured passwords 2017-11-15 14:44:45 +00:00
Ernest Borowski
9fbff7bcab mountlib: check if directory is not empty before mounting - fixes #1386 
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
 2017-11-14 21:24:31 +00:00
Nick Craig-Wood
68d0b5adbb serve webdav: this implements a webdav server for any rclone remote. 2017-11-04 10:24:11 +00:00
Nick Craig-Wood
16e16bc220 serve http: use vfs to cache the directories and support Range header 2017-11-04 10:24:11 +00:00
Nick Craig-Wood
2083ac6e2a vfs: add ECLOSED and tidy errors 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
22ee839d05 cmount,vfs: unify Read and Write handles and File and Dir where possible 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
5634659ea3 mount,vfs: unify Read and Write handles in preparation for ReadWrite handles 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
e18122e88b vfs: add tests and subsequent fixes 
* Tests for VFS layer
  * Small fixes found during testing
  * Fix Close, Flush and Release behaviour for ReadFileHandle and WriteFileHandle
  * Fix nil object bugs on File
 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
07ec8073fe mount: remove unused DirEntry struct 2017-11-03 13:00:00 +00:00
Nick Craig-Wood
8184ec4b70 vfs: add EPERM to errors 2017-11-03 13:00:00 +00:00
Nick Craig-Wood
190367d917 vfs: factor duplicated Open code into vfs from mount/cmount 2017-11-03 13:00:00 +00:00
Nick Craig-Wood
7e065440fb vfs: rename Lookup to Stat to be more in keeping with os 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
e8883e9fdb vfs: factor flags into vfsflags and remove global variables 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
1a8f824bad vfs: use os package errors where possible 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
c1aaff220d Factor new vfs module out of cmd/mountlib 
This is an OS style file system abstraction with directory caching
used in mount, cmount, serve webdav and serve http.
 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
6da6b2556b mountlib: make directory entries be returned in sorted order 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
ca19fd2d7e mountlib: Make read/write file handles support more standard interfaces 
Including Read, ReadAt, Seek, Close for read handles and Write,
WriteAt, Close for read handles.
 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
2fac74b517 mountlib: store only Node in *Dir removing DirEntry struct 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
8b6daaa877 mountlib: add DirEntry() to Node interface 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
3af9d63261 mountlib: add Remove and RemoveAll methods to Node 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
c6cd2a5280 mountlib: add parent and entry to Dir 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
0bb84efe75 mountlib: Rename Remove to RemoveName 2017-11-03 12:59:59 +00:00
Nick Craig-Wood
3ec15ac2bd mountlib: make sure Node is always set in DirEntry 
This simplifies the code and makes using the DirEntry.Node usable when
using ReadDir.
 2017-11-03 12:59:58 +00:00
Nick Craig-Wood
750690503e mountlib: make Node satisfy os.FileInfo interface 2017-11-03 12:59:58 +00:00
Nick Craig-Wood
54950d3423 mountlib: make more useful as a general purpose file system adaptor 2017-11-03 12:59:58 +00:00
Nick Craig-Wood
96665c16cb serve http: make it compile on go1.6 and go1.7 2017-10-26 21:52:29 +01:00
Nick Craig-Wood
cd146415d1 serve http: error if Range supplied (not supported yet) 
Also add Server header
 2017-10-24 23:18:36 +01:00
Nick Craig-Wood
2740c965c0 serve http: Fix timeouts 2017-10-24 23:07:46 +01:00
Nick Craig-Wood
6669165b6b serve http command to serve a remote over HTTP 
This implements a basic webserver to serve an rclone remote over HTTP.

It also sets up the framework for adding more types of server later.
 2017-10-24 13:25:49 +01:00
Dan Dascalescu
9417194751 Fix dedupe description typo 2017-10-16 21:51:31 +01:00
Nick Craig-Wood
edfab09eb9 config: add sub commands for full config file management 
Previously config sub commands were manually parsed rather than using
cobra.

Make config command have the following sub commands:

 * create    Create a new remote with name, type and options.
 * delete    Delete an existing remote <name>.
 * dump      Dump the config file as JSON.
 * edit      Enter an interactive configuration session.
 * file      Show path of configuration file in use.
 * providers List in JSON format all the providers and options.
 * show      Print (decrypted) config file, or the config for a single remote.
 * update    Update options in an existing remote.

The following changes were made to existing commands

 * listproviders was renamed to providers
 * listoptions was removed in favour of providing the output in providers
 * jsonconfig was renamed to create
 * an optional parameter was added to the show command
 2017-10-14 11:50:41 +01:00
thierry
0575623dff Add config listproviders, listoptions, jsonconfig for automated config 
Addition of a method listing the providers, a method listing the
options of a provider and method of manual configuration.
 2017-10-13 17:17:36 +01:00
Nick Craig-Wood
7f3dc9b5c4 Implement WebDAV remote #580 
This has special knowledge of Owncloud and Nextcloud to enable more
functionality such as mod times.
 2017-10-09 16:19:37 +01:00
Nick Craig-Wood
9e9297838f Implement pcloud remote - #418 2017-10-01 11:37:35 +01:00
Nick Craig-Wood
63d0734c71 tree: remove workaround for tree library bug now it is fixed 2017-09-30 15:51:14 +01:00
ishuah
7195e44dce crypt: added cryptdecode command - #1129 2017-09-24 20:02:59 +01:00
Stefan
85877f3adc config: add show/file subcommands which print the config/its path (fixes #1086) 2017-09-19 17:59:19 +02:00
Stefan Breunig
12405f9f41 fuse: re-use rcat to support uploads for all remotes (fixes #1672) 2017-09-16 22:49:08 +02:00
Stefan Breunig
80b1f2a494 rcat: configurable small files cutoff and implement proper upload verification 2017-09-11 08:26:53 +02:00
bpicode
f1f7e0e6f9 support for zsh auto-completion - #983 2017-08-31 16:21:28 +01:00
Stefan Breunig
0158ab6926 info: add check to stream files with unknown size 2017-08-22 08:00:10 +02:00
Nick Craig-Wood
bfe812ea6b dedupe: implement merging of duplicate directories - fixes #1243 2017-08-07 15:36:41 +01:00
Nick Craig-Wood
92d2e1f8d7 azureblob: rework and complete #801 
* Fixup bitrot (rclone and Azure library)
  * Implement Copy
  * Add modtime to metadata under mtime key as RFC3339Nano
  * Make multipart upload work
  * Make it pass the integration tests
  * Fix uploading of zero length blobs
  * Rename to azureblob as it seems likely we will do azurefile
  * Add docs
 2017-08-04 22:56:16 +01:00
wuyu
ec5b72f8d5 Add new QingStor remote 
Add new package qingstor to support QingStor API.

Add new unit test for its and tested through; But I commented
on some tests case because of some of the features of QingStor.

Add new docs for it.
 2017-08-04 17:25:47 +01:00
Nick Craig-Wood
7d34caac83 cmd: add os and go version to rclone version output 2017-08-04 14:25:55 +01:00
Stefan Breunig
28a18303f3 implement rcat – fixes #230, fixes #1001 2017-08-03 21:42:35 +02:00
Nick Craig-Wood
21aca68680 tree: fix when running under Windows 2017-08-01 14:46:21 +01:00
Andy Pilate
ef328c5497 Fixes typo in command dedupe definition 2017-07-27 21:17:57 +01:00
Nick Craig-Wood
f3060caf04 Implement tree command - fixes #1528 2017-07-26 23:06:48 +01:00
Nick Craig-Wood
ebc8361933 mount: Add notes on Windows limitations from Bill Zissimopoulos 2017-07-26 21:08:24 +01:00
Nick Craig-Wood
cd4895690a fstest: Factor test initialisation into Initialise() 2017-07-26 16:38:33 +01:00
Nick Craig-Wood
de54fd4c64 mount: add docs for windows install 2017-07-23 13:05:02 +01:00
Nick Craig-Wood
7b81f12dad box: add docs 
* reorder remotes so they are in alphabetical order by full name everywhere
  * update CONTRIBUTING doc
 2017-07-23 11:32:34 +01:00
Nick Craig-Wood
c223464cd0 mount: fix panic on renames - fixes #1533 
Make sure d.items is not nil and improve locking
 2017-07-22 11:00:51 +01:00
Yaroslav Halchenko
3ddcbce989 DOC: any empty directoryies -> empty directories (fixes #1546) 2017-07-22 10:24:41 +02:00
Nick Craig-Wood
e86f62c3e8 Add rclone info internal command for testing out limits of remotes 2017-07-03 15:05:27 +01:00
Nick Craig-Wood
e7d04fc103 Create fs.Directory interface and use it everywhere 2017-06-30 14:26:59 +01:00
Nick Craig-Wood
e2d7d413ef fs: rename BasicInfo to DirEntry 2017-06-30 14:26:58 +01:00
Nick Craig-Wood
aa20486485 Add --stats-log-level so can see --stats without -v - fixes #1180 
The most common use for this flag is likely to be showing the stats
without using -v by using `--stats-log-level NOTICE`.
 2017-06-26 22:50:37 +01:00
sainaen
1817d8f631 crypt: Fix typo in cryptcheck's short description 2017-06-26 12:35:20 +01:00
Nick Craig-Wood
b360527931 mount: fix hang on errored upload 
In certain circumstances if an upload failed then the mount could hang
indefinitely. This was fixed by closing the read pipe after the Put
completed.  This will cause the write side to return a pipe closed
error fixing the hang.

Fixes #1498
 2017-06-26 12:08:51 +01:00
Nick Craig-Wood
b22c4c4307 http: fix, tidy and rework ready for release 
* Fix remaining problems
  * Refactor to make testing easier and add a test suite
  * Make path parsing more robust.
  * Add single file operations
  * Add MimeType reading for objects
  * Add documentation
  * Note go1.7+ is required to build
 2017-06-20 21:27:14 +01:00
Nick Craig-Wood
dcce65b2b3 mount/cmount: factor duplicated code into mountlib 2017-06-19 14:36:51 +01:00
Nick Craig-Wood
97364fd0b6 ncdu: disable on plan9 and solaris as termbox isn't supported there 2017-06-15 20:10:54 +01:00
Nick Craig-Wood
c34f11a92f rclone ncdu for exploring a remote with a text based user interface. 2017-06-15 17:44:17 +01:00
Nick Craig-Wood
7be49249d3 Add lsjson command - fixes #1063 2017-06-14 16:49:40 +01:00
Nick Craig-Wood
7e20e16cff core: Implement Walk directory listing and use in place of Lister 
This is in preparation for removing the Lister code and replacing the
fundamental operation in the Fs with listing a single directory.
 2017-06-14 16:49:40 +01:00
Nick Craig-Wood
68333d34a1 dropbox: make setting mod time on existing files work properly 
This is a fix left over from the v2 conversion.  Dropbox ignores the
client modification on an incoming file if it was identical to the
existing file.  This change deletes the existing file first before
re-uploading the new one.
 2017-06-13 13:58:39 +01:00
Nick Craig-Wood
740b3f6ae2 Fix problems found with ineffassign 2017-06-13 11:52:36 +01:00
Nick Craig-Wood
28fcc53e45 mount test: retry umount as it fails occasionally 
This is because of the background releasing of files which happens
after all the files are closed.
 2017-06-13 10:52:10 +01:00
Nick Craig-Wood
d30cc1e119 Factor RemoteSplit into fs 2017-06-07 12:27:33 +01:00
Stefan Breunig
9782c264e9 hand dirCacheTime through again 2017-06-01 09:02:22 +01:00
Nick Craig-Wood
52e96bc0e2 dropbox: add missing dbhashsum command 
This was missed from 6381959850
 2017-05-30 19:26:06 +01:00
Nick Craig-Wood
6381959850 dropbox: support Dropbox content hashing scheme - fixes #1302 
* add support to hashing module
  * add dbhashsum to list the hashes
  * add support to dropbox module

This means objects up and downloaded to/from Dropbox will have their
hashes checked.

Note after this change local objects are calculating MD5, SHA1 and
DBHASH which is excessive and needs to be fixed.
 2017-05-29 12:04:44 +01:00
Stefan Breunig
a2e3af0523 poll for Google Drive changes when mounted 2017-05-28 17:54:52 +01:00
Sjur Fredriksen
ade61fa756 Updated FTP to follow SFTP standards, updated documentation 2017-05-25 09:30:15 +01:00
Nick Craig-Wood
8f54dc06a2 Use build tags to control when and where cmount is built 2017-05-19 17:08:04 +01:00
Nick Craig-Wood
2cae017738 mountlib: fix race condition in cache clear 2017-05-19 15:47:52 +01:00
Nick Craig-Wood
ecedcd0e7f cmount: stop failing tests on Windows 2017-05-15 17:40:44 +01:00
Nick Craig-Wood
3dff91d691 mount: add missing build constraint to fix Windows build 2017-05-15 17:40:15 +01:00
Nick Craig-Wood
b259f8b752 cmount, mount, mountlib: make --read-only reject modify operations 
Normally mount/cmount use `-o ro` to get the kernel to mark the fs as
read only.  However this is ignored by WinFsp, so in addition if
`--read-only` is in effect then return EROFS ("Read only File System")
from all methods which attempt to modify something.
 2017-05-15 16:56:47 +01:00
Nick Craig-Wood
8be8a8e41b mountlib: on read only open of file, make open pending until first read 
This fixes a problem with Windows which seems fond of opening files
just to read their attributes and closing them again.
 2017-05-15 16:56:47 +01:00
Nick Craig-Wood
f9500729b7 mountlib: fix cross platform tests 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
204a19e67f cmount: Wait for mountpoint to appear on Windows before declaring mounted 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
e6ffe3464c cmount: check for filesystem blowing up before Init is called 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
0384364c3e cmount: pass --FileSystemName under windows 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
763facfd78 cmount: implement --fuse-flag to pass commands to fuse library directly 
Useful for `--fuse-flag -h` to see exactly which options the library supports.
 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
bc88f1dafa cmount: fix openFile leak 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
0c055a1215 cmount: Statfs: reduce max size of volume for Windows 2017-05-15 16:56:46 +01:00
Nick Craig-Wood
938d7951ab cmount: allow extra options to pass to fuse with -o 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
b4466bd9b1 Add -o uid=-1 -o gid=-1 for Windows/WinFsp 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
31f76aa464 cmount: implement no-ops for Fsync, Chmod, Chown, Access, Fsyncdir and stop using fuse.FileSystemBase 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
c887c164dc cmount: add function tracing 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
115ac00222 mount, mountlib: move function tracing into mount 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
abda616f84 mountlib: make Nodes also be fmt.Stringer so they debug nicely 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
9c3048580a cmount: fix code quality warnings 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
c1d5faa32a mountlib: fix code quality warnings 2017-05-15 16:56:45 +01:00
Nick Craig-Wood
d127d8686a mountlib: pass options in fsys not as args 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
bc9856b570 Forward port 930ff266f2 to cmount branch 
compare checksums on upload/download via FUSE
 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
855071cc19 cmount: name the command mount under windows and cmount under linux 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
b179540e80 cmount: fix Getattr to work on directories 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
6a8e4690d3 mountlib: windows fixes for drive letter and timing 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
917ea6ac57 mountlib: make tests work under all platforms 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
7b47a1e842 cmount: set the correct values for uid, gid under Windows 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
bcd87009e2 Fix docs typo 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
caf85737c3 cmount: fix Windows compile (thanks Bill Zissimopoulos) 2017-05-15 16:56:44 +01:00
Nick Craig-Wood
e1516e0159 Forward port 58a82cd578 into cmount branch 
allow the fuse directory cached to be cleaned manually
 2017-05-15 16:56:43 +01:00
Nick Craig-Wood
ee1111e4c9 cmount: a new mount option based on cgofuse. 
This with the aid of WinFSP should work on Windows.

Unfinished bits
  * 1 test doesn't pass
  * docs
  * build
 2017-05-15 16:56:43 +01:00
Nick Craig-Wood
268fe0004c mount: factor filesystem code into mountlib and mounttest 2017-05-12 21:24:24 +01:00
Stefan Breunig
58a82cd578 allow the fuse directory cached to be cleaned manually (fixes #803) 2017-05-07 12:08:59 +01:00
Stefan Breunig
930ff266f2 compare checksums on upload/download via FUSE 2017-05-02 22:27:38 +01:00
Stefan Breunig
2fd86c93fc allow modTime to be changed even before all writers are closed (fixes #1197 -- again) 2017-03-31 01:28:08 +02:00
Ihor Dvoretskyi
a54692d165 OneDrive vs One Drive 
It's better to call this service as it's officially named.
 2017-03-29 18:11:33 +01:00
marvwatson
4810aa65a4 Update references from HTTP to HTTPS where possible 2017-03-29 05:38:34 -07:00
Stefan Breunig
4dc030d081 implement ModTime via FUSE for remotes that support it (fixes #1197) 2017-03-24 09:23:04 +01:00
Jérôme Vizcaino
788b6ce821 mount: umount dir when program ends with SIGINT (Ctrl+C) or SIGTERM 2017-03-18 21:24:05 +00:00
Nick Craig-Wood
1cc58e4e09 mount: fix logging for unimplemented file open modes #1195 2017-03-02 22:07:01 +00:00
Nick Craig-Wood
30e97ad9ec Fix parsing of remotes in moveto and copyto - fixes #1079 2017-02-22 22:09:33 +00:00
Nick Craig-Wood
07dc76eff0 Remove unused test file 2017-02-22 20:58:24 +00:00