Nick Craig-Wood
11da2a6c9b
Break the fs package up into smaller parts.
...
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.
The new code layout is documented in CONTRIBUTING.
2018-01-15 17:51:14 +00:00
Nick Craig-Wood
60afda007b
Move dircache, oauthutil, rest and pacer modules into lib
2018-01-12 17:07:38 +00:00
Nick Craig-Wood
b8b620f5c2
Move all backends into backend directory
2018-01-12 17:07:38 +00:00
ishuah
0a7731cf0d
cryptdecode: added option to return encrypted file names. Fixes #1923
2018-01-11 19:22:40 +03:00
Nick Craig-Wood
0eba37d8f3
lsf: add --files-only and --dirs-only flags
2018-01-06 18:04:24 +00:00
Nick Craig-Wood
c74c3b37da
lsf: add option to print hashes
2018-01-06 17:53:37 +00:00
Nick Craig-Wood
ed20fa5ee7
ls* commands: update docs and add defaults into options for lsf
2018-01-06 17:00:20 +00:00
Nick Craig-Wood
54a9fdf421
ls2: remove in favour of lsf
2018-01-06 14:41:36 +00:00
Jakub Tasiemski
0d041602cf
cmd: new command lsf
2018-01-06 14:39:31 +00:00
Nick Craig-Wood
65618afd8c
serve/http: fix serving files with : in - fixes #1939
2018-01-05 17:25:05 +00:00
Nick Craig-Wood
2a01fa9fa0
moveto,copyto: clarify error message if source doesn't exist - fixes #1022
2017-12-15 11:37:31 +00:00
lewapm
9c242edc10
rmdirs: add --leave-root flag - fixes #1874
2017-12-13 10:23:54 +00:00
Nick Craig-Wood
2cf808c825
ncdu: fix crashes on empty directories - fixes #1910
...
Up arrow or right arrow in an empty directory would crash ncdu
2017-12-12 13:54:15 +00:00
remusb
b48b537325
cache: plex integration, refactor chunk storage and worker retries ( #1899 )
2017-12-09 23:54:26 +02:00
Laurence
e767c9ac9f
Fix typo in dbhashsum description
2017-12-03 10:24:33 +00:00
Nick Craig-Wood
6d3df6f172
cmount: make tests more reliable on Windows
2017-11-28 20:39:24 +00:00
ishuah
aab8051f50
move: add --delete-empty-src-dirs flag - fixes #1854
2017-11-28 11:38:19 +03:00
Nick Craig-Wood
1248beb0b2
cachestats: Fix nil pointer if not a cache remote - fixes #1855
...
Also don't retry or show stats
2017-11-24 10:22:23 +00:00
Nick Craig-Wood
dc1c679c65
mount: support truncate properly
2017-11-20 19:42:35 +00:00
Nick Craig-Wood
af50f31f7d
mounttest: wait for Release after every Read to stop using in use files under Windows
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
2b666187a6
cmount: disable tests on windows + race detector
...
These either hang or produce incorrect results for reasons I haven't
worked out yet.
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
00b46a8b96
mounttest: wait for files to disappear from the directory listing
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
b21f227bd3
mounttest: fix crash when FUSE not present
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
e98e550021
mounttest: wait for all background Close/Release after writing a file
...
The filesystem does a certain amount of things asynchronously waiting
for the file to be released after writing it means everything should
be in a consistent state.
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
b4083b4371
vfs: rename Fsync to Sync and implement Sync on Node and Handle
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
eb3415db50
cmount: enable more tests for Windows
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
9fbd8a6419
mounttest: fixes for running under Windows
...
* don't mount and unmount between cache runs - WinFSP doesn't suport it
* use OS paths for opening things
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
61b6159a05
mount, cmount: add O_CREATE to Open calls since fuse doesn't seem to supply it
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
c560017934
vfs: add Path method to Node and use it to stop reading nil DirEntry
...
All DirEntry calls now have been checked for nil or converted to use Path.
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
7c3584f4e6
mountlib: wait for mountpoint to disappear under Windows
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
981cfb1bec
mounttest: retry directory listings to account for slow updates on Windows
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
dec21ccf63
vfs, cmount: make truncate work properly in the presence or otherwise of open files
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
94adf4f43b
cmount: translate FUSE open flags into OS flags
...
On Windows the fuse.O_* flags do not have the same values as the
os.O_* flags so translate between the two representations. They are
mostly the same which is why this hasn't caused a problem before.
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
f5f8c0c438
cmount: make Truncate call the correct Handle or Node method
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
60cdcf784c
cmount: use -o atomic_o_trunc to make sure O_TRUNC is supplied to Open()
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
57a5c67729
mounttest: run the tests for all 4 VFS cache modes
2017-11-20 12:46:21 +00:00
Nick Craig-Wood
d7908c06c9
mountlib: ensure we don't open files with read and write intent
2017-11-20 12:46:21 +00:00
Nick Craig-Wood
8951875c21
vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables
2017-11-20 12:46:21 +00:00
Nick Craig-Wood
05a1e1532b
vfs,mount,cmount,serve: Add documentation for vfs caching modes
2017-11-20 12:46:21 +00:00
Nick Craig-Wood
3fb1e96988
vfs: factor Open logic from Dir.Create into vfs.OpenFile
2017-11-20 12:00:32 +00:00
Jakub Tasiemski
3cf7c61aa0
Add touch command - fixes #1594
2017-11-20 11:16:05 +00:00
Fabian Möller
d8b3bf014d
mount: use sdnotify to signal systemd the mount is ready
...
When the NOTIFY_SOCKET environment variable is set notify systemd after
the mount is ready.
2017-11-20 11:03:10 +00:00
ishuah
31d2fb4e11
mount: Fix mount breaking on Windows - fixes #1827
2017-11-16 15:20:53 +03:00
Remus Bunduc
016abf825e
cache: first version
2017-11-15 15:23:21 +00:00
ishuah
aec2265be0
rclone: implement exit codes - #1136
2017-11-15 17:48:37 +03:00
Substantiel
2423fa40e2
config: add password sub command for setting obscured passwords
2017-11-15 14:44:45 +00:00
Ernest Borowski
9fbff7bcab
mountlib: check if directory is not empty before mounting - fixes #1386
...
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
2017-11-14 21:24:31 +00:00
Nick Craig-Wood
68d0b5adbb
serve webdav: this implements a webdav server for any rclone remote.
2017-11-04 10:24:11 +00:00
Nick Craig-Wood
16e16bc220
serve http: use vfs to cache the directories and support Range header
2017-11-04 10:24:11 +00:00
Nick Craig-Wood
2083ac6e2a
vfs: add ECLOSED and tidy errors
2017-11-04 10:24:10 +00:00
Nick Craig-Wood
22ee839d05
cmount,vfs: unify Read and Write handles and File and Dir where possible
2017-11-04 10:24:10 +00:00
Nick Craig-Wood
5634659ea3
mount,vfs: unify Read and Write handles in preparation for ReadWrite handles
2017-11-04 10:24:10 +00:00
Nick Craig-Wood
e18122e88b
vfs: add tests and subsequent fixes
...
* Tests for VFS layer
* Small fixes found during testing
* Fix Close, Flush and Release behaviour for ReadFileHandle and WriteFileHandle
* Fix nil object bugs on File
2017-11-04 10:24:10 +00:00
Nick Craig-Wood
07ec8073fe
mount: remove unused DirEntry struct
2017-11-03 13:00:00 +00:00
Nick Craig-Wood
8184ec4b70
vfs: add EPERM to errors
2017-11-03 13:00:00 +00:00
Nick Craig-Wood
190367d917
vfs: factor duplicated Open code into vfs from mount/cmount
2017-11-03 13:00:00 +00:00
Nick Craig-Wood
7e065440fb
vfs: rename Lookup to Stat to be more in keeping with os
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
e8883e9fdb
vfs: factor flags into vfsflags and remove global variables
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
1a8f824bad
vfs: use os package errors where possible
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
c1aaff220d
Factor new vfs module out of cmd/mountlib
...
This is an OS style file system abstraction with directory caching
used in mount, cmount, serve webdav and serve http.
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
6da6b2556b
mountlib: make directory entries be returned in sorted order
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
ca19fd2d7e
mountlib: Make read/write file handles support more standard interfaces
...
Including Read, ReadAt, Seek, Close for read handles and Write,
WriteAt, Close for read handles.
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
2fac74b517
mountlib: store only Node in *Dir removing DirEntry struct
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
8b6daaa877
mountlib: add DirEntry() to Node interface
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
3af9d63261
mountlib: add Remove and RemoveAll methods to Node
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
c6cd2a5280
mountlib: add parent and entry to Dir
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
0bb84efe75
mountlib: Rename Remove to RemoveName
2017-11-03 12:59:59 +00:00
Nick Craig-Wood
3ec15ac2bd
mountlib: make sure Node is always set in DirEntry
...
This simplifies the code and makes using the DirEntry.Node usable when
using ReadDir.
2017-11-03 12:59:58 +00:00
Nick Craig-Wood
750690503e
mountlib: make Node satisfy os.FileInfo interface
2017-11-03 12:59:58 +00:00
Nick Craig-Wood
54950d3423
mountlib: make more useful as a general purpose file system adaptor
2017-11-03 12:59:58 +00:00
Nick Craig-Wood
96665c16cb
serve http: make it compile on go1.6 and go1.7
2017-10-26 21:52:29 +01:00
Nick Craig-Wood
cd146415d1
serve http: error if Range supplied (not supported yet)
...
Also add Server header
2017-10-24 23:18:36 +01:00
Nick Craig-Wood
2740c965c0
serve http: Fix timeouts
2017-10-24 23:07:46 +01:00
Nick Craig-Wood
6669165b6b
serve http command to serve a remote over HTTP
...
This implements a basic webserver to serve an rclone remote over HTTP.
It also sets up the framework for adding more types of server later.
2017-10-24 13:25:49 +01:00
Dan Dascalescu
9417194751
Fix dedupe description typo
2017-10-16 21:51:31 +01:00
Nick Craig-Wood
edfab09eb9
config: add sub commands for full config file management
...
Previously config sub commands were manually parsed rather than using
cobra.
Make config command have the following sub commands:
* create Create a new remote with name, type and options.
* delete Delete an existing remote <name>.
* dump Dump the config file as JSON.
* edit Enter an interactive configuration session.
* file Show path of configuration file in use.
* providers List in JSON format all the providers and options.
* show Print (decrypted) config file, or the config for a single remote.
* update Update options in an existing remote.
The following changes were made to existing commands
* listproviders was renamed to providers
* listoptions was removed in favour of providing the output in providers
* jsonconfig was renamed to create
* an optional parameter was added to the show command
2017-10-14 11:50:41 +01:00
thierry
0575623dff
Add config listproviders, listoptions, jsonconfig for automated config
...
Addition of a method listing the providers, a method listing the
options of a provider and method of manual configuration.
2017-10-13 17:17:36 +01:00
Nick Craig-Wood
7f3dc9b5c4
Implement WebDAV remote #580
...
This has special knowledge of Owncloud and Nextcloud to enable more
functionality such as mod times.
2017-10-09 16:19:37 +01:00
Nick Craig-Wood
9e9297838f
Implement pcloud remote - #418
2017-10-01 11:37:35 +01:00
Nick Craig-Wood
63d0734c71
tree: remove workaround for tree library bug now it is fixed
2017-09-30 15:51:14 +01:00
ishuah
7195e44dce
crypt: added cryptdecode command - #1129
2017-09-24 20:02:59 +01:00
Stefan
85877f3adc
config: add show/file subcommands which print the config/its path ( fixes #1086 )
2017-09-19 17:59:19 +02:00
Stefan Breunig
12405f9f41
fuse: re-use rcat to support uploads for all remotes ( fixes #1672 )
2017-09-16 22:49:08 +02:00
Stefan Breunig
80b1f2a494
rcat: configurable small files cutoff and implement proper upload verification
2017-09-11 08:26:53 +02:00
bpicode
f1f7e0e6f9
support for zsh auto-completion - #983
2017-08-31 16:21:28 +01:00
Stefan Breunig
0158ab6926
info: add check to stream files with unknown size
2017-08-22 08:00:10 +02:00
Nick Craig-Wood
bfe812ea6b
dedupe: implement merging of duplicate directories - fixes #1243
2017-08-07 15:36:41 +01:00
Nick Craig-Wood
92d2e1f8d7
azureblob: rework and complete #801
...
* Fixup bitrot (rclone and Azure library)
* Implement Copy
* Add modtime to metadata under mtime key as RFC3339Nano
* Make multipart upload work
* Make it pass the integration tests
* Fix uploading of zero length blobs
* Rename to azureblob as it seems likely we will do azurefile
* Add docs
2017-08-04 22:56:16 +01:00
wuyu
ec5b72f8d5
Add new QingStor remote
...
Add new package qingstor to support QingStor API.
Add new unit test for its and tested through; But I commented
on some tests case because of some of the features of QingStor.
Add new docs for it.
2017-08-04 17:25:47 +01:00
Nick Craig-Wood
7d34caac83
cmd: add os and go version to rclone version output
2017-08-04 14:25:55 +01:00
Stefan Breunig
28a18303f3
implement rcat – fixes #230 , fixes #1001
2017-08-03 21:42:35 +02:00
Nick Craig-Wood
21aca68680
tree: fix when running under Windows
2017-08-01 14:46:21 +01:00
Andy Pilate
ef328c5497
Fixes typo in command dedupe definition
2017-07-27 21:17:57 +01:00
Nick Craig-Wood
f3060caf04
Implement tree command - fixes #1528
2017-07-26 23:06:48 +01:00
Nick Craig-Wood
ebc8361933
mount: Add notes on Windows limitations from Bill Zissimopoulos
2017-07-26 21:08:24 +01:00
Nick Craig-Wood
cd4895690a
fstest: Factor test initialisation into Initialise()
2017-07-26 16:38:33 +01:00
Nick Craig-Wood
de54fd4c64
mount: add docs for windows install
2017-07-23 13:05:02 +01:00
Nick Craig-Wood
7b81f12dad
box: add docs
...
* reorder remotes so they are in alphabetical order by full name everywhere
* update CONTRIBUTING doc
2017-07-23 11:32:34 +01:00
Nick Craig-Wood
c223464cd0
mount: fix panic on renames - fixes #1533
...
Make sure d.items is not nil and improve locking
2017-07-22 11:00:51 +01:00
Yaroslav Halchenko
3ddcbce989
DOC: any empty directoryies -> empty directories ( fixes #1546 )
2017-07-22 10:24:41 +02:00
Nick Craig-Wood
e86f62c3e8
Add rclone info internal command for testing out limits of remotes
2017-07-03 15:05:27 +01:00
Nick Craig-Wood
e7d04fc103
Create fs.Directory interface and use it everywhere
2017-06-30 14:26:59 +01:00
Nick Craig-Wood
e2d7d413ef
fs: rename BasicInfo to DirEntry
2017-06-30 14:26:58 +01:00
Nick Craig-Wood
aa20486485
Add --stats-log-level so can see --stats without -v - fixes #1180
...
The most common use for this flag is likely to be showing the stats
without using -v by using `--stats-log-level NOTICE`.
2017-06-26 22:50:37 +01:00
sainaen
1817d8f631
crypt: Fix typo in cryptcheck's short description
2017-06-26 12:35:20 +01:00
Nick Craig-Wood
b360527931
mount: fix hang on errored upload
...
In certain circumstances if an upload failed then the mount could hang
indefinitely. This was fixed by closing the read pipe after the Put
completed. This will cause the write side to return a pipe closed
error fixing the hang.
Fixes #1498
2017-06-26 12:08:51 +01:00
Nick Craig-Wood
b22c4c4307
http: fix, tidy and rework ready for release
...
* Fix remaining problems
* Refactor to make testing easier and add a test suite
* Make path parsing more robust.
* Add single file operations
* Add MimeType reading for objects
* Add documentation
* Note go1.7+ is required to build
2017-06-20 21:27:14 +01:00
Nick Craig-Wood
dcce65b2b3
mount/cmount: factor duplicated code into mountlib
2017-06-19 14:36:51 +01:00
Nick Craig-Wood
97364fd0b6
ncdu: disable on plan9 and solaris as termbox isn't supported there
2017-06-15 20:10:54 +01:00
Nick Craig-Wood
c34f11a92f
rclone ncdu for exploring a remote with a text based user interface.
2017-06-15 17:44:17 +01:00
Nick Craig-Wood
7be49249d3
Add lsjson command - fixes #1063
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
7e20e16cff
core: Implement Walk directory listing and use in place of Lister
...
This is in preparation for removing the Lister code and replacing the
fundamental operation in the Fs with listing a single directory.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
68333d34a1
dropbox: make setting mod time on existing files work properly
...
This is a fix left over from the v2 conversion. Dropbox ignores the
client modification on an incoming file if it was identical to the
existing file. This change deletes the existing file first before
re-uploading the new one.
2017-06-13 13:58:39 +01:00
Nick Craig-Wood
740b3f6ae2
Fix problems found with ineffassign
2017-06-13 11:52:36 +01:00
Nick Craig-Wood
28fcc53e45
mount test: retry umount as it fails occasionally
...
This is because of the background releasing of files which happens
after all the files are closed.
2017-06-13 10:52:10 +01:00
Nick Craig-Wood
d30cc1e119
Factor RemoteSplit into fs
2017-06-07 12:27:33 +01:00
Stefan Breunig
9782c264e9
hand dirCacheTime through again
2017-06-01 09:02:22 +01:00
Nick Craig-Wood
52e96bc0e2
dropbox: add missing dbhashsum command
...
This was missed from 6381959850
2017-05-30 19:26:06 +01:00
Nick Craig-Wood
6381959850
dropbox: support Dropbox content hashing scheme - fixes #1302
...
* add support to hashing module
* add dbhashsum to list the hashes
* add support to dropbox module
This means objects up and downloaded to/from Dropbox will have their
hashes checked.
Note after this change local objects are calculating MD5, SHA1 and
DBHASH which is excessive and needs to be fixed.
2017-05-29 12:04:44 +01:00
Stefan Breunig
a2e3af0523
poll for Google Drive changes when mounted
2017-05-28 17:54:52 +01:00
Sjur Fredriksen
ade61fa756
Updated FTP to follow SFTP standards, updated documentation
2017-05-25 09:30:15 +01:00
Nick Craig-Wood
8f54dc06a2
Use build tags to control when and where cmount is built
2017-05-19 17:08:04 +01:00
Nick Craig-Wood
2cae017738
mountlib: fix race condition in cache clear
2017-05-19 15:47:52 +01:00
Nick Craig-Wood
ecedcd0e7f
cmount: stop failing tests on Windows
2017-05-15 17:40:44 +01:00
Nick Craig-Wood
3dff91d691
mount: add missing build constraint to fix Windows build
2017-05-15 17:40:15 +01:00
Nick Craig-Wood
b259f8b752
cmount, mount, mountlib: make --read-only reject modify operations
...
Normally mount/cmount use `-o ro` to get the kernel to mark the fs as
read only. However this is ignored by WinFsp, so in addition if
`--read-only` is in effect then return EROFS ("Read only File System")
from all methods which attempt to modify something.
2017-05-15 16:56:47 +01:00
Nick Craig-Wood
8be8a8e41b
mountlib: on read only open of file, make open pending until first read
...
This fixes a problem with Windows which seems fond of opening files
just to read their attributes and closing them again.
2017-05-15 16:56:47 +01:00
Nick Craig-Wood
f9500729b7
mountlib: fix cross platform tests
2017-05-15 16:56:46 +01:00
Nick Craig-Wood
204a19e67f
cmount: Wait for mountpoint to appear on Windows before declaring mounted
2017-05-15 16:56:46 +01:00
Nick Craig-Wood
e6ffe3464c
cmount: check for filesystem blowing up before Init is called
2017-05-15 16:56:46 +01:00
Nick Craig-Wood
0384364c3e
cmount: pass --FileSystemName under windows
2017-05-15 16:56:46 +01:00
Nick Craig-Wood
763facfd78
cmount: implement --fuse-flag to pass commands to fuse library directly
...
Useful for `--fuse-flag -h` to see exactly which options the library supports.
2017-05-15 16:56:46 +01:00
Nick Craig-Wood
bc88f1dafa
cmount: fix openFile leak
2017-05-15 16:56:46 +01:00
Nick Craig-Wood
0c055a1215
cmount: Statfs: reduce max size of volume for Windows
2017-05-15 16:56:46 +01:00
Nick Craig-Wood
938d7951ab
cmount: allow extra options to pass to fuse with -o
2017-05-15 16:56:45 +01:00
Nick Craig-Wood
b4466bd9b1
Add -o uid=-1 -o gid=-1 for Windows/WinFsp
2017-05-15 16:56:45 +01:00
Nick Craig-Wood
31f76aa464
cmount: implement no-ops for Fsync, Chmod, Chown, Access, Fsyncdir and stop using fuse.FileSystemBase
2017-05-15 16:56:45 +01:00
Nick Craig-Wood
c887c164dc
cmount: add function tracing
2017-05-15 16:56:45 +01:00
Nick Craig-Wood
115ac00222
mount, mountlib: move function tracing into mount
2017-05-15 16:56:45 +01:00
Nick Craig-Wood
abda616f84
mountlib: make Nodes also be fmt.Stringer so they debug nicely
2017-05-15 16:56:45 +01:00
Nick Craig-Wood
9c3048580a
cmount: fix code quality warnings
2017-05-15 16:56:45 +01:00
Nick Craig-Wood
c1d5faa32a
mountlib: fix code quality warnings
2017-05-15 16:56:45 +01:00
Nick Craig-Wood
d127d8686a
mountlib: pass options in fsys not as args
2017-05-15 16:56:44 +01:00
Nick Craig-Wood
bc9856b570
Forward port 930ff266f2
to cmount branch
...
compare checksums on upload/download via FUSE
2017-05-15 16:56:44 +01:00
Nick Craig-Wood
855071cc19
cmount: name the command mount under windows and cmount under linux
2017-05-15 16:56:44 +01:00
Nick Craig-Wood
b179540e80
cmount: fix Getattr to work on directories
2017-05-15 16:56:44 +01:00
Nick Craig-Wood
6a8e4690d3
mountlib: windows fixes for drive letter and timing
2017-05-15 16:56:44 +01:00
Nick Craig-Wood
917ea6ac57
mountlib: make tests work under all platforms
2017-05-15 16:56:44 +01:00
Nick Craig-Wood
7b47a1e842
cmount: set the correct values for uid, gid under Windows
2017-05-15 16:56:44 +01:00
Nick Craig-Wood
bcd87009e2
Fix docs typo
2017-05-15 16:56:44 +01:00