extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-12-26 00:49:22 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,295 Commits 182 Branches 114 Tags 336 MiB
b49821956a
Commit Graph

379 Commits

Author SHA1 Message Date
Nick Craig-Wood
0366ea39c5 Reassign some logging levels 2017-02-11 17:56:05 +00:00
Nick Craig-Wood
80f53176d9 Rename log functions and factor into own file 2017-02-11 17:54:50 +00:00
Nick Craig-Wood
50e190ff54 cat: don't allocate buffers if not needed to reduce memory usage 2017-02-09 11:46:53 +00:00
Nick Craig-Wood
dd20a297d6 cat: Fix go routine leak 2017-02-09 11:25:36 +00:00
Nick Craig-Wood
c0ad29c06c Clarify logging and docs for --no-traverse incompatibilities - fixes #1059 2017-02-08 22:35:12 +00:00
Nick Craig-Wood
d091d4a8bb rclone cat: add --head, --tail, --offset, --count and --discard 
Fixes #819
 2017-02-08 08:09:41 +00:00
Nick Craig-Wood
48cdedc97b Re-implement sync routine to work a directory at a time 
Multiple directories (up to --checkers worth) are scanned at once.

This uses much less memory than the previous scheme - only the amount
of memory needed to hold an entire directory listing of objects.

For directory based remotes the speed is unchanged.

For bucket based remotes, instead of doing one API call to list the
whole bucket, it does multiple calls, one for each pseudo directory.
However these are done in parallel so in practice this seems to speed
up directory listings.

This replaces the existing sync method as it performs faster and uses
less memory.

The old sync method is available with the temporary --old-sync-method
flag.

Fixes #517
Fixes #439
Fixes #236
Fixes #1067
 2017-02-04 10:30:25 +00:00
Nick Craig-Wood
7c6cd3a9e1 Make --delete-after the default and refactor --delete-{before,during,after} parsing 2017-02-04 10:30:25 +00:00
Nick Craig-Wood
bcdd73369f Ignore --delete-before with --track-renames 2017-02-04 10:30:25 +00:00
Nick Craig-Wood
86bec20b56 sync: factor accumulating the rename checks 2017-02-04 10:30:25 +00:00
Nick Craig-Wood
c3b2b89473 Add ListDirSorted function to list a directory 
* fix error return of readFilesFn also
 2017-02-04 10:30:25 +00:00
Nick Craig-Wood
85f05c57d1 Clean empty directories between test runs 2017-02-04 10:30:25 +00:00
Nick Craig-Wood
726cb43be9 Complete SFTP remote #521 
* Add unit tests
  * Fix up remote so it passes tests
  * Add docs
 2017-02-04 10:29:46 +00:00
Nick Craig-Wood
288302c2cf Make fallback purge delete empty directories too. 
This was implemented to make the SFTP unit tests pass.
 2017-02-04 10:29:46 +00:00
Jack Schmidt
b9a8315696 Basic SFTP support, Issue #521 2017-02-04 10:29:18 +00:00
Nick Craig-Wood
9d331ce04b Implement --ignore-checksum flag 
Fixes #793 Fixes #863 Fixes #981
 2017-02-03 08:11:10 +00:00
Nick Craig-Wood
2192805360 rclone config: when choosing from a list, allow the value to be entered 2017-01-29 15:51:26 +00:00
Nick Craig-Wood
db0b93c0ad rclone config: allow rename and copy of remotes - fixes #641 2017-01-29 15:37:44 +00:00
Dario Giovannetti
9d36258923 Comply with XDG Base Directory specification 
Fixes #868
 2017-01-26 10:22:08 +00:00
Nick Craig-Wood
b6848a3edb Fix race in Lister.Finished which was causing the tests to be unreliable 2017-01-19 20:11:17 +00:00
Nick Craig-Wood
e2bf9eb8e9 Implement --suffix for use with --backup-dir only #98 
This also makes sure we remove files we are about to override in the
--backup-dir properly.
 2017-01-19 20:11:17 +00:00
Nick Craig-Wood
1fa258c2b4 Define a new Features() method for Fs 
Optional interfaces are becoming more important in rclone,
--track-renames and --backup-dir both rely on them.

Up to this point rclone has used interface upgrades to define optional
behaviour on Fs objects.  However when one Fs object wraps another it
is very difficult for this scheme to work accurately.  rclone has
relied on specific error messages being returned when the interface
isn't supported - this is unsatisfactory because it means you have to
call the interface to see whether it is supported.

This change enables accurate detection of optional interfaces by use
of a Features struct as returned by an obligatory Fs.Features()
method.  The Features struct contains flags and function pointers
which can be tested against nil to see whether they can be used.

As a result crypt and hubic can accurately reflect the capabilities of
the underlying Fs they are wrapping.
 2017-01-16 17:33:25 +00:00
ncw
3745c526f1 Implement --backup-dir - fixes #98 
The parameter of backup-dir specifies a remote that all deleted or
overwritten files will be copied to.
 2017-01-16 17:33:25 +00:00
Nick Craig-Wood
c123c702ab Fix fs.Overlapping and factor fs.SameConfig 2017-01-14 09:55:53 +00:00
ncw
4aae7bcca6 Factor server side move detection 2017-01-14 09:55:53 +00:00
Brandur
3b1e0b66bb Return error on not found from ListFn 
This changes `ListFn`'s implementation so that if it encounters a not
found error, instead of sending a fatal error to log, it coordinates the
return of the error between checker goroutines and sends it back to the
caller.

The main impetus here is that it allows an external program compiling
against rclone as a package to handle a not found, where it currently it
cannot.

This does change error output on a not found a little bit, we go from
this:

    2017/01/09 21:14:03 directory not found

To this:

    2017/01/09 21:13:44 Failed to ls: directory not found
 2017-01-12 10:07:59 +00:00
Nick Craig-Wood
78c1f2839e Fix filters to add ** rules to directory rules 
This fixes `--exclude ".*{,/**}"` to exclude all . files and
. directories.
 2017-01-05 19:33:49 +00:00
Nick Craig-Wood
de2d967abd Stop --track-renames hashing matching files - fixes #984 
Also only hash files of the correct size.

This speeds it up a lot.
 2017-01-05 17:58:01 +00:00
Marco Paganini
6611d92e21 Only start bandwidth ticker when necessary. 
- Only start the token ticker when the timetable entry has more than one
  entry.
- This fixes the "Scheduled bandwidth change" log message when no
  bwlimit is specified.
- Fixes #987
 2017-01-04 19:03:49 -08:00
Nick Craig-Wood
f73ee5eade Make all config file variables be settable in the environment 
These are set in the form RCLONE_CONFIG_remote_option where remote is
the uppercased remote name and option is the uppercased config file
option name.  Note that RCLONE_CONFIG_remote_TYPE must be set if
defining a new remote.

Fixes #616
 2017-01-03 22:42:47 +00:00
Nick Craig-Wood
0d75d2585f Allow all options to be set from environment variables 
The option names are munged changing - to _ making upper case and
prepending RCLONE_.  The values are as parsed by pflag.
 2017-01-03 22:42:47 +00:00
Marco Paganini
3b0f944e23 Add time-based bandwidth limits. 
- Change the --bwlimit command line parameter to accept both a limit (as
  before) or a full timetable (formatted as "hh:mm,limit
  hh:mm,limit...")
- The timetable is checked once a minute by a ticker function. A new
  tokenBucket is created every time a bandwidth change is necessary.
- This change is compatible with the SIGUSR2 change to toggle bandwidth
  limits.

This resolves #221.
 2017-01-03 21:00:38 +00:00
Nick Craig-Wood
f1221b510b Change --track-renames to use the length,hash pair stored in a map 
This makes it much faster in the case of many files and use less
memory.

This also detects use of --no-traverse and disables it.
 2017-01-03 20:37:06 +00:00
Nick Craig-Wood
274ab349f4 sync: Only allow --track-renames if have a common hash 2017-01-03 20:35:05 +00:00
Bjørn Erik Pedersen
47d3a450a4 sync: Track and perform server-side renames 
This commits adds support for tracking of file renames if `track-renames` flag is set,
and it then performs server-side renames for remotes that support it, i.e.
remotes that implement either the `Mover` or the `Copier` interface.

Fixes #888
 2017-01-03 20:35:05 +00:00
Nick Craig-Wood
5b8b379feb Version v1.35 2017-01-02 15:33:06 +00:00
Nick Craig-Wood
4dd5428b13 Fix rmdirs test and integration tests which depend on each other 2017-01-02 14:15:07 +00:00
Nick Craig-Wood
64ec220d5d Fix --no-update-modtime test on remotes which don't support hashes 2016-12-31 15:19:26 +00:00
Nick Craig-Wood
cbfec0d281 Fix tests for missing config file 2016-12-20 15:05:08 +00:00
Nick Craig-Wood
1b2dda8c4c oauthutil: Reload config file off disk before updating token 
This fixes the config file being overwritten when two rclones are running.

Fixes #682
 2016-12-19 15:04:07 +00:00
Nick Craig-Wood
4482e75f38 Fix golint 2016-12-15 21:02:41 +00:00
Nick Craig-Wood
0117aeafbf mount: this removes the async buffering as it was killing seek performance 2016-12-15 17:08:52 +00:00
Nick Craig-Wood
8083804575 Make sure wrapped retry/fatal errors are never nil to avoid panic 2016-12-13 16:02:14 +00:00
Nick Craig-Wood
fafbcc8e2f Make server side move more obvious in debug 2016-12-09 16:57:07 +00:00
Nick Craig-Wood
48a2e3844d Add optional interface DirCacheFlush for making the tests more reliable 
This is defined for the users of dircache drive, onedrive, and acd.

This helps fix the DirMove tests on acd.
 2016-12-09 16:57:07 +00:00
Nick Craig-Wood
dcf53a1d12 Allows multiple --include/--exclude/--filter options - fixes #875 2016-12-07 13:37:40 +00:00
Nick Craig-Wood
3bdfa284a9 Make rclone lsd obey the filters properly 2016-12-07 11:16:36 +00:00
Nick Craig-Wood
be4fd51289 fuse: Add bandwidth accounting and buffering 
This fixes rclone mount ignoring bwlimit and increases buffering which
should speed up transfers greatly.

Fixes #796
Fixes #690
 2016-12-04 16:57:47 +00:00
Nick Craig-Wood
c265f451f2 Implement moveto and copyto commands for choosing a destination name on copy/move 
Fixes #227
Fixes #476
 2016-12-03 23:43:52 +00:00
Nick Craig-Wood
2058652fa4 Allow overlapping remotes in move when DirMove is supported 2016-12-03 09:08:40 +00:00
Nick Craig-Wood
50b3cfccb1 Factor Move out of sync.go and add remote parameter to Move and Copy 2016-12-03 09:08:40 +00:00
Nick Craig-Wood
2756900749 Fix not transferring files that don't differ in size - fixes #911 
Due to a logic error files stored on remotes which support modtime but
not hashes weren't being transferred when updating with a file of the
same size but different modtime.  Instead the modtime of the remote
file was being set to that of the local file.

In practice this affected crypt with all remotes except Amazon Drive
and Dropbox.
 2016-11-28 17:08:15 +00:00
Nick Craig-Wood
539853df36 Fix rmdirs test 2016-11-28 12:23:24 +00:00
Scott McGillivray
f9df545e3c add --stats-unit option and improve alignment for --stats output 2016-11-28 12:18:30 +00:00
Nick Craig-Wood
f3365dd251 Make rclone rmdirs command to delete empty directories - fixes #831 2016-11-27 11:49:31 +00:00
Nick Craig-Wood
aaa1370a36 Add directory parameter to Rmdir and Mkdir #100 #831 
This will enable rclone to manage directories properly in the future.
 2016-11-26 12:02:53 +00:00
Nick Craig-Wood
d95288175f Version v1.34 2016-11-06 10:18:30 +00:00
Nick Craig-Wood
01be5bff02 Fix ogier/pflag vs spf13/pflag 2016-11-05 18:18:08 +00:00
Nick Craig-Wood
e825df6448 Fix Check on crypted file systems 2016-11-05 18:17:21 +00:00
Nick Craig-Wood
ff41b0d435 Improve error message when source remote isn't found in sync #848 2016-11-05 18:03:55 +00:00
Nick Craig-Wood
441951a93b Stop removing failed upload to cloud storage remotes - fixes #559 
We do remove a partially written file on local so we don't have
corrupted files lying around.
 2016-11-04 21:34:25 +00:00
Nick Craig-Wood
318e335137 Remove Authorization: headers from --dump-headers output 
Add in `--dump-auth` flag to put it back.
 2016-11-02 15:53:43 +00:00
Nick Craig-Wood
d1759fdfa9 Add request ID to HTTP debugging to make it easier to trace concurrent flows 2016-10-31 12:01:28 +00:00
Nick Craig-Wood
e65059e431 Fix non-windows/non-unix builds for bwlimit/SIGUSR2 feature and add a mutex 
The race detector complained whenever SIGUSR2 was sent to rclone so
this adds a mutex to prevent concurrent access.
 2016-10-30 19:20:16 +00:00
Nick Craig-Wood
5454f2abd0 Fix race in checkServerTime 2016-10-30 19:16:27 +00:00
Marco Paganini
cc4f5ba7ba Add support to toggle bandwidth limits via SIGUSR2. 
Sending rclone a SIGUSR2 signal will toggle the limiter between off and
the limit set with the --bwlimit command-line option.
 2016-10-30 17:46:59 +00:00
Nick Craig-Wood
de73063977 Fix output of crypt objects in logs 2016-10-20 17:46:51 +01:00
Nick Craig-Wood
b7875fc02a rclone check: show count of hashes that couldn't be checked #700 2016-10-17 16:48:11 +01:00
Nick Craig-Wood
8a56a6836a Check server time against local time #654 2016-10-08 14:00:50 +01:00
Nick Craig-Wood
83849e0a36 Don't show encrypted password to stop confusion - fixes #656 2016-10-08 11:26:14 +01:00
Nick Craig-Wood
d42b38699b Make ResponseHeaderTimeout be --timeout not --contimeout fixes #766 
This was causing a problem with Amazon Drive which often pauses for a
long time after uploads before returning the response.
 2016-10-08 10:12:19 +01:00
Nick Craig-Wood
d033e92234 Stop single file and --files-from operations iterating through the source bucket. 
This works by making sure directory listings that use a filter only
iterate the files provided in the filter (if any).

Single file copies now don't iterate the source or destination
buckets.

Note that this could potentially slow down very long `--files-from`
lists - this is easy to fix (with another flag probably) if it causes
anyone a problem.

Fixes #610
Fixes #769
 2016-10-07 11:39:39 +01:00
Nick Craig-Wood
e94850f322 Fix timeouts not working when set to 0 and firing too often - #766 2016-10-06 10:23:23 +01:00
Nick Craig-Wood
aef2ac5c04 Add options for Open and implement Range for all remotes 2016-10-05 21:03:56 +01:00
Nick Craig-Wood
9ea20bac42 Fix accidentally committed test in move code 2016-10-03 20:16:41 +01:00
Nick Craig-Wood
945f49ab5e Make ContentType be preserved for cloud -> cloud copies - fixes #733 2016-10-03 20:02:04 +01:00
Nick Craig-Wood
6c9a258d82 Fix move command 
* Delete src files which already existed in dst - fixes #751
  * Fix deletion of src file when dst file older
 2016-10-03 19:58:44 +01:00
Nick Craig-Wood
f2eeb4301c Make --dump-bodies imply --dump-headers 2016-09-22 08:40:37 +01:00
Nick Craig-Wood
0caac70994 Fix build for < go1.7 2016-09-13 11:36:14 +01:00
Nick Craig-Wood
83ba59749f Make failed uploads not count as "Transferred" - fixes #708 2016-09-12 18:15:58 +01:00
Nick Craig-Wood
bbf819e2d1 Move versioncheck so it happens earlier in the compile process. 2016-09-12 17:50:19 +01:00
Nick Craig-Wood
0cb9bb3b54 Redo http Transport code 
* Insert User-Agent in Transport - fixes #199
  * Update timeouts to use Context
  * Modernise transport
 2016-09-12 17:50:19 +01:00
Nick Craig-Wood
2eaac80c86 b2 with crypt: fix crash when uploading large files - fixes #673 2016-09-05 18:10:01 +01:00
Nick Craig-Wood
3996bbb8cb Version v1.33 2016-08-24 23:02:05 +01:00
Nick Craig-Wood
059743a1b0 crypt: add to integration tests 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
a54806e5c1 Fix Move when underlying remote returns ErrorCantMove 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
226c2a0d83 Implement crypt for encrypted remotes - #219 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
9985fc40f4 Make Password parameters obey Optional flag and offer to generate random ones 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
b1de4c8cba Implement password Option and re-implement editing 
Editing now shows all the options for the fs and asks one at a time
whether they should be changed.
 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
6a4e424630 Re-implement Obscure/Reveal so they use AES-CTR encryption 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
ebb67c135e Fix listToChan passing nil objects to DeleteFile 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
037a000cc8 b2: fix stats accounting for upload - fixes #602 2016-08-22 21:19:38 +01:00
Nick Craig-Wood
84eb7031bb Implement the rclone cat command 2016-08-18 22:45:32 +01:00
Nick Craig-Wood
d7b79b4481 Mark the compiled from source version with -DEV - fixes #627 2016-08-18 21:31:10 +01:00
Nick Craig-Wood
b5faaf7116 Fix double close of abort channel - fixes #592 2016-08-18 18:56:57 +01:00
Nick Craig-Wood
0805ec051f Add BasicInfo interface shared between Dir and Object 2016-08-05 17:45:27 +01:00
Nick Craig-Wood
ae56df7d4f Add --dedupe-mode only to dedupe command 2016-08-04 21:47:14 +01:00
Nick Craig-Wood
67562081f7 Version v1.32 2016-07-13 17:32:39 +01:00
Nick Craig-Wood
63f6827a0d Version v1.31 2016-07-13 12:28:01 +01:00
Nick Craig-Wood
ac3c83f966 Fix integration tests for drive 2016-07-12 21:38:15 +01:00