Commit Graph

304 Commits

Author SHA1 Message Date
Brandur
3b1e0b66bb Return error on not found from ListFn
This changes `ListFn`'s implementation so that if it encounters a not
found error, instead of sending a fatal error to log, it coordinates the
return of the error between checker goroutines and sends it back to the
caller.

The main impetus here is that it allows an external program compiling
against rclone as a package to handle a not found, where it currently it
cannot.

This does change error output on a not found a little bit, we go from
this:

    2017/01/09 21:14:03 directory not found

To this:

    2017/01/09 21:13:44 Failed to ls: directory not found
2017-01-12 10:07:59 +00:00
Nick Craig-Wood
78c1f2839e Fix filters to add ** rules to directory rules
This fixes `--exclude ".*{,/**}"` to exclude all . files and
. directories.
2017-01-05 19:33:49 +00:00
Nick Craig-Wood
de2d967abd Stop --track-renames hashing matching files - fixes #984
Also only hash files of the correct size.

This speeds it up a lot.
2017-01-05 17:58:01 +00:00
Marco Paganini
6611d92e21 Only start bandwidth ticker when necessary.
- Only start the token ticker when the timetable entry has more than one
  entry.
- This fixes the "Scheduled bandwidth change" log message when no
  bwlimit is specified.
- Fixes #987
2017-01-04 19:03:49 -08:00
Nick Craig-Wood
f73ee5eade Make all config file variables be settable in the environment
These are set in the form RCLONE_CONFIG_remote_option where remote is
the uppercased remote name and option is the uppercased config file
option name.  Note that RCLONE_CONFIG_remote_TYPE must be set if
defining a new remote.

Fixes #616
2017-01-03 22:42:47 +00:00
Nick Craig-Wood
0d75d2585f Allow all options to be set from environment variables
The option names are munged changing - to _ making upper case and
prepending RCLONE_.  The values are as parsed by pflag.
2017-01-03 22:42:47 +00:00
Marco Paganini
3b0f944e23 Add time-based bandwidth limits.
- Change the --bwlimit command line parameter to accept both a limit (as
  before) or a full timetable (formatted as "hh:mm,limit
  hh:mm,limit...")
- The timetable is checked once a minute by a ticker function. A new
  tokenBucket is created every time a bandwidth change is necessary.
- This change is compatible with the SIGUSR2 change to toggle bandwidth
  limits.

This resolves #221.
2017-01-03 21:00:38 +00:00
Nick Craig-Wood
f1221b510b Change --track-renames to use the length,hash pair stored in a map
This makes it much faster in the case of many files and use less
memory.

This also detects use of --no-traverse and disables it.
2017-01-03 20:37:06 +00:00
Nick Craig-Wood
274ab349f4 sync: Only allow --track-renames if have a common hash 2017-01-03 20:35:05 +00:00
Bjørn Erik Pedersen
47d3a450a4 sync: Track and perform server-side renames
This commits adds support for tracking of file renames if `track-renames` flag is set,
and it then performs server-side renames for remotes that support it, i.e.
remotes that implement either the `Mover` or the `Copier` interface.

Fixes #888
2017-01-03 20:35:05 +00:00
Nick Craig-Wood
5b8b379feb Version v1.35 2017-01-02 15:33:06 +00:00
Nick Craig-Wood
4dd5428b13 Fix rmdirs test and integration tests which depend on each other 2017-01-02 14:15:07 +00:00
Nick Craig-Wood
64ec220d5d Fix --no-update-modtime test on remotes which don't support hashes 2016-12-31 15:19:26 +00:00
Nick Craig-Wood
cbfec0d281 Fix tests for missing config file 2016-12-20 15:05:08 +00:00
Nick Craig-Wood
1b2dda8c4c oauthutil: Reload config file off disk before updating token
This fixes the config file being overwritten when two rclones are running.

Fixes #682
2016-12-19 15:04:07 +00:00
Nick Craig-Wood
4482e75f38 Fix golint 2016-12-15 21:02:41 +00:00
Nick Craig-Wood
0117aeafbf mount: this removes the async buffering as it was killing seek performance 2016-12-15 17:08:52 +00:00
Nick Craig-Wood
8083804575 Make sure wrapped retry/fatal errors are never nil to avoid panic 2016-12-13 16:02:14 +00:00
Nick Craig-Wood
fafbcc8e2f Make server side move more obvious in debug 2016-12-09 16:57:07 +00:00
Nick Craig-Wood
48a2e3844d Add optional interface DirCacheFlush for making the tests more reliable
This is defined for the users of dircache drive, onedrive, and acd.

This helps fix the DirMove tests on acd.
2016-12-09 16:57:07 +00:00
Nick Craig-Wood
dcf53a1d12 Allows multiple --include/--exclude/--filter options - fixes #875 2016-12-07 13:37:40 +00:00
Nick Craig-Wood
3bdfa284a9 Make rclone lsd obey the filters properly 2016-12-07 11:16:36 +00:00
Nick Craig-Wood
be4fd51289 fuse: Add bandwidth accounting and buffering
This fixes rclone mount ignoring bwlimit and increases buffering which
should speed up transfers greatly.

Fixes #796
Fixes #690
2016-12-04 16:57:47 +00:00
Nick Craig-Wood
c265f451f2 Implement moveto and copyto commands for choosing a destination name on copy/move
Fixes #227
Fixes #476
2016-12-03 23:43:52 +00:00
Nick Craig-Wood
2058652fa4 Allow overlapping remotes in move when DirMove is supported 2016-12-03 09:08:40 +00:00
Nick Craig-Wood
50b3cfccb1 Factor Move out of sync.go and add remote parameter to Move and Copy 2016-12-03 09:08:40 +00:00
Nick Craig-Wood
2756900749 Fix not transferring files that don't differ in size - fixes #911
Due to a logic error files stored on remotes which support modtime but
not hashes weren't being transferred when updating with a file of the
same size but different modtime.  Instead the modtime of the remote
file was being set to that of the local file.

In practice this affected crypt with all remotes except Amazon Drive
and Dropbox.
2016-11-28 17:08:15 +00:00
Nick Craig-Wood
539853df36 Fix rmdirs test 2016-11-28 12:23:24 +00:00
Scott McGillivray
f9df545e3c add --stats-unit option and improve alignment for --stats output 2016-11-28 12:18:30 +00:00
Nick Craig-Wood
f3365dd251 Make rclone rmdirs command to delete empty directories - fixes #831 2016-11-27 11:49:31 +00:00
Nick Craig-Wood
aaa1370a36 Add directory parameter to Rmdir and Mkdir #100 #831
This will enable rclone to manage directories properly in the future.
2016-11-26 12:02:53 +00:00
Nick Craig-Wood
d95288175f Version v1.34 2016-11-06 10:18:30 +00:00
Nick Craig-Wood
01be5bff02 Fix ogier/pflag vs spf13/pflag 2016-11-05 18:18:08 +00:00
Nick Craig-Wood
e825df6448 Fix Check on crypted file systems 2016-11-05 18:17:21 +00:00
Nick Craig-Wood
ff41b0d435 Improve error message when source remote isn't found in sync #848 2016-11-05 18:03:55 +00:00
Nick Craig-Wood
441951a93b Stop removing failed upload to cloud storage remotes - fixes #559
We do remove a partially written file on local so we don't have
corrupted files lying around.
2016-11-04 21:34:25 +00:00
Nick Craig-Wood
318e335137 Remove Authorization: headers from --dump-headers output
Add in `--dump-auth` flag to put it back.
2016-11-02 15:53:43 +00:00
Nick Craig-Wood
d1759fdfa9 Add request ID to HTTP debugging to make it easier to trace concurrent flows 2016-10-31 12:01:28 +00:00
Nick Craig-Wood
e65059e431 Fix non-windows/non-unix builds for bwlimit/SIGUSR2 feature and add a mutex
The race detector complained whenever SIGUSR2 was sent to rclone so
this adds a mutex to prevent concurrent access.
2016-10-30 19:20:16 +00:00
Nick Craig-Wood
5454f2abd0 Fix race in checkServerTime 2016-10-30 19:16:27 +00:00
Marco Paganini
cc4f5ba7ba Add support to toggle bandwidth limits via SIGUSR2.
Sending rclone a SIGUSR2 signal will toggle the limiter between off and
the limit set with the --bwlimit command-line option.
2016-10-30 17:46:59 +00:00
Nick Craig-Wood
de73063977 Fix output of crypt objects in logs 2016-10-20 17:46:51 +01:00
Nick Craig-Wood
b7875fc02a rclone check: show count of hashes that couldn't be checked #700 2016-10-17 16:48:11 +01:00
Nick Craig-Wood
8a56a6836a Check server time against local time #654 2016-10-08 14:00:50 +01:00
Nick Craig-Wood
83849e0a36 Don't show encrypted password to stop confusion - fixes #656 2016-10-08 11:26:14 +01:00
Nick Craig-Wood
d42b38699b Make ResponseHeaderTimeout be --timeout not --contimeout fixes #766
This was causing a problem with Amazon Drive which often pauses for a
long time after uploads before returning the response.
2016-10-08 10:12:19 +01:00
Nick Craig-Wood
d033e92234 Stop single file and --files-from operations iterating through the source bucket.
This works by making sure directory listings that use a filter only
iterate the files provided in the filter (if any).

Single file copies now don't iterate the source or destination
buckets.

Note that this could potentially slow down very long `--files-from`
lists - this is easy to fix (with another flag probably) if it causes
anyone a problem.

Fixes #610
Fixes #769
2016-10-07 11:39:39 +01:00
Nick Craig-Wood
e94850f322 Fix timeouts not working when set to 0 and firing too often - #766 2016-10-06 10:23:23 +01:00
Nick Craig-Wood
aef2ac5c04 Add options for Open and implement Range for all remotes 2016-10-05 21:03:56 +01:00
Nick Craig-Wood
9ea20bac42 Fix accidentally committed test in move code 2016-10-03 20:16:41 +01:00