Nick Craig-Wood
48a2e3844d
Add optional interface DirCacheFlush for making the tests more reliable
...
This is defined for the users of dircache drive, onedrive, and acd.
This helps fix the DirMove tests on acd.
2016-12-09 16:57:07 +00:00
Nick Craig-Wood
dcf53a1d12
Allows multiple --include/--exclude/--filter options - fixes #875
2016-12-07 13:37:40 +00:00
Nick Craig-Wood
3bdfa284a9
Make rclone lsd obey the filters properly
2016-12-07 11:16:36 +00:00
Nick Craig-Wood
be4fd51289
fuse: Add bandwidth accounting and buffering
...
This fixes rclone mount ignoring bwlimit and increases buffering which
should speed up transfers greatly.
Fixes #796
Fixes #690
2016-12-04 16:57:47 +00:00
Nick Craig-Wood
c265f451f2
Implement moveto and copyto commands for choosing a destination name on copy/move
...
Fixes #227
Fixes #476
2016-12-03 23:43:52 +00:00
Nick Craig-Wood
2058652fa4
Allow overlapping remotes in move when DirMove is supported
2016-12-03 09:08:40 +00:00
Nick Craig-Wood
50b3cfccb1
Factor Move out of sync.go and add remote parameter to Move and Copy
2016-12-03 09:08:40 +00:00
Nick Craig-Wood
2756900749
Fix not transferring files that don't differ in size - fixes #911
...
Due to a logic error files stored on remotes which support modtime but
not hashes weren't being transferred when updating with a file of the
same size but different modtime. Instead the modtime of the remote
file was being set to that of the local file.
In practice this affected crypt with all remotes except Amazon Drive
and Dropbox.
2016-11-28 17:08:15 +00:00
Nick Craig-Wood
539853df36
Fix rmdirs test
2016-11-28 12:23:24 +00:00
Scott McGillivray
f9df545e3c
add --stats-unit option and improve alignment for --stats output
2016-11-28 12:18:30 +00:00
Nick Craig-Wood
f3365dd251
Make rclone rmdirs command to delete empty directories - fixes #831
2016-11-27 11:49:31 +00:00
Nick Craig-Wood
aaa1370a36
Add directory parameter to Rmdir and Mkdir #100 #831
...
This will enable rclone to manage directories properly in the future.
2016-11-26 12:02:53 +00:00
Nick Craig-Wood
d95288175f
Version v1.34
2016-11-06 10:18:30 +00:00
Nick Craig-Wood
01be5bff02
Fix ogier/pflag vs spf13/pflag
2016-11-05 18:18:08 +00:00
Nick Craig-Wood
e825df6448
Fix Check on crypted file systems
2016-11-05 18:17:21 +00:00
Nick Craig-Wood
ff41b0d435
Improve error message when source remote isn't found in sync #848
2016-11-05 18:03:55 +00:00
Nick Craig-Wood
441951a93b
Stop removing failed upload to cloud storage remotes - fixes #559
...
We do remove a partially written file on local so we don't have
corrupted files lying around.
2016-11-04 21:34:25 +00:00
Nick Craig-Wood
318e335137
Remove Authorization:
headers from --dump-headers
output
...
Add in `--dump-auth` flag to put it back.
2016-11-02 15:53:43 +00:00
Nick Craig-Wood
d1759fdfa9
Add request ID to HTTP debugging to make it easier to trace concurrent flows
2016-10-31 12:01:28 +00:00
Nick Craig-Wood
e65059e431
Fix non-windows/non-unix builds for bwlimit/SIGUSR2 feature and add a mutex
...
The race detector complained whenever SIGUSR2 was sent to rclone so
this adds a mutex to prevent concurrent access.
2016-10-30 19:20:16 +00:00
Nick Craig-Wood
5454f2abd0
Fix race in checkServerTime
2016-10-30 19:16:27 +00:00
Marco Paganini
cc4f5ba7ba
Add support to toggle bandwidth limits via SIGUSR2.
...
Sending rclone a SIGUSR2 signal will toggle the limiter between off and
the limit set with the --bwlimit command-line option.
2016-10-30 17:46:59 +00:00
Nick Craig-Wood
de73063977
Fix output of crypt objects in logs
2016-10-20 17:46:51 +01:00
Nick Craig-Wood
b7875fc02a
rclone check: show count of hashes that couldn't be checked #700
2016-10-17 16:48:11 +01:00
Nick Craig-Wood
8a56a6836a
Check server time against local time #654
2016-10-08 14:00:50 +01:00
Nick Craig-Wood
83849e0a36
Don't show encrypted password to stop confusion - fixes #656
2016-10-08 11:26:14 +01:00
Nick Craig-Wood
d42b38699b
Make ResponseHeaderTimeout be --timeout
not --contimeout
fixes #766
...
This was causing a problem with Amazon Drive which often pauses for a
long time after uploads before returning the response.
2016-10-08 10:12:19 +01:00
Nick Craig-Wood
d033e92234
Stop single file and --files-from
operations iterating through the source bucket.
...
This works by making sure directory listings that use a filter only
iterate the files provided in the filter (if any).
Single file copies now don't iterate the source or destination
buckets.
Note that this could potentially slow down very long `--files-from`
lists - this is easy to fix (with another flag probably) if it causes
anyone a problem.
Fixes #610
Fixes #769
2016-10-07 11:39:39 +01:00
Nick Craig-Wood
e94850f322
Fix timeouts not working when set to 0 and firing too often - #766
2016-10-06 10:23:23 +01:00
Nick Craig-Wood
aef2ac5c04
Add options for Open and implement Range for all remotes
2016-10-05 21:03:56 +01:00
Nick Craig-Wood
9ea20bac42
Fix accidentally committed test in move code
2016-10-03 20:16:41 +01:00
Nick Craig-Wood
945f49ab5e
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-10-03 20:02:04 +01:00
Nick Craig-Wood
6c9a258d82
Fix move command
...
* Delete src files which already existed in dst - fixes #751
* Fix deletion of src file when dst file older
2016-10-03 19:58:44 +01:00
Nick Craig-Wood
f2eeb4301c
Make --dump-bodies imply --dump-headers
2016-09-22 08:40:37 +01:00
Nick Craig-Wood
0caac70994
Fix build for < go1.7
2016-09-13 11:36:14 +01:00
Nick Craig-Wood
83ba59749f
Make failed uploads not count as "Transferred" - fixes #708
2016-09-12 18:15:58 +01:00
Nick Craig-Wood
bbf819e2d1
Move versioncheck so it happens earlier in the compile process.
2016-09-12 17:50:19 +01:00
Nick Craig-Wood
0cb9bb3b54
Redo http Transport code
...
* Insert User-Agent in Transport - fixes #199
* Update timeouts to use Context
* Modernise transport
2016-09-12 17:50:19 +01:00
Nick Craig-Wood
2eaac80c86
b2 with crypt: fix crash when uploading large files - fixes #673
2016-09-05 18:10:01 +01:00
Nick Craig-Wood
3996bbb8cb
Version v1.33
2016-08-24 23:02:05 +01:00
Nick Craig-Wood
059743a1b0
crypt: add to integration tests
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
a54806e5c1
Fix Move when underlying remote returns ErrorCantMove
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
226c2a0d83
Implement crypt for encrypted remotes - #219
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
9985fc40f4
Make Password parameters obey Optional flag and offer to generate random ones
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
b1de4c8cba
Implement password Option and re-implement editing
...
Editing now shows all the options for the fs and asks one at a time
whether they should be changed.
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
6a4e424630
Re-implement Obscure/Reveal so they use AES-CTR encryption
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
ebb67c135e
Fix listToChan passing nil objects to DeleteFile
2016-08-23 17:45:37 +01:00
Nick Craig-Wood
037a000cc8
b2: fix stats accounting for upload - fixes #602
2016-08-22 21:19:38 +01:00
Nick Craig-Wood
84eb7031bb
Implement the rclone cat command
2016-08-18 22:45:32 +01:00
Nick Craig-Wood
d7b79b4481
Mark the compiled from source version with -DEV - fixes #627
2016-08-18 21:31:10 +01:00
Nick Craig-Wood
b5faaf7116
Fix double close of abort channel - fixes #592
2016-08-18 18:56:57 +01:00
Nick Craig-Wood
0805ec051f
Add BasicInfo interface shared between Dir and Object
2016-08-05 17:45:27 +01:00
Nick Craig-Wood
ae56df7d4f
Add --dedupe-mode only to dedupe command
2016-08-04 21:47:14 +01:00
Nick Craig-Wood
67562081f7
Version v1.32
2016-07-13 17:32:39 +01:00
Nick Craig-Wood
63f6827a0d
Version v1.31
2016-07-13 12:28:01 +01:00
Nick Craig-Wood
ac3c83f966
Fix integration tests for drive
2016-07-12 21:38:15 +01:00
Nick Craig-Wood
b9c8e61d39
Explicitly check the state in tests after writing files
...
...otherwise Amazon Drive will fail.
2016-07-12 21:36:39 +01:00
Nick Craig-Wood
a6056408dd
Fix move command - stop it running for overlapping fses - fixes #577
...
* Make move command check for overlapping remotes and refuse to run
* Do copy/delete rather than all the copies then all the deletes
* Doesn't purge the source - this was unexpected behaviour see #512 and #416
* Add -list-retries flag to test suite to control retries
This changes the semantics of `move` slightly. However it now errs on
the side of not deleting stuff.
2016-07-12 10:49:37 +01:00
Nick Craig-Wood
b9479cf7ab
Implement --no-update-modtime flag - fixes #511
2016-07-12 10:46:45 +01:00
Nick Craig-Wood
a20d80565b
Tidy stats output - fixes #541
2016-07-11 13:04:30 +01:00
Nick Craig-Wood
4bd9932703
Fix wording in verbose copy logs - fixes #574
2016-07-09 10:11:57 +01:00
Nick Craig-Wood
13591c7c00
Redo error handling for sync/copy/move
...
* Factor sync/copy/move into its own file
* Make fatal errors abort the sync
* Make Copy return errors
* Make Sync/Copy/Move return the last Copy error if there was one
* Prioritise returning Fatal errors
* NoRetry errors are returned if no other types of errors
2016-07-04 13:45:53 +01:00
Nick Craig-Wood
28f4061892
Add two more classes of error Fatal and NoRetry
...
These are for remotes to signal that they have a fatal error and don't
want to continue (eg cap exceeded) or that a particular file shouldn't
be retried for some reason.
2016-07-04 13:45:52 +01:00
Nick Craig-Wood
0a43ff9c13
Modify interface for accounting to take a string not an fs.Object
2016-07-02 16:58:50 +01:00
Nick Craig-Wood
9aae143833
Implement cleanup command for emptying trash / removing old versions of files
2016-07-01 16:35:36 +01:00
Nick Craig-Wood
70dc97231e
Convert more tests to use assert/require
2016-06-30 15:45:30 +01:00
Nick Craig-Wood
af4ef8ad8d
Implement --no-traverse flag to stop copy traversing the destination remote.
...
Refactor sync/copy/move
* Don't load the src listing unless doing a sync and --delete-before
* Don't load the dst listing if doing copy/move and --no-traverse is set
`rclone --no-traverse copy src dst` now won't load either of the
listings into memory so will use the minimum amount of memory.
This change will reduce the amount of memory rclone uses dramatically
too as in normal operations (copy without --notraverse or sync) as it
no longer loads the source file listing into memory at all.
Fixes #8
Fixes #544
Fixes #546
2016-06-29 17:38:50 +01:00
Nick Craig-Wood
13797a1fb8
Make retry logs be debug in main copy routine
2016-06-28 08:51:57 +01:00
Nick Craig-Wood
3ad8fb8634
Make DeleteFile and DeleteFiles return errors
2016-06-28 08:51:57 +01:00
Nick Craig-Wood
ab43005422
Make NewObject return an error
...
* make it return an error
* make a canonical error fs.ErrorNotFound
* make a test for it
* remove logs/debugs of error
2016-06-28 08:51:57 +01:00
Nick Craig-Wood
b1f131964e
Rename NewFsObject to NewObject
2016-06-28 08:51:57 +01:00
Nick Craig-Wood
1a87b69376
Get rid of LimitedFs - FIXME needs docs on copying single files
...
If remote:path points to a file make NewFs return a sentinel error
fs.ErrorIsFile and an Fs which points to the parent.
Use this to remove the LimitedFs and just add this file to the
--files-from list.
This means that server side operations can be used also.
Fixes #518
Fixes #545
2016-06-28 08:51:43 +01:00
Nick Craig-Wood
5a3b109e25
Fix issues identified by go vet -shadow - fixes #530
2016-06-21 21:17:52 +01:00
Klaus Post
e0aa4bb492
Fix incomplete local hashes.
...
Fixes #533
2016-06-19 16:51:49 +02:00
Nick Craig-Wood
bd0227450e
Version v1.30
2016-06-18 16:41:46 +01:00
Nick Craig-Wood
f438f1e9ef
Fix stats print
2016-06-18 16:41:46 +01:00
Nick Craig-Wood
d3dd672640
Document recursion requirements for Fses
2016-06-18 14:12:47 +01:00
Nick Craig-Wood
2a46be8cf3
b2: implement large file uploading - fixes #456
2016-06-18 13:38:05 +01:00
Nick Craig-Wood
1b4370bde1
Rework retry logic when copying objects
...
* Fix off by one retry logic - fixes #406
* Retry any retriable errors
* Restructure code
2016-06-18 10:55:58 +01:00
Nick Craig-Wood
2cfb3834f2
Log errors with %v
2016-06-18 09:36:47 +01:00
Nick Craig-Wood
46135d830e
Add --ignore-size flag - fixes #399
2016-06-17 17:20:08 +01:00
Nick Craig-Wood
df1092ef33
Change Fs.Put so that it must cope with existing files
...
This should fix duplicate files on drive and 409 errors on
amazonclouddrive however it will slow down the upload slightly as
another roundtrip will be needed.
None of the other Fses needed adjusting.
Fixes #483
2016-06-13 19:29:10 +01:00
Nick Craig-Wood
4c5b2833b3
Convert to using github.com/pkg/errors everywhere
2016-06-13 17:43:03 +01:00
Nick Craig-Wood
7fe653c350
Unwrap errors properly for patform specific connection retry code.
...
Include more possible errors for Windows.
For #442
2016-06-10 13:48:41 +01:00
Nick Craig-Wood
f17cb1bf50
Fix retry of Windows wsaend errors #442
...
Make the test for wsaend error less specific
2016-06-09 15:34:13 +01:00
Nick Craig-Wood
67d0375b98
Audit use of log.Print and change to Debug, Log, or ErrorLog as appropriate
2016-06-06 21:23:54 +01:00
Nick Craig-Wood
108760e17b
Log -v output to stdout by default - fixes #228
2016-06-04 18:49:27 +01:00
Nick Craig-Wood
e2788aa729
Display the transfer stats in more human readable form - fixes #428
2016-06-03 22:49:50 +01:00
Nick Craig-Wood
772f99fd74
Make SizeSuffix output without b
suffix for more useful printouts
2016-06-03 22:49:14 +01:00
Nick Craig-Wood
a21cc161de
Make 0 size files specifiable with --max-size 0b
- fixes #450
2016-06-03 21:54:27 +01:00
Nick Craig-Wood
e818b7c206
Represent -1 as "off" for SIZE values
2016-06-03 21:51:39 +01:00
Nick Craig-Wood
5723d788a4
Add b
suffix so we can specify bytes in --bwlimit, --min-size etc
...
Fixes #449
2016-06-03 21:16:48 +01:00
Nick Craig-Wood
1d6698a754
Build tweaks - fixes #484
...
* disable CGO for static builds everywhere
* override Version in release build script
* don't output symbol table in release binaries
2016-06-03 20:34:19 +01:00
Nick Craig-Wood
ccdd1ea6c4
Add --max-depth parameter
...
This will apply to ls/lsd/sync/copy etc
Fixes #412
Fixes #213
2016-06-03 17:05:39 +01:00
Nick Craig-Wood
ad85f6e413
Implement directory include filtering for efficiency
...
Fixes #395
2016-05-16 17:14:04 +01:00
Nick Craig-Wood
ac9c20b048
Make IsRetryError function
2016-05-14 17:11:19 +01:00
Nick Craig-Wood
2db35f0ce7
Dump out unexpected state in integration test
2016-05-07 21:19:26 +01:00
Nick Craig-Wood
dbfa7031d2
Factor Lister into own file, write tests and fix
2016-05-07 17:17:43 +01:00
Nick Craig-Wood
c2d0e86431
Add more tests for List() and fix resulting problems
2016-05-07 14:50:35 +01:00
Nick Craig-Wood
68ec6a9f5b
Add a directory parameter to Fs.List()
2016-05-06 16:52:34 +01:00