Nick Craig-Wood
4a0a42c2f1
swift: large file upload fixes
...
* Read metadata in file listing for 0 length files to fix syncs
* Ignore non-existent files in isManifestFile to fix errors on copy
* remove nsToSwiftFloatString - experiments with the swift program
indicate that it puts a variable number of points after the
decimal, so might as well use the one in the swift library.
* Make sure segments get deleted properly when move from segmented
to non segmented and vice versa
* Use internal list routine to detect errors on listing
* Remove the _segments container if possible
* Remove manifest first when deleting
2015-09-26 17:58:04 +01:00
Sergey Tolmachev
cc7b9af50e
swift: support files > 5GB - fixes #46
...
* Write segments to ..._segments container
* Choice of container and segment names compatible with swift tool
* See http://docs.openstack.org/developer/swift/overview_large_objects.html
* Controlled by command line flag --swift-chunk-size
* Segments removed on delete
2015-09-26 13:03:58 +01:00
Nick Craig-Wood
68fef49c55
swift: fetch headers as the only source of metadata
2015-09-26 13:03:58 +01:00
Nick Craig-Wood
5d4b149884
Version v1.21
2015-09-22 21:05:11 +01:00
Nick Craig-Wood
5f20ae707d
Make lsl output times in localtime and fix tests - fixes #141
2015-09-22 19:04:12 +01:00
Nick Craig-Wood
e9c915e6fe
Fix golint warnings
2015-09-22 18:47:16 +01:00
Nick Craig-Wood
2ed158aba3
Fixes from go vet and errcheck
2015-09-22 07:31:12 +01:00
Nick Craig-Wood
05050d53ad
Implement compliant pacing scheme for Amazon Cloud Drive
...
* Implement switchable pacing algorithm
* Add tests for pacer
2015-09-21 21:49:54 +01:00
Nick Craig-Wood
e391311512
gofmt
2015-09-17 18:42:39 +01:00
klauspost
3234c28f7c
Minor adjustments to Unc path conversion function.
2015-09-17 18:42:10 +01:00
klauspost
6fbd9cf24b
Make Google Drive Directory reads concurrent for increased speed.
...
If you use the -v flag then this will show the progress of directory
listing. Fixes #143 .
2015-09-17 18:42:10 +01:00
klauspost
bc5b63ffef
Move ACD Connection-limiter into Pacer.
2015-09-17 18:42:10 +01:00
klauspost
788ef76f1c
Show more of the filename and align output.
...
Print more of the file name, and make the output aligned, so it is nicer on frequent updates.
2015-09-17 15:53:45 +02:00
Nick Craig-Wood
0872ec3204
Allow user to override credentials again in drive, gcs and acd - fixes #139
2015-09-16 20:08:40 +01:00
klauspost
0a5870208e
Display individual transfer progress
...
Improve progress printing by displaying individual file progress, as well
as a moving average speed with ETA. Example output:
2015/09/15 16:38:21
Transferred: 183599104 Bytes (4646.49 kByte/s)
Errors: 0
Checks: 1
Transferred: 0
Elapsed time: 38.5s
Transferring:
* 01_06_14.mp3: 33% done. avg: 1280.5, cur: 1288.8 kByte/s. ETA: 1m12s
* 01_12_15.mp3: 33% done. avg: 1002.2, cur: 943.4 kByte/s. ETA: 1m17s
* 01_13_14.mp3: 48% done. avg: 1456.8, cur: 1425.2 kByte/s. ETA: 39s
* 01_19_15.mp3: 28% done. avg: 1226.9, cur: 1114.4 kByte/s. ETA: 1m37s
2015-09-16 19:42:46 +01:00
klauspost
3219334c3e
Add "-race" test to travis.
2015-09-16 12:26:28 +02:00
klauspost
79fd662676
Protect concurrent read/writes to pacer.
...
Protects all variables in the pacer from concurrent modifications. It is now safe to modify pacer settings while it is running.
I decided to not go for an RWMutex, since all accesses are very short, so the overhead of an RWMutex isn't worth it.
Fixes #138 .
2015-09-16 12:25:55 +02:00
Nick Craig-Wood
34193fd8d9
Version v1.20
2015-09-15 07:42:31 +01:00
Nick Craig-Wood
2203766f77
Retry listing in tests to work around eventual consistency
2015-09-14 21:01:25 +01:00
Nick Craig-Wood
235cbe0e57
amazon cloud drive: retry 409 errors too
2015-09-14 21:00:44 +01:00
klauspost
f50f353b5d
local: always use UNC paths on Windows - fixes #124 , fixes #130 , fixes #90
...
* Convert all paths to UNC paths on Windows.
* Update local filesystem to always use UNC paths.
* Change tests, so they can work with Windows character replacements.
* Remove "/" suffix on paths.
* Always use path/filepath
2015-09-14 19:58:03 +01:00
Klaus Post
00afe6cc96
ACD: Shorten minimum and maximum sleep.
2015-09-14 18:22:21 +01:00
Klaus Post
dd48e62b7e
Limit concurrent directory listings.
...
Never run more than "fs.Config.Checkers" directory listings at once.
2015-09-14 18:11:59 +01:00
Klaus Post
a1a780e847
Read folders in separate goroutines.
...
As proposed in the FIXME, read folders in parallel.
This appears to fix "Next token is expired" on very big directories.
The only downside is that this doesn't abort at once if an error is found.
I added some logging, so there is some output for "-v".
2015-09-14 18:11:59 +01:00
Klaus Post
fa87077211
Retry if a timeout occurs.
...
This happens more rarely than 500, but can still be encountered.
2015-09-14 18:02:35 +01:00
Klaus Post
6ac7145d2d
Retry when we get a "500 Internal server error".
...
It seems like this happens randomly, and retrying works fine for that.
2015-09-14 18:02:35 +01:00
Nick Craig-Wood
f1226f19b2
drive, googlecloudstorage: optionally use auto config for the oauth token
2015-09-12 14:17:39 +01:00
Klaus Post
3ecbf2af25
Fix missing link text.
2015-09-12 11:22:17 +02:00
Nick Craig-Wood
79f2e95bf9
Add Klaus Post to Contributors
2015-09-11 20:20:28 +01:00
klauspost
faee50b238
List multiple matched commands.
...
* When multiple commands match, list them all.
* Change example to an existing command.
2015-09-11 19:44:50 +01:00
klauspost
807d4a3c00
Improve OAUTH2 usability - fixes #131
...
Shorten the URL to be used by the user and automatically use the
returned code by the server. The browser is opened on
`http://(bindaddress)/auth `, and redirected to the actual URL. When
the code is returned it is automatically inserted, instead of
requiring a copy+paste.
This is also a workaround for the "open" package, which escapes "&"
wrongly on Windows, so the opened URL's are invalid.
2015-09-11 19:44:50 +01:00
Nick Craig-Wood
073d112204
Factor pacer module from Drive and use it in Amazon Cloud Drive for
...
smooth API pacing and retry logic.
2015-09-11 19:18:41 +01:00
Nick Craig-Wood
14f814b806
acd: Retry on 429 errors with backoff and fix upload of 0 length files
2015-09-09 23:23:37 +01:00
Nick Craig-Wood
a288c2b3a3
Make a retry error wrapper for a plain error
2015-09-09 23:22:41 +01:00
Nick Craig-Wood
fec16b0ac8
acd: Skip test on FS which don't support ModifiedTime
2015-09-09 23:21:50 +01:00
Nick Craig-Wood
dd8717797e
Implement --dump-headers and --dump-bodies debug flags
2015-09-08 21:02:48 +01:00
Nick Craig-Wood
7e7c239f09
Through popular demand add a donations page
2015-09-08 21:02:48 +01:00
Nick Craig-Wood
edd0e8abb1
Add Colin Nicholson to contributors
2015-09-08 21:02:48 +01:00
Colin Nicholson
d2b537d9a1
Fix docs link typo for googlecloudstorage - fixes #120
2015-09-08 21:02:48 +01:00
Nick Craig-Wood
8c3df224ef
Implement Amazon Cloud Drive - fixes #45
...
* Optional interfaces Copier, Mover, DirMover not done
2015-09-08 21:02:48 +01:00
Nick Craig-Wood
967fd2a778
dircache: implement FindRootParent and unify locking
2015-09-06 10:27:15 +01:00
Nick Craig-Wood
ea12e446ca
Factor DirCache from drive into its own module
2015-09-05 11:58:54 +01:00
Nick Craig-Wood
c8cd2b510f
More notes
2015-09-02 19:17:33 +01:00
Nick Craig-Wood
8b05a8322b
FAQ entry about bidirectional sync - see #118
2015-09-02 08:33:53 +01:00
Nick Craig-Wood
c98a51b26c
Lightly obscure secrets
2015-09-01 22:33:34 +01:00
Nick Craig-Wood
e2717a031e
Implement Mover and DirMover interfaces fixes #115
...
* unit tests
* local
* drive
* dropbox
2015-09-01 21:49:13 +01:00
Nick Craig-Wood
8d33ce0154
Check for source and dest being the same in sync/copy/move
2015-09-01 21:49:13 +01:00
Nick Craig-Wood
92745aa950
Add Root() to Fs interface
2015-09-01 21:49:13 +01:00
Nick Craig-Wood
cbc6bf6a89
FAQ entry on partial transfers / binary diffs - fixes #113
2015-08-31 12:47:07 +01:00
Nick Craig-Wood
f72575e75f
Remove support for go 1.2 now that go 1.5 is out
2015-08-29 18:58:48 +01:00
