Commit Graph

1779 Commits

Author SHA1 Message Date
Nick Craig-Wood
38b85e94ea vfs: rename --cache-* options to --vfs-cache-* to save confusion
..as the backend cache options are all called --cache-* too. Adjust
docs to point out what the vfs cache does vs the backend cache.
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
4b185355df fs: rcat - use in memory object and Copy for more reliable transfers 2017-12-03 12:14:15 +00:00
Nick Craig-Wood
7d15c33e42 fs: make Copy and Move return the destination object if possible 2017-12-03 12:14:15 +00:00
Nick Craig-Wood
11332a19a0 fs: make an in memory object for short transfers 2017-12-03 12:14:15 +00:00
Nick Craig-Wood
a1f8318b29 Add Laurence to contributors 2017-12-03 10:24:53 +00:00
Laurence
e767c9ac9f Fix typo in dbhashsum description 2017-12-03 10:24:33 +00:00
Nick Craig-Wood
56cfb810a8 Add Tim Cooijmans to contributors 2017-12-03 10:22:42 +00:00
Tim Cooijmans
835ca15ec8 drive: add service account support. Fixes #839. 2017-12-03 10:21:41 +00:00
remusb
4af4bbb539 cache: add support for PutStream - fixes #1836 2017-11-30 21:16:45 +02:00
remusb
47450ba326 cache: handle errors when bolt tries to start 2017-11-30 12:27:59 +02:00
Nick Craig-Wood
639e812789 fs: integration tests: add SUMMARY heading for log scraping 2017-11-29 15:55:37 +00:00
Nick Craig-Wood
1c6cad2252 fs: integration tests: add 30 minute timeout per test 2017-11-29 13:51:17 +00:00
Nick Craig-Wood
6d3df6f172 cmount: make tests more reliable on Windows 2017-11-28 20:39:24 +00:00
Nick Craig-Wood
c16ac697a9 vfs: keep track of directories in the cache also #1860
This makes managing empty directories more reliable.
2017-11-28 20:39:23 +00:00
Nick Craig-Wood
0978957a2e vfs: make sure all 96 combinations of flags for Open work 2017-11-28 20:39:23 +00:00
Nick Craig-Wood
d1b19f975d vfs: remove items from cache when deleted #1860
Also fixes Error message when items have been deleted from the cache
(eg when Moved) when the cache reaper comes to delete them.
2017-11-28 16:13:58 +00:00
ishuah
aab8051f50 move: add --delete-empty-src-dirs flag - fixes #1854 2017-11-28 11:38:19 +03:00
Nick Craig-Wood
1248beb0b2 cachestats: Fix nil pointer if not a cache remote - fixes #1855
Also don't retry or show stats
2017-11-24 10:22:23 +00:00
Nick Craig-Wood
6448c445f5 acd: Fix download of large files failing - Fixes #1501
Previously it was necessary to work around large files failing to
download with `--acd-templink-threshold`.  This change makes that flag
obsolete and all files should download.  Templinks may be useful under
some circumstances though the flag isn't being removed.

It does this by filtering `Authorization:` headers out in the
transport if the authorization is supplied in the URL.  This prevents
the "Only one auth mechanism allowed; only the X-Amz-Algorithm query
parameter, Signature query string parameter or the Authorization
header should be specified" error from AWS.
2017-11-24 09:14:25 +00:00
Nick Craig-Wood
fdb01437d8 fs: Allow the http Transport to have an optional filter request function 2017-11-24 09:07:56 +00:00
Nick Craig-Wood
729e1305b7 oauthutil: Allow the http.Client to be passed in 2017-11-24 09:07:03 +00:00
Nick Craig-Wood
02ffd43572 fs: Save the config before asking for a token - fixes #1220
Before this if the client_id/client_secret was edited it would
disappear when asking for the new token.

This means the post config is done after the user has confirmed the
config is OK which can't be helped.
2017-11-23 14:01:32 +00:00
Nick Craig-Wood
e53892f53b fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use
RepeatableReaderSized has a pre-allocated buffer which should help
with memory usage - before it grew the buffer.  Since we know the size
of the chunks, pre-allocating it should be much more efficient.

RepeatableReaderBuffer uses the buffer passed in.

RepeatableLimit* are convenience funcitions for wrapping a reader in
an io.LimitReader and then a RepeatableReader with the same buffer
size.
2017-11-23 13:53:46 +00:00
ishuah
6c62fced60 move: fixed root source directories getting deleted after move - fixes #1849 2017-11-23 12:01:35 +03:00
Nick Craig-Wood
c64ad851af Add David Minor to contributors 2017-11-23 08:57:34 +00:00
David Minor
4c116af1d0 s3: add support for ECS task IAM roles
ECS container IAM metadata is in a different place than EC2 IAM metadata.
Use defaults' RemoteCredProvider function to query the standard locations
for the credentials.

Give the ECS role precedence over the role available from the underlying
EC2 instance.
2017-11-23 08:56:56 +00:00
Nick Craig-Wood
8357a82eee dropbox: change default chunk size to 48MB now we are buffering them in memory 2017-11-22 17:15:37 +00:00
Nick Craig-Wood
483f4b8ad9 dropbox: multiparts uploads retry retry every error after the first chunk is done 2017-11-22 17:15:37 +00:00
Nick Craig-Wood
6f61da5c75 dropbox: buffer the chunks when uploading large files so they can be retried
We use fs.RepeatableReader to buffer the chunks which plays nice with
the accounting.  The default chunk size is 128M which may be too
large.

Fixes #1806
2017-11-22 17:15:37 +00:00
Nick Craig-Wood
159fce0106 fs: fix --cache-dir to have some effect 2017-11-22 17:05:02 +00:00
remusb
569c1a2ec1 cache: catch signal interrupt for bolt handle cleanup 2017-11-22 18:32:36 +02:00
remusb
2497ca5134 cache: add extra logging in Move and Copy 2017-11-22 00:38:25 +02:00
Nick Craig-Wood
cbe5d7ce64 fs: Remove X-Auth-Token: from headers when dumping for swift 2017-11-21 17:32:07 +00:00
Nick Craig-Wood
1a65a4e769 fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters
Now --dump-flag is written as --dump flag. This is a comma separated list which can contain

  * headers - HTTP headers as before
  * bodies  - HTTP bodies as before
  * requests - HTTP request bodies
  * responses - HTTP response bodies
  * auth - HTTP auth
  * filters - Filter rexeps

Leave --dump-headers and --dump-bodies for the time being but remove
the other --dump-* flags as they aren't used very often.
2017-11-21 17:32:07 +00:00
Nick Craig-Wood
bcf1ece43b Update MAINTAINERS with our new maintainer Remus Bunduc 2017-11-21 17:32:07 +00:00
ishuah
b4aa920a3d stats: show the amount of data transferred in kb/mb - fixes #1167 2017-11-21 12:40:02 +03:00
Remus Bunduc
41a97e39c8 cache: fix option help text 2017-11-21 11:25:28 +02:00
Nick Craig-Wood
abbcb2f5e0 cache: disable another unreliable test #1844 2017-11-20 21:25:38 +00:00
Nick Craig-Wood
cb6de4a2cf cache: disable unreliable test #1844 2017-11-20 19:55:00 +00:00
Nick Craig-Wood
dc1c679c65 mount: support truncate properly 2017-11-20 19:42:35 +00:00
Nick Craig-Wood
3fb4fe31d2 vfs: make sure write only handles never truncate files they shouldn't 2017-11-20 19:42:25 +00:00
Nick Craig-Wood
76b151984c vfs: cache the size of the object in the read handle 2017-11-20 17:57:13 +00:00
Nick Craig-Wood
f0ed384786 cache: fix default setting for warmup_age 2017-11-20 14:39:12 +00:00
Nick Craig-Wood
f80f7a0509 cache: use fs.CacheDir to make the default directory for the cache
NB this changes the default dir for the cache
2017-11-20 14:38:28 +00:00
Nick Craig-Wood
af50f31f7d mounttest: wait for Release after every Read to stop using in use files under Windows 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
8e2213fbbd local: add error message for cross file system moves 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
085c690798 build: add in 64bit path for WinFSP headers 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
2b666187a6 cmount: disable tests on windows + race detector
These either hang or produce incorrect results for reasons I haven't
worked out yet.
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
00b46a8b96 mounttest: wait for files to disappear from the directory listing 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
b21f227bd3 mounttest: fix crash when FUSE not present 2017-11-20 12:46:24 +00:00