Nick Craig-Wood
fdb01437d8
fs: Allow the http Transport to have an optional filter request function
2017-11-24 09:07:56 +00:00
Nick Craig-Wood
729e1305b7
oauthutil: Allow the http.Client to be passed in
2017-11-24 09:07:03 +00:00
Nick Craig-Wood
02ffd43572
fs: Save the config before asking for a token - fixes #1220
...
Before this if the client_id/client_secret was edited it would
disappear when asking for the new token.
This means the post config is done after the user has confirmed the
config is OK which can't be helped.
2017-11-23 14:01:32 +00:00
Nick Craig-Wood
e53892f53b
fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use
...
RepeatableReaderSized has a pre-allocated buffer which should help
with memory usage - before it grew the buffer. Since we know the size
of the chunks, pre-allocating it should be much more efficient.
RepeatableReaderBuffer uses the buffer passed in.
RepeatableLimit* are convenience funcitions for wrapping a reader in
an io.LimitReader and then a RepeatableReader with the same buffer
size.
2017-11-23 13:53:46 +00:00
ishuah
6c62fced60
move: fixed root source directories getting deleted after move - fixes #1849
2017-11-23 12:01:35 +03:00
Nick Craig-Wood
c64ad851af
Add David Minor to contributors
2017-11-23 08:57:34 +00:00
David Minor
4c116af1d0
s3: add support for ECS task IAM roles
...
ECS container IAM metadata is in a different place than EC2 IAM metadata.
Use defaults' RemoteCredProvider function to query the standard locations
for the credentials.
Give the ECS role precedence over the role available from the underlying
EC2 instance.
2017-11-23 08:56:56 +00:00
Nick Craig-Wood
8357a82eee
dropbox: change default chunk size to 48MB now we are buffering them in memory
2017-11-22 17:15:37 +00:00
Nick Craig-Wood
483f4b8ad9
dropbox: multiparts uploads retry retry every error after the first chunk is done
2017-11-22 17:15:37 +00:00
Nick Craig-Wood
6f61da5c75
dropbox: buffer the chunks when uploading large files so they can be retried
...
We use fs.RepeatableReader to buffer the chunks which plays nice with
the accounting. The default chunk size is 128M which may be too
large.
Fixes #1806
2017-11-22 17:15:37 +00:00
Nick Craig-Wood
159fce0106
fs: fix --cache-dir to have some effect
2017-11-22 17:05:02 +00:00
remusb
569c1a2ec1
cache: catch signal interrupt for bolt handle cleanup
2017-11-22 18:32:36 +02:00
remusb
2497ca5134
cache: add extra logging in Move and Copy
2017-11-22 00:38:25 +02:00
Nick Craig-Wood
cbe5d7ce64
fs: Remove X-Auth-Token: from headers when dumping for swift
2017-11-21 17:32:07 +00:00
Nick Craig-Wood
1a65a4e769
fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters
...
Now --dump-flag is written as --dump flag. This is a comma separated list which can contain
* headers - HTTP headers as before
* bodies - HTTP bodies as before
* requests - HTTP request bodies
* responses - HTTP response bodies
* auth - HTTP auth
* filters - Filter rexeps
Leave --dump-headers and --dump-bodies for the time being but remove
the other --dump-* flags as they aren't used very often.
2017-11-21 17:32:07 +00:00
Nick Craig-Wood
bcf1ece43b
Update MAINTAINERS with our new maintainer Remus Bunduc
2017-11-21 17:32:07 +00:00
ishuah
b4aa920a3d
stats: show the amount of data transferred in kb/mb - fixes #1167
2017-11-21 12:40:02 +03:00
Remus Bunduc
41a97e39c8
cache: fix option help text
2017-11-21 11:25:28 +02:00
Nick Craig-Wood
abbcb2f5e0
cache: disable another unreliable test #1844
2017-11-20 21:25:38 +00:00
Nick Craig-Wood
cb6de4a2cf
cache: disable unreliable test #1844
2017-11-20 19:55:00 +00:00
Nick Craig-Wood
dc1c679c65
mount: support truncate properly
2017-11-20 19:42:35 +00:00
Nick Craig-Wood
3fb4fe31d2
vfs: make sure write only handles never truncate files they shouldn't
2017-11-20 19:42:25 +00:00
Nick Craig-Wood
76b151984c
vfs: cache the size of the object in the read handle
2017-11-20 17:57:13 +00:00
Nick Craig-Wood
f0ed384786
cache: fix default setting for warmup_age
2017-11-20 14:39:12 +00:00
Nick Craig-Wood
f80f7a0509
cache: use fs.CacheDir to make the default directory for the cache
...
NB this changes the default dir for the cache
2017-11-20 14:38:28 +00:00
Nick Craig-Wood
af50f31f7d
mounttest: wait for Release after every Read to stop using in use files under Windows
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
8e2213fbbd
local: add error message for cross file system moves
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
085c690798
build: add in 64bit path for WinFSP headers
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
2b666187a6
cmount: disable tests on windows + race detector
...
These either hang or produce incorrect results for reasons I haven't
worked out yet.
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
00b46a8b96
mounttest: wait for files to disappear from the directory listing
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
b21f227bd3
mounttest: fix crash when FUSE not present
2017-11-20 12:46:24 +00:00
Nick Craig-Wood
e98e550021
mounttest: wait for all background Close/Release after writing a file
...
The filesystem does a certain amount of things asynchronously waiting
for the file to be released after writing it means everything should
be in a consistent state.
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
60945d0a37
vfs: remove misleading comment
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
b4083b4371
vfs: rename Fsync to Sync and implement Sync on Node and Handle
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
eb3415db50
cmount: enable more tests for Windows
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
9fbd8a6419
mounttest: fixes for running under Windows
...
* don't mount and unmount between cache runs - WinFSP doesn't suport it
* use OS paths for opening things
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
9738f8532b
vfs: Add FlushDirCache method
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
a5b034a992
vfs: add WaitForWriters to wait until all writers have finished
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
321b6da7af
vfs: don't remove file from writers until it is transferred
...
This means that the list of active writers is up to date
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
1b22ee5b93
vfs: fix error handling in openPending so it returns the correct error
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
eab55ce882
vfs: add open files to directories
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
61b6159a05
mount, cmount: add O_CREATE to Open calls since fuse doesn't seem to supply it
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
c560017934
vfs: add Path method to Node and use it to stop reading nil DirEntry
...
All DirEntry calls now have been checked for nil or converted to use Path.
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
7c3584f4e6
mountlib: wait for mountpoint to disappear under Windows
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
981cfb1bec
mounttest: retry directory listings to account for slow updates on Windows
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
992647b157
vfs: Don't error a r/w file open without cache; delay error until Read called
...
If we open a file for r/w without the cache we now always return a
handle and return an error if the file is ever read from. This fixes
incompatibility with cmount under windows.
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
dec21ccf63
vfs, cmount: make truncate work properly in the presence or otherwise of open files
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
94adf4f43b
cmount: translate FUSE open flags into OS flags
...
On Windows the fuse.O_* flags do not have the same values as the
os.O_* flags so translate between the two representations. They are
mostly the same which is why this hasn't caused a problem before.
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
e7f2935333
vfs: decode flags in Open/OpenFile for debug
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
f5f8c0c438
cmount: make Truncate call the correct Handle or Node method
2017-11-20 12:46:22 +00:00
