Commit Graph

2902 Commits

Author SHA1 Message Date
Nick Craig-Wood
7d74686698 fs/accounting: increase maximum burst size of token bucket
This stops occasional errors when using --bwlimit which look like this

    Token bucket error: rate: Wait(n=2255475) exceeds limiter's burst 2097152
2018-08-30 17:24:08 +01:00
Sebastian Bünger
2d7c5ebc7a jottacloud: Implement optional about interface. 2018-08-30 17:15:49 +01:00
Sebastian Bünger
86e3436d55 jottacloud: Add optional MimeTyper interface. 2018-08-30 17:15:49 +01:00
Nick Craig-Wood
f243d2a309 Add bsteiss to contributors 2018-08-30 17:08:40 +01:00
bsteiss
aaa3d7e63b s3: add support for KMS Key ID - fixes #2217
This code supports aws:kms and the kms key id for the s3 backend.
2018-08-30 17:08:27 +01:00
Nick Craig-Wood
e4c5f248c0 build: make appveyor just use latest release go version 2018-08-30 16:55:02 +01:00
Nick Craig-Wood
5afaa48d06 Add Denis to contributors 2018-08-30 16:47:32 +01:00
Denis
1c578ced1c cmd: add copyurl command - Fixes #1320 2018-08-30 16:45:41 +01:00
Nick Craig-Wood
de6ec8056f fs/accounting: fix moving average speed for file stats
Before this change the moving average for the individual file stats
would start at 0 and only converge to the correct value over 15-30
seconds.

This change starts the weighting period as 1 and moves it up once per
sample which gets the average to a better value instantly.
2018-08-28 22:55:51 +01:00
Nick Craig-Wood
66fe4a2523 fs/accounting: fix stats display which was missing transferring data
Before this change the total stats were ignoring the in flight
checking, transferring and bytes.
2018-08-28 22:21:17 +01:00
Nick Craig-Wood
f5617dadf3 fs/accounting: factor out eta and percent calculations and write tests 2018-08-28 22:21:17 +01:00
Nick Craig-Wood
5e75a9ef5c build: switch to using go1.11 modules for managing dependencies 2018-08-28 17:08:22 +01:00
Nick Craig-Wood
da1682a30e vendor: switch to using go1.11 modules 2018-08-28 16:08:48 +01:00
Nick Craig-Wood
5c75453aba version: fix test under Windows 2018-08-28 16:07:36 +01:00
Nick Craig-Wood
84ef6b2ae6 Add Alex Chen to contributors 2018-08-27 08:34:57 +01:00
Nick Craig-Wood
7194c358ad azureblob,b2,qingstor,s3,swift: remove leading / from paths - fixes #2484 2018-08-26 23:19:28 +01:00
Nick Craig-Wood
502d8b0cdd vendor: remove github.com/VividCortex/ewma dependency 2018-08-26 22:05:30 +01:00
Nick Craig-Wood
ca44fb1fba accounting: fix time to completion estimates
Previous to this change package used for this
github.com/VividCortex/ewma took a 0 average to mean reset the
statistics.  This happens quite often when transferring files though a
buffer.

Replace that implementation with a simple home grown one (with about
the same constant), without that feature.
2018-08-26 22:00:48 +01:00
Nick Craig-Wood
842ed7d2a9 swift: make it so just storage_url or auth_token can be overidden
Before this change if only one of storage_url or auth_token were
supplied then rclone would overwrite both of them when authenticating.
This effectively meant you could supply both of them or none of them
only.

Now rclone still does the authentication to read the missing
storage_url or auth_token then afterwards re-writes the auth_token or
storage_url back to what the user desired.

Fixes #2464
2018-08-26 21:54:50 +01:00
Nick Craig-Wood
b3217d2cac serve webdav: make Content-Type without reading the file and add --etag-hash
Before this change x/net/webdav would open each file to find out its
Content-Type.

Now we override the FileInfo and provide that directly from rclone.

An --etag-hash has also been implemented to override the ETag with the
hash passed in.

Fixes #2273
2018-08-26 21:50:41 +01:00
Nick Craig-Wood
94950258a4 fs: allow backends to be named using their Name or Prefix #2449
This means that, for example Google Cloud Storage can be known as
`:gcs:bucket` on the command line, as well as `:google cloud
storage:bucket`.
2018-08-26 17:59:31 +01:00
Nick Craig-Wood
8656bd2bb0 fs: Allow on the fly remotes with :backend: syntax - fixes #2449
This change allows remotes to be created on the fly without a config
file by using the remote type prefixed with a : as the remote name, Eg
:s3: to make an s3 remote.

This assumes the user is supplying the backend config via command line
flags or environment variables.
2018-08-26 17:59:31 +01:00
Nick Craig-Wood
174ca22936 mount,cmount: clip the number of blocks to 2^32-1 on macOS
OSX FUSE only supports 32 bit number of blocks which means that block
counts have been wrapping.  This causes f_bavail to be 0 which in turn
causes problems with programs like borg backup.

Fixes #2356
2018-08-26 17:32:59 +01:00
Nick Craig-Wood
4eefd05dcf version: print the release and beta versions with --check - Fixes #2348 2018-08-26 17:28:28 +01:00
Nick Craig-Wood
1cccfa7331 cmd: Make --progress work on Windows 2018-08-26 17:20:38 +01:00
Nick Craig-Wood
18685e6b0b vendor: add github.com/Azure/go-ansiterm for --progress on Windows support 2018-08-26 17:20:38 +01:00
Nick Craig-Wood
b6db90cc32 cmd: add --progress/-P flag to show progress
Fixes #2347
Fixes #1210
2018-08-26 17:20:38 +01:00
Nick Craig-Wood
b596ccdf0f build: update to use go1.11 for the build
Also select latest patch release for go using 1.yy.x feature in travis
2018-08-26 15:26:44 +01:00
Nick Craig-Wood
a64e0922b9 vendor: update github.com/ncw/swift to fix server side copy bug 2018-08-26 15:03:19 +01:00
sandeepkru
3751ceebdd azureblob: Added blob tier feature, a new configuration of azureblob-access-tier
is added to tier blobs between Hot, Cool and Archive. Addresses #2901
2018-08-21 21:52:45 +01:00
Nick Craig-Wood
9f671c5dd0 fs: fix tests for *SepList 2018-08-21 10:58:59 +01:00
Alex Chen
c6c74cb869 mountlib: fix mount --daemon not working with encrypted config - fixes #2473
This passes the configKey to the child process as an Obscured temporary file with an environment variable to the
2018-08-21 09:41:16 +01:00
Nick Craig-Wood
f9cf70e3aa jottacloud: docs, fixes and tests for MD5 calculation
* Add docs for --jottacloud-md5-memory-limit
  * Factor out readMD5 function and add tests
  * Fix accounting
  * Make sure temp file is deleted at the start (not Windows)
2018-08-21 08:57:53 +01:00
Oliver Heyme
ee4485a316 jottacloud: calculate missing MD5s - fixes #2462
If an MD5 can't be found on the source then this streams the object
into memory or on disk to calculate it.
2018-08-21 08:57:53 +01:00
Nick Craig-Wood
455219f501 crypt: fix accounting when checking hashes on upload
In e52ecba295 we forgot to unwrap and
re-wrap the accounting which mean the the accounting was no longer
first in the chain of readers.  This lead to accounting inaccuracies
in remotes which wrap and unwrap the reader again.
2018-08-21 08:57:53 +01:00
Nick Craig-Wood
1b8f4b616c fs: move CommaSepList and SpaceSepList here from config
fs can't import config so having them there means they are not usable
by rclone core.
2018-08-20 17:52:05 +01:00
Fabian Möller
f818df52b8 config: add List type 2018-08-20 17:38:51 +01:00
Cnly
29fa840d3a onedrive: Add back the check for DirMover interface 2018-08-20 17:31:28 +01:00
Nick Craig-Wood
7712a0e111 fs/asyncreader: skip some tests to work around race detector bug
The race detector currently detects a race with len(chan) against
close(chan).

See: https://github.com/golang/go/issues/27070

Skip the tests which trip this bug under the race detector.
2018-08-20 12:34:29 +01:00
Nick Craig-Wood
77806494c8 mount,cmount: adapt to sdnotify API change 2018-08-20 12:34:29 +01:00
Nick Craig-Wood
ff8de59d2b vendor: update minimum number of packages so compile with go1.11 works 2018-08-20 12:34:29 +01:00
Nick Craig-Wood
c19d1ae9a5 build: fix whitespace changes due to go1.11 gofmt changes 2018-08-20 12:26:06 +01:00
Nick Craig-Wood
64ecc2f587 build: use go1.11rc1 to make the beta releases 2018-08-20 12:26:06 +01:00
Nick Craig-Wood
7c911bf2d6 b2: fix app key support on upload to a bucket - fixes #2428 2018-08-18 19:05:32 +01:00
Nick Craig-Wood
41f709e13b yandex: fix listing/deleting files in the root - fixes #2471
Before this change `rclone ls yandex:hello.txt` would fail whereas
`rclone ls yandex:/hello.txt` would succeed.  Now they both succeed.
2018-08-18 12:12:19 +01:00
Nick Craig-Wood
6c5ccf26b1 vendor: update github.com/t3rm1n4l/go-mega to fix failed logins - fixes #2443 2018-08-18 11:46:25 +01:00
Fabian Möller
6dc5aa7454 docs: clearify buffer-size is per transfer/filehandle 2018-08-17 18:11:40 +01:00
Fabian Möller
552eb8e06b vfs: try to seek buffer on read only files 2018-08-17 18:10:28 +01:00
Nick Craig-Wood
7d35b14138 Add Martin Polden to contributors 2018-08-17 18:08:48 +01:00
Martin Polden
6199b95b61 jottacloud: Handle empty time values 2018-08-17 18:08:29 +01:00