19a4d74ee7
backend/s3: Fail fast multipart upload
...
When a part upload request fails error is returned and gCtx is cancelled.
This does not prevent from other parts being tried.
They immediately fail due to a canceled context, but are retried by rclone anyway...
Example AWS debug output
```
-----------------------------------------------------
2020/02/11 14:12:17 DEBUG: Retrying Request s3/UploadPart, attempt 4
2020/02/11 14:12:17 DEBUG: Request s3/UploadPart Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /backuptest-rclone/huge/file.db?partNumber=11&uploadId=190939b4-3c43-4b98-ac11-92303e3f11b0 HTTP/1.1
Host: 192.168.100.99:9000
User-Agent: aws-sdk-go/1.23.8 (go1.13.1; linux; amd64)
Content-Length: 5242880
Authorization: AWS4-HMAC-SHA256 Credential=miniouser/20200211/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-md5;expect;host;x-amz-content-sha256;x-amz-date, Signature=3fc03a01f651cec09b05290459e9ceb26db9a8aa00c4e1b16e8cf5617eb81da8
Content-Md5: XzY+DlipXwbL6bvGYsXftg==
Expect: 100-Continue
X-Amz-Content-Sha256: c036cbb7553a909f8b8877d4461924307f27ecb66cff928eeeafd569c3887e29
X-Amz-Date: 20200211T131217Z
Accept-Encoding: gzip
-----------------------------------------------------
http://192.168.100.99:9000/backuptest-rclone/huge/file.db?partNumber=11&uploadId=190939b4-3c43-4b98-ac11-92303e3f11b0
2020/02/11 14:12:17 DEBUG: Response s3/UploadPart Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 500 InternalServerError
Content-Length: 0
-----------------------------------------------------
UploadPartWithContext() error InternalError: We encountered an internal error. Please try again
status code: 500, request id: , host id:
2020/02/11 14:12:18 DEBUG ERROR: Request s3/UploadPart:
---[ REQUEST DUMP ERROR ]-----------------------------
context canceled
------------------------------------------------------
UploadPartWithContext() error RequestCanceled: request context canceled
caused by: context canceled
2020/02/11 14:12:20 DEBUG ERROR: Request s3/UploadPart:
---[ REQUEST DUMP ERROR ]-----------------------------
context canceled
------------------------------------------------------
UploadPartWithContext() error RequestCanceled: request context canceled
caused by: context canceled
2020/02/11 14:12:22 DEBUG ERROR: Request s3/UploadPart:
---[ REQUEST DUMP ERROR ]-----------------------------
context canceled
------------------------------------------------------
UploadPartWithContext() error RequestCanceled: request context canceled
caused by: context canceled
```
This adds a fail fast behaviour in case the context was cancelled.
2020-02-12 11:40:34 +00:00
55b5eded23
Add Frederick Zhang to contributors
2020-02-12 11:33:56 +00:00
3dbcf0af2d
backend/cache: Remove Unused Functions
...
This removes the unused functions run.writeRemoteRandomBytes() run.writeObjectRandomBytes() run.listPath() Directory.parentRemote() and Persistent.dumpRoot().
2020-02-12 11:23:57 +00:00
4e1a511f88
vfs: explicitly ignore unused variables
2020-02-12 11:20:54 +00:00
b71e1a16b1
docs: update account type during OneDrive app registration
2020-02-12 11:04:49 +00:00
ec1271818f
mount2: hide mount2 command for the moment
2020-02-11 14:28:13 +00:00
8318020387
Implement mount2 with go-fuse
...
This passes the tests and works efficiently with the non sequential vfs ReadAt fix.
2020-02-11 14:28:13 +00:00
c38d7be373
vendor: add github.com/hanwen/go-fuse/v2@master for mount2
2020-02-11 14:28:13 +00:00
dc31212c3d
Add Tim Gallant to contributors
2020-02-11 14:28:13 +00:00
ac60b36e77
backend/premiumizeme: prune unused functions
2020-02-11 12:17:35 +00:00
1d73f071f6
fs: improve log output when no changes are made - fixes #3454
...
- changes a few log messages to debug level
- adds a log message for when 0 bytes are transferred
2020-02-11 12:16:15 +00:00
5c869d5bd3
cmd: make stats be printed on non-zero exit code
...
This also rationalises the exit sequence so that dumping
goroutines/open files happens regardless of exit error state.
See: https://forum.rclone.org/t/transfer-stats-on-non-0-exit/14211
2020-02-10 16:40:43 +00:00
a54210a2e4
docs: restore missing mount --daemon docs
...
This was done as part of ebfeec9fb4
2020-02-10 15:29:39 +00:00
040d226028
docs: restore missing QingStor doc fix
...
This was done as part of 64fce8438b
2020-02-10 15:29:39 +00:00
8b664c3ec5
docs: restore lost spelling fixes
...
These came from 3d424c6e08
2020-02-10 15:29:39 +00:00
102a38bb95
docs: restore lost VFS poll interval docs
...
These came from 3d475dc0ee
2020-02-10 15:29:39 +00:00
7a54e13110
docs: restore lost VFS case insensitive docs
...
These came from 1c4e33d4ad
2020-02-10 15:29:39 +00:00
feee92c790
docs: restore lost mount share docs
...
These came from 162fdfe455
2020-02-10 15:29:39 +00:00
de93852512
docs: restore lost auth proxy logs
...
These came from f2a789ea98
2020-02-10 15:29:39 +00:00
dfb710eab7
gendocs: add autogenerated header to all docs
2020-02-10 15:29:39 +00:00
25cfeb2a64
webdav: Fix X-OC-Mtime header for Transip compatibility - fixes #3126
2020-02-10 11:57:12 +00:00
90377f5e65
s3: Specify that Minio supports URL encoding in listings
...
Thanks to @harshavardhana for pointing this out
See #3934 for background
2020-02-09 12:03:20 +00:00
f1d9bd5eab
lib/oauthutil: replace deprecated oauth2.NoContext
2020-02-07 17:49:29 +00:00
4ee3c21a9d
cmd/serve/ftp: replace deprecated os.SEEK_SET with io.SeekStart
2020-02-06 10:58:34 +00:00
fe6f4135b4
fs/rc: fix dropped error
2020-02-04 11:31:06 +00:00
3dfa63b85c
onedrive: fix occasional 416 errors on multipart uploads
...
Before this change, when uploading multipart files, onedrive would
sometimes return an unexpected 416 error and rclone would abort the
transfer.
This is usually after a 500 error which caused rclone to do a retry.
This change checks the upload position on a 416 error and works how
much of the current chunk to skip, then retries (or skips) the current
chunk as appropriate.
If the position is before the current chunk or after the current chunk
then rclone will abort the transfer.
See: https://forum.rclone.org/t/fragment-overlap-error-with-encrypted-onedrive/14001
Fixes #3131
2020-02-01 21:15:07 +00:00
ff2343475a
docs: Update README.md shields for changed CI
...
Signed-off-by: Gary Kim <gary@garykim.dev >
2020-02-01 20:17:40 +00:00
bffd7f0f14
docs: note how to use GitHub's online editor to edit rclone's docs
2020-02-01 13:44:03 +00:00
7c55fafe33
Add Durval Menezes to contributors
2020-02-01 13:41:47 +00:00
2e7fe06beb
Add Dave Koston to contributors
2020-02-01 13:41:47 +00:00
8ff91ff31b
docs: Update the "Making your own client_id" section for drive
...
...so it accurately describes the new "Enhanced Security" Google process to get your own Client ID and Client Secret to use with rclone.
2020-02-01 13:41:18 +00:00
4d1c616e97
Start v1.51.0-DEV development
2020-02-01 12:32:21 +00:00
43daecd89b
Version v1.51.0
2020-02-01 10:40:01 +00:00
9f99c20232
s3: Add StackPath Object Storage Support
2020-01-31 16:05:44 +00:00
97ed8db75d
drive: hide dangerous config from the configurator
...
This hides:
- "use_created_date"
- "use_shared_date"
- "size_as_quota"
from the configurator (rclone config) as they interfere with normal
operations and shouldn't be set in a backend config.
They can still be put in the config file by hand and will still work
as variables, etc.
This adds some more docs to "size_as_quota" also.
Fixes #3912
2020-01-31 10:09:33 +00:00
f80d98553a
dbhashsum: stop it returning UNSUPPORTED on dropbox
2020-01-29 19:49:42 +00:00
b3e7a9d01c
Add Benjapol Worakan to contributors
2020-01-29 13:38:21 +00:00
01fc063128
docs: fix spacing error under -P in docs.md
2020-01-29 13:38:03 +00:00
e71edd5577
cmd: always print elapsed time to tenth place seconds in progress
...
Before this change, the elapsed time shown with the --progress flag
would not print ".0s" so the elapsed time.
This change will make it so that the line width is kept a bit more
consistent by always printing to a fixed-point.
This does change the displayed value when the elapsed time
is less than 1s, in which it used to be that the value would be shown
in ms or smaller units.
Signed-off-by: Gary Kim <gary@garykim.dev >
2020-01-29 12:28:01 +00:00
27a34dd183
Add Motonori IWAMURO to contributors
2020-01-29 12:16:37 +00:00
7662f15939
onedrive: add support "Retry-After" header
2020-01-29 12:16:18 +00:00
bfd9f32188
lsjson: add --no-mimetype flag, speed up lsf
...
Before this changed we unconditionally fetched the MimeType. On Some
backends like s3 and swift this takes an extra transaction which meant
that `lsf` on those backends was needlessly slow.
This adds an internal option so `lsf` can declare whether it wants
MimeTypes or not depending on whether the user asked for them and an
external flag `--no-mimetype` for `lsjson`.
See: https://forum.rclone.org/t/reliably-setup-incremental-updates/14006/8
2020-01-26 16:38:00 +00:00
9c9cdf1712
fs: don't run tests for --max-duration on remote backends
...
This is a timing dependent test and to make it long enough so that it
would work with the remotes would make it too long for local tests.
The code paths are identical for local vs non-local so just run on
local.
This fixes the integration tests.
2020-01-26 09:23:03 +00:00
0e5537cd25
Add unbelauscht to contributors
2020-01-26 09:23:03 +00:00
151d0a274e
swift: Update OVH API endpoint - Fixes #3890
2020-01-24 17:43:00 +00:00
dc77ec4ba1
Add boosh to contributors
2020-01-24 13:29:36 +00:00
0d7573dd81
fs: Add --max-duration flag to control the maximum duration of a transfer session
...
This gives you more control over how long rclone will run for, making
it easier to script backups, e.g. via cron. Once the `--max-duration`
time limit is reached, no new transfers will be initiated, but those
already in-flight will be allowed to complete.
Fixes #985
2020-01-24 13:28:56 +00:00
e4d2d228bd
webdav: add Referer header to fix problems with WAFs - fixes #3868
2020-01-23 15:56:17 +00:00
ede36b001b
Add Damon Permezel to contributors
2020-01-23 15:40:23 +00:00
3afb2a4798
config: use SpaceSepList for argument to --password-command
...
This is to enable arguments with spaces in.
2020-01-23 15:39:15 +00:00
