rclone/backend
Nick Craig-Wood 7e6fac8b1e s3: re-implement multipart upload to fix memory issues
There have been quite a few reports of problems with the multipart
uploader using too much memory and not retrying possible errors.

Before this change the multipart uploader used the s3manager
abstraction in the AWS SDK.  There are numerous bug reports of this
using up too much memory.

This change re-implements a much simplified version of the s3manager
code specialized for rclone's purposes.

This should use much less memory and retry chunks properly.

See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563
See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405
See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2020-01-03 22:19:28 +00:00
..
alias build: fix up package paths after repo move 2019-07-28 18:47:38 +01:00
all New backend for Citrix Sharefile - Fixes #1543 2019-09-30 12:28:33 +01:00
amazonclouddrive amazonclouddrive: use lib/encoder 2019-09-30 22:00:24 +01:00
azureblob azureblob: disable logging to the Windows event log 2019-10-21 11:50:31 +01:00
b2 b2: remove unverified: prefix on sha1 - fixes #3654 2019-10-23 08:41:56 +01:00
box box: use lib/encoder 2019-09-30 14:05:49 +01:00
cache cache: follow move of upstream library github.com/coreos/bbolt github.com/etcd-io/bbolt 2019-11-14 21:51:34 +00:00
chunker chunker: reduce length of temporary suffix 2019-12-09 16:56:32 +00:00
crypt config: give config questions default values - fixes #3672 2019-11-05 11:53:44 +00:00
drive drive: make sure invalid auth for teamdrives always reports an error 2019-11-28 10:51:17 +00:00
dropbox hash: fix hash names for DropboxHash and CRC-32 2019-10-30 12:20:10 +00:00
fichier fichier: fix accessing files > 2GB on 32 bit systems - fixes #3581 2019-10-01 16:03:49 +01:00
ftp ftp: allow disabling EPSV mode 2019-10-10 21:00:41 +01:00
googlecloudstorage googlecloudstorage: use lib/encoder 2019-09-30 22:00:24 +01:00
googlephotos rest: add context propagation to rest library #3257 2019-09-09 23:27:07 +01:00
http http: fix race introduced in 7982aaf151 2019-09-14 08:48:13 +01:00
hubic Add context to all http.NewRequest #3257 2019-09-09 23:27:07 +01:00
jottacloud jottacloud: add support whitelabel versions 2020-01-02 15:37:33 +01:00
koofr koofr: use rclone HTTP client. 2019-11-14 11:36:44 +00:00
local local: make source file being updated errors be NoLowLevelRetry errors #3777 2019-12-06 10:54:03 +00:00
mailru mailru: comment out some debugging statements 2019-10-02 20:10:01 +01:00
mega mega: fix after lib/encoder change 2019-10-02 12:41:52 +01:00
onedrive onedrive: no trailing slash reading metadata... 2019-10-29 13:33:11 +00:00
opendrive opendrive: use lib/encoder 2019-09-30 14:05:49 +01:00
pcloud pcloud: use lib/encoder 2019-09-30 22:00:24 +01:00
premiumizeme premiumizeme: use lib/encoder 2019-10-02 11:34:08 +01:00
putio putio: add ability to resume uploads 2019-10-14 20:01:16 +01:00
qingstor vendor: update all dependencies 2019-11-14 21:51:34 +00:00
s3 s3: re-implement multipart upload to fix memory issues 2020-01-03 22:19:28 +00:00
sftp sftp: add --sftp-skip-links to skip symlinks and non regular files - fixes #3716 2019-11-24 16:10:53 +00:00
sharefile sharefile: use lib/encoder 2019-09-30 22:00:25 +01:00
swift swift: fix parsing of X-Object-Manifest 2019-11-21 13:25:02 +00:00
union docs: make the config walkthroughs consistent for each backend 2019-08-26 10:47:17 +01:00
webdav webdav: fix case of "Bearer" in Authorization: header to agree with RFC 2019-11-27 12:04:31 +00:00
yandex yandex: use lib/encoder 2019-10-05 10:22:43 +01:00