rclone/backend
Georg Welzel 40111ba5e1
plcoud: fix failing large file uploads - fixes #8147
This changes the OpenWriterAt implementation to make client/fd
handling atomic.

This PR stabilizes the situation of bigger files and multi-threaded
uploads. The root cause boils down to the old "fun" property of
pclouds fileops API: sessions are bound to TCP connections. This
forces us to use a http client with only a single connection
underneath.

With large files, we reuse the same connection for each chunk. If that
connection interrupts (e.g. because we are talking through the
internet), all chunks will fail. The probability for latter one
increases with larger files.

As the point of the whole multi-threaded feature was to speed-up large
files in the first place, this change pulls the client creation (and
hence connection handling) into each chunk. This should stabilize the
situation, as each chunk (and retry) gets its own connection.
2024-12-03 17:52:44 +00:00
..
alias config: make getting config values more consistent 2024-08-08 13:41:31 +01:00
all Add ICloud Drive backend 2024-10-02 10:19:11 +01:00
azureblob azureblob: add --azureblob-use-az to force the use of the Azure CLI for auth 2024-09-20 16:16:09 +01:00
azurefiles build: fix gocritic lint issue elseif 2024-08-15 22:08:34 +01:00
b2 build: ignore remaining gocritic lint issues 2024-08-15 22:08:34 +01:00
box box: fix server-side copying a file over existing dst - fixes #3511 2024-09-29 18:37:52 -04:00
cache build: update logging statements to make json log work - fixes #6038 2024-09-06 17:04:18 +01:00
chunker build: fix gosimple lint errors with golangci-lint v1.60.1 2024-08-14 17:46:12 +01:00
combine fs: define the optional interface SetMetadata and implement it in wrapping backends 2024-05-14 12:51:03 +01:00
compress chunkedreader: add --vfs-read-chunk-streams to parallel read chunks 2024-08-14 21:13:09 +01:00
crypt build: fix gocritic lint issue underef 2024-08-15 22:08:34 +01:00
drive drive: implement rclone backend rescue to rescue orphaned files 2024-10-21 10:15:01 +01:00
dropbox dropbox: fix server side copying over existing object 2024-11-08 18:17:55 +00:00
fichier fichier: fix server side move - fixes #7856 2024-09-06 18:20:10 +01:00
filefabric filefabric: use atomic types 2023-08-22 12:52:13 +01:00
filescom filescom: don't make an extra fetch call on each item in a list response 2024-08-19 22:31:57 +02:00
ftp ftp: implement --ftp-no-check-upload to allow upload to write only dirs 2024-09-18 12:57:01 +01:00
gofile gofile: fix server side copying over existing object 2024-11-08 14:01:51 +00:00
googlecloudstorage gcs: add access token auth with --gcs-access-token 2024-09-27 17:37:07 +01:00
googlephotos gphotos: implment --gphotos-proxy to allow download of full resolution media 2024-09-26 12:57:28 +01:00
hasher fs: define the optional interface SetMetadata and implement it in wrapping backends 2024-05-14 12:51:03 +01:00
hdfs build: fix linting issues reported by running golangci-lint with different GOOS 2024-04-22 19:29:12 +02:00
hidrive backend: fs implements the Shutdowner interface 2023-12-09 11:44:50 +00:00
http backend http: Adding no-escape flag for option to not escape URL metacharacters in path names - fixes issue #7637 2024-04-22 17:57:09 +01:00
iclouddrive Add ICloud Drive backend 2024-10-02 10:19:11 +01:00
imagekit build: fix gocritic lint issue assignop 2024-08-15 22:08:34 +01:00
internetarchive fs: implement --metadata-mapper to transform metatadata with a user supplied program 2023-11-18 17:49:35 +00:00
jottacloud jottacloud: fix setting of metadata on server side move - fixes #7900 2024-09-07 16:00:03 +01:00
koofr backend koofr: remove trailing bracket - fixes #7600 2024-04-04 20:03:26 +01:00
linkbox linkbox: fix list paging and optimized synchronization. 2024-03-24 12:05:58 +00:00
local local: fix permission and ownership on symlinks with --links and --metadata 2024-11-14 16:20:18 +00:00
mailru mailru: attempt to fix throttling by decreasing min sleep to 100ms 2024-06-08 17:44:11 +01:00
mega docs: fix typos found by codespell in docs and code comments 2023-09-23 12:20:01 +01:00
memory memory: fix incorrect list entries when rooted at subdirectory 2024-03-27 11:43:26 -04:00
netstorage build: fix gocritic lint issue valswap 2024-08-15 22:08:34 +01:00
onedrive onedrive: fix integration tests after precision change 2024-11-12 13:09:15 +00:00
opendrive opendrive: add about support to backend 2024-09-17 17:20:42 +01:00
oracleobjectstorage build: fix gocritic lint issue elseif 2024-08-15 22:08:34 +01:00
pcloud plcoud: fix failing large file uploads - fixes #8147 2024-12-03 17:52:44 +00:00
pikpak pikpak: fix fatal crash on startup with token that can't be refreshed 2024-11-08 19:34:09 +00:00
pixeldrain Add pixeldrain backend 2024-08-12 13:35:44 +01:00
premiumizeme backend: fs implements the Shutdowner interface 2023-12-09 11:44:50 +00:00
protondrive backend/protondrive: improve performance of Proton Drive backend 2024-09-18 18:15:24 +01:00
putio putio: fix server side copying over existing object 2024-11-08 18:17:55 +00:00
qingstor build: fix linting issues reported by running golangci-lint with different GOOS 2024-04-22 19:29:12 +02:00
quatrix quatrix: fix f.String() not including subpath 2024-05-13 18:32:41 +01:00
s3 s3: fix multitenant multipart uploads with CEPH 2024-11-21 11:04:49 +00:00
seafile build: fix gosimple lint errors with golangci-lint v1.60.1 2024-08-14 17:46:12 +01:00
sftp serve sftp: update github.com/pkg/sftp to v1.13.7 and fix deadlock in tests 2024-11-11 18:15:00 +00:00
sharefile tree-wide: replace /bin/bash with /usr/bin/env bash 2024-06-11 12:47:47 +01:00
sia rclone config redacted: implement support mechanism for showing redacted config 2023-07-07 16:25:14 +01:00
smb smb: fix shares not listed by updating go-smb2 2023-11-29 15:39:27 +00:00
storj build: fix linting issues reported by running golangci-lint with different GOOS 2024-04-22 19:29:12 +02:00
sugarsync sugarsync: fix server side copying over existing object 2024-11-08 18:17:55 +00:00
swift swift: add total/free space info in about command. 2024-09-06 12:46:51 +01:00
ulozto ulozto: fix upload of > 2GB files on 32 bit platforms - fixes #7960 2024-07-20 11:29:34 +01:00
union build: fix gocritic lint issue assignop 2024-08-15 22:08:34 +01:00
uptobox rclone config redacted: implement support mechanism for showing redacted config 2023-07-07 16:25:14 +01:00
webdav webdav: nextcloud: implement backoff and retry for 423 LOCKED errors 2024-09-17 16:46:02 +01:00
yandex yandex: fix server side copying over existing object 2024-11-08 18:17:55 +00:00
zoho zoho: make upload cutoff configurable 2024-09-17 20:40:42 +01:00