All user visible Durations should be fs.Duration rather than time.Duration. Suffix is then optional and defaults to s. Additional suffices d, w, M and y are supported, in addition to ms, s, m and h - which are the only ones supported by time.Duration. Absolute times can also be specified, and will be interpreted as duration relative to now.
Before this change, if not using shared key or SAS URL authentication
for the source, rclone gave this error
ManagedIdentityCredential.GetToken() requires exactly one scope
when doing server side copies.
This was introduced in:
3a5ddfcd3c azureblob: implement multipart server side copy
This fixes the problem by creating a temporary SAS URL using user
delegation to read the source blob when copying.
Fixes#8662
The seafile backend used to be able to cope with files called "." and
".." but at some point became unable to do so, causing integration
test failurs.
This adds EncodeDot to the encoding which encodes "." and ".." names.
Linkbox have started issuing 302 redirects on some of their PUT
requests when rclone uploads a file.
This is problematic for several reasons:
1. This is the wrong redirect code - it should be 307 to preserve the method
2. Since Expect/100-Continue isn't supported the whole body gets uploaded
This fixes the problem by first doing a HEAD request on the URL. This
will allow us to read the redirect Location and not upload the body to
the wrong place.
It should still work (albeit a little more inefficiently) if Linkbox
stop redirecting the PUT requests.
See: https://forum.rclone.org/t/linkbox-upload-error/51795Fixes: #8606
This removes
- TestCompressSwift: - never finishes - too slow - we have TestCompressS3 instead
- TestCryptSwift: - never finishes - too slow - we have TestCryptS3 instead
- TestChunkerChunk50bBox: - often times out - covered by other tests
This ocurred whenever there were more than 100 files in the source due
to the output channel filling up.
The fix is not to use list.NewSorter but take more care to output the
dst objects in the same order the src objects are delivered. As the
src objects are delivered sorted, no sorting is needed.
In order not to cause another deadlock, we need to send nil dst
objects which is safe since this adjusts the termination conditions
for the channels.
Thanks to @jeremy for the test script the Go tests are based on.
This commit improves error handling in two specific scenarios:
* Missing Download Links: A 5-second delay is introduced when a download
link is missing, as low-level retries aren't enough. Empirically, it
takes about 30s-1m for the link to become available. This resolves
failed integration tests: backend: TestIntegration/FsMkdir/FsPutFiles/
ObjectUpdate, vfs: TestFileReadAtNonZeroLength
* Unrecoverable 500 Errors: The shouldRetry method is updated to skip
retries for 500 errors from "idx.shub.mypikpak.com" indicating "no
record for gcid." These errors are non-recoverable, so retrying is futile.
This commit introduces a significant rewrite of PikPak's upload, specifically
targeting direct handling of file uploads rather than relying on the generic
S3 manager. The primary motivation is to address critical upload failures
reported in #8629.
* Added new `multipart.go` file for multipart uploads using AWS S3 SDK.
* Removed dependency on AWS S3 manager; replaced with custom handling.
* Updated PikPak test package with new multipart upload tests,
including configurable chunk size and upload cutoff.
* Added new configuration option `upload_cutoff` to control chunked uploads.
* Defined constraints for `chunk_size` and `upload_cutoff` (min/max values,
validation).
* Adjusted default `upload_concurrency` from 5 to 4.
