extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-11-30 04:15:26 +01:00
Code Issues Packages Projects Releases Wiki Activity
2a6675cffd
rclone/backend/s3/s3.go

6186 lines
187 KiB
Go
Raw Normal View History

backend: fix typos found by codespell
2022-09-20 08:54:47 +02:00
// Package s3 provides an interface to Amazon S3 object storage
Refactor into sub filesystems
2013-06-27 21:13:07 +02:00
package s3
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
s3: refactor to use generated code instead of reflection to copy structs
2022-07-28 12:49:19 +02:00
//go:generate go run gen_setfrom.go -o setfrom.go
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
import (
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
"bytes"
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
"context"
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
"crypto/md5"
s3: Added `--s3-disable-http2` to disable http/2 Fixes #4673 Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-13 18:11:22 +02:00
"crypto/tls"
s3: set/get the hash for multipart files - #523
2018-01-06 15:30:10 +01:00
"encoding/base64"
"encoding/hex"
s3: add IDrive e2 to provider list
2022-06-18 09:29:21 +02:00
"encoding/json"
s3: only ask for URL encoded directory listings if we need them on Ceph This works around a bug in Ceph which doesn't encode CommonPrefixes when using URL encoded directory listings. See: https://tracker.ceph.com/issues/41870
2019-09-16 21:25:55 +02:00
"encoding/xml"
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
"errors"
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
"fmt"
"io"
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
"net/http"
s3: fix encoding for control characters - Fixes #3345
2019-07-23 13:24:10 +02:00
"net/url"
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
"path"
"regexp"
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
"sort"
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
"strconv"
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
"strings"
s3: Reduce memory usage streaming files by reducing max stream upload size Before this change rclone would allow the user to stream (eg with rclone mount, rclone rcat or uploading google photos or docs) 5TB files. This meant that rclone allocated 4 * 525 MB buffers per transfer which is way too much memory by default. This change makes rclone use the configured chunk size for streamed uploads. This is 5MB by default which means that rclone can stream upload files up to 48GB by default staying below the 10,000 chunks limit. This can be increased with --s3-chunk-size if necessary. If rclone detects that a file is being streamed to s3 it will make a single NOTICE level log stating the limitation. This fixes the enormous memory usage. Fixes #3568 See: https://forum.rclone.org/t/how-much-memory-does-rclone-need/12743
2019-11-06 11:41:03 +01:00
"sync"
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
"time"
s3: use different strategy to resolve s3 region The API endpoint GetBucketLocation requires top level permission. If we do an authenticated head request to a bucket, the bucket location will be returned in the HTTP headers. Fixes #5066
2022-10-29 09:37:08 +02:00
"github.com/aws/aws-sdk-go/service/s3/s3manager"
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
s3: Fix after upstream API changes in aws-sdk-go/aws - fixes #114
2015-08-28 09:47:41 +02:00
"github.com/aws/aws-sdk-go/aws/corehandlers"
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
"github.com/aws/aws-sdk-go/aws/credentials"
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
"github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
s3: add support for ECS task IAM roles ECS container IAM metadata is in a different place than EC2 IAM metadata. Use defaults' RemoteCredProvider function to query the standard locations for the credentials. Give the ECS role precedence over the role available from the underlying EC2 instance.
2017-11-22 22:21:36 +01:00
"github.com/aws/aws-sdk-go/aws/defaults"
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
"github.com/aws/aws-sdk-go/aws/ec2metadata"
s3: Use regional s3 us-east-1 endpoint
2020-06-26 15:09:29 +02:00
"github.com/aws/aws-sdk-go/aws/endpoints"
s3: Fix after upstream API changes in aws-sdk-go/aws - fixes #114
2015-08-28 09:47:41 +02:00
"github.com/aws/aws-sdk-go/aws/request"
Update to AWS SDK 0.10.0 Tested with S3 and Dreamhost Here is a link to the release notes: http://aws.amazon.com/releasenotes/5476699172355228
2015-10-30 12:50:45 +01:00
"github.com/aws/aws-sdk-go/aws/session"
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
"github.com/aws/aws-sdk-go/service/s3"
swift: update github.com/ncw/swift to v2.0.0 The update to v2 of the swift library introduces a context parameter to each function. This required a lot of mostly mechanical changes adding context parameters. See: https://github.com/ncw/swift/issues/159 See: https://github.com/ncw/swift/issues/161
2021-01-22 18:23:51 +01:00
"github.com/ncw/swift/v2"
build: fix up package paths after repo move git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g' goimports -w `find . -name \*.go`
2019-07-28 19:47:38 +02:00
"github.com/rclone/rclone/fs"
s3: implement Purge to purge versions and `backend cleanup-hidden`
2022-07-26 16:03:32 +02:00
"github.com/rclone/rclone/fs/accounting"
s3: use chunksize lib to determine chunksize dynamically
2022-05-06 21:25:44 +02:00
"github.com/rclone/rclone/fs/chunksize"
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
"github.com/rclone/rclone/fs/config"
build: fix up package paths after repo move git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g' goimports -w `find . -name \*.go`
2019-07-28 19:47:38 +02:00
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/config/configstruct"
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/hash"
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
"github.com/rclone/rclone/fs/operations"
build: fix up package paths after repo move git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g' goimports -w `find . -name \*.go`
2019-07-28 19:47:38 +02:00
"github.com/rclone/rclone/fs/walk"
s3: cancel in progress multipart uploads and copies on rclone exit #4300
2020-06-04 12:09:27 +02:00
"github.com/rclone/rclone/lib/atexit"
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
"github.com/rclone/rclone/lib/bucket"
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
"github.com/rclone/rclone/lib/encoder"
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
"github.com/rclone/rclone/lib/multipart"
build: fix up package paths after repo move git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g' goimports -w `find . -name \*.go`
2019-07-28 19:47:38 +02:00
"github.com/rclone/rclone/lib/pacer"
s3: fix accounting for multpart uploads
2023-08-24 18:15:18 +02:00
"github.com/rclone/rclone/lib/pool"
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
"github.com/rclone/rclone/lib/readers"
build: fix up package paths after repo move git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g' goimports -w `find . -name \*.go`
2019-07-28 19:47:38 +02:00
"github.com/rclone/rclone/lib/rest"
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
"github.com/rclone/rclone/lib/version"
s3: drop binary metadata with an ERROR message Before this change, rclone would attempt to upload metadata with binary contents which fail to be uploaded by net/http. This checks the keys and values for validity as http header values before uploading. See: https://forum.rclone.org/t/invalid-metadata-key-names-result-in-a-failure-to-transfer-xattr-results-in-failure-to-upload-net-http-invalid-header-field-value-for-x-amz-meta-samba-pai/33406/
2022-10-11 18:54:22 +02:00
"golang.org/x/net/http/httpguts"
Remove filesystem flags and put in config file with editor
2014-03-15 17:06:11 +01:00
)
Refactor into sub filesystems
2013-06-27 21:13:07 +02:00
// Register with Fs
func init() {
Create separate interface for object information. Take out read-only information about a Fs in a separate struct to limit access. See discussion at #282.
2016-02-18 12:35:25 +01:00
fs.Register(&fs.RegInfo{
Add help to remote chooser in rclone config - fixes #43
2016-02-15 19:11:53 +01:00
Name: "s3",
s3: add Leviia S3 Object Storage as provider
2023-07-16 19:08:47 +02:00
Description: "Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Leviia, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi",
Add help to remote chooser in rclone config - fixes #43
2016-02-15 19:11:53 +01:00
NewFs: NewFs,
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
CommandHelp: commandHelp,
s3: add IDrive e2 to provider list
2022-06-18 09:29:21 +02:00
Config: func(ctx context.Context, name string, m configmap.Mapper, config fs.ConfigIn) (*fs.ConfigOut, error) {
switch config.State {
case "":
return nil, setEndpointValueForIDriveE2(m)
}
return nil, fmt.Errorf("unknown state %q", config.State)
},
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
MetadataInfo: &fs.MetadataInfo{
System: systemMetadataInfo,
Help: `User metadata is stored as x-amz-meta- keys. S3 metadata keys are case insensitive and are always returned in lower case.`,
},
Remove filesystem flags and put in config file with editor
2014-03-15 17:06:11 +01:00
Options: []fs.Option{{
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Name: fs.ConfigProvider,
Help: "Choose your S3 provider.",
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
// NB if you add a new provider here, then add it in the
// setQuirks function and set the correct quirks
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Examples: []fs.OptionExample{{
Value: "AWS",
Help: "Amazon Web Services (AWS) S3",
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
}, {
Value: "Alibaba",
Help: "Alibaba Cloud Object Storage System (OSS) formerly Aliyun",
s3: Fix Arvancloud Domain and region changes and alphabetise the provider
2023-06-17 13:47:23 +02:00
}, {
Value: "ArvanCloud",
Help: "Arvan Cloud Object Storage (AOS)",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "Ceph",
Help: "Ceph Object Storage",
s3: Add ChinaMobile EOS to provider list China Mobile Ecloud Elastic Object Storage (EOS) is a cloud object storage service, and is fully compatible with S3. Fixes #6054
2022-03-24 12:57:00 +01:00
}, {
Value: "ChinaMobile",
Help: "China Mobile Ecloud Elastic Object Storage (EOS)",
s3: Support Cloudflare R2 - fixes #5642
2022-05-06 13:09:29 +02:00
}, {
Value: "Cloudflare",
Help: "Cloudflare R2 Storage",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "DigitalOcean",
s3: add DigitalOcean Spaces regions sfo3, fra1, syd1
2022-12-14 15:00:04 +01:00
Help: "DigitalOcean Spaces",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "Dreamhost",
Help: "Dreamhost DreamObjects",
s3: add GCS to provider list
2023-01-09 07:54:51 +01:00
}, {
Value: "GCS",
Help: "Google Cloud Storage",
backend: add Huawei OBS to s3 provider list
2022-06-07 09:41:46 +02:00
}, {
Value: "HuaweiOBS",
Help: "Huawei Object Storage Service",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "IBMCOS",
Help: "IBM COS S3",
s3: add IDrive e2 to provider list
2022-06-18 09:29:21 +02:00
}, {
Value: "IDrive",
Help: "IDrive e2",
s3: add support for IONOS Cloud Storage
2022-09-15 17:04:34 +02:00
}, {
Value: "IONOS",
Help: "IONOS Cloud",
s3: add support for Seagate Lyve Cloud storage
2022-03-04 13:35:46 +01:00
}, {
s3: add other regions for Lyve and correct Provider name
2022-03-14 16:37:34 +01:00
Value: "LyveCloud",
s3: add support for Seagate Lyve Cloud storage
2022-03-04 13:35:46 +01:00
Help: "Seagate Lyve Cloud",
s3: add Leviia S3 Object Storage as provider
2023-07-16 19:08:47 +02:00
}, {
Value: "Leviia",
Help: "Leviia Object Storage",
s3: add Liara LOS to provider list
2022-12-06 13:25:23 +01:00
}, {
Value: "Liara",
Help: "Liara Object Storage",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "Minio",
Help: "Minio Object Storage",
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
}, {
Value: "Netease",
Help: "Netease Object Storage (NOS)",
s3: add petabox.io to s3 providers
2023-05-05 10:44:25 +02:00
}, {
Value: "Petabox",
Help: "Petabox Object Storage",
s3: Add RackCorp object storage to providers
2021-11-09 12:46:58 +01:00
}, {
Value: "RackCorp",
Help: "RackCorp Object Storage",
s3: add Scaleway provider - fixes #4338
2020-06-12 17:04:16 +02:00
}, {
Value: "Scaleway",
Help: "Scaleway Object Storage",
s3: Add SeaweedFS
2021-06-08 10:59:57 +02:00
}, {
Value: "SeaweedFS",
Help: "SeaweedFS S3",
s3: Add StackPath Object Storage Support
2020-01-31 00:21:24 +01:00
}, {
Value: "StackPath",
Help: "StackPath Object Storage",
s3: add specific provider for Storj Shared gateways - unsupported features (Copy) are turned off for Storj - enable urlEncodedListing for Storj provider - set chunksize to 64Mb
2021-09-29 15:43:06 +02:00
}, {
Value: "Storj",
Help: "Storj (S3 Compatible Gateway)",
s3: add synology to s3 provider list
2023-07-06 11:54:07 +02:00
}, {
Value: "Synology",
Help: "Synology C2 Object Storage",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "TencentCOS",
Help: "Tencent Cloud Object Storage (COS)",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "Wasabi",
Help: "Wasabi Object Storage",
s3: add Qiniu KODO to s3 provider list - fixes #6195
2022-10-13 16:49:22 +02:00
}, {
Value: "Qiniu",
Help: "Qiniu Object Storage (Kodo)",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "Other",
Help: "Any other S3 compatible provider",
}},
}, {
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
Name: "env_auth",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).\n\nOnly applies if access_key_id and secret_access_key is blank.",
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
Default: false,
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Examples: []fs.OptionExample{{
Value: "false",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Enter AWS credentials in the next step.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "true",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Get AWS credentials from the environment (env vars or IAM).",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}},
}, {
rclone config redacted: implement support mechanism for showing redacted config This introduces a new fs.Option flag, Sensitive and uses this along with IsPassword to redact the info in the config file for support purposes. It adds this flag into backends where appropriate. It was necessary to add oauthutil.SharedOptions to some backends as they were missing them. Fixes #5209
2023-07-06 18:55:53 +02:00
Name: "access_key_id",
Help: "AWS Access Key ID.\n\nLeave blank for anonymous access or runtime credentials.",
Sensitive: true,
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
rclone config redacted: implement support mechanism for showing redacted config This introduces a new fs.Option flag, Sensitive and uses this along with IsPassword to redact the info in the config file for support purposes. It adds this flag into backends where appropriate. It was necessary to add oauthutil.SharedOptions to some backends as they were missing them. Fixes #5209
2023-07-06 18:55:53 +02:00
Name: "secret_access_key",
Help: "AWS Secret Access Key (password).\n\nLeave blank for anonymous access or runtime credentials.",
Sensitive: true,
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
// References:
// 1. https://docs.aws.amazon.com/general/latest/gr/rande.html
// 2. https://docs.aws.amazon.com/general/latest/gr/s3.html
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Name: "region",
Help: "Region to connect to.",
Provider: "AWS",
Examples: []fs.OptionExample{{
Value: "us-east-1",
docs: grammar: Oxford comma Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-13 23:43:00 +02:00
Help: "The default endpoint - a good choice if you are unsure.\nUS Region, Northern Virginia, or Pacific Northwest.\nLeave location constraint empty.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "us-east-2",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "US East (Ohio) Region.\nNeeds location constraint us-east-2.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "us-west-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "US West (Northern California) Region.\nNeeds location constraint us-west-1.",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "us-west-2",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "US West (Oregon) Region.\nNeeds location constraint us-west-2.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ca-central-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Canada (Central) Region.\nNeeds location constraint ca-central-1.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "eu-west-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU (Ireland) Region.\nNeeds location constraint EU or eu-west-1.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "eu-west-2",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU (London) Region.\nNeeds location constraint eu-west-2.",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "eu-west-3",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU (Paris) Region.\nNeeds location constraint eu-west-3.",
s3: add aws endpoint eu-north-1
2019-02-02 20:44:26 +01:00
}, {
Value: "eu-north-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU (Stockholm) Region.\nNeeds location constraint eu-north-1.",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "eu-south-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU (Milan) Region.\nNeeds location constraint eu-south-1.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "eu-central-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU (Frankfurt) Region.\nNeeds location constraint eu-central-1.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ap-southeast-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Singapore) Region.\nNeeds location constraint ap-southeast-1.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ap-southeast-2",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Sydney) Region.\nNeeds location constraint ap-southeast-2.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ap-northeast-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Tokyo) Region.\nNeeds location constraint ap-northeast-1.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ap-northeast-2",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Seoul).\nNeeds location constraint ap-northeast-2.",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "ap-northeast-3",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Osaka-Local).\nNeeds location constraint ap-northeast-3.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ap-south-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Mumbai).\nNeeds location constraint ap-south-1.",
s3: Add new region Asia Patific (Hong Kong)
2020-01-02 12:10:48 +01:00
}, {
Value: "ap-east-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Hong Kong) Region.\nNeeds location constraint ap-east-1.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "sa-east-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "South America (Sao Paulo) Region.\nNeeds location constraint sa-east-1.",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "me-south-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Middle East (Bahrain) Region.\nNeeds location constraint me-south-1.",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "af-south-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Africa (Cape Town) Region.\nNeeds location constraint af-south-1.",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "cn-north-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "China (Beijing) Region.\nNeeds location constraint cn-north-1.",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "cn-northwest-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "China (Ningxia) Region.\nNeeds location constraint cn-northwest-1.",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "us-gov-east-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "AWS GovCloud (US-East) Region.\nNeeds location constraint us-gov-east-1.",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "us-gov-west-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "AWS GovCloud (US) Region.\nNeeds location constraint us-gov-west-1.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}},
s3: Add RackCorp object storage to providers
2021-11-09 12:46:58 +01:00
}, {
Name: "region",
Help: "region - the location where your bucket will be created and your data stored.\n",
Provider: "RackCorp",
Examples: []fs.OptionExample{{
Value: "global",
Help: "Global CDN (All locations) Region",
}, {
Value: "au",
Help: "Australia (All states)",
}, {
Value: "au-nsw",
Help: "NSW (Australia) Region",
}, {
Value: "au-qld",
Help: "QLD (Australia) Region",
}, {
Value: "au-vic",
Help: "VIC (Australia) Region",
}, {
Value: "au-wa",
Help: "Perth (Australia) Region",
}, {
Value: "ph",
Help: "Manila (Philippines) Region",
}, {
Value: "th",
Help: "Bangkok (Thailand) Region",
}, {
Value: "hk",
Help: "HK (Hong Kong) Region",
}, {
Value: "mn",
Help: "Ulaanbaatar (Mongolia) Region",
}, {
Value: "kg",
Help: "Bishkek (Kyrgyzstan) Region",
}, {
Value: "id",
Help: "Jakarta (Indonesia) Region",
}, {
Value: "jp",
Help: "Tokyo (Japan) Region",
}, {
Value: "sg",
Help: "SG (Singapore) Region",
}, {
Value: "de",
Help: "Frankfurt (Germany) Region",
}, {
Value: "us",
Help: "USA (AnyCast) Region",
}, {
Value: "us-east-1",
Help: "New York (USA) Region",
}, {
Value: "us-west-1",
Help: "Freemont (USA) Region",
}, {
Value: "nz",
Help: "Auckland (New Zealand) Region",
}},
s3: add Scaleway provider - fixes #4338
2020-06-12 17:04:16 +02:00
}, {
Name: "region",
Help: "Region to connect to.",
Provider: "Scaleway",
Examples: []fs.OptionExample{{
Value: "nl-ams",
Help: "Amsterdam, The Netherlands",
}, {
Value: "fr-par",
Help: "Paris, France",
s3: Add Warsaw location for Scaleway Add new location in Warsaw (Poland) to endpoints for Scaleway. More Information: https://blog.scaleway.com/scaleway-is-now-in-warsaw/ https://www.scaleway.com/en/docs/storage/object/how-to/create-a-bucket/
2022-05-17 15:24:32 +02:00
}, {
Value: "pl-waw",
Help: "Warsaw, Poland",
s3: add Scaleway provider - fixes #4338
2020-06-12 17:04:16 +02:00
}},
backend: add Huawei OBS to s3 provider list
2022-06-07 09:41:46 +02:00
}, {
Name: "region",
Help: "Region to connect to. - the location where your bucket will be created and your data stored. Need bo be same with your endpoint.\n",
Provider: "HuaweiOBS",
Examples: []fs.OptionExample{{
Value: "af-south-1",
Help: "AF-Johannesburg",
}, {
Value: "ap-southeast-2",
Help: "AP-Bangkok",
}, {
Value: "ap-southeast-3",
Help: "AP-Singapore",
}, {
Value: "cn-east-3",
Help: "CN East-Shanghai1",
}, {
Value: "cn-east-2",
Help: "CN East-Shanghai2",
}, {
Value: "cn-north-1",
Help: "CN North-Beijing1",
}, {
Value: "cn-north-4",
Help: "CN North-Beijing4",
}, {
Value: "cn-south-1",
Help: "CN South-Guangzhou",
}, {
Value: "ap-southeast-1",
Help: "CN-Hong Kong",
}, {
Value: "sa-argentina-1",
Help: "LA-Buenos Aires1",
}, {
Value: "sa-peru-1",
Help: "LA-Lima1",
}, {
Value: "na-mexico-1",
Help: "LA-Mexico City1",
}, {
Value: "sa-chile-1",
Help: "LA-Santiago2",
}, {
Value: "sa-brazil-1",
Help: "LA-Sao Paulo1",
}, {
Value: "ru-northwest-2",
Help: "RU-Moscow2",
}},
s3: Support Cloudflare R2 - fixes #5642
2022-05-06 13:09:29 +02:00
}, {
Name: "region",
Help: "Region to connect to.",
Provider: "Cloudflare",
Examples: []fs.OptionExample{{
Value: "auto",
Help: "R2 buckets are automatically distributed across Cloudflare's data centers for low latency.",
}},
s3: add Qiniu KODO to s3 provider list - fixes #6195
2022-10-13 16:49:22 +02:00
}, {
// References:
// https://developer.qiniu.com/kodo/4088/s3-access-domainname
Name: "region",
Help: "Region to connect to.",
Provider: "Qiniu",
Examples: []fs.OptionExample{{
Value: "cn-east-1",
Help: "The default endpoint - a good choice if you are unsure.\nEast China Region 1.\nNeeds location constraint cn-east-1.",
}, {
Value: "cn-east-2",
Help: "East China Region 2.\nNeeds location constraint cn-east-2.",
}, {
Value: "cn-north-1",
Help: "North China Region 1.\nNeeds location constraint cn-north-1.",
}, {
Value: "cn-south-1",
Help: "South China Region 1.\nNeeds location constraint cn-south-1.",
}, {
Value: "us-north-1",
Help: "North America Region.\nNeeds location constraint us-north-1.",
}, {
Value: "ap-southeast-1",
Help: "Southeast Asia Region 1.\nNeeds location constraint ap-southeast-1.",
}, {
Value: "ap-northeast-1",
Help: "Northeast Asia Region 1.\nNeeds location constraint ap-northeast-1.",
}},
s3: add support for IONOS Cloud Storage
2022-09-15 17:04:34 +02:00
}, {
Name: "region",
Help: "Region where your bucket will be created and your data stored.\n",
Provider: "IONOS",
Examples: []fs.OptionExample{{
Value: "de",
Help: "Frankfurt, Germany",
}, {
Value: "eu-central-2",
Help: "Berlin, Germany",
}, {
Value: "eu-south-2",
Help: "Logrono, Spain",
}},
s3: add petabox.io to s3 providers
2023-05-05 10:44:25 +02:00
}, {
Name: "region",
Help: "Region where your bucket will be created and your data stored.\n",
Provider: "Petabox",
Examples: []fs.OptionExample{{
Value: "us-east-1",
Help: "US East (N. Virginia)",
}, {
Value: "eu-central-1",
Help: "Europe (Frankfurt)",
}, {
Value: "ap-southeast-1",
Help: "Asia Pacific (Singapore)",
}, {
Value: "me-south-1",
Help: "Middle East (Bahrain)",
}, {
Value: "sa-east-1",
Help: "South America (São Paulo)",
}},
s3: add synology to s3 provider list
2023-07-06 11:54:07 +02:00
}, {
Name: "region",
Help: "Region where your data stored.\n",
Provider: "Synology",
Examples: []fs.OptionExample{{
Value: "eu-001",
Help: "Europe Region 1",
}, {
Value: "eu-002",
Help: "Europe Region 2",
}, {
Value: "us-001",
Help: "US Region 1",
}, {
Value: "us-002",
Help: "US Region 2",
}, {
Value: "tw-001",
Help: "Asia (Taiwan)",
}},
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Name: "region",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Region to connect to.\n\nLeave blank if you are using an S3 clone and you don't have a region.",
s3: add synology to s3 provider list
2023-07-06 11:54:07 +02:00
Provider: "!AWS,Alibaba,ArvanCloud,ChinaMobile,Cloudflare,IONOS,Petabox,Liara,Qiniu,RackCorp,Scaleway,Storj,Synology,TencentCOS,HuaweiOBS,IDrive",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Examples: []fs.OptionExample{{
Value: "",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Use this if unsure.\nWill use v4 signatures and an empty region.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "other-v2-signature",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Use this only if v4 signatures don't work.\nE.g. pre Jewel/v10 CEPH.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}},
}, {
Name: "endpoint",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Endpoint for S3 API.\n\nLeave blank if using AWS to use the default endpoint for the region.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "AWS",
s3: Add ChinaMobile EOS to provider list China Mobile Ecloud Elastic Object Storage (EOS) is a cloud object storage service, and is fully compatible with S3. Fixes #6054
2022-03-24 12:57:00 +01:00
}, {
// ChinaMobile endpoints: https://ecloud.10086.cn/op-help-center/doc/article/24534
Name: "endpoint",
Help: "Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.",
Provider: "ChinaMobile",
Examples: []fs.OptionExample{{
Value: "eos-wuxi-1.cmecloud.cn",
Help: "The default endpoint - a good choice if you are unsure.\nEast China (Suzhou)",
}, {
Value: "eos-jinan-1.cmecloud.cn",
Help: "East China (Jinan)",
}, {
Value: "eos-ningbo-1.cmecloud.cn",
Help: "East China (Hangzhou)",
}, {
Value: "eos-shanghai-1.cmecloud.cn",
Help: "East China (Shanghai-1)",
}, {
Value: "eos-zhengzhou-1.cmecloud.cn",
Help: "Central China (Zhengzhou)",
}, {
Value: "eos-hunan-1.cmecloud.cn",
Help: "Central China (Changsha-1)",
}, {
Value: "eos-zhuzhou-1.cmecloud.cn",
Help: "Central China (Changsha-2)",
}, {
Value: "eos-guangzhou-1.cmecloud.cn",
Help: "South China (Guangzhou-2)",
}, {
Value: "eos-dongguan-1.cmecloud.cn",
Help: "South China (Guangzhou-3)",
}, {
Value: "eos-beijing-1.cmecloud.cn",
Help: "North China (Beijing-1)",
}, {
Value: "eos-beijing-2.cmecloud.cn",
Help: "North China (Beijing-2)",
}, {
Value: "eos-beijing-4.cmecloud.cn",
Help: "North China (Beijing-3)",
}, {
Value: "eos-huhehaote-1.cmecloud.cn",
Help: "North China (Huhehaote)",
}, {
Value: "eos-chengdu-1.cmecloud.cn",
Help: "Southwest China (Chengdu)",
}, {
Value: "eos-chongqing-1.cmecloud.cn",
Help: "Southwest China (Chongqing)",
}, {
Value: "eos-guiyang-1.cmecloud.cn",
Help: "Southwest China (Guiyang)",
}, {
Value: "eos-xian-1.cmecloud.cn",
Help: "Nouthwest China (Xian)",
}, {
Value: "eos-yunnan.cmecloud.cn",
Help: "Yunnan China (Kunming)",
}, {
Value: "eos-yunnan-2.cmecloud.cn",
Help: "Yunnan China (Kunming-2)",
}, {
Value: "eos-tianjin-1.cmecloud.cn",
Help: "Tianjin China (Tianjin)",
}, {
Value: "eos-jilin-1.cmecloud.cn",
Help: "Jilin China (Changchun)",
}, {
Value: "eos-hubei-1.cmecloud.cn",
Help: "Hubei China (Xiangyan)",
}, {
Value: "eos-jiangxi-1.cmecloud.cn",
Help: "Jiangxi China (Nanchang)",
}, {
Value: "eos-gansu-1.cmecloud.cn",
Help: "Gansu China (Lanzhou)",
}, {
Value: "eos-shanxi-1.cmecloud.cn",
Help: "Shanxi China (Taiyuan)",
}, {
Value: "eos-liaoning-1.cmecloud.cn",
Help: "Liaoning China (Shenyang)",
}, {
Value: "eos-hebei-1.cmecloud.cn",
Help: "Hebei China (Shijiazhuang)",
}, {
Value: "eos-fujian-1.cmecloud.cn",
Help: "Fujian China (Xiamen)",
}, {
Value: "eos-guangxi-1.cmecloud.cn",
Help: "Guangxi China (Nanning)",
}, {
Value: "eos-anhui-1.cmecloud.cn",
Help: "Anhui China (Huainan)",
}},
s3: Add ArvanCloud AOS to provider list
2022-04-28 11:42:30 +02:00
}, {
s3: Fix Arvancloud Domain and region changes and alphabetise the provider
2023-06-17 13:47:23 +02:00
// ArvanCloud endpoints: https://www.arvancloud.ir/en/products/cloud-storage
s3: Add ArvanCloud AOS to provider list
2022-04-28 11:42:30 +02:00
Name: "endpoint",
Help: "Endpoint for Arvan Cloud Object Storage (AOS) API.",
Provider: "ArvanCloud",
Examples: []fs.OptionExample{{
s3: Fix Arvancloud Domain and region changes and alphabetise the provider
2023-06-17 13:47:23 +02:00
Value: "s3.ir-thr-at1.arvanstorage.ir",
Help: "The default endpoint - a good choice if you are unsure.\nTehran Iran (Simin)",
s3: Add ArvanCloud AOS to provider list
2022-04-28 11:42:30 +02:00
}, {
s3: Fix Arvancloud Domain and region changes and alphabetise the provider
2023-06-17 13:47:23 +02:00
Value: "s3.ir-tbz-sh1.arvanstorage.ir",
s3: Add ArvanCloud AOS to provider list
2022-04-28 11:42:30 +02:00
Help: "Tabriz Iran (Shahriar)",
}},
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Name: "endpoint",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Endpoint for IBM COS S3 API.\n\nSpecify if using an IBM COS On Premise.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "IBMCOS",
Examples: []fs.OptionExample{{
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.us.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Cross Region Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.dal.us.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Cross Region Dallas Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.wdc.us.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Cross Region Washington DC Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.sjc.us.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Cross Region San Jose Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.us.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Cross Region Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.dal.us.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Cross Region Dallas Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.wdc.us.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Cross Region Washington DC Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.sjc.us.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Cross Region San Jose Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.us-east.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Region East Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.us-east.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Region East Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.us-south.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Region South Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.us-south.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "US Region South Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.eu.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "EU Cross Region Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.fra.eu.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "EU Cross Region Frankfurt Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.mil.eu.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "EU Cross Region Milan Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.ams.eu.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "EU Cross Region Amsterdam Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.eu.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "EU Cross Region Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.fra.eu.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "EU Cross Region Frankfurt Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.mil.eu.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "EU Cross Region Milan Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.ams.eu.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "EU Cross Region Amsterdam Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.eu-gb.cloud-object-storage.appdomain.cloud",
backend: fix misspellings
2019-02-07 18:41:17 +01:00
Help: "Great Britain Endpoint",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.eu-gb.cloud-object-storage.appdomain.cloud",
backend: fix misspellings
2019-02-07 18:41:17 +01:00
Help: "Great Britain Private Endpoint",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.eu-de.cloud-object-storage.appdomain.cloud",
Help: "EU Region DE Endpoint",
}, {
Value: "s3.private.eu-de.cloud-object-storage.appdomain.cloud",
Help: "EU Region DE Private Endpoint",
}, {
Value: "s3.ap.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "APAC Cross Regional Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.tok.ap.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "APAC Cross Regional Tokyo Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.hkg.ap.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "APAC Cross Regional HongKong Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.seo.ap.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "APAC Cross Regional Seoul Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.ap.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "APAC Cross Regional Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.tok.ap.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "APAC Cross Regional Tokyo Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.hkg.ap.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "APAC Cross Regional HongKong Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.seo.ap.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "APAC Cross Regional Seoul Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.jp-tok.cloud-object-storage.appdomain.cloud",
Help: "APAC Region Japan Endpoint",
}, {
Value: "s3.private.jp-tok.cloud-object-storage.appdomain.cloud",
Help: "APAC Region Japan Private Endpoint",
}, {
Value: "s3.au-syd.cloud-object-storage.appdomain.cloud",
Help: "APAC Region Australia Endpoint",
}, {
Value: "s3.private.au-syd.cloud-object-storage.appdomain.cloud",
Help: "APAC Region Australia Private Endpoint",
}, {
Value: "s3.ams03.cloud-object-storage.appdomain.cloud",
Help: "Amsterdam Single Site Endpoint",
}, {
Value: "s3.private.ams03.cloud-object-storage.appdomain.cloud",
Help: "Amsterdam Single Site Private Endpoint",
}, {
Value: "s3.che01.cloud-object-storage.appdomain.cloud",
Help: "Chennai Single Site Endpoint",
}, {
Value: "s3.private.che01.cloud-object-storage.appdomain.cloud",
Help: "Chennai Single Site Private Endpoint",
}, {
Value: "s3.mel01.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "Melbourne Single Site Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.mel01.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "Melbourne Single Site Private Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.osl01.cloud-object-storage.appdomain.cloud",
Help: "Oslo Single Site Endpoint",
}, {
Value: "s3.private.osl01.cloud-object-storage.appdomain.cloud",
Help: "Oslo Single Site Private Endpoint",
}, {
Value: "s3.tor01.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "Toronto Single Site Endpoint",
}, {
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
Value: "s3.private.tor01.cloud-object-storage.appdomain.cloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "Toronto Single Site Private Endpoint",
s3: update IBM COS endpoints - fixes #4522
2020-08-30 18:21:11 +02:00
}, {
Value: "s3.seo01.cloud-object-storage.appdomain.cloud",
Help: "Seoul Single Site Endpoint",
}, {
Value: "s3.private.seo01.cloud-object-storage.appdomain.cloud",
Help: "Seoul Single Site Private Endpoint",
}, {
Value: "s3.mon01.cloud-object-storage.appdomain.cloud",
Help: "Montreal Single Site Endpoint",
}, {
Value: "s3.private.mon01.cloud-object-storage.appdomain.cloud",
Help: "Montreal Single Site Private Endpoint",
}, {
Value: "s3.mex01.cloud-object-storage.appdomain.cloud",
Help: "Mexico Single Site Endpoint",
}, {
Value: "s3.private.mex01.cloud-object-storage.appdomain.cloud",
Help: "Mexico Single Site Private Endpoint",
}, {
Value: "s3.sjc04.cloud-object-storage.appdomain.cloud",
Help: "San Jose Single Site Endpoint",
}, {
Value: "s3.private.sjc04.cloud-object-storage.appdomain.cloud",
Help: "San Jose Single Site Private Endpoint",
}, {
Value: "s3.mil01.cloud-object-storage.appdomain.cloud",
Help: "Milan Single Site Endpoint",
}, {
Value: "s3.private.mil01.cloud-object-storage.appdomain.cloud",
Help: "Milan Single Site Private Endpoint",
}, {
Value: "s3.hkg02.cloud-object-storage.appdomain.cloud",
Help: "Hong Kong Single Site Endpoint",
}, {
Value: "s3.private.hkg02.cloud-object-storage.appdomain.cloud",
Help: "Hong Kong Single Site Private Endpoint",
}, {
Value: "s3.par01.cloud-object-storage.appdomain.cloud",
Help: "Paris Single Site Endpoint",
}, {
Value: "s3.private.par01.cloud-object-storage.appdomain.cloud",
Help: "Paris Single Site Private Endpoint",
}, {
Value: "s3.sng01.cloud-object-storage.appdomain.cloud",
Help: "Singapore Single Site Endpoint",
}, {
Value: "s3.private.sng01.cloud-object-storage.appdomain.cloud",
Help: "Singapore Single Site Private Endpoint",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}},
s3: add support for IONOS Cloud Storage
2022-09-15 17:04:34 +02:00
}, {
Name: "endpoint",
Help: "Endpoint for IONOS S3 Object Storage.\n\nSpecify the endpoint from the same region.",
Provider: "IONOS",
Examples: []fs.OptionExample{{
Value: "s3-eu-central-1.ionoscloud.com",
Help: "Frankfurt, Germany",
}, {
Value: "s3-eu-central-2.ionoscloud.com",
Help: "Berlin, Germany",
}, {
Value: "s3-eu-south-2.ionoscloud.com",
Help: "Logrono, Spain",
}},
s3: add petabox.io to s3 providers
2023-05-05 10:44:25 +02:00
}, {
Name: "endpoint",
Help: "Endpoint for Petabox S3 Object Storage.\n\nSpecify the endpoint from the same region.",
Provider: "Petabox",
Required: true,
Examples: []fs.OptionExample{{
Value: "s3.petabox.io",
Help: "US East (N. Virginia)",
}, {
Value: "s3.us-east-1.petabox.io",
Help: "US East (N. Virginia)",
}, {
Value: "s3.eu-central-1.petabox.io",
Help: "Europe (Frankfurt)",
}, {
Value: "s3.ap-southeast-1.petabox.io",
Help: "Asia Pacific (Singapore)",
}, {
Value: "s3.me-south-1.petabox.io",
Help: "Middle East (Bahrain)",
}, {
Value: "s3.sa-east-1.petabox.io",
Help: "South America (São Paulo)",
}},
s3: add Leviia S3 Object Storage as provider
2023-07-16 19:08:47 +02:00
}, {
// Leviia endpoints: https://www.leviia.com/object-storage/
Name: "endpoint",
Help: "Endpoint for Leviia Object Storage API.",
Provider: "Leviia",
Examples: []fs.OptionExample{{
Value: "s3.leviia.com",
Help: "The default endpoint\nLeviia",
}},
s3: add Liara LOS to provider list
2022-12-06 13:25:23 +01:00
}, {
// Liara endpoints: https://liara.ir/landing/object-storage
Name: "endpoint",
Help: "Endpoint for Liara Object Storage API.",
Provider: "Liara",
Examples: []fs.OptionExample{{
Value: "storage.iran.liara.space",
Help: "The default endpoint\nIran",
}},
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
}, {
// oss endpoints: https://help.aliyun.com/document_detail/31837.html
Name: "endpoint",
Help: "Endpoint for OSS API.",
Provider: "Alibaba",
Examples: []fs.OptionExample{{
s3: update Alibaba OSS endpoints
2021-07-08 13:03:04 +02:00
Value: "oss-accelerate.aliyuncs.com",
Help: "Global Accelerate",
}, {
Value: "oss-accelerate-overseas.aliyuncs.com",
Help: "Global Accelerate (outside mainland China)",
}, {
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
Value: "oss-cn-hangzhou.aliyuncs.com",
Help: "East China 1 (Hangzhou)",
}, {
Value: "oss-cn-shanghai.aliyuncs.com",
Help: "East China 2 (Shanghai)",
}, {
Value: "oss-cn-qingdao.aliyuncs.com",
Help: "North China 1 (Qingdao)",
}, {
Value: "oss-cn-beijing.aliyuncs.com",
Help: "North China 2 (Beijing)",
}, {
Value: "oss-cn-zhangjiakou.aliyuncs.com",
Help: "North China 3 (Zhangjiakou)",
}, {
Value: "oss-cn-huhehaote.aliyuncs.com",
s3: update Alibaba OSS endpoints
2021-07-08 13:03:04 +02:00
Help: "North China 5 (Hohhot)",
}, {
Value: "oss-cn-wulanchabu.aliyuncs.com",
Help: "North China 6 (Ulanqab)",
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
}, {
Value: "oss-cn-shenzhen.aliyuncs.com",
Help: "South China 1 (Shenzhen)",
s3: update Alibaba OSS endpoints
2021-07-08 13:03:04 +02:00
}, {
Value: "oss-cn-heyuan.aliyuncs.com",
Help: "South China 2 (Heyuan)",
}, {
Value: "oss-cn-guangzhou.aliyuncs.com",
Help: "South China 3 (Guangzhou)",
}, {
Value: "oss-cn-chengdu.aliyuncs.com",
Help: "West China 1 (Chengdu)",
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
}, {
Value: "oss-cn-hongkong.aliyuncs.com",
Help: "Hong Kong (Hong Kong)",
}, {
Value: "oss-us-west-1.aliyuncs.com",
Help: "US West 1 (Silicon Valley)",
}, {
Value: "oss-us-east-1.aliyuncs.com",
Help: "US East 1 (Virginia)",
}, {
Value: "oss-ap-southeast-1.aliyuncs.com",
Help: "Southeast Asia Southeast 1 (Singapore)",
}, {
Value: "oss-ap-southeast-2.aliyuncs.com",
Help: "Asia Pacific Southeast 2 (Sydney)",
}, {
Value: "oss-ap-southeast-3.aliyuncs.com",
Help: "Southeast Asia Southeast 3 (Kuala Lumpur)",
}, {
Value: "oss-ap-southeast-5.aliyuncs.com",
Help: "Asia Pacific Southeast 5 (Jakarta)",
}, {
Value: "oss-ap-northeast-1.aliyuncs.com",
Help: "Asia Pacific Northeast 1 (Japan)",
}, {
Value: "oss-ap-south-1.aliyuncs.com",
Help: "Asia Pacific South 1 (Mumbai)",
}, {
Value: "oss-eu-central-1.aliyuncs.com",
Help: "Central Europe 1 (Frankfurt)",
}, {
Value: "oss-eu-west-1.aliyuncs.com",
Help: "West Europe (London)",
}, {
Value: "oss-me-east-1.aliyuncs.com",
Help: "Middle East 1 (Dubai)",
}},
backend: add Huawei OBS to s3 provider list
2022-06-07 09:41:46 +02:00
}, {
// obs endpoints: https://developer.huaweicloud.com/intl/en-us/endpoint?OBS
Name: "endpoint",
Help: "Endpoint for OBS API.",
Provider: "HuaweiOBS",
Examples: []fs.OptionExample{{
Value: "obs.af-south-1.myhuaweicloud.com",
Help: "AF-Johannesburg",
}, {
Value: "obs.ap-southeast-2.myhuaweicloud.com",
Help: "AP-Bangkok",
}, {
Value: "obs.ap-southeast-3.myhuaweicloud.com",
Help: "AP-Singapore",
}, {
Value: "obs.cn-east-3.myhuaweicloud.com",
Help: "CN East-Shanghai1",
}, {
Value: "obs.cn-east-2.myhuaweicloud.com",
Help: "CN East-Shanghai2",
}, {
Value: "obs.cn-north-1.myhuaweicloud.com",
Help: "CN North-Beijing1",
}, {
Value: "obs.cn-north-4.myhuaweicloud.com",
Help: "CN North-Beijing4",
}, {
Value: "obs.cn-south-1.myhuaweicloud.com",
Help: "CN South-Guangzhou",
}, {
Value: "obs.ap-southeast-1.myhuaweicloud.com",
Help: "CN-Hong Kong",
}, {
Value: "obs.sa-argentina-1.myhuaweicloud.com",
Help: "LA-Buenos Aires1",
}, {
Value: "obs.sa-peru-1.myhuaweicloud.com",
Help: "LA-Lima1",
}, {
Value: "obs.na-mexico-1.myhuaweicloud.com",
Help: "LA-Mexico City1",
}, {
Value: "obs.sa-chile-1.myhuaweicloud.com",
Help: "LA-Santiago2",
}, {
Value: "obs.sa-brazil-1.myhuaweicloud.com",
Help: "LA-Sao Paulo1",
}, {
Value: "obs.ru-northwest-2.myhuaweicloud.com",
Help: "RU-Moscow2",
}},
s3: add Scaleway provider - fixes #4338
2020-06-12 17:04:16 +02:00
}, {
Name: "endpoint",
Help: "Endpoint for Scaleway Object Storage.",
Provider: "Scaleway",
Examples: []fs.OptionExample{{
Value: "s3.nl-ams.scw.cloud",
Help: "Amsterdam Endpoint",
}, {
Value: "s3.fr-par.scw.cloud",
Help: "Paris Endpoint",
s3: Add Warsaw location for Scaleway Add new location in Warsaw (Poland) to endpoints for Scaleway. More Information: https://blog.scaleway.com/scaleway-is-now-in-warsaw/ https://www.scaleway.com/en/docs/storage/object/how-to/create-a-bucket/
2022-05-17 15:24:32 +02:00
}, {
Value: "s3.pl-waw.scw.cloud",
Help: "Warsaw Endpoint",
s3: add Scaleway provider - fixes #4338
2020-06-12 17:04:16 +02:00
}},
s3: Add StackPath Object Storage Support
2020-01-31 00:21:24 +01:00
}, {
Name: "endpoint",
Help: "Endpoint for StackPath Object Storage.",
Provider: "StackPath",
Examples: []fs.OptionExample{{
Value: "s3.us-east-2.stackpathstorage.com",
Help: "US East Endpoint",
}, {
Value: "s3.us-west-1.stackpathstorage.com",
Help: "US West Endpoint",
}, {
Value: "s3.eu-central-1.stackpathstorage.com",
Help: "EU Endpoint",
}},
s3: add GCS to provider list
2023-01-09 07:54:51 +01:00
}, {
Name: "endpoint",
Help: "Endpoint for Google Cloud Storage.",
Provider: "GCS",
Examples: []fs.OptionExample{{
Value: "https://storage.googleapis.com",
Help: "Google Cloud Storage endpoint",
}},
s3: add specific provider for Storj Shared gateways - unsupported features (Copy) are turned off for Storj - enable urlEncodedListing for Storj provider - set chunksize to 64Mb
2021-09-29 15:43:06 +02:00
}, {
Name: "endpoint",
s3/storj: update endpoints Storj switched to a single global s3 endpoint backed by a BGP routing. We want to stop advertizing the former regional endpoints and have the global one as the only option.
2022-12-22 14:44:10 +01:00
Help: "Endpoint for Storj Gateway.",
s3: add specific provider for Storj Shared gateways - unsupported features (Copy) are turned off for Storj - enable urlEncodedListing for Storj provider - set chunksize to 64Mb
2021-09-29 15:43:06 +02:00
Provider: "Storj",
Examples: []fs.OptionExample{{
s3/storj: update endpoints Storj switched to a single global s3 endpoint backed by a BGP routing. We want to stop advertizing the former regional endpoints and have the global one as the only option.
2022-12-22 14:44:10 +01:00
Value: "gateway.storjshare.io",
Help: "Global Hosted Gateway",
s3: add specific provider for Storj Shared gateways - unsupported features (Copy) are turned off for Storj - enable urlEncodedListing for Storj provider - set chunksize to 64Mb
2021-09-29 15:43:06 +02:00
}},
s3: add synology to s3 provider list
2023-07-06 11:54:07 +02:00
}, {
Name: "endpoint",
Help: "Endpoint for Synology C2 Object Storage API.",
Provider: "Synology",
Examples: []fs.OptionExample{{
Value: "eu-001.s3.synologyc2.net",
Help: "EU Endpoint 1",
}, {
Value: "eu-002.s3.synologyc2.net",
Help: "EU Endpoint 2",
}, {
Value: "us-001.s3.synologyc2.net",
Help: "US Endpoint 1",
}, {
Value: "us-002.s3.synologyc2.net",
Help: "US Endpoint 2",
}, {
Value: "tw-001.s3.synologyc2.net",
Help: "TW Endpoint 1",
}},
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
// cos endpoints: https://intl.cloud.tencent.com/document/product/436/6224
Name: "endpoint",
Help: "Endpoint for Tencent COS API.",
Provider: "TencentCOS",
Examples: []fs.OptionExample{{
Value: "cos.ap-beijing.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Beijing Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-nanjing.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Nanjing Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-shanghai.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Shanghai Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-guangzhou.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Guangzhou Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-nanjing.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Nanjing Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-chengdu.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Chengdu Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-chongqing.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Chongqing Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-hongkong.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Hong Kong (China) Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-singapore.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Singapore Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-mumbai.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Mumbai Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-seoul.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Seoul Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-bangkok.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Bangkok Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.ap-tokyo.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Tokyo Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.na-siliconvalley.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Silicon Valley Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.na-ashburn.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Virginia Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.na-toronto.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Toronto Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.eu-frankfurt.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Frankfurt Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.eu-moscow.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Moscow Region",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "cos.accelerate.myqcloud.com",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Use Tencent COS Accelerate Endpoint",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}},
s3: Add RackCorp object storage to providers
2021-11-09 12:46:58 +01:00
}, {
// RackCorp endpoints: https://www.rackcorp.com/storage/s3storage
Name: "endpoint",
Help: "Endpoint for RackCorp Object Storage.",
Provider: "RackCorp",
Examples: []fs.OptionExample{{
Value: "s3.rackcorp.com",
Help: "Global (AnyCast) Endpoint",
}, {
Value: "au.s3.rackcorp.com",
Help: "Australia (Anycast) Endpoint",
}, {
Value: "au-nsw.s3.rackcorp.com",
Help: "Sydney (Australia) Endpoint",
}, {
Value: "au-qld.s3.rackcorp.com",
Help: "Brisbane (Australia) Endpoint",
}, {
Value: "au-vic.s3.rackcorp.com",
Help: "Melbourne (Australia) Endpoint",
}, {
Value: "au-wa.s3.rackcorp.com",
Help: "Perth (Australia) Endpoint",
}, {
Value: "ph.s3.rackcorp.com",
Help: "Manila (Philippines) Endpoint",
}, {
Value: "th.s3.rackcorp.com",
Help: "Bangkok (Thailand) Endpoint",
}, {
Value: "hk.s3.rackcorp.com",
Help: "HK (Hong Kong) Endpoint",
}, {
Value: "mn.s3.rackcorp.com",
Help: "Ulaanbaatar (Mongolia) Endpoint",
}, {
Value: "kg.s3.rackcorp.com",
Help: "Bishkek (Kyrgyzstan) Endpoint",
}, {
Value: "id.s3.rackcorp.com",
Help: "Jakarta (Indonesia) Endpoint",
}, {
Value: "jp.s3.rackcorp.com",
Help: "Tokyo (Japan) Endpoint",
}, {
Value: "sg.s3.rackcorp.com",
Help: "SG (Singapore) Endpoint",
}, {
Value: "de.s3.rackcorp.com",
Help: "Frankfurt (Germany) Endpoint",
}, {
Value: "us.s3.rackcorp.com",
Help: "USA (AnyCast) Endpoint",
}, {
Value: "us-east-1.s3.rackcorp.com",
Help: "New York (USA) Endpoint",
}, {
Value: "us-west-1.s3.rackcorp.com",
Help: "Freemont (USA) Endpoint",
}, {
Value: "nz.s3.rackcorp.com",
Help: "Auckland (New Zealand) Endpoint",
}},
s3: add Qiniu KODO to s3 provider list - fixes #6195
2022-10-13 16:49:22 +02:00
}, {
// Qiniu endpoints: https://developer.qiniu.com/kodo/4088/s3-access-domainname
Name: "endpoint",
Help: "Endpoint for Qiniu Object Storage.",
Provider: "Qiniu",
Examples: []fs.OptionExample{{
Value: "s3-cn-east-1.qiniucs.com",
Help: "East China Endpoint 1",
}, {
Value: "s3-cn-east-2.qiniucs.com",
Help: "East China Endpoint 2",
}, {
Value: "s3-cn-north-1.qiniucs.com",
Help: "North China Endpoint 1",
}, {
Value: "s3-cn-south-1.qiniucs.com",
Help: "South China Endpoint 1",
}, {
Value: "s3-us-north-1.qiniucs.com",
Help: "North America Endpoint 1",
}, {
Value: "s3-ap-southeast-1.qiniucs.com",
Help: "Southeast Asia Endpoint 1",
}, {
Value: "s3-ap-northeast-1.qiniucs.com",
Help: "Northeast Asia Endpoint 1",
}},
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Name: "endpoint",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Endpoint for S3 API.\n\nRequired when using an S3 clone.",
s3: add synology to s3 provider list
2023-07-06 11:54:07 +02:00
Provider: "!AWS,ArvanCloud,IBMCOS,IDrive,IONOS,TencentCOS,HuaweiOBS,Alibaba,ChinaMobile,GCS,Liara,Scaleway,StackPath,Storj,Synology,RackCorp,Qiniu,Petabox",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Examples: []fs.OptionExample{{
s3: Update Dreamhost endpoint - fixes #2974
2019-02-13 22:10:43 +01:00
Value: "objects-us-east-1.dream.io",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Help: "Dream Objects endpoint",
Provider: "Dreamhost",
s3: add DigitalOcean Spaces regions sfo3, fra1, syd1
2022-12-14 15:00:04 +01:00
}, {
Value: "syd1.digitaloceanspaces.com",
Help: "DigitalOcean Spaces Sydney 1",
Provider: "DigitalOcean",
}, {
Value: "sfo3.digitaloceanspaces.com",
Help: "DigitalOcean Spaces San Francisco 3",
Provider: "DigitalOcean",
}, {
Value: "fra1.digitaloceanspaces.com",
Help: "DigitalOcean Spaces Frankfurt 1",
Provider: "DigitalOcean",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "nyc3.digitaloceanspaces.com",
s3: add DigitalOcean Spaces regions sfo3, fra1, syd1
2022-12-14 15:00:04 +01:00
Help: "DigitalOcean Spaces New York 3",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "DigitalOcean",
}, {
Value: "ams3.digitaloceanspaces.com",
s3: add DigitalOcean Spaces regions sfo3, fra1, syd1
2022-12-14 15:00:04 +01:00
Help: "DigitalOcean Spaces Amsterdam 3",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "DigitalOcean",
}, {
Value: "sgp1.digitaloceanspaces.com",
s3: add DigitalOcean Spaces regions sfo3, fra1, syd1
2022-12-14 15:00:04 +01:00
Help: "DigitalOcean Spaces Singapore 1",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "DigitalOcean",
}, {
s3: Add SeaweedFS
2021-06-08 10:59:57 +02:00
Value: "localhost:8333",
Help: "SeaweedFS S3 localhost",
Provider: "SeaweedFS",
s3: add support for Seagate Lyve Cloud storage
2022-03-04 13:35:46 +01:00
}, {
Value: "s3.us-east-1.lyvecloud.seagate.com",
s3: add other regions for Lyve and correct Provider name
2022-03-14 16:37:34 +01:00
Help: "Seagate Lyve Cloud US East 1 (Virginia)",
Provider: "LyveCloud",
}, {
Value: "s3.us-west-1.lyvecloud.seagate.com",
Help: "Seagate Lyve Cloud US West 1 (California)",
Provider: "LyveCloud",
}, {
Value: "s3.ap-southeast-1.lyvecloud.seagate.com",
Help: "Seagate Lyve Cloud AP Southeast 1 (Singapore)",
Provider: "LyveCloud",
s3: Add SeaweedFS
2021-06-08 10:59:57 +02:00
}, {
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Value: "s3.wasabisys.com",
s3: Add additional Wasabi locations
2022-11-11 06:19:27 +01:00
Help: "Wasabi US East 1 (N. Virginia)",
Provider: "Wasabi",
}, {
Value: "s3.us-east-2.wasabisys.com",
Help: "Wasabi US East 2 (N. Virginia)",
Provider: "Wasabi",
}, {
Value: "s3.us-central-1.wasabisys.com",
Help: "Wasabi US Central 1 (Texas)",
s3: add config info for Wasabi's US-West endpoint Wasabi has two location, US East and US West, with different endpoint URLs. When configuring S3 to use Wasabi, provide the endpoint information for both locations.
2018-11-17 17:24:00 +01:00
Provider: "Wasabi",
}, {
Value: "s3.us-west-1.wasabisys.com",
s3: Add additional Wasabi locations
2022-11-11 06:19:27 +01:00
Help: "Wasabi US West 1 (Oregon)",
Provider: "Wasabi",
}, {
Value: "s3.ca-central-1.wasabisys.com",
Help: "Wasabi CA Central 1 (Toronto)",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "Wasabi",
s3: add config info for Wasabi's EU Central endpoint Wasabi has a EU Central endpoint for a couple months now, so add it to the list. Signed-off-by: Robert Marko <robimarko@gmail.com>
2019-05-15 10:22:06 +02:00
}, {
Value: "s3.eu-central-1.wasabisys.com",
s3: Add additional Wasabi locations
2022-11-11 06:19:27 +01:00
Help: "Wasabi EU Central 1 (Amsterdam)",
Provider: "Wasabi",
}, {
Value: "s3.eu-central-2.wasabisys.com",
Help: "Wasabi EU Central 2 (Frankfurt)",
Provider: "Wasabi",
}, {
Value: "s3.eu-west-1.wasabisys.com",
Help: "Wasabi EU West 1 (London)",
Provider: "Wasabi",
}, {
Value: "s3.eu-west-2.wasabisys.com",
Help: "Wasabi EU West 2 (Paris)",
s3: add config info for Wasabi's EU Central endpoint Wasabi has a EU Central endpoint for a couple months now, so add it to the list. Signed-off-by: Robert Marko <robimarko@gmail.com>
2019-05-15 10:22:06 +02:00
Provider: "Wasabi",
s3: add Wasabi's AP-Northeast endpoint info * Wasabi starts to provide AP Northeast (Tokyo) endpoint for all customers, so add it to the list Signed-off-by: lindwurm <lindwurm.q@gmail.com>
2021-07-30 18:47:33 +02:00
}, {
Value: "s3.ap-northeast-1.wasabisys.com",
s3: Add Wasabi AP Northeast 2 endpoint info * Wasabi starts to provide AP Northeast 2 (Osaka) endpoint, so add it to the list * Rename ap-northeast-1 as "AP Northeast 1 (Tokyo)" from "AP Northeast" Signed-off-by: lindwurm <lindwurm.q@gmail.com>
2021-11-21 22:04:53 +01:00
Help: "Wasabi AP Northeast 1 (Tokyo) endpoint",
Provider: "Wasabi",
}, {
Value: "s3.ap-northeast-2.wasabisys.com",
Help: "Wasabi AP Northeast 2 (Osaka) endpoint",
s3: add Wasabi's AP-Northeast endpoint info * Wasabi starts to provide AP Northeast (Tokyo) endpoint for all customers, so add it to the list Signed-off-by: lindwurm <lindwurm.q@gmail.com>
2021-07-30 18:47:33 +02:00
Provider: "Wasabi",
s3: Add additional Wasabi locations
2022-11-11 06:19:27 +01:00
}, {
Value: "s3.ap-southeast-1.wasabisys.com",
Help: "Wasabi AP Southeast 1 (Singapore)",
Provider: "Wasabi",
}, {
Value: "s3.ap-southeast-2.wasabisys.com",
Help: "Wasabi AP Southeast 2 (Sydney)",
Provider: "Wasabi",
s3: add Liara LOS to provider list
2022-12-06 13:25:23 +01:00
}, {
Value: "storage.iran.liara.space",
Help: "Liara Iran endpoint",
Provider: "Liara",
s3: Add ArvanCloud AOS to provider list
2022-04-28 11:42:30 +02:00
}, {
s3: Fix Arvancloud Domain and region changes and alphabetise the provider
2023-06-17 13:47:23 +02:00
Value: "s3.ir-thr-at1.arvanstorage.ir",
Help: "ArvanCloud Tehran Iran (Simin) endpoint",
Provider: "ArvanCloud",
}, {
Value: "s3.ir-tbz-sh1.arvanstorage.ir",
Help: "ArvanCloud Tabriz Iran (Shahriar) endpoint",
s3: Add ArvanCloud AOS to provider list
2022-04-28 11:42:30 +02:00
Provider: "ArvanCloud",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}},
}, {
Name: "location_constraint",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Location constraint - must be set to match the Region.\n\nUsed when creating buckets only.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "AWS",
Examples: []fs.OptionExample{{
Value: "",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Empty for US Region, Northern Virginia, or Pacific Northwest",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "us-east-2",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "US East (Ohio) Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "us-west-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "US West (Northern California) Region",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "us-west-2",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "US West (Oregon) Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ca-central-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Canada (Central) Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "eu-west-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU (Ireland) Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "eu-west-2",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU (London) Region",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "eu-west-3",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU (Paris) Region",
s3: add aws endpoint eu-north-1
2019-02-02 20:44:26 +01:00
}, {
Value: "eu-north-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU (Stockholm) Region",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "eu-south-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU (Milan) Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "EU",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "EU Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ap-southeast-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Singapore) Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ap-southeast-2",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Sydney) Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ap-northeast-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Tokyo) Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ap-northeast-2",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Seoul) Region",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "ap-northeast-3",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Osaka-Local) Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ap-south-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Mumbai) Region",
s3: Add new region Asia Patific (Hong Kong)
2020-01-02 12:10:48 +01:00
}, {
Value: "ap-east-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Asia Pacific (Hong Kong) Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "sa-east-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "South America (Sao Paulo) Region",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "me-south-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Middle East (Bahrain) Region",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "af-south-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Africa (Cape Town) Region",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "cn-north-1",
Help: "China (Beijing) Region",
}, {
Value: "cn-northwest-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "China (Ningxia) Region",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "us-gov-east-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "AWS GovCloud (US-East) Region",
s3: Add missing regions for AWS Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-03 10:44:25 +02:00
}, {
Value: "us-gov-west-1",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "AWS GovCloud (US) Region",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}},
s3: Add ChinaMobile EOS to provider list China Mobile Ecloud Elastic Object Storage (EOS) is a cloud object storage service, and is fully compatible with S3. Fixes #6054
2022-03-24 12:57:00 +01:00
}, {
Name: "location_constraint",
Help: "Location constraint - must match endpoint.\n\nUsed when creating buckets only.",
Provider: "ChinaMobile",
Examples: []fs.OptionExample{{
Value: "wuxi1",
Help: "East China (Suzhou)",
}, {
Value: "jinan1",
Help: "East China (Jinan)",
}, {
Value: "ningbo1",
Help: "East China (Hangzhou)",
}, {
Value: "shanghai1",
Help: "East China (Shanghai-1)",
}, {
Value: "zhengzhou1",
Help: "Central China (Zhengzhou)",
}, {
Value: "hunan1",
Help: "Central China (Changsha-1)",
}, {
Value: "zhuzhou1",
Help: "Central China (Changsha-2)",
}, {
Value: "guangzhou1",
Help: "South China (Guangzhou-2)",
}, {
Value: "dongguan1",
Help: "South China (Guangzhou-3)",
}, {
Value: "beijing1",
Help: "North China (Beijing-1)",
}, {
Value: "beijing2",
Help: "North China (Beijing-2)",
}, {
Value: "beijing4",
Help: "North China (Beijing-3)",
}, {
Value: "huhehaote1",
Help: "North China (Huhehaote)",
}, {
Value: "chengdu1",
Help: "Southwest China (Chengdu)",
}, {
Value: "chongqing1",
Help: "Southwest China (Chongqing)",
}, {
Value: "guiyang1",
Help: "Southwest China (Guiyang)",
}, {
Value: "xian1",
Help: "Nouthwest China (Xian)",
}, {
Value: "yunnan",
Help: "Yunnan China (Kunming)",
}, {
Value: "yunnan2",
Help: "Yunnan China (Kunming-2)",
}, {
Value: "tianjin1",
Help: "Tianjin China (Tianjin)",
}, {
Value: "jilin1",
Help: "Jilin China (Changchun)",
}, {
Value: "hubei1",
Help: "Hubei China (Xiangyan)",
}, {
Value: "jiangxi1",
Help: "Jiangxi China (Nanchang)",
}, {
Value: "gansu1",
Help: "Gansu China (Lanzhou)",
}, {
Value: "shanxi1",
Help: "Shanxi China (Taiyuan)",
}, {
Value: "liaoning1",
Help: "Liaoning China (Shenyang)",
}, {
Value: "hebei1",
Help: "Hebei China (Shijiazhuang)",
}, {
Value: "fujian1",
Help: "Fujian China (Xiamen)",
}, {
Value: "guangxi1",
Help: "Guangxi China (Nanning)",
}, {
Value: "anhui1",
Help: "Anhui China (Huainan)",
}},
s3: Add ArvanCloud AOS to provider list
2022-04-28 11:42:30 +02:00
}, {
Name: "location_constraint",
Help: "Location constraint - must match endpoint.\n\nUsed when creating buckets only.",
Provider: "ArvanCloud",
Examples: []fs.OptionExample{{
Value: "ir-thr-at1",
s3: Fix Arvancloud Domain and region changes and alphabetise the provider
2023-06-17 13:47:23 +02:00
Help: "Tehran Iran (Simin)",
s3: Add ArvanCloud AOS to provider list
2022-04-28 11:42:30 +02:00
}, {
Value: "ir-tbz-sh1",
Help: "Tabriz Iran (Shahriar)",
}},
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Name: "location_constraint",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Location constraint - must match endpoint when using IBM Cloud Public.\n\nFor on-prem COS, do not make a selection from this list, hit enter.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "IBMCOS",
Examples: []fs.OptionExample{{
Value: "us-standard",
Help: "US Cross Region Standard",
}, {
Value: "us-vault",
Help: "US Cross Region Vault",
}, {
Value: "us-cold",
Help: "US Cross Region Cold",
}, {
Value: "us-flex",
Help: "US Cross Region Flex",
}, {
Value: "us-east-standard",
Help: "US East Region Standard",
}, {
Value: "us-east-vault",
Help: "US East Region Vault",
}, {
Value: "us-east-cold",
Help: "US East Region Cold",
}, {
Value: "us-east-flex",
Help: "US East Region Flex",
}, {
Value: "us-south-standard",
backend: fix misspellings
2019-02-07 18:41:17 +01:00
Help: "US South Region Standard",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "us-south-vault",
Help: "US South Region Vault",
}, {
Value: "us-south-cold",
Help: "US South Region Cold",
}, {
Value: "us-south-flex",
Help: "US South Region Flex",
}, {
Value: "eu-standard",
Help: "EU Cross Region Standard",
}, {
Value: "eu-vault",
Help: "EU Cross Region Vault",
}, {
Value: "eu-cold",
Help: "EU Cross Region Cold",
}, {
Value: "eu-flex",
Help: "EU Cross Region Flex",
}, {
Value: "eu-gb-standard",
backend: fix misspellings
2019-02-07 18:41:17 +01:00
Help: "Great Britain Standard",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "eu-gb-vault",
backend: fix misspellings
2019-02-07 18:41:17 +01:00
Help: "Great Britain Vault",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "eu-gb-cold",
backend: fix misspellings
2019-02-07 18:41:17 +01:00
Help: "Great Britain Cold",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "eu-gb-flex",
backend: fix misspellings
2019-02-07 18:41:17 +01:00
Help: "Great Britain Flex",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "ap-standard",
Help: "APAC Standard",
}, {
Value: "ap-vault",
Help: "APAC Vault",
}, {
Value: "ap-cold",
Help: "APAC Cold",
}, {
Value: "ap-flex",
Help: "APAC Flex",
}, {
Value: "mel01-standard",
Help: "Melbourne Standard",
}, {
Value: "mel01-vault",
Help: "Melbourne Vault",
}, {
Value: "mel01-cold",
Help: "Melbourne Cold",
}, {
Value: "mel01-flex",
Help: "Melbourne Flex",
}, {
Value: "tor01-standard",
Help: "Toronto Standard",
}, {
Value: "tor01-vault",
Help: "Toronto Vault",
}, {
Value: "tor01-cold",
Help: "Toronto Cold",
}, {
Value: "tor01-flex",
Help: "Toronto Flex",
}},
s3: Add RackCorp object storage to providers
2021-11-09 12:46:58 +01:00
}, {
Name: "location_constraint",
Help: "Location constraint - the location where your bucket will be located and your data stored.\n",
Provider: "RackCorp",
Examples: []fs.OptionExample{{
Value: "global",
Help: "Global CDN Region",
}, {
Value: "au",
Help: "Australia (All locations)",
}, {
Value: "au-nsw",
Help: "NSW (Australia) Region",
}, {
Value: "au-qld",
Help: "QLD (Australia) Region",
}, {
Value: "au-vic",
Help: "VIC (Australia) Region",
}, {
Value: "au-wa",
Help: "Perth (Australia) Region",
}, {
Value: "ph",
Help: "Manila (Philippines) Region",
}, {
Value: "th",
Help: "Bangkok (Thailand) Region",
}, {
Value: "hk",
Help: "HK (Hong Kong) Region",
}, {
Value: "mn",
Help: "Ulaanbaatar (Mongolia) Region",
}, {
Value: "kg",
Help: "Bishkek (Kyrgyzstan) Region",
}, {
Value: "id",
Help: "Jakarta (Indonesia) Region",
}, {
Value: "jp",
Help: "Tokyo (Japan) Region",
}, {
Value: "sg",
Help: "SG (Singapore) Region",
}, {
Value: "de",
Help: "Frankfurt (Germany) Region",
}, {
Value: "us",
Help: "USA (AnyCast) Region",
}, {
Value: "us-east-1",
Help: "New York (USA) Region",
}, {
Value: "us-west-1",
Help: "Freemont (USA) Region",
}, {
Value: "nz",
Help: "Auckland (New Zealand) Region",
}},
s3: add Qiniu KODO to s3 provider list - fixes #6195
2022-10-13 16:49:22 +02:00
}, {
Name: "location_constraint",
Help: "Location constraint - must be set to match the Region.\n\nUsed when creating buckets only.",
Provider: "Qiniu",
Examples: []fs.OptionExample{{
Value: "cn-east-1",
Help: "East China Region 1",
}, {
Value: "cn-east-2",
Help: "East China Region 2",
}, {
Value: "cn-north-1",
Help: "North China Region 1",
}, {
Value: "cn-south-1",
Help: "South China Region 1",
}, {
Value: "us-north-1",
Help: "North America Region 1",
}, {
Value: "ap-southeast-1",
Help: "Southeast Asia Region 1",
}, {
Value: "ap-northeast-1",
Help: "Northeast Asia Region 1",
}},
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Name: "location_constraint",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Location constraint - must be set to match the Region.\n\nLeave blank if not sure. Used when creating buckets only.",
s3: add Leviia S3 Object Storage as provider
2023-07-16 19:08:47 +02:00
Provider: "!AWS,Alibaba,ArvanCloud,HuaweiOBS,ChinaMobile,Cloudflare,IBMCOS,IDrive,IONOS,Leviia,Liara,Qiniu,RackCorp,Scaleway,StackPath,Storj,TencentCOS,Petabox",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Name: "acl",
s3: set ACL for server side copies to that provided by the user - fixes #2691 Before this change the ACL for objects which were server side copied was left at the default "private" settings. S3 doesn't copy the ACL from the source when you copy an object, you have to set it afresh which is what this does.
2018-10-25 23:19:36 +02:00
Help: `Canned ACL used when creating buckets and storing or copying objects.
s3: add --s3-bucket-acl to control bucket ACL - fixes #2918 Before this change buckets were created with the same ACL as objects. After this change, the user can set just --s3-acl to set the ACL of buckets and objects, or use --s3-bucket-acl as well to have a different ACL used for bucket creation. This also logs at INFO level the creation and deletion of buckets.
2019-01-16 18:23:37 +01:00
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
s3: set ACL for server side copies to that provided by the user - fixes #2691 Before this change the ACL for objects which were server side copied was left at the default "private" settings. S3 doesn't copy the ACL from the source when you copy an object, you have to set it afresh which is what this does.
2018-10-25 23:19:36 +02:00
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
docs: spelling: server-side Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-13 23:43:40 +02:00
Note that this ACL is applied when server-side copying objects as S3
s3: if bucket or object ACL is empty string then don't add X-Amz-Acl: header - fixes #5730 Before this fix it was impossible to stop rclone generating an X-Amx-Acl: header which is incompatible with GCS with uniform access control and is generally deprecated at AWS.
2022-09-14 17:24:57 +02:00
doesn't copy the ACL from the source but rather writes a fresh one.
If the acl is an empty string then no X-Amz-Acl: header is added and
the default (private) will be used.
`,
s3: add synology to s3 provider list
2023-07-06 11:54:07 +02:00
Provider: "!Storj,Synology,Cloudflare",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Examples: []fs.OptionExample{{
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
Value: "default",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets Full_CONTROL.\nNo one else has access rights (default).",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
Provider: "TencentCOS",
}, {
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Value: "private",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nNo one else has access rights (default).",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
Provider: "!IBMCOS,TencentCOS",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "public-read",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ access.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "!IBMCOS",
}, {
Value: "public-read-write",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ and WRITE access.\nGranting this on a bucket is generally not recommended.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "!IBMCOS",
}, {
Value: "authenticated-read",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nThe AuthenticatedUsers group gets READ access.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "!IBMCOS",
}, {
Value: "bucket-owner-read",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Object owner gets FULL_CONTROL.\nBucket owner gets READ access.\nIf you specify this canned ACL when creating a bucket, Amazon S3 ignores it.",
s3: Add ChinaMobile EOS to provider list China Mobile Ecloud Elastic Object Storage (EOS) is a cloud object storage service, and is fully compatible with S3. Fixes #6054
2022-03-24 12:57:00 +01:00
Provider: "!IBMCOS,ChinaMobile",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "bucket-owner-full-control",
Help: "Both the object owner and the bucket owner get FULL_CONTROL over the object.\nIf you specify this canned ACL when creating a bucket, Amazon S3 ignores it.",
s3: Add ChinaMobile EOS to provider list China Mobile Ecloud Elastic Object Storage (EOS) is a cloud object storage service, and is fully compatible with S3. Fixes #6054
2022-03-24 12:57:00 +01:00
Provider: "!IBMCOS,ChinaMobile",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "private",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nNo one else has access rights (default).\nThis acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS.",
config,s3: hierarchical configuration support #2140 This introduces a method of making provider specific configuration within a remote. This is useful particularly in s3. This commit does the basic configuration in S3 for IBM COS.
2018-04-12 18:05:53 +02:00
Provider: "IBMCOS",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "public-read",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ access.\nThis acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "IBMCOS",
}, {
Value: "public-read-write",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ and WRITE access.\nThis acl is available on IBM Cloud (Infra), On-Premise IBM COS.",
config,s3: hierarchical configuration support #2140 This introduces a method of making provider specific configuration within a remote. This is useful particularly in s3. This commit does the basic configuration in S3 for IBM COS.
2018-04-12 18:05:53 +02:00
Provider: "IBMCOS",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Value: "authenticated-read",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nThe AuthenticatedUsers group gets READ access.\nNot supported on Buckets.\nThis acl is available on IBM Cloud (Infra) and On-Premise IBM COS.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "IBMCOS",
}},
s3: add --s3-bucket-acl to control bucket ACL - fixes #2918 Before this change buckets were created with the same ACL as objects. After this change, the user can set just --s3-acl to set the ACL of buckets and objects, or use --s3-bucket-acl as well to have a different ACL used for bucket creation. This also logs at INFO level the creation and deletion of buckets.
2019-01-16 18:23:37 +01:00
}, {
Name: "bucket_acl",
Help: `Canned ACL used when creating buckets.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when only when creating buckets. If it
s3: if bucket or object ACL is empty string then don't add X-Amz-Acl: header - fixes #5730 Before this fix it was impossible to stop rclone generating an X-Amx-Acl: header which is incompatible with GCS with uniform access control and is generally deprecated at AWS.
2022-09-14 17:24:57 +02:00
isn't set then "acl" is used instead.
If the "acl" and "bucket_acl" are empty strings then no X-Amz-Acl:
header is added and the default (private) will be used.
`,
s3: add --s3-bucket-acl to control bucket ACL - fixes #2918 Before this change buckets were created with the same ACL as objects. After this change, the user can set just --s3-acl to set the ACL of buckets and objects, or use --s3-bucket-acl as well to have a different ACL used for bucket creation. This also logs at INFO level the creation and deletion of buckets.
2019-01-16 18:23:37 +01:00
Advanced: true,
Examples: []fs.OptionExample{{
Value: "private",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nNo one else has access rights (default).",
s3: add --s3-bucket-acl to control bucket ACL - fixes #2918 Before this change buckets were created with the same ACL as objects. After this change, the user can set just --s3-acl to set the ACL of buckets and objects, or use --s3-bucket-acl as well to have a different ACL used for bucket creation. This also logs at INFO level the creation and deletion of buckets.
2019-01-16 18:23:37 +01:00
}, {
Value: "public-read",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ access.",
s3: add --s3-bucket-acl to control bucket ACL - fixes #2918 Before this change buckets were created with the same ACL as objects. After this change, the user can set just --s3-acl to set the ACL of buckets and objects, or use --s3-bucket-acl as well to have a different ACL used for bucket creation. This also logs at INFO level the creation and deletion of buckets.
2019-01-16 18:23:37 +01:00
}, {
Value: "public-read-write",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ and WRITE access.\nGranting this on a bucket is generally not recommended.",
s3: add --s3-bucket-acl to control bucket ACL - fixes #2918 Before this change buckets were created with the same ACL as objects. After this change, the user can set just --s3-acl to set the ACL of buckets and objects, or use --s3-bucket-acl as well to have a different ACL used for bucket creation. This also logs at INFO level the creation and deletion of buckets.
2019-01-16 18:23:37 +01:00
}, {
Value: "authenticated-read",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Owner gets FULL_CONTROL.\nThe AuthenticatedUsers group gets READ access.",
s3: add --s3-bucket-acl to control bucket ACL - fixes #2918 Before this change buckets were created with the same ACL as objects. After this change, the user can set just --s3-acl to set the ACL of buckets and objects, or use --s3-bucket-acl as well to have a different ACL used for bucket creation. This also logs at INFO level the creation and deletion of buckets.
2019-01-16 18:23:37 +01:00
}},
s3: add requester pays option - fixes #301
2020-12-03 03:30:06 +01:00
}, {
Name: "requester_pays",
Help: "Enables requester pays option when interacting with S3 bucket.",
Provider: "AWS",
Default: false,
Advanced: true,
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Name: "server_side_encryption",
Help: "The server-side encryption algorithm used when storing this object in S3.",
s3: update Arvancloud default values and correct docs
2022-05-02 17:04:01 +02:00
Provider: "AWS,Ceph,ChinaMobile,Minio",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
}, {
Value: "AES256",
Help: "AES256",
s3: add support for KMS Key ID - fixes #2217 This code supports aws:kms and the kms key id for the s3 backend.
2018-08-30 18:08:27 +02:00
}, {
s3: Add ChinaMobile EOS to provider list China Mobile Ecloud Elastic Object Storage (EOS) is a cloud object storage service, and is fully compatible with S3. Fixes #6054
2022-03-24 12:57:00 +01:00
Value: "aws:kms",
Help: "aws:kms",
Provider: "!ChinaMobile",
s3: add support for KMS Key ID - fixes #2217 This code supports aws:kms and the kms key id for the s3 backend.
2018-08-30 18:08:27 +02:00
}},
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
}, {
Name: "sse_customer_algorithm",
Help: "If using SSE-C, the server-side encryption algorithm used when storing this object in S3.",
s3: update Arvancloud default values and correct docs
2022-05-02 17:04:01 +02:00
Provider: "AWS,Ceph,ChinaMobile,Minio",
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
}, {
Value: "AES256",
Help: "AES256",
}},
s3: add support for KMS Key ID - fixes #2217 This code supports aws:kms and the kms key id for the s3 backend.
2018-08-30 18:08:27 +02:00
}, {
Name: "sse_kms_key_id",
Help: "If using KMS ID you must provide the ARN of Key.",
s3: update Arvancloud default values and correct docs
2022-05-02 17:04:01 +02:00
Provider: "AWS,Ceph,Minio",
s3: add support for KMS Key ID - fixes #2217 This code supports aws:kms and the kms key id for the s3 backend.
2018-08-30 18:08:27 +02:00
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
}, {
Value: "arn:aws:kms:us-east-1:*",
Help: "arn:aws:kms:*",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}},
rclone config redacted: implement support mechanism for showing redacted config This introduces a new fs.Option flag, Sensitive and uses this along with IsPassword to redact the info in the config file for support purposes. It adds this flag into backends where appropriate. It was necessary to add oauthutil.SharedOptions to some backends as they were missing them. Fixes #5209
2023-07-06 18:55:53 +02:00
Sensitive: true,
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
}, {
s3: add --s3-sse-customer-key-base64 to supply keys with binary data Fixes #6400
2022-09-17 18:28:44 +02:00
Name: "sse_customer_key",
Help: `To use SSE-C you may provide the secret encryption key used to encrypt/decrypt your data.
Alternatively you can provide --sse-customer-key-base64.`,
Provider: "AWS,Ceph,ChinaMobile,Minio",
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
}},
rclone config redacted: implement support mechanism for showing redacted config This introduces a new fs.Option flag, Sensitive and uses this along with IsPassword to redact the info in the config file for support purposes. It adds this flag into backends where appropriate. It was necessary to add oauthutil.SharedOptions to some backends as they were missing them. Fixes #5209
2023-07-06 18:55:53 +02:00
Sensitive: true,
s3: add --s3-sse-customer-key-base64 to supply keys with binary data Fixes #6400
2022-09-17 18:28:44 +02:00
}, {
Name: "sse_customer_key_base64",
Help: `If using SSE-C you must provide the secret encryption key encoded in base64 format to encrypt/decrypt your data.
Alternatively you can provide --sse-customer-key.`,
s3: update Arvancloud default values and correct docs
2022-05-02 17:04:01 +02:00
Provider: "AWS,Ceph,ChinaMobile,Minio",
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
}},
rclone config redacted: implement support mechanism for showing redacted config This introduces a new fs.Option flag, Sensitive and uses this along with IsPassword to redact the info in the config file for support purposes. It adds this flag into backends where appropriate. It was necessary to add oauthutil.SharedOptions to some backends as they were missing them. Fixes #5209
2023-07-06 18:55:53 +02:00
Sensitive: true,
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
}, {
s3: complete sse-c implementation This now can complete all operations with SSE-C enabled. Fixes #2827 See: https://forum.rclone.org/t/issues-with-aws-s3-sse-c-getting-strange-log-entries-and-errors/20553
2020-11-20 12:15:48 +01:00
Name: "sse_customer_key_md5",
Help: `If using SSE-C you may provide the secret encryption key MD5 checksum (optional).
If you leave it blank, this is calculated automatically from the sse_customer_key provided.
`,
s3: update Arvancloud default values and correct docs
2022-05-02 17:04:01 +02:00
Provider: "AWS,Ceph,ChinaMobile,Minio",
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
}},
rclone config redacted: implement support mechanism for showing redacted config This introduces a new fs.Option flag, Sensitive and uses this along with IsPassword to redact the info in the config file for support purposes. It adds this flag into backends where appropriate. It was necessary to add oauthutil.SharedOptions to some backends as they were missing them. Fixes #5209
2023-07-06 18:55:53 +02:00
Sensitive: true,
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}, {
Name: "storage_class",
docs: move documentation for options from docs/content into backends In the following commit, the documentation will be autogenerated.
2018-10-01 19:36:15 +02:00
Help: "The storage class to use when storing new objects in S3.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
Provider: "AWS",
Examples: []fs.OptionExample{{
Value: "",
Help: "Default",
}, {
Value: "STANDARD",
Help: "Standard storage class",
}, {
Value: "REDUCED_REDUNDANCY",
Help: "Reduced redundancy storage class",
}, {
Value: "STANDARD_IA",
Help: "Standard Infrequent Access storage class",
}, {
Value: "ONEZONE_IA",
Help: "One Zone Infrequent Access storage class",
s3: add GLACIER storage class Fixes #923
2018-12-06 22:53:05 +01:00
}, {
Value: "GLACIER",
Help: "Glacier storage class",
s3: add support for "Glacier Deep Archive" storage class - fixes #3088
2019-04-05 05:14:05 +02:00
}, {
Value: "DEEP_ARCHIVE",
Help: "Glacier Deep Archive storage class",
s3: Add INTELLIGENT_TIERING storage class For Intelligent-Tiering: https://aws.amazon.com/s3/storage-classes/#Unknown_or_changing_access
2019-04-04 19:04:28 +02:00
}, {
Value: "INTELLIGENT_TIERING",
Help: "Intelligent-Tiering storage class",
s3: Add GLACIER_IR storage class
2021-12-02 06:55:57 +01:00
}, {
Value: "GLACIER_IR",
Help: "Glacier Instant Retrieval storage class",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 17:08:00 +02:00
}},
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
}, {
s3: Add Alibaba OSS to integration tests and fix storage classes
2019-01-12 21:41:47 +01:00
// Mapping from here: https://www.alibabacloud.com/help/doc-detail/64919.htm
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
Name: "storage_class",
Help: "The storage class to use when storing new objects in OSS.",
Provider: "Alibaba",
Examples: []fs.OptionExample{{
s3: Add Alibaba OSS to integration tests and fix storage classes
2019-01-12 21:41:47 +01:00
Value: "",
Help: "Default",
}, {
Value: "STANDARD",
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
Help: "Standard storage class",
}, {
s3: Add Alibaba OSS to integration tests and fix storage classes
2019-01-12 21:41:47 +01:00
Value: "GLACIER",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Archive storage mode",
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
}, {
s3: Add Alibaba OSS to integration tests and fix storage classes
2019-01-12 21:41:47 +01:00
Value: "STANDARD_IA",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Infrequent access storage mode",
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
}},
s3: Add ChinaMobile EOS to provider list China Mobile Ecloud Elastic Object Storage (EOS) is a cloud object storage service, and is fully compatible with S3. Fixes #6054
2022-03-24 12:57:00 +01:00
}, {
// Mapping from here: https://ecloud.10086.cn/op-help-center/doc/article/24495
Name: "storage_class",
Help: "The storage class to use when storing new objects in ChinaMobile.",
Provider: "ChinaMobile",
Examples: []fs.OptionExample{{
Value: "",
Help: "Default",
}, {
Value: "STANDARD",
Help: "Standard storage class",
}, {
Value: "GLACIER",
Help: "Archive storage mode",
}, {
Value: "STANDARD_IA",
Help: "Infrequent access storage mode",
}},
s3: add Liara LOS to provider list
2022-12-06 13:25:23 +01:00
}, {
// Mapping from here: https://liara.ir/landing/object-storage
Name: "storage_class",
Help: "The storage class to use when storing new objects in Liara",
Provider: "Liara",
Examples: []fs.OptionExample{{
Value: "STANDARD",
Help: "Standard storage class",
}},
s3: Add ArvanCloud AOS to provider list
2022-04-28 11:42:30 +02:00
}, {
s3: Fix Arvancloud Domain and region changes and alphabetise the provider
2023-06-17 13:47:23 +02:00
// Mapping from here: https://www.arvancloud.ir/en/products/cloud-storage
s3: Add ArvanCloud AOS to provider list
2022-04-28 11:42:30 +02:00
Name: "storage_class",
Help: "The storage class to use when storing new objects in ArvanCloud.",
Provider: "ArvanCloud",
Examples: []fs.OptionExample{{
Value: "STANDARD",
Help: "Standard storage class",
}},
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
// Mapping from here: https://intl.cloud.tencent.com/document/product/436/30925
Name: "storage_class",
Help: "The storage class to use when storing new objects in Tencent COS.",
Provider: "TencentCOS",
Examples: []fs.OptionExample{{
Value: "",
Help: "Default",
}, {
Value: "STANDARD",
Help: "Standard storage class",
}, {
Value: "ARCHIVE",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Archive storage mode",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}, {
Value: "STANDARD_IA",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Infrequent access storage mode",
docs: add Tencent COS to s3 provider list - fixes #4468 * add Tencent COS to s3 provider list. Co-authored-by: wjielai <wjielai@tencent.com>
2020-09-08 17:34:25 +02:00
}},
s3: add Scaleway provider - fixes #4338
2020-06-12 17:04:16 +02:00
}, {
s3: update Scaleway storage classes There are now 3 classes: * "STANDARD" - Multi-AZ, all regions * "ONEZONE_IA" - Single-AZ, FR-PAR only * "GLACIER" - Archive, FR-PAR and NL-AMS only
2023-04-18 23:21:35 +02:00
// Mapping from here: https://www.scaleway.com/en/docs/storage/object/quickstart/
s3: add Scaleway provider - fixes #4338
2020-06-12 17:04:16 +02:00
Name: "storage_class",
Help: "The storage class to use when storing new objects in S3.",
Provider: "Scaleway",
Examples: []fs.OptionExample{{
Value: "",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: "Default.",
s3: add Scaleway provider - fixes #4338
2020-06-12 17:04:16 +02:00
}, {
Value: "STANDARD",
s3: update Scaleway storage classes There are now 3 classes: * "STANDARD" - Multi-AZ, all regions * "ONEZONE_IA" - Single-AZ, FR-PAR only * "GLACIER" - Archive, FR-PAR and NL-AMS only
2023-04-18 23:21:35 +02:00
Help: "The Standard class for any upload.\nSuitable for on-demand content like streaming or CDN.\nAvailable in all regions.",
s3: add Scaleway provider - fixes #4338
2020-06-12 17:04:16 +02:00
}, {
Value: "GLACIER",
s3: update Scaleway storage classes There are now 3 classes: * "STANDARD" - Multi-AZ, all regions * "ONEZONE_IA" - Single-AZ, FR-PAR only * "GLACIER" - Archive, FR-PAR and NL-AMS only
2023-04-18 23:21:35 +02:00
Help: "Archived storage.\nPrices are lower, but it needs to be restored first to be accessed.\nAvailable in FR-PAR and NL-AMS regions.",
}, {
Value: "ONEZONE_IA",
Help: "One Zone - Infrequent Access.\nA good choice for storing secondary backup copies or easily re-creatable data.\nAvailable in the FR-PAR region only.",
s3: add Scaleway provider - fixes #4338
2020-06-12 17:04:16 +02:00
}},
s3: add Qiniu KODO to s3 provider list - fixes #6195
2022-10-13 16:49:22 +02:00
}, {
// Mapping from here: https://developer.qiniu.com/kodo/5906/storage-type
Name: "storage_class",
Help: "The storage class to use when storing new objects in Qiniu.",
Provider: "Qiniu",
Examples: []fs.OptionExample{{
Value: "STANDARD",
Help: "Standard storage class",
}, {
Value: "LINE",
Help: "Infrequent access storage mode",
}, {
Value: "GLACIER",
Help: "Archive storage mode",
}, {
Value: "DEEP_ARCHIVE",
Help: "Deep archive storage mode",
}},
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
}, {
Name: "upload_cutoff",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: `Cutoff for switching to chunked upload.
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
Any files larger than this will be uploaded in chunks of chunk_size.
Use binary prefixes for size and rate units Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M. Extends binary suffix output with letter i, e.g. Ki and Mi. Centralizes creation of bit/byte unit strings.
2021-03-02 20:11:57 +01:00
The minimum is 0 and the maximum is 5 GiB.`,
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
Default: defaultUploadCutoff,
Advanced: true,
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
}, {
docs: move documentation for options from docs/content into backends In the following commit, the documentation will be autogenerated.
2018-10-01 19:36:15 +02:00
Name: "chunk_size",
Help: `Chunk size to use for uploading.
s3: Reduce memory usage streaming files by reducing max stream upload size Before this change rclone would allow the user to stream (eg with rclone mount, rclone rcat or uploading google photos or docs) 5TB files. This meant that rclone allocated 4 * 525 MB buffers per transfer which is way too much memory by default. This change makes rclone use the configured chunk size for streamed uploads. This is 5MB by default which means that rclone can stream upload files up to 48GB by default staying below the 10,000 chunks limit. This can be increased with --s3-chunk-size if necessary. If rclone detects that a file is being streamed to s3 it will make a single NOTICE level log stating the limitation. This fixes the enormous memory usage. Fixes #3568 See: https://forum.rclone.org/t/how-much-memory-does-rclone-need/12743
2019-11-06 11:41:03 +01:00
When uploading files larger than upload_cutoff or files with unknown
docs: spelling: e.g. Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-13 23:49:58 +02:00
size (e.g. from "rclone rcat" or uploaded with "rclone mount" or google
s3: Reduce memory usage streaming files by reducing max stream upload size Before this change rclone would allow the user to stream (eg with rclone mount, rclone rcat or uploading google photos or docs) 5TB files. This meant that rclone allocated 4 * 525 MB buffers per transfer which is way too much memory by default. This change makes rclone use the configured chunk size for streamed uploads. This is 5MB by default which means that rclone can stream upload files up to 48GB by default staying below the 10,000 chunks limit. This can be increased with --s3-chunk-size if necessary. If rclone detects that a file is being streamed to s3 it will make a single NOTICE level log stating the limitation. This fixes the enormous memory usage. Fixes #3568 See: https://forum.rclone.org/t/how-much-memory-does-rclone-need/12743
2019-11-06 11:41:03 +01:00
photos or google docs) they will be uploaded as multipart uploads
using this chunk size.
docs: move documentation for options from docs/content into backends In the following commit, the documentation will be autogenerated.
2018-10-01 19:36:15 +02:00
Note that "--s3-upload-concurrency" chunks of this size are buffered
in memory per transfer.
docs: spelling: high-speed Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-13 23:50:53 +02:00
If you are transferring large files over high-speed links and you have
s3: Reduce memory usage streaming files by reducing max stream upload size Before this change rclone would allow the user to stream (eg with rclone mount, rclone rcat or uploading google photos or docs) 5TB files. This meant that rclone allocated 4 * 525 MB buffers per transfer which is way too much memory by default. This change makes rclone use the configured chunk size for streamed uploads. This is 5MB by default which means that rclone can stream upload files up to 48GB by default staying below the 10,000 chunks limit. This can be increased with --s3-chunk-size if necessary. If rclone detects that a file is being streamed to s3 it will make a single NOTICE level log stating the limitation. This fixes the enormous memory usage. Fixes #3568 See: https://forum.rclone.org/t/how-much-memory-does-rclone-need/12743
2019-11-06 11:41:03 +01:00
enough memory, then increasing this will speed up the transfers.
Rclone will automatically increase the chunk size when uploading a
large file of known size to stay below the 10,000 chunks limit.
Files of unknown size are uploaded with the configured
Use binary prefixes for size and rate units Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M. Extends binary suffix output with letter i, e.g. Ki and Mi. Centralizes creation of bit/byte unit strings.
2021-03-02 20:11:57 +01:00
chunk_size. Since the default chunk size is 5 MiB and there can be at
s3: Reduce memory usage streaming files by reducing max stream upload size Before this change rclone would allow the user to stream (eg with rclone mount, rclone rcat or uploading google photos or docs) 5TB files. This meant that rclone allocated 4 * 525 MB buffers per transfer which is way too much memory by default. This change makes rclone use the configured chunk size for streamed uploads. This is 5MB by default which means that rclone can stream upload files up to 48GB by default staying below the 10,000 chunks limit. This can be increased with --s3-chunk-size if necessary. If rclone detects that a file is being streamed to s3 it will make a single NOTICE level log stating the limitation. This fixes the enormous memory usage. Fixes #3568 See: https://forum.rclone.org/t/how-much-memory-does-rclone-need/12743
2019-11-06 11:41:03 +01:00
most 10,000 chunks, this means that by default the maximum size of
Use binary prefixes for size and rate units Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M. Extends binary suffix output with letter i, e.g. Ki and Mi. Centralizes creation of bit/byte unit strings.
2021-03-02 20:11:57 +01:00
a file you can stream upload is 48 GiB. If you wish to stream upload
docs/s3: add note about chunk size decreasing progress accuracy
2022-06-16 22:29:36 +02:00
larger files then you will need to increase chunk_size.
Increasing the chunk size decreases the accuracy of the progress
statistics displayed with "-P" flag. Rclone treats chunk as sent when
it's buffered by the AWS SDK, when in fact it may still be uploading.
A bigger chunk size means a bigger AWS SDK buffer and progress
reporting more deviating from the truth.
`,
backend: add fstests.ChunkedUploadConfig - azureblob - b2 - drive - dropbox - onedrive - s3 - swift
2018-09-07 13:02:27 +02:00
Default: minChunkSize,
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
Advanced: true,
s3: add `max_upload_parts` support - fixes #4159 * s3: add `max_upload_parts` support This allows to configure a maximum amount of chunks used to upload file: - Support Scaleway which has a limit of 1k chunks currently - Reduce a cost on S3 when each request costs some money at the expense of memory used Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2020-06-08 19:22:34 +02:00
}, {
Name: "max_upload_parts",
Help: `Maximum number of parts in a multipart upload.
This option defines the maximum number of multipart chunks to use
when doing a multipart upload.
This can be useful if a service does not support the AWS S3
specification of 10,000 chunks.
Rclone will automatically increase the chunk size when uploading a
large file of a known size to stay below this number of chunks limit.
`,
Default: maxUploadParts,
Advanced: true,
s3: add --s3-copy-cutoff for size to switch to multipart copy Before this change we used the same (relatively low limits) for server side copy as we did for multipart uploads. It doesn't make sense to use the same limits since no data is being downloaded or uploaded for a server side copy. This change introduces a new parameter --s3-copy-cutoff to control when the switch from single to multipart server size copy happens and defaults it to the maximum 5GB. This makes server side copies much more efficient. It also fixes the erroneous error when trying to set the modification time of a file bigger than 5GB. See #3778
2019-12-02 18:14:57 +01:00
}, {
Name: "copy_cutoff",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: `Cutoff for switching to multipart copy.
s3: add --s3-copy-cutoff for size to switch to multipart copy Before this change we used the same (relatively low limits) for server side copy as we did for multipart uploads. It doesn't make sense to use the same limits since no data is being downloaded or uploaded for a server side copy. This change introduces a new parameter --s3-copy-cutoff to control when the switch from single to multipart server size copy happens and defaults it to the maximum 5GB. This makes server side copies much more efficient. It also fixes the erroneous error when trying to set the modification time of a file bigger than 5GB. See #3778
2019-12-02 18:14:57 +01:00
docs: spelling: server-side Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-13 23:43:40 +02:00
Any files larger than this that need to be server-side copied will be
s3: add --s3-copy-cutoff for size to switch to multipart copy Before this change we used the same (relatively low limits) for server side copy as we did for multipart uploads. It doesn't make sense to use the same limits since no data is being downloaded or uploaded for a server side copy. This change introduces a new parameter --s3-copy-cutoff to control when the switch from single to multipart server size copy happens and defaults it to the maximum 5GB. This makes server side copies much more efficient. It also fixes the erroneous error when trying to set the modification time of a file bigger than 5GB. See #3778
2019-12-02 18:14:57 +01:00
copied in chunks of this size.
Use binary prefixes for size and rate units Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M. Extends binary suffix output with letter i, e.g. Ki and Mi. Centralizes creation of bit/byte unit strings.
2021-03-02 20:11:57 +01:00
The minimum is 0 and the maximum is 5 GiB.`,
s3: add --s3-copy-cutoff for size to switch to multipart copy Before this change we used the same (relatively low limits) for server side copy as we did for multipart uploads. It doesn't make sense to use the same limits since no data is being downloaded or uploaded for a server side copy. This change introduces a new parameter --s3-copy-cutoff to control when the switch from single to multipart server size copy happens and defaults it to the maximum 5GB. This makes server side copies much more efficient. It also fixes the erroneous error when trying to set the modification time of a file bigger than 5GB. See #3778
2019-12-02 18:14:57 +01:00
Default: fs.SizeSuffix(maxSizeForCopy),
Advanced: true,
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
}, {
s3: improve docs for --s3-disable-checksum
2020-04-23 20:47:48 +02:00
Name: "disable_checksum",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: `Don't store MD5 checksum with object metadata.
s3: improve docs for --s3-disable-checksum
2020-04-23 20:47:48 +02:00
Normally rclone will calculate the MD5 checksum of the input before
uploading it so it can add it to metadata on the object. This is great
for data integrity checking but can cause long delays for large files
Revert "s3: fix InvalidRequest copying to a locked bucket from a source with no MD5SUM" This reverts commit e5a1bcb1ce771bc80fee0072565bb4bfa1e86dca. This causes a lot of integration test failures so may need to be optional.
2023-03-21 12:43:35 +01:00
to start uploading.`,
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
Default: false,
Advanced: true,
s3: implement `profile` and `shared_credentials_file` options It is impossible to use two different profiles at the same time - these config vars enable that. See: https://forum.rclone.org/t/s3-source-destination-named-profile/17417
2020-06-26 12:14:40 +02:00
}, {
Name: "shared_credentials_file",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: `Path to the shared credentials file.
s3: implement `profile` and `shared_credentials_file` options It is impossible to use two different profiles at the same time - these config vars enable that. See: https://forum.rclone.org/t/s3-source-destination-named-profile/17417
2020-06-26 12:14:40 +02:00
If env_auth = true then rclone can use a shared credentials file.
If this variable is empty rclone will look for the
"AWS_SHARED_CREDENTIALS_FILE" env variable. If the env value is empty
it will default to the current user's home directory.
Linux/OSX: "$HOME/.aws/credentials"
Windows: "%USERPROFILE%\.aws\credentials"
`,
Advanced: true,
}, {
Name: "profile",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: `Profile to use in the shared credentials file.
s3: implement `profile` and `shared_credentials_file` options It is impossible to use two different profiles at the same time - these config vars enable that. See: https://forum.rclone.org/t/s3-source-destination-named-profile/17417
2020-06-26 12:14:40 +02:00
If env_auth = true then rclone can use a shared credentials file. This
variable controls which profile is used in that file.
If empty it will default to the environment variable "AWS_PROFILE" or
"default" if that environment variable is also not set.
`,
Advanced: true,
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
}, {
rclone config redacted: implement support mechanism for showing redacted config This introduces a new fs.Option flag, Sensitive and uses this along with IsPassword to redact the info in the config file for support purposes. It adds this flag into backends where appropriate. It was necessary to add oauthutil.SharedOptions to some backends as they were missing them. Fixes #5209
2023-07-06 18:55:53 +02:00
Name: "session_token",
Help: "An AWS session token.",
Advanced: true,
Sensitive: true,
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
}, {
docs: move documentation for options from docs/content into backends In the following commit, the documentation will be autogenerated.
2018-10-01 19:36:15 +02:00
Name: "upload_concurrency",
Help: `Concurrency for multipart uploads.
This is the number of chunks of the same file that are uploaded
concurrently.
docs: grammar: uncountable: links Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-13 23:59:13 +02:00
If you are uploading small numbers of large files over high-speed links
docs: move documentation for options from docs/content into backends In the following commit, the documentation will be autogenerated.
2018-10-01 19:36:15 +02:00
and these uploads do not fully utilize your bandwidth, then increasing
this may help to speed up the transfers.`,
s3: change --s3-upload-concurrency default to 4 to increase perfomance #2772 Increasing the --s3-upload-concurrency to 4 (from 2) gives an additional 45% throughput at the cost of 10MB extra memory per transfer. After testing the upload perfoc
2018-12-02 18:51:14 +01:00
Default: 4,
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
Advanced: true,
s3: Add --s3-force-path-style - fixes #2401
2018-07-18 17:40:59 +02:00
}, {
docs: move documentation for options from docs/content into backends In the following commit, the documentation will be autogenerated.
2018-10-01 19:36:15 +02:00
Name: "force_path_style",
Help: `If true use path style access if false use virtual hosted style.
If this is true (the default) then rclone will use path style access,
if false then rclone will use virtual path style. See [the AWS S3
docs](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro)
for more info.
docs: grammar: Oxford comma Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-13 23:43:00 +02:00
Some providers (e.g. AWS, Aliyun OSS, Netease COS, or Tencent COS) require this set to
s3: force path style bucket access to off for AWS deprecation AWS are deprecating path style bucket access so rclone should stop using it by default for this provider. This change shouldn't break any workflows as all AWS endpoints support virtual hosted style lookups of buckets. It may even improve performance. See: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/
2020-01-05 18:53:45 +01:00
false - rclone will do this automatically based on the provider
setting.`,
s3: Add --s3-force-path-style - fixes #2401
2018-07-18 17:40:59 +02:00
Default: true,
Advanced: true,
s3: make --s3-v2-auth flag This is an alternative to setting the region to "other-v2-signature" which is inconvenient for multi-region providers.
2018-10-09 14:03:37 +02:00
}, {
Name: "v2_auth",
Help: `If true use v2 authentication.
If this is false (the default) then rclone will use v4 authentication.
If it is set then rclone will use v2 authentication.
docs: spelling: e.g. Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-13 23:49:58 +02:00
Use this only if v4 signatures don't work, e.g. pre Jewel/v10 CEPH.`,
s3: make --s3-v2-auth flag This is an alternative to setting the region to "other-v2-signature" which is inconvenient for multi-region providers.
2018-10-09 14:03:37 +02:00
Default: false,
Advanced: true,
s3: suppport S3 Accelerated endpoints with --s3-use-accelerate-endpoint Fixes #3123
2019-04-26 11:19:00 +02:00
}, {
Name: "use_accelerate_endpoint",
Provider: "AWS",
Help: `If true use the AWS S3 accelerated endpoint.
See: [AWS S3 Transfer acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration-examples.html)`,
Default: false,
Advanced: true,
s3: add option for multipart failiure behaviour This is needed for resuming uploads across different sessions.
2019-07-24 11:03:38 +02:00
}, {
Name: "leave_parts_on_error",
Provider: "AWS",
Help: `If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery.
It should be set to true for resuming uploads across different sessions.
WARNING: Storing parts of an incomplete multipart upload counts towards space usage on S3 and will add additional costs if not cleaned up.
`,
Default: false,
Advanced: true,
s3: introduce list_chunk option for bucket listing The S3 ListObject API returns paginated bucket listings, with "MaxKeys" items for each GET call. The default value is 1000 entries, but for buckets with millions of objects it might make sense to request more elements per request, if the backend supports it. This commit adds a "list_chunk" option for the user to specify a lower or higher value. This commit does not add safe guards around this value - if a user decides to request a too large list, it might result in connection timeouts (on the server or client). In AWS S3, there is a fixed limit of 1000, some other services might have one too. In Ceph, this can be configured in RadosGW.
2019-12-26 12:05:00 +01:00
}, {
Name: "list_chunk",
Help: `Size of listing chunk (response list for each ListObject S3 request).
This option is also known as "MaxKeys", "max-items", or "page-size" from the AWS S3 specification.
Most services truncate the response list to 1000 objects even if requested more than that.
In AWS S3 this is a global maximum and cannot be changed, see [AWS S3](https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html).
In Ceph, this can be increased with the "rgw list buckets max chunk" option.
`,
Default: 1000,
Advanced: true,
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
}, {
Name: "list_version",
Help: `Version of ListObjects to use: 1,2 or 0 for auto.
When S3 originally launched it only provided the ListObjects call to
enumerate objects in a bucket.
However in May 2016 the ListObjectsV2 call was introduced. This is
much higher performance and should be used if at all possible.
If set to the default, 0, rclone will guess according to the provider
set which list objects method to call. If it guesses wrong, then it
may be set manually here.
`,
Default: 0,
Advanced: true,
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
}, {
Name: "list_url_encode",
Help: `Whether to url encode listings: true/false/unset
Some providers support URL encoding listings and where this is
available this is more reliable when using control characters in file
names. If this is set to unset (the default) then rclone will choose
according to the provider setting what to apply, but you can override
rclone's choice here.
`,
Default: fs.Tristate{},
Advanced: true,
s3: add --s3-no-check-bucket for minimising rclone transactions and perms Fixes #4449
2020-07-22 13:02:17 +02:00
}, {
Name: "no_check_bucket",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: `If set, don't attempt to check the bucket exists or create it.
s3: add --s3-no-check-bucket for minimising rclone transactions and perms Fixes #4449
2020-07-22 13:02:17 +02:00
This can be useful when trying to minimise the number of transactions
rclone does if you know the bucket exists already.
s3: update help for --s3-no-check-bucket #4913
2021-01-10 18:53:32 +01:00
It can also be needed if the user you are using does not have bucket
creation permissions. Before v1.52.0 this would have passed silently
due to a bug.
s3: add --s3-no-head parameter to minimise transactions on upload See: https://forum.rclone.org/t/prevent-head-on-amazon-s3-family/21935
2021-01-29 12:13:42 +01:00
`,
Default: false,
Advanced: true,
}, {
Name: "no_head",
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: `If set, don't HEAD uploaded objects to check integrity.
s3: add --s3-no-head parameter to minimise transactions on upload See: https://forum.rclone.org/t/prevent-head-on-amazon-s3-family/21935
2021-01-29 12:13:42 +01:00
This can be useful when trying to minimise the number of transactions
rclone does.
Setting it means that if rclone receives a 200 OK message after
uploading an object with PUT then it will assume that it got uploaded
properly.
In particular it will assume:
- the metadata, including modtime, storage class and content type was as uploaded
- the size was as uploaded
It reads the following items from the response for a single part PUT:
- the MD5SUM
- The uploaded date
For multipart uploads these items aren't read.
If an source object of unknown length is uploaded then rclone **will** do a
HEAD request.
Setting this flag increases the chance for undetected upload failures,
in particular an incorrect size, so it isn't recommended for normal
operation. In practice the chance of an undetected upload failure is
very small even with this flag.
s3: add --s3-no-check-bucket for minimising rclone transactions and perms Fixes #4449
2020-07-22 13:02:17 +02:00
`,
Default: false,
Advanced: true,
s3: add --s3-no-head-object This stops rclone doing any HEAD requests on objects.
2021-04-28 12:05:54 +02:00
}, {
Name: "no_head_object",
azureblob: add --azureblob-no-head-object
2021-09-06 11:41:54 +02:00
Help: `If set, do not do HEAD before GET when getting objects.`,
s3: add --s3-no-head-object This stops rclone doing any HEAD requests on objects.
2021-04-28 12:05:54 +02:00
Default: false,
Advanced: true,
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
}, {
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
Advanced: true,
backends: move encoding definitions from fs/encodings
2020-01-14 22:51:49 +01:00
// Any UTF-8 character is valid in a key, however it can't handle
// invalid UTF-8 and / have a special meaning.
//
// The SDK can't seem to handle uploading files called '.'
//
// FIXME would be nice to add
// - initial / encoding
// - doubled / encoding
// - trailing / encoding
// so that AWS keys are always valid file names
backend/s3: use memory pool for buffer allocations Currently each multipart upload allocated his own buffers, which after file upload was garbaged. Next files couldn't leverage already allocated memory which resulted in inefficent memory management. This change introduces backend memory pool keeping memory chunks which can be used during object operations. Fixes #3967
2020-02-19 11:17:25 +01:00
Default: encoder.EncodeInvalidUtf8 |
backends: move encoding definitions from fs/encodings
2020-01-14 22:51:49 +01:00
encoder.EncodeSlash |
backend/s3: use memory pool for buffer allocations Currently each multipart upload allocated his own buffers, which after file upload was garbaged. Next files couldn't leverage already allocated memory which resulted in inefficent memory management. This change introduces backend memory pool keeping memory chunks which can be used during object operations. Fixes #3967
2020-02-19 11:17:25 +01:00
encoder.EncodeDot,
}, {
Name: "memory_pool_flush_time",
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
Default: fs.Duration(time.Minute),
backend/s3: use memory pool for buffer allocations Currently each multipart upload allocated his own buffers, which after file upload was garbaged. Next files couldn't leverage already allocated memory which resulted in inefficent memory management. This change introduces backend memory pool keeping memory chunks which can be used during object operations. Fixes #3967
2020-02-19 11:17:25 +01:00
Advanced: true,
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
Hide: fs.OptionHideBoth,
Help: `How often internal memory buffer pools will be flushed. (no longer used)`,
backend/s3: use memory pool for buffer allocations Currently each multipart upload allocated his own buffers, which after file upload was garbaged. Next files couldn't leverage already allocated memory which resulted in inefficent memory management. This change introduces backend memory pool keeping memory chunks which can be used during object operations. Fixes #3967
2020-02-19 11:17:25 +01:00
}, {
Name: "memory_pool_use_mmap",
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
Default: false,
backend/s3: use memory pool for buffer allocations Currently each multipart upload allocated his own buffers, which after file upload was garbaged. Next files couldn't leverage already allocated memory which resulted in inefficent memory management. This change introduces backend memory pool keeping memory chunks which can be used during object operations. Fixes #3967
2020-02-19 11:17:25 +01:00
Advanced: true,
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
Hide: fs.OptionHideBoth,
Help: `Whether to use mmap buffers in internal memory pool. (no longer used)`,
s3: Added `--s3-disable-http2` to disable http/2 Fixes #4673 Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-13 18:11:22 +02:00
}, {
Name: "disable_http2",
Default: false,
Advanced: true,
docs: punctuation cleanup See #5538
2021-08-16 11:30:01 +02:00
Help: `Disable usage of http2 for S3 backends.
s3: Added `--s3-disable-http2` to disable http/2 Fixes #4673 Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-13 18:11:22 +02:00
There is currently an unsolved issue with the s3 (specifically minio) backend
and HTTP/2. HTTP/2 is enabled by default for the s3 backend but can be
disabled here. When the issue is solved this flag will be removed.
See: https://github.com/rclone/rclone/issues/4673, https://github.com/rclone/rclone/issues/3631
`,
s3: add support to use CDN URL to download the file The egress charges while using a CloudFront CDN url is cheaper when compared to accessing the file directly from S3. So added a download URL advanced option, which when set downloads the file using it.
2021-10-14 12:19:38 +02:00
}, {
Name: "download_url",
Help: `Custom endpoint for downloads.
This is usually set to a CloudFront CDN URL as AWS S3 offers
cheaper egress for data downloaded through the CloudFront network.`,
Advanced: true,
s3: empty directory markers - #3453
2022-09-26 08:59:57 +02:00
}, {
Name: "directory_markers",
Default: false,
Advanced: true,
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
Help: `Upload an empty object with a trailing slash when a new directory is created
s3: empty directory markers - #3453
2022-09-26 08:59:57 +02:00
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
Empty folders are unsupported for bucket based remotes, this option creates an empty
object ending with "/", to persist the folder.
s3: empty directory markers - #3453
2022-09-26 08:59:57 +02:00
`,
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
}, {
Name: "use_multipart_etag",
Help: `Whether to use ETag in multipart uploads for verification
This should be true, false or left unset to use the default for the provider.
`,
Default: fs.Tristate{},
Advanced: true,
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
}, {
Name: "use_presigned_request",
Help: `Whether to use a presigned request or PutObject for single part uploads
If this is false rclone will use PutObject from the AWS SDK to upload
an object.
Versions of rclone < 1.59 use presigned requests to upload a single
part object and setting this flag to true will re-enable that
functionality. This shouldn't be necessary except in exceptional
circumstances or for testing.
`,
Default: false,
Advanced: true,
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
}, {
Name: "versions",
Help: "Include old versions in directory listings.",
Default: false,
Advanced: true,
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
}, {
Name: "version_at",
Help: `Show file versions as they were at the specified time.
The parameter should be a date, "2006-01-02", datetime "2006-01-02
15:04:05" or a duration for that long ago, eg "100d" or "1h".
Note that when using this no file write operations are permitted,
so you can't upload files or delete them.
See [the time option docs](/docs/#time-option) for valid formats.
`,
Default: fs.Time{},
Advanced: true,
s3: add --s3-decompress flag to download gzip-encoded files Before this change, if an object compressed with "Content-Encoding: gzip" was downloaded, a length and hash mismatch would occur since the go runtime automatically decompressed the object on download. If --s3-decompress is set, this change erases the length and hash on compressed objects so they can be downloaded successfully, at the cost of not being able to check the length or the hash of the downloaded object. If --s3-decompress is not set the compressed files will be downloaded as-is providing compressed objects with intact size and hash information. See #2658
2022-07-29 18:01:59 +02:00
}, {
Name: "decompress",
Help: `If set this will decompress gzip encoded objects.
It is possible to upload objects to S3 with "Content-Encoding: gzip"
all: fix spelling across the project * abcdefghijklmnopqrstuvwxyz * accounting * additional * allowed * almost * already * appropriately * arise * bandwidth * behave * bidirectional * brackets * cached * characters * cloud * committing * concatenating * configured * constructs * current * cutoff * deferred * different * directory * disposition * dropbox * either way * error * excess * experiments * explicitly * externally * files * github * gzipped * hierarchies * huffman * hyphen * implicitly * independent * insensitive * integrity * libraries * literally * metadata * mimics * missing * modification * multipart * multiple * nightmare * nonexistent * number * obscure * ourselves * overridden * potatoes * preexisting * priority * received * remote * replacement * represents * reproducibility * response * satisfies * sensitive * separately * separator * specifying * string * successful * synchronization * syncing * šenfeld * take * temporarily * testcontents * that * the * themselves * throttling * timeout * transaction * transferred * unnecessary * using * webbrowser * which * with * workspace Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-08-14 04:56:32 +02:00
set. Normally rclone will download these files as compressed objects.
s3: add --s3-decompress flag to download gzip-encoded files Before this change, if an object compressed with "Content-Encoding: gzip" was downloaded, a length and hash mismatch would occur since the go runtime automatically decompressed the object on download. If --s3-decompress is set, this change erases the length and hash on compressed objects so they can be downloaded successfully, at the cost of not being able to check the length or the hash of the downloaded object. If --s3-decompress is not set the compressed files will be downloaded as-is providing compressed objects with intact size and hash information. See #2658
2022-07-29 18:01:59 +02:00
If this flag is set then rclone will decompress these files with
"Content-Encoding: gzip" as they are received. This means that rclone
can't check the size and hash but the file contents will be decompressed.
`,
Advanced: true,
Default: false,
s3: add provider quirk --s3-might-gzip to fix corrupted on transfer: sizes differ Before this change, some files were giving this error when downloaded from Cloudflare and other providers. ERROR corrupted on transfer: sizes differ NNN vs MMM This is because these providers auto gzips the object when rclone wasn't expecting it to. (AWS does not gzip objects without their being uploaded gzipped). This patch adds a quirk to for fix the problem and a flag to control it. The quirk `might_gzip` is set to `true` for all providers except AWS. See: https://forum.rclone.org/t/s3-error-corrupted-on-transfer-sizes-differ-nnn-vs-mmm/33694/ Fixes: #6533
2022-10-29 00:23:29 +02:00
}, {
Name: "might_gzip",
Help: strings.ReplaceAll(`Set this if the backend might gzip objects.
Normally providers will not alter objects when they are downloaded. If
an object was not uploaded with |Content-Encoding: gzip| then it won't
be set on download.
However some providers may gzip objects even if they weren't uploaded
with |Content-Encoding: gzip| (eg Cloudflare).
A symptom of this would be receiving errors like
ERROR corrupted on transfer: sizes differ NNN vs MMM
If you set this flag and rclone downloads an object with
Content-Encoding: gzip set and chunked transfer encoding, then rclone
will decompress the object on the fly.
If this is set to unset (the default) then rclone will choose
according to the provider setting what to apply, but you can override
rclone's choice here.
s3: add GCS to provider list
2023-01-09 07:54:51 +01:00
`, "|", "`"),
Default: fs.Tristate{},
Advanced: true,
}, {
Name: "use_accept_encoding_gzip",
Help: strings.ReplaceAll(`Whether to send |Accept-Encoding: gzip| header.
By default, rclone will append |Accept-Encoding: gzip| to the request to download
compressed objects whenever possible.
However some providers such as Google Cloud Storage may alter the HTTP headers, breaking
the signature of the request.
A symptom of this would be receiving errors like
SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided.
In this case, you might want to try disabling this option.
s3: add provider quirk --s3-might-gzip to fix corrupted on transfer: sizes differ Before this change, some files were giving this error when downloaded from Cloudflare and other providers. ERROR corrupted on transfer: sizes differ NNN vs MMM This is because these providers auto gzips the object when rclone wasn't expecting it to. (AWS does not gzip objects without their being uploaded gzipped). This patch adds a quirk to for fix the problem and a flag to control it. The quirk `might_gzip` is set to `true` for all providers except AWS. See: https://forum.rclone.org/t/s3-error-corrupted-on-transfer-sizes-differ-nnn-vs-mmm/33694/ Fixes: #6533
2022-10-29 00:23:29 +02:00
`, "|", "`"),
Default: fs.Tristate{},
Advanced: true,
s3: add --s3-no-system-metadata to suppress read and write of system metadata See: https://forum.rclone.org/t/problems-with-content-disposition-and-backblaze-b2-using-s3/33292/
2022-10-12 09:55:58 +02:00
}, {
Name: "no_system_metadata",
Help: `Suppress setting and reading of system metadata`,
Advanced: true,
Default: false,
s3: add --s3-sts-endpoint to specify STS endpoint See: https://forum.rclone.org/t/s3-profile-failing-when-explicit-s3-endpoint-is-present/36063/
2023-03-02 10:56:09 +01:00
}, {
Name: "sts_endpoint",
Help: "Endpoint for STS.\n\nLeave blank if using AWS to use the default endpoint for the region.",
Provider: "AWS",
Advanced: true,
backend/s3: use memory pool for buffer allocations Currently each multipart upload allocated his own buffers, which after file upload was garbaged. Next files couldn't leverage already allocated memory which resulted in inefficent memory management. This change introduces backend memory pool keeping memory chunks which can be used during object operations. Fixes #3967
2020-02-19 11:17:25 +01:00
},
}})
Refactor into sub filesystems
2013-06-27 21:13:07 +02:00
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
// Constants
const (
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
metaMtime = "mtime" // the meta key to store mtime in - e.g. X-Amz-Meta-Mtime
metaMD5Hash = "md5chksum" // the meta key to store md5hash in
Use binary prefixes for size and rate units Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M. Extends binary suffix output with letter i, e.g. Ki and Mi. Centralizes creation of bit/byte unit strings.
2021-03-02 20:11:57 +01:00
// The maximum size of object we can COPY - this should be 5 GiB but is < 5 GB for b2 compatibility
s3: reduce the default --s3-copy-cutoff to < 5GB The maximum value for the --s3--copy-cutoff should be 5GiB as tested with AWS S3. However b2 have implemented this as 5GB rather than 5GiB so having the default at 5 GiB makes the b2s3 server side copy of a large file by default. This patch sets the default to 4768 MiB which is slightly less than 5GB. This should have very little effect on anything. If in future rclone can lower this limit more if Copy can multithread. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/76
2020-09-01 19:53:29 +02:00
// See https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/76
maxSizeForCopy = 4768 * 1024 * 1024
maxUploadParts = 10000 // maximum allowed number of parts in a multi-part upload
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
minChunkSize = fs.SizeSuffix(1024 * 1024 * 5)
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
defaultUploadCutoff = fs.SizeSuffix(200 * 1024 * 1024)
maxUploadCutoff = fs.SizeSuffix(5 * 1024 * 1024 * 1024)
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
minSleep = 10 * time.Millisecond // In case of error, start at 10ms sleep.
s3: Cap expiry duration to 1 Week and return error when sharing dir
2020-06-18 18:50:50 +02:00
maxExpireDuration = fs.Duration(7 * 24 * time.Hour) // max expiry is 1 week
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
)
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
// globals
var (
errNotWithVersionAt = errors.New("can't modify or delete files in --s3-version-at mode")
)
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
// system metadata keys which this backend owns
var systemMetadataInfo = map[string]fs.MetadataHelp{
"cache-control": {
Help: "Cache-Control header",
Type: "string",
Example: "no-cache",
},
"content-disposition": {
Help: "Content-Disposition header",
Type: "string",
Example: "inline",
},
"content-encoding": {
Help: "Content-Encoding header",
Type: "string",
Example: "gzip",
},
"content-language": {
Help: "Content-Language header",
Type: "string",
Example: "en-US",
},
"content-type": {
Help: "Content-Type header",
Type: "string",
Example: "text/plain",
},
// "tagging": {
// Help: "x-amz-tagging header",
// Type: "string",
// Example: "tag1=value1&tag2=value2",
// },
"tier": {
Help: "Tier of the object",
Type: "string",
Example: "GLACIER",
ReadOnly: true,
},
"mtime": {
Help: "Time of last modification, read from rclone metadata",
Type: "RFC 3339",
Example: "2006-01-02T15:04:05.999999999Z07:00",
},
"btime": {
Help: "Time of file birth (creation) read from Last-Modified header",
Type: "RFC 3339",
Example: "2006-01-02T15:04:05.999999999Z07:00",
ReadOnly: true,
},
}
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
// Options defines the configuration for this backend
type Options struct {
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
Provider string `config:"provider"`
EnvAuth bool `config:"env_auth"`
AccessKeyID string `config:"access_key_id"`
SecretAccessKey string `config:"secret_access_key"`
Region string `config:"region"`
Endpoint string `config:"endpoint"`
s3: add --s3-sts-endpoint to specify STS endpoint See: https://forum.rclone.org/t/s3-profile-failing-when-explicit-s3-endpoint-is-present/36063/
2023-03-02 10:56:09 +01:00
STSEndpoint string `config:"sts_endpoint"`
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
LocationConstraint string `config:"location_constraint"`
ACL string `config:"acl"`
BucketACL string `config:"bucket_acl"`
s3: add requester pays option - fixes #301
2020-12-03 03:30:06 +01:00
RequesterPays bool `config:"requester_pays"`
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
ServerSideEncryption string `config:"server_side_encryption"`
SSEKMSKeyID string `config:"sse_kms_key_id"`
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
SSECustomerAlgorithm string `config:"sse_customer_algorithm"`
SSECustomerKey string `config:"sse_customer_key"`
s3: add --s3-sse-customer-key-base64 to supply keys with binary data Fixes #6400
2022-09-17 18:28:44 +02:00
SSECustomerKeyBase64 string `config:"sse_customer_key_base64"`
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
SSECustomerKeyMD5 string `config:"sse_customer_key_md5"`
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
StorageClass string `config:"storage_class"`
UploadCutoff fs.SizeSuffix `config:"upload_cutoff"`
CopyCutoff fs.SizeSuffix `config:"copy_cutoff"`
ChunkSize fs.SizeSuffix `config:"chunk_size"`
azureblob,b2,s3: fix chunksize calculations producing too many parts Before this fix, the chunksize calculator was using the previous size of the object, not the new size of the object to calculate the chunk sizes. This meant that uploading a replacement object which needed a new chunk size would fail, using too many parts. This fix fixes the calculator to take the size explicitly.
2022-08-09 11:44:54 +02:00
MaxUploadParts int `config:"max_upload_parts"`
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
DisableChecksum bool `config:"disable_checksum"`
s3: implement `profile` and `shared_credentials_file` options It is impossible to use two different profiles at the same time - these config vars enable that. See: https://forum.rclone.org/t/s3-source-destination-named-profile/17417
2020-06-26 12:14:40 +02:00
SharedCredentialsFile string `config:"shared_credentials_file"`
Profile string `config:"profile"`
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
SessionToken string `config:"session_token"`
UploadConcurrency int `config:"upload_concurrency"`
ForcePathStyle bool `config:"force_path_style"`
V2Auth bool `config:"v2_auth"`
UseAccelerateEndpoint bool `config:"use_accelerate_endpoint"`
LeavePartsOnError bool `config:"leave_parts_on_error"`
ListChunk int64 `config:"list_chunk"`
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
ListVersion int `config:"list_version"`
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
ListURLEncode fs.Tristate `config:"list_url_encode"`
s3: add --s3-no-check-bucket for minimising rclone transactions and perms Fixes #4449
2020-07-22 13:02:17 +02:00
NoCheckBucket bool `config:"no_check_bucket"`
s3: add --s3-no-head parameter to minimise transactions on upload See: https://forum.rclone.org/t/prevent-head-on-amazon-s3-family/21935
2021-01-29 12:13:42 +01:00
NoHead bool `config:"no_head"`
s3: add --s3-no-head-object This stops rclone doing any HEAD requests on objects.
2021-04-28 12:05:54 +02:00
NoHeadObject bool `config:"no_head_object"`
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
Enc encoder.MultiEncoder `config:"encoding"`
s3: Added `--s3-disable-http2` to disable http/2 Fixes #4673 Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-13 18:11:22 +02:00
DisableHTTP2 bool `config:"disable_http2"`
s3: add support to use CDN URL to download the file The egress charges while using a CloudFront CDN url is cheaper when compared to accessing the file directly from S3. So added a download URL advanced option, which when set downloads the file using it.
2021-10-14 12:19:38 +02:00
DownloadURL string `config:"download_url"`
s3: empty directory markers - #3453
2022-09-26 08:59:57 +02:00
DirectoryMarkers bool `config:"directory_markers"`
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
UseMultipartEtag fs.Tristate `config:"use_multipart_etag"`
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
UsePresignedRequest bool `config:"use_presigned_request"`
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
Versions bool `config:"versions"`
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
VersionAt fs.Time `config:"version_at"`
s3: add --s3-decompress flag to download gzip-encoded files Before this change, if an object compressed with "Content-Encoding: gzip" was downloaded, a length and hash mismatch would occur since the go runtime automatically decompressed the object on download. If --s3-decompress is set, this change erases the length and hash on compressed objects so they can be downloaded successfully, at the cost of not being able to check the length or the hash of the downloaded object. If --s3-decompress is not set the compressed files will be downloaded as-is providing compressed objects with intact size and hash information. See #2658
2022-07-29 18:01:59 +02:00
Decompress bool `config:"decompress"`
s3: add provider quirk --s3-might-gzip to fix corrupted on transfer: sizes differ Before this change, some files were giving this error when downloaded from Cloudflare and other providers. ERROR corrupted on transfer: sizes differ NNN vs MMM This is because these providers auto gzips the object when rclone wasn't expecting it to. (AWS does not gzip objects without their being uploaded gzipped). This patch adds a quirk to for fix the problem and a flag to control it. The quirk `might_gzip` is set to `true` for all providers except AWS. See: https://forum.rclone.org/t/s3-error-corrupted-on-transfer-sizes-differ-nnn-vs-mmm/33694/ Fixes: #6533
2022-10-29 00:23:29 +02:00
MightGzip fs.Tristate `config:"might_gzip"`
s3: add GCS to provider list
2023-01-09 07:54:51 +01:00
UseAcceptEncodingGzip fs.Tristate `config:"use_accept_encoding_gzip"`
s3: add --s3-no-system-metadata to suppress read and write of system metadata See: https://forum.rclone.org/t/problems-with-content-disposition-and-backblaze-b2-using-s3/33292/
2022-10-12 09:55:58 +02:00
NoSystemMetadata bool `config:"no_system_metadata"`
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
}
s3: add support for setting storage class in config and command line
2016-09-01 23:27:50 +02:00
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
// Fs represents a remote s3 server
type Fs struct {
s3: add --s3-decompress flag to download gzip-encoded files Before this change, if an object compressed with "Content-Encoding: gzip" was downloaded, a length and hash mismatch would occur since the go runtime automatically decompressed the object on download. If --s3-decompress is set, this change erases the length and hash on compressed objects so they can be downloaded successfully, at the cost of not being able to check the length or the hash of the downloaded object. If --s3-decompress is not set the compressed files will be downloaded as-is providing compressed objects with intact size and hash information. See #2658
2022-07-29 18:01:59 +02:00
name string // the name of the remote
root string // root of the bucket - ignore all objects above this
opt Options // parsed options
ci *fs.ConfigInfo // global config
ctx context.Context // global context for reading config
features *fs.Features // optional features
c *s3.S3 // the connection to the s3 server
ses *session.Session // the s3 session
rootBucket string // bucket part of root (if any)
rootDirectory string // directory part of root (if any)
cache *bucket.Cache // cache for bucket creation status
pacer *fs.Pacer // To pace the API calls
srv *http.Client // a plain http client
srvRest *rest.Client // the rest connection to the server
etagIsNotMD5 bool // if set ETags are not MD5s
versioningMu sync.Mutex
versioning fs.Tristate // if set bucket is using versions
warnCompressed sync.Once // warn once about compressed files
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
// Object describes a s3 object
type Object struct {
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
// Will definitely have everything but meta which may be nil
//
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 23:13:24 +02:00
// List will read everything but meta & mimeType - to fill
// that in you need to call readMetaData
s3: change metadata storage to normal map with lowercase keys
2022-06-22 16:40:30 +02:00
fs *Fs // what this object is part of
remote string // The remote path
md5 string // md5sum of the object
bytes int64 // size of the object
lastModified time.Time // Last modified
meta map[string]string // The object metadata if known - may be nil - with lower case keys
mimeType string // MimeType of object - may be ""
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
versionID *string // If present this points to an object version
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
// Metadata as pointers to strings as they often won't be present
storageClass *string // e.g. GLACIER
cacheControl *string // Cache-Control: header
contentDisposition *string // Content-Disposition: header
contentEncoding *string // Content-Encoding: header
contentLanguage *string // Content-Language: header
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
// ------------------------------------------------------------
Fix golint warnings
2015-09-22 19:47:16 +02:00
// Name of the remote (as passed into NewFs)
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
func (f *Fs) Name() string {
Add Name() to Fs interface to return name as passed to NewFs
2015-08-22 17:53:11 +02:00
return f.name
}
Fix golint warnings
2015-09-22 19:47:16 +02:00
// Root of the remote (as passed into NewFs)
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
func (f *Fs) Root() string {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
return f.root
Add Root() to Fs interface
2015-09-01 21:45:27 +02:00
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
// String converts this Fs to a string
func (f *Fs) String() string {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
if f.rootBucket == "" {
Fix lint issues reported by staticcheck Used staticcheck 2022.1.2 (v0.3.2) See: staticcheck.io
2022-06-08 22:25:17 +02:00
return "S3 root"
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
}
if f.rootDirectory == "" {
return fmt.Sprintf("S3 bucket %s", f.rootBucket)
s3: support sub-bucket paths
2014-05-05 19:25:32 +02:00
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
return fmt.Sprintf("S3 bucket %s path %s", f.rootBucket, f.rootDirectory)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
Define a new Features() method for Fs Optional interfaces are becoming more important in rclone, --track-renames and --backup-dir both rely on them. Up to this point rclone has used interface upgrades to define optional behaviour on Fs objects. However when one Fs object wraps another it is very difficult for this scheme to work accurately. rclone has relied on specific error messages being returned when the interface isn't supported - this is unsatisfactory because it means you have to call the interface to see whether it is supported. This change enables accurate detection of optional interfaces by use of a Features struct as returned by an obligatory Fs.Features() method. The Features struct contains flags and function pointers which can be tested against nil to see whether they can be used. As a result crypt and hubic can accurately reflect the capabilities of the underlying Fs they are wrapping.
2017-01-13 18:21:47 +01:00
// Features returns the optional features of this Fs
func (f *Fs) Features() *fs.Features {
return f.features
}
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
// retryErrorCodes is a slice of error codes that we will retry
// See: https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
var retryErrorCodes = []int{
s3: Added error handling for error code 429 indicating too many requests Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-12-01 15:25:25 +01:00
429, // Too Many Requests
backend/s3: Added 500 as retryErrorCode The error code 500 Internal Error indicates that Amazon S3 is unable to handle the request at that time. The error code 503 Slow Down typically indicates that the requests to the S3 bucket are very high, exceeding the request rates described in Request Rate and Performance Guidelines. Because Amazon S3 is a distributed service, a very small percentage of 5xx errors are expected during normal use of the service. All requests that return 5xx errors from Amazon S3 can and should be retried, so we recommend that applications making requests to Amazon S3 have a fault-tolerance mechanism to recover from these errors. https://aws.amazon.com/premiumsupport/knowledge-center/http-5xx-errors-s3/
2020-02-11 16:30:32 +01:00
500, // Internal Server Error - "We encountered an internal error. Please try again."
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
503, // Service Unavailable/Slow Down - "Reduce your request rate"
}
build: reformat comments to pass go1.19 vet See: https://go.dev/doc/go1.19#go-doc
2022-08-05 17:35:41 +02:00
// S3 is pretty resilient, and the built in retry handling is probably sufficient
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
// as it should notice closed connections and timeouts which are the most likely
// sort of failure modes
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
func (f *Fs) shouldRetry(ctx context.Context, err error) (bool, error) {
if fserrors.ContextError(ctx, &err) {
return false, err
}
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
// If this is an awserr object, try and extract more useful information to determine if we should retry
if awsError, ok := err.(awserr.Error); ok {
backend: fix misspellings
2019-02-07 18:41:17 +01:00
// Simple case, check the original embedded error in case it's generically retryable
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
if fserrors.ShouldRetry(awsError.OrigErr()) {
return true, err
}
s3: retry RequestTimeout errors See: https://forum.rclone.org/t/s3-failed-upload-large-files-bad-request-400/27695
2021-11-24 13:48:57 +01:00
// If it is a timeout then we want to retry that
if awsError.Code() == "RequestTimeout" {
return true, err
}
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
// Failing that, if it's a RequestFailure it's probably got an http status code we can check
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
if reqErr, ok := err.(awserr.RequestFailure); ok {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
// 301 if wrong region for bucket - can only update if running from a bucket
if f.rootBucket != "" {
if reqErr.StatusCode() == http.StatusMovedPermanently {
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
urfbErr := f.updateRegionForBucket(ctx, f.rootBucket)
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
if urfbErr != nil {
fs.Errorf(f, "Failed to update region for bucket: %v", urfbErr)
return false, err
}
return true, err
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
}
}
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
for _, e := range retryErrorCodes {
if reqErr.StatusCode() == e {
return true, err
}
}
}
}
s3: Support Alibaba Cloud (Aliyun) OSS The existing s3 backend passed all integration tests with OSS provided `force_path_style = false`. This makes sure that is so and adds documentation and configuration for OSS. Thanks to @luolibin for their work on the OSS backend which we ended up not needing. Fixes #1641 Fixes #1237
2019-01-12 17:46:45 +01:00
// Ok, not an awserr, check for generic failure conditions
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
return fserrors.ShouldRetry(err), err
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
// parsePath parses a remote 'url'
func parsePath(path string) (root string) {
root = strings.Trim(path, "/")
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
return
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
// split returns bucket and bucketPath from the rootRelativePath
// relative to f.root
func (f *Fs) split(rootRelativePath string) (bucketName, bucketPath string) {
s3: use bucket.Join instead of path.Join to preserve paths Before this change, path.Join would remove the trailing / from objects which had them. The simplified bucket.Join does not.
2023-03-03 11:45:29 +01:00
bucketName, bucketPath = bucket.Split(bucket.Join(f.root, rootRelativePath))
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
return f.opt.Enc.FromStandardName(bucketName), f.opt.Enc.FromStandardPath(bucketPath)
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
}
// split returns bucket and bucketPath from the object
func (o *Object) split() (bucket, bucketPath string) {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
bucket, bucketPath = o.fs.split(o.remote)
// If there is an object version, then the path may have a
// version suffix, if so remove it.
//
// If we are unlucky enough to have a file name with a valid
// version path where this wasn't required (eg using
// --s3-version-at) then this will go wrong.
if o.versionID != nil {
_, bucketPath = version.Remove(bucketPath)
}
return bucket, bucketPath
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
}
s3: Added `--s3-disable-http2` to disable http/2 Fixes #4673 Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-13 18:11:22 +02:00
// getClient makes an http client according to the options
fs: deglobalise the config #4685 This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.
2020-11-05 12:33:32 +01:00
func getClient(ctx context.Context, opt *Options) *http.Client {
s3: Added `--s3-disable-http2` to disable http/2 Fixes #4673 Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-13 18:11:22 +02:00
// TODO: Do we need cookies too?
fshttp: read config from ctx not passed in ConfigInfo #4685
2020-11-13 16:24:43 +01:00
t := fshttp.NewTransportCustom(ctx, func(t *http.Transport) {
s3: Added `--s3-disable-http2` to disable http/2 Fixes #4673 Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-13 18:11:22 +02:00
if opt.DisableHTTP2 {
t.TLSNextProto = map[string]func(string, *tls.Conn) http.RoundTripper{}
}
})
return &http.Client{
Transport: t,
}
}
s3: fix AWS STS failing if --s3-endpoint is set Before this change if an --s3-profile was set which used AWS STS (eg to assume a role) and --s3-endpoint was set then rclone would use the value from --s3-endpoint to contact the STS server which did not work. This fix implements an endpoint resolver which only overrides the "s3" service if --s3-endpoint is set. It sends the "sts" service (and any other service) to the default resolver. Fixes #6443 See: https://forum.rclone.org/t/s3-profile-failing-when-explicit-s3-endpoint-is-present/36063/
2023-03-01 17:18:04 +01:00
// Default name resolver
var defaultResolver = endpoints.DefaultResolver()
// resolve (service, region) to endpoint
//
// Used to set endpoint for s3 services and not for other services
s3: add --s3-sts-endpoint to specify STS endpoint See: https://forum.rclone.org/t/s3-profile-failing-when-explicit-s3-endpoint-is-present/36063/
2023-03-02 10:56:09 +01:00
type resolver map[string]string
// Add a service to the resolver, ignoring empty urls
func (r resolver) addService(service, url string) {
if url == "" {
return
}
if !strings.HasPrefix(url, "http") {
url = "https://" + url
}
r[service] = url
}
s3: fix AWS STS failing if --s3-endpoint is set Before this change if an --s3-profile was set which used AWS STS (eg to assume a role) and --s3-endpoint was set then rclone would use the value from --s3-endpoint to contact the STS server which did not work. This fix implements an endpoint resolver which only overrides the "s3" service if --s3-endpoint is set. It sends the "sts" service (and any other service) to the default resolver. Fixes #6443 See: https://forum.rclone.org/t/s3-profile-failing-when-explicit-s3-endpoint-is-present/36063/
2023-03-01 17:18:04 +01:00
// EndpointFor return the endpoint for s3 if set or the default if not
s3: add --s3-sts-endpoint to specify STS endpoint See: https://forum.rclone.org/t/s3-profile-failing-when-explicit-s3-endpoint-is-present/36063/
2023-03-02 10:56:09 +01:00
func (r resolver) EndpointFor(service, region string, opts ...func(*endpoints.Options)) (endpoints.ResolvedEndpoint, error) {
s3: fix AWS STS failing if --s3-endpoint is set Before this change if an --s3-profile was set which used AWS STS (eg to assume a role) and --s3-endpoint was set then rclone would use the value from --s3-endpoint to contact the STS server which did not work. This fix implements an endpoint resolver which only overrides the "s3" service if --s3-endpoint is set. It sends the "sts" service (and any other service) to the default resolver. Fixes #6443 See: https://forum.rclone.org/t/s3-profile-failing-when-explicit-s3-endpoint-is-present/36063/
2023-03-01 17:18:04 +01:00
fs.Debugf(nil, "Resolving service %q region %q", service, region)
s3: add --s3-sts-endpoint to specify STS endpoint See: https://forum.rclone.org/t/s3-profile-failing-when-explicit-s3-endpoint-is-present/36063/
2023-03-02 10:56:09 +01:00
url, ok := r[service]
if ok {
s3: fix AWS STS failing if --s3-endpoint is set Before this change if an --s3-profile was set which used AWS STS (eg to assume a role) and --s3-endpoint was set then rclone would use the value from --s3-endpoint to contact the STS server which did not work. This fix implements an endpoint resolver which only overrides the "s3" service if --s3-endpoint is set. It sends the "sts" service (and any other service) to the default resolver. Fixes #6443 See: https://forum.rclone.org/t/s3-profile-failing-when-explicit-s3-endpoint-is-present/36063/
2023-03-01 17:18:04 +01:00
return endpoints.ResolvedEndpoint{
URL: url,
SigningRegion: region,
}, nil
}
return defaultResolver.EndpointFor(service, region, opts...)
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
// s3Connection makes a connection to s3
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
func s3Connection(ctx context.Context, opt *Options, client *http.Client) (*s3.S3, *session.Session, error) {
s3: Use a combination of SDK retries and rclone retries - fixes #5509 This reverts commit dc06973796392edf Revert "s3: use rclone's low level retries instead of AWS SDK to fix listing retries" Which in turn reverted 5470d34740d03e15 "backend/s3: use low-level-retries as the number of SDK retries" So we are back where we started. It then modifies it to set the AWS SDK to `--low-level-retries` retries, but set the rclone retries to 2 so that directory listings can be retried.
2021-08-13 18:27:49 +02:00
ci := fs.GetConfig(ctx)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
// Make the auth
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
v := credentials.Value{
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
AccessKeyID: opt.AccessKeyID,
SecretAccessKey: opt.SecretAccessKey,
SessionToken: opt.SessionToken,
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
}
s3: add support for ECS task IAM roles ECS container IAM metadata is in a different place than EC2 IAM metadata. Use defaults' RemoteCredProvider function to query the standard locations for the credentials. Give the ECS role precedence over the role available from the underlying EC2 instance.
2017-11-22 22:21:36 +01:00
lowTimeoutClient := &http.Client{Timeout: 1 * time.Second} // low timeout to ec2 metadata service
s3: Added `--s3-disable-http2` to disable http/2 Fixes #4673 Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-13 18:11:22 +02:00
s3: add support for ECS task IAM roles ECS container IAM metadata is in a different place than EC2 IAM metadata. Use defaults' RemoteCredProvider function to query the standard locations for the credentials. Give the ECS role precedence over the role available from the underlying EC2 instance.
2017-11-22 22:21:36 +01:00
def := defaults.Get()
def.Config.HTTPClient = lowTimeoutClient
backend/s3: replace deprecated session.New() with session.NewSession()
2020-03-04 01:19:59 +01:00
// start a new AWS session
awsSession, err := session.NewSession()
if err != nil {
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
return nil, nil, fmt.Errorf("NewSession: %w", err)
backend/s3: replace deprecated session.New() with session.NewSession()
2020-03-04 01:19:59 +01:00
}
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
// first provider to supply a credential set "wins"
providers := []credentials.Provider{
// use static credentials if they're present (checked by provider)
&credentials.StaticProvider{Value: v},
// * Access Key ID: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY
// * Secret Access Key: AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY
&credentials.EnvProvider{},
s3: Look in S3 named profile files for credentials - fixes #2243
2018-04-16 13:14:35 +02:00
// A SharedCredentialsProvider retrieves credentials
// from the current user's home directory. It checks
// AWS_SHARED_CREDENTIALS_FILE and AWS_PROFILE too.
s3: implement `profile` and `shared_credentials_file` options It is impossible to use two different profiles at the same time - these config vars enable that. See: https://forum.rclone.org/t/s3-source-destination-named-profile/17417
2020-06-26 12:14:40 +02:00
&credentials.SharedCredentialsProvider{
Filename: opt.SharedCredentialsFile, // If empty will look for "AWS_SHARED_CREDENTIALS_FILE" env variable.
Profile: opt.Profile, // If empty will look gor "AWS_PROFILE" env var or "default" if not set.
},
s3: Look in S3 named profile files for credentials - fixes #2243
2018-04-16 13:14:35 +02:00
s3: add support for ECS task IAM roles ECS container IAM metadata is in a different place than EC2 IAM metadata. Use defaults' RemoteCredProvider function to query the standard locations for the credentials. Give the ECS role precedence over the role available from the underlying EC2 instance.
2017-11-22 22:21:36 +01:00
// Pick up IAM role if we're in an ECS task
defaults.RemoteCredProvider(*def.Config, def.Handlers),
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
// Pick up IAM role in case we're on EC2
&ec2rolecreds.EC2RoleProvider{
backend/s3: replace deprecated session.New() with session.NewSession()
2020-03-04 01:19:59 +01:00
Client: ec2metadata.New(awsSession, &aws.Config{
s3: add support for ECS task IAM roles ECS container IAM metadata is in a different place than EC2 IAM metadata. Use defaults' RemoteCredProvider function to query the standard locations for the credentials. Give the ECS role precedence over the role available from the underlying EC2 instance.
2017-11-22 22:21:36 +01:00
HTTPClient: lowTimeoutClient,
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
}),
backend/s3: fix ExpiryWindow value ExpiryWindow accepts duration but it was set to value 3. This changes it to 3 * time.Minute since default is 5 min.
2019-10-31 16:42:16 +01:00
ExpiryWindow: 3 * time.Minute,
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
},
}
cred := credentials.NewChainCredentials(providers)
s3: allow anonymous access to public repositories - fixes #154 When setting up the remote, leave both the access key and secret key blank.
2015-09-29 10:58:03 +02:00
switch {
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
case opt.EnvAuth:
s3: Fix empty checks in auth
2016-02-09 18:19:33 +01:00
// No need for empty checks if "env_auth" is true
case v.AccessKeyID == "" && v.SecretAccessKey == "":
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
// if no access key/secret and iam is explicitly disabled then fall back to anon interaction
cred = credentials.AnonymousCredentials
case v.AccessKeyID == "":
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
return nil, nil, errors.New("access_key_id not found")
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
case v.SecretAccessKey == "":
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
return nil, nil, errors.New("secret_access_key not found")
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
if opt.Region == "" {
opt.Region = "us-east-1"
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
setQuirks(opt)
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
awsConfig := aws.NewConfig().
s3: Use a combination of SDK retries and rclone retries - fixes #5509 This reverts commit dc06973796392edf Revert "s3: use rclone's low level retries instead of AWS SDK to fix listing retries" Which in turn reverted 5470d34740d03e15 "backend/s3: use low-level-retries as the number of SDK retries" So we are back where we started. It then modifies it to set the AWS SDK to `--low-level-retries` retries, but set the rclone retries to 2 so that directory listings can be retried.
2021-08-13 18:27:49 +02:00
WithMaxRetries(ci.LowLevelRetries).
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 14:11:27 +01:00
WithCredentials(cred).
s3: fix Wasabi HEAD requests returning stale data by using only 1 transport In this commit fc5b14b62046b240 s3: Added `--s3-disable-http2` to disable http/2 We created our own transport so we could disable http/2. However the added function is called twice meaning that we create two HTTP transports. This didn't happen with the original code because the default transport is cached by fshttp. Rclone normally does a PUT followed by a HEAD request to check an upload has been successful. With the two transports, the PUT and the HEAD were being done on different HTTP transports. This means that it wasn't re-using the same HTTP connection, so the HEAD request showed the previous object value. This caused rclone to declare the upload was corrupted, delete the object and try again. This patch makes sure we only create one transport and use it for both PUT and HEAD requests which fixes the problem with Wasabi. See: https://forum.rclone.org/t/each-time-rclone-is-run-1-3-fails-2-3-succeeds/22545
2021-03-05 12:04:57 +01:00
WithHTTPClient(client).
s3: suppport S3 Accelerated endpoints with --s3-use-accelerate-endpoint Fixes #3123
2019-04-26 11:19:00 +02:00
WithS3ForcePathStyle(opt.ForcePathStyle).
s3: Use regional s3 us-east-1 endpoint
2020-06-26 15:09:29 +02:00
WithS3UseAccelerate(opt.UseAccelerateEndpoint).
WithS3UsEast1RegionalEndpoint(endpoints.RegionalS3UsEast1Endpoint)
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
if opt.Region != "" {
awsConfig.WithRegion(opt.Region)
}
s3: add --s3-sts-endpoint to specify STS endpoint See: https://forum.rclone.org/t/s3-profile-failing-when-explicit-s3-endpoint-is-present/36063/
2023-03-02 10:56:09 +01:00
if opt.Endpoint != "" || opt.STSEndpoint != "" {
// If endpoints are set, override the relevant services only
r := make(resolver)
r.addService("s3", opt.Endpoint)
r.addService("sts", opt.STSEndpoint)
awsConfig.WithEndpointResolver(r)
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
}
s3: make v2 signatures work for ceph
2015-08-10 12:02:34 +02:00
// awsConfig.WithLogLevel(aws.LogDebugWithSigning)
s3: fix role_arn, credential_source, ... When the env_auth option is enabled, the AWS SDK's session constructor now loads configuration from ~/.aws/config and environment variables, and credentials per the selected (or default) AWS_PROFILE's settings. This is accomplished by **NOT** including any Credential provider in the aws.Config passed to the session constructor: If the Config.Credentials is non-nil, that will always be used and the user's configuration re role_arn, credential_source, source_profile, etc... from the shared config will be completely ignored. (The conditional creation and configuration of the stscreds Credential provider is complicated enough that it is not worth re-creating that logic.)
2018-11-07 03:50:28 +01:00
awsSessionOpts := session.Options{
Config: *awsConfig,
}
if opt.EnvAuth && opt.AccessKeyID == "" && opt.SecretAccessKey == "" {
// Enable loading config options from ~/.aws/config (selected by AWS_PROFILE env)
awsSessionOpts.SharedConfigState = session.SharedConfigEnable
s3: fix --s3-profile which wasn't working - fixes #4757
2021-01-19 19:21:08 +01:00
// Set the name of the profile if supplied
awsSessionOpts.Profile = opt.Profile
s3: fix shared_credentials_file auth after reverting incorrect fix #5762 Before this change the `shared_credentials_file` config option was being ignored. The correct value is passed into the SDK but it only sets the credentials in the default provider. Unfortunately we wipe the default provider in order to install our own chain if env_auth is true. This patch restores the shared credentials file in the session options, exactly the same as how we restore the profile. Original fix: 1605f9e14dbbe58e s3: Fix shared_credentials_file auth
2021-10-29 13:39:34 +02:00
// Set the shared config file if supplied
if opt.SharedCredentialsFile != "" {
awsSessionOpts.SharedConfigFiles = []string{opt.SharedCredentialsFile}
}
s3: fix IAM Role for Service Account not working and other auth problems This patch reverts this commit 1605f9e14dbbe58e s3: Fix shared_credentials_file auth It unfortunately had the side effect of making the s3 SDK ignore the config in our custom chain and use the default provider. This means that advanced auth was being ignored such as --s3-profile with role_arn. Fixes #5468 Fixes #5762
2021-10-27 16:50:07 +02:00
// The session constructor (aws/session/mergeConfigSrcs) will only use the user's preferred credential source
// (from the shared config file) if the passed-in Options.Config.Credentials is nil.
awsSessionOpts.Config.Credentials = nil
s3: fix role_arn, credential_source, ... When the env_auth option is enabled, the AWS SDK's session constructor now loads configuration from ~/.aws/config and environment variables, and credentials per the selected (or default) AWS_PROFILE's settings. This is accomplished by **NOT** including any Credential provider in the aws.Config passed to the session constructor: If the Config.Credentials is non-nil, that will always be used and the user's configuration re role_arn, credential_source, source_profile, etc... from the shared config will be completely ignored. (The conditional creation and configuration of the stscreds Credential provider is complicated enough that it is not worth re-creating that logic.)
2018-11-07 03:50:28 +01:00
}
ses, err := session.NewSessionWithOptions(awsSessionOpts)
if err != nil {
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
return nil, nil, err
}
c := s3.New(ses)
if opt.V2Auth || opt.Region == "other-v2-signature" {
fs.Debugf(nil, "Using v2 auth")
signer := func(req *request.Request) {
// Ignore AnonymousCredentials object
if req.Config.Credentials == credentials.AnonymousCredentials {
return
s3: make v2 signatures work for ceph
2015-08-10 12:02:34 +02:00
}
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
sign(v.AccessKeyID, v.SecretAccessKey, req.HTTPRequest)
s3: make v2 signatures work for ceph
2015-08-10 12:02:34 +02:00
}
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
c.Handlers.Sign.Clear()
c.Handlers.Sign.PushBackNamed(corehandlers.BuildContentLengthHandler)
c.Handlers.Sign.PushBack(signer)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
return c, ses, nil
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
backend: add fstests.ChunkedUploadConfig - azureblob - b2 - drive - dropbox - onedrive - s3 - swift
2018-09-07 13:02:27 +02:00
func checkUploadChunkSize(cs fs.SizeSuffix) error {
if cs < minChunkSize {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return fmt.Errorf("%s is less than %s", cs, minChunkSize)
backend: add fstests.ChunkedUploadConfig - azureblob - b2 - drive - dropbox - onedrive - s3 - swift
2018-09-07 13:02:27 +02:00
}
return nil
}
func (f *Fs) setUploadChunkSize(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
err = checkUploadChunkSize(cs)
if err == nil {
old, f.opt.ChunkSize = f.opt.ChunkSize, cs
}
return
}
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
func checkUploadCutoff(cs fs.SizeSuffix) error {
if cs > maxUploadCutoff {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return fmt.Errorf("%s is greater than %s", cs, maxUploadCutoff)
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
}
return nil
}
func (f *Fs) setUploadCutoff(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
err = checkUploadCutoff(cs)
if err == nil {
old, f.opt.UploadCutoff = f.opt.UploadCutoff, cs
}
return
}
s3: add IDrive e2 to provider list
2022-06-18 09:29:21 +02:00
// setEndpointValueForIDriveE2 gets user region endpoint against the Access Key details by calling the API
func setEndpointValueForIDriveE2(m configmap.Mapper) (err error) {
value, ok := m.Get(fs.ConfigProvider)
if !ok || value != "IDrive" {
return
}
value, ok = m.Get("access_key_id")
if !ok || value == "" {
return
}
client := &http.Client{Timeout: time.Second * 3}
// API to get user region endpoint against the Access Key details: https://www.idrive.com/e2/guides/get_region_endpoint
resp, err := client.Post("https://api.idrivee2.com/api/service/get_region_end_point",
"application/json",
strings.NewReader(`{"access_key": "`+value+`"}`))
if err != nil {
return
}
defer fs.CheckClose(resp.Body, &err)
decoder := json.NewDecoder(resp.Body)
var data = &struct {
RespCode int `json:"resp_code"`
RespMsg string `json:"resp_msg"`
DomainName string `json:"domain_name"`
}{}
if err = decoder.Decode(data); err == nil && data.RespCode == 0 {
m.Set("endpoint", data.DomainName)
}
return
}
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
// Set the provider quirks
//
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
// There should be no testing against opt.Provider anywhere in the
// code except in here to localise the setting of the quirks.
//
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
// These should be differences from AWS S3
func setQuirks(opt *Options) {
var (
s3: add GCS to provider list
2023-01-09 07:54:51 +01:00
listObjectsV2 = true
virtualHostStyle = true
urlEncodeListings = true
useMultipartEtag = true
useAcceptEncodingGzip = true
mightGzip = true // assume all providers might gzip until proven otherwise
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
)
switch opt.Provider {
case "AWS":
// No quirks
s3: add provider quirk --s3-might-gzip to fix corrupted on transfer: sizes differ Before this change, some files were giving this error when downloaded from Cloudflare and other providers. ERROR corrupted on transfer: sizes differ NNN vs MMM This is because these providers auto gzips the object when rclone wasn't expecting it to. (AWS does not gzip objects without their being uploaded gzipped). This patch adds a quirk to for fix the problem and a flag to control it. The quirk `might_gzip` is set to `true` for all providers except AWS. See: https://forum.rclone.org/t/s3-error-corrupted-on-transfer-sizes-differ-nnn-vs-mmm/33694/ Fixes: #6533
2022-10-29 00:23:29 +02:00
mightGzip = false // Never auto gzips objects
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "Alibaba":
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
useMultipartEtag = false // Alibaba seems to calculate multipart Etags differently from AWS
backend: add Huawei OBS to s3 provider list
2022-06-07 09:41:46 +02:00
case "HuaweiOBS":
// Huawei OBS PFS is not support listObjectV2, and if turn on the urlEncodeListing, marker will not work and keep list same page forever.
urlEncodeListings = false
listObjectsV2 = false
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "Ceph":
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
listObjectsV2 = false
virtualHostStyle = false
urlEncodeListings = false
s3: Add ChinaMobile EOS to provider list China Mobile Ecloud Elastic Object Storage (EOS) is a cloud object storage service, and is fully compatible with S3. Fixes #6054
2022-03-24 12:57:00 +01:00
case "ChinaMobile":
listObjectsV2 = false
virtualHostStyle = false
urlEncodeListings = false
s3: Support Cloudflare R2 - fixes #5642
2022-05-06 13:09:29 +02:00
case "Cloudflare":
virtualHostStyle = false
useMultipartEtag = false // currently multipart Etags are random
s3: Add ArvanCloud AOS to provider list
2022-04-28 11:42:30 +02:00
case "ArvanCloud":
listObjectsV2 = false
virtualHostStyle = false
urlEncodeListings = false
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "DigitalOcean":
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
urlEncodeListings = false
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "Dreamhost":
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
urlEncodeListings = false
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "IBMCOS":
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
listObjectsV2 = false // untested
virtualHostStyle = false
urlEncodeListings = false
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
useMultipartEtag = false // untested
s3: add IDrive e2 to provider list
2022-06-18 09:29:21 +02:00
case "IDrive":
virtualHostStyle = false
s3: add support for IONOS Cloud Storage
2022-09-15 17:04:34 +02:00
case "IONOS":
// listObjectsV2 supported - https://api.ionos.com/docs/s3/#Basic-Operations-get-Bucket-list-type-2
virtualHostStyle = false
urlEncodeListings = false
s3: add petabox.io to s3 providers
2023-05-05 10:44:25 +02:00
case "Petabox":
// No quirks
s3: add Liara LOS to provider list
2022-12-06 13:25:23 +01:00
case "Liara":
virtualHostStyle = false
urlEncodeListings = false
useMultipartEtag = false
s3: add other regions for Lyve and correct Provider name
2022-03-14 16:37:34 +01:00
case "LyveCloud":
useMultipartEtag = false // LyveCloud seems to calculate multipart Etags differently from AWS
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "Minio":
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
virtualHostStyle = false
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "Netease":
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
listObjectsV2 = false // untested
urlEncodeListings = false
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
useMultipartEtag = false // untested
s3: Add RackCorp object storage to providers
2021-11-09 12:46:58 +01:00
case "RackCorp":
// No quirks
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
useMultipartEtag = false // untested
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "Scaleway":
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
// Scaleway can only have 1000 parts in an upload
if opt.MaxUploadParts > 1000 {
opt.MaxUploadParts = 1000
}
urlEncodeListings = false
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "SeaweedFS":
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
listObjectsV2 = false // untested
virtualHostStyle = false
urlEncodeListings = false
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
useMultipartEtag = false // untested
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "StackPath":
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
listObjectsV2 = false // untested
virtualHostStyle = false
urlEncodeListings = false
s3: add specific provider for Storj Shared gateways - unsupported features (Copy) are turned off for Storj - enable urlEncodedListing for Storj provider - set chunksize to 64Mb
2021-09-29 15:43:06 +02:00
case "Storj":
// Force chunk size to >= 64 MiB
if opt.ChunkSize < 64*fs.Mebi {
opt.ChunkSize = 64 * fs.Mebi
}
s3: add synology to s3 provider list
2023-07-06 11:54:07 +02:00
case "Synology":
useMultipartEtag = false
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "TencentCOS":
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
listObjectsV2 = false // untested
useMultipartEtag = false // untested
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
case "Wasabi":
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
// No quirks
s3: add Leviia S3 Object Storage as provider
2023-07-16 19:08:47 +02:00
case "Leviia":
// No quirks
s3: add Qiniu KODO to s3 provider list - fixes #6195
2022-10-13 16:49:22 +02:00
case "Qiniu":
useMultipartEtag = false
urlEncodeListings = false
s3: add Qiniu KODO quirks virtualHostStyle is false
2023-05-17 13:23:34 +02:00
virtualHostStyle = false
s3: add GCS to provider list
2023-01-09 07:54:51 +01:00
case "GCS":
// Google break request Signature by mutating accept-encoding HTTP header
// https://github.com/rclone/rclone/issues/6670
useAcceptEncodingGzip = false
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
case "Other":
listObjectsV2 = false
virtualHostStyle = false
urlEncodeListings = false
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
useMultipartEtag = false
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
default:
fs.Logf("s3", "s3 provider %q not known - please set correctly", opt.Provider)
listObjectsV2 = false
virtualHostStyle = false
urlEncodeListings = false
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
useMultipartEtag = false
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
}
// Path Style vs Virtual Host style
if virtualHostStyle || opt.UseAccelerateEndpoint {
opt.ForcePathStyle = false
}
// Set to see if we need to URL encode listings
if !opt.ListURLEncode.Valid {
opt.ListURLEncode.Valid = true
opt.ListURLEncode.Value = urlEncodeListings
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
}
// Set the correct list version if not manually set
if opt.ListVersion == 0 {
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
if listObjectsV2 {
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
opt.ListVersion = 2
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
} else {
opt.ListVersion = 1
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
}
}
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
// Set the correct use multipart Etag for error checking if not manually set
if !opt.UseMultipartEtag.Valid {
opt.UseMultipartEtag.Valid = true
opt.UseMultipartEtag.Value = useMultipartEtag
}
s3: add provider quirk --s3-might-gzip to fix corrupted on transfer: sizes differ Before this change, some files were giving this error when downloaded from Cloudflare and other providers. ERROR corrupted on transfer: sizes differ NNN vs MMM This is because these providers auto gzips the object when rclone wasn't expecting it to. (AWS does not gzip objects without their being uploaded gzipped). This patch adds a quirk to for fix the problem and a flag to control it. The quirk `might_gzip` is set to `true` for all providers except AWS. See: https://forum.rclone.org/t/s3-error-corrupted-on-transfer-sizes-differ-nnn-vs-mmm/33694/ Fixes: #6533
2022-10-29 00:23:29 +02:00
// set MightGzip if not manually set
if !opt.MightGzip.Valid {
opt.MightGzip.Valid = true
opt.MightGzip.Value = mightGzip
}
s3: add GCS to provider list
2023-01-09 07:54:51 +01:00
// set UseAcceptEncodingGzip if not manually set
if !opt.UseAcceptEncodingGzip.Valid {
opt.UseAcceptEncodingGzip.Valid = true
opt.UseAcceptEncodingGzip.Value = useAcceptEncodingGzip
}
s3: fallback to ListObject v1 on unsupported providers This implements a quirks system for providers and notes which providers we have tested to support ListObjectsV2. For those providers which don't support ListObjectsV2 we use the original ListObjects call.
2021-11-03 19:01:23 +01:00
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
// setRoot changes the root of the Fs
func (f *Fs) setRoot(root string) {
f.root = parsePath(root)
f.rootBucket, f.rootDirectory = bucket.Split(f.root)
}
s3: if bucket or object ACL is empty string then don't add X-Amz-Acl: header - fixes #5730 Before this fix it was impossible to stop rclone generating an X-Amx-Acl: header which is incompatible with GCS with uniform access control and is generally deprecated at AWS.
2022-09-14 17:24:57 +02:00
// return a pointer to the string if non empty or nil if it is empty
func stringPointerOrNil(s string) *string {
if s == "" {
return nil
}
return &s
}
s3: add support for setting storage class in config and command line
2016-09-01 23:27:50 +02:00
// NewFs constructs an Fs from the path, bucket:path
fs: Add context to NewFs #3257 #4685 This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.
2020-11-05 16:18:51 +01:00
func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, error) {
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
// Parse config into Options struct
opt := new(Options)
err := configstruct.Set(m, opt)
if err != nil {
return nil, err
}
backend: add fstests.ChunkedUploadConfig - azureblob - b2 - drive - dropbox - onedrive - s3 - swift
2018-09-07 13:02:27 +02:00
err = checkUploadChunkSize(opt.ChunkSize)
if err != nil {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return nil, fmt.Errorf("s3: chunk size: %w", err)
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 19:06:57 +02:00
}
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
err = checkUploadCutoff(opt.UploadCutoff)
if err != nil {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return nil, fmt.Errorf("s3: upload cutoff: %w", err)
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
}
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
if opt.Versions && opt.VersionAt.IsSet() {
backend: fix typos found by codespell
2022-09-20 08:54:47 +02:00
return nil, errors.New("s3: can't use --s3-versions and --s3-version-at at the same time")
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
}
s3: add --s3-bucket-acl to control bucket ACL - fixes #2918 Before this change buckets were created with the same ACL as objects. After this change, the user can set just --s3-acl to set the ACL of buckets and objects, or use --s3-bucket-acl as well to have a different ACL used for bucket creation. This also logs at INFO level the creation and deletion of buckets.
2019-01-16 18:23:37 +01:00
if opt.BucketACL == "" {
opt.BucketACL = opt.ACL
}
s3: add --s3-sse-customer-key-base64 to supply keys with binary data Fixes #6400
2022-09-17 18:28:44 +02:00
if opt.SSECustomerKeyBase64 != "" && opt.SSECustomerKey != "" {
return nil, errors.New("s3: can't use sse_customer_key and sse_customer_key_base64 at the same time")
} else if opt.SSECustomerKeyBase64 != "" {
// Decode the base64-encoded key and store it in the SSECustomerKey field
decoded, err := base64.StdEncoding.DecodeString(opt.SSECustomerKeyBase64)
if err != nil {
return nil, fmt.Errorf("s3: Could not decode sse_customer_key_base64: %w", err)
}
opt.SSECustomerKey = string(decoded)
}
s3: complete sse-c implementation This now can complete all operations with SSE-C enabled. Fixes #2827 See: https://forum.rclone.org/t/issues-with-aws-s3-sse-c-getting-strange-log-entries-and-errors/20553
2020-11-20 12:15:48 +01:00
if opt.SSECustomerKey != "" && opt.SSECustomerKeyMD5 == "" {
// calculate CustomerKeyMD5 if not supplied
md5sumBinary := md5.Sum([]byte(opt.SSECustomerKey))
opt.SSECustomerKeyMD5 = base64.StdEncoding.EncodeToString(md5sumBinary[:])
}
s3: fix Wasabi HEAD requests returning stale data by using only 1 transport In this commit fc5b14b62046b240 s3: Added `--s3-disable-http2` to disable http/2 We created our own transport so we could disable http/2. However the added function is called twice meaning that we create two HTTP transports. This didn't happen with the original code because the default transport is cached by fshttp. Rclone normally does a PUT followed by a HEAD request to check an upload has been successful. With the two transports, the PUT and the HEAD were being done on different HTTP transports. This means that it wasn't re-using the same HTTP connection, so the HEAD request showed the previous object value. This caused rclone to declare the upload was corrupted, delete the object and try again. This patch makes sure we only create one transport and use it for both PUT and HEAD requests which fixes the problem with Wasabi. See: https://forum.rclone.org/t/each-time-rclone-is-run-1-3-fails-2-3-succeeds/22545
2021-03-05 12:04:57 +01:00
srv := getClient(ctx, opt)
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
c, ses, err := s3Connection(ctx, opt, srv)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
if err != nil {
return nil, err
}
backend/s3: use memory pool for buffer allocations Currently each multipart upload allocated his own buffers, which after file upload was garbaged. Next files couldn't leverage already allocated memory which resulted in inefficent memory management. This change introduces backend memory pool keeping memory chunks which can be used during object operations. Fixes #3967
2020-02-19 11:17:25 +01:00
fs: deglobalise the config #4685 This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.
2020-11-05 12:33:32 +01:00
ci := fs.GetConfig(ctx)
s3: Use a combination of SDK retries and rclone retries - fixes #5509 This reverts commit dc06973796392edf Revert "s3: use rclone's low level retries instead of AWS SDK to fix listing retries" Which in turn reverted 5470d34740d03e15 "backend/s3: use low-level-retries as the number of SDK retries" So we are back where we started. It then modifies it to set the AWS SDK to `--low-level-retries` retries, but set the rclone retries to 2 so that directory listings can be retried.
2021-08-13 18:27:49 +02:00
pc := fs.NewPacer(ctx, pacer.NewS3(pacer.MinSleep(minSleep)))
// Set pacer retries to 2 (1 try and 1 retry) because we are
// relying on SDK retry mechanism, but we allow 2 attempts to
// retry directory listings after XMLSyntaxError
pc.SetRetries(2)
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
f := &Fs{
s3: add support to use CDN URL to download the file The egress charges while using a CloudFront CDN url is cheaper when compared to accessing the file directly from S3. So added a download URL advanced option, which when set downloads the file using it.
2021-10-14 12:19:38 +02:00
name: name,
opt: *opt,
ci: ci,
ctx: ctx,
c: c,
ses: ses,
s3: Use a combination of SDK retries and rclone retries - fixes #5509 This reverts commit dc06973796392edf Revert "s3: use rclone's low level retries instead of AWS SDK to fix listing retries" Which in turn reverted 5470d34740d03e15 "backend/s3: use low-level-retries as the number of SDK retries" So we are back where we started. It then modifies it to set the AWS SDK to `--low-level-retries` retries, but set the rclone retries to 2 so that directory listings can be retried.
2021-08-13 18:27:49 +02:00
pacer: pc,
s3: add support to use CDN URL to download the file The egress charges while using a CloudFront CDN url is cheaper when compared to accessing the file directly from S3. So added a download URL advanced option, which when set downloads the file using it.
2021-10-14 12:19:38 +02:00
cache: bucket.NewCache(),
srv: srv,
srvRest: rest.NewClient(fshttp.NewClient(ctx)),
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
}
s3: fix hashes on small files with aws:kms and sse-c If rclone is configured for server side encryption - either aws:kms or sse-c (but not sse-s3) then don't treat the ETags returned on objects as MD5 hashes. This fixes being able to upload small files. Fixes #1824
2020-11-20 13:15:56 +01:00
if opt.ServerSideEncryption == "aws:kms" || opt.SSECustomerAlgorithm != "" {
// From: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
//
// Objects encrypted by SSE-S3 or plaintext have ETags that are an MD5
// digest of their data.
//
// Objects encrypted by SSE-C or SSE-KMS have ETags that are not an
// MD5 digest of their object data.
f.etagIsNotMD5 = true
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
f.setRoot(root)
fs: Add CanHaveEmptyDirectories and BucketBased feature flags to all remotes
2017-08-09 16:27:43 +02:00
f.features = (&fs.Features{
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
ReadMimeType: true,
WriteMimeType: true,
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
ReadMetadata: true,
WriteMetadata: true,
UserMetadata: true,
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
BucketBased: true,
BucketBasedRootOK: true,
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
SetTier: true,
GetTier: true,
fs: define SlowModTime and SlowHash features in the relevant backends
2020-06-19 11:28:34 +02:00
SlowModTime: true,
fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685
2020-11-05 17:00:40 +01:00
}).Fill(ctx, f)
s3: fix incorrect tier support for StorJ and IDrive when pointing at a file Fixes #6734
2023-02-02 19:12:00 +01:00
if opt.Provider == "Storj" {
f.features.SetTier = false
f.features.GetTier = false
}
if opt.Provider == "IDrive" {
f.features.SetTier = false
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
if opt.DirectoryMarkers {
f.features.CanHaveEmptyDirectories = true
}
s3: fix incorrect tier support for StorJ and IDrive when pointing at a file Fixes #6734
2023-02-02 19:12:00 +01:00
// f.listMultipartUploads()
s3: don't check to see if remote is object if it ends with / Before this change, rclone would always check the root to see if it was an object. This change doesn't check to see if the root is an object if the path ends with a / This avoids a transaction where rclone HEADs the path to see if it exists. See #4990
2021-02-07 18:18:52 +01:00
if f.rootBucket != "" && f.rootDirectory != "" && !opt.NoHeadObject && !strings.HasSuffix(root, "/") {
s3: complete sse-c implementation This now can complete all operations with SSE-C enabled. Fixes #2827 See: https://forum.rclone.org/t/issues-with-aws-s3-sse-c-getting-strange-log-entries-and-errors/20553
2020-11-20 12:15:48 +01:00
// Check to see if the (bucket,directory) is actually an existing file
oldRoot := f.root
newRoot, leaf := path.Split(oldRoot)
f.setRoot(newRoot)
_, err := f.NewObject(ctx, leaf)
if err != nil {
s3: fix failed to create file system with folder level permissions policy Before this change, if folder level access permissions policy was in use, with trailing `/` marking the folders then rclone would HEAD the path without a trailing `/` to work out if it was a file or a folder. This returned a permission denied error, which rclone returned to the user. Failed to create file system for "s3:bucket/path/": Forbidden: Forbidden status code: 403, request id: XXXX, host id: Previous to this change 53aa03cc44b2a425 s3: complete sse-c implementation rclone would assume any errors when HEAD-ing the object implied it didn't exist and this test would not fail. This change reverts the functionality of the test to work as it did before, meaning any errors on HEAD will make rclone assume the object does not exist and the path is referring to a directory. Fixes #4990
2021-02-08 14:22:12 +01:00
// File doesn't exist or is a directory so return old f
f.setRoot(oldRoot)
return f, nil
Implement single file operations for all file systems
2014-05-05 20:52:52 +02:00
}
s3: complete sse-c implementation This now can complete all operations with SSE-C enabled. Fixes #2827 See: https://forum.rclone.org/t/issues-with-aws-s3-sse-c-getting-strange-log-entries-and-errors/20553
2020-11-20 12:15:48 +01:00
// return an error with an fs which points to the parent
return f, fs.ErrorIsFile
Implement single file operations for all file systems
2014-05-05 20:52:52 +02:00
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
return f, nil
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
// getMetaDataListing gets the metadata from the object unconditionally from the listing
//
// This is needed to find versioned objects from their paths.
//
// It may return info == nil and err == nil if a HEAD would be more appropriate
func (f *Fs) getMetaDataListing(ctx context.Context, wantRemote string) (info *s3.Object, versionID *string, err error) {
bucket, bucketPath := f.split(wantRemote)
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
// Strip the version string off if using versions
if f.opt.Versions {
var timestamp time.Time
timestamp, bucketPath = version.Remove(bucketPath)
// If the path had no version string return no info, to force caller to look it up
if timestamp.IsZero() {
return nil, nil, nil
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
}
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
err = f.list(ctx, listOpt{
bucket: bucket,
directory: bucketPath,
s3: fix --s3-versions on individual objects Before this fix attempting to access an s3 versioned object by name in a subdirectory of root would not find the object. This fixes the problem and introduced an integraton test. See: https://forum.rclone.org/t/s3-versions-cant-retrieve-old-version/36900
2023-03-21 13:44:45 +01:00
prefix: f.rootDirectory,
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
recurse: true,
withVersions: f.opt.Versions,
findFile: true,
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
versionAt: f.opt.VersionAt,
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
}, func(gotRemote string, object *s3.Object, objectVersionID *string, isDirectory bool) error {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
if isDirectory {
return nil
}
if wantRemote != gotRemote {
return nil
}
info = object
versionID = objectVersionID
return errEndList // read only 1 item
})
if err != nil {
if err == fs.ErrorDirNotFound {
return nil, nil, fs.ErrorObjectNotFound
}
return nil, nil, err
}
if info == nil {
return nil, nil, fs.ErrorObjectNotFound
}
return info, versionID, nil
}
s3: fix nil pointer exception when using Versions This was caused by a9bd0c8de6e74460 s3: reduce memory consumption for s3 objects Which assumed that the StorageClass would always be set, but it isn't set for Versions.
2022-12-08 13:41:23 +01:00
// stringClonePointer clones the string pointed to by sp into new
// memory. This is useful to stop us keeping references to small
// strings carved out of large XML responses.
func stringClonePointer(sp *string) *string {
if sp == nil {
return nil
}
var s = *sp
return &s
}
Rename NewFsObject to NewObject
2016-06-25 22:58:34 +02:00
// Return an Object from a path
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
//
build: reformat comments to pass go1.19 vet See: https://go.dev/doc/go1.19#go-doc
2022-08-05 17:35:41 +02:00
// If it can't be found it returns the error ErrorObjectNotFound.
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *s3.Object, versionID *string) (obj fs.Object, err error) {
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
o := &Object{
fs: f,
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
remote: remote,
}
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
if info == nil && ((f.opt.Versions && version.Match(remote)) || f.opt.VersionAt.IsSet()) {
// If versions, have to read the listing to find the correct version ID
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
info, versionID, err = f.getMetaDataListing(ctx, remote)
if err != nil {
return nil, err
}
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
if info != nil {
// Set info but not meta
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
if info.LastModified == nil {
Rename log functions and factor into own file
2017-02-09 12:01:20 +01:00
fs.Logf(o, "Failed to read last modified")
Refactor into sub filesystems
2013-06-27 21:13:07 +02:00
o.lastModified = time.Now()
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
} else {
o.lastModified = *info.LastModified
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827
2020-11-23 12:23:06 +01:00
o.setMD5FromEtag(aws.StringValue(info.ETag))
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
o.bytes = aws.Int64Value(info.Size)
s3: fix nil pointer exception when using Versions This was caused by a9bd0c8de6e74460 s3: reduce memory consumption for s3 objects Which assumed that the StorageClass would always be set, but it isn't set for Versions.
2022-12-08 13:41:23 +01:00
o.storageClass = stringClonePointer(info.StorageClass)
s3: fix excess memory usage when using versions Before this change, we were taking the version ID straight from the XML blob returned by the SDK and thus pinning the XML into memory which bulked up the average memory per object from about 400 bytes to 4k. Copying the string fixes the excess memory usage.
2022-12-13 12:50:51 +01:00
o.versionID = stringClonePointer(versionID)
s3: add --s3-no-head-object This stops rclone doing any HEAD requests on objects.
2021-04-28 12:05:54 +02:00
} else if !o.fs.opt.NoHeadObject {
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
err := o.readMetaData(ctx) // reads info and meta, returning an error
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
if err != nil {
Make NewObject return an error * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error
2016-06-25 22:23:20 +02:00
return nil, err
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
}
Make NewObject return an error * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error
2016-06-25 22:23:20 +02:00
return o, nil
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
Make NewObject return an error * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error
2016-06-25 22:23:20 +02:00
// NewObject finds the Object at remote. If it can't be found
// it returns the error fs.ErrorObjectNotFound.
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
return f.newObjectWithInfo(ctx, remote, nil, nil)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
// Gets the bucket location
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
func (f *Fs) getBucketLocation(ctx context.Context, bucket string) (string, error) {
s3: use different strategy to resolve s3 region The API endpoint GetBucketLocation requires top level permission. If we do an authenticated head request to a bucket, the bucket location will be returned in the HTTP headers. Fixes #5066
2022-10-29 09:37:08 +02:00
region, err := s3manager.GetBucketRegion(ctx, f.ses, bucket, "", func(r *request.Request) {
r.Config.S3ForcePathStyle = aws.Bool(f.opt.ForcePathStyle)
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
})
if err != nil {
return "", err
}
s3: use different strategy to resolve s3 region The API endpoint GetBucketLocation requires top level permission. If we do an authenticated head request to a bucket, the bucket location will be returned in the HTTP headers. Fixes #5066
2022-10-29 09:37:08 +02:00
return region, nil
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
}
// Updates the region for the bucket by reading the region from the
// bucket then updating the session.
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
func (f *Fs) updateRegionForBucket(ctx context.Context, bucket string) error {
region, err := f.getBucketLocation(ctx, bucket)
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
if err != nil {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return fmt.Errorf("reading bucket location failed: %w", err)
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
}
if aws.StringValue(f.c.Config.Endpoint) != "" {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return fmt.Errorf("can't set region to %q as endpoint is set", region)
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
}
if aws.StringValue(f.c.Config.Region) == region {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return fmt.Errorf("region is already %q - not updating", region)
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
}
// Make a new session with the new region
oldRegion := f.opt.Region
f.opt.Region = region
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
c, ses, err := s3Connection(f.ctx, &f.opt, f.srv)
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
if err != nil {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return fmt.Errorf("creating new session failed: %w", err)
s3: Auto detect region for buckets on operation failure - fixes #2915 If an incorrect region error is returned while using a bucket then the region is updated, the session is remade and the operation is retried.
2019-01-16 14:35:19 +01:00
}
f.c = c
f.ses = ses
fs.Logf(f, "Switched region to %q from %q", region, oldRegion)
return nil
}
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
// Common interface for bucket listers
type bucketLister interface {
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
List(ctx context.Context) (resp *s3.ListObjectsV2Output, versionIDs []*string, err error)
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
URLEncodeListings(bool)
}
// V1 bucket lister
type v1List struct {
f *Fs
req s3.ListObjectsInput
}
// Create a new V1 bucket lister
func (f *Fs) newV1List(req *s3.ListObjectsV2Input) bucketLister {
l := &v1List{
f: f,
}
// Convert v2 req into v1 req
s3: refactor to use generated code instead of reflection to copy structs
2022-07-28 12:49:19 +02:00
//structs.SetFrom(&l.req, req)
setFrom_s3ListObjectsInput_s3ListObjectsV2Input(&l.req, req)
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
return l
}
// List a bucket with V1 listing
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
func (ls *v1List) List(ctx context.Context) (resp *s3.ListObjectsV2Output, versionIDs []*string, err error) {
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
respv1, err := ls.f.c.ListObjectsWithContext(ctx, &ls.req)
if err != nil {
return nil, nil, err
}
// Set up the request for next time
ls.req.Marker = respv1.NextMarker
if aws.BoolValue(respv1.IsTruncated) && ls.req.Marker == nil {
if len(respv1.Contents) == 0 {
return nil, nil, errors.New("s3 protocol error: received listing v1 with IsTruncated set, no NextMarker and no Contents")
}
// Use the last Key received if no NextMarker and isTruncated
ls.req.Marker = respv1.Contents[len(respv1.Contents)-1].Key
}
// If we are URL encoding then must decode the marker
if ls.req.Marker != nil && ls.req.EncodingType != nil {
*ls.req.Marker, err = url.QueryUnescape(*ls.req.Marker)
if err != nil {
return nil, nil, fmt.Errorf("failed to URL decode Marker %q: %w", *ls.req.Marker, err)
}
}
// convert v1 resp into v2 resp
resp = new(s3.ListObjectsV2Output)
s3: refactor to use generated code instead of reflection to copy structs
2022-07-28 12:49:19 +02:00
//structs.SetFrom(resp, respv1)
setFrom_s3ListObjectsV2Output_s3ListObjectsOutput(resp, respv1)
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
return resp, nil, nil
}
// URL Encode the listings
func (ls *v1List) URLEncodeListings(encode bool) {
if encode {
ls.req.EncodingType = aws.String(s3.EncodingTypeUrl)
} else {
ls.req.EncodingType = nil
}
}
// V2 bucket lister
type v2List struct {
f *Fs
req s3.ListObjectsV2Input
}
// Create a new V2 bucket lister
func (f *Fs) newV2List(req *s3.ListObjectsV2Input) bucketLister {
return &v2List{
f: f,
req: *req,
}
}
// Do a V2 listing
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
func (ls *v2List) List(ctx context.Context) (resp *s3.ListObjectsV2Output, versionIDs []*string, err error) {
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
resp, err = ls.f.c.ListObjectsV2WithContext(ctx, &ls.req)
s3: fix for unchecked err value in s3 listv2
2022-11-10 12:52:59 +01:00
if err != nil {
return nil, nil, err
}
s3: fix detection of listing routines which don't support v2 properly In this commit ab849b3613fe4935 s3: fix listing loop when using v2 listing on v1 server The ContinuationToken was tested for existence, but it is the NextContinuationToken that we are interested in. See: #6600
2022-12-13 13:19:00 +01:00
if aws.BoolValue(resp.IsTruncated) && (resp.NextContinuationToken == nil || *resp.NextContinuationToken == "") {
return nil, nil, errors.New("s3 protocol error: received listing v2 with IsTruncated set and no NextContinuationToken. Should you be using `--s3-list-version 1`?")
s3: fix listing loop when using v2 listing on v1 server Before this change, rclone would enter a listing loop if it used v2 listing on a v1 server and the list exceeded 1000 items. This change detects the problem and gives the user a helpful message. Fixes #6600
2022-12-05 17:59:48 +01:00
}
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
ls.req.ContinuationToken = resp.NextContinuationToken
s3: fix for unchecked err value in s3 listv2
2022-11-10 12:52:59 +01:00
return resp, nil, nil
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
}
// URL Encode the listings
func (ls *v2List) URLEncodeListings(encode bool) {
if encode {
ls.req.EncodingType = aws.String(s3.EncodingTypeUrl)
} else {
ls.req.EncodingType = nil
}
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
// Versions bucket lister
type versionsList struct {
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
f *Fs
req s3.ListObjectVersionsInput
versionAt time.Time // set if we want only versions before this
usingVersionAt bool // set if we need to use versionAt
hidden bool // set to see hidden versions
lastKeySent string // last Key sent to the receiving function
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
}
// Create a new Versions bucket lister
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
func (f *Fs) newVersionsList(req *s3.ListObjectsV2Input, hidden bool, versionAt time.Time) bucketLister {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
l := &versionsList{
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
f: f,
versionAt: versionAt,
usingVersionAt: !versionAt.IsZero(),
hidden: hidden,
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
}
// Convert v2 req into withVersions req
s3: refactor to use generated code instead of reflection to copy structs
2022-07-28 12:49:19 +02:00
//structs.SetFrom(&l.req, req)
setFrom_s3ListObjectVersionsInput_s3ListObjectsV2Input(&l.req, req)
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
return l
}
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
// Any s3.Object or s3.ObjectVersion with this as their Size are delete markers
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
var isDeleteMarker = new(int64)
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
// Compare two s3.ObjectVersions, sorted alphabetically by key with
// the newest first if the Keys match or the one with IsLatest set if
// everything matches.
func versionLess(a, b *s3.ObjectVersion) bool {
if a == nil || a.Key == nil || a.LastModified == nil {
return true
}
if b == nil || b.Key == nil || b.LastModified == nil {
return false
}
if *a.Key < *b.Key {
return true
}
if *a.Key > *b.Key {
return false
}
dt := (*a.LastModified).Sub(*b.LastModified)
if dt > 0 {
return true
}
if dt < 0 {
return false
}
if aws.BoolValue(a.IsLatest) {
return true
}
return false
}
// Merge the DeleteMarkers into the Versions.
//
// These are delivered by S3 sorted by key then by LastUpdated
// newest first but annoyingly the SDK splits them up into two
// so we need to merge them back again
//
// We do this by converting the s3.DeleteEntry into
// s3.ObjectVersion with Size = isDeleteMarker to tell them apart
//
// We then merge them back into the Versions in the correct order
func mergeDeleteMarkers(oldVersions []*s3.ObjectVersion, deleteMarkers []*s3.DeleteMarkerEntry) (newVersions []*s3.ObjectVersion) {
newVersions = make([]*s3.ObjectVersion, 0, len(oldVersions)+len(deleteMarkers))
for _, deleteMarker := range deleteMarkers {
var obj = new(s3.ObjectVersion)
s3: refactor to use generated code instead of reflection to copy structs
2022-07-28 12:49:19 +02:00
//structs.SetFrom(obj, deleteMarker)
setFrom_s3ObjectVersion_s3DeleteMarkerEntry(obj, deleteMarker)
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
obj.Size = isDeleteMarker
for len(oldVersions) > 0 && versionLess(oldVersions[0], obj) {
newVersions = append(newVersions, oldVersions[0])
oldVersions = oldVersions[1:]
}
newVersions = append(newVersions, obj)
}
// Merge any remaining versions
newVersions = append(newVersions, oldVersions...)
return newVersions
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
// List a bucket with versions
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
func (ls *versionsList) List(ctx context.Context) (resp *s3.ListObjectsV2Output, versionIDs []*string, err error) {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
respVersions, err := ls.f.c.ListObjectVersionsWithContext(ctx, &ls.req)
if err != nil {
return nil, nil, err
}
// Set up the request for next time
ls.req.KeyMarker = respVersions.NextKeyMarker
ls.req.VersionIdMarker = respVersions.NextVersionIdMarker
// If we are URL encoding then must decode the marker
if ls.req.KeyMarker != nil && ls.req.EncodingType != nil {
*ls.req.KeyMarker, err = url.QueryUnescape(*ls.req.KeyMarker)
if err != nil {
return nil, nil, fmt.Errorf("failed to URL decode KeyMarker %q: %w", *ls.req.KeyMarker, err)
}
}
// convert Versions resp into v2 resp
resp = new(s3.ListObjectsV2Output)
s3: refactor to use generated code instead of reflection to copy structs
2022-07-28 12:49:19 +02:00
//structs.SetFrom(resp, respVersions)
setFrom_s3ListObjectsV2Output_s3ListObjectVersionsOutput(resp, respVersions)
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
// Merge in delete Markers as s3.ObjectVersion if we need them
if ls.hidden || ls.usingVersionAt {
respVersions.Versions = mergeDeleteMarkers(respVersions.Versions, respVersions.DeleteMarkers)
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
// Convert the Versions and the DeleteMarkers into an array of s3.Object
//
// These are returned in the order that they are stored with the most recent first.
// With the annoyance that the Versions and DeleteMarkers are split into two
objs := make([]*s3.Object, 0, len(respVersions.Versions))
for _, objVersion := range respVersions.Versions {
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
if ls.usingVersionAt {
if objVersion.LastModified.After(ls.versionAt) {
// Ignore versions that were created after the specified time
continue
}
if *objVersion.Key == ls.lastKeySent {
// Ignore versions before the already returned version
continue
}
}
ls.lastKeySent = *objVersion.Key
// Don't send delete markers if we don't want hidden things
if !ls.hidden && objVersion.Size == isDeleteMarker {
continue
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
var obj = new(s3.Object)
s3: refactor to use generated code instead of reflection to copy structs
2022-07-28 12:49:19 +02:00
//structs.SetFrom(obj, objVersion)
setFrom_s3Object_s3ObjectVersion(obj, objVersion)
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
// Adjust the file names
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
if !ls.usingVersionAt && !aws.BoolValue(objVersion.IsLatest) {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
if obj.Key != nil && objVersion.LastModified != nil {
*obj.Key = version.Add(*obj.Key, *objVersion.LastModified)
}
}
objs = append(objs, obj)
versionIDs = append(versionIDs, objVersion.VersionId)
}
resp.Contents = objs
return resp, versionIDs, nil
}
// URL Encode the listings
func (ls *versionsList) URLEncodeListings(encode bool) {
if encode {
ls.req.EncodingType = aws.String(s3.EncodingTypeUrl)
} else {
ls.req.EncodingType = nil
}
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
// listFn is called from list to handle an object.
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
type listFn func(remote string, object *s3.Object, versionID *string, isDirectory bool) error
// errEndList is a sentinel used to end the list iteration now.
// listFn should return it to end the iteration with no errors.
var errEndList = errors.New("end list")
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
// list options
type listOpt struct {
s3: make purge remove directory markers too See: https://forum.rclone.org/t/cannot-purge-aws-s3/36169/
2023-02-27 16:32:59 +01:00
bucket string // bucket to list
directory string // directory with bucket
prefix string // prefix to remove from listing
addBucket bool // if set, the bucket is added to the start of the remote
recurse bool // if set, recurse to read sub directories
withVersions bool // if set, versions are produced
hidden bool // if set, return delete markers as objects with size == isDeleteMarker
findFile bool // if set, it will look for files called (bucket, directory)
versionAt fs.Time // if set only show versions <= this time
noSkipMarkers bool // if set return dir marker objects
s3: add rclone backend restore-status command This command shows the restore status of objects being retrieved from GLACIER. See: https://forum.rclone.org/t/aws-s3-glacier-monitor-restore-status-command-for-glacier-restoring-process/41373/7
2023-09-06 13:42:03 +02:00
restoreStatus bool // if set return restore status in listing too
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
}
// list lists the objects into the function supplied with the opt
// supplied.
func (f *Fs) list(ctx context.Context, opt listOpt, fn listFn) error {
s3: fix --s3-versions on individual objects Before this fix attempting to access an s3 versioned object by name in a subdirectory of root would not find the object. This fixes the problem and introduced an integraton test. See: https://forum.rclone.org/t/s3-versions-cant-retrieve-old-version/36900
2023-03-21 13:44:45 +01:00
if opt.prefix != "" {
opt.prefix += "/"
}
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
if !opt.findFile {
if opt.directory != "" {
opt.directory += "/"
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
}
Add a directory parameter to Fs.List()
2016-04-23 22:46:52 +02:00
}
s3: support sub-bucket paths
2014-05-05 19:25:32 +02:00
delimiter := ""
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
if !opt.recurse {
s3: support sub-bucket paths
2014-05-05 19:25:32 +02:00
delimiter = "/"
}
s3: only ask for URL encoded directory listings if we need them on Ceph This works around a bug in Ceph which doesn't encode CommonPrefixes when using URL encoded directory listings. See: https://tracker.ceph.com/issues/41870
2019-09-16 21:25:55 +02:00
// URL encode the listings so we can use control characters in object names
// See: https://github.com/aws/aws-sdk-go/issues/1914
//
// However this doesn't work perfectly under Ceph (and hence DigitalOcean/Dreamhost) because
// it doesn't encode CommonPrefixes.
// See: https://tracker.ceph.com/issues/41870
//
// This does not work under IBM COS also: See https://github.com/rclone/rclone/issues/3345
// though maybe it does on some versions.
//
// This does work with minio but was only added relatively recently
// https://github.com/minio/minio/pull/7265
//
// So we enable only on providers we know supports it properly, all others can retry when a
// XML Syntax error is detected.
s3: collect the provider quirks into a single function and update This removes the checks against the provider throughout the code and puts them into a single setQuirks function for easy maintenance when adding a new provider. It also updates the quirks with the results of testing against backends we have access to. This also adds a list_url_encode parameter so that quirk can be manually set.
2021-11-03 20:40:16 +01:00
urlEncodeListings := f.opt.ListURLEncode.Value
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
req := s3.ListObjectsV2Input{
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
Bucket: &opt.bucket,
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
Delimiter: &delimiter,
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
Prefix: &opt.directory,
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
MaxKeys: &f.opt.ListChunk,
}
s3: add rclone backend restore-status command This command shows the restore status of objects being retrieved from GLACIER. See: https://forum.rclone.org/t/aws-s3-glacier-monitor-restore-status-command-for-glacier-restoring-process/41373/7
2023-09-06 13:42:03 +02:00
if opt.restoreStatus {
restoreStatus := "RestoreStatus"
req.OptionalObjectAttributes = []*string{&restoreStatus}
}
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
if f.opt.RequesterPays {
req.RequestPayer = aws.String(s3.RequestPayerRequester)
}
var listBucket bucketLister
switch {
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
case opt.withVersions || opt.versionAt.IsSet():
listBucket = f.newVersionsList(&req, opt.hidden, time.Time(opt.versionAt))
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
case f.opt.ListVersion == 1:
listBucket = f.newV1List(&req)
default:
listBucket = f.newV2List(&req)
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
foundItems := 0
s3: list an unlimited number of items - fixes #22
2015-02-10 18:58:29 +01:00
for {
s3: Use ListObjectsV2 for faster listings Using ListObjectsV2 with a continuation token is about 5-6x faster than ListObjectsV2 with a marker.
2021-03-02 23:36:50 +01:00
var resp *s3.ListObjectsV2Output
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
var err error
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
var versionIDs []*string
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
err = f.pacer.Call(func() (bool, error) {
s3: factor different listing versions into separate objects
2022-07-25 16:56:17 +02:00
listBucket.URLEncodeListings(urlEncodeListings)
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
resp, versionIDs, err = listBucket.List(ctx)
s3: only ask for URL encoded directory listings if we need them on Ceph This works around a bug in Ceph which doesn't encode CommonPrefixes when using URL encoded directory listings. See: https://tracker.ceph.com/issues/41870
2019-09-16 21:25:55 +02:00
if err != nil && !urlEncodeListings {
if awsErr, ok := err.(awserr.RequestFailure); ok {
if origErr := awsErr.OrigErr(); origErr != nil {
if _, ok := origErr.(*xml.SyntaxError); ok {
// Retry the listing with URL encoding as there were characters that XML can't encode
urlEncodeListings = true
fs.Debugf(f, "Retrying listing because of characters which can't be XML encoded")
return true, err
}
}
}
}
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
})
s3: list an unlimited number of items - fixes #22
2015-02-10 18:58:29 +01:00
if err != nil {
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
if awsErr, ok := err.(awserr.RequestFailure); ok {
if awsErr.StatusCode() == http.StatusNotFound {
err = fs.ErrorDirNotFound
}
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
if f.rootBucket == "" {
// if listing from the root ignore wrong region requests returning
// empty directory
if reqErr, ok := err.(awserr.RequestFailure); ok {
// 301 if wrong region for bucket
if reqErr.StatusCode() == http.StatusMovedPermanently {
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
fs.Errorf(f, "Can't change region for bucket %q with no bucket specified", opt.bucket)
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
return nil
}
}
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
return err
}
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
if !opt.recurse {
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
foundItems += len(resp.CommonPrefixes)
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
for _, commonPrefix := range resp.CommonPrefixes {
if commonPrefix.Prefix == nil {
Rename log functions and factor into own file
2017-02-09 12:01:20 +01:00
fs.Logf(f, "Nil common prefix received")
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
continue
s3: list an unlimited number of items - fixes #22
2015-02-10 18:58:29 +01:00
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
remote := *commonPrefix.Prefix
s3: only ask for URL encoded directory listings if we need them on Ceph This works around a bug in Ceph which doesn't encode CommonPrefixes when using URL encoded directory listings. See: https://tracker.ceph.com/issues/41870
2019-09-16 21:25:55 +02:00
if urlEncodeListings {
remote, err = url.QueryUnescape(remote)
if err != nil {
fs.Logf(f, "failed to URL decode %q in listing common prefix: %v", *commonPrefix.Prefix, err)
continue
}
s3: fix encoding for control characters - Fixes #3345
2019-07-23 13:24:10 +02:00
}
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
remote = f.opt.Enc.ToStandardPath(remote)
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
if !strings.HasPrefix(remote, opt.prefix) {
Rename log functions and factor into own file
2017-02-09 12:01:20 +01:00
fs.Logf(f, "Odd name received %q", remote)
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
continue
}
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
remote = remote[len(opt.prefix):]
if opt.addBucket {
s3: use bucket.Join instead of path.Join to preserve paths Before this change, path.Join would remove the trailing / from objects which had them. The simplified bucket.Join does not.
2023-03-03 11:45:29 +01:00
remote = bucket.Join(opt.bucket, remote)
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
}
Fix lint issues reported by staticcheck Used staticcheck 2022.1.2 (v0.3.2) See: staticcheck.io
2022-06-08 22:25:17 +02:00
remote = strings.TrimSuffix(remote, "/")
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
err = fn(remote, &s3.Object{Key: &remote}, nil, true)
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
if err != nil {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
if err == errEndList {
return nil
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
return err
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
foundItems += len(resp.Contents)
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
for i, object := range resp.Contents {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
remote := aws.StringValue(object.Key)
s3: only ask for URL encoded directory listings if we need them on Ceph This works around a bug in Ceph which doesn't encode CommonPrefixes when using URL encoded directory listings. See: https://tracker.ceph.com/issues/41870
2019-09-16 21:25:55 +02:00
if urlEncodeListings {
remote, err = url.QueryUnescape(remote)
if err != nil {
fs.Logf(f, "failed to URL decode %q in listing: %v", aws.StringValue(object.Key), err)
continue
}
s3: fix encoding for control characters - Fixes #3345
2019-07-23 13:24:10 +02:00
}
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
remote = f.opt.Enc.ToStandardPath(remote)
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
if !strings.HasPrefix(remote, opt.prefix) {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
fs.Logf(f, "Odd name received %q", remote)
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
continue
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
isDirectory := (remote == "" || strings.HasSuffix(remote, "/")) && object.Size != nil && *object.Size == 0
s3: ignore zero length directory markers - fixes #1621
2018-03-19 18:41:46 +01:00
// is this a directory marker?
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
if isDirectory {
if opt.noSkipMarkers {
// process directory markers as files
isDirectory = false
} else {
// Don't insert the root directory
if remote == opt.directory {
continue
}
// process directory markers as directories
remote = strings.TrimRight(remote, "/")
}
s3: ignore zero length directory markers - fixes #1621
2018-03-19 18:41:46 +01:00
}
s3: fix "Entry doesn't belong in directory" errors when using directory markers Before this change we were incorrectly identifying the root directory of the listing and adding it into the listing. This caused higher layers of rclone to emit the error above. See #7038
2023-06-10 15:18:59 +02:00
remote = remote[len(opt.prefix):]
if opt.addBucket {
remote = bucket.Join(opt.bucket, remote)
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
if versionIDs != nil {
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
err = fn(remote, object, versionIDs[i], isDirectory)
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
} else {
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
err = fn(remote, object, nil, isDirectory)
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
if err != nil {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
if err == errEndList {
return nil
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
return err
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
}
s3: list an unlimited number of items - fixes #22
2015-02-10 18:58:29 +01:00
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
if !aws.BoolValue(resp.IsTruncated) {
break
}
s3: support sub-bucket paths
2014-05-05 19:25:32 +02:00
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
if f.opt.DirectoryMarkers && foundItems == 0 && opt.directory != "" {
// Determine whether the directory exists or not by whether it has a marker
req := s3.HeadObjectInput{
Bucket: &opt.bucket,
Key: &opt.directory,
}
_, err := f.headObject(ctx, &req)
if err != nil {
if err == fs.ErrorObjectNotFound {
return fs.ErrorDirNotFound
}
return err
}
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
return nil
s3: support sub-bucket paths
2014-05-05 19:25:32 +02:00
}
fs: rename BasicInfo to DirEntry
2017-06-30 11:54:14 +02:00
// Convert a list item into a DirEntry
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
func (f *Fs) itemToDirEntry(ctx context.Context, remote string, object *s3.Object, versionID *string, isDirectory bool) (fs.DirEntry, error) {
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
if isDirectory {
size := int64(0)
if object.Size != nil {
size = *object.Size
}
Create fs.Directory interface and use it everywhere
2017-06-30 14:37:29 +02:00
d := fs.NewDir(remote, time.Time{}).SetSize(size)
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
return d, nil
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
o, err := f.newObjectWithInfo(ctx, remote, object, versionID)
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
if err != nil {
return nil, err
}
return o, nil
}
// listDir lists files and directories to out
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
func (f *Fs) listDir(ctx context.Context, bucket, directory, prefix string, addBucket bool) (entries fs.DirEntries, err error) {
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
// List the objects and directories
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
err = f.list(ctx, listOpt{
bucket: bucket,
directory: directory,
prefix: prefix,
addBucket: addBucket,
withVersions: f.opt.Versions,
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
versionAt: f.opt.VersionAt,
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
}, func(remote string, object *s3.Object, versionID *string, isDirectory bool) error {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
entry, err := f.itemToDirEntry(ctx, remote, object, versionID, isDirectory)
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
if err != nil {
return err
}
if entry != nil {
entries = append(entries, entry)
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
}
return nil
})
if err != nil {
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
return nil, err
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
}
azureblob,b2,gcs,qingstor,s3,swift: Don't check for bucket/container presense if listing was OK In a typical rclone copy to a bucket/container based remote, before this change we were doing a list, followed by a HEAD of the bucket to check it existed before doing the copy. The fact the list succeeded means the bucket exists so mark it OK at that point. Issue #1421
2018-03-01 13:11:34 +01:00
// bucket must be present if listing succeeded
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
f.cache.MarkOK(bucket)
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
return entries, nil
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
}
// listBuckets lists the buckets to out
s3,b2,googlecloudstorage,swift,qingstor,azureblob: fixes after code review #3421 - change the interface of listBuckets() removing dir parameter and adding context - add makeBucket() and use in place of Mkdir("") - this fixes some corner cases in Copy/Update - mark all the listed buckets OK in ListR Thanks to @yparitcher for the review.
2019-08-22 22:30:55 +02:00
func (f *Fs) listBuckets(ctx context.Context) (entries fs.DirEntries, err error) {
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
req := s3.ListBucketsInput{}
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
var resp *s3.ListBucketsOutput
err = f.pacer.Call(func() (bool, error) {
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
resp, err = f.c.ListBucketsWithContext(ctx, &req)
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
})
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
if err != nil {
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
return nil, err
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 21:06:21 +02:00
}
for _, bucket := range resp.Buckets {
backend: adjust backends to have encoding parameter Fixes #3761 Fixes #3836 Fixes #3841
2020-01-14 18:33:35 +01:00
bucketName := f.opt.Enc.ToStandardName(aws.StringValue(bucket.Name))
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
f.cache.MarkOK(bucketName)
d := fs.NewDir(bucketName, aws.TimeValue(bucket.CreationDate))
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
entries = append(entries, d)
s3: support sub-bucket paths
2014-05-05 19:25:32 +02:00
}
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
return entries, nil
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
// List the objects and directories in dir into entries. The
// entries can be returned in any order but should be for a
// complete directory.
//
// dir should be "" to list the root, and should not have
// trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
bucket, directory := f.split(dir)
if bucket == "" {
s3,b2,googlecloudstorage,swift,qingstor,azureblob: fixes after code review #3421 - change the interface of listBuckets() removing dir parameter and adding context - add makeBucket() and use in place of Mkdir("") - this fixes some corner cases in Copy/Update - mark all the listed buckets OK in ListR Thanks to @yparitcher for the review.
2019-08-22 22:30:55 +02:00
if directory != "" {
return nil, fs.ErrorListBucketRequired
}
return f.listBuckets(ctx)
s3: support sub-bucket paths
2014-05-05 19:25:32 +02:00
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
return f.listDir(ctx, bucket, directory, f.rootDirectory, f.rootBucket == "")
Implement fs.ListDir / lsd command
2013-01-23 23:43:20 +01:00
}
Add placeholder support for ListR interface. The ListR interface will be implemented by remotes that can do a recursive directory listing more efficiently than just recursing through the directories. These include the bucket based remotes.
2017-06-05 17:14:24 +02:00
// ListR lists the objects and directories of the Fs starting
// from dir recursively into out.
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
//
// dir should be "" to start from the root, and should not
// have trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
//
// It should call callback for each tranche of entries read.
// These need not be returned in any particular order. If
// callback returns an error then the listing will stop
// immediately.
//
// Don't implement this unless you have a more efficient way
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
// of listing recursively than doing a directory traversal.
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (err error) {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
bucket, directory := f.split(dir)
Break the fs package up into smaller parts. The purpose of this is to make it easier to maintain and eventually to allow the rclone backends to be re-used in other projects without having to use the rclone configuration system. The new code layout is documented in CONTRIBUTING.
2018-01-12 17:30:54 +01:00
list := walk.NewListRHelper(callback)
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
listR := func(bucket, directory, prefix string, addBucket bool) error {
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
return f.list(ctx, listOpt{
bucket: bucket,
directory: directory,
prefix: prefix,
addBucket: addBucket,
recurse: true,
withVersions: f.opt.Versions,
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
versionAt: f.opt.VersionAt,
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
}, func(remote string, object *s3.Object, versionID *string, isDirectory bool) error {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
entry, err := f.itemToDirEntry(ctx, remote, object, versionID, isDirectory)
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
if err != nil {
return err
}
return list.Add(entry)
})
}
if bucket == "" {
s3,b2,googlecloudstorage,swift,qingstor,azureblob: fixes after code review #3421 - change the interface of listBuckets() removing dir parameter and adding context - add makeBucket() and use in place of Mkdir("") - this fixes some corner cases in Copy/Update - mark all the listed buckets OK in ListR Thanks to @yparitcher for the review.
2019-08-22 22:30:55 +02:00
entries, err := f.listBuckets(ctx)
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
if err != nil {
return err
}
for _, entry := range entries {
err = list.Add(entry)
if err != nil {
return err
}
bucket := entry.Remote()
err = listR(bucket, "", f.rootDirectory, true)
if err != nil {
return err
}
s3,b2,googlecloudstorage,swift,qingstor,azureblob: fixes after code review #3421 - change the interface of listBuckets() removing dir parameter and adding context - add makeBucket() and use in place of Mkdir("") - this fixes some corner cases in Copy/Update - mark all the listed buckets OK in ListR Thanks to @yparitcher for the review.
2019-08-22 22:30:55 +02:00
// bucket must be present if listing succeeded
f.cache.MarkOK(bucket)
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
}
} else {
err = listR(bucket, directory, f.rootDirectory, f.rootBucket == "")
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
if err != nil {
return err
}
s3,b2,googlecloudstorage,swift,qingstor,azureblob: fixes after code review #3421 - change the interface of listBuckets() removing dir parameter and adding context - add makeBucket() and use in place of Mkdir("") - this fixes some corner cases in Copy/Update - mark all the listed buckets OK in ListR Thanks to @yparitcher for the review.
2019-08-22 22:30:55 +02:00
// bucket must be present if listing succeeded
f.cache.MarkOK(bucket)
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 23:43:31 +02:00
}
return list.Flush()
Add placeholder support for ListR interface. The ListR interface will be implemented by remotes that can do a recursive directory listing more efficiently than just recursing through the directories. These include the bucket based remotes.
2017-06-05 17:14:24 +02:00
}
Rename NewFsObject to NewObject
2016-06-25 22:58:34 +02:00
// Put the Object into the bucket
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
// Temporary Object under construction
fs := &Object{
fs: f,
Create separate interface for object information. Take out read-only information about a Fs in a separate struct to limit access. See discussion at #282.
2016-02-18 12:35:25 +01:00
remote: src.Remote(),
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
}
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
return fs, fs.Update(ctx, in, src, options...)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: allow streaming upload of files with unknown file size (see #1614)
2017-09-15 20:20:32 +02:00
// PutStream uploads to the remote path with the modTime given of indeterminate size
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
return f.Put(ctx, in, src, options...)
s3: allow streaming upload of files with unknown file size (see #1614)
2017-09-15 20:20:32 +02:00
}
s3: Check if directory exists during Mkdir If you dont have privs to create a bucket in S3 but it exists, don't fail with an auth error, but detect that the mkdir was not needed and return successfully.
2016-02-24 02:58:55 +01:00
// Check if the bucket exists
s3: work around eventual consistency in bucket creation Deleting a bucket then testing its existence can give the wrong result. Work around by keeping a flag as to whether we have deleted the bucket.
2017-06-29 13:26:14 +02:00
//
// NB this can return incorrect results if called immediately after bucket deletion
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
func (f *Fs) bucketExists(ctx context.Context, bucket string) (bool, error) {
s3: Check if directory exists during Mkdir If you dont have privs to create a bucket in S3 but it exists, don't fail with an auth error, but detect that the mkdir was not needed and return successfully.
2016-02-24 02:58:55 +01:00
req := s3.HeadBucketInput{
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
Bucket: &bucket,
s3: Check if directory exists during Mkdir If you dont have privs to create a bucket in S3 but it exists, don't fail with an auth error, but detect that the mkdir was not needed and return successfully.
2016-02-24 02:58:55 +01:00
}
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
err := f.pacer.Call(func() (bool, error) {
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
_, err := f.c.HeadBucketWithContext(ctx, &req)
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
})
s3: Check if directory exists during Mkdir If you dont have privs to create a bucket in S3 but it exists, don't fail with an auth error, but detect that the mkdir was not needed and return successfully.
2016-02-24 02:58:55 +01:00
if err == nil {
return true, nil
}
if err, ok := err.(awserr.RequestFailure); ok {
if err.StatusCode() == http.StatusNotFound {
return false, nil
}
}
return false, err
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
// Create directory marker file and parents
func (f *Fs) createDirectoryMarker(ctx context.Context, bucket, dir string) error {
if !f.opt.DirectoryMarkers || bucket == "" {
return nil
}
// Object to be uploaded
o := &Object{
fs: f,
meta: map[string]string{
metaMtime: swift.TimeToFloatString(time.Now()),
},
}
for {
_, bucketPath := f.split(dir)
// Don't create the directory marker if it is the bucket or at the very root
if bucketPath == "" {
break
}
o.remote = dir + "/"
// Check to see if object already exists
_, err := o.headObject(ctx)
if err == nil {
return nil
}
// Upload it if not
fs.Debugf(o, "Creating directory marker")
content := io.Reader(strings.NewReader(""))
s3: fix directory marker code #3453 Use Update to upload the directory markers
2023-04-28 18:22:45 +02:00
err = o.Update(ctx, content, o)
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
if err != nil {
return fmt.Errorf("creating directory marker failed: %w", err)
}
// Now check parent directory exists
dir = path.Dir(dir)
if dir == "/" || dir == "." {
break
}
}
return nil
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
// Mkdir creates the bucket if it doesn't exist
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (f *Fs) Mkdir(ctx context.Context, dir string) error {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
bucket, _ := f.split(dir)
s3: empty directory markers - #3453
2022-09-26 08:59:57 +02:00
e := f.makeBucket(ctx, bucket)
if e != nil {
return e
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
return f.createDirectoryMarker(ctx, bucket, dir)
}
// mkdirParent creates the parent bucket/directory if it doesn't exist
func (f *Fs) mkdirParent(ctx context.Context, remote string) error {
remote = strings.TrimRight(remote, "/")
dir := path.Dir(remote)
if dir == "/" || dir == "." {
dir = ""
s3: empty directory markers - #3453
2022-09-26 08:59:57 +02:00
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
return f.Mkdir(ctx, dir)
s3,b2,googlecloudstorage,swift,qingstor,azureblob: fixes after code review #3421 - change the interface of listBuckets() removing dir parameter and adding context - add makeBucket() and use in place of Mkdir("") - this fixes some corner cases in Copy/Update - mark all the listed buckets OK in ListR Thanks to @yparitcher for the review.
2019-08-22 22:30:55 +02:00
}
// makeBucket creates the bucket if it doesn't exist
func (f *Fs) makeBucket(ctx context.Context, bucket string) error {
s3: add --s3-no-check-bucket for minimising rclone transactions and perms Fixes #4449
2020-07-22 13:02:17 +02:00
if f.opt.NoCheckBucket {
return nil
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
return f.cache.Create(bucket, func() error {
req := s3.CreateBucketInput{
Bucket: &bucket,
s3: if bucket or object ACL is empty string then don't add X-Amz-Acl: header - fixes #5730 Before this fix it was impossible to stop rclone generating an X-Amx-Acl: header which is incompatible with GCS with uniform access control and is generally deprecated at AWS.
2022-09-14 17:24:57 +02:00
ACL: stringPointerOrNil(f.opt.BucketACL),
s3: work around eventual consistency in bucket creation Deleting a bucket then testing its existence can give the wrong result. Work around by keeping a flag as to whether we have deleted the bucket.
2017-06-29 13:26:14 +02:00
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
if f.opt.LocationConstraint != "" {
req.CreateBucketConfiguration = &s3.CreateBucketConfiguration{
LocationConstraint: &f.opt.LocationConstraint,
}
s3: work around eventual consistency in bucket creation Deleting a bucket then testing its existence can give the wrong result. Work around by keeping a flag as to whether we have deleted the bucket.
2017-06-29 13:26:14 +02:00
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
err := f.pacer.Call(func() (bool, error) {
_, err := f.c.CreateBucketWithContext(ctx, &req)
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
})
if err == nil {
fs.Infof(f, "Bucket %q created with ACL %q", bucket, f.opt.BucketACL)
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
}
s3: fix detection of BucketAlreadyOwnedByYou and BucketAlreadyExists error This was being silently ignored until this commit e2bf91452a263ff8 s3: report errors on bucket creation (mkdir) correctly
2020-04-22 19:01:59 +02:00
if awsErr, ok := err.(awserr.Error); ok {
if code := awsErr.Code(); code == "BucketAlreadyOwnedByYou" || code == "BucketAlreadyExists" {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
err = nil
}
S3: Fix mkdir not to error if bucket exists and implement SetModTime
2013-01-08 23:31:16 +01:00
}
s3: report errors on bucket creation (mkdir) correctly Before this fix errors on bucket creation were being silently swallowed. See: https://forum.rclone.org/t/rclone-with-brand-new-aws-account-for-s3/15590
2020-04-15 14:13:13 +02:00
return err
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
}, func() (bool, error) {
return f.bucketExists(ctx, bucket)
})
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
swift, s3, googlecloudstorage: Don't delete the container/bucket if fs wasn't at root - fixes #172
2015-11-07 16:31:04 +01:00
// Rmdir deletes the bucket if the fs is at the root
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
//
// Returns an error if it isn't empty
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (f *Fs) Rmdir(ctx context.Context, dir string) error {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
bucket, directory := f.split(dir)
s3: empty directory markers - #3453
2022-09-26 08:59:57 +02:00
// Remove directory marker file
if f.opt.DirectoryMarkers && bucket != "" && dir != "" {
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
o := &Object{
s3: empty directory markers - #3453
2022-09-26 08:59:57 +02:00
fs: f,
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
remote: dir + "/",
}
fs.Debugf(o, "Removing directory marker")
err := o.Remove(ctx)
if err != nil {
return fmt.Errorf("removing directory marker failed: %w", err)
s3: empty directory markers - #3453
2022-09-26 08:59:57 +02:00
}
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
if bucket == "" || directory != "" {
swift, s3, googlecloudstorage: Don't delete the container/bucket if fs wasn't at root - fixes #172
2015-11-07 16:31:04 +01:00
return nil
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
return f.cache.Remove(bucket, func() error {
req := s3.DeleteBucketInput{
Bucket: &bucket,
}
err := f.pacer.Call(func() (bool, error) {
_, err := f.c.DeleteBucketWithContext(ctx, &req)
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
})
if err == nil {
fs.Infof(f, "Bucket %q deleted", bucket)
}
return err
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
})
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
Fix golint warnings
2015-09-22 19:47:16 +02:00
// Precision of the remote
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
func (f *Fs) Precision() time.Duration {
Implement Precision interface in all filesystems * drive: add mime type on upload * drive: correct info on upload to fix crash * local: measure time precision by making a local file and using Chtimes * swift: move snet parameter here * core: add -modify-window and compute the optimum one * core: use modify window when checking times
2013-01-19 00:21:02 +01:00
return time.Nanosecond
}
s3: fix server side copy and set modtime on files with + in - fixes #2001 This was broken in 64ea94c1a4e9d7a104b393a17d2e883593e20809 when putting a work-around for Digital Ocean. PathEscape has now been adjusted so it works with both providers.
2018-01-23 11:50:50 +01:00
// pathEscape escapes s as for a URL path. It uses rest.URLPathEscape
// but also escapes '+' for S3 and Digital Ocean spaces compatibility
func pathEscape(s string) string {
refactor: replace strings.Replace with strings.ReplaceAll strings.ReplaceAll(s, old, new) is a wrapper function for strings.Replace(s, old, new, -1). But strings.ReplaceAll is more readable and removes the hardcoded -1. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-05-16 18:11:45 +02:00
return strings.ReplaceAll(rest.URLPathEscape(s), "+", "%2B")
s3: fix server side copy and set modtime on files with + in - fixes #2001 This was broken in 64ea94c1a4e9d7a104b393a17d2e883593e20809 when putting a work-around for Digital Ocean. PathEscape has now been adjusted so it works with both providers.
2018-01-23 11:50:50 +01:00
}
docs: spelling: server-side Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-13 23:43:40 +02:00
// copy does a server-side copy
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
//
// It adds the boiler plate to the req passed in and calls the s3
// method
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
func (f *Fs) copy(ctx context.Context, req *s3.CopyObjectInput, dstBucket, dstPath, srcBucket, srcPath string, src *Object) error {
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
req.Bucket = &dstBucket
s3: if bucket or object ACL is empty string then don't add X-Amz-Acl: header - fixes #5730 Before this fix it was impossible to stop rclone generating an X-Amx-Acl: header which is incompatible with GCS with uniform access control and is generally deprecated at AWS.
2022-09-14 17:24:57 +02:00
req.ACL = stringPointerOrNil(f.opt.ACL)
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
req.Key = &dstPath
s3: use bucket.Join instead of path.Join to preserve paths Before this change, path.Join would remove the trailing / from objects which had them. The simplified bucket.Join does not.
2023-03-03 11:45:29 +01:00
source := pathEscape(bucket.Join(srcBucket, srcPath))
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
if src.versionID != nil {
source += fmt.Sprintf("?versionId=%s", *src.versionID)
}
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
req.CopySource = &source
s3: add requester pays option - fixes #301
2020-12-03 03:30:06 +01:00
if f.opt.RequesterPays {
req.RequestPayer = aws.String(s3.RequestPayerRequester)
}
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
if f.opt.ServerSideEncryption != "" {
req.ServerSideEncryption = &f.opt.ServerSideEncryption
}
s3: complete sse-c implementation This now can complete all operations with SSE-C enabled. Fixes #2827 See: https://forum.rclone.org/t/issues-with-aws-s3-sse-c-getting-strange-log-entries-and-errors/20553
2020-11-20 12:15:48 +01:00
if f.opt.SSECustomerAlgorithm != "" {
req.SSECustomerAlgorithm = &f.opt.SSECustomerAlgorithm
req.CopySourceSSECustomerAlgorithm = &f.opt.SSECustomerAlgorithm
}
if f.opt.SSECustomerKey != "" {
req.SSECustomerKey = &f.opt.SSECustomerKey
req.CopySourceSSECustomerKey = &f.opt.SSECustomerKey
}
if f.opt.SSECustomerKeyMD5 != "" {
req.SSECustomerKeyMD5 = &f.opt.SSECustomerKeyMD5
req.CopySourceSSECustomerKeyMD5 = &f.opt.SSECustomerKeyMD5
}
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
if f.opt.SSEKMSKeyID != "" {
req.SSEKMSKeyId = &f.opt.SSEKMSKeyID
}
if req.StorageClass == nil && f.opt.StorageClass != "" {
req.StorageClass = &f.opt.StorageClass
}
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
if src.bytes >= int64(f.opt.CopyCutoff) {
return f.copyMultipart(ctx, req, dstBucket, dstPath, srcBucket, srcPath, src)
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
}
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
return f.pacer.Call(func() (bool, error) {
_, err := f.c.CopyObjectWithContext(ctx, req)
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
})
}
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
func calculateRange(partSize, partIndex, numParts, totalSize int64) string {
start := partIndex * partSize
var ends string
if partIndex == numParts-1 {
s3: fix multipart copy - fixes #3778 Before this change multipart copies were giving the error Range specified is not valid for source object of size This was due to an off by one error in the range source introduced in 7b1274e29ac51231 "s3: support for multipart copy"
2019-12-02 18:00:54 +01:00
if totalSize >= 1 {
ends = strconv.FormatInt(totalSize-1, 10)
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
}
} else {
ends = strconv.FormatInt(start+partSize-1, 10)
}
return fmt.Sprintf("bytes=%v-%v", start, ends)
}
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
func (f *Fs) copyMultipart(ctx context.Context, copyReq *s3.CopyObjectInput, dstBucket, dstPath, srcBucket, srcPath string, src *Object) (err error) {
info, err := src.headObject(ctx)
if err != nil {
return err
}
req := &s3.CreateMultipartUploadInput{}
// Fill in the request from the head info
s3: refactor to use generated code instead of reflection to copy structs
2022-07-28 12:49:19 +02:00
//structs.SetFrom(req, info)
setFrom_s3CreateMultipartUploadInput_s3HeadObjectOutput(req, info)
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
// If copy metadata was set then set the Metadata to that read
// from the head request
if aws.StringValue(copyReq.MetadataDirective) == s3.MetadataDirectiveCopy {
copyReq.Metadata = info.Metadata
}
// Overwrite any from the copyReq
s3: refactor to use generated code instead of reflection to copy structs
2022-07-28 12:49:19 +02:00
//structs.SetFrom(req, copyReq)
setFrom_s3CreateMultipartUploadInput_s3CopyObjectInput(req, copyReq)
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
req.Bucket = &dstBucket
req.Key = &dstPath
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
var cout *s3.CreateMultipartUploadOutput
if err := f.pacer.Call(func() (bool, error) {
var err error
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
cout, err = f.c.CreateMultipartUploadWithContext(ctx, req)
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
}); err != nil {
return err
}
uid := cout.UploadId
s3: cancel in progress multipart uploads and copies on rclone exit #4300
2020-06-04 12:09:27 +02:00
defer atexit.OnError(&err, func() {
// Try to abort the upload, but ignore the error.
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
fs.Debugf(src, "Cancelling multipart copy")
s3: cancel in progress multipart uploads and copies on rclone exit #4300
2020-06-04 12:09:27 +02:00
_ = f.pacer.Call(func() (bool, error) {
_, err := f.c.AbortMultipartUploadWithContext(context.Background(), &s3.AbortMultipartUploadInput{
Bucket: &dstBucket,
Key: &dstPath,
UploadId: uid,
RequestPayer: req.RequestPayer,
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
})
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
s3: cancel in progress multipart uploads and copies on rclone exit #4300
2020-06-04 12:09:27 +02:00
})
})()
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
srcSize := src.bytes
s3: add --s3-copy-cutoff for size to switch to multipart copy Before this change we used the same (relatively low limits) for server side copy as we did for multipart uploads. It doesn't make sense to use the same limits since no data is being downloaded or uploaded for a server side copy. This change introduces a new parameter --s3-copy-cutoff to control when the switch from single to multipart server size copy happens and defaults it to the maximum 5GB. This makes server side copies much more efficient. It also fixes the erroneous error when trying to set the modification time of a file bigger than 5GB. See #3778
2019-12-02 18:14:57 +01:00
partSize := int64(f.opt.CopyCutoff)
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
numParts := (srcSize-1)/partSize + 1
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
fs.Debugf(src, "Starting multipart copy with %d parts", numParts)
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
var parts []*s3.CompletedPart
for partNum := int64(1); partNum <= numParts; partNum++ {
if err := f.pacer.Call(func() (bool, error) {
partNum := partNum
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
uploadPartReq := &s3.UploadPartCopyInput{}
s3: refactor to use generated code instead of reflection to copy structs
2022-07-28 12:49:19 +02:00
//structs.SetFrom(uploadPartReq, copyReq)
setFrom_s3UploadPartCopyInput_s3CopyObjectInput(uploadPartReq, copyReq)
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
uploadPartReq.Bucket = &dstBucket
uploadPartReq.Key = &dstPath
uploadPartReq.PartNumber = &partNum
uploadPartReq.UploadId = uid
uploadPartReq.CopySourceRange = aws.String(calculateRange(partSize, partNum-1, numParts, srcSize))
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
uout, err := f.c.UploadPartCopyWithContext(ctx, uploadPartReq)
if err != nil {
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
}
parts = append(parts, &s3.CompletedPart{
PartNumber: &partNum,
ETag: uout.CopyPartResult.ETag,
})
return false, nil
}); err != nil {
return err
}
}
return f.pacer.Call(func() (bool, error) {
_, err := f.c.CompleteMultipartUploadWithContext(ctx, &s3.CompleteMultipartUploadInput{
Bucket: &dstBucket,
Key: &dstPath,
MultipartUpload: &s3.CompletedMultipartUpload{
Parts: parts,
},
RequestPayer: req.RequestPayer,
UploadId: uid,
})
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
s3: support for multipart copy Fixes #2375 Fixes #3579
2019-10-04 17:49:06 +02:00
})
}
docs: spelling: server-side Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-13 23:43:40 +02:00
// Copy src to this remote using server-side copy operations.
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 19:48:08 +01:00
//
build: reformat comments to pass go1.19 vet See: https://go.dev/doc/go1.19#go-doc
2022-08-05 17:35:41 +02:00
// This is stored with the remote path given.
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 19:48:08 +01:00
//
build: reformat comments to pass go1.19 vet See: https://go.dev/doc/go1.19#go-doc
2022-08-05 17:35:41 +02:00
// It returns the destination Object and a possible error.
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 19:48:08 +01:00
//
// Will only be called if src.Fs().Name() == f.Name()
//
// If it isn't possible then return fs.ErrorCantCopy
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
if f.opt.VersionAt.IsSet() {
return nil, errNotWithVersionAt
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
dstBucket, dstPath := f.split(remote)
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
err := f.mkdirParent(ctx, remote)
swift, s3, gcs: create container if necessary on server side copy
2017-06-28 22:14:53 +02:00
if err != nil {
return nil, err
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
srcObj, ok := src.(*Object)
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 19:48:08 +01:00
if !ok {
Rename log functions and factor into own file
2017-02-09 12:01:20 +01:00
fs.Debugf(src, "Can't copy - not same remote type")
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 19:48:08 +01:00
return nil, fs.ErrorCantCopy
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
srcBucket, srcPath := srcObj.split()
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 19:48:08 +01:00
req := s3.CopyObjectInput{
MetadataDirective: aws.String(s3.MetadataDirectiveCopy),
}
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
err = f.copy(ctx, &req, dstBucket, dstPath, srcBucket, srcPath, srcObj)
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 19:48:08 +01:00
if err != nil {
return nil, err
}
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
return f.NewObject(ctx, remote)
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 19:48:08 +01:00
}
Add support for multiple hash types. Add support for multiple hash types with negotiation of common hash types for comparison. Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
// Hashes returns the supported hash sets.
Break the fs package up into smaller parts. The purpose of this is to make it easier to maintain and eventually to allow the rclone backends to be re-used in other projects without having to use the rclone configuration system. The new code layout is documented in CONTRIBUTING.
2018-01-12 17:30:54 +01:00
func (f *Fs) Hashes() hash.Set {
config/hash: rename more symbols after factoring into own package
2018-01-18 21:27:52 +01:00
return hash.Set(hash.MD5)
Add support for multiple hash types. Add support for multiple hash types with negotiation of common hash types for comparison. Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
}
s3: implement link sharing with PublicLink
2020-06-04 23:03:12 +02:00
// PublicLink generates a public link to the remote path (usually readable by anyone)
func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration, unlink bool) (link string, err error) {
s3: Cap expiry duration to 1 Week and return error when sharing dir
2020-06-18 18:50:50 +02:00
if strings.HasSuffix(remote, "/") {
return "", fs.ErrorCantShareDirectories
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
obj, err := f.NewObject(ctx, remote)
if err != nil {
s3: implement link sharing with PublicLink
2020-06-04 23:03:12 +02:00
return "", err
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
o := obj.(*Object)
s3: Cap expiry duration to 1 Week and return error when sharing dir
2020-06-18 18:50:50 +02:00
if expire > maxExpireDuration {
fs.Logf(f, "Public Link: Reducing expiry to %v as %v is greater than the max time allowed", maxExpireDuration, expire)
expire = maxExpireDuration
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
bucket, bucketPath := o.split()
s3: implement link sharing with PublicLink
2020-06-04 23:03:12 +02:00
httpReq, _ := f.c.GetObjectRequest(&s3.GetObjectInput{
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
Bucket: &bucket,
Key: &bucketPath,
VersionId: o.versionID,
s3: implement link sharing with PublicLink
2020-06-04 23:03:12 +02:00
})
return httpReq.Presign(time.Duration(expire))
}
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
var commandHelp = []fs.CommandHelp{{
Name: "restore",
Short: "Restore objects from GLACIER to normal storage",
Long: `This command can be used to restore one or more objects from GLACIER
to normal storage.
Usage Examples:
docs: rclone backend restore
2023-07-10 19:12:54 +02:00
rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS
rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS
rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
docs: use --interactive instead of -i in examples to avoid confusion
2023-01-20 21:47:36 +01:00
This flag also obeys the filters. Test first with --interactive/-i or --dry-run flags
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
docs: rclone backend restore
2023-07-10 19:12:54 +02:00
rclone --interactive backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
All the objects shown will be marked for restore, then
docs: rclone backend restore
2023-07-10 19:12:54 +02:00
rclone backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
It returns a list of status dictionaries with Remote and Status
Spelling fixes Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-09 02:17:24 +02:00
keys. The Status will be OK if it was successful or an error message
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
if not.
[
{
"Status": "OK",
s3: add rclone backend restore-status command This command shows the restore status of objects being retrieved from GLACIER. See: https://forum.rclone.org/t/aws-s3-glacier-monitor-restore-status-command-for-glacier-restoring-process/41373/7
2023-09-06 13:42:03 +02:00
"Remote": "test.txt"
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
},
{
"Status": "OK",
s3: add rclone backend restore-status command This command shows the restore status of objects being retrieved from GLACIER. See: https://forum.rclone.org/t/aws-s3-glacier-monitor-restore-status-command-for-glacier-restoring-process/41373/7
2023-09-06 13:42:03 +02:00
"Remote": "test/file4.txt"
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
}
]
`,
Opts: map[string]string{
"priority": "Priority of restore: Standard|Expedited|Bulk",
"lifetime": "Lifetime of the active copy in days",
"description": "The optional description for the job.",
},
s3: add rclone backend restore-status command This command shows the restore status of objects being retrieved from GLACIER. See: https://forum.rclone.org/t/aws-s3-glacier-monitor-restore-status-command-for-glacier-restoring-process/41373/7
2023-09-06 13:42:03 +02:00
}, {
Name: "restore-status",
Short: "Show the restore status for objects being restored from GLACIER to normal storage",
Long: `This command can be used to show the status for objects being restored from GLACIER
to normal storage.
Usage Examples:
rclone backend restore-status s3:bucket/path/to/object
rclone backend restore-status s3:bucket/path/to/directory
rclone backend restore-status -o all s3:bucket/path/to/directory
This command does not obey the filters.
It returns a list of status dictionaries.
[
{
"Remote": "file.txt",
"VersionID": null,
"RestoreStatus": {
"IsRestoreInProgress": true,
"RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
},
"StorageClass": "GLACIER"
},
{
"Remote": "test.pdf",
"VersionID": null,
"RestoreStatus": {
"IsRestoreInProgress": false,
"RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
},
"StorageClass": "DEEP_ARCHIVE"
}
]
`,
Opts: map[string]string{
"all": "if set then show all objects, not just ones with restore status",
},
s3: implement cleanup and backend command to list & remove multipart uploads This implements `rclone cleanup` to remove multipart uploads over 24 hours old. It also implements the backend command `list-multipart-uploads` to see which ones are available and `cleanup` to delete them with a configurable expiry interval. See #4302
2020-06-25 17:11:05 +02:00
}, {
Name: "list-multipart-uploads",
Short: "List the unfinished multipart uploads",
Long: `This command lists the unfinished multipart uploads in JSON format.
rclone backend list-multipart s3:bucket/path/to/object
It returns a dictionary of buckets with values as lists of unfinished
multipart uploads.
You can call it with no bucket in which case it lists all bucket, with
a bucket or with a bucket and path.
{
"rclone": [
{
"Initiated": "2020-06-26T14:20:36Z",
"Initiator": {
"DisplayName": "XXX",
"ID": "arn:aws:iam::XXX:user/XXX"
},
"Key": "KEY",
"Owner": {
"DisplayName": null,
"ID": "XXX"
},
"StorageClass": "STANDARD",
"UploadId": "XXX"
}
],
"rclone-1000files": [],
"rclone-dst": []
}
`,
}, {
Name: "cleanup",
Short: "Remove unfinished multipart uploads.",
Long: `This command removes unfinished multipart uploads of age greater than
max-age which defaults to 24 hours.
docs: use --interactive instead of -i in examples to avoid confusion
2023-01-20 21:47:36 +01:00
Note that you can use --interactive/-i or --dry-run with this command to see what
it would do.
s3: implement cleanup and backend command to list & remove multipart uploads This implements `rclone cleanup` to remove multipart uploads over 24 hours old. It also implements the backend command `list-multipart-uploads` to see which ones are available and `cleanup` to delete them with a configurable expiry interval. See #4302
2020-06-25 17:11:05 +02:00
rclone backend cleanup s3:bucket/path/to/object
rclone backend cleanup -o max-age=7w s3:bucket/path/to/object
Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc.
`,
Opts: map[string]string{
"max-age": "Max age of upload to delete",
},
s3: implement Purge to purge versions and `backend cleanup-hidden`
2022-07-26 16:03:32 +02:00
}, {
Name: "cleanup-hidden",
Short: "Remove old versions of files.",
Long: `This command removes any old hidden versions of files
on a versions enabled bucket.
docs: use --interactive instead of -i in examples to avoid confusion
2023-01-20 21:47:36 +01:00
Note that you can use --interactive/-i or --dry-run with this command to see what
it would do.
s3: implement Purge to purge versions and `backend cleanup-hidden`
2022-07-26 16:03:32 +02:00
rclone backend cleanup-hidden s3:bucket/path/to/dir
`,
s3: implement backend versioning command to get/set bucket versioning
2022-07-25 17:05:17 +02:00
}, {
Name: "versioning",
Short: "Set/get versioning support for a bucket.",
Long: `This command sets versioning support if a parameter is
passed and then returns the current versioning status for the bucket
supplied.
rclone backend versioning s3:bucket # read status only
rclone backend versioning s3:bucket Enabled
rclone backend versioning s3:bucket Suspended
It may return "Enabled", "Suspended" or "Unversioned". Note that once versioning
has been enabled the status can't be set back to "Unversioned".
`,
s3: implement backend set command to update running config
2023-07-15 19:56:58 +02:00
}, {
Name: "set",
Short: "Set command for updating the config parameters.",
Long: `This set command can be used to update the config parameters
for a running s3 backend.
Usage Examples:
rclone backend set s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
rclone rc backend/command command=set fs=s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
rclone rc backend/command command=set fs=s3: -o session_token=X -o access_key_id=X -o secret_access_key=X
The option keys are named as they are in the config file.
This rebuilds the connection to the s3 backend when it is called with
the new parameters. Only new parameters need be passed as the values
will default to those currently in use.
It doesn't return anything.
`,
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
}}
// Command the backend to run a named command
//
// The command run is name
// args may be used to read arguments from
// opts may be used to read optional arguments from
//
// The result should be capable of being JSON encoded
// If it is a string or a []string it will be shown to the user
// otherwise it will be JSON encoded and shown to the user like that
func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[string]string) (out interface{}, err error) {
switch name {
case "restore":
req := s3.RestoreObjectInput{
//Bucket: &f.rootBucket,
//Key: &encodedDirectory,
RestoreRequest: &s3.RestoreRequest{},
}
if lifetime := opt["lifetime"]; lifetime != "" {
ilifetime, err := strconv.ParseInt(lifetime, 10, 64)
if err != nil {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return nil, fmt.Errorf("bad lifetime: %w", err)
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
}
req.RestoreRequest.Days = &ilifetime
}
if priority := opt["priority"]; priority != "" {
req.RestoreRequest.GlacierJobParameters = &s3.GlacierJobParameters{
Tier: &priority,
}
}
if description := opt["description"]; description != "" {
req.RestoreRequest.Description = &description
}
type status struct {
Status string
Remote string
}
var (
outMu sync.Mutex
out = []status{}
)
err = operations.ListFn(ctx, f, func(obj fs.Object) {
// Remember this is run --checkers times concurrently
o, ok := obj.(*Object)
st := status{Status: "OK", Remote: obj.Remote()}
defer func() {
outMu.Lock()
out = append(out, st)
outMu.Unlock()
}()
if operations.SkipDestructive(ctx, obj, "restore") {
return
}
if !ok {
st.Status = "Not an S3 object"
return
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
if o.storageClass == nil || (*o.storageClass != "GLACIER" && *o.storageClass != "DEEP_ARCHIVE") {
s3: backend restore command to skip non-GLACIER objects
2022-05-12 21:42:37 +02:00
st.Status = "Not GLACIER or DEEP_ARCHIVE storage class"
return
}
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
bucket, bucketPath := o.split()
reqCopy := req
reqCopy.Bucket = &bucket
reqCopy.Key = &bucketPath
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
reqCopy.VersionId = o.versionID
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
err = f.pacer.Call(func() (bool, error) {
_, err = f.c.RestoreObject(&reqCopy)
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
})
if err != nil {
st.Status = err.Error()
}
})
if err != nil {
return out, err
}
return out, nil
s3: add rclone backend restore-status command This command shows the restore status of objects being retrieved from GLACIER. See: https://forum.rclone.org/t/aws-s3-glacier-monitor-restore-status-command-for-glacier-restoring-process/41373/7
2023-09-06 13:42:03 +02:00
case "restore-status":
_, all := opt["all"]
return f.restoreStatus(ctx, all)
s3: implement cleanup and backend command to list & remove multipart uploads This implements `rclone cleanup` to remove multipart uploads over 24 hours old. It also implements the backend command `list-multipart-uploads` to see which ones are available and `cleanup` to delete them with a configurable expiry interval. See #4302
2020-06-25 17:11:05 +02:00
case "list-multipart-uploads":
return f.listMultipartUploadsAll(ctx)
case "cleanup":
maxAge := 24 * time.Hour
if opt["max-age"] != "" {
maxAge, err = fs.ParseDuration(opt["max-age"])
if err != nil {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return nil, fmt.Errorf("bad max-age: %w", err)
s3: implement cleanup and backend command to list & remove multipart uploads This implements `rclone cleanup` to remove multipart uploads over 24 hours old. It also implements the backend command `list-multipart-uploads` to see which ones are available and `cleanup` to delete them with a configurable expiry interval. See #4302
2020-06-25 17:11:05 +02:00
}
}
return nil, f.cleanUp(ctx, maxAge)
s3: implement Purge to purge versions and `backend cleanup-hidden`
2022-07-26 16:03:32 +02:00
case "cleanup-hidden":
return nil, f.CleanUpHidden(ctx)
s3: implement backend versioning command to get/set bucket versioning
2022-07-25 17:05:17 +02:00
case "versioning":
return f.setGetVersioning(ctx, arg...)
s3: implement backend set command to update running config
2023-07-15 19:56:58 +02:00
case "set":
newOpt := f.opt
err := configstruct.Set(configmap.Simple(opt), &newOpt)
if err != nil {
return nil, fmt.Errorf("reading config: %w", err)
}
c, ses, err := s3Connection(f.ctx, &newOpt, f.srv)
if err != nil {
return nil, fmt.Errorf("updating session: %w", err)
}
f.c = c
f.ses = ses
f.opt = newOpt
keys := []string{}
for k := range opt {
keys = append(keys, k)
}
fs.Logf(f, "Updated config values: %s", strings.Join(keys, ", "))
return nil, nil
s3: add backend restore command to restore objects from GLACIER See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-24 12:02:34 +02:00
default:
return nil, fs.ErrorCommandNotFound
}
}
s3: add rclone backend restore-status command This command shows the restore status of objects being retrieved from GLACIER. See: https://forum.rclone.org/t/aws-s3-glacier-monitor-restore-status-command-for-glacier-restoring-process/41373/7
2023-09-06 13:42:03 +02:00
// Returned from "restore-status"
type restoreStatusOut struct {
Remote string
VersionID *string
RestoreStatus *s3.RestoreStatus
StorageClass *string
}
// Recursively enumerate the current fs to find objects with a restore status
func (f *Fs) restoreStatus(ctx context.Context, all bool) (out []restoreStatusOut, err error) {
fs.Debugf(f, "all = %v", all)
bucket, directory := f.split("")
out = []restoreStatusOut{}
err = f.list(ctx, listOpt{
bucket: bucket,
directory: directory,
prefix: f.rootDirectory,
addBucket: f.rootBucket == "",
recurse: true,
withVersions: f.opt.Versions,
versionAt: f.opt.VersionAt,
restoreStatus: true,
}, func(remote string, object *s3.Object, versionID *string, isDirectory bool) error {
entry, err := f.itemToDirEntry(ctx, remote, object, versionID, isDirectory)
if err != nil {
return err
}
if entry != nil {
if o, ok := entry.(*Object); ok && (all || object.RestoreStatus != nil) {
out = append(out, restoreStatusOut{
Remote: o.remote,
VersionID: o.versionID,
RestoreStatus: object.RestoreStatus,
StorageClass: object.StorageClass,
})
}
}
return nil
})
if err != nil {
return nil, err
}
// bucket must be present if listing succeeded
f.cache.MarkOK(bucket)
return out, nil
}
s3: implement cleanup and backend command to list & remove multipart uploads This implements `rclone cleanup` to remove multipart uploads over 24 hours old. It also implements the backend command `list-multipart-uploads` to see which ones are available and `cleanup` to delete them with a configurable expiry interval. See #4302
2020-06-25 17:11:05 +02:00
// listMultipartUploads lists all outstanding multipart uploads for (bucket, key)
//
// Note that rather lazily we treat key as a prefix so it matches
Spelling fixes Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-09 02:17:24 +02:00
// directories and objects. This could surprise the user if they ask
s3: implement cleanup and backend command to list & remove multipart uploads This implements `rclone cleanup` to remove multipart uploads over 24 hours old. It also implements the backend command `list-multipart-uploads` to see which ones are available and `cleanup` to delete them with a configurable expiry interval. See #4302
2020-06-25 17:11:05 +02:00
// for "dir" and it returns "dirKey"
func (f *Fs) listMultipartUploads(ctx context.Context, bucket, key string) (uploads []*s3.MultipartUpload, err error) {
var (
keyMarker *string
uploadIDMarker *string
)
uploads = []*s3.MultipartUpload{}
for {
req := s3.ListMultipartUploadsInput{
Bucket: &bucket,
MaxUploads: &f.opt.ListChunk,
KeyMarker: keyMarker,
UploadIdMarker: uploadIDMarker,
Prefix: &key,
}
var resp *s3.ListMultipartUploadsOutput
err = f.pacer.Call(func() (bool, error) {
resp, err = f.c.ListMultipartUploads(&req)
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return f.shouldRetry(ctx, err)
s3: implement cleanup and backend command to list & remove multipart uploads This implements `rclone cleanup` to remove multipart uploads over 24 hours old. It also implements the backend command `list-multipart-uploads` to see which ones are available and `cleanup` to delete them with a configurable expiry interval. See #4302
2020-06-25 17:11:05 +02:00
})
if err != nil {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return nil, fmt.Errorf("list multipart uploads bucket %q key %q: %w", bucket, key, err)
s3: implement cleanup and backend command to list & remove multipart uploads This implements `rclone cleanup` to remove multipart uploads over 24 hours old. It also implements the backend command `list-multipart-uploads` to see which ones are available and `cleanup` to delete them with a configurable expiry interval. See #4302
2020-06-25 17:11:05 +02:00
}
uploads = append(uploads, resp.Uploads...)
if !aws.BoolValue(resp.IsTruncated) {
break
}
keyMarker = resp.NextKeyMarker
uploadIDMarker = resp.NextUploadIdMarker
}
return uploads, nil
}
func (f *Fs) listMultipartUploadsAll(ctx context.Context) (uploadsMap map[string][]*s3.MultipartUpload, err error) {
uploadsMap = make(map[string][]*s3.MultipartUpload)
bucket, directory := f.split("")
if bucket != "" {
uploads, err := f.listMultipartUploads(ctx, bucket, directory)
if err != nil {
return uploadsMap, err
}
uploadsMap[bucket] = uploads
return uploadsMap, nil
}
entries, err := f.listBuckets(ctx)
if err != nil {
return uploadsMap, err
}
for _, entry := range entries {
bucket := entry.Remote()
uploads, listErr := f.listMultipartUploads(ctx, bucket, "")
if listErr != nil {
err = listErr
fs.Errorf(f, "%v", err)
}
uploadsMap[bucket] = uploads
}
return uploadsMap, err
}
// cleanUpBucket removes all pending multipart uploads for a given bucket over the age of maxAge
func (f *Fs) cleanUpBucket(ctx context.Context, bucket string, maxAge time.Duration, uploads []*s3.MultipartUpload) (err error) {
fs.Infof(f, "cleaning bucket %q of pending multipart uploads older than %v", bucket, maxAge)
for _, upload := range uploads {
if upload.Initiated != nil && upload.Key != nil && upload.UploadId != nil {
age := time.Since(*upload.Initiated)
what := fmt.Sprintf("pending multipart upload for bucket %q key %q dated %v (%v ago)", bucket, *upload.Key, upload.Initiated, age)
if age > maxAge {
fs.Infof(f, "removing %s", what)
if operations.SkipDestructive(ctx, what, "remove pending upload") {
continue
}
req := s3.AbortMultipartUploadInput{
Bucket: &bucket,
UploadId: upload.UploadId,
Key: upload.Key,
}
_, abortErr := f.c.AbortMultipartUpload(&req)
if abortErr != nil {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
err = fmt.Errorf("failed to remove %s: %w", what, abortErr)
s3: implement cleanup and backend command to list & remove multipart uploads This implements `rclone cleanup` to remove multipart uploads over 24 hours old. It also implements the backend command `list-multipart-uploads` to see which ones are available and `cleanup` to delete them with a configurable expiry interval. See #4302
2020-06-25 17:11:05 +02:00
fs.Errorf(f, "%v", err)
}
} else {
fs.Debugf(f, "ignoring %s", what)
}
}
}
return err
}
// CleanUp removes all pending multipart uploads
func (f *Fs) cleanUp(ctx context.Context, maxAge time.Duration) (err error) {
uploadsMap, err := f.listMultipartUploadsAll(ctx)
if err != nil {
return err
}
for bucket, uploads := range uploadsMap {
cleanErr := f.cleanUpBucket(ctx, bucket, maxAge, uploads)
if err != nil {
fs.Errorf(f, "Failed to cleanup bucket %q: %v", bucket, cleanErr)
err = cleanErr
}
}
return err
}
s3: implement backend versioning command to get/set bucket versioning
2022-07-25 17:05:17 +02:00
// Read whether the bucket is versioned or not
func (f *Fs) isVersioned(ctx context.Context) bool {
f.versioningMu.Lock()
defer f.versioningMu.Unlock()
if !f.versioning.Valid {
_, _ = f.setGetVersioning(ctx)
fs.Debugf(f, "bucket is versioned: %v", f.versioning.Value)
}
return f.versioning.Value
}
// Set or get bucket versioning.
//
// Pass no arguments to get, or pass "Enabled" or "Suspended"
//
// Updates f.versioning
func (f *Fs) setGetVersioning(ctx context.Context, arg ...string) (status string, err error) {
if len(arg) > 1 {
return "", errors.New("too many arguments")
}
if f.rootBucket == "" {
return "", errors.New("need a bucket")
}
if len(arg) == 1 {
var versioning = s3.VersioningConfiguration{
Status: aws.String(arg[0]),
}
// Disabled is indicated by the parameter missing
if *versioning.Status == "Disabled" {
versioning.Status = aws.String("")
}
req := s3.PutBucketVersioningInput{
Bucket: &f.rootBucket,
VersioningConfiguration: &versioning,
}
err := f.pacer.Call(func() (bool, error) {
_, err = f.c.PutBucketVersioningWithContext(ctx, &req)
return f.shouldRetry(ctx, err)
})
if err != nil {
return "", err
}
}
req := s3.GetBucketVersioningInput{
Bucket: &f.rootBucket,
}
var resp *s3.GetBucketVersioningOutput
err = f.pacer.Call(func() (bool, error) {
resp, err = f.c.GetBucketVersioningWithContext(ctx, &req)
return f.shouldRetry(ctx, err)
})
f.versioning.Valid = true
f.versioning.Value = false
if err != nil {
fs.Errorf(f, "Failed to read versioning status, assuming unversioned: %v", err)
return "", err
}
if resp.Status == nil {
return "Unversioned", err
}
f.versioning.Value = true
return *resp.Status, err
}
s3: implement cleanup and backend command to list & remove multipart uploads This implements `rclone cleanup` to remove multipart uploads over 24 hours old. It also implements the backend command `list-multipart-uploads` to see which ones are available and `cleanup` to delete them with a configurable expiry interval. See #4302
2020-06-25 17:11:05 +02:00
// CleanUp removes all pending multipart uploads older than 24 hours
func (f *Fs) CleanUp(ctx context.Context) (err error) {
return f.cleanUp(ctx, 24*time.Hour)
}
s3: implement Purge to purge versions and `backend cleanup-hidden`
2022-07-26 16:03:32 +02:00
// purge deletes all the files and directories
//
// if oldOnly is true then it deletes only non current files.
//
// Implemented here so we can make sure we delete old versions.
func (f *Fs) purge(ctx context.Context, dir string, oldOnly bool) error {
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
if f.opt.VersionAt.IsSet() {
return errNotWithVersionAt
}
s3: implement Purge to purge versions and `backend cleanup-hidden`
2022-07-26 16:03:32 +02:00
bucket, directory := f.split(dir)
if bucket == "" {
return errors.New("can't purge from root")
}
versioned := f.isVersioned(ctx)
if !versioned && oldOnly {
fs.Infof(f, "bucket is not versioned so not removing old versions")
return nil
}
var errReturn error
var checkErrMutex sync.Mutex
var checkErr = func(err error) {
if err == nil {
return
}
checkErrMutex.Lock()
defer checkErrMutex.Unlock()
if errReturn == nil {
errReturn = err
}
}
// Delete Config.Transfers in parallel
delChan := make(fs.ObjectsChan, f.ci.Transfers)
delErr := make(chan error, 1)
go func() {
delErr <- operations.DeleteFiles(ctx, delChan)
}()
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
checkErr(f.list(ctx, listOpt{
s3: make purge remove directory markers too See: https://forum.rclone.org/t/cannot-purge-aws-s3/36169/
2023-02-27 16:32:59 +01:00
bucket: bucket,
directory: directory,
prefix: f.rootDirectory,
addBucket: f.rootBucket == "",
recurse: true,
withVersions: versioned,
hidden: true,
noSkipMarkers: true,
s3: refactor f.list() to take an options struct as it had too many parameters
2022-07-27 11:46:28 +02:00
}, func(remote string, object *s3.Object, versionID *string, isDirectory bool) error {
s3: implement Purge to purge versions and `backend cleanup-hidden`
2022-07-26 16:03:32 +02:00
if isDirectory {
return nil
}
s3: fix purging of root directory with --s3-directory-markers - fixes #7247
2023-08-25 18:39:16 +02:00
// If the root is a dirmarker it will have lost its trailing /
if remote == "" {
remote = "/"
}
s3: implement Purge to purge versions and `backend cleanup-hidden`
2022-07-26 16:03:32 +02:00
oi, err := f.newObjectWithInfo(ctx, remote, object, versionID)
if err != nil {
fs.Errorf(object, "Can't create object %+v", err)
return nil
}
accounting: Make checkers show what they are doing Before this change, all types of checkers showed "checking" after the file name despite the fact that not all of them were checking. After this change, they can show - checking - deleting - hashing - importing - listing - merging - moving - renaming See: https://forum.rclone.org/t/what-is-rclone-checking-during-a-purge/35931/
2023-02-06 11:30:22 +01:00
tr := accounting.Stats(ctx).NewCheckingTransfer(oi, "checking")
s3: implement Purge to purge versions and `backend cleanup-hidden`
2022-07-26 16:03:32 +02:00
// Work out whether the file is the current version or not
isCurrentVersion := !versioned || !version.Match(remote)
fs.Debugf(nil, "%q version %v", remote, version.Match(remote))
if oldOnly && isCurrentVersion {
// Check current version of the file
if object.Size == isDeleteMarker {
fs.Debugf(remote, "Deleting current version (id %q) as it is a delete marker", aws.StringValue(versionID))
delChan <- oi
} else {
fs.Debugf(remote, "Not deleting current version %q", aws.StringValue(versionID))
}
} else {
if object.Size == isDeleteMarker {
fs.Debugf(remote, "Deleting delete marker (id %q)", aws.StringValue(versionID))
} else {
fs.Debugf(remote, "Deleting (id %q)", aws.StringValue(versionID))
}
delChan <- oi
}
tr.Done(ctx, nil)
return nil
}))
close(delChan)
checkErr(<-delErr)
if !oldOnly {
checkErr(f.Rmdir(ctx, dir))
}
return errReturn
}
// Purge deletes all the files and directories including the old versions.
func (f *Fs) Purge(ctx context.Context, dir string) error {
return f.purge(ctx, dir, false)
}
// CleanUpHidden deletes all the hidden files.
func (f *Fs) CleanUpHidden(ctx context.Context) error {
return f.purge(ctx, "", true)
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
// ------------------------------------------------------------
Fix golint warnings
2015-09-22 19:47:16 +02:00
// Fs returns the parent Fs
Create separate interface for object information. Take out read-only information about a Fs in a separate struct to limit access. See discussion at #282.
2016-02-18 12:35:25 +01:00
func (o *Object) Fs() fs.Info {
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
return o.fs
Factor the generic code into fs and add some more intefaces
2014-03-28 18:56:04 +01:00
}
// Return a string version
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
func (o *Object) String() string {
Factor the generic code into fs and add some more intefaces
2014-03-28 18:56:04 +01:00
if o == nil {
return "<nil>"
}
return o.remote
}
Fix golint warnings
2015-09-22 19:47:16 +02:00
// Remote returns the remote path
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
func (o *Object) Remote() string {
Refactor into sub filesystems
2013-06-27 21:13:07 +02:00
return o.remote
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: ignore etags from multipart uploads which aren't md5sums - fixes #56
2015-05-09 11:37:43 +02:00
var matchMd5 = regexp.MustCompile(`^[0-9a-f]{32}$`)
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827
2020-11-23 12:23:06 +01:00
// Set the MD5 from the etag
func (o *Object) setMD5FromEtag(etag string) {
if o.fs.etagIsNotMD5 {
o.md5 = ""
return
}
if etag == "" {
o.md5 = ""
return
}
hash := strings.Trim(strings.ToLower(etag), `"`)
// Check the etag is a valid md5sum
if !matchMd5.MatchString(hash) {
o.md5 = ""
return
}
o.md5 = hash
}
Add support for multiple hash types. Add support for multiple hash types with negotiation of common hash types for comparison. Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
// Hash returns the Md5sum of an object returning a lowercase hex string
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
config/hash: rename more symbols after factoring into own package
2018-01-18 21:27:52 +01:00
if t != hash.MD5 {
return "", hash.ErrUnsupported
Add support for multiple hash types. Add support for multiple hash types with negotiation of common hash types for comparison. Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
}
s3: add --s3-decompress flag to download gzip-encoded files Before this change, if an object compressed with "Content-Encoding: gzip" was downloaded, a length and hash mismatch would occur since the go runtime automatically decompressed the object on download. If --s3-decompress is set, this change erases the length and hash on compressed objects so they can be downloaded successfully, at the cost of not being able to check the length or the hash of the downloaded object. If --s3-decompress is not set the compressed files will be downloaded as-is providing compressed objects with intact size and hash information. See #2658
2022-07-29 18:01:59 +02:00
// If decompressing, erase the hash
if o.bytes < 0 {
return "", nil
}
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827
2020-11-23 12:23:06 +01:00
// If we haven't got an MD5, then check the metadata
if o.md5 == "" {
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
err := o.readMetaData(ctx)
s3: set/get the hash for multipart files - #523
2018-01-06 15:30:10 +01:00
if err != nil {
return "", err
}
s3: ignore etags from multipart uploads which aren't md5sums - fixes #56
2015-05-09 11:37:43 +02:00
}
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827
2020-11-23 12:23:06 +01:00
return o.md5, nil
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
// Size returns the size of an object in bytes
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
func (o *Object) Size() int64 {
Refactor into sub filesystems
2013-06-27 21:13:07 +02:00
return o.bytes
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
func (o *Object) headObject(ctx context.Context) (resp *s3.HeadObjectOutput, err error) {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
bucket, bucketPath := o.split()
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
req := s3.HeadObjectInput{
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
Bucket: &bucket,
Key: &bucketPath,
VersionId: o.versionID,
s3: make reading metadata more reliable to work around eventual consistency problems
2014-07-28 23:32:15 +02:00
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
return o.fs.headObject(ctx, &req)
}
func (f *Fs) headObject(ctx context.Context, req *s3.HeadObjectInput) (resp *s3.HeadObjectOutput, err error) {
if f.opt.RequesterPays {
s3: add requester pays option - fixes #301
2020-12-03 03:30:06 +01:00
req.RequestPayer = aws.String(s3.RequestPayerRequester)
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
if f.opt.SSECustomerAlgorithm != "" {
req.SSECustomerAlgorithm = &f.opt.SSECustomerAlgorithm
s3: complete sse-c implementation This now can complete all operations with SSE-C enabled. Fixes #2827 See: https://forum.rclone.org/t/issues-with-aws-s3-sse-c-getting-strange-log-entries-and-errors/20553
2020-11-20 12:15:48 +01:00
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
if f.opt.SSECustomerKey != "" {
req.SSECustomerKey = &f.opt.SSECustomerKey
s3: complete sse-c implementation This now can complete all operations with SSE-C enabled. Fixes #2827 See: https://forum.rclone.org/t/issues-with-aws-s3-sse-c-getting-strange-log-entries-and-errors/20553
2020-11-20 12:15:48 +01:00
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
if f.opt.SSECustomerKeyMD5 != "" {
req.SSECustomerKeyMD5 = &f.opt.SSECustomerKeyMD5
s3: complete sse-c implementation This now can complete all operations with SSE-C enabled. Fixes #2827 See: https://forum.rclone.org/t/issues-with-aws-s3-sse-c-getting-strange-log-entries-and-errors/20553
2020-11-20 12:15:48 +01:00
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
err = f.pacer.Call(func() (bool, error) {
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
var err error
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
resp, err = f.c.HeadObjectWithContext(ctx, req)
return f.shouldRetry(ctx, err)
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
})
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
if err != nil {
Make NewObject return an error * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error
2016-06-25 22:23:20 +02:00
if awsErr, ok := err.(awserr.RequestFailure); ok {
if awsErr.StatusCode() == http.StatusNotFound {
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
return nil, fs.ErrorObjectNotFound
Make NewObject return an error * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error
2016-06-25 22:23:20 +02:00
}
}
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
return nil, err
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
if req.Bucket != nil {
f.cache.MarkOK(*req.Bucket)
}
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
return resp, nil
}
// readMetaData gets the metadata if it hasn't already been fetched
//
// it also sets the info
func (o *Object) readMetaData(ctx context.Context) (err error) {
if o.meta != nil {
return nil
}
resp, err := o.headObject(ctx)
if err != nil {
return err
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
o.setMetaData(resp)
// resp.ETag, resp.ContentLength, resp.LastModified, resp.Metadata, resp.ContentType, resp.StorageClass)
s3: add --s3-no-head-object This stops rclone doing any HEAD requests on objects.
2021-04-28 12:05:54 +02:00
return nil
}
s3: change metadata storage to normal map with lowercase keys
2022-06-22 16:40:30 +02:00
// Convert S3 metadata with pointers into a map[string]string
// while lowercasing the keys
func s3MetadataToMap(s3Meta map[string]*string) map[string]string {
meta := make(map[string]string, len(s3Meta))
for k, v := range s3Meta {
if v != nil {
meta[strings.ToLower(k)] = *v
}
}
return meta
}
// Convert our metadata back into S3 metadata
func mapToS3Metadata(meta map[string]string) map[string]*string {
s3Meta := make(map[string]*string, len(meta))
for k, v := range meta {
s3Meta[k] = aws.String(v)
}
return s3Meta
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
func (o *Object) setMetaData(resp *s3.HeadObjectOutput) {
s3: Treat missing Content-Length as 0 for some ceph installations
2014-05-16 17:27:53 +02:00
// Ignore missing Content-Length assuming it is 0
// Some versions of ceph do this due their apache proxies
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
if resp.ContentLength != nil {
o.bytes = *resp.ContentLength
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
o.setMD5FromEtag(aws.StringValue(resp.ETag))
o.meta = s3MetadataToMap(resp.Metadata)
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827
2020-11-23 12:23:06 +01:00
// Read MD5 from metadata if present
if md5sumBase64, ok := o.meta[metaMD5Hash]; ok {
s3: change metadata storage to normal map with lowercase keys
2022-06-22 16:40:30 +02:00
md5sumBytes, err := base64.StdEncoding.DecodeString(md5sumBase64)
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827
2020-11-23 12:23:06 +01:00
if err != nil {
s3: change metadata storage to normal map with lowercase keys
2022-06-22 16:40:30 +02:00
fs.Debugf(o, "Failed to read md5sum from metadata %q: %v", md5sumBase64, err)
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827
2020-11-23 12:23:06 +01:00
} else if len(md5sumBytes) != 16 {
s3: change metadata storage to normal map with lowercase keys
2022-06-22 16:40:30 +02:00
fs.Debugf(o, "Failed to read md5sum from metadata %q: wrong length", md5sumBase64)
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827
2020-11-23 12:23:06 +01:00
} else {
o.md5 = hex.EncodeToString(md5sumBytes)
}
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
if resp.LastModified == nil {
Refactor into sub filesystems
2013-06-27 21:13:07 +02:00
o.lastModified = time.Now()
s3: Use the ETag on multipart transfers to verify the transfer was OK Before this rclone ignored the ETag on multipart uploads which missed an opportunity for a whole file integrity check. This adds that check which means that we now check even harder that multipart uploads have arrived properly. See #5993
2022-02-16 18:50:11 +01:00
fs.Logf(o, "Failed to read last modified")
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
} else {
s3: try to keep the maximum precision in ModTime with --user-server-modtime Before this change if --user-server-modtime was in use the ModTime could change for an object as we receive it accurate to the nearest ms in listings, but only accurate to the nearest second in HEAD and GET requests. Normally AWS returns the milliseconds as .000 in listings, but if versions are in use it may not. Storj S3 also seems to return milliseconds. This patch tries to keep the maximum precision in the last modified time, so it doesn't update a last modified time with a truncated version if the times were the same to the nearest second. See: https://forum.rclone.org/t/cache-fingerprint-miss-behavior-leading-to-false-positive-stalen-cache/33404/
2022-10-08 12:27:17 +02:00
// Try to keep the maximum precision in lastModified. If we read
// it from listings then it may have millisecond precision, but
// if we read it from a HEAD/GET request then it will have
// second precision.
equalToWithinOneSecond := o.lastModified.Truncate(time.Second).Equal((*resp.LastModified).Truncate(time.Second))
newHasNs := (*resp.LastModified).Nanosecond() != 0
if !equalToWithinOneSecond || newHasNs {
o.lastModified = *resp.LastModified
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
o.mimeType = aws.StringValue(resp.ContentType)
// Set system metadata
o.storageClass = resp.StorageClass
o.cacheControl = resp.CacheControl
o.contentDisposition = resp.ContentDisposition
o.contentEncoding = resp.ContentEncoding
o.contentLanguage = resp.ContentLanguage
s3: add --s3-decompress flag to download gzip-encoded files Before this change, if an object compressed with "Content-Encoding: gzip" was downloaded, a length and hash mismatch would occur since the go runtime automatically decompressed the object on download. If --s3-decompress is set, this change erases the length and hash on compressed objects so they can be downloaded successfully, at the cost of not being able to check the length or the hash of the downloaded object. If --s3-decompress is not set the compressed files will be downloaded as-is providing compressed objects with intact size and hash information. See #2658
2022-07-29 18:01:59 +02:00
// If decompressing then size and md5sum are unknown
if o.fs.opt.Decompress && aws.StringValue(o.contentEncoding) == "gzip" {
o.bytes = -1
o.md5 = ""
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
// ModTime returns the modification time of the object
//
// It attempts to read the objects mtime and if that isn't present the
// LastModified returned in the http headers
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (o *Object) ModTime(ctx context.Context) time.Time {
fs: deglobalise the config #4685 This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.
2020-11-05 12:33:32 +01:00
if o.fs.ci.UseServerModTime {
s3,swift: Add --use-server-modtime `--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.
2018-04-13 14:32:17 +02:00
return o.lastModified
}
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
err := o.readMetaData(ctx)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
if err != nil {
Rename log functions and factor into own file
2017-02-09 12:01:20 +01:00
fs.Logf(o, "Failed to read metadata: %v", err)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
return time.Now()
}
// read mtime out of metadata if available
Refactor into sub filesystems
2013-06-27 21:13:07 +02:00
d, ok := o.meta[metaMtime]
s3: change metadata storage to normal map with lowercase keys
2022-06-22 16:40:30 +02:00
if !ok {
Rename log functions and factor into own file
2017-02-09 12:01:20 +01:00
// fs.Debugf(o, "No metadata")
Refactor into sub filesystems
2013-06-27 21:13:07 +02:00
return o.lastModified
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: change metadata storage to normal map with lowercase keys
2022-06-22 16:40:30 +02:00
modTime, err := swift.FloatStringToTime(d)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
if err != nil {
Rename log functions and factor into own file
2017-02-09 12:01:20 +01:00
fs.Logf(o, "Failed to read mtime from object: %v", err)
Refactor into sub filesystems
2013-06-27 21:13:07 +02:00
return o.lastModified
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
return modTime
}
Fix golint warnings
2015-09-22 19:47:16 +02:00
// SetModTime sets the modification time of the local fs object
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
err := o.readMetaData(ctx)
S3: Fix mkdir not to error if bucket exists and implement SetModTime
2013-01-08 23:31:16 +01:00
if err != nil {
Change the interface of SetModTime to return an error - #348
2016-03-22 16:07:10 +01:00
return err
S3: Fix mkdir not to error if bucket exists and implement SetModTime
2013-01-08 23:31:16 +01:00
}
s3: change metadata storage to normal map with lowercase keys
2022-06-22 16:40:30 +02:00
o.meta[metaMtime] = swift.TimeToFloatString(modTime)
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
// Can't update metadata here, so return this error to force a recopy
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
if o.storageClass != nil && (*o.storageClass == "GLACIER" || *o.storageClass == "DEEP_ARCHIVE") {
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
return fs.ErrorCantSetModTime
}
s3: Fix corrupting Content-Type on mod time update This fixes an issue where updating the modification time resets the content-type to the S3 default of binary/octet-stream which breaks static websites that expect an html file to have a content-type of text/html.
2016-01-02 09:58:48 +01:00
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
// Copy the object to itself to update the metadata
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
bucket, bucketPath := o.split()
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
req := s3.CopyObjectInput{
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
ContentType: aws.String(fs.MimeType(ctx, o)), // Guess the content type
s3: change metadata storage to normal map with lowercase keys
2022-06-22 16:40:30 +02:00
Metadata: mapToS3Metadata(o.meta),
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
MetadataDirective: aws.String(s3.MetadataDirectiveReplace), // replace metadata with that passed in
s3: make SetModTime work for GLACIER while syncing - Fixes #3224 Before this change rclone would fail with Failed to set modification time: InvalidObjectState: Operation is not valid for the source object's storage class when attempting to set the modification time of an object in GLACIER. After this change rclone will re-upload the object as part of a sync if it needs to change the modification time. See: https://forum.rclone.org/t/suspected-bug-in-s3-or-compatible-sync-logic-to-glacier/10187
2019-06-03 16:28:19 +02:00
}
s3: add requester pays option - fixes #301
2020-12-03 03:30:06 +01:00
if o.fs.opt.RequesterPays {
req.RequestPayer = aws.String(s3.RequestPayerRequester)
}
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
return o.fs.copy(ctx, &req, bucket, bucketPath, bucket, bucketPath, o)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
Fix golint warnings
2015-09-22 19:47:16 +02:00
// Storable raturns a boolean indicating if this object is storable
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
func (o *Object) Storable() bool {
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
return true
}
s3: add support to use CDN URL to download the file The egress charges while using a CloudFront CDN url is cheaper when compared to accessing the file directly from S3. So added a download URL advanced option, which when set downloads the file using it.
2021-10-14 12:19:38 +02:00
func (o *Object) downloadFromURL(ctx context.Context, bucketPath string, options ...fs.OpenOption) (in io.ReadCloser, err error) {
url := o.fs.opt.DownloadURL + bucketPath
var resp *http.Response
opts := rest.Opts{
Method: "GET",
RootURL: url,
Options: options,
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srvRest.Call(ctx, &opts)
return o.fs.shouldRetry(ctx, err)
})
if err != nil {
return nil, err
}
backend/http: support content-range response header
2022-03-20 11:12:48 +01:00
contentLength := rest.ParseSizeFromHeaders(resp.Header)
if contentLength < 0 {
fs.Debugf(o, "Failed to parse file size from headers")
s3: add support to use CDN URL to download the file The egress charges while using a CloudFront CDN url is cheaper when compared to accessing the file directly from S3. So added a download URL advanced option, which when set downloads the file using it.
2021-10-14 12:19:38 +02:00
}
backend/http: support content-range response header
2022-03-20 11:12:48 +01:00
lastModified, err := http.ParseTime(resp.Header.Get("Last-Modified"))
s3: add support to use CDN URL to download the file The egress charges while using a CloudFront CDN url is cheaper when compared to accessing the file directly from S3. So added a download URL advanced option, which when set downloads the file using it.
2021-10-14 12:19:38 +02:00
if err != nil {
fs.Debugf(o, "Failed to parse last modified from string %s, %v", resp.Header.Get("Last-Modified"), err)
}
metaData := make(map[string]*string)
for key, value := range resp.Header {
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
key = strings.ToLower(key)
if strings.HasPrefix(key, "x-amz-meta-") {
s3: add support to use CDN URL to download the file The egress charges while using a CloudFront CDN url is cheaper when compared to accessing the file directly from S3. So added a download URL advanced option, which when set downloads the file using it.
2021-10-14 12:19:38 +02:00
metaKey := strings.TrimPrefix(key, "x-amz-meta-")
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
metaData[metaKey] = &value[0]
s3: add support to use CDN URL to download the file The egress charges while using a CloudFront CDN url is cheaper when compared to accessing the file directly from S3. So added a download URL advanced option, which when set downloads the file using it.
2021-10-14 12:19:38 +02:00
}
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
header := func(k string) *string {
v := resp.Header.Get(k)
if v == "" {
return nil
}
return &v
}
var head = s3.HeadObjectOutput{
ETag: header("Etag"),
backend/http: support content-range response header
2022-03-20 11:12:48 +01:00
ContentLength: &contentLength,
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
LastModified: &lastModified,
Metadata: metaData,
CacheControl: header("Cache-Control"),
ContentDisposition: header("Content-Disposition"),
ContentEncoding: header("Content-Encoding"),
ContentLanguage: header("Content-Language"),
ContentType: header("Content-Type"),
StorageClass: header("X-Amz-Storage-Class"),
}
o.setMetaData(&head)
s3: add support to use CDN URL to download the file The egress charges while using a CloudFront CDN url is cheaper when compared to accessing the file directly from S3. So added a download URL advanced option, which when set downloads the file using it.
2021-10-14 12:19:38 +02:00
return resp.Body, err
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
// Open an object for read
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.ReadCloser, err error) {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
bucket, bucketPath := o.split()
s3: add support to use CDN URL to download the file The egress charges while using a CloudFront CDN url is cheaper when compared to accessing the file directly from S3. So added a download URL advanced option, which when set downloads the file using it.
2021-10-14 12:19:38 +02:00
if o.fs.opt.DownloadURL != "" {
return o.downloadFromURL(ctx, bucketPath, options...)
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
req := s3.GetObjectInput{
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
Bucket: &bucket,
Key: &bucketPath,
VersionId: o.versionID,
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
}
s3: add requester pays option - fixes #301
2020-12-03 03:30:06 +01:00
if o.fs.opt.RequesterPays {
req.RequestPayer = aws.String(s3.RequestPayerRequester)
}
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
if o.fs.opt.SSECustomerAlgorithm != "" {
req.SSECustomerAlgorithm = &o.fs.opt.SSECustomerAlgorithm
}
if o.fs.opt.SSECustomerKey != "" {
req.SSECustomerKey = &o.fs.opt.SSECustomerKey
}
if o.fs.opt.SSECustomerKeyMD5 != "" {
req.SSECustomerKeyMD5 = &o.fs.opt.SSECustomerKeyMD5
}
s3: add support for HTTPOption
2020-02-10 10:02:04 +01:00
httpReq, resp := o.fs.c.GetObjectRequest(&req)
azureblob/b2/dropbox/gcs/koofr/qingstor/s3: fix 0 length files In 0386d22cc946dfc5 we introduced a test for 0 length files read the way mount does. This test failed on these backends which we fix up here.
2019-08-06 16:18:08 +02:00
fs.FixRangeOption(options, o.bytes)
s3: add --s3-decompress flag to download gzip-encoded files Before this change, if an object compressed with "Content-Encoding: gzip" was downloaded, a length and hash mismatch would occur since the go runtime automatically decompressed the object on download. If --s3-decompress is set, this change erases the length and hash on compressed objects so they can be downloaded successfully, at the cost of not being able to check the length or the hash of the downloaded object. If --s3-decompress is not set the compressed files will be downloaded as-is providing compressed objects with intact size and hash information. See #2658
2022-07-29 18:01:59 +02:00
// Override the automatic decompression in the transport to
// download compressed files as-is
s3: add GCS to provider list
2023-01-09 07:54:51 +01:00
if o.fs.opt.UseAcceptEncodingGzip.Value {
httpReq.HTTPRequest.Header.Set("Accept-Encoding", "gzip")
}
s3: add --s3-decompress flag to download gzip-encoded files Before this change, if an object compressed with "Content-Encoding: gzip" was downloaded, a length and hash mismatch would occur since the go runtime automatically decompressed the object on download. If --s3-decompress is set, this change erases the length and hash on compressed objects so they can be downloaded successfully, at the cost of not being able to check the length or the hash of the downloaded object. If --s3-decompress is not set the compressed files will be downloaded as-is providing compressed objects with intact size and hash information. See #2658
2022-07-29 18:01:59 +02:00
Add options for Open and implement Range for all remotes
2016-09-10 12:29:57 +02:00
for _, option := range options {
switch option.(type) {
case *fs.RangeOption, *fs.SeekOption:
_, value := option.Header()
req.Range = &value
s3: add support for HTTPOption
2020-02-10 10:02:04 +01:00
case *fs.HTTPOption:
key, value := option.Header()
httpReq.HTTPRequest.Header.Add(key, value)
Add options for Open and implement Range for all remotes
2016-09-10 12:29:57 +02:00
default:
if option.Mandatory() {
Rename log functions and factor into own file
2017-02-09 12:01:20 +01:00
fs.Logf(o, "Unsupported mandatory option: %v", option)
Add options for Open and implement Range for all remotes
2016-09-10 12:29:57 +02:00
}
}
}
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
err = o.fs.pacer.Call(func() (bool, error) {
var err error
s3: add support for HTTPOption
2020-02-10 10:02:04 +01:00
httpReq.HTTPRequest = httpReq.HTTPRequest.WithContext(ctx)
err = httpReq.Send()
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return o.fs.shouldRetry(ctx, err)
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
})
s3: Error message for objects in glacier
2017-09-09 14:02:26 +02:00
if err, ok := err.(awserr.RequestFailure); ok {
if err.Code() == "InvalidObjectState" {
Remove github.com/pkg/errors and replace with std library version This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.
2021-11-04 11:12:57 +01:00
return nil, fmt.Errorf("Object in GLACIER, restore first: bucket=%q, key=%q", bucket, bucketPath)
s3: Error message for objects in glacier
2017-09-09 14:02:26 +02:00
}
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
if err != nil {
return nil, err
}
s3: Use the ETag on multipart transfers to verify the transfer was OK Before this rclone ignored the ETag on multipart uploads which missed an opportunity for a whole file integrity check. This adds that check which means that we now check even harder that multipart uploads have arrived properly. See #5993
2022-02-16 18:50:11 +01:00
s3: add --s3-no-head-object This stops rclone doing any HEAD requests on objects.
2021-04-28 12:05:54 +02:00
// read size from ContentLength or ContentRange
size := resp.ContentLength
if resp.ContentRange != nil {
var contentRange = *resp.ContentRange
slash := strings.IndexRune(contentRange, '/')
if slash >= 0 {
i, err := strconv.ParseInt(contentRange[slash+1:], 10, 64)
if err == nil {
size = &i
} else {
fs.Debugf(o, "Failed to find parse integer from in %q: %v", contentRange, err)
}
} else {
fs.Debugf(o, "Failed to find length in %q", contentRange)
}
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
var head s3.HeadObjectOutput
s3: refactor to use generated code instead of reflection to copy structs
2022-07-28 12:49:19 +02:00
//structs.SetFrom(&head, resp)
setFrom_s3HeadObjectOutput_s3GetObjectOutput(&head, resp)
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
head.ContentLength = size
o.setMetaData(&head)
s3: add --s3-decompress flag to download gzip-encoded files Before this change, if an object compressed with "Content-Encoding: gzip" was downloaded, a length and hash mismatch would occur since the go runtime automatically decompressed the object on download. If --s3-decompress is set, this change erases the length and hash on compressed objects so they can be downloaded successfully, at the cost of not being able to check the length or the hash of the downloaded object. If --s3-decompress is not set the compressed files will be downloaded as-is providing compressed objects with intact size and hash information. See #2658
2022-07-29 18:01:59 +02:00
// Decompress body if necessary
if aws.StringValue(resp.ContentEncoding) == "gzip" {
s3: add provider quirk --s3-might-gzip to fix corrupted on transfer: sizes differ Before this change, some files were giving this error when downloaded from Cloudflare and other providers. ERROR corrupted on transfer: sizes differ NNN vs MMM This is because these providers auto gzips the object when rclone wasn't expecting it to. (AWS does not gzip objects without their being uploaded gzipped). This patch adds a quirk to for fix the problem and a flag to control it. The quirk `might_gzip` is set to `true` for all providers except AWS. See: https://forum.rclone.org/t/s3-error-corrupted-on-transfer-sizes-differ-nnn-vs-mmm/33694/ Fixes: #6533
2022-10-29 00:23:29 +02:00
if o.fs.opt.Decompress || (resp.ContentLength == nil && o.fs.opt.MightGzip.Value) {
s3: add --s3-decompress flag to download gzip-encoded files Before this change, if an object compressed with "Content-Encoding: gzip" was downloaded, a length and hash mismatch would occur since the go runtime automatically decompressed the object on download. If --s3-decompress is set, this change erases the length and hash on compressed objects so they can be downloaded successfully, at the cost of not being able to check the length or the hash of the downloaded object. If --s3-decompress is not set the compressed files will be downloaded as-is providing compressed objects with intact size and hash information. See #2658
2022-07-29 18:01:59 +02:00
return readers.NewGzipReader(resp.Body)
}
o.fs.warnCompressed.Do(func() {
fs.Logf(o, "Not decompressing 'Content-Encoding: gzip' compressed file. Use --s3-decompress to override")
})
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
return resp.Body, nil
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
s3: Reduce memory usage streaming files by reducing max stream upload size Before this change rclone would allow the user to stream (eg with rclone mount, rclone rcat or uploading google photos or docs) 5TB files. This meant that rclone allocated 4 * 525 MB buffers per transfer which is way too much memory by default. This change makes rclone use the configured chunk size for streamed uploads. This is 5MB by default which means that rclone can stream upload files up to 48GB by default staying below the 10,000 chunks limit. This can be increased with --s3-chunk-size if necessary. If rclone detects that a file is being streamed to s3 it will make a single NOTICE level log stating the limitation. This fixes the enormous memory usage. Fixes #3568 See: https://forum.rclone.org/t/how-much-memory-does-rclone-need/12743
2019-11-06 11:41:03 +01:00
var warnStreamUpload sync.Once
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
// state of ChunkWriter
type s3ChunkWriter struct {
chunkSize int64
size int64
f *Fs
bucket *string
key *string
build: fix lint errors when re-enabling revive var-naming
2023-08-29 13:57:33 +02:00
uploadID *string
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
multiPartUploadInput *s3.CreateMultipartUploadInput
completedPartsMu sync.Mutex
completedParts []*s3.CompletedPart
eTag string
versionID string
md5sMu sync.Mutex
md5s []byte
ui uploadInfo
o *Object
}
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
// OpenChunkWriter returns the chunk size and a ChunkWriter
//
// Pass in the remote and the src object
// You can also use options to hint at the desired chunk size
backends: change OpenChunkWriter interface to allow backend concurrency override Before this change the concurrency used for an upload was rather inconsistent. - if size below `--backend-upload-cutoff` (default 200M) do single part upload. - if size below `--multi-thread-cutoff` (default 256M) or using streaming uploads (eg `rclone rcat) do multipart upload using `--backend-upload-concurrency` to set the concurrency used by the uploader. - otherwise do multipart upload using `--multi-thread-streams` to set the concurrency. This change makes the default for the concurrency used be the `--backend-upload-concurrency`. If `--multi-thread-streams` is set and larger than the `--backend-upload-concurrency` then that will be used instead. This means that if the user sets `--backend-upload-concurrency` then it will be obeyed for all multipart/multi-thread transfers and the user can override them all with `--multi-thread-streams`. See: #7056
2023-09-01 18:25:15 +02:00
func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectInfo, options ...fs.OpenOption) (info fs.ChunkWriterInfo, writer fs.ChunkWriter, err error) {
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
// Temporary Object under construction
o := &Object{
fs: f,
remote: remote,
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
}
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui, err := o.prepareUpload(ctx, src, options)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
if err != nil {
backends: change OpenChunkWriter interface to allow backend concurrency override Before this change the concurrency used for an upload was rather inconsistent. - if size below `--backend-upload-cutoff` (default 200M) do single part upload. - if size below `--multi-thread-cutoff` (default 256M) or using streaming uploads (eg `rclone rcat) do multipart upload using `--backend-upload-concurrency` to set the concurrency used by the uploader. - otherwise do multipart upload using `--multi-thread-streams` to set the concurrency. This change makes the default for the concurrency used be the `--backend-upload-concurrency`. If `--multi-thread-streams` is set and larger than the `--backend-upload-concurrency` then that will be used instead. This means that if the user sets `--backend-upload-concurrency` then it will be obeyed for all multipart/multi-thread transfers and the user can override them all with `--multi-thread-streams`. See: #7056
2023-09-01 18:25:15 +02:00
return info, nil, fmt.Errorf("failed to prepare upload: %w", err)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
}
//structs.SetFrom(&mReq, req)
var mReq s3.CreateMultipartUploadInput
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
setFrom_s3CreateMultipartUploadInput_s3PutObjectInput(&mReq, ui.req)
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
s3: add `max_upload_parts` support - fixes #4159 * s3: add `max_upload_parts` support This allows to configure a maximum amount of chunks used to upload file: - Support Scaleway which has a limit of 1k chunks currently - Reduce a cost on S3 when each request costs some money at the expense of memory used Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2020-06-08 19:22:34 +02:00
uploadParts := f.opt.MaxUploadParts
if uploadParts < 1 {
uploadParts = 1
} else if uploadParts > maxUploadParts {
uploadParts = maxUploadParts
}
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
size := src.Size()
s3: add `max_upload_parts` support - fixes #4159 * s3: add `max_upload_parts` support This allows to configure a maximum amount of chunks used to upload file: - Support Scaleway which has a limit of 1k chunks currently - Reduce a cost on S3 when each request costs some money at the expense of memory used Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2020-06-08 19:22:34 +02:00
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
// calculate size of parts
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
chunkSize := f.opt.ChunkSize
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
Use binary prefixes for size and rate units Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M. Extends binary suffix output with letter i, e.g. Ki and Mi. Centralizes creation of bit/byte unit strings.
2021-03-02 20:11:57 +01:00
// size can be -1 here meaning we don't know the size of the incoming file. We use ChunkSize
// buffers here (default 5 MiB). With a maximum number of parts (10,000) this will be a file of
// 48 GiB which seems like a not too unreasonable limit.
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
if size == -1 {
warnStreamUpload.Do(func() {
fs.Logf(f, "Streaming uploads using chunk size %v will have maximum file size of %v",
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
f.opt.ChunkSize, fs.SizeSuffix(int64(chunkSize)*int64(uploadParts)))
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
})
} else {
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
chunkSize = chunksize.Calculator(src, size, uploadParts, chunkSize)
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
}
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
var mOut *s3.CreateMultipartUploadOutput
err = f.pacer.Call(func() (bool, error) {
mOut, err = f.c.CreateMultipartUploadWithContext(ctx, &mReq)
return f.shouldRetry(ctx, err)
})
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
if err != nil {
backends: change OpenChunkWriter interface to allow backend concurrency override Before this change the concurrency used for an upload was rather inconsistent. - if size below `--backend-upload-cutoff` (default 200M) do single part upload. - if size below `--multi-thread-cutoff` (default 256M) or using streaming uploads (eg `rclone rcat) do multipart upload using `--backend-upload-concurrency` to set the concurrency used by the uploader. - otherwise do multipart upload using `--multi-thread-streams` to set the concurrency. This change makes the default for the concurrency used be the `--backend-upload-concurrency`. If `--multi-thread-streams` is set and larger than the `--backend-upload-concurrency` then that will be used instead. This means that if the user sets `--backend-upload-concurrency` then it will be obeyed for all multipart/multi-thread transfers and the user can override them all with `--multi-thread-streams`. See: #7056
2023-09-01 18:25:15 +02:00
return info, nil, fmt.Errorf("create multipart upload failed: %w", err)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
}
chunkWriter := &s3ChunkWriter{
chunkSize: int64(chunkSize),
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
size: size,
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
f: f,
bucket: mOut.Bucket,
key: mOut.Key,
build: fix lint errors when re-enabling revive var-naming
2023-08-29 13:57:33 +02:00
uploadID: mOut.UploadId,
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
multiPartUploadInput: &mReq,
completedParts: make([]*s3.CompletedPart, 0),
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui: ui,
o: o,
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
}
backends: change OpenChunkWriter interface to allow backend concurrency override Before this change the concurrency used for an upload was rather inconsistent. - if size below `--backend-upload-cutoff` (default 200M) do single part upload. - if size below `--multi-thread-cutoff` (default 256M) or using streaming uploads (eg `rclone rcat) do multipart upload using `--backend-upload-concurrency` to set the concurrency used by the uploader. - otherwise do multipart upload using `--multi-thread-streams` to set the concurrency. This change makes the default for the concurrency used be the `--backend-upload-concurrency`. If `--multi-thread-streams` is set and larger than the `--backend-upload-concurrency` then that will be used instead. This means that if the user sets `--backend-upload-concurrency` then it will be obeyed for all multipart/multi-thread transfers and the user can override them all with `--multi-thread-streams`. See: #7056
2023-09-01 18:25:15 +02:00
info = fs.ChunkWriterInfo{
ChunkSize: int64(chunkSize),
Concurrency: o.fs.opt.UploadConcurrency,
LeavePartsOnError: o.fs.opt.LeavePartsOnError,
}
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
fs.Debugf(o, "open chunk writer: started multipart upload: %v", *mOut.UploadId)
backends: change OpenChunkWriter interface to allow backend concurrency override Before this change the concurrency used for an upload was rather inconsistent. - if size below `--backend-upload-cutoff` (default 200M) do single part upload. - if size below `--multi-thread-cutoff` (default 256M) or using streaming uploads (eg `rclone rcat) do multipart upload using `--backend-upload-concurrency` to set the concurrency used by the uploader. - otherwise do multipart upload using `--multi-thread-streams` to set the concurrency. This change makes the default for the concurrency used be the `--backend-upload-concurrency`. If `--multi-thread-streams` is set and larger than the `--backend-upload-concurrency` then that will be used instead. This means that if the user sets `--backend-upload-concurrency` then it will be obeyed for all multipart/multi-thread transfers and the user can override them all with `--multi-thread-streams`. See: #7056
2023-09-01 18:25:15 +02:00
return info, chunkWriter, err
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
}
backend/s3: use memory pool for buffer allocations Currently each multipart upload allocated his own buffers, which after file upload was garbaged. Next files couldn't leverage already allocated memory which resulted in inefficent memory management. This change introduces backend memory pool keeping memory chunks which can be used during object operations. Fixes #3967
2020-02-19 11:17:25 +01:00
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
// add a part number and etag to the completed parts
func (w *s3ChunkWriter) addCompletedPart(partNum *int64, eTag *string) {
w.completedPartsMu.Lock()
defer w.completedPartsMu.Unlock()
w.completedParts = append(w.completedParts, &s3.CompletedPart{
PartNumber: partNum,
ETag: eTag,
})
}
// addMd5 adds a binary md5 to the md5 calculated so far
func (w *s3ChunkWriter) addMd5(md5binary *[]byte, chunkNumber int64) {
w.md5sMu.Lock()
defer w.md5sMu.Unlock()
start := chunkNumber * md5.Size
end := start + md5.Size
if extend := end - int64(len(w.md5s)); extend > 0 {
w.md5s = append(w.md5s, make([]byte, extend)...)
}
copy(w.md5s[start:end], (*md5binary)[:])
}
// WriteChunk will write chunk number with reader bytes, where chunk number >= 0
fs: add context.Ctx to ChunkWriter methods WriteChunk in particular needs a different context from that which OpenChunkWriter was used with so add it to all the methods.
2023-08-19 18:30:55 +02:00
func (w *s3ChunkWriter) WriteChunk(ctx context.Context, chunkNumber int, reader io.ReadSeeker) (int64, error) {
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
if chunkNumber < 0 {
err := fmt.Errorf("invalid chunk number provided: %v", chunkNumber)
return -1, err
}
s3: fix accounting for multpart uploads
2023-08-24 18:15:18 +02:00
// Only account after the checksum reads have been done
if do, ok := reader.(pool.DelayAccountinger); ok {
// To figure out this number, do a transfer and if the accounted size is 0 or a
// multiple of what it should be, increase or decrease this number.
do.DelayAccounting(3)
}
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
// create checksum of buffer for integrity checking
// currently there is no way to calculate the md5 without reading the chunk a 2nd time (1st read is in uploadMultipart)
// possible in AWS SDK v2 with trailers?
m := md5.New()
currentChunkSize, err := io.Copy(m, reader)
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
if err != nil {
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
return -1, err
}
s3: fix multpart streaming uploads of 0 length files
2023-09-03 13:37:20 +02:00
// If no data read and not the first chunk, don't write the chunk
if currentChunkSize == 0 && chunkNumber != 0 {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
return 0, nil
}
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
md5sumBinary := m.Sum([]byte{})
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
w.addMd5(&md5sumBinary, int64(chunkNumber))
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
md5sum := base64.StdEncoding.EncodeToString(md5sumBinary[:])
// S3 requires 1 <= PartNumber <= 10000
s3PartNumber := aws.Int64(int64(chunkNumber + 1))
uploadPartReq := &s3.UploadPartInput{
Body: reader,
Bucket: w.bucket,
Key: w.key,
PartNumber: s3PartNumber,
build: fix lint errors when re-enabling revive var-naming
2023-08-29 13:57:33 +02:00
UploadId: w.uploadID,
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
ContentMD5: &md5sum,
ContentLength: aws.Int64(currentChunkSize),
RequestPayer: w.multiPartUploadInput.RequestPayer,
SSECustomerAlgorithm: w.multiPartUploadInput.SSECustomerAlgorithm,
SSECustomerKey: w.multiPartUploadInput.SSECustomerKey,
SSECustomerKeyMD5: w.multiPartUploadInput.SSECustomerKeyMD5,
}
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
var uout *s3.UploadPartOutput
err = w.f.pacer.Call(func() (bool, error) {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
// rewind the reader on retry and after reading md5
_, err = reader.Seek(0, io.SeekStart)
if err != nil {
return false, err
}
uout, err = w.f.c.UploadPartWithContext(ctx, uploadPartReq)
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
if err != nil {
if chunkNumber <= 8 {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
return w.f.shouldRetry(ctx, err)
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
}
// retry all chunks once have done the first few
return true, err
}
return false, nil
})
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
if err != nil {
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
return -1, fmt.Errorf("failed to upload chunk %d with %v bytes: %w", chunkNumber+1, currentChunkSize, err)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
}
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
w.addCompletedPart(s3PartNumber, uout.ETag)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
fs.Debugf(w.o, "multipart upload wrote chunk %d with %v bytes and etag %v", chunkNumber+1, currentChunkSize, *uout.ETag)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
return currentChunkSize, err
}
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
// Abort the multpart upload
fs: add context.Ctx to ChunkWriter methods WriteChunk in particular needs a different context from that which OpenChunkWriter was used with so add it to all the methods.
2023-08-19 18:30:55 +02:00
func (w *s3ChunkWriter) Abort(ctx context.Context) error {
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
err := w.f.pacer.Call(func() (bool, error) {
_, err := w.f.c.AbortMultipartUploadWithContext(context.Background(), &s3.AbortMultipartUploadInput{
Bucket: w.bucket,
Key: w.key,
build: fix lint errors when re-enabling revive var-naming
2023-08-29 13:57:33 +02:00
UploadId: w.uploadID,
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
RequestPayer: w.multiPartUploadInput.RequestPayer,
})
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
return w.f.shouldRetry(ctx, err)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
})
if err != nil {
build: fix lint errors when re-enabling revive var-naming
2023-08-29 13:57:33 +02:00
return fmt.Errorf("failed to abort multipart upload %q: %w", *w.uploadID, err)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
}
build: fix lint errors when re-enabling revive var-naming
2023-08-29 13:57:33 +02:00
fs.Debugf(w.o, "multipart upload %q aborted", *w.uploadID)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
return err
}
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
// Close and finalise the multipart upload
fs: add context.Ctx to ChunkWriter methods WriteChunk in particular needs a different context from that which OpenChunkWriter was used with so add it to all the methods.
2023-08-19 18:30:55 +02:00
func (w *s3ChunkWriter) Close(ctx context.Context) (err error) {
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
// sort the completed parts by part number
sort.Slice(w.completedParts, func(i, j int) bool {
return *w.completedParts[i].PartNumber < *w.completedParts[j].PartNumber
})
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
var resp *s3.CompleteMultipartUploadOutput
err = w.f.pacer.Call(func() (bool, error) {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
resp, err = w.f.c.CompleteMultipartUploadWithContext(ctx, &s3.CompleteMultipartUploadInput{
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
Bucket: w.bucket,
Key: w.key,
MultipartUpload: &s3.CompletedMultipartUpload{
Parts: w.completedParts,
},
RequestPayer: w.multiPartUploadInput.RequestPayer,
build: fix lint errors when re-enabling revive var-naming
2023-08-29 13:57:33 +02:00
UploadId: w.uploadID,
s3: fix retry logic, logging and error reporting for chunk upload - move retries into correct place into lowest level functions - fix logging and error reporting
2023-08-15 15:27:21 +02:00
})
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
return w.f.shouldRetry(ctx, err)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
})
if err != nil {
build: fix lint errors when re-enabling revive var-naming
2023-08-29 13:57:33 +02:00
return fmt.Errorf("failed to complete multipart upload %q: %w", *w.uploadID, err)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
}
if resp != nil {
if resp.ETag != nil {
w.eTag = *resp.ETag
}
if resp.VersionId != nil {
w.versionID = *resp.VersionId
}
}
build: fix lint errors when re-enabling revive var-naming
2023-08-29 13:57:33 +02:00
fs.Debugf(w.o, "multipart upload %q finished", *w.uploadID)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
return err
}
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
func (o *Object) uploadMultipart(ctx context.Context, src fs.ObjectInfo, in io.Reader, options ...fs.OpenOption) (wantETag, gotETag string, versionID *string, ui uploadInfo, err error) {
chunkWriter, err := multipart.UploadMultipart(ctx, src, in, multipart.UploadMultipartOptions{
backends: change OpenChunkWriter interface to allow backend concurrency override Before this change the concurrency used for an upload was rather inconsistent. - if size below `--backend-upload-cutoff` (default 200M) do single part upload. - if size below `--multi-thread-cutoff` (default 256M) or using streaming uploads (eg `rclone rcat) do multipart upload using `--backend-upload-concurrency` to set the concurrency used by the uploader. - otherwise do multipart upload using `--multi-thread-streams` to set the concurrency. This change makes the default for the concurrency used be the `--backend-upload-concurrency`. If `--multi-thread-streams` is set and larger than the `--backend-upload-concurrency` then that will be used instead. This means that if the user sets `--backend-upload-concurrency` then it will be obeyed for all multipart/multi-thread transfers and the user can override them all with `--multi-thread-streams`. See: #7056
2023-09-01 18:25:15 +02:00
Open: o.fs,
OpenOptions: options,
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
})
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
if err != nil {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
return wantETag, gotETag, versionID, ui, err
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
}
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
var s3cw *s3ChunkWriter = chunkWriter.(*s3ChunkWriter)
gotETag = s3cw.eTag
versionID = aws.String(s3cw.versionID)
hashOfHashes := md5.Sum(s3cw.md5s)
wantETag = fmt.Sprintf("%s-%d", hex.EncodeToString(hashOfHashes[:]), len(s3cw.completedParts))
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
return wantETag, gotETag, versionID, s3cw.ui, nil
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
}
s3: unwrap SDK errors to reveal underlying errors on upload The SDK doesn't wrap errors in a Go standard way so they can't be unwrapped and tested for - eg fatal error. The code looks for a Serialization or RequestError and returns the unwrapped underlying error if possible. This fixes the fs/operations integration tests checking for fatal errors being returned.
2022-06-17 15:34:57 +02:00
// unWrapAwsError unwraps AWS errors, looking for a non AWS error
//
// It returns true if one was found and the error, or false and the
// error passed in.
func unWrapAwsError(err error) (found bool, outErr error) {
if awsErr, ok := err.(awserr.Error); ok {
var origErrs []error
staticcheck: awserr.BatchError is deprecated: Replaced with BatchedErrors
2022-06-24 18:36:46 +02:00
if batchErr, ok := awsErr.(awserr.BatchedErrors); ok {
s3: unwrap SDK errors to reveal underlying errors on upload The SDK doesn't wrap errors in a Go standard way so they can't be unwrapped and tested for - eg fatal error. The code looks for a Serialization or RequestError and returns the unwrapped underlying error if possible. This fixes the fs/operations integration tests checking for fatal errors being returned.
2022-06-17 15:34:57 +02:00
origErrs = batchErr.OrigErrs()
} else {
origErrs = []error{awsErr.OrigErr()}
}
for _, origErr := range origErrs {
found, newErr := unWrapAwsError(origErr)
if found {
return found, newErr
}
}
return false, err
}
return true, err
}
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
// Upload a single part using PutObject
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
func (o *Object) uploadSinglepartPutObject(ctx context.Context, req *s3.PutObjectInput, size int64, in io.Reader) (etag string, lastModified time.Time, versionID *string, err error) {
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
r, resp := o.fs.c.PutObjectRequest(req)
if req.ContentLength != nil && *req.ContentLength == 0 {
// Can't upload zero length files like this for some reason
r.Body = bytes.NewReader([]byte{})
} else {
Replace deprecated ioutil As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code.
2022-08-20 16:38:02 +02:00
r.SetStreamingBody(io.NopCloser(in))
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
}
r.SetContext(ctx)
r.HTTPRequest.Header.Set("X-Amz-Content-Sha256", "UNSIGNED-PAYLOAD")
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
s3: simplify PutObject code to use the Request.SetStreamingBody method In this commit e5974ac4b0a847e8 s3: use PutObject from the aws SDK to upload single part objects rclone was made to upload objects to s3 using PUT requests rather than using signed uploads. However this change missed the fact that there is a supported way to do this in the SDK using the SetStreamingBody method on the Request. This therefore reverts a lot of the previous commit to do with making an unsigned connection and other complication and uses the SDK facility.
2022-06-16 23:55:45 +02:00
err := r.Send()
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
return o.fs.shouldRetry(ctx, err)
})
if err != nil {
s3: unwrap SDK errors to reveal underlying errors on upload The SDK doesn't wrap errors in a Go standard way so they can't be unwrapped and tested for - eg fatal error. The code looks for a Serialization or RequestError and returns the unwrapped underlying error if possible. This fixes the fs/operations integration tests checking for fatal errors being returned.
2022-06-17 15:34:57 +02:00
// Return the underlying error if we have a
// Serialization or RequestError error if possible
//
// These errors are synthesized locally in the SDK
// (not returned from the server) and we'd rather have
// the underlying error if there is one.
if do, ok := err.(awserr.Error); ok && (do.Code() == request.ErrCodeSerialization || do.Code() == request.ErrCodeRequestError) {
if found, newErr := unWrapAwsError(err); found {
err = newErr
}
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
return etag, lastModified, nil, err
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
}
lastModified = time.Now()
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
if resp != nil {
etag = aws.StringValue(resp.ETag)
versionID = resp.VersionId
}
return etag, lastModified, versionID, nil
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
}
// Upload a single part using a presigned request
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
func (o *Object) uploadSinglepartPresignedRequest(ctx context.Context, req *s3.PutObjectInput, size int64, in io.Reader) (etag string, lastModified time.Time, versionID *string, err error) {
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
// Create the request
putObj, _ := o.fs.c.PutObjectRequest(req)
// Sign it so we can upload using a presigned request.
//
// Note the SDK didn't used to support streaming to
// PutObject so we used this work-around.
url, headers, err := putObj.PresignRequest(15 * time.Minute)
if err != nil {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
return etag, lastModified, nil, fmt.Errorf("s3 upload: sign request: %w", err)
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
}
if o.fs.opt.V2Auth && headers == nil {
headers = putObj.HTTPRequest.Header
}
// Set request to nil if empty so as not to make chunked encoding
if size == 0 {
in = nil
}
// create the vanilla http request
httpReq, err := http.NewRequestWithContext(ctx, "PUT", url, in)
if err != nil {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
return etag, lastModified, nil, fmt.Errorf("s3 upload: new request: %w", err)
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
}
// set the headers we signed and the length
httpReq.Header = headers
httpReq.ContentLength = size
var resp *http.Response
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
var err error
resp, err = o.fs.srv.Do(httpReq)
if err != nil {
return o.fs.shouldRetry(ctx, err)
}
body, err := rest.ReadBody(resp)
if err != nil {
return o.fs.shouldRetry(ctx, err)
}
if resp.StatusCode >= 200 && resp.StatusCode < 299 {
return false, nil
}
err = fmt.Errorf("s3 upload: %s: %s", resp.Status, body)
return fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
})
if err != nil {
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
return etag, lastModified, nil, err
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
}
if resp != nil {
if date, err := http.ParseTime(resp.Header.Get("Date")); err != nil {
lastModified = date
}
etag = resp.Header.Get("Etag")
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
vID := resp.Header.Get("x-amz-version-id")
if vID != "" {
versionID = &vID
}
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
}
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
return etag, lastModified, versionID, nil
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
}
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
// Info needed for an upload
type uploadInfo struct {
req *s3.PutObjectInput
md5sumHex string
}
// Prepare object for being uploaded
func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options []fs.OpenOption) (ui uploadInfo, err error) {
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
bucket, bucketPath := o.split()
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
// Create parent dir/bucket if not saving directory marker
if !strings.HasSuffix(o.remote, "/") {
err := o.fs.mkdirParent(ctx, o.remote)
if err != nil {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
return ui, err
s3: empty directory markers further work #3453 - Report correct feature flag - Fix test failures due to that - don't output the root directory marker - Don't create the directory marker if it is the bucket or root - Create directories when uploading files
2023-04-26 11:59:17 +02:00
}
swift, b2, gcs, s3: Fix moveto and copyto We now make sure the container/bucket is created before creating any objects.
2017-06-07 15:16:50 +02:00
}
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
modTime := src.ModTime(ctx)
Factor Fs.Put into Object.Update and call Update rather than Put if possible
2014-04-18 18:04:21 +02:00
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req = &s3.PutObjectInput{
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
Bucket: &bucket,
s3: if bucket or object ACL is empty string then don't add X-Amz-Acl: header - fixes #5730 Before this fix it was impossible to stop rclone generating an X-Amx-Acl: header which is incompatible with GCS with uniform access control and is generally deprecated at AWS.
2022-09-14 17:24:57 +02:00
ACL: stringPointerOrNil(o.fs.opt.ACL),
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
Key: &bucketPath,
}
// Fetch metadata if --metadata is in use
meta, err := fs.GetMetadataOptions(ctx, src, options)
if err != nil {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
return ui, fmt.Errorf("failed to read metadata from source object: %w", err)
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
}
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.Metadata = make(map[string]*string, len(meta)+2)
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
// merge metadata into request and user metadata
for k, v := range meta {
pv := aws.String(v)
k = strings.ToLower(k)
s3: add --s3-no-system-metadata to suppress read and write of system metadata See: https://forum.rclone.org/t/problems-with-content-disposition-and-backblaze-b2-using-s3/33292/
2022-10-12 09:55:58 +02:00
if o.fs.opt.NoSystemMetadata {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.Metadata[k] = pv
s3: add --s3-no-system-metadata to suppress read and write of system metadata See: https://forum.rclone.org/t/problems-with-content-disposition-and-backblaze-b2-using-s3/33292/
2022-10-12 09:55:58 +02:00
continue
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
switch k {
case "cache-control":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.CacheControl = pv
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
case "content-disposition":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.ContentDisposition = pv
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
case "content-encoding":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.ContentEncoding = pv
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
case "content-language":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.ContentLanguage = pv
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
case "content-type":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.ContentType = pv
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
case "x-amz-tagging":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.Tagging = pv
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
case "tier":
// ignore
case "mtime":
// mtime in meta overrides source ModTime
metaModTime, err := time.Parse(time.RFC3339Nano, v)
if err != nil {
fs.Debugf(o, "failed to parse metadata %s: %q: %v", k, v, err)
} else {
modTime = metaModTime
}
case "btime":
// write as metadata since we can't set it
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.Metadata[k] = pv
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
default:
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.Metadata[k] = pv
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
// Set the mtime in the meta data
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.Metadata[metaMtime] = aws.String(swift.TimeToFloatString(modTime))
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
s3: fix missing x-amz-meta-md5chksum headers for multipart uploads This reverts "s3: fix DisableChecksum condition" which introduced the problem. This reverts commit c05bb63f9679e09ecb078e7477f3efd02846c2ad. The code was correct as it stands - the comment was incorrect and this commit updates it. See: https://forum.rclone.org/t/s3-upload-md5-check-sum/13706
2020-01-07 20:23:08 +01:00
// read the md5sum if available
Spelling fixes Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-09 02:17:24 +02:00
// - for non multipart
s3: fix missing x-amz-meta-md5chksum headers for multipart uploads This reverts "s3: fix DisableChecksum condition" which introduced the problem. This reverts commit c05bb63f9679e09ecb078e7477f3efd02846c2ad. The code was correct as it stands - the comment was incorrect and this commit updates it. See: https://forum.rclone.org/t/s3-upload-md5-check-sum/13706
2020-01-07 20:23:08 +01:00
// - so we can add a ContentMD5
s3: Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C Before this change, small objects uploaded with SSE-AWS/SSE-C would not have MD5 sums. This change adds metadata for these objects in the same way that the metadata is stored for multipart uploaded objects. See: #1824 #2827
2020-11-23 12:53:31 +01:00
// - so we can add the md5sum in the metadata as metaMD5Hash if using SSE/SSE-C
s3: fix missing x-amz-meta-md5chksum headers for multipart uploads This reverts "s3: fix DisableChecksum condition" which introduced the problem. This reverts commit c05bb63f9679e09ecb078e7477f3efd02846c2ad. The code was correct as it stands - the comment was incorrect and this commit updates it. See: https://forum.rclone.org/t/s3-upload-md5-check-sum/13706
2020-01-07 20:23:08 +01:00
// - for multipart provided checksums aren't disabled
// - so we can add the md5sum in the metadata as metaMD5Hash
s3: fix multipart upload with --no-head flag - Fixes #5956 Before this change a multipart upload with the --no-head flag returned the MD5SUM as a base64 string rather than a Hex string as the rest of rclone was expecting.
2022-01-28 17:18:40 +01:00
var md5sumBase64 string
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
size := src.Size()
multipart := size < 0 || size >= int64(o.fs.opt.UploadCutoff)
s3: fix missing x-amz-meta-md5chksum headers for multipart uploads This reverts "s3: fix DisableChecksum condition" which introduced the problem. This reverts commit c05bb63f9679e09ecb078e7477f3efd02846c2ad. The code was correct as it stands - the comment was incorrect and this commit updates it. See: https://forum.rclone.org/t/s3-upload-md5-check-sum/13706
2020-01-07 20:23:08 +01:00
if !multipart || !o.fs.opt.DisableChecksum {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.md5sumHex, err = src.Hash(ctx, hash.MD5)
if err == nil && matchMd5.MatchString(ui.md5sumHex) {
hashBytes, err := hex.DecodeString(ui.md5sumHex)
s3: set/get the hash for multipart files - #523
2018-01-06 15:30:10 +01:00
if err == nil {
s3: fix multipart upload with --no-head flag - Fixes #5956 Before this change a multipart upload with the --no-head flag returned the MD5SUM as a base64 string rather than a Hex string as the rest of rclone was expecting.
2022-01-28 17:18:40 +01:00
md5sumBase64 = base64.StdEncoding.EncodeToString(hashBytes)
s3: Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C Before this change, small objects uploaded with SSE-AWS/SSE-C would not have MD5 sums. This change adds metadata for these objects in the same way that the metadata is stored for multipart uploaded objects. See: #1824 #2827
2020-11-23 12:53:31 +01:00
if (multipart || o.fs.etagIsNotMD5) && !o.fs.opt.DisableChecksum {
// Set the md5sum as metadata on the object if
// - a multipart upload
// - the Etag is not an MD5, eg when using SSE/SSE-C
// provided checksums aren't disabled
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.Metadata[metaMD5Hash] = &md5sumBase64
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
}
s3: set/get the hash for multipart files - #523
2018-01-06 15:30:10 +01:00
}
}
}
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
// Set the content type if it isn't set already
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
if ui.req.ContentType == nil {
ui.req.ContentType = aws.String(fs.MimeType(ctx, src))
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
}
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
if size >= 0 {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.ContentLength = &size
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
}
s3: fix multipart upload with --no-head flag - Fixes #5956 Before this change a multipart upload with the --no-head flag returned the MD5SUM as a base64 string rather than a Hex string as the rest of rclone was expecting.
2022-01-28 17:18:40 +01:00
if md5sumBase64 != "" {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.ContentMD5 = &md5sumBase64
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
}
s3: add requester pays option - fixes #301
2020-12-03 03:30:06 +01:00
if o.fs.opt.RequesterPays {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.RequestPayer = aws.String(s3.RequestPayerRequester)
s3: add requester pays option - fixes #301
2020-12-03 03:30:06 +01:00
}
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
if o.fs.opt.ServerSideEncryption != "" {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.ServerSideEncryption = &o.fs.opt.ServerSideEncryption
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
}
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
if o.fs.opt.SSECustomerAlgorithm != "" {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.SSECustomerAlgorithm = &o.fs.opt.SSECustomerAlgorithm
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
}
if o.fs.opt.SSECustomerKey != "" {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.SSECustomerKey = &o.fs.opt.SSECustomerKey
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
}
if o.fs.opt.SSECustomerKeyMD5 != "" {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.SSECustomerKeyMD5 = &o.fs.opt.SSECustomerKeyMD5
backend/s3: add SSE-C support for AWS, Ceph, and MinIO
2020-03-30 12:26:52 +02:00
}
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
if o.fs.opt.SSEKMSKeyID != "" {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.SSEKMSKeyId = &o.fs.opt.SSEKMSKeyID
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
}
if o.fs.opt.StorageClass != "" {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.StorageClass = &o.fs.opt.StorageClass
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
}
s3: fix --header-upload - Fixes #4303 Before this change we were setting the headers on the PUT request for normal and multipart uploads. For normal uploads this caused the error 403 Forbidden: There were headers present in the request which were not signed After this fix we set the headers in the object upload request itself as the s3 SDK expects. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type - X-Amz-Tagging - X-Amz-Meta- Note for the last of those are for setting custom metadata in the form "X-Amz-Meta-Key: value". This now works for multipart uploads and single part uploads See also #59
2020-06-05 12:45:54 +02:00
// Apply upload options
for _, option := range options {
key, value := option.Header()
lowerKey := strings.ToLower(key)
switch lowerKey {
case "":
// ignore
case "cache-control":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.CacheControl = aws.String(value)
s3: fix --header-upload - Fixes #4303 Before this change we were setting the headers on the PUT request for normal and multipart uploads. For normal uploads this caused the error 403 Forbidden: There were headers present in the request which were not signed After this fix we set the headers in the object upload request itself as the s3 SDK expects. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type - X-Amz-Tagging - X-Amz-Meta- Note for the last of those are for setting custom metadata in the form "X-Amz-Meta-Key: value". This now works for multipart uploads and single part uploads See also #59
2020-06-05 12:45:54 +02:00
case "content-disposition":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.ContentDisposition = aws.String(value)
s3: fix --header-upload - Fixes #4303 Before this change we were setting the headers on the PUT request for normal and multipart uploads. For normal uploads this caused the error 403 Forbidden: There were headers present in the request which were not signed After this fix we set the headers in the object upload request itself as the s3 SDK expects. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type - X-Amz-Tagging - X-Amz-Meta- Note for the last of those are for setting custom metadata in the form "X-Amz-Meta-Key: value". This now works for multipart uploads and single part uploads See also #59
2020-06-05 12:45:54 +02:00
case "content-encoding":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.ContentEncoding = aws.String(value)
s3: fix --header-upload - Fixes #4303 Before this change we were setting the headers on the PUT request for normal and multipart uploads. For normal uploads this caused the error 403 Forbidden: There were headers present in the request which were not signed After this fix we set the headers in the object upload request itself as the s3 SDK expects. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type - X-Amz-Tagging - X-Amz-Meta- Note for the last of those are for setting custom metadata in the form "X-Amz-Meta-Key: value". This now works for multipart uploads and single part uploads See also #59
2020-06-05 12:45:54 +02:00
case "content-language":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.ContentLanguage = aws.String(value)
s3: fix --header-upload - Fixes #4303 Before this change we were setting the headers on the PUT request for normal and multipart uploads. For normal uploads this caused the error 403 Forbidden: There were headers present in the request which were not signed After this fix we set the headers in the object upload request itself as the s3 SDK expects. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type - X-Amz-Tagging - X-Amz-Meta- Note for the last of those are for setting custom metadata in the form "X-Amz-Meta-Key: value". This now works for multipart uploads and single part uploads See also #59
2020-06-05 12:45:54 +02:00
case "content-type":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.ContentType = aws.String(value)
s3: fix --header-upload - Fixes #4303 Before this change we were setting the headers on the PUT request for normal and multipart uploads. For normal uploads this caused the error 403 Forbidden: There were headers present in the request which were not signed After this fix we set the headers in the object upload request itself as the s3 SDK expects. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type - X-Amz-Tagging - X-Amz-Meta- Note for the last of those are for setting custom metadata in the form "X-Amz-Meta-Key: value". This now works for multipart uploads and single part uploads See also #59
2020-06-05 12:45:54 +02:00
case "x-amz-tagging":
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.Tagging = aws.String(value)
s3: fix --header-upload - Fixes #4303 Before this change we were setting the headers on the PUT request for normal and multipart uploads. For normal uploads this caused the error 403 Forbidden: There were headers present in the request which were not signed After this fix we set the headers in the object upload request itself as the s3 SDK expects. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type - X-Amz-Tagging - X-Amz-Meta- Note for the last of those are for setting custom metadata in the form "X-Amz-Meta-Key: value". This now works for multipart uploads and single part uploads See also #59
2020-06-05 12:45:54 +02:00
default:
const amzMetaPrefix = "x-amz-meta-"
if strings.HasPrefix(lowerKey, amzMetaPrefix) {
metaKey := lowerKey[len(amzMetaPrefix):]
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui.req.Metadata[metaKey] = aws.String(value)
s3: fix --header-upload - Fixes #4303 Before this change we were setting the headers on the PUT request for normal and multipart uploads. For normal uploads this caused the error 403 Forbidden: There were headers present in the request which were not signed After this fix we set the headers in the object upload request itself as the s3 SDK expects. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type - X-Amz-Tagging - X-Amz-Meta- Note for the last of those are for setting custom metadata in the form "X-Amz-Meta-Key: value". This now works for multipart uploads and single part uploads See also #59
2020-06-05 12:45:54 +02:00
} else {
fs.Errorf(o, "Don't know how to set key %q on upload", key)
}
}
}
s3: re-implement multipart upload to fix memory issues There have been quite a few reports of problems with the multipart uploader using too much memory and not retrying possible errors. Before this change the multipart uploader used the s3manager abstraction in the AWS SDK. There are numerous bug reports of this using up too much memory. This change re-implements a much simplified version of the s3manager code specialized for rclone's purposes. This should use much less memory and retry chunks properly. See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563 See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405 See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2019-12-31 00:17:06 +01:00
s3: drop binary metadata with an ERROR message Before this change, rclone would attempt to upload metadata with binary contents which fail to be uploaded by net/http. This checks the keys and values for validity as http header values before uploading. See: https://forum.rclone.org/t/invalid-metadata-key-names-result-in-a-failure-to-transfer-xattr-results-in-failure-to-upload-net-http-invalid-header-field-value-for-x-amz-meta-samba-pai/33406/
2022-10-11 18:54:22 +02:00
// Check metadata keys and values are valid
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
for key, value := range ui.req.Metadata {
s3: drop binary metadata with an ERROR message Before this change, rclone would attempt to upload metadata with binary contents which fail to be uploaded by net/http. This checks the keys and values for validity as http header values before uploading. See: https://forum.rclone.org/t/invalid-metadata-key-names-result-in-a-failure-to-transfer-xattr-results-in-failure-to-upload-net-http-invalid-header-field-value-for-x-amz-meta-samba-pai/33406/
2022-10-11 18:54:22 +02:00
if !httpguts.ValidHeaderFieldName(key) {
fs.Errorf(o, "Dropping invalid metadata key %q", key)
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
delete(ui.req.Metadata, key)
s3: drop binary metadata with an ERROR message Before this change, rclone would attempt to upload metadata with binary contents which fail to be uploaded by net/http. This checks the keys and values for validity as http header values before uploading. See: https://forum.rclone.org/t/invalid-metadata-key-names-result-in-a-failure-to-transfer-xattr-results-in-failure-to-upload-net-http-invalid-header-field-value-for-x-amz-meta-samba-pai/33406/
2022-10-11 18:54:22 +02:00
} else if value == nil {
fs.Errorf(o, "Dropping nil metadata value for key %q", key)
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
delete(ui.req.Metadata, key)
s3: drop binary metadata with an ERROR message Before this change, rclone would attempt to upload metadata with binary contents which fail to be uploaded by net/http. This checks the keys and values for validity as http header values before uploading. See: https://forum.rclone.org/t/invalid-metadata-key-names-result-in-a-failure-to-transfer-xattr-results-in-failure-to-upload-net-http-invalid-header-field-value-for-x-amz-meta-samba-pai/33406/
2022-10-11 18:54:22 +02:00
} else if !httpguts.ValidHeaderFieldValue(*value) {
fs.Errorf(o, "Dropping invalid metadata value %q for key %q", *value, key)
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
delete(ui.req.Metadata, key)
s3: drop binary metadata with an ERROR message Before this change, rclone would attempt to upload metadata with binary contents which fail to be uploaded by net/http. This checks the keys and values for validity as http header values before uploading. See: https://forum.rclone.org/t/invalid-metadata-key-names-result-in-a-failure-to-transfer-xattr-results-in-failure-to-upload-net-http-invalid-header-field-value-for-x-amz-meta-samba-pai/33406/
2022-10-11 18:54:22 +02:00
}
}
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
return ui, nil
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
}
// Update the Object from in with modTime and size
func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
if o.fs.opt.VersionAt.IsSet() {
return errNotWithVersionAt
}
size := src.Size()
multipart := size < 0 || size >= int64(o.fs.opt.UploadCutoff)
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
var wantETag string // Multipart upload Etag to check
s3: Check multipart upload ETag when --s3-no-head is in use Before this change if --s3-no-head was in use rclone didn't check the multipart upload ETag at all. However the ETag is returned in the final POST request when completing the object. This change uses that ETag from the final POST if --s3-no-head is in use, otherwise it uses the ETag from a fresh HEAD request. See: https://forum.rclone.org/t/in-some-cases-rclone-does-not-use-etag-to-verify-files/36095/
2023-02-13 11:31:31 +01:00
var gotETag string // Etag we got from the upload
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
var lastModified time.Time // Time we got from the upload
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
var versionID *string // versionID we got from the upload
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
var err error
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
var ui uploadInfo
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
if multipart {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
wantETag, gotETag, versionID, ui, err = o.uploadMultipart(ctx, src, in)
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
} else {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
ui, err = o.prepareUpload(ctx, src, options)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
if err != nil {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
return fmt.Errorf("failed to prepare upload: %w", err)
s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056
2023-07-18 20:37:31 +02:00
}
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
if o.fs.opt.UsePresignedRequest {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
gotETag, lastModified, versionID, err = o.uploadSinglepartPresignedRequest(ctx, ui.req, size, in)
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
} else {
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
gotETag, lastModified, versionID, err = o.uploadSinglepartPutObject(ctx, ui.req, size, in)
s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 Before this change rclone would use multipart uploads for any size of file. However multipart uploads are less efficient for smaller files and don't have MD5 checksums so it is advantageous to use single part uploads if possible. This implements single part uploads for all files smaller than the upload_cutoff size. Streamed files must be uploaded as multipart files though.
2018-11-26 22:09:23 +01:00
}
s3: Read metadata after Update or Put
2014-07-19 13:37:11 +02:00
}
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
if err != nil {
return err
}
s3: ignore versionIDs from uploads unless using --s3-versions or --s3-versions-at Before this change, when a new object was created s3 returns its versionID (on a versioned bucket) and rclone recorded it in the object. This means that when rclone came to delete the object it would delete it with the versionID. However it is common to forbid actions with versionIDs on buckets so as to preserve the historical record and these operations would fail whereas they succeeded in pre-v1.60.0 versions. This patch fixes the problem by not recording versions of objects supplied by the S3 API on upload unless `--s3-versions` or `--s3-version-at` is used. This makes rclone behave as it did before v1.60.0 when version support was introduced. See: https://forum.rclone.org/t/s3-and-intermittent-403-errors-with-file-renames-and-drag-and-drop-operations-in-windows-explorer/34773
2022-12-16 12:44:58 +01:00
// Only record versionID if we are using --s3-versions or --s3-version-at
if o.fs.opt.Versions || o.fs.opt.VersionAt.IsSet() {
o.versionID = versionID
} else {
o.versionID = nil
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
s3: add --s3-no-head parameter to minimise transactions on upload See: https://forum.rclone.org/t/prevent-head-on-amazon-s3-family/21935
2021-01-29 12:13:42 +01:00
// User requested we don't HEAD the object after uploading it
// so make up the object as best we can assuming it got
// uploaded properly. If size < 0 then we need to do the HEAD.
s3: Check multipart upload ETag when --s3-no-head is in use Before this change if --s3-no-head was in use rclone didn't check the multipart upload ETag at all. However the ETag is returned in the final POST request when completing the object. This change uses that ETag from the final POST if --s3-no-head is in use, otherwise it uses the ETag from a fresh HEAD request. See: https://forum.rclone.org/t/in-some-cases-rclone-does-not-use-etag-to-verify-files/36095/
2023-02-13 11:31:31 +01:00
var head *s3.HeadObjectOutput
s3: add --s3-no-head parameter to minimise transactions on upload See: https://forum.rclone.org/t/prevent-head-on-amazon-s3-family/21935
2021-01-29 12:13:42 +01:00
if o.fs.opt.NoHead && size >= 0 {
s3: Check multipart upload ETag when --s3-no-head is in use Before this change if --s3-no-head was in use rclone didn't check the multipart upload ETag at all. However the ETag is returned in the final POST request when completing the object. This change uses that ETag from the final POST if --s3-no-head is in use, otherwise it uses the ETag from a fresh HEAD request. See: https://forum.rclone.org/t/in-some-cases-rclone-does-not-use-etag-to-verify-files/36095/
2023-02-13 11:31:31 +01:00
head = new(s3.HeadObjectOutput)
//structs.SetFrom(head, &req)
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
setFrom_s3HeadObjectOutput_s3PutObjectInput(head, ui.req)
head.ETag = &ui.md5sumHex // doesn't matter quotes are missing
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
head.ContentLength = &size
s3: Check multipart upload ETag when --s3-no-head is in use Before this change if --s3-no-head was in use rclone didn't check the multipart upload ETag at all. However the ETag is returned in the final POST request when completing the object. This change uses that ETag from the final POST if --s3-no-head is in use, otherwise it uses the ETag from a fresh HEAD request. See: https://forum.rclone.org/t/in-some-cases-rclone-does-not-use-etag-to-verify-files/36095/
2023-02-13 11:31:31 +01:00
// We get etag back from single and multipart upload so fill it in here
if gotETag != "" {
head.ETag = &gotETag
s3: use PutObject from the aws SDK to upload single part objects Before this change rclone used presigned requests to upload single part objects. This was because of a limitation in the SDK which didn't allow non seekable io.Readers to be passed in. This is incompatible with some S3 backends, and rclone wasn't adding the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` header which was incompatible with other S3 backends. The SDK now allows for this so rclone can use PutObject directly. This sets the `X-Amz-Content-Sha256: UNSIGNED-PAYLOAD` flag on the PUT request. However rclone will add a `Content-Md5` header if at all possible so the body data is still protected. Note that the old behaviour can still be configured if required with the `use_presigned_request` config parameter. Fixes #5422
2022-05-03 18:39:01 +02:00
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
if lastModified.IsZero() {
lastModified = time.Now()
s3: add --s3-no-head parameter to minimise transactions on upload See: https://forum.rclone.org/t/prevent-head-on-amazon-s3-family/21935
2021-01-29 12:13:42 +01:00
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
head.LastModified = &lastModified
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
head.VersionId = versionID
s3: Check multipart upload ETag when --s3-no-head is in use Before this change if --s3-no-head was in use rclone didn't check the multipart upload ETag at all. However the ETag is returned in the final POST request when completing the object. This change uses that ETag from the final POST if --s3-no-head is in use, otherwise it uses the ETag from a fresh HEAD request. See: https://forum.rclone.org/t/in-some-cases-rclone-does-not-use-etag-to-verify-files/36095/
2023-02-13 11:31:31 +01:00
} else {
// Read the metadata from the newly created object
o.meta = nil // wipe old metadata
head, err = o.headObject(ctx)
if err != nil {
return err
}
s3: Use the ETag on multipart transfers to verify the transfer was OK Before this rclone ignored the ETag on multipart uploads which missed an opportunity for a whole file integrity check. This adds that check which means that we now check even harder that multipart uploads have arrived properly. See #5993
2022-02-16 18:50:11 +01:00
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
o.setMetaData(head)
s3: Check multipart upload ETag when --s3-no-head is in use Before this change if --s3-no-head was in use rclone didn't check the multipart upload ETag at all. However the ETag is returned in the final POST request when completing the object. This change uses that ETag from the final POST if --s3-no-head is in use, otherwise it uses the ETag from a fresh HEAD request. See: https://forum.rclone.org/t/in-some-cases-rclone-does-not-use-etag-to-verify-files/36095/
2023-02-13 11:31:31 +01:00
// Check multipart upload ETag if required
s3: add --s3-use-multipart-etag provider quirk #5993 Before this change the new multipart upload ETag checking code was failing in the integration tests with Alibaba OSS. Apparently Alibaba calculate the ETag in a different way to AWS. This introduces a new provider quirk with a flag to disable the checking of the ETag for multipart uploads. Mulpart Etag checking has been enabled for all providers that we can test for and work, and left disabled for the others.
2022-02-27 16:47:31 +01:00
if o.fs.opt.UseMultipartEtag.Value && !o.fs.etagIsNotMD5 && wantETag != "" && head.ETag != nil && *head.ETag != "" {
s3: Use the ETag on multipart transfers to verify the transfer was OK Before this rclone ignored the ETag on multipart uploads which missed an opportunity for a whole file integrity check. This adds that check which means that we now check even harder that multipart uploads have arrived properly. See #5993
2022-02-16 18:50:11 +01:00
gotETag := strings.Trim(strings.ToLower(*head.ETag), `"`)
if wantETag != gotETag {
return fmt.Errorf("multipart upload corrupted: Etag differ: expecting %s but got %s", wantETag, gotETag)
}
fs.Debugf(o, "Multipart upload Etag: %s OK", wantETag)
}
Factor Fs.Put into Object.Update and call Update rather than Put if possible
2014-04-18 18:04:21 +02:00
return err
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
// Remove an object
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (o *Object) Remove(ctx context.Context) error {
s3: implement --s3-version-at flag - Fixes #1776
2022-07-26 18:58:57 +02:00
if o.fs.opt.VersionAt.IsSet() {
return errNotWithVersionAt
}
s3: make all operations work from the root #3421
2019-08-09 12:29:36 +02:00
bucket, bucketPath := o.split()
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
req := s3.DeleteObjectInput{
s3: implement --s3-versions flag - See #1776
2022-07-25 17:06:15 +02:00
Bucket: &bucket,
Key: &bucketPath,
VersionId: o.versionID,
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
}
s3: add requester pays option - fixes #301
2020-12-03 03:30:06 +01:00
if o.fs.opt.RequesterPays {
req.RequestPayer = aws.String(s3.RequestPayerRequester)
}
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
err := o.fs.pacer.Call(func() (bool, error) {
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
_, err := o.fs.c.DeleteObjectWithContext(ctx, &req)
backends: Add context checking to remaining backends #4504 This is a follow up to 4013bc4a4cd768e2 which missed some backends. It adds a ctx parameter to shouldRetry and checks it.
2021-03-16 16:50:02 +01:00
return o.fs.shouldRetry(ctx, err)
S3: Use (custom) pacer, to retry operations when reasonable - fixes #2503
2018-09-03 06:41:04 +02:00
})
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 13:09:02 +01:00
return err
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
}
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 23:13:24 +02:00
// MimeType of an Object if known, "" otherwise
Add context propagation to rclone - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.
2019-06-17 10:34:30 +02:00
func (o *Object) MimeType(ctx context.Context) string {
err := o.readMetaData(ctx)
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 23:13:24 +02:00
if err != nil {
Rename log functions and factor into own file
2017-02-09 12:01:20 +01:00
fs.Logf(o, "Failed to read metadata: %v", err)
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 23:13:24 +02:00
return ""
}
return o.mimeType
}
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
// SetTier performs changing storage class
func (o *Object) SetTier(tier string) (err error) {
ctx := context.TODO()
tier = strings.ToUpper(tier)
bucket, bucketPath := o.split()
req := s3.CopyObjectInput{
MetadataDirective: aws.String(s3.MetadataDirectiveCopy),
StorageClass: aws.String(tier),
}
s3: preserve metadata when doing multipart copy Before this change the s3 multipart server side copy was not preserving the metadata of the object. This was most noticeable because the modtime was not preserved. This change fetches the metadata from the object before starting the copy and overwrites it if requires. It will also mean any other metadata is preserved. See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/70
2020-07-30 11:52:32 +02:00
err = o.fs.copy(ctx, &req, bucket, bucketPath, bucket, bucketPath, o)
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
if err != nil {
return err
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
o.storageClass = &tier
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
return err
}
// GetTier returns storage class as string
func (o *Object) GetTier() string {
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
if o.storageClass == nil || *o.storageClass == "" {
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
return "STANDARD"
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
return *o.storageClass
}
// Metadata returns metadata for an object
//
// It should return nil if there is no Metadata
func (o *Object) Metadata(ctx context.Context) (metadata fs.Metadata, err error) {
err = o.readMetaData(ctx)
if err != nil {
return nil, err
}
metadata = make(fs.Metadata, len(o.meta)+7)
for k, v := range o.meta {
switch k {
case metaMtime:
if modTime, err := swift.FloatStringToTime(v); err == nil {
metadata["mtime"] = modTime.Format(time.RFC3339Nano)
}
case metaMD5Hash:
// don't write hash metadata
default:
metadata[k] = v
}
}
if o.mimeType != "" {
metadata["content-type"] = o.mimeType
}
// metadata["x-amz-tagging"] = ""
if !o.lastModified.IsZero() {
metadata["btime"] = o.lastModified.Format(time.RFC3339Nano)
}
// Set system metadata
setMetadata := func(k string, v *string) {
s3: add --s3-no-system-metadata to suppress read and write of system metadata See: https://forum.rclone.org/t/problems-with-content-disposition-and-backblaze-b2-using-s3/33292/
2022-10-12 09:55:58 +02:00
if o.fs.opt.NoSystemMetadata {
return
}
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
if v == nil || *v == "" {
return
}
metadata[k] = *v
}
setMetadata("cache-control", o.cacheControl)
setMetadata("content-disposition", o.contentDisposition)
setMetadata("content-encoding", o.contentEncoding)
setMetadata("content-language", o.contentLanguage)
s3: fix missing "tier" metadata Before this change if the storage class wasn't set on the object, we didn't set the "tier" metadata. This made it impossible to filter on tier using the metadata filters. This returns the "tier" metadata as STANDARD if the storage class isn't set on the object. See: https://forum.rclone.org/t/copy-from-s3-to-another-s3-filter-by-storage-class/37861
2023-04-27 16:27:31 +02:00
metadata["tier"] = o.GetTier()
s3: implement reading and writing of metadata #111
2022-05-24 13:32:39 +02:00
return metadata, nil
s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier - Read the storage class for each object - Implement SetTier/GetTier - Check the storage class on the **object** before using SetModTime This updates the fix in 1a2fb52 so that SetModTime works when you are using objects which have been migrated to GLACIER but you aren't using GLACIER as a storage class. Fixes #3522
2019-09-09 21:44:50 +02:00
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 19:53:35 +01:00
// Check the interfaces are satisfied
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
var (
s3: factor generic multipart upload into lib/multipart #7056 This makes the memory controls of the s3 backend inoperative and replaced with the global ones. --s3-memory-pool-flush-time --s3-memory-pool-use-mmap By using the buffered reader this fixes excessive memory use when uploading large files as it will share memory pages between all readers. Fixes #7141
2023-08-15 21:38:02 +02:00
_ fs.Fs = &Fs{}
_ fs.Purger = &Fs{}
_ fs.Copier = &Fs{}
_ fs.PutStreamer = &Fs{}
_ fs.ListRer = &Fs{}
_ fs.Commander = &Fs{}
_ fs.CleanUpper = &Fs{}
_ fs.OpenChunkWriter = &Fs{}
_ fs.Object = &Object{}
_ fs.MimeTyper = &Object{}
_ fs.GetTierer = &Object{}
_ fs.SetTierer = &Object{}
_ fs.Metadataer = &Object{}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 12:14:46 +01:00
)
Reference in New Issue Copy Permalink