s3: update to using AWS SDK v2 - fixes #4989

SDK v2 conversion

Changes

  - `--s3-sts-endpoint` is no longer supported
  - `--s3-use-unsigned-payload` to control use of trailer checksums (needed for non AWS)
This commit is contained in:
Nick Craig-Wood
2024-08-03 11:35:32 +01:00
parent a1f52bcf50
commit fd9049c83d
9 changed files with 558 additions and 552 deletions

View File

@@ -571,15 +571,21 @@ The different authentication methods are tried in this order:
- Session Token: `AWS_SESSION_TOKEN` (optional)
- Or, use a [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html):
- Profile files are standard files used by AWS CLI tools
- By default it will use the profile in your home directory (e.g. `~/.aws/credentials` on unix based systems) file and the "default" profile, to change set these environment variables:
- `AWS_SHARED_CREDENTIALS_FILE` to control which file.
- `AWS_PROFILE` to control which profile to use.
- By default it will use the profile in your home directory (e.g. `~/.aws/credentials` on unix based systems) file and the "default" profile, to change set these environment variables or config keys:
- `AWS_SHARED_CREDENTIALS_FILE` to control which file or the `shared_credentials_file` config key.
- `AWS_PROFILE` to control which profile to use or the `profile` config key.
- Or, run `rclone` in an ECS task with an IAM role (AWS only).
- Or, run `rclone` on an EC2 instance with an IAM role (AWS only).
- Or, run `rclone` in an EKS pod with an IAM role that is associated with a service account (AWS only).
- Or, use [process credentials](https://docs.aws.amazon.com/sdkref/latest/guide/feature-process-credentials.html) to read config from an external program.
With `env_auth = true` rclone (which uses the SDK for Go v2) should support
[all authentication methods](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html)
that the `aws` CLI tool does and the other AWS SDKs.
If none of these option actually end up providing `rclone` with AWS
credentials then S3 interaction will be non-authenticated (see below).
credentials then S3 interaction will be non-authenticated (see the
[anonymous access](#anonymous-access) section for more info).
### S3 Permissions
@@ -2171,7 +2177,7 @@ It doesn't return anything.
{{< rem autogenerated options stop >}}
### Anonymous access to public buckets
### Anonymous access to public buckets {#anonymous-access}
If you want to use rclone to access a public bucket, configure with a
blank `access_key_id` and `secret_access_key`. Your config should end
@@ -2181,15 +2187,6 @@ up looking like this:
[anons3]
type = s3
provider = AWS
env_auth = false
access_key_id =
secret_access_key =
region = us-east-1
endpoint =
location_constraint =
acl = private
server_side_encryption =
storage_class =
```
Then use it as normal with the name of the public bucket, e.g.
@@ -2198,6 +2195,10 @@ Then use it as normal with the name of the public bucket, e.g.
You will be able to list and copy data but not upload it.
You can also do this entirely on the command line
rclone lsd :s3,provider=AWS:1000genomes
## Providers
### AWS S3