From da70088b11e633999f2f920d3c603d97c292135e Mon Sep 17 00:00:00 2001 From: Pawel Palucha Date: Wed, 11 Sep 2024 09:23:49 +0200 Subject: [PATCH] docs: add section for improving performance for s3 --- docs/content/s3.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/content/s3.md b/docs/content/s3.md index db52f5c26..9031e57fe 100644 --- a/docs/content/s3.md +++ b/docs/content/s3.md @@ -401,6 +401,38 @@ there for more details. Setting this flag increases the chance for undetected upload failures. +### Increasing performance + +#### Using server-side copy + +If you are copying objects between S3 buckets in the same region, you should +use server-side copy. +This is much faster than downloading and re-uploading the objects, as no data is transferred. + +For rclone to use server-side copy, you must use the same remote for the source and destination. + + rclone copy s3:source-bucket s3:destination-bucket + +When using server-side copy, the performance is limited by the rate at which rclone issues +API requests to S3. +See below for how to increase the number of API requests rclone makes. + +#### Increasing the rate of API requests + +You can increase the rate of API requests to S3 by increasing the parallelism using `--transfers` and `--checkers` +options. + +Rclone uses a very conservative defaults for these settings, as not all providers support high rates of requests. +Depending on your provider, you can increase significantly the number of transfers and checkers. + +For example, with AWS S3, if you can increase the number of checkers to values like 200. +If you are doing a server-side copy, you can also increase the number of transfers to 200. + + rclone sync --transfers 200 --checkers 200 --checksum s3:source-bucket s3:destination-bucket + +You will need to experiment with these values to find the optimal settings for your setup. + + ### Versions When bucket versioning is enabled (this can be done with rclone with