extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-12-22 15:11:56 +01:00
Code Issues Packages Projects Releases Wiki Activity

docs: s3: add section on using too much memory #7974

Browse Source
This commit is contained in:
Nick Craig-Wood 2024-07-30 09:51:30 +01:00
parent 3e8cee148a
commit 1ac18e5765
1 changed files with 22 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
docs/content/s3.md
View File

@ -5018,6 +5018,28 @@ nodes across the network.
For more detailed comparison please check the documentation of the For more detailed comparison please check the documentation of the
[storj](/storj) backend. [storj](/storj) backend.
## Memory usage {memory}
The most common cause of rclone using lots of memory is a single
directory with millions of files in. Despite s3 not really having the
concepts of directories, rclone does the sync on a directory by
directory basis to be compatible with normal filing systems.
Rclone loads each directory into memory as rclone objects. Each rclone
object takes 0.5k-1k of memory, so approximately 1GB per 1,000,000
files, and the sync for that directory does not begin until it is
entirely loaded in memory. So the sync can take a long time to start
for large directories.
To sync a directory with 100,000,000 files in you would need approximately
100 GB of memory. At some point the amount of memory becomes difficult
to provide so there is
[a workaround for this](https://github.com/rclone/rclone/wiki/Big-syncs-with-millions-of-files)
which involves a bit of scripting.
At some point rclone will gain a sync mode which is effectively this
workaround but built in to rclone.
## Limitations ## Limitations
`rclone about` is not supported by the S3 backend. Backends without `rclone about` is not supported by the S3 backend. Backends without
@ -5028,7 +5050,6 @@ remote.
See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/) See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
### Synology C2 Object Storage {#synology-c2} ### Synology C2 Object Storage {#synology-c2}
[Synology C2 Object Storage](https://c2.synology.com/en-global/object-storage/overview) provides a secure, S3-compatible, and cost-effective cloud storage solution without API request, download fees, and deletion penalty. [Synology C2 Object Storage](https://c2.synology.com/en-global/object-storage/overview) provides a secure, S3-compatible, and cost-effective cloud storage solution without API request, download fees, and deletion penalty.