diff --git a/docs/content/docs.md b/docs/content/docs.md index a3ddf79f9..d297e817e 100644 --- a/docs/content/docs.md +++ b/docs/content/docs.md @@ -2121,34 +2121,50 @@ there were IO errors`. ### --fast-list ### When doing anything which involves a directory listing (e.g. `sync`, -`copy`, `ls` - in fact nearly every command), rclone normally lists a -directory and processes it before using more directory lists to -process any subdirectories. This can be parallelised and works very -quickly using the least amount of memory. +`copy`, `ls` - in fact nearly every command), rclone has different +strategies to choose from. -However, some remotes have a way of listing all files beneath a -directory in one (or a small number) of transactions. These tend to -be the bucket-based remotes (e.g. S3, B2, GCS, Swift). +The basic strategy is to list one directory and processes it before using +more directory lists to process any subdirectories. This is a mandatory +backend feature, called `List`, which means it is supported by all backends. +This strategy uses small amount of memory, and because it can be parallelised +it is fast for operations involving processing of the list results. -If you use the `--fast-list` flag then rclone will use this method for -listing directories. This will have the following consequences for -the listing: +Some backends provide the support for an alternative strategy, where all +files beneath a directory can be listed in one (or a small number) of +transactions. Rclone supports this alternative strategy through an optional +backend feature called [`ListR`](/overview/#listr). You can see in the storage +system overview documentation's [optional features](/overview/#optional-features) +section which backends it is enabled for (these tend to be the bucket-based +ones, e.g. S3, B2, GCS, Swift). This strategy requires fewer transactions +for highly recursive operations, which is important on backends where this +is charged or heavily rate limited. It may be faster (due to fewer transactions) +or slower (because it can't be parallelized) depending on different parameters, +and may require more memory if rclone has to keep the whole listing in memory. - * It **will** use fewer transactions (important if you pay for them) - * It **will** use more memory. Rclone has to load the whole listing into memory. - * It *may* be faster because it uses fewer transactions - * It *may* be slower because it can't be parallelized +Which listing strategy rclone picks for a given operation is complicated, but +in general it tries to choose the best possible. It will prefer `ListR` in +situations where it doesn't need to store the listed files in memory, e.g. +for unlimited recursive `ls` command variants. In other situations it will +prefer `List`, e.g. for `sync` and `copy`, where it needs to keep the listed +files in memory, and is performing operations on them where parallelization +may be a huge advantage. -rclone should always give identical results with and without -`--fast-list`. +Rclone is not able to take all relevant parameters into account for deciding +the best strategy, and therefore allows you to influence the choice in two ways: +You can stop rclone from using `ListR` by disabling the feature, using the +[--disable](#disable-feature-feature) option (`--disable ListR`), or you can +allow rclone to use `ListR` where it would normally choose not to do so due to +higher memory usage, using the `--fast-list` option. Rclone should always +produce identical results either way. Using `--disable ListR` or `--fast-list` +on a remote which doesn't support `ListR` does nothing, rclone will just ignore +it. -If you pay for transactions and can fit your entire sync listing into -memory then `--fast-list` is recommended. If you have a very big sync -to do then don't use `--fast-list` otherwise you will run out of -memory. - -If you use `--fast-list` on a remote which doesn't support it, then -rclone will just ignore it. +A rule of thumb is that if you pay for transactions and can fit your entire +sync listing into memory, then `--fast-list` is recommended. If you have a +very big sync to do, then don't use `--fast-list`, otherwise you will run out +of memory. Run some tests and compare before you decide, and if in doubt then +just leave the default, let rclone decide, i.e. not use `--fast-list`. ### --timeout=TIME ###