drive: add docs for --fast-list and add to integration tests

This commit is contained in:
Nick Craig-Wood 2018-08-06 21:38:50 +01:00
parent dc5a734522
commit f7c0b2407d
3 changed files with 42 additions and 3 deletions

View File

@ -312,6 +312,45 @@ d) Delete this remote
y/e/d> y
```
### --fast-list ###
This remote supports `--fast-list` which allows you to use fewer
transactions in exchange for more memory. See the [rclone
docs](/docs/#fast-list) for more details.
It does this by combining multiple `list` calls into a single API request.
This works by combining many `'%s' in parents` filters into one expression.
To list the contents of directories a, b and c, the the following requests will be send by the regular `List` function:
```
trashed=false and 'a' in parents
trashed=false and 'b' in parents
trashed=false and 'c' in parents
```
These can now be combined into a single request:
```
trashed=false and ('a' in parents or 'b' in parents or 'c' in parents)
```
The implementation of `ListR` will put up to 50 `parents` filters into one request.
It will use the `--checkers` value to specify the number of requests to run in parallel.
In tests, these batch requests were up to 20x faster than the regular method.
Running the following command against different sized folders gives:
```
rclone lsjson -vv -R --checkers=6 gdrive:folder
```
small folder (220 directories, 700 files):
- without `--fast-list`: 38s
- with `--fast-list`: 10s
large folder (10600 directories, 39000 files):
- without `--fast-list`: 22:05 min
- with `--fast-list`: 58s
### Modified time ###
Google drive stores modification times accurate to 1 ms.

View File

@ -134,7 +134,7 @@ operations more efficient.
| Dropbox | Yes | Yes | Yes | Yes | No [#575](https://github.com/ncw/rclone/issues/575) | No | Yes | Yes | Yes |
| FTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/ncw/rclone/issues/2178) | No |
| Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/ncw/rclone/issues/2178) | No |
| Google Drive | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |
| Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| HTTP | No | No | No | No | No | No | No | No [#2178](https://github.com/ncw/rclone/issues/2178) | No |
| Hubic | Yes † | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/ncw/rclone/issues/2178) | Yes |
| Mega | Yes | No | Yes | Yes | No | No | No | No [#2178](https://github.com/ncw/rclone/issues/2178) | Yes |

View File

@ -45,7 +45,7 @@ var (
{
Name: "TestCryptDrive:",
SubDir: false,
FastList: false,
FastList: true,
},
{
Name: "TestCryptSwift:",
@ -55,7 +55,7 @@ var (
{
Name: "TestDrive:",
SubDir: false,
FastList: false,
FastList: true,
},
{
Name: "TestDropbox:",