diff --git a/MANUAL.html b/MANUAL.html index 6ab9d478d..d4b683fec 100644 --- a/MANUAL.html +++ b/MANUAL.html @@ -81,7 +81,7 @@

rclone(1) User Manual

Nick Craig-Wood

-

Jun 14, 2024

+

Sep 08, 2024

Rclone syncs your files to cloud storage

rclone logo

@@ -154,7 +154,9 @@
  • Dropbox
  • Enterprise File Fabric
  • Fastmail Files
    • +
  • Files.com
  • FTP
    • +
  • Gofile
  • Google Cloud Storage
  • Google Drive
  • Google Photos
    • @@ -193,6 +195,7 @@
  • pCloud
  • Petabox
  • PikPak
    • +
  • Pixeldrain
  • premiumize.me
  • put.io
  • Proton Drive
    • @@ -441,6 +444,7 @@ go build

    There are other make targets that can be used for more advanced builds, such as cross-compiling for all supported os/architectures, and packaging results into release artifacts. See Makefile and cross-compile.go for details.

    Another alternative method for source installation is to download the source, build and install rclone - all in one operation, as a regular Go package. The source will be stored it in the Go module cache, and the resulting executable will be in your GOPATH bin folder ($(go env GOPATH)/bin, which corresponds to ~/go/bin/rclone by default).

    go install github.com/rclone/rclone@latest
    +

    In some situations, rclone executable size might be too big for deployment in very restricted environments when all backends with large SDKs are included. To limit binary size unused backends can be commented out in backends/all/all.go and unused commands in cmd/all/all.go before building with go build or make

    Ansible installation

    This can be done with Stefan Weichinger's ansible role.

    Instructions

    @@ -510,12 +514,15 @@ go build
  • Digi Storage
  • Dropbox
  • Enterprise File Fabric
    • +
  • Files.com
  • FTP
    • +
  • Gofile
  • Google Cloud Storage
  • Google Drive
  • Google Photos
  • Hasher - to handle checksums for other remotes
  • HDFS
    • +
  • Hetzner Storage Box
  • HiDrive
  • HTTP
  • Internet Archive
    • @@ -533,11 +540,13 @@ go build
  • Oracle Object Storage
  • Pcloud
  • PikPak
    • +
  • Pixeldrain
  • premiumize.me
  • put.io
  • Proton Drive
  • QingStor
  • Quatrix by Maytech
    • +
  • rsync.net
  • Seafile
  • SFTP
  • Sia
    • @@ -576,7 +585,7 @@ rclone sync --interactive /local/path remote:path # syncs /local/path to the rem

    Options

      -h, --help   help for config

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    See Also

    The default number of parallel checks is 8. See the --checkers=N option for more information.

    rclone cryptcheck remote:path cryptedremote:path [flags]
    -

    Options

    +

    Options

          --combined string         Make a combined report of changes to this file
       --differ string           Report all non-matching files to this file
       --error string            Report all files with errors (hashing or reading) to this file
@@ -2417,11 +2487,12 @@ if src is directory
       --missing-on-dst string   Report all files missing from the destination to this file
       --missing-on-src string   Report all files missing from the source to this file
       --one-way                 Check one way only, source files must exist on remote
    -

    Check Options

    -

    Flags used for rclone check.

    +

    Options shared with other commands are described next. See the global flags page for global options not listed here.

    +

    Check Options

    +

    Flags used for check commands

          --max-backlog int   Maximum number of objects in sync or check backlog (default 10000)
    -

    Filter Options

    -

    Flags for filtering directory listings.

    +

    Filter Options

    +

    Flags for filtering directory listings

          --delete-excluded                     Delete files on dest excluded from sync
       --exclude stringArray                 Exclude files matching pattern
       --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
@@ -2444,19 +2515,18 @@ if src is directory
       --metadata-include-from stringArray   Read metadata include patterns from file (use - to read from stdin)
       --min-age Duration                    Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
       --min-size SizeSuffix                 Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
    -

    Listing Options

    -

    Flags for listing directories.

    +

    Listing Options

    +

    Flags for listing directories

          --default-time Time   Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
       --fast-list           Use recursive list if available; uses more memory but fewer transactions
    -

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    See Also

    rclone cryptdecode

    Cryptdecode returns unencrypted file names.

    -

    Synopsis

    -

    rclone cryptdecode returns unencrypted file names when provided with a list of encrypted file names. List limit is 10 items.

    +

    Synopsis

    +

    Returns unencrypted file names when provided with a list of encrypted file names. List limit is 10 items.

    If you supply the --reverse flag, it will return encrypted file names.

    use it like this

    rclone cryptdecode encryptedremote: encryptedfilename1 encryptedfilename2
@@ -2464,99 +2534,31 @@ if src is directory
 rclone cryptdecode --reverse encryptedremote: filename1 filename2

    Another way to accomplish this is by using the rclone backend encode (or decode) command. See the documentation on the crypt overlay for more info.

    rclone cryptdecode encryptedremote: encryptedfilename [flags]
    -

    Options

    +

    Options

      -h, --help      help for cryptdecode
       --reverse   Reverse cryptdecode, encrypts filenames

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    See Also

    rclone deletefile

    Remove a single file from remote.

    -

    Synopsis

    +

    Synopsis

    Remove a single file from remote. Unlike delete it cannot be used to remove a directory and it doesn't obey include/exclude filters - if the specified file exists, it will always be removed.

    rclone deletefile remote:path [flags]
    -

    Options

    +

    Options

      -h, --help   help for deletefile
    -

    Important Options

    -

    Important flags useful for most commands.

    +

    Options shared with other commands are described next. See the global flags page for global options not listed here.

    +

    Important Options

    +

    Important flags useful for most commands

      -n, --dry-run         Do a trial run with no permanent changes
   -i, --interactive     Enable interactive mode
   -v, --verbose count   Print lots more stuff (repeat for more)
    -

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    See Also

    -

    rclone genautocomplete

    -

    Output completion script for a given shell.

    -

    Synopsis

    -

    Generates a shell completion script for rclone. Run with --help to list the supported shells.

    -

    Options

    -
      -h, --help   help for genautocomplete
    -

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    - -

    rclone genautocomplete bash

    -

    Output bash completion script for rclone.

    -

    Synopsis

    -

    Generates a bash shell autocompletion script for rclone.

    -

    This writes to /etc/bash_completion.d/rclone by default so will probably need to be run with sudo or as root, e.g.

    -
    sudo rclone genautocomplete bash
    -

    Logout and login again to use the autocompletion scripts, or source them directly

    -
    . /etc/bash_completion
    -

    If you supply a command line argument the script will be written there.

    -

    If output_file is "-", then the output will be written to stdout.

    -
    rclone genautocomplete bash [output_file] [flags]
    -

    Options

    -
      -h, --help   help for bash
    -

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    - -

    rclone genautocomplete fish

    -

    Output fish completion script for rclone.

    -

    Synopsis

    -

    Generates a fish autocompletion script for rclone.

    -

    This writes to /etc/fish/completions/rclone.fish by default so will probably need to be run with sudo or as root, e.g.

    -
    sudo rclone genautocomplete fish
    -

    Logout and login again to use the autocompletion scripts, or source them directly

    -
    . /etc/fish/completions/rclone.fish
    -

    If you supply a command line argument the script will be written there.

    -

    If output_file is "-", then the output will be written to stdout.

    -
    rclone genautocomplete fish [output_file] [flags]
    -

    Options

    -
      -h, --help   help for fish
    -

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    - -

    rclone genautocomplete zsh

    -

    Output zsh completion script for rclone.

    -

    Synopsis

    -

    Generates a zsh autocompletion script for rclone.

    -

    This writes to /usr/share/zsh/vendor-completions/_rclone by default so will probably need to be run with sudo or as root, e.g.

    -
    sudo rclone genautocomplete zsh
    -

    Logout and login again to use the autocompletion scripts, or source them directly

    -
    autoload -U compinit && compinit
    -

    If you supply a command line argument the script will be written there.

    -

    If output_file is "-", then the output will be written to stdout.

    -
    rclone genautocomplete zsh [output_file] [flags]
    -

    Options

    -
      -h, --help   help for zsh
    -

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    -

    rclone gendocs

    Output markdown docs for rclone to the directory supplied.

    Synopsis

    @@ -2565,7 +2567,7 @@ rclone cryptdecode --reverse encryptedremote: filename1 filename2

    Options

      -h, --help   help for gendocs

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    See Also

    @@ -2576,38 +2578,38 @@ rclone cryptdecode --reverse encryptedremote: filename1 filename2

    Installation on Linux

    1. Skip this step if your version of git-annex is 10.20240430 or newer. Otherwise, you must create a symlink somewhere on your PATH with a particular name. This symlink helps git-annex tell rclone it wants to run the "gitannex" subcommand.

      -
      # Create the helper symlink in "$HOME/bin".
-ln -s "$(realpath rclone)" "$HOME/bin/git-annex-remote-rclone-builtin"
-
-# Verify the new symlink is on your PATH.
-which git-annex-remote-rclone-builtin
      2. +
      # Create the helper symlink in "$HOME/bin".
+ln -s "$(realpath rclone)" "$HOME/bin/git-annex-remote-rclone-builtin"
+
+# Verify the new symlink is on your PATH.
+which git-annex-remote-rclone-builtin

    2. Add a new remote to your git-annex repo. This new remote will connect git-annex with the rclone gitannex subcommand.

      Start by asking git-annex to describe the remote's available configuration parameters.

      -
      # If you skipped step 1:
-git annex initremote MyRemote type=rclone --whatelse
-
-# If you created a symlink in step 1:
-git annex initremote MyRemote type=external externaltype=rclone-builtin --whatelse
      +
      # If you skipped step 1:
+git annex initremote MyRemote type=rclone --whatelse
+
+# If you created a symlink in step 1:
+git annex initremote MyRemote type=external externaltype=rclone-builtin --whatelse

      NOTE: If you're porting an existing git-annex-remote-rclone remote to use rclone gitannex, you can probably reuse the configuration parameters verbatim without renaming them. Check parameter synonyms with --whatelse as shown above.

      The following example creates a new git-annex remote named "MyRemote" that will use the rclone remote named "SomeRcloneRemote". That rclone remote must be one configured in your rclone.conf file, which can be located with rclone config file.

      -
      git annex initremote MyRemote         \
-    type=external                     \
-    externaltype=rclone-builtin       \
-    encryption=none                   \
-    rcloneremotename=SomeRcloneRemote \
-    rcloneprefix=git-annex-content    \
-    rclonelayout=nodir
      3. +
      git annex initremote MyRemote         \
+    type=external                     \
+    externaltype=rclone-builtin       \
+    encryption=none                   \
+    rcloneremotename=SomeRcloneRemote \
+    rcloneprefix=git-annex-content    \
+    rclonelayout=nodir

    3. Before you trust this command with your precious data, be sure to test the remote. This command is very new and has not been tested on many rclone backends. Caveat emptor!

      -
      git annex testremote MyRemote
      4. +
      git annex testremote MyRemote

    Happy annexing!

    rclone gitannex [flags]

    Options

      -h, --help   help for gitannex

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    See Also

    @@ -2636,8 +2638,9 @@ Supported hashes are: --download Download the file and hash it locally; if this flag is not specified, the hash is requested from the remote -h, --help help for hashsum --output-file string Output hashsums to a file rather than the terminal -

    Filter Options

    -

    Flags for filtering directory listings.

    +

    Options shared with other commands are described next. See the global flags page for global options not listed here.

    +

    Filter Options

    +

    Flags for filtering directory listings

          --delete-excluded                     Delete files on dest excluded from sync
       --exclude stringArray                 Exclude files matching pattern
       --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
@@ -2660,19 +2663,18 @@ Supported hashes are:
       --metadata-include-from stringArray   Read metadata include patterns from file (use - to read from stdin)
       --min-age Duration                    Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
       --min-size SizeSuffix                 Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
    -

    Listing Options

    -

    Flags for listing directories.

    +

    Listing Options

    +

    Flags for listing directories

          --default-time Time   Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
       --fast-list           Use recursive list if available; uses more memory but fewer transactions
    -

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    See Also

    rclone link

    Generate public link to file/folder.

    Synopsis

    -

    rclone link will create, retrieve or remove a public link to the given file or folder.

    +

    Create, retrieve or remove a public link to the given file or folder.

    rclone link remote:path/to/file
 rclone link remote:path/to/folder/
 rclone link --unlink remote:path/to/folder/
@@ -2686,21 +2688,28 @@ rclone link --expire 1d remote:path/to/file
    -h, --help help for link --unlink Remove existing public link to file/folder

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    See Also

    rclone listremotes

    List all the remotes in the config file and defined in environment variables.

    Synopsis

    -

    rclone listremotes lists all the available remotes from the config file.

    -

    When used with the --long flag it lists the types and the descriptions too.

    -
    rclone listremotes [flags]
    +

    Lists all the available remotes from the config file, or the remotes matching an optional filter.

    +

    Prints the result in human-readable format by default, and as a simple list of remote names, or if used with flag --long a tabular format including the remote names, types and descriptions. Using flag --json produces machine-readable output instead, which always includes all attributes - including the source (file or environment).

    +

    Result can be filtered by a filter argument which applies to all attributes, and/or filter flags specific for each attribute. The values must be specified according to regular rclone filtering pattern syntax.

    +
    rclone listremotes [<filter>] [flags]

    Options

    -
      -h, --help   help for listremotes
-      --long   Show the type and the description as well as names
    +
          --description string   Filter remotes by description
+  -h, --help                 help for listremotes
+      --json                 Format output as JSON
+      --long                 Show type and description in addition to name
+      --name string          Filter remotes by name
+      --order-by string      Instructions on how to order the result, e.g. 'type,name=descending'
+      --source string        Filter remotes by source, e.g. 'file' or 'environment'
+      --type string          Filter remotes by type

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    See Also

    @@ -2797,8 +2806,9 @@ rclone lsf remote:path --format pt --time-format max -R, --recursive Recurse into the listing -s, --separator string Separator for the items in the format (default ";") -t, --time-format string Specify a custom time format, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05) -

    Filter Options

    -

    Flags for filtering directory listings.

    +

    Options shared with other commands are described next. See the global flags page for global options not listed here.

    +

    Filter Options

    +

    Flags for filtering directory listings

          --delete-excluded                     Delete files on dest excluded from sync
       --exclude stringArray                 Exclude files matching pattern
       --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
@@ -2821,12 +2831,11 @@ rclone lsf remote:path --format pt --time-format max
    --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -

    Listing Options

    -

    Flags for listing directories.

    +

    Listing Options

    +

    Flags for listing directories

          --default-time Time   Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
       --fast-list           Use recursive list if available; uses more memory but fewer transactions
    -

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    See Also

    @@ -2834,7 +2843,7 @@ rclone lsf remote:path --format pt --time-format max

    List directories and objects in the path in JSON format.

    Synopsis

    List directories and objects in the path in JSON format.

    -

    The output is an array of Items, where each Item looks like this

    +

    The output is an array of Items, where each Item looks like this:

    {
   "Hashes" : {
      "SHA-1" : "f572d396fae9206628714fb2ce00f72e94f2258f",
@@ -2854,18 +2863,28 @@ rclone lsf remote:path --format pt --time-format max
    "Size" : 6, "Tier" : "hot", } -

    If --hash is not specified, the Hashes property will be omitted. The types of hash can be specified with the --hash-type parameter (which may be repeated). If --hash-type is set then it implies --hash.

    -

    If --no-modtime is specified then ModTime will be blank. This can speed things up on remotes where reading the ModTime takes an extra request (e.g. s3, swift).

    -

    If --no-mimetype is specified then MimeType will be blank. This can speed things up on remotes where reading the MimeType takes an extra request (e.g. s3, swift).

    -

    If --encrypted is not specified the Encrypted will be omitted.

    -

    If --dirs-only is not specified files in addition to directories are returned

    -

    If --files-only is not specified directories in addition to the files will be returned.

    -

    If --metadata is set then an additional Metadata key will be returned. This will have metadata in rclone standard format as a JSON object.

    -

    if --stat is set then a single JSON blob will be returned about the item pointed to. This will return an error if the item isn't found. However on bucket based backends (like s3, gcs, b2, azureblob etc) if the item isn't found it will return an empty directory as it isn't possible to tell empty directories from missing directories there.

    +

    The exact set of properties included depends on the backend:

    + +

    Different options may also affect which properties are included:

    + +

    The default is to list directories and files/objects, but this can be changed with the following options:

    + +

    If --stat is set then the the output is not an array of items, but instead a single JSON blob will be returned about the item pointed to. This will return an error if the item isn't found, however on bucket based backends (like s3, gcs, b2, azureblob etc) if the item isn't found it will return an empty directory, as it isn't possible to tell empty directories from missing directories there.

    The Path field will only show folders below the remote path being listed. If "remote:path" contains the file "subfolder/file.txt", the Path for "file.txt" will be "subfolder/file.txt", not "remote:path/subfolder/file.txt". When used without --recursive the Path will always be the same as Name.

    -

    If the directory is a bucket in a bucket-based backend, then "IsBucket" will be set to true. This key won't be present unless it is "true".

    The time is in RFC3339 format with up to nanosecond precision. The number of decimal digits in the seconds will depend on the precision that the remote can hold the times, so if times are accurate to the nearest millisecond (e.g. Google Drive) then 3 digits will always be shown ("2017-05-31T16:15:57.034+01:00") whereas if the times are accurate to the nearest second (Dropbox, Box, WebDav, etc.) no digits will be shown ("2017-05-31T16:15:57+01:00").

    -

    The whole output can be processed as a JSON blob, or alternatively it can be processed line by line as each item is written one to a line.

    +

    The whole output can be processed as a JSON blob, or alternatively it can be processed line by line as each item is written on individual lines (except with --stat).

    Any of the filtering options can be applied to this command.

    There are several related list commands

    -

    If none of these option actually end up providing rclone with AWS credentials then S3 interaction will be non-authenticated (see below).

    +

    With env_auth = true rclone (which uses the SDK for Go v2) should support all authentication methods that the aws CLI tool does and the other AWS SDKs.

    +

    If none of these option actually end up providing rclone with AWS credentials then S3 interaction will be non-authenticated (see the anonymous access section for more info).

    S3 Permissions

    When using the sync subcommand of rclone the following minimum permissions are required to be available on the bucket being written to:

    --s3-sts-endpoint

    -

    Endpoint for STS.

    +

    Endpoint for STS (deprecated).

    Leave blank if using AWS to use the default endpoint for the region.

    Properties:

    +

    --s3-sdk-log-mode

    +

    Set to debug the SDK

    +

    This can be set to a comma separated list of the following functions:

    + +

    Use Off to disable and All to set all log levels. You will need to use -vv to see the debug level logs.

    +

    Properties:

    +

    --s3-description

    Description of the remote.

    Properties:

    @@ -15420,13 +16028,14 @@ Windows: "%USERPROFILE%\.aws\credentials"

    See the backend command for more info on how to pass options and arguments.

    These can be run on a running backend using the rc command backend/command.

    restore

    -

    Restore objects from GLACIER to normal storage

    +

    Restore objects from GLACIER or INTELLIGENT-TIERING archive tier

    rclone backend restore remote: [options] [<arguments>+]
    -

    This command can be used to restore one or more objects from GLACIER to normal storage.

    +

    This command can be used to restore one or more objects from GLACIER to normal storage or from INTELLIGENT-TIERING Archive Access / Deep Archive Access tier to the Frequent Access tier.

    Usage Examples:

    rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS
 rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS
-rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS
    +rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS +rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY

    This flag also obeys the filters. Test first with --interactive/-i or --dry-run flags

    rclone --interactive backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1

    All the objects shown will be marked for restore, then

    @@ -15445,13 +16054,13 @@ rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS

    Options:

    restore-status

    -

    Show the restore status for objects being restored from GLACIER to normal storage

    +

    Show the restore status for objects being restored from GLACIER or INTELLIGENT-TIERING storage

    rclone backend restore-status remote: [options] [<arguments>+]
    -

    This command can be used to show the status for objects being restored from GLACIER to normal storage.

    +

    This command can be used to show the status for objects being restored from GLACIER to normal storage or from INTELLIGENT-TIERING Archive Access / Deep Archive Access tier to the Frequent Access tier.

    Usage Examples:

    rclone backend restore-status s3:bucket/path/to/object
 rclone backend restore-status s3:bucket/path/to/directory
@@ -15476,6 +16085,15 @@ rclone backend restore-status -o all s3:bucket/path/to/directory
    "RestoreExpiryDate": "2023-09-06T12:29:19+01:00" }, "StorageClass": "DEEP_ARCHIVE" + }, + { + "Remote": "test.gz", + "VersionID": null, + "RestoreStatus": { + "IsRestoreInProgress": true, + "RestoreExpiryDate": "null" + }, + "StorageClass": "INTELLIGENT_TIERING" } ]

    Options:

    @@ -15546,23 +16164,16 @@ rclone rc backend/command command=set fs=s3: -o session_token=X -o access_key_id

    The option keys are named as they are in the config file.

    This rebuilds the connection to the s3 backend when it is called with the new parameters. Only new parameters need be passed as the values will default to those currently in use.

    It doesn't return anything.

    -

    Anonymous access to public buckets

    +

    Anonymous access to public buckets

    If you want to use rclone to access a public bucket, configure with a blank access_key_id and secret_access_key. Your config should end up looking like this:

    [anons3]
 type = s3
-provider = AWS
-env_auth = false
-access_key_id =
-secret_access_key =
-region = us-east-1
-endpoint =
-location_constraint =
-acl = private
-server_side_encryption =
-storage_class =
    +provider = AWS

    Then use it as normal with the name of the public bucket, e.g.

    rclone lsd anons3:1000genomes

    You will be able to list and copy data but not upload it.

    +

    You can also do this entirely on the command line

    +
    rclone lsd :s3,provider=AWS:1000genomes

    Providers

    AWS S3

    This is the provider used as main example and described in the configuration section above.

    @@ -15690,6 +16301,7 @@ endpoint = https://ACCOUNT_ID.r2.cloudflarestorage.com acl = private

    Now run rclone lsf r2: to see your buckets and rclone lsf r2:bucket to look within a bucket.

    For R2 tokens with the "Object Read & Write" permission, you may also need to add no_check_bucket = true for object uploads to work correctly.

    +

    Note that Cloudflare decompresses files uploaded with Content-Encoding: gzip by default which is a deviation from what AWS does. If this is causing a problem then upload the files with --header-upload "Cache-Control: no-transform"

    Dreamhost

    Dreamhost DreamObjects is an object storage system based on CEPH.

    To use rclone with Dreamhost, configure as above but leave the region blank and set the endpoint. You should end up with something like this in your config:

    @@ -17710,7 +18322,7 @@ cos s3

    For Netease NOS configure as per the configurator rclone config setting the provider Netease. This will automatically set force_path_style = false which is necessary for it to run properly.

    Petabox

    Here is an example of making a Petabox configuration. First run:

    -
    rclone config
    +
    rclone config

    This will guide you through an interactive setup process.

    No remotes found, make a new one?
 n) New remote
@@ -17921,6 +18533,11 @@ y/n> n

    Use the the native protocol to take advantage of client-side encryption as well as to achieve the best possible download performance. Uploads will be erasure-coded locally, thus a 1gb upload will result in 2.68gb of data being uploaded to storage nodes across the network.

    Use this backend and the S3 compatible Hosted Gateway to increase upload performance and reduce the load on your systems and network. Uploads will be encrypted and erasure-coded server-side, thus a 1GB upload will result in only in 1GB of data being uploaded to storage nodes across the network.

    For more detailed comparison please check the documentation of the storj backend.

    +

    Memory usage {memory}

    +

    The most common cause of rclone using lots of memory is a single directory with millions of files in. Despite s3 not really having the concepts of directories, rclone does the sync on a directory by directory basis to be compatible with normal filing systems.

    +

    Rclone loads each directory into memory as rclone objects. Each rclone object takes 0.5k-1k of memory, so approximately 1GB per 1,000,000 files, and the sync for that directory does not begin until it is entirely loaded in memory. So the sync can take a long time to start for large directories.

    +

    To sync a directory with 100,000,000 files in you would need approximately 100 GB of memory. At some point the amount of memory becomes difficult to provide so there is a workaround for this which involves a bit of scripting.

    +

    At some point rclone will gain a sync mode which is effectively this workaround but built in to rclone.

    Limitations

    rclone about is not supported by the S3 backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

    See List of backends that do not support rclone about and rclone about

    @@ -18070,12 +18687,13 @@ key> 0123456789abcdef0123456789abcdef0123456789 Endpoint for the service - leave blank normally. endpoint> Remote config --------------------- -[remote] -account = 123456789abc -key = 0123456789abcdef0123456789abcdef0123456789 -endpoint = --------------------- +Configuration complete. +Options: +- type: b2 +- account: 123456789abc +- key: 0123456789abcdef0123456789abcdef0123456789 +- endpoint: +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -18129,7 +18747,9 @@ y/e/d> y

    Backblaze recommends that you do lots of transfers simultaneously for maximum speed. In tests from my SSD equipped laptop the optimum setting is about --transfers 32 though higher numbers may be used for a slight speed improvement. The optimum number for you may vary depending on your hardware, how big the files are, how much you want to load your computer, etc. The default of --transfers 4 is definitely too low for Backblaze B2 though.

    Note that uploading big files (bigger than 200 MiB by default) will use a 96 MiB RAM buffer by default. There can be at most --transfers of these in use at any moment, so this sets the upper limit on the memory used.

    Versions

    -

    When rclone uploads a new version of a file it creates a new version of it. Likewise when you delete a file, the old version will be marked hidden and still be available. Conversely, you may opt in to a "hard delete" of files with the --b2-hard-delete flag which would permanently remove the file instead of hiding it.

    +

    The default setting of B2 is to keep old versions of files. This means when rclone uploads a new version of a file it creates a new version of it. Likewise when you delete a file, the old version will be marked hidden and still be available.

    +

    Whether B2 keeps old versions of files or not can be adjusted on a per bucket basis using the "Lifecycle settings" on the B2 control panel or when creating the bucket using the --b2-lifecycle flag or after creation using the rclone backend lifecycle command.

    +

    You may opt in to a "hard delete" of files with the --b2-hard-delete flag which permanently removes files on deletion instead of hiding them.

    Old versions of files, where available, are visible using the --b2-versions flag.

    It is also possible to view a bucket as it was at a certain point in time, using the --b2-version-at flag. This will show the file versions as they were at that time, showing files that have been deleted afterwards, and hiding files that were created since.

    If you wish to remove all the old versions, and unfinished large file uploads, then you can use the rclone cleanup remote:bucket command which will delete all the old versions of files, leaving the current ones intact. You can also supply a path and only old versions under that path will be deleted, e.g. rclone cleanup remote:bucket/path/to/stuff.

    @@ -18525,12 +19145,13 @@ If your browser doesn't open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -client_id = -client_secret = -token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} --------------------- +Configuration complete. +Options: +- type: box +- client_id: +- client_secret: +- token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -18579,11 +19200,11 @@ e/n/d/r/c/s/q> e Choose a number from below, or type in an existing value 1 > remote remote> remote --------------------- -[remote] -type = box -token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"} --------------------- +Configuration complete. +Options: +- type: box +- token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"} +Keep this "remote" remote? Edit remote Value "client_id" = "" Edit? (y/n)> @@ -18611,11 +19232,11 @@ If your browser doesn't open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = box -token = {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"} --------------------- +Configuration complete. +Options: +- type: box +- token: {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -19638,12 +20259,12 @@ If your browser doesn't open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = sharefile -endpoint = https://XXX.sharefile.com -token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} --------------------- +Configuration complete. +Options: +- type: sharefile +- endpoint: https://XXX.sharefile.com +- token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -20532,11 +21153,11 @@ Embedded spaces can be added using quotes "dir=remote:path with space" "dir2=remote2:path with space" Enter a fs.SpaceSepList value. upstreams> images=s3:imagesbucket files=drive:important/files --------------------- -[remote] -type = combine -upstreams = images=s3:imagesbucket files=drive:important/files --------------------- +Configuration complete. +Options: +- type: combine +- upstreams: images=s3:imagesbucket files=drive:important/files +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -20617,12 +21238,13 @@ Remote config Please visit: https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX --------------------- -[remote] -app_key = -app_secret = -token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX --------------------- +Configuration complete. +Options: +- type: dropbox +- app_key: +- app_secret: +- token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -20862,7 +21484,7 @@ y/e/d> y

    --dropbox-batch-size

    Max number of files in upload batch.

    This sets the batch size of files to upload. It has to be less than 1000.

    -

    By default this is 0 which means rclone which calculate the batch size depending on the setting of batch_mode.

    +

    By default this is 0 which means rclone will calculate the batch size depending on the setting of batch_mode.

    +

    Files.com

    +

    Files.com is a cloud storage service that provides a secure and easy way to store and share files.

    +

    The initial setup for filescom involves authenticating with your Files.com account. You can do this by providing your site subdomain, username, and password. Alternatively, you can authenticate using an API Key from Files.com. rclone config walks you through it.

    +

    Configuration

    +

    Here is an example of how to make a remote called remote. First run:

    +
    rclone config
    +

    This will guide you through an interactive setup process:

    +
    No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+
+Enter name for new remote.
+name> remote
+
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+XX / Files.com
+  \ "filescom"
+[snip]
+Storage> filescom
+
+Option site.
+Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com)
+Enter a value. Press Enter to leave empty.
+site> mysite
+
+Option username.
+The username used to authenticate with Files.com.
+Enter a value. Press Enter to leave empty.
+username> user
+
+Option password.
+The password used to authenticate with Files.com.
+Choose an alternative below. Press Enter for the default (n).
+y) Yes, type in my own password
+g) Generate random password
+n) No, leave this optional password blank (default)
+y/g/n> y
+Enter the password:
+password:
+Confirm the password:
+password:
+
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+
+Configuration complete.
+Options:
+- type: filescom
+- site: mysite
+- username: user
+- password: *** ENCRYPTED ***
+Keep this "remote" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
    +

    Once configured you can use rclone.

    +

    See all files in the top level:

    +
    rclone lsf remote:
    +

    Make a new directory in the root:

    +
    rclone mkdir remote:dir
    +

    Recursively List the contents:

    +
    rclone ls remote:
    +

    Sync /home/local/directory to the remote directory, deleting any excess files in the directory.

    +
    rclone sync --interactive /home/local/directory remote:dir
    +

    Standard options

    +

    Here are the Standard options specific to filescom (Files.com).

    +

    --filescom-site

    +

    Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com).

    +

    Properties:

    + +

    --filescom-username

    +

    The username used to authenticate with Files.com.

    +

    Properties:

    + +

    --filescom-password

    +

    The password used to authenticate with Files.com.

    +

    NB Input to this must be obscured - see rclone obscure.

    +

    Properties:

    + +

    Advanced options

    +

    Here are the Advanced options specific to filescom (Files.com).

    +

    --filescom-api-key

    +

    The API key used to authenticate with Files.com.

    +

    Properties:

    + +

    --filescom-encoding

    +

    The encoding for the backend.

    +

    See the encoding section in the overview for more info.

    +

    Properties:

    + +

    --filescom-description

    +

    Description of the remote.

    +

    Properties:

    +

    FTP

    FTP is the File Transfer Protocol. Rclone FTP support is provided using the github.com/jlaffaye/ftp package.

    Limitations of Rclone's FTP backend

    Paths are specified as remote:path. If the path does not begin with a / it is relative to the home directory of the user. An empty path remote: refers to the user's home directory.

    -

    Configuration

    +

    Configuration

    To create an FTP configuration named remote, run

    rclone config

    Rclone config guides you through an interactive setup process. A minimal rclone FTP remote definition only requires host, username and password. For an anonymous FTP server, see below.

    @@ -21180,12 +21934,12 @@ Use FTP over TLS (Explicit) Enter a boolean value (true or false). Press Enter for the default ("false"). explicit_tls> Remote config --------------------- -[remote] -type = ftp -host = ftp.example.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: ftp +- host: ftp.example.com +- pass: *** ENCRYPTED *** +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -21247,7 +22001,7 @@ rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=IXs2wc8OJOz7SYLBk47

    This backend's interactive configuration wizard provides a selection of sensible encoding settings for major FTP servers: ProFTPd, PureFTPd, VsFTPd. Just hit a selection number when prompted.

    -

    Standard options

    +

    Standard options

    Here are the Standard options specific to ftp (FTP).

    --ftp-host

    FTP host to connect to.

    @@ -21307,7 +22061,7 @@ rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=IXs2wc8OJOz7SYLBk47
  • Type: bool
  • Default: false
    • -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to ftp (FTP).

    --ftp-concurrency

    Maximum number of FTP simultaneous connections, 0 for unlimited.

    @@ -21497,9 +22251,224 @@ rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=IXs2wc8OJOz7SYLBk47

    File modification time (timestamps) is supported to 1 second resolution for major FTP servers: ProFTPd, PureFTPd, VsFTPd, and FileZilla FTP server. The VsFTPd server has non-standard implementation of time related protocol commands and needs a special configuration setting: writing_mdtm = true.

    Support for precise file time with other FTP servers varies depending on what protocol extensions they advertise. If all the MLSD, MDTM and MFTM extensions are present, rclone will use them together to provide precise time. Otherwise the times you see on the FTP server through rclone are those of the last file upload.

    You can use the following command to check whether rclone can use precise time with your FTP server: rclone backend features your_ftp_remote: (the trailing colon is important). Look for the number in the line tagged by Precision designating the remote time precision expressed as nanoseconds. A value of 1000000000 means that file time precision of 1 second is available. A value of 3153600000000000000 (or another large number) means "unsupported".

    +

    Gofile

    +

    Gofile is a content storage and distribution platform. Its aim is to provide as much service as possible for free or at a very low price.

    +

    The initial setup for Gofile involves logging in to the web interface and going to the "My Profile" section. Copy the "Account API token" for use in the config file.

    +

    Note that if you wish to connect rclone to Gofile you will need a premium account.

    +

    Configuration

    +

    Here is an example of how to make a remote called remote. First run:

    +
     rclone config
    +

    This will guide you through an interactive setup process:

    +
    No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+
+Enter name for new remote.
+name> remote
+
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+XX / Gofile
+   \ (gofile)
+Storage> gofile
+
+Option access_token.
+API Access token
+You can get this from the web control panel.
+Enter a value. Press Enter to leave empty.
+access_token> YOURACCESSTOKEN
+
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+
+Configuration complete.
+Options:
+- type: gofile
+- access_token: YOURACCESSTOKEN
+Keep this "remote" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
    +

    Once configured you can then use rclone like this,

    +

    List directories and files in the top level of your Gofile

    +
    rclone lsf remote:
    +

    To copy a local directory to an Gofile directory called backup

    +
    rclone copy /home/source remote:backup
    +

    Modification times and hashes

    +

    Gofile supports modification times with a resolution of 1 second.

    +

    Gofile supports MD5 hashes, so you can use the --checksum flag.

    +

    Restricted filename characters

    +

    In addition to the default restricted characters set the following characters are also replaced:

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    CharacterValueReplacement
    !0x21！
    "0x22＂
    *0x2A＊
    :0x3A：
    <0x3C＜
    >0x3E＞
    ?0x3F？
    \0x5C＼
    |0x7C｜
    +

    File names can also not start or end with the following characters. These only get replaced if they are the first or last character in the name:

    + + + + + + + + + + + + + + + +
    CharacterValueReplacement
    .0x2E．
    +

    Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

    + +

    Gofile supports rclone link to make public links to files or directories. If you specify a directory it will download as a zip file. You can use the --expire flag to specify the time the link should be valid. Note that rclone link --unlink removes all the public links for a file.

    +

    Root folder ID

    +

    You can set the root_folder_id for rclone. This is the directory (identified by its Folder ID) that rclone considers to be the root of your Gofile drive.

    +

    Normally you will leave this blank and rclone will determine the correct root to use itself and fill in the value in the config file.

    +

    However you can set this to restrict rclone to a specific folder hierarchy.

    +

    In order to do this you will have to find the Folder ID of the directory you wish rclone to display.

    +

    You can do this with rclone

    +
    $ rclone lsf -Fip --dirs-only remote:
+d6341f53-ee65-4f29-9f59-d11e8070b2a0;Files/
+f4f5c9b8-6ece-478b-b03e-4538edfe5a1c;Photos/
+d50e356c-29ca-4b27-a3a7-494d91026e04;Videos/
    +

    The ID to use is the part before the ; so you could set

    +
    root_folder_id = d6341f53-ee65-4f29-9f59-d11e8070b2a0
    +

    To restrict rclone to the Files directory.

    +

    Standard options

    +

    Here are the Standard options specific to gofile (Gofile).

    +

    --gofile-access-token

    +

    API Access token

    +

    You can get this from the web control panel.

    +

    Properties:

    + +

    Advanced options

    +

    Here are the Advanced options specific to gofile (Gofile).

    +

    --gofile-root-folder-id

    +

    ID of the root folder

    +

    Leave this blank normally, rclone will fill it in automatically.

    +

    If you want rclone to be restricted to a particular folder you can fill it in - see the docs for more info.

    +

    Properties:

    + +

    --gofile-account-id

    +

    Account ID

    +

    Leave this blank normally, rclone will fill it in automatically.

    +

    Properties:

    + +

    --gofile-list-chunk

    +

    Number of items to list in each call

    +

    Properties:

    + +

    --gofile-encoding

    +

    The encoding for the backend.

    +

    See the encoding section in the overview for more info.

    +

    Properties:

    + +

    --gofile-description

    +

    Description of the remote.

    +

    Properties:

    + +

    Limitations

    +

    Gofile only supports filenames up to 255 characters in length, where a character is a unicode character.

    +

    Directories should not be cached for more than 24h otherwise files in the directory may not be downloadable. In practice this means when using a VFS based rclone command such as rclone mount you should make sure --dir-cache-time is less than 24h.

    +

    Note that Gofile is currently limited to a total of 100,000 items. If you attempt to upload more than that you will get error-limit-100000. This limit may be lifted in the future.

    +

    Duplicated files

    +

    Gofile is capable of having files with duplicated file names. For instance two files called hello.txt in the same directory.

    +

    Rclone cannot sync that to a normal file system but it can be fixed with the rclone dedupe command.

    +

    Duplicated files cause problems with the syncing and you will see messages in the log about duplicates.

    +

    Use rclone dedupe to fix duplicated files.

    Google Cloud Storage

    Paths are specified as remote:bucket (or remote: for the lsd command.) You may put subdirectories in too, e.g. remote:bucket/path/to/dir.

    -

    Configuration

    +

    Configuration

    The initial setup for google cloud storage involves getting a token from Google Cloud Storage which you need to do in your browser. rclone config walks you through it.

    Here is an example of how to make a remote called remote. First run:

     rclone config
    @@ -21610,16 +22579,16 @@ If your browser doesn't open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = google cloud storage -client_id = -client_secret = -token = {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null} -project_number = 12345678 -object_acl = private -bucket_acl = private --------------------- +Configuration complete. +Options: +- type: google cloud storage +- client_id: +- client_secret: +- token: {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null} +- project_number: 12345678 +- object_acl: private +- bucket_acl: private +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -21639,7 +22608,7 @@ y/e/d> y

    You can set up rclone with Google Cloud Storage in an unattended mode, i.e. not tied to a specific end-user Google account. This is useful when you want to synchronise files onto machines that don't have actively logged-in users, for example build machines.

    To get credentials for Google Cloud Platform IAM Service Accounts, please head to the Service Account section of the Google Developer Console. Service Accounts behave just like normal User permissions in Google Cloud Storage ACLs, so you can limit their access (e.g. make them read only). After creating an account, a JSON file containing the Service Account's credentials will be downloaded onto your machines. These credentials are what rclone will use for authentication.

    To use a Service Account instead of OAuth2 token flow, enter the path to your Service Account credentials at the service_account_file prompt and rclone won't use the browser based authentication flow. If you'd rather stuff the contents of the credentials file into the rclone config file, you can set service_account_credentials with the actual contents of the file instead, or set the equivalent environment variable.

    -

    Anonymous Access

    +

    Anonymous Access

    For downloads of objects that permit public access you can configure rclone to use anonymous access by setting anonymous to true. With unauthorized access you can't write or create files but only read or list those buckets and objects that have public read access.

    Application Default Credentials

    If no other source of credentials is provided, rclone will fall back to Application Default Credentials this is useful both when you already have configured authentication for your developer account, or in production when running on a google compute host. Note that if running in docker, you may need to run additional commands on your google compute machine - see this page.

    @@ -21663,7 +22632,7 @@ y/e/d> y

    Google Cloud Storage stores md5sum natively. Google's gsutil tool stores modification time with one-second precision as goog-reserved-file-mtime in file metadata.

    To ensure compatibility with gsutil, rclone stores modification time in 2 separate metadata entries. mtime uses RFC3339 format with one-nanosecond precision. goog-reserved-file-mtime uses Unix timestamp format with one-second precision. To get modification time from object metadata, rclone reads the metadata in the following order: mtime, goog-reserved-file-mtime, object updated time.

    Note that rclone's default modify window is 1ns. Files uploaded by gsutil only contain timestamps with one-second precision. If you use rclone to sync files previously uploaded by gsutil, rclone will attempt to update modification time for all these files. To avoid these possibly unnecessary updates, use --modify-window 1s.

    -

    Restricted filename characters

    +

    Restricted filename characters

    @@ -21696,7 +22665,7 @@ y/e/d> y

    Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

    -

    Standard options

    +

    Standard options

    Here are the Standard options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)).

    --gcs-client-id

    OAuth Client Id.

    @@ -22081,7 +23050,7 @@ y/e/d> y -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)).

    --gcs-token

    OAuth Access Token as a JSON blob.

    @@ -22172,13 +23141,13 @@ y/e/d> y
  • Type: string
  • Required: false
    • -

    Limitations

    +

    Limitations

    rclone about is not supported by the Google Cloud Storage backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

    See List of backends that do not support rclone about and rclone about

    Google Drive

    Paths are specified as drive:path

    Drive paths may be as deep as required, e.g. drive:directory/subdirectory.

    -

    Configuration

    +

    Configuration

    The initial setup for drive involves getting a token from Google drive which you need to do in your browser. rclone config walks you through it.

    Here is an example of how to make a remote called remote. First run:

     rclone config
    @@ -22237,15 +23206,16 @@ Configure this as a Shared Drive (Team Drive)? y) Yes n) No y/n> n --------------------- -[remote] -client_id = -client_secret = -scope = drive -root_folder_id = -service_account_file = -token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"} --------------------- +Configuration complete. +Options: +type: drive +- client_id: +- client_secret: +- scope: drive +- root_folder_id: +- service_account_file: +- token: {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -22277,7 +23247,7 @@ y/e/d> y

    This gives rclone its own private area to store files. Rclone will not be able to see any other files on your drive and you won't be able to see rclone's files from the web interface either.

    drive.metadata.readonly

    This allows read only access to file names only. It does not allow rclone to download or upload data, or rename or delete files or directories.

    -

    Root folder ID

    +

    Root folder ID

    This option has been moved to the advanced section. You can set the root_folder_id for rclone. This is the directory (identified by its Folder ID) that rclone considers to be the root of your drive.

    Normally you will leave this blank and rclone will determine the correct root to use itself.

    However you can set this to restrict rclone to a specific folder hierarchy or to access data within the "Computers" tab on the drive web interface (where files from Google's Backup and Sync desktop program go).

    @@ -22289,38 +23259,44 @@ y/e/d> y

    Service Account support

    You can set up rclone with Google Drive in an unattended mode, i.e. not tied to a specific end-user Google account. This is useful when you want to synchronise files onto machines that don't have actively logged-in users, for example build machines.

    To use a Service Account instead of OAuth2 token flow, enter the path to your Service Account credentials at the service_account_file prompt during rclone config and rclone won't use the browser based authentication flow. If you'd rather stuff the contents of the credentials file into the rclone config file, you can set service_account_credentials with the actual contents of the file instead, or set the equivalent environment variable.

    -

    Use case - Google Apps/G-suite account and individual Drive

    -

    Let's say that you are the administrator of a Google Apps (old) or G-suite account. The goal is to store data on an individual's Drive account, who IS a member of the domain. We'll call the domain example.com, and the user foo@example.com.

    +

    Use case - Google Workspace account and individual Drive

    +

    Let's say that you are the administrator of a Google Workspace. The goal is to read or write data on an individual's Drive account, who IS a member of the domain. We'll call the domain example.com, and the user foo@example.com.

    There's a few steps we need to go through to accomplish this:

    1. Create a service account for example.com
    +

    If you ever need to remove access, press the "Delete service account key" button.

    2. Allowing API access to example.com Google Drive
    3. Configure rclone, assuming a new install
    rclone config
 
 n/s/q> n         # New
 name>gdrive      # Gdrive is an example name
-Storage>         # Select the number shown for Google Drive
+Storage>         # Type drive
 client_id>       # Can be left blank
 client_secret>   # Can be left blank
-scope>           # Select your scope, 1 for example
+scope>           # Select the scope use used in step 2
 root_folder_id>  # Can be left blank
-service_account_file> /home/foo/myJSONfile.json # This is where the JSON file goes!
+service_account_file> /home/foo/myJSONfile.json # Path to the JSON file you downloaded in step 1.
 y/n>             # Auto config, n
    4. Verify that it's working
    @@ -22334,7 +23310,7 @@ y/n> # Auto config, n
  • gdrive:backup - use the remote called gdrive, work in the folder named backup.
    • -

    Note: in case you configured a specific root folder on gdrive and rclone is unable to access the contents of that folder when using --drive-impersonate, do this instead: - in the gdrive web interface, share your root folder with the user/email of the new Service Account you created/selected at step #1 - use rclone without specifying the --drive-impersonate option, like this: rclone -v lsf gdrive:backup

    +

    Note: in case you configured a specific root folder on gdrive and rclone is unable to access the contents of that folder when using --drive-impersonate, do this instead: - in the gdrive web interface, share your root folder with the user/email of the new Service Account you created/selected at step 1 - use rclone without specifying the --drive-impersonate option, like this: rclone -v lsf gdrive:backup

    Shared drives (team drives)

    If you want to configure the remote to point to a Google Shared Drive (previously known as Team Drives) then answer y to the question Configure this as a Shared Drive (Team Drive)?.

    This will fetch the list of Shared Drives from google and allow you to configure which one you want to use. You can also type in a Shared Drive ID if you prefer.

    @@ -22352,13 +23328,14 @@ Choose a number from below, or type in your own value 3 / Rclone Test 3 \ "zzzzzzzzzzzzzzzzzzzz" Enter a Shared Drive ID> 1 --------------------- -[remote] -client_id = -client_secret = -token = {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} -team_drive = xxxxxxxxxxxxxxxxxxxx --------------------- +Configuration complete. +Options: +- type: drive +- client_id: +- client_secret: +- token: {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} +- team_drive: xxxxxxxxxxxxxxxxxxxx +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -22385,10 +23362,10 @@ trashed=false and 'c' in parents
  • without --fast-list: 22:05 min
  • with --fast-list: 58s
    • -

    Modification times and hashes

    +

    Modification times and hashes

    Google drive stores modification times accurate to 1 ms.

    Hash algorithms MD5, SHA1 and SHA256 are supported. Note, however, that a small fraction of files uploaded may not have SHA1 or SHA256 hashes especially if they were uploaded before 2018.

    -

    Restricted filename characters

    +

    Restricted filename characters

    Only Invalid UTF-8 bytes will be replaced, as they can't be used in JSON strings.

    In contrast to other backends, / can also be used in names and . or .. are valid names.

    Revisions

    @@ -22674,7 +23651,7 @@ trashed=false and 'c' in parents -

    Standard options

    +

    Standard options

    Here are the Standard options specific to drive (Google Drive).

    --drive-client-id

    Google Application Client Id Setting your own is recommended. See https://rclone.org/drive/#making-your-own-client-id for how to create your own. If you leave this blank, it will use an internal key which is low performance.

    @@ -22751,7 +23728,7 @@ trashed=false and 'c' in parents
  • Type: bool
  • Default: false
    • -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to drive (Google Drive).

    --drive-token

    OAuth Access Token as a JSON blob.

    @@ -23549,7 +24526,7 @@ rclone backend copyid drive: ID1 path1 ID2 path2 "webViewLink": "https://drive.google.com/file/d/0AxBe_CDEF4zkGHI4d0FjYko2QkD/view?usp=drivesdk\u0026resourcekey=0-ABCDEFGHIXJQpIGqBJq3MC" } ] -

    Limitations

    +

    Limitations

    Drive has quite a lot of rate limiting. This causes rclone to be limited to transferring about 2 files per second only. Individual files may be transferred much faster at 100s of MiB/s but lots of small files can take a long time.

    Server side copies are also subject to a separate rate limit. If you see User rate limit exceeded errors, wait at least 24 hours and retry. You can disable server-side copies with --disable copy to download and upload the files if you prefer.

    Limitations of Google Docs

    @@ -23557,7 +24534,7 @@ rclone backend copyid drive: ID1 path1 ID2 path2

    This is because rclone can't find out the size of the Google docs without downloading them.

    Google docs will transfer correctly with rclone sync, rclone copy etc as rclone knows to ignore the size when doing the transfer.

    However an unfortunate consequence of this is that you may not be able to download Google docs using rclone mount. If it doesn't work you will get a 0 sized file. If you try again the doc may gain its correct size and be downloadable. Whether it will work on not depends on the application accessing the mount and the OS you are running - experiment to find out if it does work for you!

    -

    Duplicated files

    +

    Duplicated files

    Sometimes, for no reason I've been able to track down, drive will duplicate a file that rclone uploads. Drive unlike all the other remotes can have duplicated files.

    Duplicated files cause problems with the syncing and you will see messages in the log about duplicates.

    Use rclone dedupe to fix duplicated files.

    @@ -23601,7 +24578,7 @@ rclone backend copyid drive: ID1 path1 ID2 path2

    Google Photos

    The rclone backend for Google Photos is a specialized backend for transferring photos and videos to and from Google Photos.

    NB The Google Photos API which rclone uses has quite a few limitations, so please read the limitations section carefully to make sure it is suitable for your use.

    -

    Configuration

    +

    Configuration

    The initial setup for google cloud storage involves getting a token from Google Photos which you need to do in your browser. rclone config walks you through it.

    Here is an example of how to make a remote called remote. First run:

     rclone config
    @@ -23657,11 +24634,11 @@ Got code *** are stored in full resolution at original quality. These uploads *** will count towards storage in your Google Account. --------------------- -[remote] -type = google photos -token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"} --------------------- +Configuration complete. +Options: +- type: google photos +- token: {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -23756,7 +24733,7 @@ y/e/d> y

    This means that you can use the album path pretty much like a normal filesystem and it is a good target for repeated syncing.

    The shared-album directory shows albums shared with you or by you. This is similar to the Sharing tab in the Google Photos web interface.

    -

    Standard options

    +

    Standard options

    Here are the Standard options specific to google photos (Google Photos).

    --gphotos-client-id

    OAuth Client Id.

    @@ -23788,7 +24765,7 @@ y/e/d> y
  • Type: bool
  • Default: false
    • -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to google photos (Google Photos).

    --gphotos-token

    OAuth Access Token as a JSON blob.

    @@ -23881,7 +24858,7 @@ y/e/d> y

    --gphotos-batch-size

    Max number of files in upload batch.

    This sets the batch size of files to upload. It has to be less than 50.

    -

    By default this is 0 which means rclone which calculate the batch size depending on the setting of batch_mode.

    +

    By default this is 0 which means rclone will calculate the batch size depending on the setting of batch_mode.

    -

    Limitations

    +

    Limitations

    Only images and videos can be uploaded. If you attempt to upload non videos or images or formats that Google Photos doesn't understand, rclone will upload the file, then Google Photos will give an error when it is put turned into a media item.

    Note that all media items uploaded to Google Photos through the API are stored in full resolution at "original quality" and will count towards your storage quota in your Google Account. The API does not offer a way to upload in "high quality" mode..

    rclone about is not supported by the Google Photos backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

    @@ -24039,7 +25016,7 @@ rclone backend drop Hasher: 
    rclone backend stickyimport hasher:path/to/data sha1 remote:/path/to/sum.sha1

    stickyimport is similar to import but works much faster because it does not need to stat existing files and skips initial tree walk. Instead of binding cache entries to file fingerprints it creates sticky entries bound to the file name alone ignoring size, modification time etc. Such hash entries can be replaced only by purge, delete, backend drop or by full re-read/re-write of the files.

    Configuration reference

    -

    Standard options

    +

    Standard options

    Here are the Standard options specific to hasher (Better checksums for other remotes).

    --hasher-remote

    Remote to cache checksums for (e.g. myRemote:path).

    @@ -24068,7 +25045,7 @@ rclone backend drop Hasher:
  • Type: Duration
  • Default: off
    • -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to hasher (Better checksums for other remotes).

    --hasher-auto-size

    Auto-update checksum for files smaller than this size (disabled by default).

    @@ -24133,6 +25110,7 @@ rclone backend drop Hasher:

    Other operations

    -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to hdfs (Hadoop distributed file system).

    --hdfs-service-principal-name

    Kerberos service principal name for the namenode.

    @@ -24329,7 +25307,7 @@ username = root
  • Type: string
  • Required: false
    • -

    Limitations

    +

    Limitations

    -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to hidrive (HiDrive).

    --hidrive-token

    OAuth Access Token as a JSON blob.

    @@ -24634,7 +25612,7 @@ rclone lsd remote:/users/test/path
  • Type: string
  • Required: false
    • -

    Limitations

    +

    Limitations

    HiDrive is able to store symbolic links (symlinks) by design, for example, when unpacked from a zip archive.

    There exists no direct mechanism to manage native symlinks in remotes. As such this implementation has chosen to ignore any native symlinks present in the remote. rclone will not be able to access or show any symlinks stored in the hidrive-remote. This means symlinks cannot be individually removed, copied, or moved, except when removing, copying, or moving the parent folder.

    @@ -24648,7 +25626,7 @@ rclone lsd remote:/users/test/path

    The remote: represents the configured url, and any path following it will be resolved relative to this url, according to the URL standard. This means with remote url https://beta.rclone.org/branch and path fix, the resolved URL will be https://beta.rclone.org/branch/fix, while with path /fix the resolved URL will be https://beta.rclone.org/fix as the absolute path is resolved from the root of the domain.

    If the path following the remote: ends with / it will be assumed to point to a directory. If the path does not end with /, then a HEAD request is sent and the response used to decide if it it is treated as a file or a directory (run with -vv to see details). When --http-no-head is specified, a path without ending / is always assumed to be a file. If rclone incorrectly assumes the path is a file, the solution is to specify the path with ending /. When you know the path is a directory, ending it with / is always better as it avoids the initial HEAD request.

    To just download a single file it is easier to use copyurl.

    -

    Configuration

    +

    Configuration

    Here is an example of how to make a remote called remote. First run:

     rclone config

    This will guide you through an interactive setup process:

    @@ -24671,10 +25649,11 @@ Choose a number from below, or type in your own value \ "https://example.com" url> https://beta.rclone.org Remote config --------------------- -[remote] -url = https://beta.rclone.org --------------------- +Configuration complete. +Options: +- type: http +- url: https://beta.rclone.org +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -24711,7 +25690,7 @@ e/n/d/r/c/s/q> q 
    rclone lsd --http-url https://beta.rclone.org :http:

    or:

    rclone lsd :http,url='https://beta.rclone.org':
    -

    Standard options

    +

    Standard options

    Here are the Standard options specific to http (HTTP).

    --http-url

    URL of HTTP host to connect to.

    @@ -24732,7 +25711,7 @@ e/n/d/r/c/s/q> q
  • Type: bool
  • Default: false
    • -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to http (HTTP).

    --http-headers

    Set HTTP headers for all transactions.

    @@ -24802,7 +25781,7 @@ rclone rc backend/command command=set fs=remote: -o url=https://example.comThe option keys are named as they are in the config file.

    This rebuilds the connection to the http backend when it is called with the new parameters. Only new parameters need be passed as the values will default to those currently in use.

    It doesn't return anything.

    -

    Limitations

    +

    Limitations

    rclone about is not supported by the HTTP backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

    See List of backends that do not support rclone about and rclone about

    ImageKit

    @@ -24811,7 +25790,7 @@ rclone rc backend/command command=set fs=remote: -o url=https://example.comImageKit.io provides real-time image and video optimizations, transformations, and CDN delivery. Over 1,000 businesses and 70,000 developers trust ImageKit with their images and videos on the web.

    Accounts & Pricing

    To use this backend, you need to create an account on ImageKit. Start with a free plan with generous usage limits. Then, as your requirements grow, upgrade to a plan that best fits your needs. See the pricing details.

    -

    Configuration

    +

    Configuration

    Here is an example of making an imagekit configuration.

    Firstly create a ImageKit.io account and choose a plan.

    You will need to log in and get the publicKey and privateKey for your account from the developer section.

    @@ -24878,7 +25857,7 @@ y/e/d> y

    ImageKit does not support modification times or hashes yet.

    Checksums

    No checksums are supported.

    -

    Standard options

    +

    Standard options

    Here are the Standard options specific to imagekit (ImageKit.io).

    --imagekit-endpoint

    You can find your ImageKit.io URL endpoint in your dashboard

    @@ -24907,7 +25886,7 @@ y/e/d> y
  • Type: string
  • Required: true
    • -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to imagekit (ImageKit.io).

    --imagekit-only-signed

    If you have configured Restrict unsigned image URLs in your dashboard settings, set this to true.

    @@ -25083,7 +26062,7 @@ y/e/d> y

    These auto-created files can be excluded from the sync using metadata filtering.

    rclone sync ... --metadata-exclude "source=metadata" --metadata-exclude "format=Metadata"

    Which excludes from the sync any files which have the source=metadata or format=Metadata flags which are added to Internet Archive auto-created files.

    -

    Configuration

    +

    Configuration

    Here is an example of making an internetarchive configuration. Most applies to the other providers as well, any differences are described below.

    First run

    rclone config
    @@ -25142,17 +26121,17 @@ Edit advanced config? y) Yes n) No (default) y/n> n --------------------- -[remote] -type = internetarchive -access_key_id = XXXX -secret_access_key = XXXX --------------------- +Configuration complete. +Options: +- type: internetarchive +- access_key_id: XXXX +- secret_access_key: XXXX +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y -

    Standard options

    +

    Standard options

    Here are the Standard options specific to internetarchive (Internet Archive).

    --internetarchive-access-key-id

    IAS3 Access Key.

    @@ -25174,7 +26153,7 @@ y/e/d> y
  • Type: string
  • Required: false
    • -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to internetarchive (Internet Archive).

    --internetarchive-endpoint

    IAS3 Endpoint.

    @@ -25380,7 +26359,7 @@ Response: {"error":"invalid_grant","error_description&q

    Onlime has sold access to Jottacloud proper, while providing localized support to Danish Customers, but have recently set up their own hosting, transferring their customers from Jottacloud servers to their own ones.

    This, of course, necessitates using their servers for authentication, but otherwise functionality and architecture seems equivalent to Jottacloud.

    To setup rclone to use Onlime Cloud Storage, choose Onlime Cloud authentication in the setup. The rest of the setup is identical to the default setup.

    -

    Configuration

    +

    Configuration

    Here is an example of how to make a remote called remote with the default setup. First run:

    rclone config

    This will guide you through an interactive setup process:

    @@ -25452,18 +26431,18 @@ Press Enter for the default (Archive). 2 > Shared 3 > Sync config_mountpoint> 1 --------------------- -[remote] -type = jottacloud -configVersion = 1 -client_id = jottacli -client_secret = -tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token -token = {........} -username = 2940e57271a93d987d6f8a21 -device = Jotta -mountpoint = Archive --------------------- +Configuration complete. +Options: +- type: jottacloud +- configVersion: 1 +- client_id: jottacli +- client_secret: +- tokenURL: https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token +- token: {........} +- username: 2940e57271a93d987d6f8a21 +- device: Jotta +- mountpoint: Archive +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -25484,11 +26463,11 @@ y/e/d> y

    This backend supports --fast-list which allows you to use fewer transactions in exchange for more memory. See the rclone docs for more details.

    Note that the implementation in Jottacloud always uses only a single API request to get the entire list, so for large folders this could lead to long wait time before the first results are shown.

    Note also that with rclone version 1.58 and newer, information about MIME types and metadata item utime are not available when using --fast-list.

    -

    Modification times and hashes

    +

    Modification times and hashes

    Jottacloud allows modification times to be set on objects accurate to 1 second. These will be used to detect whether objects need syncing or not.

    Jottacloud supports MD5 type hashes, so you can use the --checksum flag.

    Note that Jottacloud requires the MD5 hash before upload so if the source does not have an MD5 checksum then the file will be cached temporarily on disk (in location given by --temp-dir) before it is uploaded. Small files will be cached in memory - see the --jottacloud-md5-memory-limit flag. When uploading from local disk the source checksum is always available, so this does not apply. Starting with rclone version 1.52 the same is true for encrypted remotes (in older versions the crypt backend would not calculate hashes for uploads from local disk, so the Jottacloud backend had to do it as described above).

    -

    Restricted filename characters

    +

    Restricted filename characters

    In addition to the default restricted characters set the following characters are also replaced:

    @@ -25544,7 +26523,7 @@ y/e/d> y

    Versioning can be disabled by --jottacloud-no-versions option. This is achieved by deleting the remote file prior to uploading a new version. If the upload the fails no version of the file will be available in the remote.

    Quota information

    To view your current quota you can use the rclone about remote: command which will display your usage limit (unless it is unlimited) and the current usage.

    -

    Standard options

    +

    Standard options

    Here are the Standard options specific to jottacloud (Jottacloud).

    --jottacloud-client-id

    OAuth Client Id.

    @@ -25566,7 +26545,7 @@ y/e/d> y
  • Type: string
  • Required: false
    • -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to jottacloud (Jottacloud).

    --jottacloud-token

    OAuth Access Token as a JSON blob.

    @@ -25714,7 +26693,7 @@ y/e/d> y

    See the metadata docs for more info.

    -

    Limitations

    +

    Limitations

    Note that Jottacloud is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

    There are quite a few characters that can't be in Jottacloud file names. Rclone will map these names to and from an identical looking unicode equivalent. For example if a file has a ? in it will be mapped to ？ instead.

    Jottacloud only supports filenames up to 255 characters in length.

    @@ -25723,7 +26702,7 @@ y/e/d> y

    Koofr

    Paths are specified as remote:path

    Paths may be as deep as required, e.g. remote:directory/subdirectory.

    -

    Configuration

    +

    Configuration

    The initial setup for Koofr involves creating an application password for rclone. You can do that by opening the Koofr web application, giving the password a nice name like rclone and clicking on generate.

    Here is an example of how to make a remote called koofr. First run:

     rclone config
    @@ -25791,7 +26770,7 @@ y/e/d> y 
    rclone ls koofr:

    To copy a local directory to an Koofr directory called backup

    rclone copy /home/source koofr:backup
    -

    Restricted filename characters

    +

    Restricted filename characters

    In addition to the default restricted characters set the following characters are also replaced:

    @@ -25810,7 +26789,7 @@ y/e/d> y

    Invalid UTF-8 bytes will also be replaced, as they can't be used in XML strings.

    -

    Standard options

    +

    Standard options

    Here are the Standard options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers).

    --koofr-provider

    Choose your storage provider.

    @@ -25866,7 +26845,7 @@ y/e/d> y
  • Type: string
  • Required: true
    • -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers).

    --koofr-mountid

    Mount ID of the mount to use.

    @@ -25907,7 +26886,7 @@ y/e/d> y
  • Type: string
  • Required: false
    • -

    Limitations

    +

    Limitations

    Note that Koofr is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

    Providers

    Koofr

    @@ -26037,7 +27016,7 @@ d) Delete this remote y/e/d> y

    Linkbox

    Linkbox is a private cloud drive.

    -

    Configuration

    +

    Configuration

    Here is an example of making a remote for Linkbox.

    First run:

     rclone config
    @@ -26073,7 +27052,7 @@ e) Edit this remote d) Delete this remote y/e/d> y -

    Standard options

    +

    Standard options

    Here are the Standard options specific to linkbox (Linkbox).

    Token from https://www.linkbox.to/admin/account

    @@ -26084,7 +27063,7 @@ y/e/d> y
  • Type: string
  • Required: true
    • -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to linkbox (Linkbox).

    Description of the remote.

    @@ -26095,7 +27074,7 @@ y/e/d> y
  • Type: string
  • Required: false
    • -

    Limitations

    +

    Limitations

    Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

    Mail.ru Cloud

    Mail.ru Cloud is a cloud storage provided by a Russian internet company Mail.Ru Group. The official desktop client is Disk-O:, available on Windows and Mac OS.

    @@ -26110,7 +27089,7 @@ y/e/d> y
  • Storage keeps hash for all files and performs transparent deduplication, the hash algorithm is a modified SHA1
  • If a particular file is already present in storage, one can quickly submit file hash instead of long file upload (this optimization is supported by rclone)
    • -

    Configuration

    +

    Configuration

    Here is an example of making a mailru configuration.

    First create a Mail.ru Cloud account and choose a tariff.

    You will need to log in and create an app password for rclone. Rclone will not work with your normal username and password - it will give an error like oauth2: server response missing access_token.

    @@ -26170,13 +27149,13 @@ y) Yes n) No y/n> n Remote config --------------------- -[remote] -type = mailru -user = username@mail.ru -pass = *** ENCRYPTED *** -speedup_enable = true --------------------- +Configuration complete. +Options: +- type: mailru +- user: username@mail.ru +- pass: *** ENCRYPTED *** +- speedup_enable: true +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -26190,14 +27169,14 @@ y/e/d> y 
    rclone ls remote:directory

    Sync /home/local/directory to the remote path, deleting any excess files in the path.

    rclone sync --interactive /home/local/directory remote:directory
    -

    Modification times and hashes

    +

    Modification times and hashes

    Files support a modification time attribute with up to 1 second precision. Directories do not have a modification time, which is shown as "Jan 1 1970".

    File hashes are supported, with a custom Mail.ru algorithm based on SHA1. If file size is less than or equal to the SHA1 block size (20 bytes), its hash is simply its data right-padded with zero bytes. Hashes of a larger file is computed as a SHA1 of the file data bytes concatenated with a decimal representation of the data length.

    Emptying Trash

    Removing a file or directory actually moves it to the trash, which is not visible to rclone but can be seen in a web browser. The trashed file still occupies part of total quota. If you wish to empty your trash and free some quota, you can use the rclone cleanup remote: command, which will permanently delete all your trashed files. This command does not take any path arguments.

    Quota information

    To view your current quota you can use the rclone about remote: command which will display your usage limit (quota) and the current usage.

    -

    Restricted filename characters

    +

    Restricted filename characters

    In addition to the default restricted characters set the following characters are also replaced:

    @@ -26251,7 +27230,7 @@ y/e/d> y

    Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

    -

    Standard options

    +

    Standard options

    Here are the Standard options specific to mailru (Mail.ru Cloud).

    --mailru-client-id

    OAuth Client Id.

    @@ -26314,7 +27293,7 @@ y/e/d> y -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to mailru (Mail.ru Cloud).

    --mailru-token

    OAuth Access Token as a JSON blob.

    @@ -26482,7 +27461,7 @@ y/e/d> y
  • Type: string
  • Required: false
    • -

    Limitations

    +

    Limitations

    File size limits depend on your account. A single file size is limited by 2G for a free account and unlimited for paid tariffs. Please refer to the Mail.ru site for the total uploaded size limits.

    Note that Mailru is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

    Mega

    @@ -26490,7 +27469,7 @@ y/e/d> y

    This is an rclone backend for Mega which supports the file transfer features of Mega using the same client side encryption.

    Paths are specified as remote:path

    Paths may be as deep as required, e.g. remote:directory/subdirectory.

    -

    Configuration

    +

    Configuration

    Here is an example of how to make a remote called remote. First run:

     rclone config

    This will guide you through an interactive setup process:

    @@ -26519,12 +27498,12 @@ password: Confirm the password: password: Remote config --------------------- -[remote] -type = mega -user = you@example.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: mega +- user: you@example.com +- pass: *** ENCRYPTED *** +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -26537,9 +27516,9 @@ y/e/d> y 
    rclone ls remote:

    To copy a local directory to an Mega directory called backup

    rclone copy /home/source remote:backup
    -

    Modification times and hashes

    +

    Modification times and hashes

    Mega does not support modification times or hashes yet.

    -

    Restricted filename characters

    +

    Restricted filename characters

    @@ -26562,7 +27541,7 @@ y/e/d> y

    Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

    -

    Duplicated files

    +

    Duplicated files

    Mega can have two files with exactly the same name and path (unlike a normal file system).

    Duplicated files cause problems with the syncing and you will see messages in the log about duplicates.

    Use rclone dedupe to fix duplicated files.

    @@ -26591,7 +27570,7 @@ me@example.com:/$

    Note that once blocked, the use of other tools (such as megacmd) is not a sure workaround: following megacmd login times have been observed in succession for blocked remote: 7 minutes, 20 min, 30min, 30 min, 30min. Web access looks unaffected though.

    Investigation is continuing in relation to workarounds based on timeouts, pacers, retrials and tpslimits - if you discover something relevant, please post on the forum.

    So, if rclone was working nicely and suddenly you are unable to log-in and you are sure the user and the password are correct, likely you have got the remote blocked for a while.

    -

    Standard options

    +

    Standard options

    Here are the Standard options specific to mega (Mega).

    --mega-user

    User name.

    @@ -26612,7 +27591,7 @@ me@example.com:/$
  • Type: string
  • Required: true
    • -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to mega (Mega).

    --mega-debug

    Output more debug from Mega.

    @@ -26665,13 +27644,13 @@ me@example.com:/$

    Process killed

    On accounts with large files or something else, memory usage can significantly increase when executing list/sync instructions. When running on cloud providers (like AWS with EC2), check if the instance type has sufficient memory/CPU to execute the commands. Use the resource monitoring tools to inspect after sending the commands. Look at this issue.

    -

    Limitations

    +

    Limitations

    This backend uses the go-mega go library which is an opensource go library implementing the Mega API. There doesn't appear to be any documentation for the mega protocol beyond the mega C++ SDK source code so there are likely quite a few errors still remaining in this library.

    Mega allows duplicate files which may confuse rclone.

    Memory

    The memory backend is an in RAM backend. It does not persist its data - use the local backend for that.

    The memory backend behaves like a bucket-based remote (e.g. like s3). Because it has no parameters you can just use it with the :memory: remote name.

    -

    Configuration

    +

    Configuration

    You can configure it as a remote like this with rclone config too if you want to:

    No remotes found, make a new one?
 n) New remote
@@ -26691,10 +27670,10 @@ Storage> memory
 
 Remote config
 
---------------------
-[remote]
-type = memory
---------------------
+Configuration complete.
+Options:
+- type: memory
+Keep this "remote" remote?
 y) Yes this is OK (default)
 e) Edit this remote
 d) Delete this remote
@@ -26703,11 +27682,11 @@ y/e/d> y
    rclone mount :memory: /mnt/tmp
 rclone serve webdav :memory:
 rclone serve sftp :memory:
    -

    Modification times and hashes

    +

    Modification times and hashes

    The memory backend supports MD5 hashes and modification times accurate to 1 nS.

    -

    Restricted filename characters

    +

    Restricted filename characters

    The memory backend replaces the default restricted characters set.

    -

    Advanced options

    +

    Advanced options

    Here are the Advanced options specific to memory (In memory object storage system.).

    --memory-description

    Description of the remote.

    @@ -26722,7 +27701,7 @@ rclone serve sftp :memory:

    Paths are specified as remote: You may put subdirectories in too, e.g. remote:/path/to/dir. If you have a CP code you can use that as the folder after the domain such as <domain>/<cpcode>/<internal directories within cpcode>.

    For example, this is commonly configured with or without a CP code: * With a CP code. [your-domain-prefix]-nsu.akamaihd.net/123456/subdirectory/ * Without a CP code. [your-domain-prefix]-nsu.akamaihd.net

    See all buckets rclone lsd remote: The initial setup for Netstorage involves getting an account and secret. Use rclone config to walk you through the setup process.

    -

    Configuration

    +

    Configuration

    Here's an example of how to make a remote called ns1.

    1. To begin the interactive configuration process, enter this command:
      2. @@ -26830,7 +27809,7 @@ y/e/d> y

      Purge

      NetStorage remote supports the purge feature by using the "quick-delete" NetStorage API action. The quick-delete action is disabled by default for security reasons and can be enabled for the account through the Akamai portal. Rclone will first try to use quick-delete action for the purge command and if this functionality is disabled then will fall back to a standard delete method.

      Note: Read the NetStorage Usage API for considerations when using "quick-delete". In general, using quick-delete method will not delete the tree immediately and objects targeted for quick-delete may still be accessible.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to netstorage (Akamai NetStorage).

      --netstorage-host

      Domain+path of NetStorage host to connect to.

      @@ -26862,7 +27841,7 @@ y/e/d> y
    2. Type: string
    3. Required: true
      4. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to netstorage (Akamai NetStorage).

      --netstorage-protocol

      Select between HTTP or HTTPS protocol.

      @@ -26911,7 +27890,7 @@ y/e/d> y

      The desired path location (including applicable sub-directories) ending in the object that will be the target of the symlink (for example, /links/mylink). Include the file extension for the object, if applicable. rclone backend symlink <src> <path>

      Microsoft Azure Blob Storage

      Paths are specified as remote:container (or remote: for the lsd command.) You may put subdirectories in too, e.g. remote:container/path/to/dir.

      -

      Configuration

      +

      Configuration

      Here is an example of making a Microsoft Azure Blob Storage configuration. For a remote called remote. First run:

       rclone config

      This will guide you through an interactive setup process:

      @@ -26935,12 +27914,13 @@ key> base64encodedkey== Endpoint for the service - leave blank normally. endpoint> Remote config --------------------- -[remote] -account = account_name -key = base64encodedkey== -endpoint = --------------------- +Configuration complete. +Options: +- type: azureblob +- account: account_name +- key: base64encodedkey== +- endpoint: +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -26955,13 +27935,13 @@ y/e/d> y 
      rclone sync --interactive /home/local/directory remote:container

      --fast-list

      This remote supports --fast-list which allows you to use fewer transactions in exchange for more memory. See the rclone docs for more details.

      -

      Modification times and hashes

      +

      Modification times and hashes

      The modification time is stored as metadata on the object with the mtime key. It is stored using RFC3339 Format time with nanosecond precision. The metadata is supplied during directory listings so there is no performance overhead to using it.

      If you wish to use the Azure standard LastModified time stored on the object as the modified time, then use the --use-server-modtime flag. Note that rclone can't set LastModified, so using the --update flag when syncing is recommended if using --use-server-modtime.

      MD5 hashes are stored with blobs. However blobs that were uploaded in chunks only have an MD5 if the source remote was capable of MD5 hashes, e.g. the local disk.

      Performance

      When uploading large files, increasing the value of --azureblob-upload-concurrency will increase performance at the cost of using more memory. The default of 16 is set quite conservatively to use less memory. It maybe be necessary raise it to 64 or higher to fully utilize a 1 GBit/s link with a single file transfer.

      -

      Restricted filename characters

      +

      Restricted filename characters

      In addition to the default restricted characters set the following characters are also replaced:

      @@ -27104,7 +28084,10 @@ container/

      If use_msi is set then managed service identity credentials are used. This authentication only works when running in an Azure service. env_auth needs to be unset to use this.

      However if you have multiple user identities to choose from these must be explicitly specified using exactly one of the msi_object_id, msi_client_id, or msi_mi_res_id parameters.

      If none of msi_object_id, msi_client_id, or msi_mi_res_id is set, this is is equivalent to using env_auth.

      -

      Standard options

      +

      Anonymous

      +

      If you want to access resources with public anonymous access then set account only. You can do this without making an rclone config:

      +
      rclone lsf :azureblob,account=ACCOUNT:CONTAINER
      +

      Standard options

      Here are the Standard options specific to azureblob (Microsoft Azure Blob Storage).

      --azureblob-account

      Azure Storage Account Name.

      @@ -27200,7 +28183,7 @@ container/
    4. Type: string
    5. Required: false
      6. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to azureblob (Microsoft Azure Blob Storage).

      --azureblob-client-send-certificate-chain

      Send the certificate chain when using certificate auth.

      @@ -27512,7 +28495,7 @@ container/
    6. Content-Type

      7. Eg --header-upload "Content-Type: text/potato"

      -

      Limitations

      +

      Limitations

      MD5 sums are only uploaded with chunked files if the source has an MD5 sum. This will always be the case for a local to azure copy.

      rclone about is not supported by the Microsoft Azure Blob storage backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

      See List of backends that do not support rclone about and rclone about

      @@ -27522,7 +28505,7 @@ container/

      Also, if you want to access a storage emulator instance running on a different machine, you can override the endpoint parameter in the advanced settings, setting it to http(s)://<host>:<port>/devstoreaccount1 (e.g. http://10.254.2.5:10000/devstoreaccount1).

      Microsoft Azure Files Storage

      Paths are specified as remote: You may put subdirectories in too, e.g. remote:path/to/dir.

      -

      Configuration

      +

      Configuration

      Here is an example of making a Microsoft Azure Files Storage configuration. For a remote called remote. First run:

       rclone config

      This will guide you through an interactive setup process:

      @@ -27602,7 +28585,7 @@ y/e/d>

      The modified time is stored as Azure standard LastModified time on files

      Performance

      When uploading large files, increasing the value of --azurefiles-upload-concurrency will increase performance at the cost of using more memory. The default of 16 is set quite conservatively to use less memory. It maybe be necessary raise it to 64 or higher to fully utilize a 1 GBit/s link with a single file transfer.

      -

      Restricted filename characters

      +

      Restricted filename characters

      In addition to the default restricted characters set the following characters are also replaced:

      @@ -27767,7 +28750,7 @@ y/e/d>

      If use_msi is set then managed service identity credentials are used. This authentication only works when running in an Azure service. env_auth needs to be unset to use this.

      However if you have multiple user identities to choose from these must be explicitly specified using exactly one of the msi_object_id, msi_client_id, or msi_mi_res_id parameters.

      If none of msi_object_id, msi_client_id, or msi_mi_res_id is set, this is is equivalent to using env_auth.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to azurefiles (Microsoft Azure Files).

      --azurefiles-account

      Azure Storage Account Name.

      @@ -27882,7 +28865,7 @@ y/e/d>
    7. Type: string
    8. Required: false
      9. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to azurefiles (Microsoft Azure Files).

      --azurefiles-client-send-certificate-chain

      Send the certificate chain when using certificate auth.

      @@ -28052,12 +29035,12 @@ y/e/d>
    9. Content-Type

      10. Eg --header-upload "Content-Type: text/potato"

      -

      Limitations

      +

      Limitations

      MD5 sums are only uploaded with chunked files if the source has an MD5 sum. This will always be the case for a local to azure copy.

      Microsoft OneDrive

      Paths are specified as remote:path

      Paths may be as deep as required, e.g. remote:directory/subdirectory.

      -

      Configuration

      +

      Configuration

      The initial setup for OneDrive involves getting a token from Microsoft which you need to do in your browser. rclone config walks you through it.

      Here is an example of how to make a remote called remote. First run:

       rclone config
      @@ -28123,13 +29106,13 @@ Is that okay? y) Yes n) No y/n> y --------------------- -[remote] -type = onedrive -token = {"access_token":"youraccesstoken","token_type":"Bearer","refresh_token":"yourrefreshtoken","expiry":"2018-08-26T22:39:52.486512262+08:00"} -drive_id = b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk -drive_type = business --------------------- +Configuration complete. +Options: +- type: onedrive +- token: {"access_token":"youraccesstoken","token_type":"Bearer","refresh_token":"yourrefreshtoken","expiry":"2018-08-26T22:39:52.486512262+08:00"} +- drive_id: b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk +- drive_type: business +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -28169,7 +29152,7 @@ y/e/d> y
    10. In the rclone config, set token_url to https://login.microsoftonline.com/YOUR_TENANT_ID/oauth2/v2.0/token.

      11. Note: If you have a special region, you may need a different host in step 4 and 5. Here are some hints.

      -

      Modification times and hashes

      +

      Modification times and hashes

      OneDrive allows modification times to be set on objects accurate to 1 second. These will be used to detect whether objects need syncing or not.

      OneDrive Personal, OneDrive for Business and Sharepoint Server support QuickXorHash.

      Before rclone 1.62 the default hash for Onedrive Personal was SHA1. For rclone 1.62 and above the default for all Onedrive backends is QuickXorHash.

      @@ -28182,7 +29165,7 @@ y/e/d> y

      This can be useful with rclone mount and rclone rc vfs/refresh recursive=true) to very quickly fill the mount with information about all the files.

      The API used for the recursive listing (ListR) only supports listing from the root of the drive. This will become increasingly inefficient the further away you get from the root as rclone will have to discard files outside of the directory you are using.

      Some commands (like rclone lsf -R) will use ListR by default - you can turn this off with --disable ListR if you need to.

      -

      Restricted filename characters

      +

      Restricted filename characters

      In addition to the default restricted characters set the following characters are also replaced:

      @@ -28282,7 +29265,7 @@ y/e/d> y

      Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

      Deleting files

      Any files you delete with rclone will end up in the trash. Microsoft doesn't provide an API to permanently delete files, nor to empty the trash, so you will have to do that with one of Microsoft's apps or via the OneDrive website.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to onedrive (Microsoft OneDrive).

      --onedrive-client-id

      OAuth Client Id.

      @@ -28332,7 +29315,7 @@ y/e/d> y -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to onedrive (Microsoft OneDrive).

      --onedrive-token

      OAuth Access Token as a JSON blob.

      @@ -28677,75 +29660,75 @@ rclone rc vfs/refresh recursive=true

      Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are "read", "write", "read,write", and "off" (the default). "write" supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use "read,write" instead of "write" if you wish to update/remove permissions.

      Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business.

      Example for OneDrive Personal:

      -
      [
-    {
-        "id": "1234567890ABC!123",
-        "grantedTo": {
-            "user": {
-                "id": "ryan@contoso.com"
-            },
-            "application": {},
-            "device": {}
-        },
-        "invitation": {
-            "email": "ryan@contoso.com"
-        },
-        "link": {
-            "webUrl": "https://1drv.ms/t/s!1234567890ABC"
-        },
-        "roles": [
-            "read"
-        ],
-        "shareId": "s!1234567890ABC"
-    }
-]
      +
      [
+    {
+        "id": "1234567890ABC!123",
+        "grantedTo": {
+            "user": {
+                "id": "ryan@contoso.com"
+            },
+            "application": {},
+            "device": {}
+        },
+        "invitation": {
+            "email": "ryan@contoso.com"
+        },
+        "link": {
+            "webUrl": "https://1drv.ms/t/s!1234567890ABC"
+        },
+        "roles": [
+            "read"
+        ],
+        "shareId": "s!1234567890ABC"
+    }
+]

      Example for OneDrive Business:

      -
      [
-    {
-        "id": "48d31887-5fad-4d73-a9f5-3c356e68a038",
-        "grantedToIdentities": [
-            {
-                "user": {
-                    "displayName": "ryan@contoso.com"
-                },
-                "application": {},
-                "device": {}
-            }
-        ],
-        "link": {
-            "type": "view",
-            "scope": "users",
-            "webUrl": "https://contoso.sharepoint.com/:w:/t/design/a577ghg9hgh737613bmbjf839026561fmzhsr85ng9f3hjck2t5s"
-        },
-        "roles": [
-            "read"
-        ],
-        "shareId": "u!LKj1lkdlals90j1nlkascl"
-    },
-    {
-        "id": "5D33DD65C6932946",
-        "grantedTo": {
-            "user": {
-                "displayName": "John Doe",
-                "id": "efee1b77-fb3b-4f65-99d6-274c11914d12"
-            },
-            "application": {},
-            "device": {}
-        },
-        "roles": [
-            "owner"
-        ],
-        "shareId": "FWxc1lasfdbEAGM5fI7B67aB5ZMPDMmQ11U"
-    }
-]
      +
      [
+    {
+        "id": "48d31887-5fad-4d73-a9f5-3c356e68a038",
+        "grantedToIdentities": [
+            {
+                "user": {
+                    "displayName": "ryan@contoso.com"
+                },
+                "application": {},
+                "device": {}
+            }
+        ],
+        "link": {
+            "type": "view",
+            "scope": "users",
+            "webUrl": "https://contoso.sharepoint.com/:w:/t/design/a577ghg9hgh737613bmbjf839026561fmzhsr85ng9f3hjck2t5s"
+        },
+        "roles": [
+            "read"
+        ],
+        "shareId": "u!LKj1lkdlals90j1nlkascl"
+    },
+    {
+        "id": "5D33DD65C6932946",
+        "grantedTo": {
+            "user": {
+                "displayName": "John Doe",
+                "id": "efee1b77-fb3b-4f65-99d6-274c11914d12"
+            },
+            "application": {},
+            "device": {}
+        },
+        "roles": [
+            "owner"
+        ],
+        "shareId": "FWxc1lasfdbEAGM5fI7B67aB5ZMPDMmQ11U"
+    }
+]

      To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this.

      When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous".

      Example request to add a "read" permission with --metadata-mapper:

      -
      {
-    "Metadata": {
-        "permissions": "[{\"grantedToIdentities\":[{\"user\":{\"id\":\"ryan@contoso.com\"}}],\"roles\":[\"read\"]}]"
-    }
-}
      +
      {
+    "Metadata": {
+        "permissions": "[{\"grantedToIdentities\":[{\"user\":{\"id\":\"ryan@contoso.com\"}}],\"roles\":[\"read\"]}]"
+    }
+}

      Note that adding a permission can fail if a conflicting permission already exists for the file/folder.

      To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed.

      To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that the owner role will be ignored, as it cannot be removed.

      @@ -28895,7 +29878,7 @@ rclone rc vfs/refresh recursive=true

      See the metadata docs for more info.

      -

      Limitations

      +

      Limitations

      If you don't use rclone for 90 days the refresh token will expire. This will result in authorization problems. This is easy to fix by running the rclone config reconnect remote: command to get a new token and refresh token.

      Naming

      Note that OneDrive is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

      @@ -28990,7 +29973,7 @@ ERROR : 20230203_123826234_iOS.heic: vfs cache: failed to download: vfs reader:

      OpenDrive

      Paths are specified as remote:path

      Paths may be as deep as required, e.g. remote:directory/subdirectory.

      -

      Configuration

      +

      Configuration

      Here is an example of how to make a remote called remote. First run:

       rclone config

      This will guide you through an interactive setup process:

      @@ -29016,11 +29999,12 @@ Enter the password: password: Confirm the password: password: --------------------- -[remote] -username = -password = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: opendrive +- username: +- password: *** ENCRYPTED *** +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -29031,10 +30015,10 @@ y/e/d> y 
      rclone ls remote:

      To copy a local directory to an OpenDrive directory called backup

      rclone copy /home/source remote:backup
      -

      Modification times and hashes

      +

      Modification times and hashes

      OpenDrive allows modification times to be set on objects accurate to 1 second. These will be used to detect whether objects need syncing or not.

      The MD5 hash algorithm is supported.

      -

      Restricted filename characters

      +

      Restricted filename characters

      @@ -29134,7 +30118,7 @@ y/e/d> y

      Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to opendrive (OpenDrive).

      --opendrive-username

      Username.

      @@ -29155,7 +30139,7 @@ y/e/d> y
    11. Type: string
    12. Required: true
      13. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to opendrive (OpenDrive).

      --opendrive-encoding

      The encoding for the backend.

      @@ -29186,7 +30170,7 @@ y/e/d> y
    13. Type: string
    14. Required: false
      15. -

      Limitations

      +

      Limitations

      Note that OpenDrive is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

      There are quite a few characters that can't be in OpenDrive file names. These can't occur on Windows platforms, but on non-Windows platforms they are common. Rclone will map these names to and from an identical looking unicode equivalent. For example if a file has a ? in it will be mapped to ？ instead.

      rclone about is not supported by the OpenDrive backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

      @@ -29200,7 +30184,7 @@ y/e/d> y

      Paths are specified as remote:bucket (or remote: for the lsd command.) You may put subdirectories in too, e.g. remote:bucket/path/to/dir.

      Sample command to transfer local artifacts to remote:bucket in oracle object storage:

      rclone -vvv --progress --stats-one-line --max-stats-groups 10 --log-format date,time,UTC,longfile --fast-list --buffer-size 256Mi --oos-no-check-bucket --oos-upload-cutoff 10Mi --multi-thread-cutoff 16Mi --multi-thread-streams 3000 --transfers 3000 --checkers 64 --retries 2 --oos-chunk-size 10Mi --oos-upload-concurrency 10000 --oos-attempt-resume-upload --oos-leave-parts-on-error sync ./artifacts remote:bucket -vv

      -

      Configuration

      +

      Configuration

      Here is an example of making an oracle object storage configuration. rclone config walks you through it.

      Here is an example of how to make a remote called remote. First run:

       rclone config
      @@ -29378,7 +30362,7 @@ namespace = id<redacted>34 compartment = ocid1.compartment.oc1..aa<redacted>ba region = us-ashburn-1 provider = no_auth -

      Modification times and hashes

      +

      Modification times and hashes

      The modification time is stored as metadata on the object as opc-meta-mtime as floating point since the epoch, accurate to 1 ns.

      If the modification time needs to be updated rclone will attempt to perform a server side copy to update the modification if the object can be copied in a single part. In the case the object is larger than 5Gb, the object will be uploaded rather than copied.

      Note that reading this from the object takes an additional HEAD request as the metadata isn't returned in object listings.

      @@ -29391,7 +30375,7 @@ provider = no_auth

      Multipart uploads will use --transfers * --oos-upload-concurrency * --oos-chunk-size extra memory. Single part uploads to not use extra memory.

      Single part transfers can be faster than multipart transfers or slower depending on your latency from oos - the more latency, the more likely single part transfers will be faster.

      Increasing --oos-upload-concurrency will increase throughput (8 would be a sensible value) and increasing --oos-chunk-size also increases throughput (16M would be sensible). Increasing either of these will use more memory. The default values are high enough to gain most of the possible performance without using too much memory.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to oracleobjectstorage (Oracle Cloud Infrastructure Object Storage).

      --oos-provider

      Choose your Auth Provider

      @@ -29506,7 +30490,7 @@ provider = no_auth -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to oracleobjectstorage (Oracle Cloud Infrastructure Object Storage).

      --oos-storage-tier

      The storage class to use when storing new objects in storage. https://docs.oracle.com/en-us/iaas/Content/Object/Concepts/understandingstoragetiers.htm

      @@ -29825,7 +30809,7 @@ if not.

      Mounting Buckets

      QingStor

      Paths are specified as remote:bucket (or remote: for the lsd command.) You may put subdirectories in too, e.g. remote:bucket/path/to/dir.

      -

      Configuration

      +

      Configuration

      Here is an example of making an QingStor configuration. First run

      rclone config

      This will guide you through an interactive setup process.

      @@ -29871,15 +30855,16 @@ Number of connection retry. Leave blank will use the default value "3". connection_retries> Remote config --------------------- -[remote] -env_auth = false -access_key_id = access_key -secret_access_key = secret_key -endpoint = -zone = pek3a -connection_retries = --------------------- +Configuration complete. +Options: +- type: qingstor +- env_auth: false +- access_key_id: access_key +- secret_access_key: secret_key +- endpoint: +- zone: pek3a +- connection_retries: +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -29917,10 +30902,10 @@ y/e/d> y -

      Restricted filename characters

      +

      Restricted filename characters

      The control characters 0x00-0x1F and / are replaced as in the default restricted characters set. Note that 0x7F is not replaced.

      Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to qingstor (QingCloud Object Storage).

      --qingstor-env-auth

      Get QingStor credentials from runtime.

      @@ -30001,7 +30986,7 @@ y/e/d> y -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to qingstor (QingCloud Object Storage).

      --qingstor-connection-retries

      Number of connection retries.

      @@ -30065,7 +31050,7 @@ y/e/d> y
    15. Type: string
    16. Required: false
      17. -

      Limitations

      +

      Limitations

      rclone about is not supported by the qingstor backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

      See List of backends that do not support rclone about and rclone about

      Quatrix

      @@ -30074,7 +31059,7 @@ y/e/d> y

      Paths may be as deep as required, e.g., remote:directory/subdirectory.

      The initial setup for Quatrix involves getting an API Key from Quatrix. You can get the API key in the user's profile at https://<account>/profile/api-keys or with the help of the API - https://docs.maytech.net/quatrix/quatrix-api/api-explorer#/API-Key/post_api_key_create.

      See complete Swagger documentation for Quatrix - https://docs.maytech.net/quatrix/quatrix-api/api-explorer

      -

      Configuration

      +

      Configuration

      Here is an example of how to make a remote called remote. First run:

       rclone config

      This will guide you through an interactive setup process:

      @@ -30096,11 +31081,12 @@ api_key> your_api_key Host name of Quatrix account. host> example.quatrix.it --------------------- -[remote] -api_key = your_api_key -host = example.quatrix.it --------------------- +Configuration complete. +Options: +- type: quatrix +- api_key: your_api_key +- host: example.quatrix.it +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -30132,12 +31118,12 @@ e/n/d/r/c/s/q> e Choose a number from below, or type in an existing value 1 > remote remote> remote --------------------- -[remote] -type = quatrix -host = some_host.quatrix.it -api_key = your_api_key --------------------- +Configuration complete. +Options: +- type: quatrix +- host: some_host.quatrix.it +- api_key: your_api_key +Keep this "remote" remote? Edit remote Option api_key. API key for accessing Quatrix account @@ -30147,26 +31133,26 @@ Option host. Host name of Quatrix account Enter a string value. Press Enter for the default (some_host.quatrix.it). --------------------- -[remote] -type = quatrix -host = some_host.quatrix.it -api_key = your_api_key --------------------- +Configuration complete. +Options: +- type: quatrix +- host: some_host.quatrix.it +- api_key: your_api_key +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y -

      Modification times and hashes

      +

      Modification times and hashes

      Quatrix allows modification times to be set on objects accurate to 1 microsecond. These will be used to detect whether objects need syncing or not.

      Quatrix does not support hashes, so you cannot use the --checksum flag.

      -

      Restricted filename characters

      +

      Restricted filename characters

      File names in Quatrix are case sensitive and have limitations like the maximum length of a filename is 255, and the minimum length is 1. A file name cannot be equal to . or .. nor contain / , \ or non-printable ascii.

      Transfers

      For files above 50 MiB rclone will use a chunked transfer. Rclone will upload up to --transfers chunks at the same time (shared among all multipart uploads). Chunks are buffered in memory, and the minimal chunk size is 10_000_000 bytes by default, and it can be changed in the advanced configuration, so increasing --transfers will increase the memory use. The chunk size has a maximum size limit, which is set to 100_000_000 bytes by default and can be changed in the advanced configuration. The size of the uploaded chunk will dynamically change depending on the upload speed. The total memory use equals the number of transfers multiplied by the minimal chunk size. In case there's free memory allocated for the upload (which equals the difference of maximal_summary_chunk_size and minimal_chunk_size * transfers), the chunk size may increase in case of high upload speed. As well as it can decrease in case of upload speed problems. If no free memory is available, all chunks will equal minimal_chunk_size.

      Deleting files

      Files you delete with rclone will end up in Trash and be stored there for 30 days. Quatrix also provides an API to permanently delete files and an API to empty the Trash so that you can remove files permanently from your account.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to quatrix (Quatrix by Maytech).

      --quatrix-api-key

      API key for accessing Quatrix account

      @@ -30186,7 +31172,7 @@ y/e/d> y
    17. Type: string
    18. Required: true
      19. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to quatrix (Quatrix by Maytech).

      --quatrix-encoding

      The encoding for the backend.

      @@ -30263,7 +31249,7 @@ y/e/d> y

      rclone interacts with Sia network by talking to the Sia daemon via HTTP API which is usually available on port 9980. By default you will run the daemon locally on the same computer so it's safe to leave the API password blank (the API URL will be http://127.0.0.1:9980 making external access impossible).

      However, if you want to access Sia daemon running on another node, for example due to memory constraints or because you want to share single daemon between several rclone and Sia-UI instances, you'll need to make a few more provisions: - Ensure you have Sia daemon installed directly or in a docker container because Sia-UI does not support this mode natively. - Run it on externally accessible port, for example provide --api-addr :9980 and --disable-api-security arguments on the daemon command line. - Enforce API password for the siad daemon via environment variable SIA_API_PASSWORD or text file named apipassword in the daemon directory. - Set rclone backend option api_password taking it from above locations.

      Notes: 1. If your wallet is locked, rclone cannot unlock it automatically. You should either unlock it in advance by using Sia-UI or via command line siac wallet unlock. Alternatively you can make siad unlock your wallet automatically upon startup by running it with environment variable SIA_WALLET_PASSWORD. 2. If siad cannot find the SIA_API_PASSWORD variable or the apipassword file in the SIA_DIR directory, it will generate a random password and store in the text file named apipassword under YOUR_HOME/.sia/ directory on Unix or C:\Users\YOUR_HOME\AppData\Local\Sia\apipassword on Windows. Remember this when you configure password in rclone. 3. The only way to use siad without API password is to run it on localhost with command line argument --authorize-api=false, but this is insecure and strongly discouraged.

      -

      Configuration

      +

      Configuration

      Here is an example of how to make a sia remote called mySia. First, run:

       rclone config

      This will guide you through an interactive setup process:

      @@ -30323,7 +31309,7 @@ y/e/d> y
    19. Upload a local directory to the Sia directory called backup
      20. rclone copy /home/source mySia:backup
      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to sia (Sia Decentralized Cloud).

      --sia-api-url

      Sia daemon API URL, like http://sia.daemon.host:9980.

      @@ -30346,7 +31332,7 @@ y/e/d> y
    20. Type: string
    21. Required: false
      22. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to sia (Sia Decentralized Cloud).

      --sia-user-agent

      Siad User Agent

      @@ -30377,7 +31363,7 @@ y/e/d> y
    22. Type: string
    23. Required: false
      24. -

      Limitations

      +

      Limitations

      Paths are specified as remote:container (or remote: for the lsd command.) You may put subdirectories in too, e.g. remote:container/path/to/dir.

      -

      Configuration

      +

      Configuration

      Here is an example of making a swift configuration. First run

      rclone config

      This will guide you through an interactive setup process.

      @@ -30535,11 +31521,11 @@ rclone lsd myremote:

      --update and --use-server-modtime

      As noted below, the modified time is stored on metadata on the object. It is used by default for all operations that require checking the time a file was last updated. It allows rclone to treat the remote more like a true filesystem, but it is inefficient because it requires an extra API call to retrieve the metadata.

      For many operations, the time the object was last uploaded to the remote is sufficient to determine if it is "dirty". By using --update along with --use-server-modtime, you can avoid the extra API call and simply upload files whose local modtime is newer than the time it was last uploaded.

      -

      Modification times and hashes

      +

      Modification times and hashes

      The modified time is stored as metadata on the object as X-Object-Meta-Mtime as floating point since the epoch accurate to 1 ns.

      This is a de facto standard (used in the official python-swiftclient amongst others) for storing the modification time for an object.

      The MD5 hash algorithm is supported.

      -

      Restricted filename characters

      +

      Restricted filename characters

      @@ -30562,7 +31548,7 @@ rclone lsd myremote:

      Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to swift (OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)).

      --swift-env-auth

      Get swift credentials from environment variables in standard OpenStack form.

      @@ -30800,7 +31786,7 @@ rclone lsd myremote: -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to swift (OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)).

      --swift-leave-parts-on-error

      If true avoid calling abort upload on a failure.

      @@ -30812,6 +31798,30 @@ rclone lsd myremote:
    24. Type: bool
    25. Default: false
      26. +

      --swift-fetch-until-empty-page

      +

      When paginating, always fetch unless we received an empty page.

      +

      Consider using this option if rclone listings show fewer objects than expected, or if repeated syncs copy unchanged objects.

      +

      It is safe to enable this, but rclone may make more API calls than necessary.

      +

      This is one of a pair of workarounds to handle implementations of the Swift API that do not implement pagination as expected. See also "partial_page_fetch_threshold".

      +

      Properties:

      + +

      --swift-partial-page-fetch-threshold

      +

      When paginating, fetch if the current page is within this percentage of the limit.

      +

      Consider using this option if rclone listings show fewer objects than expected, or if repeated syncs copy unchanged objects.

      +

      It is safe to enable this, but rclone may make more API calls than necessary.

      +

      This is one of a pair of workarounds to handle implementations of the Swift API that do not implement pagination as expected. See also "fetch_until_empty_page".

      +

      Properties:

      +

      --swift-chunk-size

      Above this size files will be chunked.

      Above this size files will be chunked into a a _segments container or a .file-segments directory. (See the use_segments_container option for more info). Default for this is 5 GiB which is its maximum value, which means only files above this size will be chunked.

      @@ -30880,7 +31890,7 @@ rclone lsd myremote:
    26. Type: string
    27. Required: false
      28. -

      Limitations

      +

      Limitations

      The Swift API doesn't return a correct MD5SUM for segmented files (Dynamic or Static Large Objects) so rclone won't check or use the MD5SUM for these.

      Troubleshooting

      Rclone gives Failed to create file system for "remote:": Bad Request

      @@ -30900,7 +31910,7 @@ rclone lsd myremote:

      pCloud

      Paths are specified as remote:path

      Paths may be as deep as required, e.g. remote:directory/subdirectory.

      -

      Configuration

      +

      Configuration

      The initial setup for pCloud involves getting a token from pCloud which you need to do in your browser. rclone config walks you through it.

      Here is an example of how to make a remote called remote. First run:

       rclone config
      @@ -30934,12 +31944,13 @@ If your browser doesn't open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -client_id = -client_secret = -token = {"access_token":"XXX","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"} --------------------- +Configuration complete. +Options: +- type: pcloud +- client_id: +- client_secret: +- token: {"access_token":"XXX","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -30953,10 +31964,10 @@ y/e/d> y 
      rclone ls remote:

      To copy a local directory to a pCloud directory called backup

      rclone copy /home/source remote:backup
      -

      Modification times and hashes

      +

      Modification times and hashes

      pCloud allows modification times to be set on objects accurate to 1 second. These will be used to detect whether objects need syncing or not. In order to set a Modification time pCloud requires the object be re-uploaded.

      pCloud supports MD5 and SHA1 hashes in the US region, and SHA1 and SHA256 hashes in the EU region, so you can use the --checksum flag.

      -

      Restricted filename characters

      +

      Restricted filename characters

      In addition to the default restricted characters set the following characters are also replaced:

      @@ -30979,13 +31990,13 @@ y/e/d> y

      Deleted files will be moved to the trash. Your subscription level will determine how long items stay in the trash. rclone cleanup can be used to empty the trash.

      Emptying the trash

      Due to an API limitation, the rclone cleanup command will only work if you set your username and password in the advanced options for this backend. Since we generally want to avoid storing user passwords in the rclone config file, we advise you to only set this up if you need the rclone cleanup command to work.

      -

      Root folder ID

      +

      Root folder ID

      You can set the root_folder_id for rclone. This is the directory (identified by its Folder ID) that rclone considers to be the root of your pCloud drive.

      Normally you will leave this blank and rclone will determine the correct root to use itself.

      However you can set this to restrict rclone to a specific folder hierarchy.

      In order to do this you will have to find the Folder ID of the directory you wish rclone to display. This will be the folder field of the URL when you open the relevant folder in the pCloud web interface.

      So if the folder you want rclone to use has a URL which looks like https://my.pcloud.com/#page=filemanager&folder=5xxxxxxxx8&tpl=foldergrid in the browser, then you use 5xxxxxxxx8 as the root_folder_id in the config.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to pcloud (Pcloud).

      --pcloud-client-id

      OAuth Client Id.

      @@ -31007,7 +32018,7 @@ y/e/d> y
    28. Type: string
    29. Required: false
      30. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to pcloud (Pcloud).

      --pcloud-token

      OAuth Access Token as a JSON blob.

      @@ -31110,7 +32121,7 @@ y/e/d> y

      PikPak

      PikPak is a private cloud drive.

      Paths are specified as remote:path, and may be as deep as required, e.g. remote:directory/subdirectory.

      -

      Configuration

      +

      Configuration

      Here is an example of making a remote for PikPak.

      First run:

       rclone config
      @@ -31163,10 +32174,10 @@ y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y -

      Modification times and hashes

      +

      Modification times and hashes

      PikPak keeps modification times on objects, and updates them when uploading objects, but it does not support changing only the modification time

      The MD5 hash algorithm is supported.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to pikpak (PikPak).

      --pikpak-user

      Pikpak username.

      @@ -31187,7 +32198,7 @@ y/e/d> y
    30. Type: string
    31. Required: true
      32. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to pikpak (PikPak).

      --pikpak-client-id

      OAuth Client Id.

      @@ -31350,15 +32361,174 @@ rclone backend decompress pikpak:dirpath {filename} -o delete-src-file

      -

      Limitations

      +

      Limitations

      Hashes may be empty

      PikPak supports MD5 hash, but sometimes given empty especially for user-uploaded files.

      Deleted files still visible with trashed-only

      Deleted files will still be visible with --pikpak-trashed-only even after the trash emptied. This goes away after few days.

      +

      Pixeldrain

      +

      This is the backend for Pixeldrain's premium filesystem feature. This is not the same as pixeldrain's free file sharing product. The filesystem requires either a Pro subscription or the Prepaid plan. More information on subscriptions.

      +

      An overview of the filesystem's features and limitations is available in the filesystem guide on pixeldrain.

      +

      Usage with account

      +

      To use the personal filesystem you will need a pixeldrain account and either the Prepaid plan or one of the Patreon-based subscriptions. After registering and subscribing, your personal filesystem will be available at this link: https://pixeldrain.com/d/me.

      +

      Go to the API keys page on your account and generate a new API key for rclone. Then run rclone config and use the API key to create a new backend.

      +

      Example:

      +
      No remotes found, make a new one?
+n) New remote
+d) Delete remote
+c) Copy remote
+s) Set configuration password
+q) Quit config
+n/d/c/s/q> n
+
+Enter name for new remote.
+name> pixeldrainfs
+
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+...
+XX / Pixeldrain Filesystem
+   \ (pixeldrain)
+...
+Storage> pixeldrain
+
+Option api_key.
+API key for your pixeldrain account.
+Found on https://pixeldrain.com/user/api_keys.
+Enter a value. Press Enter to leave empty.
+api_key> b1bb1e81-9b7b-406b-986a-c9b20be76e15
+
+Option directory_id.
+Root of the filesystem to use. Set to 'me' to use your personal filesystem.
+Set to a shared directory ID to use a shared directory.
+Enter a string value. Press Enter for the default (me).
+directory_id>
+
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n>
+
+Configuration complete.
+Options:
+- type: pixeldrain
+- api_key: b1bb1e81-9b7b-406b-986a-c9b20be76e15
+Keep this "pixeldrainfs" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d>
+
+Current remotes:
+
+Name                 Type
+====                 ====
+pixeldrainfs         pixeldrain
+
+e) Edit existing remote
+n) New remote
+d) Delete remote
+r) Rename remote
+c) Copy remote
+s) Set configuration password
+q) Quit config
+e/n/d/r/c/s/q> q
      +

      Usage without account

      +

      It is possible to gain read-only access to publicly shared directories through rclone. For this you only need a directory ID. The directory ID can be found in the URL of a shared directory, the URL will look like this https://pixeldrain.com/d/abcd1234 where abcd1234 is the directory ID. Directory IDs in your own filesystem can also be listed with the lsf command:

      +

      rclone lsf Pixeldrain: --dirs-only -Fpi

      +

      This will print directories in your Pixeldrain home directory and their public IDs.

      +

      Enter this directory ID in the rclone config and you will be able to access the directory.

      +

      Standard options

      +

      Here are the Standard options specific to pixeldrain (Pixeldrain Filesystem).

      +

      --pixeldrain-api-key

      +

      API key for your pixeldrain account. Found on https://pixeldrain.com/user/api_keys.

      +

      Properties:

      + +

      --pixeldrain-root-folder-id

      +

      Root of the filesystem to use.

      +

      Set to 'me' to use your personal filesystem. Set to a shared directory ID to use a shared directory.

      +

      Properties:

      + +

      Advanced options

      +

      Here are the Advanced options specific to pixeldrain (Pixeldrain Filesystem).

      +

      --pixeldrain-api-url

      +

      The API endpoint to connect to. In the vast majority of cases it's fine to leave this at default. It is only intended to be changed for testing purposes.

      +

      Properties:

      + +

      --pixeldrain-description

      +

      Description of the remote.

      +

      Properties:

      + +

      Metadata

      +

      Pixeldrain supports file modes and creation times.

      +

      Here are the possible system metadata items for the pixeldrain backend.

      +
      +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameHelpTypeExampleRead Only
      btimeTime of file birth (creation)RFC 33392006-01-02T15:04:05.999999999Z07:00N
      modeFile modeoctal, unix style755N
      mtimeTime of last modificationRFC 33392006-01-02T15:04:05.999999999Z07:00N
      +

      See the metadata docs for more info.

      premiumize.me

      Paths are specified as remote:path

      Paths may be as deep as required, e.g. remote:directory/subdirectory.

      -

      Configuration

      +

      Configuration

      The initial setup for premiumize.me involves getting a token from premiumize.me which you need to do in your browser. rclone config walks you through it.

      Here is an example of how to make a remote called remote. First run:

       rclone config
      @@ -31391,11 +32561,11 @@ If your browser doesn't open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = premiumizeme -token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2029-08-07T18:44:15.548915378+01:00"} --------------------- +Configuration complete. +Options: +- type: premiumizeme +- token: {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2029-08-07T18:44:15.548915378+01:00"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -31409,9 +32579,9 @@ y/e/d> 
      rclone ls remote:

      To copy a local directory to an premiumize.me directory called backup

      rclone copy /home/source remote:backup
      -

      Modification times and hashes

      +

      Modification times and hashes

      premiumize.me does not support modification times or hashes, therefore syncing will default to --size-only checking. Note that using --update will work.

      -

      Restricted filename characters

      +

      Restricted filename characters

      In addition to the default restricted characters set the following characters are also replaced:

      @@ -31435,7 +32605,7 @@ y/e/d>

      Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to premiumizeme (premiumize.me).

      --premiumizeme-client-id

      OAuth Client Id.

      @@ -31467,7 +32637,7 @@ y/e/d>
    32. Type: string
    33. Required: false
      34. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to premiumizeme (premiumize.me).

      --premiumizeme-token

      OAuth Access Token as a JSON blob.

      @@ -31517,7 +32687,7 @@ y/e/d>
    34. Type: string
    35. Required: false
      36. -

      Limitations

      +

      Limitations

      Note that premiumize.me is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

      premiumize.me file names can't have the \ or " characters in. rclone maps these to and from an identical looking unicode equivalents ＼ and ＂

      premiumize.me only supports filenames up to 255 characters in length.

      @@ -31561,12 +32731,12 @@ Option 2fa. Enter a value. Press Enter to leave empty. 2fa> 123456 Remote config --------------------- -[remote] -type = protondrive -user = you@protonmail.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: protondrive +- user: you@protonmail.com +- pass: *** ENCRYPTED *** +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -31579,18 +32749,18 @@ y/e/d> y 
      rclone ls remote:

      To copy a local directory to an Proton Drive directory called backup

      rclone copy /home/source remote:backup
      -

      Modification times and hashes

      +

      Modification times and hashes

      Proton Drive Bridge does not support updating modification times yet.

      The SHA1 hash algorithm is supported.

      -

      Restricted filename characters

      +

      Restricted filename characters

      Invalid UTF-8 bytes will be replaced, also left and right spaces will be removed (code reference)

      -

      Duplicated files

      +

      Duplicated files

      Proton Drive can not have two files with exactly the same name and path. If the conflict occurs, depending on the advanced config, the file might or might not be overwritten.

      Mailbox password

      Please set your mailbox password in the advanced config section.

      Caching

      The cache is currently built for the case when the rclone is the only instance performing operations to the mount point. The event system, which is the proton API system that provides visibility of what has changed on the drive, is yet to be implemented, so updates from other clients won’t be reflected in the cache. Thus, if there are concurrent clients accessing the same mount point, then we might have a problem with caching the stale data.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to protondrive (Proton Drive).

      --protondrive-username

      The username of your proton account

      @@ -31622,7 +32792,7 @@ y/e/d> y
    36. Type: string
    37. Required: false
      38. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to protondrive (Proton Drive).

      --protondrive-mailbox-password

      The mailbox password of your two-password proton account.

      @@ -31734,7 +32904,7 @@ y/e/d> y
    38. Type: string
    39. Required: false
      40. -

      Limitations

      +

      Limitations

      This backend uses the Proton-API-Bridge, which is based on go-proton-api, a fork of the official repo.

      There is no official API documentation available from Proton Drive. But, thanks to Proton open sourcing proton-go-api and the web, iOS, and Android client codebases, we don't need to completely reverse engineer the APIs by observing the web client traffic!

      proton-go-api provides the basic building blocks of API calls and error handling, such as 429 exponential back-off, but it is pretty much just a barebone interface to the Proton API. For example, the encryption and decryption of the Proton Drive file are not provided in this library.

      @@ -31742,7 +32912,7 @@ y/e/d> y

      put.io

      Paths are specified as remote:path

      put.io paths may be as deep as required, e.g. remote:directory/subdirectory.

      -

      Configuration

      +

      Configuration

      The initial setup for put.io involves getting a token from put.io which you need to do in your browser. rclone config walks you through it.

      Here is an example of how to make a remote called remote. First run:

       rclone config
      @@ -31807,7 +32977,7 @@ e/n/d/r/c/s/q> q 
      rclone ls remote:

      To copy a local directory to a put.io directory called backup

      rclone copy /home/source remote:backup
      -

      Restricted filename characters

      +

      Restricted filename characters

      In addition to the default restricted characters set the following characters are also replaced:

      @@ -31826,7 +32996,7 @@ e/n/d/r/c/s/q> q

      Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to putio (Put.io).

      --putio-client-id

      OAuth Client Id.

      @@ -31848,7 +33018,7 @@ e/n/d/r/c/s/q> q
    40. Type: string
    41. Required: false
      42. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to putio (Put.io).

      --putio-token

      OAuth Access Token as a JSON blob.

      @@ -31898,7 +33068,7 @@ e/n/d/r/c/s/q> q
    42. Type: string
    43. Required: false
      44. -

      Limitations

      +

      Limitations

      put.io has rate limiting. When you hit a limit, rclone automatically retries after waiting the amount of time requested by the server.

      If you want to avoid ever hitting these limits, you may use the --tpslimit flag with a low number. Note that the imposed limits may be different for different operations, and may change over time.

      Proton Drive

      @@ -31941,12 +33111,12 @@ Option 2fa. Enter a value. Press Enter to leave empty. 2fa> 123456 Remote config --------------------- -[remote] -type = protondrive -user = you@protonmail.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: protondrive +- user: you@protonmail.com +- pass: *** ENCRYPTED *** +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -31959,18 +33129,18 @@ y/e/d> y 
      rclone ls remote:

      To copy a local directory to an Proton Drive directory called backup

      rclone copy /home/source remote:backup
      -

      Modification times and hashes

      +

      Modification times and hashes

      Proton Drive Bridge does not support updating modification times yet.

      The SHA1 hash algorithm is supported.

      -

      Restricted filename characters

      +

      Restricted filename characters

      Invalid UTF-8 bytes will be replaced, also left and right spaces will be removed (code reference)

      -

      Duplicated files

      +

      Duplicated files

      Proton Drive can not have two files with exactly the same name and path. If the conflict occurs, depending on the advanced config, the file might or might not be overwritten.

      Mailbox password

      Please set your mailbox password in the advanced config section.

      Caching

      The cache is currently built for the case when the rclone is the only instance performing operations to the mount point. The event system, which is the proton API system that provides visibility of what has changed on the drive, is yet to be implemented, so updates from other clients won’t be reflected in the cache. Thus, if there are concurrent clients accessing the same mount point, then we might have a problem with caching the stale data.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to protondrive (Proton Drive).

      --protondrive-username

      The username of your proton account

      @@ -32002,7 +33172,7 @@ y/e/d> y
    44. Type: string
    45. Required: false
      46. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to protondrive (Proton Drive).

      --protondrive-mailbox-password

      The mailbox password of your two-password proton account.

      @@ -32114,14 +33284,14 @@ y/e/d> y
    46. Type: string
    47. Required: false
      48. -

      Limitations

      +

      Limitations

      This backend uses the Proton-API-Bridge, which is based on go-proton-api, a fork of the official repo.

      There is no official API documentation available from Proton Drive. But, thanks to Proton open sourcing proton-go-api and the web, iOS, and Android client codebases, we don't need to completely reverse engineer the APIs by observing the web client traffic!

      proton-go-api provides the basic building blocks of API calls and error handling, such as 429 exponential back-off, but it is pretty much just a barebone interface to the Proton API. For example, the encryption and decryption of the Proton Drive file are not provided in this library.

      The Proton-API-Bridge, attempts to bridge the gap, so rclone can be built on top of this quickly. This codebase handles the intricate tasks before and after calling Proton APIs, particularly the complex encryption scheme, allowing developers to implement features for other software on top of this codebase. There are likely quite a few errors in this library, as there isn't official documentation available.

      Seafile

      This is a backend for the Seafile storage service: - It works with both the free community edition or the professional edition. - Seafile versions 6.x, 7.x, 8.x and 9.x are all supported. - Encrypted libraries are also supported. - It supports 2FA enabled users - Using a Library API Token is not supported

      -

      Configuration

      +

      Configuration

      There are two distinct modes you can setup your remote: - you point your remote to the root of the server, meaning you don't specify a library during the configuration: Paths are specified as remote:library. You may put subdirectories in too, e.g. remote:library/path/to/dir. - you point your remote to a specific library during the configuration: Paths are specified as remote:path/to/dir. This is the recommended mode when using encrypted libraries. (This mode is possibly slightly faster than the root mode)

      Configuration in root mode

      Here is an example of making a seafile configuration for a user with no two-factor authentication. First run

      @@ -32280,7 +33450,7 @@ y/e/d> y 
      rclone sync --interactive /home/local/directory seafile:

      --fast-list

      Seafile version 7+ supports --fast-list which allows you to use fewer transactions in exchange for more memory. See the rclone docs for more details. Please note this is not supported on seafile server version 6.x

      -

      Restricted filename characters

      +

      Restricted filename characters

      In addition to the default restricted characters set the following characters are also replaced:

      @@ -32322,7 +33492,7 @@ http://my.seafile.server/d/9ea2455f6f55478bbb0d/

      It has been actively developed using the seafile docker image of these versions: - 6.3.4 community edition - 7.0.5 community edition - 7.1.3 community edition - 9.0.10 community edition

      Versions below 6.0 are not supported. Versions between 6.0 and 6.3 haven't been tested and might not work properly.

      Each new version of rclone is automatically tested against the latest docker image of the seafile community server.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to seafile (seafile).

      --seafile-url

      URL of seafile host to connect to.

      @@ -32398,7 +33568,7 @@ http://my.seafile.server/d/9ea2455f6f55478bbb0d/
    48. Type: string
    49. Required: false
      50. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to seafile (seafile).

      --seafile-create-library

      Should rclone create a library if it doesn't exist.

      @@ -32439,7 +33609,7 @@ http://my.seafile.server/d/9ea2455f6f55478bbb0d/

      Paths are specified as remote:path. If the path does not begin with a / it is relative to the home directory of the user. An empty path remote: refers to the user's home directory. For example, rclone lsd remote: would list the home directory of the user configured in the rclone remote config (i.e /home/sftpuser). However, rclone lsd remote:/ would list the root directory for remote machine (i.e. /)

      Note that some SFTP servers will need the leading / - Synology is a good example of this. rsync.net and Hetzner, on the other hand, requires users to OMIT the leading /.

      Note that by default rclone will try to execute shell commands on the server, see shell access considerations.

      -

      Configuration

      +

      Configuration

      Here is an example of making an SFTP configuration. First run

      rclone config

      This will guide you through an interactive setup process.

      @@ -32475,14 +33645,15 @@ y/g/n> n Path to unencrypted PEM-encoded private key file, leave blank to use ssh-agent. key_file> Remote config --------------------- -[remote] -host = example.com -user = sftpuser -port = -pass = -key_file = --------------------- +Configuration complete. +Options: +- type: sftp +- host: example.com +- user: sftpuser +- port: +- pass: +- key_file: +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -32575,14 +33746,14 @@ known_hosts_file = ~/.ssh/known_hosts

      The options md5sum_command and sha1_command can be used to customize the command to be executed for calculation of checksums. You can for example set a specific path to where md5sum and sha1sum executables are located, or use them to specify some other tools that print checksums in compatible format. The value can include command-line arguments, or even shell script blocks as with PowerShell. Rclone has subcommands md5sum and sha1sum that use compatible format, which means if you have an rclone executable on the server it can be used. As mentioned above, they will be automatically picked up if found in PATH, but if not you can set something like /path/to/rclone md5sum as the value of option md5sum_command to make sure a specific executable is used.

      Remote checksumming is recommended and enabled by default. First time rclone is using a SFTP remote, if options md5sum_command or sha1_command are not set, it will check if any of the default commands for each of them, as described above, can be used. The result will be saved in the remote configuration, so next time it will use the same. Value none will be set if none of the default commands could be used for a specific algorithm, and this algorithm will not be supported by the remote.

      Disabling the checksumming may be required if you are connecting to SFTP servers which are not under your control, and to which the execution of remote shell commands is prohibited. Set the configuration option disable_hashcheck to true to disable checksumming entirely, or set shell_type to none to disable all functionality based on remote shell command execution.

      -

      Modification times and hashes

      +

      Modification times and hashes

      Modified times are stored on the server to 1 second precision.

      Modified times are used in syncing and are fully supported.

      Some SFTP servers disable setting/modifying the file modification time after upload (for example, certain configurations of ProFTPd with mod_sftp). If you are using one of these servers, you can set the option set_modtime = false in your RClone backend configuration to disable this behaviour.

      About command

      The about command returns the total space, free space, and used space on the remote for the disk of the specified path on the remote or, if not set, the disk of the root on the remote.

      SFTP usually supports the about command, but it depends on the server. If the server implements the vendor-specific VFS statistics extension, which is normally the case with OpenSSH instances, it will be used. If not, but the same login has access to a Unix shell, where the df command is available (e.g. in the remote's PATH), then this will be used instead. If the server shell is PowerShell, probably with a Windows OpenSSH server, rclone will use a built-in shell command (see shell access). If none of the above is applicable, about will fail.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to sftp (SSH/SFTP).

      --sftp-host

      SSH host to connect to.

      @@ -32624,7 +33795,11 @@ known_hosts_file = ~/.ssh/known_hosts

      --sftp-key-pem

      Raw PEM-encoded private key.

      -

      If specified, will override key_file parameter.

      +

      Note that this should be on a single line with line endings replaced with '', eg

      +
      key_pem = -----BEGIN RSA PRIVATE KEY-----\nMaMbaIXtE\n0gAMbMbaSsd\nMbaass\n-----END RSA PRIVATE KEY-----
      +

      This will generate the single line correctly:

      +
      awk '{printf "%s\\n", $0}' < ~/.ssh/id_rsa
      +

      If specified, it will override the key_file parameter.

      Properties:

      -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to sftp (SSH/SFTP).

      --sftp-known-hosts-file

      Optional path to known_hosts file.

      @@ -32945,9 +34120,9 @@ server_command = sudo /usr/libexec/openssh/sftp-server

      --sftp-connections

      Maximum number of SFTP simultaneous connections, 0 for unlimited.

      Note that setting this is very likely to cause deadlocks so it should be used with care.

      -

      If you are doing a sync or copy then make sure concurrency is one more than the sum of --transfers and --checkers.

      +

      If you are doing a sync or copy then make sure connections is one more than the sum of --transfers and --checkers.

      If you use --check-first then it just needs to be one more than the maximum of --checkers and --transfers.

      -

      So for concurrency 3 you'd use --checkers 2 --transfers 2 --check-first or --checkers 1 --transfers 1.

      +

      So for connections 3 you'd use --checkers 2 --transfers 2 --check-first or --checkers 1 --transfers 1.

      Properties:

      -

      Limitations

      +

      Limitations

      On some SFTP servers (e.g. Synology) the paths are different for SSH and SFTP so the hashes can't be calculated properly. For them using disable_hashcheck is a good idea.

      The only ssh agent supported under Windows is Putty's pageant.

      The Go SSH library disables the use of the aes128-cbc cipher by default, due to security concerns. This can be re-enabled on a per-connection basis by setting the use_insecure_cipher setting in the configuration file to true. Further details on the insecurity of this cipher can be found in this paper.

      @@ -33078,7 +34253,7 @@ server_command = sudo /usr/libexec/openssh/sftp-server

      The first path segment must be the name of the share, which you entered when you started to share on Windows. On smbd, it's the section title in smb.conf (usually in /etc/samba/) file. You can find shares by querying the root if you're unsure (e.g. rclone lsd remote:).

      You can't access to the shared printers from rclone, obviously.

      You can't use Anonymous access for logging in. You have to use the guest user with an empty password instead. The rclone client tries to avoid 8.3 names when uploading files by encoding trailing spaces and periods. Alternatively, the local backend on Windows can access SMB servers using UNC paths, by \\server\share. This doesn't apply to non-Windows OSes, such as Linux and macOS.

      -

      Configuration

      +

      Configuration

      Here is an example of making a SMB configuration.

      First run

      rclone config
      @@ -33153,7 +34328,7 @@ y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> d -

      Standard options

      +

      Standard options

      Here are the Standard options specific to smb (SMB / CIFS).

      --smb-host

      SMB server hostname to connect to.

      @@ -33214,7 +34389,7 @@ y/e/d> d
    50. Type: string
    51. Required: false
      52. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to smb (SMB / CIFS).

      --smb-idle-timeout

      Max time before closing idle connections.

      @@ -33325,7 +34500,7 @@ y/e/d> d
    52. S3 backend: secret encryption key is shared with the gateway
      53. -

      Configuration

      +

      Configuration

      To make a new Storj configuration you need one of the following: * Access Grant that someone else shared with you. * API Key of a Storj project you are a member of.

      Here is an example of how to make a remote called remote. First run:

       rclone config
      @@ -33359,11 +34534,11 @@ Access Grant. Enter a string value. Press Enter for the default (""). access_grant> your-access-grant-received-by-someone-else Remote config --------------------- -[remote] -type = storj -access_grant = your-access-grant-received-by-someone-else --------------------- +Configuration complete. +Options: +- type: storj +- access_grant: your-access-grant-received-by-someone-else +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -33410,19 +34585,19 @@ Encryption Passphrase. To access existing objects enter passphrase used for uplo Enter a string value. Press Enter for the default (""). passphrase> your-human-readable-encryption-passphrase Remote config --------------------- -[remote] -type = storj -satellite_address = 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777 -api_key = your-api-key-for-your-storj-project -passphrase = your-human-readable-encryption-passphrase -access_grant = the-access-grant-generated-from-the-api-key-and-passphrase --------------------- +Configuration complete. +Options: +- type: storj +- satellite_address: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777 +- api_key: your-api-key-for-your-storj-project +- passphrase: your-human-readable-encryption-passphrase +- access_grant: the-access-grant-generated-from-the-api-key-and-passphrase +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y -

      Standard options

      +

      Standard options

      Here are the Standard options specific to storj (Storj Decentralized Cloud Storage).

      --storj-provider

      Choose an authentication method.

      @@ -33501,7 +34676,7 @@ y/e/d> y
    53. Type: string
    54. Required: false
      55. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to storj (Storj Decentralized Cloud Storage).

      --storj-description

      Description of the remote.

      @@ -33566,7 +34741,7 @@ y/e/d> y
      rclone sync --interactive --progress remote-us:bucket/path/to/dir/ remote-europe:bucket/path/to/dir/

      Or even between another cloud storage and Storj.

      rclone sync --interactive --progress s3:bucket/path/to/dir/ storj:bucket/path/to/dir/
      -

      Limitations

      +

      Limitations

      rclone about is not supported by the rclone Storj backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

      See List of backends that do not support rclone about and rclone about

      Known issues

      @@ -33574,7 +34749,7 @@ y/e/d> y

      To fix these, please raise your system limits. You can do this issuing a ulimit -n 65536 just before you run rclone. To change the limits more permanently you can add this to your shell startup script, e.g. $HOME/.bashrc, or change the system-wide configuration, usually /etc/sysctl.conf and/or /etc/security/limits.conf, but please refer to your operating system manual.

      SugarSync

      SugarSync is a cloud service that enables active synchronization of files across computers and other devices for file backup, access, syncing, and sharing.

      -

      Configuration

      +

      Configuration

      The initial setup for SugarSync involves getting a token from SugarSync which you can do with rclone. rclone config walks you through it.

      Here is an example of how to make a remote called remote. First run:

       rclone config
      @@ -33619,11 +34794,11 @@ Remote config Username (email address)> nick@craig-wood.com Your Sugarsync password is only required during setup and will not be stored. password: --------------------- -[remote] -type = sugarsync -refresh_token = https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX --------------------- +Configuration complete. +Options: +- type: sugarsync +- refresh_token: https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -33639,15 +34814,15 @@ y/e/d> y

      Paths are specified as remote:path

      Paths may be as deep as required, e.g. remote:directory/subdirectory.

      NB you can't create files in the top level folder you have to create a folder, which rclone will create as a "Sync Folder" with SugarSync.

      -

      Modification times and hashes

      +

      Modification times and hashes

      SugarSync does not support modification times or hashes, therefore syncing will default to --size-only checking. Note that using --update will work as rclone can read the time files were uploaded.

      -

      Restricted filename characters

      +

      Restricted filename characters

      SugarSync replaces the default restricted characters set except for DEL.

      Invalid UTF-8 bytes will also be replaced, as they can't be used in XML strings.

      Deleting files

      Deleted files will be moved to the "Deleted items" folder by default.

      However you can supply the flag --sugarsync-hard-delete or set the config parameter hard_delete = true if you would like files to be deleted straight away.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to sugarsync (Sugarsync).

      --sugarsync-app-id

      Sugarsync App ID.

      @@ -33688,7 +34863,7 @@ y/e/d> y
    55. Type: bool
    56. Default: false
      57. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to sugarsync (Sugarsync).

      --sugarsync-refresh-token

      Sugarsync refresh token.

      @@ -33769,16 +34944,14 @@ y/e/d> y
    57. Type: string
    58. Required: false
      59. -

      Limitations

      +

      Limitations

      rclone about is not supported by the SugarSync backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

      See List of backends that do not support rclone about and rclone about

      -

      Tardigrade

      -

      The Tardigrade backend has been renamed to be the Storj backend. Old configuration files will continue to work.

      Uloz.to

      Paths are specified as remote:path

      Paths may be as deep as required, e.g. remote:directory/subdirectory.

      The initial setup for Uloz.to involves filling in the user credentials. rclone config walks you through it.

      -

      Configuration

      +

      Configuration

      Here is an example of how to make a remote called remote. First run:

       rclone config

      This will guide you through an interactive setup process:

      @@ -33838,10 +35011,10 @@ y/e/d> y
      rclone copy /home/source remote:backup

      User credentials

      The only reliable method is to authenticate the user using username and password. Uloz.to offers an API key as well, but it's reserved for the use of Uloz.to's in-house application and using it in different circumstances is unreliable.

      -

      Modification times and hashes

      +

      Modification times and hashes

      Uloz.to doesn't allow the user to set a custom modification time, or retrieve the hashes after upload. As a result, the integration uses a free form field the API provides to encode client-provided timestamps and hashes. Timestamps are stored with microsecond precision.

      A server calculated MD5 hash of the file is verified upon upload. Afterwards, the backend only serves the client-side calculated hashes. Hashes can also be retrieved upon creating a file download link, but it's impractical for list-like use cases.

      -

      Restricted filename characters

      +

      Restricted filename characters

      In addition to the default restricted characters set the following characters are also replaced:

      @@ -33865,13 +35038,13 @@ y/e/d> y

      Deleting files

      By default, files are moved to the recycle bin whereas folders are deleted immediately. Trashed files are permanently deleted after 30 days in the recycle bin.

      Emptying the trash is currently not implemented in rclone.

      -

      Root folder ID

      +

      Root folder ID

      You can set the root_folder_slug for rclone. This is the folder (identified by its Folder slug) that rclone considers to be the root of your Uloz.to drive.

      Normally you will leave this blank and rclone will determine the correct root to use itself. However you can set this to restrict rclone to a specific folder hierarchy.

      In order to do this you will have to find the Folder slug of the folder you wish to use as root. This will be the last segment of the URL when you open the relevant folder in the Uloz.to web interface.

      For example, for exploring a folder with URL https://uloz.to/fm/my-files/foobar, foobar should be used as the root slug.

      root_folder_slug can be used alongside a specific path in the remote path. For example, if your remote's root_folder_slug corresponds to /foo/bar, remote:baz/qux will refer to ABSOLUTE_ULOZTO_ROOT/foo/bar/baz/qux.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to ulozto (Uloz.to).

      --ulozto-app-token

      The application token identifying the app. An app API key can be either found in the API doc https://uloz.to/upload-resumable-api-beta or obtained from customer service.

      @@ -33901,7 +35074,7 @@ y/e/d> y
    59. Type: string
    60. Required: false
      61. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to ulozto (Uloz.to).

      --ulozto-root-folder-slug

      If set, rclone will use this folder as the root folder for all operations. For example, if the slug identifies 'foo/bar/', 'ulozto:baz' is equivalent to 'ulozto:foo/bar/baz' without any root slug set.

      @@ -33940,7 +35113,7 @@ y/e/d> y
    61. Type: string
    62. Required: false
      63. -

      Limitations

      +

      Limitations

      Uloz.to file names can't have the \ character in. rclone maps this to and from an identical looking unicode equivalent ＼ (U+FF3C Fullwidth Reverse Solidus).

      Uloz.to only supports filenames up to 255 characters in length.

      Uloz.to rate limits access to the API, but exact details are undisclosed. Practical testing reveals that hitting the rate limit during normal use is very rare, although not impossible with higher number of concurrently uploaded files.

      @@ -33950,7 +35123,7 @@ y/e/d> y

      This is a Backend for Uptobox file storage service. Uptobox is closer to a one-click hoster than a traditional cloud storage provider and therefore not suitable for long term storage.

      Paths are specified as remote:path

      Paths may be as deep as required, e.g. remote:directory/subdirectory.

      -

      Configuration

      +

      Configuration

      To configure an Uptobox backend you'll need your personal api token. You'll find it in your account settings

      Here is an example of how to make a remote called remote with the default setup. First run:

      rclone config
      @@ -34004,9 +35177,9 @@ y/e/d> 
      rclone ls remote:

      To copy a local directory to an Uptobox directory called backup

      rclone copy /home/source remote:backup
      -

      Modification times and hashes

      +

      Modification times and hashes

      Uptobox supports neither modified times nor checksums. All timestamps will read as that set by --default-time.

      -

      Restricted filename characters

      +

      Restricted filename characters

      In addition to the default restricted characters set the following characters are also replaced:

      @@ -34030,7 +35203,7 @@ y/e/d>

      Invalid UTF-8 bytes will also be replaced, as they can't be used in XML strings.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to uptobox (Uptobox).

      --uptobox-access-token

      Your access token.

      @@ -34042,7 +35215,7 @@ y/e/d>
    63. Type: string
    64. Required: false
      65. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to uptobox (Uptobox).

      --uptobox-private

      Set to make uploaded files private

      @@ -34072,7 +35245,7 @@ y/e/d>
    65. Type: string
    66. Required: false
      67. -

      Limitations

      +

      Limitations

      Uptobox will delete inactive files that have not been accessed in 60 days.

      rclone about is not supported by this backend an overview of used space can however been seen in the uptobox web interface.

      Union

      @@ -34086,7 +35259,7 @@ y/e/d>

      Subfolders can be used in upstream remotes. Assume a union remote named backup with the remotes mydrive:private/backup. Invoking rclone mkdir backup:desktop is exactly the same as invoking rclone mkdir mydrive:private/backup/desktop.

      There is no special handling of paths containing .. segments. Invoking rclone mkdir backup:../desktop is exactly the same as invoking rclone mkdir mydrive:private/backup/../desktop.

      -

      Configuration

      +

      Configuration

      Here is an example of how to make a union called remote for local folders. First run:

       rclone config

      This will guide you through an interactive setup process:

      @@ -34120,11 +35293,11 @@ Cache time of usage and free space (in seconds). This option is only useful when Enter a signed integer. Press Enter for the default ("120"). cache_time> Remote config --------------------- -[remote] -type = union -upstreams = remote1:dir1 remote2:dir2 remote3:dir3 --------------------- +Configuration complete. +Options: +- type: union +- upstreams: remote1:dir1 remote2:dir2 remote3:dir3 +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -34319,7 +35492,7 @@ upstreams = /local:writeback remote:dir

      When files are written, they will be written to both remote:dir and /local.

      As many remotes as desired can be added to upstreams but there should only be one :writeback tag.

      Rclone does not manage the :writeback remote in any way other than writing files back to it. So if you need to expire old files or manage the size then you will have to do this yourself.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to union (Union merges the contents of several upstream fs).

      --union-upstreams

      List of space separated upstreams.

      @@ -34368,7 +35541,7 @@ upstreams = /local:writeback remote:dir
    67. Type: int
    68. Default: 120
      69. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to union (Union merges the contents of several upstream fs).

      --union-min-free-space

      Minimum viable free space for lfs/eplfs policies.

      @@ -34389,13 +35562,13 @@ upstreams = /local:writeback remote:dir
    69. Type: string
    70. Required: false
      71. -

      Metadata

      +

      Metadata

      Any metadata supported by the underlying remote is read and written.

      See the metadata docs for more info.

      WebDAV

      Paths are specified as remote:path

      Paths may be as deep as required, e.g. remote:directory/subdirectory.

      -

      Configuration

      +

      Configuration

      To configure the WebDAV remote you will need to have a URL for it, and a username and password. If you know what kind of system you are connecting to then rclone can enable extra features.

      Here is an example of how to make a remote called remote. First run:

       rclone config
      @@ -34449,15 +35622,15 @@ password: Bearer token instead of user/pass (e.g. a Macaroon) bearer_token> Remote config --------------------- -[remote] -type = webdav -url = https://example.com/remote.php/webdav/ -vendor = nextcloud -user = user -pass = *** ENCRYPTED *** -bearer_token = --------------------- +Configuration complete. +Options: +- type: webdav +- url: https://example.com/remote.php/webdav/ +- vendor: nextcloud +- user: user +- pass: *** ENCRYPTED *** +- bearer_token: +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -34469,10 +35642,10 @@ y/e/d> y 
      rclone ls remote:

      To copy a local directory to an WebDAV directory called backup

      rclone copy /home/source remote:backup
      -

      Modification times and hashes

      +

      Modification times and hashes

      Plain WebDAV does not support modified times. However when used with Fastmail Files, Owncloud or Nextcloud rclone will support modified times.

      Likewise plain WebDAV does not support hashes, however when used with Fastmail Files, Owncloud or Nextcloud rclone will support SHA1 and MD5 hashes. Depending on the exact version of Owncloud or Nextcloud hashes may appear on all objects, or only on objects which had a hash uploaded with them.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to webdav (WebDAV).

      --webdav-url

      URL of http host to connect to.

      @@ -34553,7 +35726,7 @@ y/e/d> y
    71. Type: string
    72. Required: false
      73. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to webdav (WebDAV).

      --webdav-bearer-token-command

      Command to run to get a bearer token.

      @@ -34626,6 +35799,15 @@ y/e/d> y
    73. Type: bool
    74. Default: false
      75. +

      --webdav-unix-socket

      +

      Path to a unix domain socket to dial to, instead of opening a TCP connection directly

      +

      Properties:

      +

      --webdav-description

      Description of the remote.

      Properties:

      @@ -34713,7 +35895,7 @@ vendor = other bearer_token_command = oidc-token XDC

      Yandex Disk

      Yandex Disk is a cloud storage solution created by Yandex.

      -

      Configuration

      +

      Configuration

      Here is an example of making a yandex configuration. First run

      rclone config

      This will guide you through an interactive setup process:

      @@ -34745,12 +35927,13 @@ If your browser doesn't open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -client_id = -client_secret = -token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"OAuth","expiry":"2016-12-29T12:27:11.362788025Z"} --------------------- +Configuration complete. +Options: +- type: yandex +- client_id: +- client_secret: +- token: {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"OAuth","expiry":"2016-12-29T12:27:11.362788025Z"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -34767,17 +35950,17 @@ y/e/d> y

      Sync /home/local/directory to the remote path, deleting any excess files in the path.

      rclone sync --interactive /home/local/directory remote:directory

      Yandex paths may be as deep as required, e.g. remote:directory/subdirectory.

      -

      Modification times and hashes

      +

      Modification times and hashes

      Modified times are supported and are stored accurate to 1 ns in custom metadata called rclone_modified in RFC3339 with nanoseconds format.

      The MD5 hash algorithm is natively supported by Yandex Disk.

      Emptying Trash

      If you wish to empty your trash you can use the rclone cleanup remote: command which will permanently delete all your trashed files. This command does not take any path arguments.

      Quota information

      To view your current quota you can use the rclone about remote: command which will display your usage limit (quota) and the current usage.

      -

      Restricted filename characters

      +

      Restricted filename characters

      The default restricted characters set are replaced.

      Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to yandex (Yandex Disk).

      --yandex-client-id

      OAuth Client Id.

      @@ -34799,7 +35982,7 @@ y/e/d> y
    75. Type: string
    76. Required: false
      77. -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to yandex (Yandex Disk).

      --yandex-token

      OAuth Access Token as a JSON blob.

      @@ -34849,6 +36032,15 @@ y/e/d> y
    77. Type: Encoding
    78. Default: Slash,Del,Ctl,InvalidUtf8,Dot
      79. +

      --yandex-spoof-ua

      +

      Set the user agent to match an official version of the yandex disk client. May help with upload performance.

      +

      Properties:

      +

      --yandex-description

      Description of the remote.

      Properties:

      @@ -34858,13 +36050,13 @@ y/e/d> y
    79. Type: string
    80. Required: false
      81. -

      Limitations

      +

      Limitations

      When uploading very large files (bigger than about 5 GiB) you will need to increase the --timeout parameter. This is because Yandex pauses (perhaps to calculate the MD5SUM for the entire file) before returning confirmation that the file has been uploaded. The default handling of timeouts in rclone is to assume a 5 minute pause is an error and close the connection - you'll see net/http: timeout awaiting response headers errors in the logs if this is happening. Setting the timeout to twice the max size of file in GiB should be enough, so if you want to upload a 30 GiB file set a timeout of 2 * 30 = 60m, that is --timeout 60m.

      Having a Yandex Mail account is mandatory to use the Yandex.Disk subscription. Token generation will work without a mail account, but Rclone won't be able to complete any actions.

      [403 - DiskUnsupportedUserAccountTypeError] User account type is not supported.

      Zoho Workdrive

      Zoho WorkDrive is a cloud storage solution created by Zoho.

      -

      Configuration

      +

      Configuration

      Here is an example of making a zoho configuration. First run

      rclone config

      This will guide you through an interactive setup process:

      @@ -34915,12 +36107,12 @@ Choose a number from below, or type in your own value 1 / General \ "4u2869d2aa6fca04f4f2f896b6539243b85b1" Enter a Workspace ID> 1 --------------------- -[remote] -type = zoho -token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Zoho-oauthtoken","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2020-10-12T00:54:52.370275223+02:00"} -root_folder_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --------------------- +Configuration complete. +Options: +- type: zoho +- token: {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Zoho-oauthtoken","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2020-10-12T00:54:52.370275223+02:00"} +- root_folder_id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -34937,14 +36129,14 @@ y/e/d>

      Sync /home/local/directory to the remote path, deleting any excess files in the path.

      rclone sync --interactive /home/local/directory remote:directory

      Zoho paths may be as deep as required, eg remote:directory/subdirectory.

      -

      Modification times and hashes

      +

      Modification times and hashes

      Modified times are currently not supported for Zoho Workdrive

      No hash algorithms are supported.

      Usage information

      To view your current quota you can use the rclone about remote: command which will display your current usage.

      -

      Restricted filename characters

      +

      Restricted filename characters

      Only control characters and invalid UTF-8 are replaced. In addition most Unicode full-width characters are not supported at all and will be removed from filenames during upload.

      -

      Standard options

      +

      Standard options

      Here are the Standard options specific to zoho (Zoho).

      --zoho-client-id

      OAuth Client Id.

      @@ -35003,7 +36195,7 @@ y/e/d> -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to zoho (Zoho).

      --zoho-token

      OAuth Access Token as a JSON blob.

      @@ -35065,7 +36257,7 @@ y/e/d>

      Local paths are specified as normal filesystem paths, e.g. /path/to/wherever, so

      rclone sync --interactive /home/source /tmp/destination

      Will sync /home/source to /tmp/destination.

      -

      Configuration

      +

      Configuration

      For consistencies sake one can also configure a remote of type local in the config file, and access the local filesystem using rclone remote paths, e.g. remote:path/to/wherever, but it is probably easier not to.

      Modification times

      Rclone reads and writes the modification times using an accuracy determined by the OS. Typically this is 1ns on Linux, 10 ns on Windows and 1 Second on OS X.

      @@ -35339,8 +36531,7 @@ y/e/d>

      Invalid UTF-8 bytes will also be replaced, as they can't be converted to UTF-16.

      Paths on Windows

      -

      On Windows there are many ways of specifying a path to a file system resource. Local paths can be absolute, like C:\path\to\wherever, or relative, like ..\wherever. Network paths in UNC format, \\server\share, are also supported. Path separator can be either \ (as in C:\path\to\wherever) or / (as in C:/path/to/wherever). Length of these paths are limited to 259 characters for files and 247 characters for directories, but there is an alternative extended-length path format increasing the limit to (approximately) 32,767 characters. This format requires absolute paths and the use of prefix \\?\, e.g. \\?\D:\some\very\long\path. For convenience rclone will automatically convert regular paths into the corresponding extended-length paths, so in most cases you do not have to worry about this (read more below).

      -

      Note that Windows supports using the same prefix \\?\ to specify path to volumes identified by their GUID, e.g. \\?\Volume{b75e2c83-0000-0000-0000-602f00000000}\some\path. This is not supported in rclone, due to an issue in go.

      +

      On Windows there are many ways of specifying a path to a file system resource. Local paths can be absolute, like C:\path\to\wherever, or relative, like ..\wherever. Network paths in UNC format, \\server\share, are also supported. Path separator can be either \ (as in C:\path\to\wherever) or / (as in C:/path/to/wherever). Length of these paths are limited to 259 characters for files and 247 characters for directories, but there is an alternative extended-length path format increasing the limit to (approximately) 32,767 characters. This format requires absolute paths and the use of prefix \\?\, e.g. \\?\D:\some\very\long\path. For convenience rclone will automatically convert regular paths into the corresponding extended-length paths, so in most cases you do not have to worry about this (read more below). Using the same prefix \\?\ it is also possible to specify path to volumes identified by their GUID, e.g. \\?\Volume{b75e2c83-0000-0000-0000-602f00000000}\some\path.

      Long paths

      Rclone handles long paths automatically, by converting all paths to extended-length path format, which allows paths up to 32,767 characters.

      This conversion will ensure paths are absolute and prefix them with the \\?\. This is why you will see that your paths, for instance .\files is shown as path \\?\C:\files in the output, and \\server\share as \\?\UNC\server\share.

      @@ -35442,7 +36633,7 @@ $ tree /tmp/c 0 file2

      NB Rclone (like most unix tools such as du, rsync and tar) treats a bind mount to the same device as being on the same filesystem.

      NB This flag is only available on Unix based systems. On systems where it isn't supported (e.g. Windows) it will be ignored.

      -

      Advanced options

      +

      Advanced options

      Here are the Advanced options specific to local (Local Disk).

      --local-nounc

      Disable UNC (long path names) conversion on Windows.

      @@ -35566,6 +36757,19 @@ $ tree /tmp/c
    81. Type: bool
    82. Default: false
      83. +

      --local-no-clone

      +

      Disable reflink cloning for server-side copies.

      +

      Normally, for local-to-local transfers, rclone will "clone" the file when possible, and fall back to "copying" only when cloning is not supported.

      +

      Cloning creates a shallow copy (or "reflink") which initially shares blocks with the original file. Unlike a "hardlink", the two files are independent and neither will affect the other if subsequently modified.

      +

      Cloning is usually preferable to copying, as it is much faster and is deduplicated by default (i.e. having two identical files does not consume more storage than having just one.) However, for use cases where data redundancy is preferable, --local-no-clone can be used to disable cloning and force "deep" copies.

      +

      Currently, cloning is only supported when using APFS on macOS (support for other platforms may be added in the future.)

      +

      Properties:

      +

      --local-no-preallocate

      Disable preallocation of disk space for transferred files.

      Preallocation of disk space helps prevent filesystem fragmentation. However, some virtual filesystem layers (such as Google Drive File Stream) may incorrectly set the actual file size equal to the preallocated space, causing checksum and file size checks to fail. Use this flag to disable preallocation.

      @@ -35653,7 +36857,7 @@ $ tree /tmp/c
    83. Type: string
    84. Required: false
      85. -

      Metadata

      +

      Metadata

      Depending on which OS is in use the local backend may return only some of the system metadata. Setting system metadata is supported on all OSes but setting user metadata is only supported on linux, freebsd, netbsd, macOS and Solaris. It is not supported on Windows yet (see pkg/attrs#47).

      User metadata is stored as extended attributes (which may not be supported by all file systems) under the "user.*" prefix.

      Metadata is supported on files and directories.

      @@ -35745,6 +36949,196 @@ $ tree /tmp/c
    85. "error": return an error based on option value

      86. Changelog

      +

      v1.68.0 - 2024-09-08

      +

      See commits

      +

      v1.67.0 - 2024-06-14

      See commits

      Bugs and Limitations

      -

      Limitations

      +

      Limitations

      Directory timestamps aren't preserved on some backends

      As of v1.66, rclone supports syncing directory modtimes, if the backend supports it. Some backends do not support it -- see overview for a complete list. Additionally, note that empty directories are not synced by default (this can be enabled with --create-empty-src-dirs.)

      Rclone struggles with millions of files in a directory/bucket

      @@ -43704,7 +45098,7 @@ yyyy/mm/dd hh:mm:ss Fatal error: config failed to refresh token: failed to start

      Rclone is using too much memory or appears to have a memory leak

      Rclone is written in Go which uses a garbage collector. The default settings for the garbage collector mean that it runs when the heap size has doubled.

      However it is possible to tune the garbage collector to use less memory by setting GOGC to a lower value, say export GOGC=20. This will make the garbage collector work harder, reducing memory size at the expense of CPU usage.

      -

      The most common cause of rclone using lots of memory is a single directory with thousands or millions of files in. Rclone has to load this entirely into memory as rclone objects. Each rclone object takes 0.5k-1k of memory.

      +

      The most common cause of rclone using lots of memory is a single directory with millions of files in. Rclone has to load this entirely into memory as rclone objects. Each rclone object takes 0.5k-1k of memory. There is a workaround for this which involves a bit of scripting.

      Rclone changes fullwidth Unicode punctuation marks in file names

      For example: On a Windows system, you have a file with name Test：1.jpg, where ： is the Unicode fullwidth colon symbol. When using rclone to copy this to your Google Drive, you will notice that the file gets renamed to Test:1.jpg, where : is the regular (halfwidth) colon.

      The reason for such renames is the way rclone handles different restricted filenames on different cloud storage systems. It tries to avoid ambiguous file names as much and allow moving files between many cloud storage systems transparently, by replacing invalid characters with similar looking Unicode characters when transferring to one storage system, and replacing back again when transferring to a different storage system where the original characters are supported. When the same Unicode characters are intentionally used in file names, this replacement strategy leads to unwanted renames. Read more here.

      @@ -44585,6 +45979,30 @@ THE SOFTWARE.
    86. Michał Dzienisiewicz
    87. Florian Klink
    88. Bill Fraser
      89. +
    89. Thearas
      90. +
    90. Filipe Herculano
      91. +
    91. Russ Bubley
      92. +
    92. Paul Collins
      93. +
    93. Tomasz Melcer
      94. +
    94. itsHenry
      95. +
    95. Ke Wang
      96. +
    96. AThePeanut4
      97. +
    97. Tobias Markus
      98. +
    98. Ernie Hershey
      99. +
    99. Will Miles
      100. +
    100. David Seifert
      101. +
    101. Fornax
      102. +
    102. Sam Harrison
      103. +
    103. Péter Bozsó
      104. +
    104. Georg Welzel
      105. +
    105. John Oxley
      106. +
    106. Pawel Palucha
      107. +
    107. crystalstall
      108. +
    108. nipil
      109. +
    109. yuval-cloudinary
      110. +
    110. Mathieu Moreau
      111. +
    111. fsantagostinobietti
      112. +
    112. Oleg Kunitsyn

      113. Contact the rclone project

      Forum

      diff --git a/MANUAL.md b/MANUAL.md index 27e5b2153..11e520ba8 100644 --- a/MANUAL.md +++ b/MANUAL.md @@ -1,6 +1,6 @@ % rclone(1) User Manual % Nick Craig-Wood -% Jun 14, 2024 +% Sep 08, 2024 # Rclone syncs your files to cloud storage @@ -119,7 +119,9 @@ WebDAV or S3, that work out of the box.) - Dropbox - Enterprise File Fabric - Fastmail Files +- Files.com - FTP +- Gofile - Google Cloud Storage - Google Drive - Google Photos @@ -158,6 +160,7 @@ WebDAV or S3, that work out of the box.) - pCloud - Petabox - PikPak +- Pixeldrain - premiumize.me - put.io - Proton Drive @@ -653,6 +656,11 @@ which corresponds to `~/go/bin/rclone` by default). go install github.com/rclone/rclone@latest ``` +In some situations, rclone executable size might be too big for deployment +in very restricted environments when all backends with large SDKs are included. +To limit binary size unused backends can be commented out in `backends/all/all.go` +and unused commands in `cmd/all/all.go` before building with `go build` or `make` + ## Ansible installation {#ansible} This can be done with [Stefan Weichinger's ansible @@ -872,12 +880,15 @@ See the following for detailed instructions for * [Digi Storage](https://rclone.org/koofr/#digi-storage) * [Dropbox](https://rclone.org/dropbox/) * [Enterprise File Fabric](https://rclone.org/filefabric/) + * [Files.com](https://rclone.org/filescom/) * [FTP](https://rclone.org/ftp/) + * [Gofile](https://rclone.org/gofile/) * [Google Cloud Storage](https://rclone.org/googlecloudstorage/) * [Google Drive](https://rclone.org/drive/) * [Google Photos](https://rclone.org/googlephotos/) * [Hasher](https://rclone.org/hasher/) - to handle checksums for other remotes * [HDFS](https://rclone.org/hdfs/) + * [Hetzner Storage Box](https://rclone.org/sftp/#hetzner-storage-box) * [HiDrive](https://rclone.org/hidrive/) * [HTTP](https://rclone.org/http/) * [Internet Archive](https://rclone.org/internetarchive/) @@ -895,11 +906,13 @@ See the following for detailed instructions for * [Oracle Object Storage](https://rclone.org/oracleobjectstorage/) * [Pcloud](https://rclone.org/pcloud/) * [PikPak](https://rclone.org/pikpak/) + * [Pixeldrain](https://rclone.org/pixeldrain/) * [premiumize.me](https://rclone.org/premiumizeme/) * [put.io](https://rclone.org/putio/) * [Proton Drive](https://rclone.org/protondrive/) * [QingStor](https://rclone.org/qingstor/) * [Quatrix by Maytech](https://rclone.org/quatrix/) + * [rsync.net](https://rclone.org/sftp/#rsync-net) * [Seafile](https://rclone.org/seafile/) * [SFTP](https://rclone.org/sftp/) * [Sia](https://rclone.org/sia/) @@ -979,10 +992,9 @@ rclone config [flags] -h, --help help for config ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. * [rclone config create](https://rclone.org/commands/rclone_config_create/) - Create a new remote with name, type and options. @@ -990,6 +1002,7 @@ See the [global flags page](https://rclone.org/flags/) for global options not li * [rclone config disconnect](https://rclone.org/commands/rclone_config_disconnect/) - Disconnects user from remote * [rclone config dump](https://rclone.org/commands/rclone_config_dump/) - Dump the config file as JSON. * [rclone config edit](https://rclone.org/commands/rclone_config_edit/) - Enter an interactive configuration session. +* [rclone config encryption](https://rclone.org/commands/rclone_config_encryption/) - set, remove and check the encryption for the config file * [rclone config file](https://rclone.org/commands/rclone_config_file/) - Show path of configuration file in use. * [rclone config password](https://rclone.org/commands/rclone_config_password/) - Update password in an existing remote. * [rclone config paths](https://rclone.org/commands/rclone_config_paths/) - Show paths used for configuration, cache, temp etc. @@ -1007,7 +1020,6 @@ Copy files from source to dest, skipping identical files. ## Synopsis - Copy the source to the destination. Does not transfer files that are identical on source and destination, testing by size and modification time or MD5SUM. Doesn't delete files from the destination. If you @@ -1085,15 +1097,17 @@ rclone copy source:path dest:path [flags] -h, --help help for copy ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -1125,9 +1139,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -1135,9 +1149,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -1164,18 +1178,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -1185,7 +1197,6 @@ Make source and dest identical, modifying destination only. ## Synopsis - Sync the source to the destination, changing the destination only. Doesn't transfer files that are identical on source and destination, testing by size and modification time or MD5SUM. @@ -1294,15 +1305,17 @@ rclone sync source:path dest:path [flags] -t, --timeformat string Specify a custom time format, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -1334,9 +1347,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Sync Options +### Sync Options -Flags just used for `rclone sync`. +Flags used for sync commands ``` --backup-dir string Make backups into hierarchy based in DIR @@ -1353,9 +1366,9 @@ Flags just used for `rclone sync`. --track-renames-strategy string Strategies to use when synchronizing using track-renames hash|modtime|leaf (default "hash") ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -1363,9 +1376,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -1392,18 +1405,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -1413,7 +1424,6 @@ Move files from source to dest. ## Synopsis - Moves the contents of the source directory to the destination directory. Rclone will error if the source and destination overlap and the remote does not support a server-side directory move operation. @@ -1465,15 +1475,17 @@ rclone move source:path dest:path [flags] -h, --help help for move ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -1505,9 +1517,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -1515,9 +1527,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -1544,18 +1556,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -1565,7 +1575,6 @@ Remove the files in path. ## Synopsis - Remove the files in path. Unlike [purge](https://rclone.org/commands/rclone_purge/) it obeys include/exclude filters so can be used to selectively delete files. @@ -1605,10 +1614,12 @@ rclone delete remote:path [flags] --rmdirs rmdirs removes empty directories but leaves root intact ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -1616,9 +1627,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -1645,18 +1656,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -1666,7 +1675,6 @@ Remove the path and all of its contents. ## Synopsis - Remove the path and all of its contents. Note that this does not obey include/exclude filters - everything will be removed. Use the [delete](https://rclone.org/commands/rclone_delete/) command if you want to selectively @@ -1687,10 +1695,12 @@ rclone purge remote:path [flags] -h, --help help for purge ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -1698,9 +1708,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -1718,10 +1726,12 @@ rclone mkdir remote:path [flags] -h, --help help for mkdir ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -1729,9 +1739,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -1741,7 +1749,6 @@ Remove the empty directory at path. ## Synopsis - This removes empty directory given by path. Will not remove the path if it has any objects in it, not even empty subdirectories. Use command [rmdirs](https://rclone.org/commands/rclone_rmdirs/) (or [delete](https://rclone.org/commands/rclone_delete/) @@ -1760,10 +1767,12 @@ rclone rmdir remote:path [flags] -h, --help help for rmdir ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -1771,9 +1780,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -1783,7 +1790,6 @@ Checks the files in the source and destination match. ## Synopsis - Checks the files in the source and destination match. It compares sizes and hashes (MD5 or SHA1) and logs a report of files that don't match. It doesn't alter the source or destination. @@ -1847,18 +1853,20 @@ rclone check source:path dest:path [flags] --one-way Check one way only, source files must exist on remote ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Check Options +### Check Options -Flags used for `rclone check`. +Flags used for check commands ``` --max-backlog int Maximum number of objects in sync or check backlog (default 10000) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -1885,18 +1893,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -1906,7 +1912,6 @@ List the objects in the path with size and path. ## Synopsis - Lists the objects in the source path to standard output in a human readable format with size and path. Recurses by default. @@ -1952,10 +1957,12 @@ rclone ls remote:path [flags] -h, --help help for ls ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -1982,18 +1989,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2003,7 +2008,6 @@ List all directories/containers/buckets in the path. ## Synopsis - Lists the directories in the source path to standard output. Does not recurse by default. Use the `-R` flag to recurse. @@ -2060,10 +2064,12 @@ rclone lsd remote:path [flags] -R, --recursive Recurse into the listing ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -2090,18 +2096,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2111,7 +2115,6 @@ List the objects in path with modification time, size and path. ## Synopsis - Lists the objects in the source path to standard output in a human readable format with modification time, size and path. Recurses by default. @@ -2157,10 +2160,12 @@ rclone lsl remote:path [flags] -h, --help help for lsl ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -2187,18 +2192,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2208,7 +2211,6 @@ Produces an md5sum file for all the objects in the path. ## Synopsis - Produces an md5sum file for all the objects in the path. This is in the same format as the standard md5sum tool produces. @@ -2241,10 +2243,12 @@ rclone md5sum remote:path [flags] --output-file string Output hashsums to a file rather than the terminal ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -2271,18 +2275,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2292,7 +2294,6 @@ Produces an sha1sum file for all the objects in the path. ## Synopsis - Produces an sha1sum file for all the objects in the path. This is in the same format as the standard sha1sum tool produces. @@ -2328,10 +2329,12 @@ rclone sha1sum remote:path [flags] --output-file string Output hashsums to a file rather than the terminal ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -2358,18 +2361,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2379,7 +2380,6 @@ Prints the total size and number of objects in remote:path. ## Synopsis - Counts objects in the path and calculates the total size. Prints the result to standard output. @@ -2410,10 +2410,12 @@ rclone size remote:path [flags] --json Format output as JSON ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -2440,18 +2442,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2461,7 +2461,6 @@ Show the version number. ## Synopsis - Show the rclone version number, the go version, the build target OS and architecture, the runtime OS and kernel version and bitness, build tags and the type of executable (static or dynamic). @@ -2511,10 +2510,9 @@ rclone version [flags] -h, --help help for version ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2524,7 +2522,6 @@ Clean up the remote if possible. ## Synopsis - Clean up the remote if possible. Empty the trash or delete old file versions. Not supported by all remotes. @@ -2539,10 +2536,12 @@ rclone cleanup remote:path [flags] -h, --help help for cleanup ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -2550,9 +2549,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2562,8 +2559,6 @@ Interactively find duplicate filenames and delete/rename them. ## Synopsis - - By default `dedupe` interactively finds files with duplicate names and offers to delete all but one or rename them to be different. This is known as deduping by name. @@ -2682,10 +2677,12 @@ rclone dedupe [mode] remote:path [flags] -h, --help help for dedupe ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -2693,9 +2690,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2705,8 +2700,7 @@ Get quota information from the remote. ## Synopsis - -`rclone about` prints quota information about a remote to standard +Prints quota information about a remote to standard output. The output is typically used, free, quota and trash contents. E.g. Typical output from `rclone about remote:` is: @@ -2765,10 +2759,9 @@ rclone about remote: [flags] --json Format output as JSON ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2778,7 +2771,6 @@ Remote authorization. ## Synopsis - Remote authorization. Used to authorize a remote or headless rclone from a machine with a browser - use as instructed by rclone config. @@ -2800,10 +2792,9 @@ rclone authorize [flags] --template string The path to a custom Go template for generating HTML responses ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2813,7 +2804,6 @@ Run a backend-specific command. ## Synopsis - This runs a backend-specific command. The commands themselves (except for "help" and "features") are defined by the backends and you should see the backend docs for definitions. @@ -2853,10 +2843,12 @@ rclone backend remote:path [opts] [flags] -o, --option stringArray Option in the form name=value or name ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -2864,9 +2856,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2928,15 +2918,17 @@ rclone bisync remote1:path1 remote2:path2 [flags] --workdir string Use custom working dir - useful for testing. (default: {WORKDIR}) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -2968,9 +2960,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -2978,9 +2970,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -3007,9 +2999,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3019,8 +3009,7 @@ Concatenates any files and sends them to stdout. ## Synopsis - -rclone cat sends any files to standard output. +Sends any files to standard output. You can use it like this to output a single file @@ -3068,10 +3057,12 @@ rclone cat remote:path [flags] --tail int Only print the last N characters ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -3098,18 +3089,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3119,7 +3108,6 @@ Checks the files in the destination against a SUM file. ## Synopsis - Checks that hashsums of destination files match the SUM file. It compares hashes (MD5, SHA1, etc) and logs a report of files which don't match. It doesn't alter the file system. @@ -3176,10 +3164,12 @@ rclone checksum sumfile dst:path [flags] --one-way Check one way only, source files must exist on remote ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -3206,18 +3196,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3227,7 +3215,6 @@ Output completion script for a given shell. ## Synopsis - Generates a shell completion script for rclone. Run with `--help` to list the supported shells. @@ -3238,10 +3225,9 @@ Run with `--help` to list the supported shells. -h, --help help for completion ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. * [rclone completion bash](https://rclone.org/commands/rclone_completion_bash/) - Output bash completion script for rclone. @@ -3255,12 +3241,11 @@ Output bash completion script for rclone. ## Synopsis - Generates a bash shell autocompletion script for rclone. By default, when run without any arguments, - rclone genautocomplete bash + rclone completion bash the generated script will be written to @@ -3295,10 +3280,9 @@ rclone completion bash [output_file] [flags] -h, --help help for bash ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone completion](https://rclone.org/commands/rclone_completion/) - Output completion script for a given shell. @@ -3308,13 +3292,12 @@ Output fish completion script for rclone. ## Synopsis - Generates a fish autocompletion script for rclone. This writes to /etc/fish/completions/rclone.fish by default so will probably need to be run with sudo or as root, e.g. - sudo rclone genautocomplete fish + sudo rclone completion fish Logout and login again to use the autocompletion scripts, or source them directly @@ -3337,10 +3320,9 @@ rclone completion fish [output_file] [flags] -h, --help help for fish ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone completion](https://rclone.org/commands/rclone_completion/) - Output completion script for a given shell. @@ -3350,7 +3332,6 @@ Output powershell completion script for rclone. ## Synopsis - Generate the autocompletion script for powershell. To load completions in your current shell session: @@ -3373,10 +3354,9 @@ rclone completion powershell [output_file] [flags] -h, --help help for powershell ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone completion](https://rclone.org/commands/rclone_completion/) - Output completion script for a given shell. @@ -3386,13 +3366,12 @@ Output zsh completion script for rclone. ## Synopsis - Generates a zsh autocompletion script for rclone. This writes to /usr/share/zsh/vendor-completions/_rclone by default so will probably need to be run with sudo or as root, e.g. - sudo rclone genautocomplete zsh + sudo rclone completion zsh Logout and login again to use the autocompletion scripts, or source them directly @@ -3415,10 +3394,9 @@ rclone completion zsh [output_file] [flags] -h, --help help for zsh ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone completion](https://rclone.org/commands/rclone_completion/) - Output completion script for a given shell. @@ -3428,7 +3406,6 @@ Create a new remote with name, type and options. ## Synopsis - Create a new remote of `name` with `type` and options. The options should be passed in pairs of `key` `value` or as `key=value`. @@ -3548,10 +3525,9 @@ rclone config create name type [key value]* [flags] --state string State - use with --continue ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3569,10 +3545,9 @@ rclone config delete name [flags] -h, --help help for delete ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3582,7 +3557,6 @@ Disconnects user from remote ## Synopsis - This disconnects the remote: passed in to the cloud storage system. This normally means revoking the oauth token. @@ -3600,10 +3574,9 @@ rclone config disconnect remote: [flags] -h, --help help for disconnect ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3621,10 +3594,9 @@ rclone config dump [flags] -h, --help help for dump ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3649,12 +3621,144 @@ rclone config edit [flags] -h, --help help for edit ``` +See the [global flags page](https://rclone.org/flags/) for global options not listed here. + +## See Also + +* [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. + +# rclone config encryption + +set, remove and check the encryption for the config file + +## Synopsis + +This command sets, clears and checks the encryption for the config file using +the subcommands below. + + +## Options + +``` + -h, --help help for encryption +``` See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. +* [rclone config encryption check](https://rclone.org/commands/rclone_config_encryption_check/) - Check that the config file is encrypted +* [rclone config encryption remove](https://rclone.org/commands/rclone_config_encryption_remove/) - Remove the config file encryption password +* [rclone config encryption set](https://rclone.org/commands/rclone_config_encryption_set/) - Set or change the config file encryption password + +# rclone config encryption check + +Check that the config file is encrypted + +## Synopsis + +This checks the config file is encrypted and that you can decrypt it. + +It will attempt to decrypt the config using the password you supply. + +If decryption fails it will return a non-zero exit code if using +`--password-command`, otherwise it will prompt again for the password. + +If the config file is not encrypted it will return a non zero exit code. + + +``` +rclone config encryption check [flags] +``` + +## Options + +``` + -h, --help help for check +``` + +See the [global flags page](https://rclone.org/flags/) for global options not listed here. + +## See Also + +* [rclone config encryption](https://rclone.org/commands/rclone_config_encryption/) - set, remove and check the encryption for the config file + +# rclone config encryption remove + +Remove the config file encryption password + +## Synopsis + +Remove the config file encryption password + +This removes the config file encryption, returning it to un-encrypted. + +If `--password-command` is in use, this will be called to supply the old config +password. + +If the config was not encrypted then no error will be returned and +this command will do nothing. + + +``` +rclone config encryption remove [flags] +``` + +## Options + +``` + -h, --help help for remove +``` + +See the [global flags page](https://rclone.org/flags/) for global options not listed here. + +## See Also + +* [rclone config encryption](https://rclone.org/commands/rclone_config_encryption/) - set, remove and check the encryption for the config file + +# rclone config encryption set + +Set or change the config file encryption password + +## Synopsis + +This command sets or changes the config file encryption password. + +If there was no config password set then it sets a new one, otherwise +it changes the existing config password. + +Note that if you are changing an encryption password using +`--password-command` then this will be called once to decrypt the +config using the old password and then again to read the new +password to re-encrypt the config. + +When `--password-command` is called to change the password then the +environment variable `RCLONE_PASSWORD_CHANGE=1` will be set. So if +changing passwords programatically you can use the environment +variable to distinguish which password you must supply. + +Alternatively you can remove the password first (with `rclone config +encryption remove`), then set it again with this command which may be +easier if you don't mind the unecrypted config file being on the disk +briefly. + + +``` +rclone config encryption set [flags] +``` + +## Options + +``` + -h, --help help for set +``` + +See the [global flags page](https://rclone.org/flags/) for global options not listed here. + +## See Also + +* [rclone config encryption](https://rclone.org/commands/rclone_config_encryption/) - set, remove and check the encryption for the config file # rclone config file @@ -3670,10 +3774,9 @@ rclone config file [flags] -h, --help help for file ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3683,7 +3786,6 @@ Update password in an existing remote. ## Synopsis - Update an existing remote's password. The password should be passed in pairs of `key` `password` or as `key=password`. The `password` should be passed in in clear (unobscured). @@ -3707,10 +3809,9 @@ rclone config password name [key value]+ [flags] -h, --help help for password ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3728,10 +3829,9 @@ rclone config paths [flags] -h, --help help for paths ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3749,10 +3849,9 @@ rclone config providers [flags] -h, --help help for providers ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3762,7 +3861,6 @@ Re-authenticates user with remote. ## Synopsis - This reconnects remote: passed in to the cloud storage system. To disconnect the remote use "rclone config disconnect". @@ -3780,10 +3878,9 @@ rclone config reconnect remote: [flags] -h, --help help for reconnect ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3815,10 +3912,9 @@ rclone config redacted [] [flags] -h, --help help for redacted ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3836,10 +3932,9 @@ rclone config show [] [flags] -h, --help help for show ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3857,10 +3952,9 @@ rclone config touch [flags] -h, --help help for touch ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -3870,7 +3964,6 @@ Update options in an existing remote. ## Synopsis - Update an existing remote's options. The options should be passed in pairs of `key` `value` or as `key=value`. @@ -3990,10 +4083,9 @@ rclone config update name [key value]+ [flags] --state string State - use with --continue ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4003,7 +4095,6 @@ Prints info about logged in user of remote. ## Synopsis - This prints the details of the person logged in to the cloud storage system. @@ -4019,10 +4110,9 @@ rclone config userinfo remote: [flags] --json Format output as JSON ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4032,7 +4122,6 @@ Copy files from source to dest, skipping identical files. ## Synopsis - If source:path is a file or directory then it copies it to a file or directory named dest:path. @@ -4072,15 +4161,17 @@ rclone copyto source:path dest:path [flags] -h, --help help for copyto ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -4112,9 +4203,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -4122,9 +4213,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -4151,18 +4242,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -4172,7 +4261,6 @@ Copy the contents of the URL supplied content to dest:path. ## Synopsis - Download a URL's content and copy it to the destination without saving it in temporary storage. @@ -4218,10 +4306,12 @@ rclone copyurl https://example.com dest:path [flags] --stdout Write the output to stdout rather than a file ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -4229,9 +4319,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -4241,10 +4329,9 @@ Cryptcheck checks the integrity of an encrypted remote. ## Synopsis - -rclone cryptcheck checks a remote against a [crypted](https://rclone.org/crypt/) remote. -This is the equivalent of running rclone [check](https://rclone.org/commands/rclone_check/), -but able to check the checksums of the encrypted remote. +Checks a remote against a [crypted](https://rclone.org/crypt/) remote. This is the equivalent +of running rclone [check](https://rclone.org/commands/rclone_check/), but able to check the +checksums of the encrypted remote. For it to work the underlying remote of the cryptedremote must support some kind of checksum. @@ -4307,18 +4394,20 @@ rclone cryptcheck remote:path cryptedremote:path [flags] --one-way Check one way only, source files must exist on remote ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Check Options +### Check Options -Flags used for `rclone check`. +Flags used for check commands ``` --max-backlog int Maximum number of objects in sync or check backlog (default 10000) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -4345,18 +4434,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -4366,9 +4453,8 @@ Cryptdecode returns unencrypted file names. ## Synopsis - -rclone cryptdecode returns unencrypted file names when provided with -a list of encrypted file names. List limit is 10 items. +Returns unencrypted file names when provided with a list of encrypted file +names. List limit is 10 items. If you supply the `--reverse` flag, it will return encrypted file names. @@ -4393,10 +4479,9 @@ rclone cryptdecode encryptedremote: encryptedfilename [flags] --reverse Reverse cryptdecode, encrypts filenames ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -4406,7 +4491,6 @@ Remove a single file from remote. ## Synopsis - Remove a single file from remote. Unlike `delete` it cannot be used to remove a directory and it doesn't obey include/exclude filters - if the specified file exists, it will always be removed. @@ -4422,10 +4506,12 @@ rclone deletefile remote:path [flags] -h, --help help for deletefile ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -4433,168 +4519,16 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. -# rclone genautocomplete - -Output completion script for a given shell. - -# Synopsis - - -Generates a shell completion script for rclone. -Run with `--help` to list the supported shells. - - -# Options - -``` - -h, --help help for genautocomplete -``` - -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO - -* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. -* [rclone genautocomplete bash](https://rclone.org/commands/rclone_genautocomplete_bash/) - Output bash completion script for rclone. -* [rclone genautocomplete fish](https://rclone.org/commands/rclone_genautocomplete_fish/) - Output fish completion script for rclone. -* [rclone genautocomplete zsh](https://rclone.org/commands/rclone_genautocomplete_zsh/) - Output zsh completion script for rclone. - -# rclone genautocomplete bash - -Output bash completion script for rclone. - -# Synopsis - - -Generates a bash shell autocompletion script for rclone. - -This writes to /etc/bash_completion.d/rclone by default so will -probably need to be run with sudo or as root, e.g. - - sudo rclone genautocomplete bash - -Logout and login again to use the autocompletion scripts, or source -them directly - - . /etc/bash_completion - -If you supply a command line argument the script will be written -there. - -If output_file is "-", then the output will be written to stdout. - - -``` -rclone genautocomplete bash [output_file] [flags] -``` - -# Options - -``` - -h, --help help for bash -``` - -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO - -* [rclone genautocomplete](https://rclone.org/commands/rclone_genautocomplete/) - Output completion script for a given shell. - -# rclone genautocomplete fish - -Output fish completion script for rclone. - -# Synopsis - - -Generates a fish autocompletion script for rclone. - -This writes to /etc/fish/completions/rclone.fish by default so will -probably need to be run with sudo or as root, e.g. - - sudo rclone genautocomplete fish - -Logout and login again to use the autocompletion scripts, or source -them directly - - . /etc/fish/completions/rclone.fish - -If you supply a command line argument the script will be written -there. - -If output_file is "-", then the output will be written to stdout. - - -``` -rclone genautocomplete fish [output_file] [flags] -``` - -# Options - -``` - -h, --help help for fish -``` - -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO - -* [rclone genautocomplete](https://rclone.org/commands/rclone_genautocomplete/) - Output completion script for a given shell. - -# rclone genautocomplete zsh - -Output zsh completion script for rclone. - -# Synopsis - - -Generates a zsh autocompletion script for rclone. - -This writes to /usr/share/zsh/vendor-completions/_rclone by default so will -probably need to be run with sudo or as root, e.g. - - sudo rclone genautocomplete zsh - -Logout and login again to use the autocompletion scripts, or source -them directly - - autoload -U compinit && compinit - -If you supply a command line argument the script will be written -there. - -If output_file is "-", then the output will be written to stdout. - - -``` -rclone genautocomplete zsh [output_file] [flags] -``` - -# Options - -``` - -h, --help help for zsh -``` - -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO - -* [rclone genautocomplete](https://rclone.org/commands/rclone_genautocomplete/) - Output completion script for a given shell. - # rclone gendocs Output markdown docs for rclone to the directory supplied. ## Synopsis - This produces markdown docs for the rclone commands to the directory supplied. These are in a format suitable for hugo to render into the rclone.org website. @@ -4609,10 +4543,9 @@ rclone gendocs output_directory [flags] -h, --help help for gendocs ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -4703,10 +4636,9 @@ rclone gitannex [flags] -h, --help help for gitannex ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -4716,7 +4648,6 @@ Produces a hashsum file for all the objects in the path. ## Synopsis - Produces a hash file for all the objects in the path using the hash named. The output is in the same format as the standard md5sum/sha1sum tool. @@ -4765,10 +4696,12 @@ rclone hashsum [ remote:path] [flags] --output-file string Output hashsums to a file rather than the terminal ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -4795,18 +4728,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -4816,8 +4747,7 @@ Generate public link to file/folder. ## Synopsis -rclone link will create, retrieve or remove a public link to the given -file or folder. +Create, retrieve or remove a public link to the given file or folder. rclone link remote:path/to/file rclone link remote:path/to/folder/ @@ -4851,10 +4781,9 @@ rclone link remote:path [flags] --unlink Remove existing public link to file/folder ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -4865,26 +4794,40 @@ List all the remotes in the config file and defined in environment variables. ## Synopsis -rclone listremotes lists all the available remotes from the config file. +Lists all the available remotes from the config file, or the remotes matching +an optional filter. -When used with the `--long` flag it lists the types and the descriptions too. +Prints the result in human-readable format by default, and as a simple list of +remote names, or if used with flag `--long` a tabular format including +the remote names, types and descriptions. Using flag `--json` produces +machine-readable output instead, which always includes all attributes - including +the source (file or environment). + +Result can be filtered by a filter argument which applies to all attributes, +and/or filter flags specific for each attribute. The values must be specified +according to regular rclone filtering pattern syntax. ``` -rclone listremotes [flags] +rclone listremotes [] [flags] ``` ## Options ``` - -h, --help help for listremotes - --long Show the type and the description as well as names + --description string Filter remotes by description + -h, --help help for listremotes + --json Format output as JSON + --long Show type and description in addition to name + --name string Filter remotes by name + --order-by string Instructions on how to order the result, e.g. 'type,name=descending' + --source string Filter remotes by source, e.g. 'file' or 'environment' + --type string Filter remotes by type ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -4894,7 +4837,6 @@ List directories and objects in remote:path formatted for parsing. ## Synopsis - List the contents of the source path (directories and objects) to standard output in a form which is easy to parse by scripts. By default this will just be the names of the objects and directories, @@ -5047,10 +4989,12 @@ rclone lsf remote:path [flags] -t, --time-format string Specify a custom time format, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -5077,18 +5021,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -5100,7 +5042,7 @@ List directories and objects in the path in JSON format. List directories and objects in the path in JSON format. -The output is an array of Items, where each Item looks like this +The output is an array of Items, where each Item looks like this: { "Hashes" : { @@ -5122,44 +5064,50 @@ The output is an array of Items, where each Item looks like this "Tier" : "hot", } -If `--hash` is not specified, the Hashes property will be omitted. The -types of hash can be specified with the `--hash-type` parameter (which -may be repeated). If `--hash-type` is set then it implies `--hash`. +The exact set of properties included depends on the backend: -If `--no-modtime` is specified then ModTime will be blank. This can -speed things up on remotes where reading the ModTime takes an extra -request (e.g. s3, swift). +- The property IsBucket will only be included for bucket-based remotes, and only + for directories that are buckets. It will always be omitted when value is not true. +- Properties Encrypted and EncryptedPath will only be included for encrypted + remotes, and (as mentioned below) only if the `--encrypted` option is set. -If `--no-mimetype` is specified then MimeType will be blank. This can -speed things up on remotes where reading the MimeType takes an extra -request (e.g. s3, swift). +Different options may also affect which properties are included: -If `--encrypted` is not specified the Encrypted will be omitted. +- If `--hash` is not specified, the Hashes property will be omitted. The + types of hash can be specified with the `--hash-type` parameter (which + may be repeated). If `--hash-type` is set then it implies `--hash`. +- If `--no-modtime` is specified then ModTime will be blank. This can + speed things up on remotes where reading the ModTime takes an extra + request (e.g. s3, swift). +- If `--no-mimetype` is specified then MimeType will be blank. This can + speed things up on remotes where reading the MimeType takes an extra + request (e.g. s3, swift). +- If `--encrypted` is not specified the Encrypted and EncryptedPath + properties will be omitted - even for encrypted remotes. +- If `--metadata` is set then an additional Metadata property will be + returned. This will have [metadata](https://rclone.org/docs/#metadata) in rclone standard format + as a JSON object. -If `--dirs-only` is not specified files in addition to directories are -returned +The default is to list directories and files/objects, but this can be changed +with the following options: -If `--files-only` is not specified directories in addition to the files -will be returned. +- If `--dirs-only` is specified then directories will be returned + only, no files/objects. +- If `--files-only` is specified then files will be returned only, + no directories. -If `--metadata` is set then an additional Metadata key will be returned. -This will have metadata in rclone standard format as a JSON object. - -if `--stat` is set then a single JSON blob will be returned about the -item pointed to. This will return an error if the item isn't found. -However on bucket based backends (like s3, gcs, b2, azureblob etc) if -the item isn't found it will return an empty directory as it isn't -possible to tell empty directories from missing directories there. +If `--stat` is set then the the output is not an array of items, +but instead a single JSON blob will be returned about the item pointed to. +This will return an error if the item isn't found, however on bucket based +backends (like s3, gcs, b2, azureblob etc) if the item isn't found it will +return an empty directory, as it isn't possible to tell empty directories +from missing directories there. The Path field will only show folders below the remote path being listed. If "remote:path" contains the file "subfolder/file.txt", the Path for "file.txt" will be "subfolder/file.txt", not "remote:path/subfolder/file.txt". When used without `--recursive` the Path will always be the same as Name. -If the directory is a bucket in a bucket-based backend, then -"IsBucket" will be set to true. This key won't be present unless it is -"true". - The time is in RFC3339 format with up to nanosecond precision. The number of decimal digits in the seconds will depend on the precision that the remote can hold the times, so if times are accurate to the @@ -5169,7 +5117,8 @@ accurate to the nearest second (Dropbox, Box, WebDav, etc.) no digits will be shown ("2017-05-31T16:15:57+01:00"). The whole output can be processed as a JSON blob, or alternatively it -can be processed line by line as each item is written one to a line. +can be processed line by line as each item is written on individual lines +(except with `--stat`). Any of the filtering options can be applied to this command. @@ -5215,10 +5164,12 @@ rclone lsjson remote:path [flags] --stat Just return the info for the pointed to file ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -5245,18 +5196,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -5960,6 +5909,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -5975,6 +5929,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -6102,9 +6080,9 @@ rclone mount remote:path /path/to/mountpoint [flags] --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for mount @@ -6120,7 +6098,7 @@ rclone mount remote:path /path/to/mountpoint [flags] --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -6133,6 +6111,7 @@ rclone mount remote:path /path/to/mountpoint [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -6142,10 +6121,12 @@ rclone mount remote:path /path/to/mountpoint [flags] --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -6172,9 +6153,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -6184,7 +6163,6 @@ Move file or directory from source to dest. ## Synopsis - If source:path is a file or directory then it moves it to a file or directory named dest:path. @@ -6227,15 +6205,17 @@ rclone moveto source:path dest:path [flags] -h, --help help for moveto ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -6267,9 +6247,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -6277,9 +6257,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -6306,18 +6286,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -6327,7 +6305,6 @@ Explore a remote with a text based user interface. ## Synopsis - This displays a text based user interface allowing the navigation of a remote. It is most useful for answering the question - "What is using all my disk space?". @@ -6380,7 +6357,7 @@ These flags have the following meaning: This an homage to the [ncdu tool](https://dev.yorhel.nl/ncdu) but for rclone remotes. It is missing lots of features at the moment -but is useful as it stands. +but is useful as it stands. Unlike ncdu it does not show excluded files. Note that it might take some time to delete big files/directories. The UI won't respond in the meantime since the deletion is done synchronously. @@ -6400,10 +6377,12 @@ rclone ncdu remote:path [flags] -h, --help help for ncdu ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -6430,18 +6409,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -7145,6 +7122,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -7160,6 +7142,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -7288,16 +7294,18 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for nfsmount --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki) --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset) --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only) + --nfs-cache-dir string The directory the NFS handle cache will use if set --nfs-cache-handle-limit int max file handles cached simultaneously (min 5) (default 1000000) + --nfs-cache-type memory|disk|symlink Type of NFS handle cache to use (default memory) --no-checksum Don't compare checksums on up/download --no-modtime Don't read/write the modification time (can speed things up) --no-seek Don't allow seeking in files @@ -7306,9 +7314,9 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] -o, --option stringArray Option for libfuse/WinFsp (repeat if required) --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access - --sudo Use sudo to run the mount command as root. + --sudo Use sudo to run the mount/umount commands as root. --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -7321,6 +7329,7 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -7330,10 +7339,12 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -7360,9 +7371,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -7406,10 +7415,9 @@ rclone obscure password [flags] -h, --help help for obscure ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -7419,8 +7427,6 @@ Run a command against a running rclone. ## Synopsis - - This runs a command against a running rclone. Use the `--url` flag to specify an non default URL to connect on. This can be either a ":port" which is taken to mean "http://localhost:port" or a @@ -7431,6 +7437,13 @@ A username and password can be passed in with `--user` and `--pass`. Note that `--rc-addr`, `--rc-user`, `--rc-pass` will be read also for `--url`, `--user`, `--pass`. +The `--unix-socket` flag can be used to connect over a unix socket like this + + # start server on /tmp/my.socket + rclone rcd --rc-addr unix:///tmp/my.socket + # Connect to it + rclone rc --unix-socket /tmp/my.socket core/stats + Arguments should be passed in as parameter=value. The result will be returned as a JSON object by default. @@ -7477,21 +7490,21 @@ rclone rc commands parameter [flags] ## Options ``` - -a, --arg stringArray Argument placed in the "arg" array - -h, --help help for rc - --json string Input JSON - use instead of key=value args - --loopback If set connect to this rclone instance not via HTTP - --no-output If set, don't output the JSON result - -o, --opt stringArray Option in the form name=value or name placed in the "opt" array - --pass string Password to use to connect to rclone remote control - --url string URL to connect to rclone remote control (default "http://localhost:5572/") - --user string Username to use to rclone remote control + -a, --arg stringArray Argument placed in the "arg" array + -h, --help help for rc + --json string Input JSON - use instead of key=value args + --loopback If set connect to this rclone instance not via HTTP + --no-output If set, don't output the JSON result + -o, --opt stringArray Option in the form name=value or name placed in the "opt" array + --pass string Password to use to connect to rclone remote control + --unix-socket string Path to a unix domain socket to dial to, instead of opening a TCP connection directly + --url string URL to connect to rclone remote control (default "http://localhost:5572/") + --user string Username to use to rclone remote control ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -7501,9 +7514,7 @@ Copies standard input to file on remote. ## Synopsis - -rclone rcat reads from standard input (stdin) and copies it to a -single remote file. +Reads from standard input (stdin) and copies it to a single remote file. echo "hello world" | rclone rcat remote:path/to/file ffmpeg - | rclone rcat remote:path/to/file @@ -7544,10 +7555,12 @@ rclone rcat remote:path [flags] --size int File size hint to preallocate (default -1) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -7555,9 +7568,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -7592,6 +7603,7 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. `--rc-addr` may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to accomplish the same. `--rc-server-read-timeout` and `--rc-server-write-timeout` can be used to control the timeouts on the server. Note that this is the total time @@ -7624,6 +7636,20 @@ certificate authority certificate. values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). +## Socket activation + +Instead of the listening addresses specified above, rclone will listen to all +FDs passed by the service manager, if any (and ignore any arguments passed by --rc-addr`). + +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 over TCP. ### Template `--rc-template` allows a user to specify a custom markup template for HTTP @@ -7699,19 +7725,21 @@ rclone rcd * [flags] -h, --help help for rcd ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## RC Options +### RC Options -Flags to control the Remote Control API. +Flags to control the Remote Control API ``` --rc Enable the remote control server - --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572]) + --rc-addr stringArray IPaddress:Port or :Port to bind server to (default ["localhost:5572"]) --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from --rc-baseurl string Prefix for URLs - leave blank for root --rc-cert string TLS PEM key (concatenation of certificate and CA certificate) --rc-client-ca string Client certificate authority to verify clients with - --rc-enable-metrics Enable prometheus metrics on /metrics + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server --rc-files string Path to local files to serve on the HTTP server --rc-htpasswd string A htpasswd file - if not provided no authentication is done --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s) @@ -7736,9 +7764,7 @@ Flags to control the Remote Control API. --rc-web-gui-update Check and update to latest version of web gui ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -7748,7 +7774,6 @@ Remove empty directories under the path. ## Synopsis - This recursively removes any empty directories (including directories that only contain empty directories), that it finds under the path. The root path itself will also be removed if it is empty, unless @@ -7780,10 +7805,12 @@ rclone rmdirs remote:path [flags] --leave-root Do not remove root directory if empty ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -7791,9 +7818,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -7868,10 +7893,9 @@ rclone selfupdate [flags] --version string Install the given rclone version (default: latest) ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -7899,10 +7923,9 @@ rclone serve [opts] [flags] -h, --help help for serve ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. * [rclone serve dlna](https://rclone.org/commands/rclone_serve_dlna/) - Serve remote:path over DLNA @@ -7932,6 +7955,10 @@ based on media formats or file extensions. Additionally, there is no media transcoding support. This means that some players might show files that they are not able to play back correctly. +Rclone will add external subtitle files (.srt) to videos if they have the same +filename as the video file itself (except the extension), either in the same +directory as the video, or in a "Subs" subdirectory. + ## Server options Use `--addr` to specify which IP address and port the server should @@ -8176,6 +8203,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -8191,6 +8223,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -8309,8 +8365,8 @@ rclone serve dlna remote:path [flags] --addr string The ip:port or :port to bind the DLNA http server to (default ":7879") --announce-interval Duration The interval between SSDP announcements (default 12m0s) --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for dlna --interface stringArray The interface to use for SSDP (repeat as necessary) @@ -8322,7 +8378,7 @@ rclone serve dlna remote:path [flags] --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -8335,6 +8391,7 @@ rclone serve dlna remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -8342,10 +8399,12 @@ rclone serve dlna remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -8372,9 +8431,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -8655,6 +8712,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -8670,6 +8732,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -8798,9 +8884,9 @@ rclone serve docker [flags] --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --forget-state Skip restoring previous state --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) @@ -8820,7 +8906,7 @@ rclone serve docker [flags] --socket-addr string Address or absolute path (default: /run/docker/plugins/rclone.sock) --socket-gid int GID for unix socket (default: current process GID) (default 1000) --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -8833,6 +8919,7 @@ rclone serve docker [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -8842,10 +8929,12 @@ rclone serve docker [flags] --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -8872,9 +8961,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -9136,6 +9223,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -9151,6 +9243,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -9351,8 +9467,8 @@ rclone serve ftp remote:path [flags] --auth-proxy string A program to use to create the backend from the auth --cert string TLS PEM key (concatenation of certificate and CA certificate) --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for ftp --key string TLS PEM Private key @@ -9365,7 +9481,7 @@ rclone serve ftp remote:path [flags] --public-ip string Public IP address to advertise for passive connections --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication (default "anonymous") --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -9379,6 +9495,7 @@ rclone serve ftp remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -9386,10 +9503,12 @@ rclone serve ftp remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -9416,9 +9535,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -9455,6 +9572,7 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. `--addr` may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to accomplish the same. `--server-read-timeout` and `--server-write-timeout` can be used to control the timeouts on the server. Note that this is the total time @@ -9487,6 +9605,20 @@ certificate authority certificate. values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). +## Socket activation + +Instead of the listening addresses specified above, rclone will listen to all +FDs passed by the service manager, if any (and ignore any arguments passed by --addr`). + +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 over TCP. ### Template `--template` allows a user to specify a custom markup template for HTTP @@ -9782,6 +9914,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -9797,6 +9934,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -9993,15 +10154,15 @@ rclone serve http remote:path [flags] ## Options ``` - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for http --htpasswd string A htpasswd file - if not provided no authentication is done @@ -10020,7 +10181,7 @@ rclone serve http remote:path [flags] --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --template string User-specified template --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -10034,6 +10195,7 @@ rclone serve http remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -10041,10 +10203,12 @@ rclone serve http remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -10071,9 +10235,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -10085,34 +10247,69 @@ Serve the remote as an NFS mount Create an NFS server that serves the given remote over the network. -The primary purpose for this command is to enable [mount command](https://rclone.org/commands/rclone_mount/) on recent macOS versions where -installing FUSE is very cumbersome. +This implements an NFSv3 server to serve any rclone remote via NFS. -Since this is running on NFSv3, no authentication method is available. Any client -will be able to access the data. To limit access, you can use serve NFS on loopback address -and rely on secure tunnels (such as SSH). For this reason, by default, a random TCP port is chosen and loopback interface is used for the listening address; -meaning that it is only available to the local machine. If you want other machines to access the -NFS mount over local network, you need to specify the listening address and port using `--addr` flag. +The primary purpose for this command is to enable the [mount +command](https://rclone.org/commands/rclone_mount/) on recent macOS versions where +installing FUSE is very cumbersome. -Modifying files through NFS protocol requires VFS caching. Usually you will need to specify `--vfs-cache-mode` -in order to be able to write to the mountpoint (full is recommended). If you don't specify VFS cache mode, -the mount will be read-only. Note also that `--nfs-cache-handle-limit` controls the maximum number of cached file handles stored by the caching handler. -This should not be set too low or you may experience errors when trying to access files. The default is `1000000`, but consider lowering this limit if -the server's system resource usage causes problems. +This server does not implement any authentication so any client will be +able to access the data. To limit access, you can use `serve nfs` on +the loopback address or rely on secure tunnels (such as SSH) or use +firewalling. + +For this reason, by default, a random TCP port is chosen and the +loopback interface is used for the listening address by default; +meaning that it is only available to the local machine. If you want +other machines to access the NFS mount over local network, you need to +specify the listening address and port using the `--addr` flag. + +Modifying files through the NFS protocol requires VFS caching. Usually +you will need to specify `--vfs-cache-mode` in order to be able to +write to the mountpoint (`full` is recommended). If you don't specify +VFS cache mode, the mount will be read-only. + +`--nfs-cache-type` controls the type of the NFS handle cache. By +default this is `memory` where new handles will be randomly allocated +when needed. These are stored in memory. If the server is restarted +the handle cache will be lost and connected NFS clients will get stale +handle errors. + +`--nfs-cache-type disk` uses an on disk NFS handle cache. Rclone +hashes the path of the object and stores it in a file named after the +hash. These hashes are stored on disk the directory controlled by +`--cache-dir` or the exact directory may be specified with +`--nfs-cache-dir`. Using this means that the NFS server can be +restarted at will without affecting the connected clients. + +`--nfs-cache-type symlink` is similar to `--nfs-cache-type disk` in +that it uses an on disk cache, but the cache entries are held as +symlinks. Rclone will use the handle of the underlying file as the NFS +handle which improves performance. This sort of cache can't be backed +up and restored as the underlying handles will change. This is Linux +only. + +`--nfs-cache-handle-limit` controls the maximum number of cached NFS +handles stored by the caching handler. This should not be set too low +or you may experience errors when trying to access files. The default +is `1000000`, but consider lowering this limit if the server's system +resource usage causes problems. This is only used by the `memory` type +cache. To serve NFS over the network use following command: rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full -We specify a specific port that we can use in the mount command: - -To mount the server under Linux/macOS, use the following command: +This specifies a port that can be used in the mount command. To mount +the server under Linux/macOS, use the following command: - mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint + mount -t nfs -o port=$PORT,mountport=$PORT,tcp $HOSTNAME:/ path/to/mountpoint -Where `$PORT` is the same port number we used in the serve nfs command. +Where `$PORT` is the same port number used in the `serve nfs` command +and `$HOSTNAME` is the network address of the machine that `serve nfs` +was run on. -This feature is only available on Unix platforms. +This command is only available on Unix platforms. ## VFS - Virtual File System @@ -10346,6 +10543,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -10361,6 +10563,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -10478,18 +10704,20 @@ rclone serve nfs remote:path [flags] ``` --addr string IPaddress:Port or :Port to bind server to --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for nfs + --nfs-cache-dir string The directory the NFS handle cache will use if set --nfs-cache-handle-limit int max file handles cached simultaneously (min 5) (default 1000000) + --nfs-cache-type memory|disk|symlink Type of NFS handle cache to use (default memory) --no-checksum Don't compare checksums on up/download --no-modtime Don't read/write the modification time (can speed things up) --no-seek Don't allow seeking in files --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -10502,6 +10730,7 @@ rclone serve nfs remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -10509,10 +10738,12 @@ rclone serve nfs remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -10539,9 +10770,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -10648,6 +10877,7 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. `--addr` may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to accomplish the same. `--server-read-timeout` and `--server-write-timeout` can be used to control the timeouts on the server. Note that this is the total time @@ -10680,6 +10910,20 @@ certificate authority certificate. values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). +## Socket activation + +Instead of the listening addresses specified above, rclone will listen to all +FDs passed by the service manager, if any (and ignore any arguments passed by --addr`). + +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 over TCP. ### Authentication By default this will serve files without needing a login. @@ -10716,7 +10960,7 @@ rclone serve restic remote:path [flags] ## Options ``` - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --append-only Disallow deletion of repository data --baseurl string Prefix for URLs - leave blank for root @@ -10738,10 +10982,9 @@ rclone serve restic remote:path [flags] --user string User name for authentication ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -10887,6 +11130,33 @@ metadata which will be set as the modification time of the file. Other operations will return error `Unimplemented`. +### Authentication + +By default this will serve files without needing a login. + +You can either use an htpasswd file which can take lots of users, or +set a single username and password with the `--user` and `--pass` flags. + +If no static users are configured by either of the above methods, and client +certificates are required by the `--client-ca` flag passed to the server, the +client certificate common name will be considered as the username. + +Use `--htpasswd /path/to/htpasswd` to provide an htpasswd file. This is +in standard apache format and supports MD5, SHA1 and BCrypt for basic +authentication. Bcrypt is recommended. + +To create an htpasswd file: + + touch htpasswd + htpasswd -B htpasswd user + htpasswd -B htpasswd anotherUser + +The password file can be updated while rclone is running. + +Use `--realm` to set the authentication realm. + +Use `--salt` to change the password hashing salt from the default. + ## Server options Use `--addr` to specify which IP address and port the server should @@ -10902,6 +11172,7 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. `--addr` may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to accomplish the same. `--server-read-timeout` and `--server-write-timeout` can be used to control the timeouts on the server. Note that this is the total time @@ -10934,6 +11205,20 @@ certificate authority certificate. values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). +## Socket activation + +Instead of the listening addresses specified above, rclone will listen to all +FDs passed by the service manager, if any (and ignore any arguments passed by --addr`). + +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 over TCP. ## VFS - Virtual File System This command uses the VFS layer. This adapts the cloud storage objects @@ -11166,6 +11451,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -11181,6 +11471,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -11296,19 +11610,21 @@ rclone serve s3 remote:path [flags] ## Options ``` - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-key stringArray Set key pair for v4 authorization: access_key_id,secret_access_key + --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --etag-hash string Which hash to use for the ETag, or auto or blank for off (default "MD5") - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --force-path-style If true use path style access if false use virtual hosted style (default true) (default true) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for s3 + --htpasswd string A htpasswd file - if not provided no authentication is done --key string TLS PEM Private key --max-header-bytes int Maximum size of request header (default 4096) --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") @@ -11316,12 +11632,16 @@ rclone serve s3 remote:path [flags] --no-cleanup Not to cleanup empty folder after object is deleted --no-modtime Don't read/write the modification time (can speed things up) --no-seek Don't allow seeking in files + --pass string Password for authentication --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access + --realm string Realm for authentication + --salt string Password hashing salt (default "dlPL2MqE") --server-read-timeout Duration Timeout for server reading data (default 1h0m0s) --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) + --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -11334,6 +11654,7 @@ rclone serve s3 remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -11341,10 +11662,12 @@ rclone serve s3 remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -11371,9 +11694,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -11416,6 +11737,17 @@ directory. By default the server binds to localhost:2022 - if you want it to be reachable externally then supply `--addr :2022` for example. +This also supports being run with socket activation, in which case it will +listen on the first passed FD. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command: + + systemd-socket-activate -l 2222 -- rclone serve sftp :local:vfs/ + +This will socket-activate rclone on the first connection to port 2222 over TCP. + Note that the default of `--vfs-cache-mode off` is fine for the rclone sftp backend, but it may not be with other SFTP clients. @@ -11667,6 +11999,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -11682,6 +12019,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -11882,8 +12243,8 @@ rclone serve sftp remote:path [flags] --auth-proxy string A program to use to create the backend from the auth --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys") --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for sftp --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate) @@ -11896,7 +12257,7 @@ rclone serve sftp remote:path [flags] --read-only Only allow read-only access --stdio Run an sftp server on stdin/stdout --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -11910,6 +12271,7 @@ rclone serve sftp remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -11917,10 +12279,12 @@ rclone serve sftp remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -11947,9 +12311,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -12001,6 +12363,19 @@ Create a new DWORD BasicAuthLevel with value 2. https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint +## Serving over a unix socket + +You can serve the webdav on a unix socket like this: + + rclone serve webdav --addr unix:///tmp/my.socket remote:path + +and connect to it like this using rclone and the webdav backend: + + rclone --webdav-unix-socket /tmp/my.socket --webdav-url http://localhost lsf :webdav: + +Note that there is no authentication on http protocol - this is expected to be +done by the permissions on the socket. + ## Server options Use `--addr` to specify which IP address and port the server should @@ -12016,6 +12391,7 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. `--addr` may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to accomplish the same. `--server-read-timeout` and `--server-write-timeout` can be used to control the timeouts on the server. Note that this is the total time @@ -12048,6 +12424,20 @@ certificate authority certificate. values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). +## Socket activation + +Instead of the listening addresses specified above, rclone will listen to all +FDs passed by the service manager, if any (and ignore any arguments passed by --addr`). + +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 over TCP. ### Template `--template` allows a user to specify a custom markup template for HTTP @@ -12343,6 +12733,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -12358,6 +12753,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -12554,17 +12973,17 @@ rclone serve webdav remote:path [flags] ## Options ``` - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --disable-dir-list Disable HTML directory list on GET request for a directory --etag-hash string Which hash to use for the ETag, or auto or blank for off - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for webdav --htpasswd string A htpasswd file - if not provided no authentication is done @@ -12583,7 +13002,7 @@ rclone serve webdav remote:path [flags] --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --template string User-specified template --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -12597,6 +13016,7 @@ rclone serve webdav remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -12604,10 +13024,12 @@ rclone serve webdav remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -12634,9 +13056,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -12646,10 +13066,9 @@ Changes storage class/tier of objects in remote. ## Synopsis - -rclone settier changes storage tier or class at remote if supported. -Few cloud storage services provides different storage classes on objects, -for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, +Changes storage tier or class at remote if supported. Few cloud storage +services provides different storage classes on objects, for example +AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline etc. Note that, certain tier changes make objects not available to access immediately. @@ -12680,10 +13099,9 @@ rclone settier tier remote:path [flags] -h, --help help for settier ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -12711,10 +13129,9 @@ so reading their documentation first is recommended. -h, --help help for test ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. * [rclone test changenotify](https://rclone.org/commands/rclone_test_changenotify/) - Log any change notify requests for the remote passed in. @@ -12739,10 +13156,9 @@ rclone test changenotify remote: [flags] --poll-interval Duration Time to wait between polling for changes (default 10s) ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command @@ -12769,10 +13185,9 @@ rclone test histogram [remote:path] [flags] -h, --help help for histogram ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command @@ -12782,10 +13197,10 @@ Discovers file name or other limitations for paths. ## Synopsis -rclone info discovers what filenames and upload methods are possible -to write to the paths passed in and how long they can be. It can take some -time. It will write test files into the remote:path passed in. It outputs -a bit of go code for each one. +Discovers what filenames and upload methods are possible to write to the +paths passed in and how long they can be. It can take some time. It will +write test files into the remote:path passed in. It outputs a bit of go +code for each one. **NB** this can create undeletable files and other hazards - use with care @@ -12809,10 +13224,9 @@ rclone test info [remote:path]+ [flags] --write-json string Write results to file ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command @@ -12836,10 +13250,9 @@ rclone test makefile []+ [flags] --zero Fill files with ASCII 0x00 ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command @@ -12870,10 +13283,9 @@ rclone test makefiles [flags] --zero Fill files with ASCII 0x00 ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command @@ -12891,10 +13303,9 @@ rclone test memory remote:path [flags] -h, --help help for memory ``` - See the [global flags page](https://rclone.org/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command @@ -12904,7 +13315,6 @@ Create new file or change file modification time. ## Synopsis - Set the modification time on file(s) as specified by remote:path to have the current time. @@ -12940,10 +13350,12 @@ rclone touch remote:path [flags] -t, --timestamp string Use specified time instead of the current time of day ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -12951,9 +13363,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -12980,18 +13392,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -13001,9 +13411,7 @@ List the contents of the remote in a tree like fashion. ## Synopsis - -rclone tree lists the contents of a remote in a similar way to the -unix tree command. +Lists the contents of a remote in a similar way to the unix tree command. For example @@ -13058,10 +13466,12 @@ rclone tree remote:path [flags] --version Sort files alphanumerically by version ``` +Options shared with other commands are described next. +See the [global flags page](https://rclone.org/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -13088,18 +13498,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](https://rclone.org/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -13664,12 +14072,20 @@ for upload:download, e.g.`10M:1M`. characters. It is optional. - `HH:MM` is an hour from 00:00 to 23:59. +Entries can be separated by spaces or semicolons. + +**Note:** Semicolons can be used as separators instead of spaces to avoid parsing issues in environments like Docker. + An example of a typical timetable to avoid link saturation during daytime working hours could be: +Using spaces as separators: `--bwlimit "08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off"` -In this example, the transfer bandwidth will be set to 512 KiB/s +Using semicolons as separators: +`--bwlimit "08:00,512k;12:00,10M;13:00,512k;18:00,30M;23:00,off"` + +In these examples, the transfer bandwidth will be set to 512 KiB/s at 8am every day. At noon, it will rise to 10 MiB/s, and drop back to 512 KiB/sec at 1pm. At 6pm, the bandwidth limit will be set to 30 MiB/s, and at 11pm it will be completely disabled (full speed). @@ -13677,8 +14093,12 @@ Anything between 11pm and 8am will remain unlimited. An example of timetable with `WEEKDAY` could be: +Using spaces as separators: `--bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off"` +Using semicolons as separators: +`--bwlimit "Mon-00:00,512;Fri-23:59,10M;Sat-10:00,1M;Sun-20:00,off"` + It means that, the transfer bandwidth will be set to 512 KiB/s on Monday. It will rise to 10 MiB/s before the end of Friday. At 10:00 on Saturday it will be set to 1 MiB/s. From 20:00 on Sunday it will @@ -14270,11 +14690,12 @@ flag set) such as: - local - ftp - sftp +- pcloud Without `--inplace` (the default) rclone will first upload to a temporary file with an extension like this, where `XXXXXX` represents a -random string and `.partial` is [--partial-suffix](#partial-suffix) value -(`.partial` by default). +hash of the source file's fingerprint and `.partial` is +[--partial-suffix](#partial-suffix) value (`.partial` by default). original-file-name.XXXXXX.partial @@ -14846,11 +15267,12 @@ Suffix length limit is 16 characters. The default is `.partial`. -### --password-command SpaceSepList ### +### --password-command SpaceSepList {#password-command} This flag supplies a program which should supply the config password when run. This is an alternative to rclone prompting for the password -or setting the `RCLONE_CONFIG_PASS` variable. +or setting the `RCLONE_CONFIG_PASS` variable. It is also used when +setting the config password for the first time. The argument to this should be a command with a space separated list of arguments. If one of the arguments has a space in then enclose it @@ -14864,6 +15286,11 @@ Eg --password-command 'echo "hello with space"' --password-command 'echo "hello with ""quotes"" and space"' +Note that when changing the configuration password the environment +variable `RCLONE_PASSWORD_CHANGE=1` will be set. This can be used to +distinguish initial decryption of the config file from the new +password. + See the [Configuration Encryption](#configuration-encryption) for more info. See a [Windows PowerShell example on the Wiki](https://github.com/rclone/rclone/wiki/Windows-Powershell-use-rclone-password-command-for-Config-file-password). @@ -15467,6 +15894,12 @@ encryption from your configuration. There is no way to recover the configuration if you lose your password. +You can also use + +- [rclone config encryption set](https://rclone.org/commands/rclone_config_encryption_set/) to set the config encryption directly +- [rclone config encryption remove](https://rclone.org/commands/rclone_config_encryption_remove/) to remove it +- [rclone config encryption check](https://rclone.org/commands/rclone_config_encryption_check/) to check that it is encrypted properly. + rclone uses [nacl secretbox](https://godoc.org/golang.org/x/crypto/nacl/secretbox) which in turn uses XSalsa20 and Poly1305 to encrypt and authenticate your configuration with secret-key cryptography. @@ -15499,7 +15932,7 @@ An alternate means of supplying the password is to provide a script which will retrieve the password and print on standard output. This script should have a fully specified path name and not rely on any environment variables. The script is supplied either via -`--password-command="..."` command line argument or via the +[`--password-command="..."`](#password-command) command line argument or via the `RCLONE_PASSWORD_COMMAND` environment variable. One useful example of this is using the `passwordstore` application @@ -15536,12 +15969,63 @@ a configuration file, you can avoid it being loaded by overriding the location, e.g. with one of the documented special values for memory-only configuration. Since only backend options can be stored in configuration files, this is normally unnecessary for commands -that do not operate on backends, e.g. `genautocomplete`. However, +that do not operate on backends, e.g. `completion`. However, it will be relevant for commands that do operate on backends in general, but are used without referencing a stored remote, e.g. listing local filesystem paths, or [connection strings](#connection-strings): `rclone --config="" ls .` +Configuration Encryption Cheatsheet +----------------------------------- +You can quickly apply a configuration encryption without plain-text +at rest or transfer. Detailed instructions for popular OSes: + +### Mac ### + +* Generate and store a password + +`security add-generic-password -a rclone -s config -w $(openssl rand -base64 40)` + +* Add the retrieval instruction to your .zprofile / .profile + +`export RCLONE_PASSWORD_COMMAND="/usr/bin/security find-generic-password -a rclone -s config -w"` + +### Linux ### + +* Prerequisite + +Linux doesn't come with a default password manager. Let's install +the "pass" utility using a package manager, e.g. `apt install pass`, + `yum install pass`, + [etc.](https://www.passwordstore.org/#download); then initialize a + password store: + +`pass init rclone` + +* Generate and store a password + +`echo $(openssl rand -base64 40) | pass insert -m rclone/config` + +* Add the retrieval instruction + +`export RCLONE_PASSWORD_COMMAND="/usr/bin/pass rclone/config"` + +### Windows ### + +* Generate and store a password + +`New-Object -TypeName PSCredential -ArgumentList "rclone", (ConvertTo-SecureString -String ([System.Web.Security.Membership]::GeneratePassword(40, 10)) -AsPlainText -Force) | Export-Clixml -Path "rclone-credential.xml"` + +* Add the password retrieval instruction + +`[Environment]::SetEnvironmentVariable("RCLONE_PASSWORD_COMMAND", "[System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR((Import-Clixml -Path "rclone-credential.xml").Password))")` + +### Encrypt the config file (all systems) ### + +* Execute `rclone config` -> `s` + +* Add/update the password from previous steps + Developer options ----------------- @@ -15695,6 +16179,17 @@ Rclone prefixes all log messages with their level in capitals, e.g. INFO which makes it easy to grep the log file for different kinds of information. +Metrics +------- + +Rclone can publish metrics in the OpenMetrics/Prometheus format. + +To enable the metrics endpoint, use the `--metrics-addr` flag. Metrics can also be published on the `--rc-addr` port if the `--rc` flag and `--rc-enable-metrics` flags are supplied or if using rclone rcd `--rc-enable-metrics` + +Rclone provides extensive configuration options for the metrics HTTP endpoint. These settings are grouped under the Metrics section and have a prefix `--metrics-*`. + +When metrics are enabled with `--rc-enable-metrics`, they will be published on the same port as the rc API. In this case, the `--metrics-*` flags will be ignored, and the HTTP endpoint configuration will be managed by the `--rc-*` parameters. + Exit Code --------- @@ -15860,37 +16355,33 @@ two ways of doing it, described below. ## Configuring using rclone authorize ## -On the headless box run `rclone` config but answer `N` to the `Use web browser -to automatically authenticate?` question. +On the headless box run `rclone` config but answer `N` to the `Use auto config?` question. ``` -... -Remote config -Use web browser to automatically authenticate rclone with remote? - * Say Y if the machine running rclone has a web browser you can use - * Say N if running rclone on a (remote) machine without web browser access -If not sure try Y. If Y failed, try N. +Use auto config? + * Say Y if not sure + * Say N if you are working on a remote or headless machine + y) Yes (default) n) No y/n> n + +Option config_token. For this to work, you will need rclone available on a machine that has a web browser available. - For more help and alternate methods see: https://rclone.org/remote_setup/ - Execute the following on the machine with the web browser (same rclone version recommended): - - rclone authorize "dropbox" - -Then paste the result below: -result> + rclone authorize "onedrive" +Then paste the result. +Enter a value. +config_token> ``` Then on your main desktop machine ``` -rclone authorize "dropbox" +rclone authorize "onedrive" If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... @@ -15903,7 +16394,7 @@ SECRET_TOKEN Then back to the headless box, paste in the code ``` -result> SECRET_TOKEN +config_token> SECRET_TOKEN -------------------- [acd12] client_id = @@ -15945,16 +16436,13 @@ Linux and MacOS users can utilize SSH Tunnel to redirect the headless box port 5 ``` ssh -L localhost:53682:localhost:53682 username@remote_server ``` -Then on the headless box run `rclone` config and answer `Y` to the `Use web -browser to automatically authenticate?` question. +Then on the headless box run `rclone config` and answer `Y` to the `Use auto config?` question. ``` -... -Remote config -Use web browser to automatically authenticate rclone with remote? - * Say Y if the machine running rclone has a web browser you can use - * Say N if running rclone on a (remote) machine without web browser access -If not sure try Y. If Y failed, try N. +Use auto config? + * Say Y if not sure + * Say N if you are working on a remote or headless machine + y) Yes (default) n) No y/n> y @@ -16630,7 +17118,7 @@ remote or flag value. The fix then is to quote values containing spaces. ### `--min-size` - Don't transfer any file smaller than this Controls the minimum size file within the scope of an rclone command. -Default units are `KiB` but abbreviations `K`, `M`, `G`, `T` or `P` are valid. +Default units are `KiB` but abbreviations `B`, `K`, `M`, `G`, `T` or `P` are valid. E.g. `rclone ls remote: --min-size 50k` lists files on `remote:` of 50 KiB size or larger. @@ -16640,7 +17128,7 @@ See [the size option docs](https://rclone.org/docs/#size-option) for more info. ### `--max-size` - Don't transfer any file larger than this Controls the maximum size file within the scope of an rclone command. -Default units are `KiB` but abbreviations `K`, `M`, `G`, `T` or `P` are valid. +Default units are `KiB` but abbreviations `B`, `K`, `M`, `G`, `T` or `P` are valid. E.g. `rclone ls remote: --max-size 1G` lists files on `remote:` of 1 GiB size or smaller. @@ -16960,6 +17448,7 @@ Default Off. ### --rc-enable-metrics Enable OpenMetrics/Prometheus compatible endpoint at `/metrics`. +If more control over the metrics is desired (for example running it on a different port or with different auth) then endpoint can be enabled with the `--metrics-*` flags instead. Default Off. @@ -17260,6 +17749,76 @@ call and taken by the [options/set](#options-set) calls as well as the - `BandwidthSpec` - this will be set and returned as a string, eg "1M". +### Option blocks {#option-blocks} + +The calls [options/info](#options-info) (for the main config) and +[config/providers](#config-providers) (for the backend config) may be +used to get information on the rclone configuration options. This can +be used to build user interfaces for displaying and setting any rclone +option. + +These consist of arrays of `Option` blocks. These have the following +format. Each block describes a single option. + +| Field | Type | Optional | Description | +|-------|------|----------|-------------| +| Name | string | N | name of the option in snake_case | +| FieldName | string | N | name of the field used in the rc - if blank use Name | +| Help | string | N | help, started with a single sentence on a single line | +| Groups | string | Y | groups this option belongs to - comma separated string for options classification | +| Provider | string | Y | set to filter on provider | +| Default | any | N | default value, if set (and not to nil or "") then Required does nothing | +| Value | any | N | value to be set by flags | +| Examples | Examples | Y | predefined values that can be selected from list (multiple-choice option) | +| ShortOpt | string | Y | the short command line option for this | +| Hide | Visibility | N | if non zero, this option is hidden from the configurator or the command line | +| Required | bool | N | this option is required, meaning value cannot be empty unless there is a default | +| IsPassword | bool | N | set if the option is a password | +| NoPrefix | bool | N | set if the option for this should not use the backend prefix | +| Advanced | bool | N | set if this is an advanced config option | +| Exclusive | bool | N | set if the answer can only be one of the examples (empty string allowed unless Required or Default is set) | +| Sensitive | bool | N | set if this option should be redacted when using `rclone config redacted` | + +An example of this might be the `--log-level` flag. Note that the +`Name` of the option becomes the command line flag with `_` replaced +with `-`. + +``` +{ + "Advanced": false, + "Default": 5, + "DefaultStr": "NOTICE", + "Examples": [ + { + "Help": "", + "Value": "EMERGENCY" + }, + { + "Help": "", + "Value": "ALERT" + }, + ... + ], + "Exclusive": true, + "FieldName": "LogLevel", + "Groups": "Logging", + "Help": "Log level DEBUG|INFO|NOTICE|ERROR", + "Hide": 0, + "IsPassword": false, + "Name": "log_level", + "NoPrefix": true, + "Required": true, + "Sensitive": false, + "Type": "LogLevel", + "Value": null, + "ValueStr": "NOTICE" +}, +``` + +Note that the `Help` may be multiple lines separated by `\n`. The +first line will always be a short sentence and this is the sentence +shown when running `rclone help flags`. + ## Specifying remotes to work on Remotes are specified with the `fs=`, `srcFs=`, `dstFs=` @@ -17286,7 +17845,7 @@ For example this JSON is equivalent to `remote:/tmp` ``` { "_name": "remote", - "_path": "/tmp" + "_root": "/tmp" } ``` @@ -17296,7 +17855,7 @@ And this is equivalent to `:sftp,host='example.com':/tmp` { "type": "sftp", "host": "example.com", - "_path": "/tmp" + "_root": "/tmp" } ``` @@ -17305,7 +17864,7 @@ And this is equivalent to `/tmp/dir` ``` { type = "local", - _ path = "/tmp/dir" + _root = "/tmp/dir" } ``` @@ -17498,7 +18057,12 @@ See the [config paths](https://rclone.org/commands/rclone_config_paths/) command Returns a JSON object: - providers - array of objects -See the [config providers](https://rclone.org/commands/rclone_config_providers/) command for more information on the above. +See the [config providers](https://rclone.org/commands/rclone_config_providers/) command +for more information on the above. + +Note that the Options blocks are in the same format as returned by +"options/info". They are described in the +[option blocks](#option-blocks) section. **Authentication is required for this call.** @@ -18500,6 +19064,11 @@ Returns: Returns an object where keys are option block names and values are an object with the current option values in. +Parameters: + +- blocks: optional string of comma separated blocks to include + - all are included if this is missing or "" + Note that these are the global options which are unaffected by use of the _config and _filter parameters. If you wish to read the parameters set in _config then use options/config and for _filter use options/filter. @@ -18507,6 +19076,19 @@ set in _config then use options/config and for _filter use options/filter. This shows the internal names of the option within rclone which should map to the external options very easily with a few exceptions. +### options/info: Get info about all the global options {#options-info} + +Returns an object where keys are option block names and values are an +array of objects with info about each options. + +Parameters: + +- blocks: optional string of comma separated blocks to include + - all are included if this is missing or "" + +These objects are in the same format as returned by "config/providers". They are +described in the [option blocks](#option-blocks) section. + ### options/local: Get the currently active config for this call {#options-local} Returns an object with the keys "config" and "filter". @@ -18790,6 +19372,73 @@ If poll-interval is updated or disabled temporarily, some changes might not get picked up by the polling function, depending on the used remote. +This command takes an "fs" parameter. If this parameter is not +supplied and if there is only one VFS in use then that VFS will be +used. If there is more than one VFS in use then the "fs" parameter +must be supplied. + +### vfs/queue: Queue info for a VFS. {#vfs-queue} + +This returns info about the upload queue for the selected VFS. + +This is only useful if `--vfs-cache-mode` > off. If you call it when +the `--vfs-cache-mode` is off, it will return an empty result. + + { + "queued": // an array of files queued for upload + [ + { + "name": "file", // string: name (full path) of the file, + "id": 123, // integer: id of this item in the queue, + "size": 79, // integer: size of the file in bytes + "expiry": 1.5 // float: time until file is eligible for transfer, lowest goes first + "tries": 1, // integer: number of times we have tried to upload + "delay": 5.0, // float: seconds between upload attempts + "uploading": false, // boolean: true if item is being uploaded + }, + ], + } + +The `expiry` time is the time until the file is elegible for being +uploaded in floating point seconds. This may go negative. As rclone +only transfers `--transfers` files at once, only the lowest +`--transfers` expiry times will have `uploading` as `true`. So there +may be files with negative expiry times for which `uploading` is +`false`. + + +This command takes an "fs" parameter. If this parameter is not +supplied and if there is only one VFS in use then that VFS will be +used. If there is more than one VFS in use then the "fs" parameter +must be supplied. + +### vfs/queue-set-expiry: Set the expiry time for an item queued for upload. {#vfs-queue-set-expiry} + +Use this to adjust the `expiry` time for an item in the upload queue. +You will need to read the `id` of the item using `vfs/queue` before +using this call. + +You can then set `expiry` to a floating point number of seconds from +now when the item is eligible for upload. If you want the item to be +uploaded as soon as possible then set it to a large negative number (eg +-1000000000). If you want the upload of the item to be delayed +for a long time then set it to a large positive number. + +Setting the `expiry` of an item which has already has started uploading +will have no effect - the item will carry on being uploaded. + +This will return an error if called with `--vfs-cache-mode` off or if +the `id` passed is not found. + +This takes the following parameters + +- `fs` - select the VFS in use (optional) +- `id` - a numeric ID as returned from `vfs/queue` +- `expiry` - a new expiry time as floating point seconds + +This returns an empty result on success, or an error. + + This command takes an "fs" parameter. If this parameter is not supplied and if there is only one VFS in use then that VFS will be used. If there is more than one VFS in use then the "fs" parameter @@ -19096,7 +19745,9 @@ Here is an overview of the major features of each cloud storage system. | Citrix ShareFile | MD5 | R/W | Yes | No | - | - | | Dropbox | DBHASH ¹ | R | Yes | No | - | - | | Enterprise File Fabric | - | R/W | Yes | No | R/W | - | +| Files.com | MD5, CRC32 | DR/W | Yes | No | R | - | | FTP | - | R/W ¹⁰ | No | No | - | - | +| Gofile | MD5 | DR/W | No | Yes | R | - | | Google Cloud Storage | MD5 | R/W | No | No | R/W | - | | Google Drive | MD5, SHA1, SHA256 | DR/W | No | Yes | R/W | DRWU | | Google Photos | - | - | No | Yes | R | - | @@ -19118,6 +19769,7 @@ Here is an overview of the major features of each cloud storage system. | Oracle Object Storage | MD5 | R/W | No | No | R/W | - | | pCloud | MD5, SHA1 ⁷ | R | No | No | W | - | | PikPak | MD5 | R | No | No | R | - | +| Pixeldrain | SHA256 | R/W | No | No | R | RW | | premiumize.me | - | - | Yes | No | R | - | | put.io | CRC-32 | R/W | No | Yes | R | - | | Proton Drive | SHA1 | R/W | No | No | R | - | @@ -19420,8 +20072,8 @@ have a Windows file system with Unicode fullwidth characters remote rather than being translated to regular (halfwidth) `*`, `?` and `:`. The `--backend-encoding` flags allow you to change that. You can -disable the encoding completely with `--backend-encoding None` or set -`encoding = None` in the config file. +disable the encoding completely with `--backend-encoding Raw` or set +`encoding = Raw` in the config file. Encoding takes a comma separated list of encodings. You can see the list of all possible values by passing an invalid value to this @@ -19440,6 +20092,7 @@ will show you the defaults for the backends. | Dollar | `$` | `＄` | | Dot | `.` or `..` as entire string | `．`, `．．` | | DoubleQuote | `"` | `＂` | +| Exclamation | `!` | `！` | | Hash | `#` | `＃` | | InvalidUtf8 | An invalid UTF-8 character (e.g. latin1) | `�` | | LeftCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the left of a string | `␍`, `␊`, `␉`, `␋` | @@ -19447,7 +20100,7 @@ will show you the defaults for the backends. | LeftSpace | SPACE on the left of a string | `␠` | | LeftTilde | `~` on the left of a string | `～` | | LtGt | `<`, `>` | `＜`, `＞` | -| None | No characters are encoded | | +| None ¹ | NUL 0x00 | ␀ | | Percent | `%` | `％` | | Pipe | \| | `｜` | | Question | `?` | `？` | @@ -19459,6 +20112,10 @@ will show you the defaults for the backends. | Slash | `/` | `／` | | SquareBracket | `[`, `]` | `［`, `］` | +¹ Encoding from NUL 0x00 to ␀ is always implicit except when using Raw. +It was previously incorrectly documented as disabling encoding, +and to maintain backward compatibility, its behavior has not been changed. + ##### Encoding example: FTP To take a specific example, the FTP backend's default encoding is @@ -19502,7 +20159,7 @@ the default value but without `Colon,Question,Asterisk`: --local-encoding "Slash,LtGt,DoubleQuote,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot" ``` -Alternatively, you can disable the conversion of any characters with `--local-encoding None`. +Alternatively, you can disable the conversion of any characters with `--local-encoding Raw`. Instead of using command-line argument `--local-encoding`, you may also set it as [environment variable](https://rclone.org/docs/#environment-variables) `RCLONE_LOCAL_ENCODING`, @@ -19566,14 +20223,16 @@ upon backend-specific capabilities. | Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | No | No | No | No | Yes | | Dropbox | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes | | Enterprise File Fabric | Yes | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | +| Files.com | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | No | Yes | | FTP | No | No | Yes | Yes | No | No | Yes | No | No | No | Yes | +| Gofile | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes | | Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No | No | No | No | | Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | | Google Photos | No | No | No | No | No | No | No | No | No | No | No | | HDFS | Yes | No | Yes | Yes | No | No | Yes | No | No | Yes | Yes | | HiDrive | Yes | Yes | Yes | Yes | No | No | Yes | No | No | No | Yes | | HTTP | No | No | No | No | No | No | No | No | No | No | Yes | -| ImageKit | Yes | Yes | Yes | No | No | No | No | No | No | No | Yes | +| ImageKit | Yes | Yes | Yes | No | No | No | No | No | No | No | Yes | | Internet Archive | No | Yes | No | No | Yes | Yes | No | No | Yes | Yes | No | | Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | | Koofr | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes | @@ -19588,6 +20247,7 @@ upon backend-specific capabilities. | Oracle Object Storage | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | No | | pCloud | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes | | PikPak | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes | +| Pixeldrain | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes | | premiumize.me | Yes | No | Yes | Yes | No | No | No | No | Yes | Yes | Yes | | put.io | Yes | No | Yes | Yes | Yes | No | Yes | No | No | Yes | Yes | | Proton Drive | Yes | No | Yes | Yes | Yes | No | No | No | No | Yes | Yes | @@ -19711,12 +20371,12 @@ split into groups. ## Copy -Flags for anything which can Copy a file. +Flags for anything which can copy a file. ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -19751,7 +20411,7 @@ Flags for anything which can Copy a file. ## Sync -Flags just used for `rclone sync`. +Flags used for sync commands. ``` --backup-dir string Make backups into hierarchy based in DIR @@ -19782,7 +20442,7 @@ Important flags useful for most commands. ## Check -Flags used for `rclone check`. +Flags used for check commands. ``` --max-backlog int Maximum number of objects in sync or check backlog (default 10000) @@ -19791,7 +20451,7 @@ Flags used for `rclone check`. ## Networking -General networking and HTTP stuff. +Flags for general networking and HTTP stuff. ``` --bind string Local address to bind to for outgoing connections, IPv4, IPv6 or name @@ -19801,7 +20461,7 @@ General networking and HTTP stuff. --client-cert string Client SSL certificate (PEM) for mutual TLS auth --client-key string Client SSL private key (PEM) for mutual TLS auth --contimeout Duration Connect timeout (default 1m0s) - --disable-http-keep-alives Disable HTTP keep-alives and use each connection once. + --disable-http-keep-alives Disable HTTP keep-alives and use each connection once --disable-http2 Disable HTTP/2 in the global transport --dscp string Set DSCP value to connections, value or name, e.g. CS1, LE, DF, AF21 --expect-continue-timeout Duration Timeout when using expect / 100-continue in HTTP (default 1s) @@ -19814,7 +20474,7 @@ General networking and HTTP stuff. --tpslimit float Limit HTTP transactions per second to this --tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --use-cookies Enable session cookiejar - --user-agent string Set the user-agent to a specified string (default "rclone/v1.67.0") + --user-agent string Set the user-agent to a specified string (default "rclone/v1.68.0") ``` @@ -19831,7 +20491,7 @@ Flags helpful for increasing performance. ## Config -General configuration of rclone. +Flags for general configuration of rclone. ``` --ask-password Allow prompt for password for encrypted configuration (default true) @@ -19915,7 +20575,7 @@ Flags for listing directories. ## Logging -Logging and statistics. +Flags for logging and statistics. ``` --log-file string Log everything to this file @@ -19934,7 +20594,7 @@ Logging and statistics. --stats-one-line-date-format string Enable --stats-one-line-date and use custom formatted date: Enclose date string in double quotes ("), see https://golang.org/pkg/time/#Time.Format --stats-unit string Show data rate in stats as either 'bits' or 'bytes' per second (default "bytes") --syslog Use Syslog for logging - --syslog-facility string Facility for syslog, e.g. KERN,USER,... (default "DAEMON") + --syslog-facility string Facility for syslog, e.g. KERN,USER (default "DAEMON") --use-json-log Use json log format -v, --verbose count Print lots more stuff (repeat for more) ``` @@ -19963,12 +20623,12 @@ Flags to control the Remote Control API. ``` --rc Enable the remote control server - --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572]) + --rc-addr stringArray IPaddress:Port or :Port to bind server to (default ["localhost:5572"]) --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from --rc-baseurl string Prefix for URLs - leave blank for root --rc-cert string TLS PEM key (concatenation of certificate and CA certificate) --rc-client-ca string Client certificate authority to verify clients with - --rc-enable-metrics Enable prometheus metrics on /metrics + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server --rc-files string Path to local files to serve on the HTTP server --rc-htpasswd string A htpasswd file - if not provided no authentication is done --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s) @@ -19994,9 +20654,34 @@ Flags to control the Remote Control API. ``` +## Metrics + +Flags to control the Metrics HTTP endpoint.. + +``` + --metrics-addr stringArray IPaddress:Port or :Port to bind metrics server to (default [""]) + --metrics-allow-origin string Origin which cross-domain request (CORS) can be executed from + --metrics-baseurl string Prefix for URLs - leave blank for root + --metrics-cert string TLS PEM key (concatenation of certificate and CA certificate) + --metrics-client-ca string Client certificate authority to verify clients with + --metrics-htpasswd string A htpasswd file - if not provided no authentication is done + --metrics-key string TLS PEM Private key + --metrics-max-header-bytes int Maximum size of request header (default 4096) + --metrics-min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") + --metrics-pass string Password for authentication + --metrics-realm string Realm for authentication + --metrics-salt string Password hashing salt (default "dlPL2MqE") + --metrics-server-read-timeout Duration Timeout for server reading data (default 1h0m0s) + --metrics-server-write-timeout Duration Timeout for server writing data (default 1h0m0s) + --metrics-template string User-specified template + --metrics-user string User name for authentication + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server +``` + + ## Backend -Backend only flags. These can be set in the config file also. +Backend-only flags (these can be set in the config file also). ``` --alias-description string Description of the remote @@ -20220,6 +20905,12 @@ Backend only flags. These can be set in the config file also. --filefabric-token-expiry string Token expiry time --filefabric-url string URL of the Enterprise File Fabric to connect to --filefabric-version string Version read from the file fabric + --filescom-api-key string The API key used to authenticate with Files.com + --filescom-description string Description of the remote + --filescom-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot) + --filescom-password string The password used to authenticate with Files.com (obscured) + --filescom-site string Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com) + --filescom-username string The username used to authenticate with Files.com --ftp-ask-password Allow asking for FTP password when needed --ftp-close-timeout Duration Maximum time to wait for a response to close (default 1m0s) --ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited @@ -20263,6 +20954,12 @@ Backend only flags. These can be set in the config file also. --gcs-token string OAuth Access Token as a JSON blob --gcs-token-url string Token server url --gcs-user-project string User project + --gofile-access-token string API Access token + --gofile-account-id string Account ID + --gofile-description string Description of the remote + --gofile-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftPeriod,RightPeriod,InvalidUtf8,Dot,Exclamation) + --gofile-list-chunk int Number of items to list in each call (default 1000) + --gofile-root-folder-id string ID of the root folder --gphotos-auth-url string Auth server URL --gphotos-batch-commit-timeout Duration Max time to wait for a batch to finish committing (default 10m0s) --gphotos-batch-mode string Upload file batching sync|async|off (default "sync") @@ -20354,6 +21051,7 @@ Backend only flags. These can be set in the config file also. --local-description string Description of the remote --local-encoding Encoding The encoding for the backend (default Slash,Dot) --local-no-check-updated Don't check to see if the files change during upload + --local-no-clone Disable reflink cloning for server-side copies --local-no-preallocate Disable preallocation of disk space for transferred files --local-no-set-modtime Disable setting modtime --local-no-sparse Disable sparse files for multi-thread downloads @@ -20470,6 +21168,10 @@ Backend only flags. These can be set in the config file also. --pikpak-upload-concurrency int Concurrency for multipart uploads (default 5) --pikpak-use-trash Send files to the trash instead of deleting permanently (default true) --pikpak-user string Pikpak username + --pixeldrain-api-key string API key for your pixeldrain account + --pixeldrain-api-url string The API endpoint to connect to. In the vast majority of cases it's fine to leave (default "https://pixeldrain.com/api") + --pixeldrain-description string Description of the remote + --pixeldrain-root-folder-id string Root of the filesystem to use (default "me") --premiumizeme-auth-url string Auth server URL --premiumizeme-client-id string OAuth Client Id --premiumizeme-client-secret string OAuth Client Secret @@ -20544,6 +21246,7 @@ Backend only flags. These can be set in the config file also. --s3-provider string Choose your S3 provider --s3-region string Region to connect to --s3-requester-pays Enables requester pays option when interacting with S3 bucket + --s3-sdk-log-mode Bits Set to debug the SDK (default Off) --s3-secret-access-key string AWS Secret Access Key (password) --s3-server-side-encryption string The server-side encryption algorithm used when storing this object in S3 --s3-session-token string An AWS session token @@ -20554,7 +21257,6 @@ Backend only flags. These can be set in the config file also. --s3-sse-customer-key-md5 string If using SSE-C you may provide the secret encryption key MD5 checksum (optional) --s3-sse-kms-key-id string If using KMS ID you must provide the ARN of Key --s3-storage-class string The storage class to use when storing new objects in S3 - --s3-sts-endpoint string Endpoint for STS --s3-upload-concurrency int Concurrency for multipart uploads and copies (default 4) --s3-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi) --s3-use-accelerate-endpoint If true use the AWS S3 accelerated endpoint @@ -20564,6 +21266,7 @@ Backend only flags. These can be set in the config file also. --s3-use-multipart-etag Tristate Whether to use ETag in multipart uploads for verification (default unset) --s3-use-multipart-uploads Tristate Set if rclone should use multipart uploads (default unset) --s3-use-presigned-request Whether to use a presigned request or PutObject for single part uploads + --s3-use-unsigned-payload Tristate Whether to use an unsigned payload in PutObject (default unset) --s3-v2-auth If true use v2 authentication --s3-version-at Time Show file versions as they were at the specified time (default off) --s3-version-deleted Show deleted file markers when using versions @@ -20672,10 +21375,12 @@ Backend only flags. These can be set in the config file also. --swift-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8) --swift-endpoint-type string Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) (default "public") --swift-env-auth Get swift credentials from environment variables in standard OpenStack form + --swift-fetch-until-empty-page When paginating, always fetch unless we received an empty page --swift-key string API key or password (OS_PASSWORD) --swift-leave-parts-on-error If true avoid calling abort upload on a failure --swift-no-chunk Don't chunk files during streaming upload --swift-no-large-objects Disable support for static and dynamic large objects + --swift-partial-page-fetch-threshold int When paginating, fetch if the current page is within this percentage of the limit --swift-region string Region name - optional (OS_REGION_NAME) --swift-storage-policy string The storage policy to use when creating a new container --swift-storage-url string Storage URL - optional (OS_STORAGE_URL) @@ -20713,6 +21418,7 @@ Backend only flags. These can be set in the config file also. --webdav-owncloud-exclude-shares Exclude ownCloud shares --webdav-pacer-min-sleep Duration Minimum time to sleep between API calls (default 10ms) --webdav-pass string Password (obscured) + --webdav-unix-socket string Path to a unix domain socket to dial to, instead of opening a TCP connection directly --webdav-url string URL of http host to connect to --webdav-user string User name --webdav-vendor string Name of the WebDAV site/service/software you are using @@ -20722,6 +21428,7 @@ Backend only flags. These can be set in the config file also. --yandex-description string Description of the remote --yandex-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot) --yandex-hard-delete Delete files permanently rather than putting them into the trash + --yandex-spoof-ua Set the user agent to match an official version of the yandex disk client. May help with upload performance (default true) --yandex-token string OAuth Access Token as a JSON blob --yandex-token-url string Token server url --zoho-auth-url string Auth server URL @@ -23097,6 +23804,12 @@ about _Unison_ and synchronization in general. ## Changelog +### `v1.68` +* Fixed an issue affecting backends that round modtimes to a lower precision. + +### `v1.67` +* Added integration tests against all backends. + ### `v1.66` * Copies and deletes are now handled in one operation instead of two * `--track-renames` and `--backup-dir` are now supported @@ -23284,7 +23997,7 @@ $ rclone hashsum sha256 -C SHA256SUMS rclone-v1.63.1-windows-amd64.zip You can verify the signatures and hashes in one command line like this: ``` -$ gpg --decrypt SHA256SUMS | sha256sum -c --ignore-missing +$ h=$(gpg --decrypt SHA256SUMS) && echo "$h" | sha256sum - -c --ignore-missing gpg: Signature made Mon 17 Jul 2023 15:03:17 BST gpg: using DSA key FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA gpg: Good signature from "Nick Craig-Wood " [ultimate] @@ -23339,11 +24052,11 @@ y) Yes n) No y/n> Remote config --------------------- -[remote] -type = fichier -api_key = example_key --------------------- +Configuration complete. +Options: +- type: fichier +- api_key: example_key +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -23557,10 +24270,11 @@ Remote or path to alias. Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path". remote> /mnt/storage/backup Remote config --------------------- -[remote] -remote = /mnt/storage/backup --------------------- +Configuration complete. +Options: +- type: alias +- remote: /mnt/storage/backup +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -23878,20 +24592,20 @@ Choose a number from below, or type in your own value \ "GLACIER_IR" storage_class> 1 Remote config --------------------- -[remote] -type = s3 -provider = AWS -env_auth = false -access_key_id = XXX -secret_access_key = YYY -region = us-east-1 -endpoint = -location_constraint = -acl = private -server_side_encryption = -storage_class = --------------------- +Configuration complete. +Options: +- type: s3 +- provider: AWS +- env_auth: false +- access_key_id: XXX +- secret_access_key: YYY +- region: us-east-1 +- endpoint: +- location_constraint: +- acl: private +- server_side_encryption: +- storage_class: +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -24197,15 +24911,21 @@ The different authentication methods are tried in this order: - Session Token: `AWS_SESSION_TOKEN` (optional) - Or, use a [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html): - Profile files are standard files used by AWS CLI tools - - By default it will use the profile in your home directory (e.g. `~/.aws/credentials` on unix based systems) file and the "default" profile, to change set these environment variables: - - `AWS_SHARED_CREDENTIALS_FILE` to control which file. - - `AWS_PROFILE` to control which profile to use. + - By default it will use the profile in your home directory (e.g. `~/.aws/credentials` on unix based systems) file and the "default" profile, to change set these environment variables or config keys: + - `AWS_SHARED_CREDENTIALS_FILE` to control which file or the `shared_credentials_file` config key. + - `AWS_PROFILE` to control which profile to use or the `profile` config key. - Or, run `rclone` in an ECS task with an IAM role (AWS only). - Or, run `rclone` on an EC2 instance with an IAM role (AWS only). - Or, run `rclone` in an EKS pod with an IAM role that is associated with a service account (AWS only). + - Or, use [process credentials](https://docs.aws.amazon.com/sdkref/latest/guide/feature-process-credentials.html) to read config from an external program. + +With `env_auth = true` rclone (which uses the SDK for Go v2) should support +[all authentication methods](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html) +that the `aws` CLI tool does and the other AWS SDKs. If none of these option actually end up providing `rclone` with AWS -credentials then S3 interaction will be non-authenticated (see below). +credentials then S3 interaction will be non-authenticated (see the +[anonymous access](#anonymous-access) section for more info). ### S3 Permissions @@ -25029,6 +25749,10 @@ Some providers (e.g. AWS, Aliyun OSS, Netease COS, or Tencent COS) require this false - rclone will do this automatically based on the provider setting. +Note that if your bucket isn't a valid DNS name, i.e. has '.' or '_' in, +you'll need to set this to true. + + Properties: - Config: force_path_style @@ -25318,6 +26042,24 @@ Properties: - Type: Tristate - Default: unset +#### --s3-use-unsigned-payload + +Whether to use an unsigned payload in PutObject + +Rclone has to avoid the AWS SDK seeking the body when calling +PutObject. The AWS provider can add checksums in the trailer to avoid +seeking but other providers can't. + +This should be true, false or left unset to use the default for the provider. + + +Properties: + +- Config: use_unsigned_payload +- Env Var: RCLONE_S3_USE_UNSIGNED_PAYLOAD +- Type: Tristate +- Default: unset + #### --s3-use-presigned-request Whether to use a presigned request or PutObject for single part uploads @@ -25475,7 +26217,7 @@ Properties: #### --s3-sts-endpoint -Endpoint for STS. +Endpoint for STS (deprecated). Leave blank if using AWS to use the default endpoint for the region. @@ -25538,6 +26280,33 @@ Properties: - Type: Tristate - Default: unset +#### --s3-sdk-log-mode + +Set to debug the SDK + +This can be set to a comma separated list of the following functions: + +- `Signing` +- `Retries` +- `Request` +- `RequestWithBody` +- `Response` +- `ResponseWithBody` +- `DeprecatedUsage` +- `RequestEventMessage` +- `ResponseEventMessage` + +Use `Off` to disable and `All` to set all log levels. You will need to +use `-vv` to see the debug level logs. + + +Properties: + +- Config: sdk_log_mode +- Env Var: RCLONE_S3_SDK_LOG_MODE +- Type: Bits +- Default: Off + #### --s3-description Description of the remote. @@ -25586,18 +26355,19 @@ These can be run on a running backend using the rc command ### restore -Restore objects from GLACIER to normal storage +Restore objects from GLACIER or INTELLIGENT-TIERING archive tier rclone backend restore remote: [options] [+] -This command can be used to restore one or more objects from GLACIER -to normal storage. +This command can be used to restore one or more objects from GLACIER to normal storage +or from INTELLIGENT-TIERING Archive Access / Deep Archive Access tier to the Frequent Access tier. Usage Examples: rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS + rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY This flag also obeys the filters. Test first with --interactive/-i or --dry-run flags @@ -25627,17 +26397,17 @@ if not. Options: - "description": The optional description for the job. -- "lifetime": Lifetime of the active copy in days +- "lifetime": Lifetime of the active copy in days, ignored for INTELLIGENT-TIERING storage - "priority": Priority of restore: Standard|Expedited|Bulk ### restore-status -Show the restore status for objects being restored from GLACIER to normal storage +Show the restore status for objects being restored from GLACIER or INTELLIGENT-TIERING storage rclone backend restore-status remote: [options] [+] -This command can be used to show the status for objects being restored from GLACIER -to normal storage. +This command can be used to show the status for objects being restored from GLACIER to normal storage +or from INTELLIGENT-TIERING Archive Access / Deep Archive Access tier to the Frequent Access tier. Usage Examples: @@ -25667,6 +26437,15 @@ It returns a list of status dictionaries. "RestoreExpiryDate": "2023-09-06T12:29:19+01:00" }, "StorageClass": "DEEP_ARCHIVE" + }, + { + "Remote": "test.gz", + "VersionID": null, + "RestoreStatus": { + "IsRestoreInProgress": true, + "RestoreExpiryDate": "null" + }, + "StorageClass": "INTELLIGENT_TIERING" } ] @@ -25795,7 +26574,7 @@ It doesn't return anything. -### Anonymous access to public buckets +### Anonymous access to public buckets {#anonymous-access} If you want to use rclone to access a public bucket, configure with a blank `access_key_id` and `secret_access_key`. Your config should end @@ -25805,15 +26584,6 @@ up looking like this: [anons3] type = s3 provider = AWS -env_auth = false -access_key_id = -secret_access_key = -region = us-east-1 -endpoint = -location_constraint = -acl = private -server_side_encryption = -storage_class = ``` Then use it as normal with the name of the public bucket, e.g. @@ -25822,6 +26592,10 @@ Then use it as normal with the name of the public bucket, e.g. You will be able to list and copy data but not upload it. +You can also do this entirely on the command line + + rclone lsd :s3,provider=AWS:1000genomes + ## Providers ### AWS S3 @@ -26011,7 +26785,14 @@ acl = private Now run `rclone lsf r2:` to see your buckets and `rclone lsf r2:bucket` to look within a bucket. -For R2 tokens with the "Object Read & Write" permission, you may also need to add `no_check_bucket = true` for object uploads to work correctly. +For R2 tokens with the "Object Read & Write" permission, you may also +need to add `no_check_bucket = true` for object uploads to work +correctly. + +Note that Cloudflare decompresses files uploaded with +`Content-Encoding: gzip` by default which is a deviation from what AWS +does. If this is causing a problem then upload the files with +`--header-upload "Cache-Control: no-transform"` ### Dreamhost @@ -28642,6 +29423,28 @@ nodes across the network. For more detailed comparison please check the documentation of the [storj](/storj) backend. +## Memory usage {memory} + +The most common cause of rclone using lots of memory is a single +directory with millions of files in. Despite s3 not really having the +concepts of directories, rclone does the sync on a directory by +directory basis to be compatible with normal filing systems. + +Rclone loads each directory into memory as rclone objects. Each rclone +object takes 0.5k-1k of memory, so approximately 1GB per 1,000,000 +files, and the sync for that directory does not begin until it is +entirely loaded in memory. So the sync can take a long time to start +for large directories. + +To sync a directory with 100,000,000 files in you would need approximately +100 GB of memory. At some point the amount of memory becomes difficult +to provide so there is +[a workaround for this](https://github.com/rclone/rclone/wiki/Big-syncs-with-millions-of-files) +which involves a bit of scripting. + +At some point rclone will gain a sync mode which is effectively this +workaround but built in to rclone. + ## Limitations `rclone about` is not supported by the S3 backend. Backends without @@ -28652,7 +29455,6 @@ remote. See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/) - ### Synology C2 Object Storage {#synology-c2} [Synology C2 Object Storage](https://c2.synology.com/en-global/object-storage/overview) provides a secure, S3-compatible, and cost-effective cloud storage solution without API request, download fees, and deletion penalty. @@ -28825,12 +29627,13 @@ key> 0123456789abcdef0123456789abcdef0123456789 Endpoint for the service - leave blank normally. endpoint> Remote config --------------------- -[remote] -account = 123456789abc -key = 0123456789abcdef0123456789abcdef0123456789 -endpoint = --------------------- +Configuration complete. +Options: +- type: b2 +- account: 123456789abc +- key: 0123456789abcdef0123456789abcdef0123456789 +- endpoint: +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -28946,12 +29749,21 @@ used. ### Versions -When rclone uploads a new version of a file it creates a [new version +The default setting of B2 is to keep old versions of files. This means +when rclone uploads a new version of a file it creates a [new version of it](https://www.backblaze.com/docs/cloud-storage-file-versions). Likewise when you delete a file, the old version will be marked hidden -and still be available. Conversely, you may opt in to a "hard delete" -of files with the `--b2-hard-delete` flag which would permanently remove -the file instead of hiding it. +and still be available. + +Whether B2 keeps old versions of files or not can be adjusted on a per +bucket basis using the "Lifecycle settings" on the B2 control panel or +when creating the bucket using the [--b2-lifecycle](#b2-lifecycle) +flag or after creation using the [rclone backend lifecycle](#lifecycle) +command. + +You may opt in to a "hard delete" of files with the `--b2-hard-delete` +flag which permanently removes files on deletion instead of hiding +them. Old versions of files, where available, are visible using the `--b2-versions` flag. @@ -29616,12 +30428,13 @@ If your browser doesn't open automatically go to the following link: http://127. Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -client_id = -client_secret = -token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} --------------------- +Configuration complete. +Options: +- type: box +- client_id: +- client_secret: +- token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -29704,11 +30517,11 @@ e/n/d/r/c/s/q> e Choose a number from below, or type in an existing value 1 > remote remote> remote --------------------- -[remote] -type = box -token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"} --------------------- +Configuration complete. +Options: +- type: box +- token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"} +Keep this "remote" remote? Edit remote Value "client_id" = "" Edit? (y/n)> @@ -29736,11 +30549,11 @@ If your browser doesn't open automatically go to the following link: http://127. Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = box -token = {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"} --------------------- +Configuration complete. +Options: +- type: box +- token: {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -31322,12 +32135,12 @@ If your browser doesn't open automatically go to the following link: http://127. Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = sharefile -endpoint = https://XXX.sharefile.com -token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} --------------------- +Configuration complete. +Options: +- type: sharefile +- endpoint: https://XXX.sharefile.com +- token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -32659,11 +33472,11 @@ Embedded spaces can be added using quotes "dir=remote:path with space" "dir2=remote2:path with space" Enter a fs.SpaceSepList value. upstreams> images=s3:imagesbucket files=drive:important/files --------------------- -[remote] -type = combine -upstreams = images=s3:imagesbucket files=drive:important/files --------------------- +Configuration complete. +Options: +- type: combine +- upstreams: images=s3:imagesbucket files=drive:important/files +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -32792,12 +33605,13 @@ Remote config Please visit: https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX --------------------- -[remote] -app_key = -app_secret = -token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX --------------------- +Configuration complete. +Options: +- type: dropbox +- app_key: +- app_secret: +- token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -33160,7 +33974,7 @@ Max number of files in upload batch. This sets the batch size of files to upload. It has to be less than 1000. -By default this is 0 which means rclone which calculate the batch size +By default this is 0 which means rclone will calculate the batch size depending on the setting of batch_mode. - batch_mode: async - default batch_size is 100 @@ -33350,12 +34164,12 @@ y) Yes n) No (default) y/n> n Remote config --------------------- -[remote] -type = filefabric -url = https://yourfabric.smestorage.com/ -permanent_token = xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx --------------------- +Configuration complete. +Options: +- type: filefabric +- url: https://yourfabric.smestorage.com/ +- permanent_token: xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -33562,6 +34376,182 @@ Properties: +# Files.com + +[Files.com](https://www.files.com/) is a cloud storage service that provides a +secure and easy way to store and share files. + +The initial setup for filescom involves authenticating with your Files.com +account. You can do this by providing your site subdomain, username, and +password. Alternatively, you can authenticate using an API Key from +[Files.com](https://www.files.com/docs/sdk-and-apis/api-keys/). +`rclone config` walks you through it. + +## Configuration + +Here is an example of how to make a remote called `remote`. First run: + + rclone config + +This will guide you through an interactive setup process: + + No remotes found, make a new one? + n) New remote + s) Set configuration password + q) Quit config + n/s/q> n + + Enter name for new remote. + name> remote + + Option Storage. + Type of storage to configure. + Choose a number from below, or type in your own value. + [snip] + XX / Files.com + \ "filescom" + [snip] + Storage> filescom + + Option site. + Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com) + Enter a value. Press Enter to leave empty. + site> mysite + + Option username. + The username used to authenticate with Files.com. + Enter a value. Press Enter to leave empty. + username> user + + Option password. + The password used to authenticate with Files.com. + Choose an alternative below. Press Enter for the default (n). + y) Yes, type in my own password + g) Generate random password + n) No, leave this optional password blank (default) + y/g/n> y + Enter the password: + password: + Confirm the password: + password: + + Edit advanced config? + y) Yes + n) No (default) + y/n> n + + Configuration complete. + Options: + - type: filescom + - site: mysite + - username: user + - password: *** ENCRYPTED *** + Keep this "remote" remote? + y) Yes this is OK (default) + e) Edit this remote + d) Delete this remote + y/e/d> y + +Once configured you can use rclone. + +See all files in the top level: + + rclone lsf remote: + +Make a new directory in the root: + + rclone mkdir remote:dir + +Recursively List the contents: + + rclone ls remote: + +Sync `/home/local/directory` to the remote directory, deleting any +excess files in the directory. + + rclone sync --interactive /home/local/directory remote:dir + + +### Standard options + +Here are the Standard options specific to filescom (Files.com). + +#### --filescom-site + +Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com). + +Properties: + +- Config: site +- Env Var: RCLONE_FILESCOM_SITE +- Type: string +- Required: false + +#### --filescom-username + +The username used to authenticate with Files.com. + +Properties: + +- Config: username +- Env Var: RCLONE_FILESCOM_USERNAME +- Type: string +- Required: false + +#### --filescom-password + +The password used to authenticate with Files.com. + +**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). + +Properties: + +- Config: password +- Env Var: RCLONE_FILESCOM_PASSWORD +- Type: string +- Required: false + +### Advanced options + +Here are the Advanced options specific to filescom (Files.com). + +#### --filescom-api-key + +The API key used to authenticate with Files.com. + +Properties: + +- Config: api_key +- Env Var: RCLONE_FILESCOM_API_KEY +- Type: string +- Required: false + +#### --filescom-encoding + +The encoding for the backend. + +See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. + +Properties: + +- Config: encoding +- Env Var: RCLONE_FILESCOM_ENCODING +- Type: Encoding +- Default: Slash,BackSlash,Del,Ctl,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot + +#### --filescom-description + +Description of the remote. + +Properties: + +- Config: description +- Env Var: RCLONE_FILESCOM_DESCRIPTION +- Type: string +- Required: false + + + # FTP FTP is the File Transfer Protocol. Rclone FTP support is provided using the @@ -33630,12 +34620,12 @@ Use FTP over TLS (Explicit) Enter a boolean value (true or false). Press Enter for the default ("false"). explicit_tls> Remote config --------------------- -[remote] -type = ftp -host = ftp.example.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: ftp +- host: ftp.example.com +- pass: *** ENCRYPTED *** +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -34075,6 +35065,268 @@ designating the remote time precision expressed as nanoseconds. A value of `1000000000` means that file time precision of 1 second is available. A value of `3153600000000000000` (or another large number) means "unsupported". +# Gofile + +[Gofile](https://gofile.io) is a content storage and distribution +platform. Its aim is to provide as much service as possible for free +or at a very low price. + +The initial setup for Gofile involves logging in to the web interface +and going to the "My Profile" section. Copy the "Account API token" +for use in the config file. + +Note that if you wish to connect rclone to Gofile you will need a +premium account. + +## Configuration + +Here is an example of how to make a remote called `remote`. First run: + + rclone config + +This will guide you through an interactive setup process: + +``` +No remotes found, make a new one? +n) New remote +s) Set configuration password +q) Quit config +n/s/q> n + +Enter name for new remote. +name> remote + +Option Storage. +Type of storage to configure. +Choose a number from below, or type in your own value. +XX / Gofile + \ (gofile) +Storage> gofile + +Option access_token. +API Access token +You can get this from the web control panel. +Enter a value. Press Enter to leave empty. +access_token> YOURACCESSTOKEN + +Edit advanced config? +y) Yes +n) No (default) +y/n> n + +Configuration complete. +Options: +- type: gofile +- access_token: YOURACCESSTOKEN +Keep this "remote" remote? +y) Yes this is OK (default) +e) Edit this remote +d) Delete this remote +y/e/d> y +``` + +Once configured you can then use `rclone` like this, + +List directories and files in the top level of your Gofile + + rclone lsf remote: + +To copy a local directory to an Gofile directory called backup + + rclone copy /home/source remote:backup + +### Modification times and hashes + +Gofile supports modification times with a resolution of 1 second. + +Gofile supports MD5 hashes, so you can use the `--checksum` flag. + +### Restricted filename characters + +In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) +the following characters are also replaced: + +| Character | Value | Replacement | +| --------- |:-----:|:-----------:| +| ! | 0x21 | ！ | +| " | 0x22 | ＂ | +| * | 0x2A | ＊ | +| : | 0x3A | ： | +| < | 0x3C | ＜ | +| > | 0x3E | ＞ | +| ? | 0x3F | ？ | +| \ | 0x5C | ＼ | +| \| | 0x7C | ｜ | + + +File names can also not start or end with the following characters. +These only get replaced if they are the first or last character in the +name: + +| Character | Value | Replacement | +| --------- |:-----:|:-----------:| +| . | 0x2E | ． | + +Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), +as they can't be used in JSON strings. + +### Public Links + +Gofile supports `rclone link` to make public links to files or +directories. If you specify a directory it will download as a `zip` +file. You can use the `--expire` flag to specify the time the link +should be valid. Note that `rclone link --unlink` removes all the +public links for a file. + +### Root folder ID + +You can set the `root_folder_id` for rclone. This is the directory +(identified by its `Folder ID`) that rclone considers to be the root +of your Gofile drive. + +Normally you will leave this blank and rclone will determine the +correct root to use itself and fill in the value in the config file. + +However you can set this to restrict rclone to a specific folder +hierarchy. + +In order to do this you will have to find the `Folder ID` of the +directory you wish rclone to display. + +You can do this with rclone + +``` +$ rclone lsf -Fip --dirs-only remote: +d6341f53-ee65-4f29-9f59-d11e8070b2a0;Files/ +f4f5c9b8-6ece-478b-b03e-4538edfe5a1c;Photos/ +d50e356c-29ca-4b27-a3a7-494d91026e04;Videos/ +``` + +The ID to use is the part before the `;` so you could set + +``` +root_folder_id = d6341f53-ee65-4f29-9f59-d11e8070b2a0 +``` + +To restrict rclone to the `Files` directory. + + +### Standard options + +Here are the Standard options specific to gofile (Gofile). + +#### --gofile-access-token + +API Access token + +You can get this from the web control panel. + +Properties: + +- Config: access_token +- Env Var: RCLONE_GOFILE_ACCESS_TOKEN +- Type: string +- Required: false + +### Advanced options + +Here are the Advanced options specific to gofile (Gofile). + +#### --gofile-root-folder-id + +ID of the root folder + +Leave this blank normally, rclone will fill it in automatically. + +If you want rclone to be restricted to a particular folder you can +fill it in - see the docs for more info. + + +Properties: + +- Config: root_folder_id +- Env Var: RCLONE_GOFILE_ROOT_FOLDER_ID +- Type: string +- Required: false + +#### --gofile-account-id + +Account ID + +Leave this blank normally, rclone will fill it in automatically. + + +Properties: + +- Config: account_id +- Env Var: RCLONE_GOFILE_ACCOUNT_ID +- Type: string +- Required: false + +#### --gofile-list-chunk + +Number of items to list in each call + +Properties: + +- Config: list_chunk +- Env Var: RCLONE_GOFILE_LIST_CHUNK +- Type: int +- Default: 1000 + +#### --gofile-encoding + +The encoding for the backend. + +See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. + +Properties: + +- Config: encoding +- Env Var: RCLONE_GOFILE_ENCODING +- Type: Encoding +- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftPeriod,RightPeriod,InvalidUtf8,Dot,Exclamation + +#### --gofile-description + +Description of the remote. + +Properties: + +- Config: description +- Env Var: RCLONE_GOFILE_DESCRIPTION +- Type: string +- Required: false + + + +## Limitations + +Gofile only supports filenames up to 255 characters in length, where a +character is a unicode character. + +Directories should not be cached for more than 24h otherwise files in +the directory may not be downloadable. In practice this means when +using a VFS based rclone command such as `rclone mount` you should +make sure `--dir-cache-time` is less than `24h`. + +Note that Gofile is currently limited to a total of 100,000 items. If +you attempt to upload more than that you will get +`error-limit-100000`. This limit may be lifted in the future. + +### Duplicated files + +Gofile is capable of having files with duplicated file names. For +instance two files called `hello.txt` in the same directory. + +Rclone cannot sync that to a normal file system but it can be fixed +with the `rclone dedupe` command. + +Duplicated files cause problems with the syncing and you will see +messages in the log about duplicates. + +Use `rclone dedupe` to fix duplicated files. + # Google Cloud Storage Paths are specified as `remote:bucket` (or `remote:` for the `lsd` @@ -34199,16 +35451,16 @@ If your browser doesn't open automatically go to the following link: http://127. Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = google cloud storage -client_id = -client_secret = -token = {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null} -project_number = 12345678 -object_acl = private -bucket_acl = private --------------------- +Configuration complete. +Options: +- type: google cloud storage +- client_id: +- client_secret: +- token: {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null} +- project_number: 12345678 +- object_acl: private +- bucket_acl: private +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -34865,15 +36117,16 @@ Configure this as a Shared Drive (Team Drive)? y) Yes n) No y/n> n --------------------- -[remote] -client_id = -client_secret = -scope = drive -root_folder_id = -service_account_file = -token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"} --------------------- +Configuration complete. +Options: +type: drive +- client_id: +- client_secret: +- scope: drive +- root_folder_id: +- service_account_file: +- token: {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -35001,42 +36254,49 @@ credentials file into the rclone config file, you can set `service_account_credentials` with the actual contents of the file instead, or set the equivalent environment variable. -#### Use case - Google Apps/G-suite account and individual Drive +#### Use case - Google Workspace account and individual Drive -Let's say that you are the administrator of a Google Apps (old) or -G-suite account. -The goal is to store data on an individual's Drive account, who IS -a member of the domain. -We'll call the domain **example.com**, and the user -**foo@example.com**. +Let's say that you are the administrator of a Google Workspace. The +goal is to read or write data on an individual's Drive account, who IS +a member of the domain. We'll call the domain **example.com**, and the +user **foo@example.com**. There's a few steps we need to go through to accomplish this: ##### 1. Create a service account for example.com + - To create a service account and obtain its credentials, go to the [Google Developer Console](https://console.developers.google.com). - - You must have a project - create one if you don't. + - You must have a project - create one if you don't and make sure you are on the selected project. - Then go to "IAM & admin" -> "Service Accounts". - Use the "Create Service Account" button. Fill in "Service account name" and "Service account ID" with something that identifies your client. - Select "Create And Continue". Step 2 and 3 are optional. - - These credentials are what rclone will use for authentication. + - Click on the newly created service account + - Click "Keys" and then "Add Key" and then "Create new key" + - Choose type "JSON" and click create + - This will download a small JSON file that rclone will use for authentication. + If you ever need to remove access, press the "Delete service account key" button. ##### 2. Allowing API access to example.com Google Drive - - Go to example.com's admin console + + - Go to example.com's [Workspace Admin Console](https://admin.google.com) - Go into "Security" (or use the search bar) - - Select "Show more" and then "Advanced settings" - - Select "Manage API client access" in the "Authentication" section - - In the "Client Name" field enter the service account's + - Select "Access and data control" and then "API controls" + - Click "Manage domain-wide delegation" + - Click "Add new" + - In the "Client ID" field enter the service account's "Client ID" - this can be found in the Developer Console under "IAM & Admin" -> "Service Accounts", then "View Client ID" for the newly created service account. It is a ~21 character numerical string. - - In the next field, "One or More API Scopes", enter + - In the next field, "OAuth Scopes", enter `https://www.googleapis.com/auth/drive` -to grant access to Google Drive specifically. +to grant read/write access to Google Drive specifically. +You can also use `https://www.googleapis.com/auth/drive.readonly` for read only access. + - Click "Authorise" ##### 3. Configure rclone, assuming a new install @@ -35045,17 +36305,18 @@ rclone config n/s/q> n # New name>gdrive # Gdrive is an example name -Storage> # Select the number shown for Google Drive +Storage> # Type drive client_id> # Can be left blank client_secret> # Can be left blank -scope> # Select your scope, 1 for example +scope> # Select the scope use used in step 2 root_folder_id> # Can be left blank -service_account_file> /home/foo/myJSONfile.json # This is where the JSON file goes! +service_account_file> /home/foo/myJSONfile.json # Path to the JSON file you downloaded in step 1. y/n> # Auto config, n ``` ##### 4. Verify that it's working + - `rclone -v --drive-impersonate foo@example.com lsf gdrive:backup` - The arguments do: - `-v` - verbose logging @@ -35066,7 +36327,7 @@ the magic, pretending to be user foo. the folder named backup. Note: in case you configured a specific root folder on gdrive and rclone is unable to access the contents of that folder when using `--drive-impersonate`, do this instead: - - in the gdrive web interface, share your root folder with the user/email of the new Service Account you created/selected at step #1 + - in the gdrive web interface, share your root folder with the user/email of the new Service Account you created/selected at step 1 - use rclone without specifying the `--drive-impersonate` option, like this: `rclone -v lsf gdrive:backup` @@ -35097,13 +36358,14 @@ Choose a number from below, or type in your own value 3 / Rclone Test 3 \ "zzzzzzzzzzzzzzzzzzzz" Enter a Shared Drive ID> 1 --------------------- -[remote] -client_id = -client_secret = -token = {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} -team_drive = xxxxxxxxxxxxxxxxxxxx --------------------- +Configuration complete. +Options: +- type: drive +- client_id: +- client_secret: +- token: {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} +- team_drive: xxxxxxxxxxxxxxxxxxxx +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -36691,11 +37953,11 @@ Got code *** are stored in full resolution at original quality. These uploads *** will count towards storage in your Google Account. --------------------- -[remote] -type = google photos -token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"} --------------------- +Configuration complete. +Options: +- type: google photos +- token: {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -37019,7 +38281,7 @@ Max number of files in upload batch. This sets the batch size of files to upload. It has to be less than 50. -By default this is 0 which means rclone which calculate the batch size +By default this is 0 which means rclone will calculate the batch size depending on the setting of batch_mode. - batch_mode: async - default batch_size is 50 @@ -37510,6 +38772,7 @@ The `rclone hashsum` (or `md5sum` or `sha1sum`) command will: ### Other operations +- any time a hash is requested, follow the logic from 1-4 from `hashsum` above - whenever a file is uploaded or downloaded **in full**, capture the stream to calculate all supported hashes on the fly and update database - server-side `move` will update keys of existing cache entries @@ -37581,12 +38844,12 @@ y) Yes n) No (default) y/n> n Remote config --------------------- -[remote] -type = hdfs -namenode = namenode.hadoop:8020 -username = root --------------------- +Configuration complete. +Options: +- type: hdfs +- namenode: namenode.hadoop:8020 +- username: root +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -37828,11 +39091,11 @@ If your browser doesn't open automatically go to the following link: http://127. Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = hidrive -token = {"access_token":"xxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxx","expiry":"xxxxxxxxxxxxxxxxxxxxxxx"} --------------------- +Configuration complete. +Options: +- type: hidrive +- token: {"access_token":"xxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxx","expiry":"xxxxxxxxxxxxxxxxxxxxxxx"} +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -38289,10 +39552,11 @@ Choose a number from below, or type in your own value \ "https://example.com" url> https://beta.rclone.org Remote config --------------------- -[remote] -url = https://beta.rclone.org --------------------- +Configuration complete. +Options: +- type: http +- url: https://beta.rclone.org +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -38877,12 +40141,12 @@ Edit advanced config? y) Yes n) No (default) y/n> n --------------------- -[remote] -type = internetarchive -access_key_id = XXXX -secret_access_key = XXXX --------------------- +Configuration complete. +Options: +- type: internetarchive +- access_key_id: XXXX +- secret_access_key: XXXX +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -39205,18 +40469,18 @@ Press Enter for the default (Archive). 2 > Shared 3 > Sync config_mountpoint> 1 --------------------- -[remote] -type = jottacloud -configVersion = 1 -client_id = jottacli -client_secret = -tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token -token = {........} -username = 2940e57271a93d987d6f8a21 -device = Jotta -mountpoint = Archive --------------------- +Configuration complete. +Options: +- type: jottacloud +- configVersion: 1 +- client_id: jottacli +- client_secret: +- tokenURL: https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token +- token: {........} +- username: 2940e57271a93d987d6f8a21 +- device: Jotta +- mountpoint: Archive +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -40066,13 +41330,13 @@ y) Yes n) No y/n> n Remote config --------------------- -[remote] -type = mailru -user = username@mail.ru -pass = *** ENCRYPTED *** -speedup_enable = true --------------------- +Configuration complete. +Options: +- type: mailru +- user: username@mail.ru +- pass: *** ENCRYPTED *** +- speedup_enable: true +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -40459,12 +41723,12 @@ password: Confirm the password: password: Remote config --------------------- -[remote] -type = mega -user = you@example.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: mega +- user: you@example.com +- pass: *** ENCRYPTED *** +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -40747,10 +42011,10 @@ Storage> memory Remote config --------------------- -[remote] -type = memory --------------------- +Configuration complete. +Options: +- type: memory +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -41114,12 +42378,13 @@ key> base64encodedkey== Endpoint for the service - leave blank normally. endpoint> Remote config --------------------- -[remote] -account = account_name -key = base64encodedkey== -endpoint = --------------------- +Configuration complete. +Options: +- type: azureblob +- account: account_name +- key: base64encodedkey== +- endpoint: +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -41363,6 +42628,13 @@ be explicitly specified using exactly one of the `msi_object_id`, If none of `msi_object_id`, `msi_client_id`, or `msi_mi_res_id` is set, this is is equivalent to using `env_auth`. +#### Anonymous {#anonymous} + +If you want to access resources with public anonymous access then set +`account` only. You can do this without making an rclone config: + + rclone lsf :azureblob,account=ACCOUNT:CONTAINER + ### Standard options @@ -42770,13 +44042,13 @@ Is that okay? y) Yes n) No y/n> y --------------------- -[remote] -type = onedrive -token = {"access_token":"youraccesstoken","token_type":"Bearer","refresh_token":"yourrefreshtoken","expiry":"2018-08-26T22:39:52.486512262+08:00"} -drive_id = b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk -drive_type = business --------------------- +Configuration complete. +Options: +- type: onedrive +- token: {"access_token":"youraccesstoken","token_type":"Bearer","refresh_token":"yourrefreshtoken","expiry":"2018-08-26T22:39:52.486512262+08:00"} +- drive_id: b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk +- drive_type: business +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -43839,11 +45111,12 @@ Enter the password: password: Confirm the password: password: --------------------- -[remote] -username = -password = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: opendrive +- username: +- password: *** ENCRYPTED *** +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -44886,15 +46159,16 @@ Number of connection retry. Leave blank will use the default value "3". connection_retries> Remote config --------------------- -[remote] -env_auth = false -access_key_id = access_key -secret_access_key = secret_key -endpoint = -zone = pek3a -connection_retries = --------------------- +Configuration complete. +Options: +- type: qingstor +- env_auth: false +- access_key_id: access_key +- secret_access_key: secret_key +- endpoint: +- zone: pek3a +- connection_retries: +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -45198,11 +46472,12 @@ api_key> your_api_key Host name of Quatrix account. host> example.quatrix.it --------------------- -[remote] -api_key = your_api_key -host = example.quatrix.it --------------------- +Configuration complete. +Options: +- type: quatrix +- api_key: your_api_key +- host: example.quatrix.it +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -45248,12 +46523,12 @@ e/n/d/r/c/s/q> e Choose a number from below, or type in an existing value 1 > remote remote> remote --------------------- -[remote] -type = quatrix -host = some_host.quatrix.it -api_key = your_api_key --------------------- +Configuration complete. +Options: +- type: quatrix +- host: some_host.quatrix.it +- api_key: your_api_key +Keep this "remote" remote? Edit remote Option api_key. API key for accessing Quatrix account @@ -45263,12 +46538,12 @@ Option host. Host name of Quatrix account Enter a string value. Press Enter for the default (some_host.quatrix.it). --------------------- -[remote] -type = quatrix -host = some_host.quatrix.it -api_key = your_api_key --------------------- +Configuration complete. +Options: +- type: quatrix +- host: some_host.quatrix.it +- api_key: your_api_key +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -46141,6 +47416,48 @@ Properties: - Type: bool - Default: false +#### --swift-fetch-until-empty-page + +When paginating, always fetch unless we received an empty page. + +Consider using this option if rclone listings show fewer objects +than expected, or if repeated syncs copy unchanged objects. + +It is safe to enable this, but rclone may make more API calls than +necessary. + +This is one of a pair of workarounds to handle implementations +of the Swift API that do not implement pagination as expected. See +also "partial_page_fetch_threshold". + +Properties: + +- Config: fetch_until_empty_page +- Env Var: RCLONE_SWIFT_FETCH_UNTIL_EMPTY_PAGE +- Type: bool +- Default: false + +#### --swift-partial-page-fetch-threshold + +When paginating, fetch if the current page is within this percentage of the limit. + +Consider using this option if rclone listings show fewer objects +than expected, or if repeated syncs copy unchanged objects. + +It is safe to enable this, but rclone may make more API calls than +necessary. + +This is one of a pair of workarounds to handle implementations +of the Swift API that do not implement pagination as expected. See +also "fetch_until_empty_page". + +Properties: + +- Config: partial_page_fetch_threshold +- Env Var: RCLONE_SWIFT_PARTIAL_PAGE_FETCH_THRESHOLD +- Type: int +- Default: 0 + #### --swift-chunk-size Above this size files will be chunked. @@ -46363,12 +47680,13 @@ If your browser doesn't open automatically go to the following link: http://127. Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -client_id = -client_secret = -token = {"access_token":"XXX","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"} --------------------- +Configuration complete. +Options: +- type: pcloud +- client_id: +- client_secret: +- token: {"access_token":"XXX","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -46980,6 +48298,182 @@ PikPak supports MD5 hash, but sometimes given empty especially for user-uploaded Deleted files will still be visible with `--pikpak-trashed-only` even after the trash emptied. This goes away after few days. +# Pixeldrain + +This is the backend for Pixeldrain's premium filesystem feature. This is not the +same as pixeldrain's free file sharing product. The filesystem requires either a +Pro subscription or the Prepaid plan. [More information on +subscriptions](https://pixeldrain.com/#pro). + +An overview of the filesystem's features and limitations is available in the +[filesystem guide](https://pixeldrain.com/filesystem) on pixeldrain. + +### Usage with account + +To use the personal filesystem you will need a [pixeldrain +account](https://pixeldrain.com/register) and either the Prepaid plan or one of +the Patreon-based subscriptions. After registering and subscribing, your +personal filesystem will be available at this link: https://pixeldrain.com/d/me. + +Go to the [API keys page](https://pixeldrain.com/user/api_keys) on your account +and generate a new API key for rclone. Then run `rclone config` and use the API +key to create a new backend. + +Example: + +``` +No remotes found, make a new one? +n) New remote +d) Delete remote +c) Copy remote +s) Set configuration password +q) Quit config +n/d/c/s/q> n + +Enter name for new remote. +name> pixeldrainfs + +Option Storage. +Type of storage to configure. +Choose a number from below, or type in your own value. +... +XX / Pixeldrain Filesystem + \ (pixeldrain) +... +Storage> pixeldrain + +Option api_key. +API key for your pixeldrain account. +Found on https://pixeldrain.com/user/api_keys. +Enter a value. Press Enter to leave empty. +api_key> b1bb1e81-9b7b-406b-986a-c9b20be76e15 + +Option directory_id. +Root of the filesystem to use. Set to 'me' to use your personal filesystem. +Set to a shared directory ID to use a shared directory. +Enter a string value. Press Enter for the default (me). +directory_id> + +Edit advanced config? +y) Yes +n) No (default) +y/n> + +Configuration complete. +Options: +- type: pixeldrain +- api_key: b1bb1e81-9b7b-406b-986a-c9b20be76e15 +Keep this "pixeldrainfs" remote? +y) Yes this is OK (default) +e) Edit this remote +d) Delete this remote +y/e/d> + +Current remotes: + +Name Type +==== ==== +pixeldrainfs pixeldrain + +e) Edit existing remote +n) New remote +d) Delete remote +r) Rename remote +c) Copy remote +s) Set configuration password +q) Quit config +e/n/d/r/c/s/q> q +``` + +### Usage without account + +It is possible to gain read-only access to publicly shared directories through +rclone. For this you only need a directory ID. The directory ID can be found in +the URL of a shared directory, the URL will look like this +`https://pixeldrain.com/d/abcd1234` where `abcd1234` is the directory ID. +Directory IDs in your own filesystem can also be listed with the `lsf` command: + +`rclone lsf Pixeldrain: --dirs-only -Fpi` + +This will print directories in your `Pixeldrain` home directory and their public +IDs. + +Enter this directory ID in the rclone config and you will be able to access the +directory. + + +### Standard options + +Here are the Standard options specific to pixeldrain (Pixeldrain Filesystem). + +#### --pixeldrain-api-key + +API key for your pixeldrain account. +Found on https://pixeldrain.com/user/api_keys. + +Properties: + +- Config: api_key +- Env Var: RCLONE_PIXELDRAIN_API_KEY +- Type: string +- Required: false + +#### --pixeldrain-root-folder-id + +Root of the filesystem to use. + +Set to 'me' to use your personal filesystem. Set to a shared directory ID to use a shared directory. + +Properties: + +- Config: root_folder_id +- Env Var: RCLONE_PIXELDRAIN_ROOT_FOLDER_ID +- Type: string +- Default: "me" + +### Advanced options + +Here are the Advanced options specific to pixeldrain (Pixeldrain Filesystem). + +#### --pixeldrain-api-url + +The API endpoint to connect to. In the vast majority of cases it's fine to leave +this at default. It is only intended to be changed for testing purposes. + +Properties: + +- Config: api_url +- Env Var: RCLONE_PIXELDRAIN_API_URL +- Type: string +- Default: "https://pixeldrain.com/api" + +#### --pixeldrain-description + +Description of the remote. + +Properties: + +- Config: description +- Env Var: RCLONE_PIXELDRAIN_DESCRIPTION +- Type: string +- Required: false + +### Metadata + +Pixeldrain supports file modes and creation times. + +Here are the possible system metadata items for the pixeldrain backend. + +| Name | Help | Type | Example | Read Only | +|------|------|------|---------|-----------| +| btime | Time of file birth (creation) | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | N | +| mode | File mode | octal, unix style | 755 | N | +| mtime | Time of last modification | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | N | + +See the [metadata](https://rclone.org/docs/#metadata) docs for more info. + + + # premiumize.me Paths are specified as `remote:path` @@ -47026,11 +48520,11 @@ If your browser doesn't open automatically go to the following link: http://127. Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = premiumizeme -token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2029-08-07T18:44:15.548915378+01:00"} --------------------- +Configuration complete. +Options: +- type: premiumizeme +- token: {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2029-08-07T18:44:15.548915378+01:00"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -47262,12 +48756,12 @@ Option 2fa. Enter a value. Press Enter to leave empty. 2fa> 123456 Remote config --------------------- -[remote] -type = protondrive -user = you@protonmail.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: protondrive +- user: you@protonmail.com +- pass: *** ENCRYPTED *** +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -47843,12 +49337,12 @@ Option 2fa. Enter a value. Press Enter to leave empty. 2fa> 123456 Remote config --------------------- -[remote] -type = protondrive -user = you@protonmail.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: protondrive +- user: you@protonmail.com +- pass: *** ENCRYPTED *** +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -48615,14 +50109,15 @@ y/g/n> n Path to unencrypted PEM-encoded private key file, leave blank to use ssh-agent. key_file> Remote config --------------------- -[remote] -host = example.com -user = sftpuser -port = -pass = -key_file = --------------------- +Configuration complete. +Options: +- type: sftp +- host: example.com +- user: sftpuser +- port: +- pass: +- key_file: +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -48986,7 +50481,15 @@ Properties: Raw PEM-encoded private key. -If specified, will override key_file parameter. +Note that this should be on a single line with line endings replaced with '\n', eg + + key_pem = -----BEGIN RSA PRIVATE KEY-----\nMaMbaIXtE\n0gAMbMbaSsd\nMbaass\n-----END RSA PRIVATE KEY----- + +This will generate the single line correctly: + + awk '{printf "%s\\n", $0}' < ~/.ssh/id_rsa + +If specified, it will override the key_file parameter. Properties: @@ -49443,13 +50946,13 @@ Maximum number of SFTP simultaneous connections, 0 for unlimited. Note that setting this is very likely to cause deadlocks so it should be used with care. -If you are doing a sync or copy then make sure concurrency is one more +If you are doing a sync or copy then make sure connections is one more than the sum of `--transfers` and `--checkers`. If you use `--check-first` then it just needs to be one more than the maximum of `--checkers` and `--transfers`. -So for `concurrency 3` you'd use `--checkers 2 --transfers 2 +So for `connections 3` you'd use `--checkers 2 --transfers 2 --check-first` or `--checkers 1 --transfers 1`. @@ -50061,11 +51564,11 @@ Access Grant. Enter a string value. Press Enter for the default (""). access_grant> your-access-grant-received-by-someone-else Remote config --------------------- -[remote] -type = storj -access_grant = your-access-grant-received-by-someone-else --------------------- +Configuration complete. +Options: +- type: storj +- access_grant: your-access-grant-received-by-someone-else +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -50116,14 +51619,14 @@ Encryption Passphrase. To access existing objects enter passphrase used for uplo Enter a string value. Press Enter for the default (""). passphrase> your-human-readable-encryption-passphrase Remote config --------------------- -[remote] -type = storj -satellite_address = 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777 -api_key = your-api-key-for-your-storj-project -passphrase = your-human-readable-encryption-passphrase -access_grant = the-access-grant-generated-from-the-api-key-and-passphrase --------------------- +Configuration complete. +Options: +- type: storj +- satellite_address: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777 +- api_key: your-api-key-for-your-storj-project +- passphrase: your-human-readable-encryption-passphrase +- access_grant: the-access-grant-generated-from-the-api-key-and-passphrase +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -50414,11 +51917,11 @@ Remote config Username (email address)> nick@craig-wood.com Your Sugarsync password is only required during setup and will not be stored. password: --------------------- -[remote] -type = sugarsync -refresh_token = https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX --------------------- +Configuration complete. +Options: +- type: sugarsync +- refresh_token: https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -50646,11 +52149,6 @@ remote. See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/) -# Tardigrade - -The Tardigrade backend has been renamed to be the [Storj backend](https://rclone.org/storj/). -Old configuration files will continue to work. - # Uloz.to Paths are specified as `remote:path` @@ -51144,11 +52642,11 @@ Cache time of usage and free space (in seconds). This option is only useful when Enter a signed integer. Press Enter for the default ("120"). cache_time> Remote config --------------------- -[remote] -type = union -upstreams = remote1:dir1 remote2:dir2 remote3:dir3 --------------------- +Configuration complete. +Options: +- type: union +- upstreams: remote1:dir1 remote2:dir2 remote3:dir3 +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -51450,15 +52948,15 @@ password: Bearer token instead of user/pass (e.g. a Macaroon) bearer_token> Remote config --------------------- -[remote] -type = webdav -url = https://example.com/remote.php/webdav/ -vendor = nextcloud -user = user -pass = *** ENCRYPTED *** -bearer_token = --------------------- +Configuration complete. +Options: +- type: webdav +- url: https://example.com/remote.php/webdav/ +- vendor: nextcloud +- user: user +- pass: *** ENCRYPTED *** +- bearer_token: +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -51672,6 +53170,17 @@ Properties: - Type: bool - Default: false +#### --webdav-unix-socket + +Path to a unix domain socket to dial to, instead of opening a TCP connection directly + +Properties: + +- Config: unix_socket +- Env Var: RCLONE_WEBDAV_UNIX_SOCKET +- Type: string +- Required: false + #### --webdav-description Description of the remote. @@ -51920,12 +53429,13 @@ If your browser doesn't open automatically go to the following link: http://127. Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -client_id = -client_secret = -token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"OAuth","expiry":"2016-12-29T12:27:11.362788025Z"} --------------------- +Configuration complete. +Options: +- type: yandex +- client_id: +- client_secret: +- token: {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"OAuth","expiry":"2016-12-29T12:27:11.362788025Z"} +Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -52084,6 +53594,17 @@ Properties: - Type: Encoding - Default: Slash,Del,Ctl,InvalidUtf8,Dot +#### --yandex-spoof-ua + +Set the user agent to match an official version of the yandex disk client. May help with upload performance. + +Properties: + +- Config: spoof_ua +- Env Var: RCLONE_YANDEX_SPOOF_UA +- Type: bool +- Default: true + #### --yandex-description Description of the remote. @@ -52176,12 +53697,12 @@ Choose a number from below, or type in your own value 1 / General \ "4u2869d2aa6fca04f4f2f896b6539243b85b1" Enter a Workspace ID> 1 --------------------- -[remote] -type = zoho -token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Zoho-oauthtoken","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2020-10-12T00:54:52.370275223+02:00"} -root_folder_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --------------------- +Configuration complete. +Options: +- type: zoho +- token: {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Zoho-oauthtoken","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2020-10-12T00:54:52.370275223+02:00"} +- root_folder_id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -52500,12 +54021,8 @@ This format requires absolute paths and the use of prefix `\\?\`, e.g. `\\?\D:\some\very\long\path`. For convenience rclone will automatically convert regular paths into the corresponding extended-length paths, so in most cases you do not have to worry about this (read more [below](#long-paths)). - -Note that Windows supports using the same prefix `\\?\` to -specify path to volumes identified by their GUID, e.g. -`\\?\Volume{b75e2c83-0000-0000-0000-602f00000000}\some\path`. -This is *not* supported in rclone, due to an [issue](https://github.com/golang/go/issues/39785) -in go. +Using the same prefix `\\?\` it is also possible to specify path to volumes +identified by their GUID, e.g. `\\?\Volume{b75e2c83-0000-0000-0000-602f00000000}\some\path`. #### Long paths #### @@ -52886,6 +54403,32 @@ Properties: - Type: bool - Default: false +#### --local-no-clone + +Disable reflink cloning for server-side copies. + +Normally, for local-to-local transfers, rclone will "clone" the file when +possible, and fall back to "copying" only when cloning is not supported. + +Cloning creates a shallow copy (or "reflink") which initially shares blocks with +the original file. Unlike a "hardlink", the two files are independent and +neither will affect the other if subsequently modified. + +Cloning is usually preferable to copying, as it is much faster and is +deduplicated by default (i.e. having two identical files does not consume more +storage than having just one.) However, for use cases where data redundancy is +preferable, --local-no-clone can be used to disable cloning and force "deep" copies. + +Currently, cloning is only supported when using APFS on macOS (support for other +platforms may be added in the future.) + +Properties: + +- Config: no_clone +- Env Var: RCLONE_LOCAL_NO_CLONE +- Type: bool +- Default: false + #### --local-no-preallocate Disable preallocation of disk space for transferred files. @@ -53059,6 +54602,127 @@ Options: # Changelog +## v1.68.0 - 2024-09-08 + +[See commits](https://github.com/rclone/rclone/compare/v1.67.0...v1.68.0) + +* New backends + * [Files.com](/filescom) (Sam Harrison) + * [Gofile](https://rclone.org/gofile/) (Nick Craig-Wood) + * [Pixeldrain](https://rclone.org/pixeldrain/) (Fornax) +* Changed backends + * [S3](https://rclone.org/s3/) backend updated to use [AWS SDKv2](https://github.com/aws/aws-sdk-go-v2) as v1 is now unsupported. + * The matrix of providers and auth methods is huge and there could be problems with obscure combinations. + * Please report problems in a [new issue](https://github.com/rclone/rclone/issues/new/choose) on Github. +* New commands + * [config encryption](https://rclone.org/commands/rclone_config_encryption/): set, remove and check to manage config file encryption (Nick Craig-Wood) +* New Features + * build + * Update to go1.23 and make go1.21 the minimum required version (Nick Craig-Wood) + * Update all dependencies (Nick Craig-Wood) + * Disable wasm/js build due to [go bug #64856](https://github.com/golang/go/issues/64856) (Nick Craig-Wood) + * Enable custom linting rules with ruleguard via gocritic (albertony) + * Update logging statements to make `--use-json-log` work always (albertony) + * Adding new code quality tests and fixing the fallout (albertony) + * config + * Internal config re-organised to be more consistent and make it available from the rc (Nick Craig-Wood) + * Avoid remotes with empty names from the environment (albertony) + * Make listing of remotes more consistent (albertony) + * Make getting config values more consistent (albertony) + * Use `--password-command` to set config file password if supplied (Nick Craig-Wood) + * doc fixes (albertony, crystalstall, David Seifert, Eng Zer Jun, Ernie Hershey, Florian Klink, John Oxley, kapitainsky, Mathieu Moreau, Nick Craig-Wood, nipil, Pétr Bozsó, Russ Bubley, Sam Harrison, Thearas, URenko, Will Miles, yuval-cloudinary) + * fs: Allow semicolons as well as spaces in `--bwlimit` timetable parsing (Kyle Reynolds) + * help + * Global flags help command now takes glob filter (albertony) + * Make help command output less distracting (albertony) + * lib/encoder: Add Raw encoding for use where no encoding at all is required, eg `--local-encoding Raw` (URenko) + * listremotes: Added options for filtering, ordering and json output (albertony) + * nfsmount + * Make the `--sudo` flag work for umount as well as mount (Nick Craig-Wood) + * Add `-o tcp` option to NFS mount options to fix mounting under Linux (Nick Craig-Wood) + * operations: copy: generate stable partial suffix (Georg Welzel) + * rc + * Add [options/info](https://rclone.org/rc/#options-info) call to enumerate options (Nick Craig-Wood) + * Add option blocks parameter to [options/get](https://rclone.org/rc/#options-get) and [options/info](/rc/#options-info) (Nick Craig-Wood) + * Add [vfs/queue](https://rclone.org/rc/#vfs-queue) to show the status of the upload queue (Nick Craig-Wood) + * Add [vfs/queue-set-expiry](https://rclone.org/rc/#vfs-queue-set-expiry) to adjust expiry of items in the VFS queue (Nick Craig-Wood) + * Add `--unix-socket` option to `rc` command (Florian Klink) + * Prevent unmount rc command from sending a `STOPPING=1` sd-notify message (AThePeanut4) + * rcserver: Implement [prometheus metrics](https://rclone.org/docs/#metrics) on a dedicated port (Oleg Kunitsyn) + * serve dlna + * Also look at "Subs" subdirectory (Florian Klink) + * Don't swallow `video.{idx,sub}` (Florian Klink) + * Set more correct mime type (Florian Klink) + * serve nfs + * Implement on disk cache for file handles selected with `--nfs-cache-type` (Nick Craig-Wood) + * Add tracing to filesystem calls (Nick Craig-Wood) + * Mask unimplemented error from chmod (Nick Craig-Wood) + * Unify the nfs library logging with rclone's logging better (Nick Craig-Wood) + * Fix incorrect user id and group id exported to NFS (Nick Craig-Wood) + * serve s3 + * Implement `--auth-proxy` (Sawjan Gurung) + * Update to AWS SDKv2 by updating `github.com/rclone/gofakes3` (Nick Craig-Wood) +* Bug Fixes + * bisync: Fix sync time problems with backends that round time (eg Dropbox) (nielash) + * serve dlna: Fix panic: invalid argument to Int63n (Nick Craig-Wood) +* VFS + * Add [--vfs-read-chunk-streams](https://rclone.org/commands/rclone_mount/#vfs-read-chunk-streams-0-1) to parallel read chunks from files (Nick Craig-Wood) + * This can increase mount performance on high bandwidth or large latency links + * Fix cache encoding with special characters (URenko) +* Local + * Fix encoding of root path fix (URenko) + * Add server-side copy (using clone) with xattrs on macOS (nielash) + * `--local-no-clone` flag to disable cloning for server-side copies (nielash) + * Support setting custom `--metadata` during server-side Copy (nielash) +* Azure Blob + * Allow anonymous access for public resources (Nick Craig-Wood) +* B2 + * Include custom upload headers in large file info (Pat Patterson) +* Drive + * Fix copying Google Docs to a backend which only supports SHA1 (Nick Craig-Wood) +* Fichier + * Fix detection of Flood Detected error (Nick Craig-Wood) + * Fix server side move (Nick Craig-Wood) +* HTTP + * Reload client certificates on expiry (Saleh Dindar) + * Support listening on passed FDs (Florian Klink) +* Jottacloud + * Fix setting of metadata on server side move (albertony) +* Onedrive + * Fix nil pointer error when uploading small files (Nick Craig-Wood) +* Pcloud + * Implement `SetModTime` (Georg Welzel) + * Implement `OpenWriterAt` feature to enable multipart uploads (Georg Welzel) +* Pikpak + * Improve data consistency by ensuring async tasks complete (wiserain) + * Implement custom hash to replace wrong sha1 (wiserain) + * Fix error with `copyto` command (wiserain) + * Optimize file move by removing unnecessary `readMetaData()` call (wiserain) + * Non-buffered hash calculation for local source files (wiserain) + * Optimize upload by pre-fetching gcid from API (wiserain) + * Correct file transfer progress for uploads by hash (wiserain) + * Update to using AWS SDK v2 (wiserain) +* S3 + * Update to using AWS SDK v2 (Nick Craig-Wood) + * Add `--s3-sdk-log-mode` to control SDKv2 debugging (Nick Craig-Wood) + * Fix incorrect region for Magalu provider (Filipe Herculano) + * Allow restoring from intelligent-tiering storage class (Pawel Palucha) +* SFTP + * Use `uint32` for mtime to save memory (Tomasz Melcer) + * Ignore useless errors when closing the connection pool (Nick Craig-Wood) + * Support listening on passed FDs (Florian Klink) +* Swift + * Add workarounds for bad listings in Ceph RGW (Paul Collins) + * Add total/free space info in `about` command. (fsantagostinobietti) +* Ulozto + * Fix upload of > 2GB files on 32 bit platforms (Tobias Markus) +* WebDAV + * Add `--webdav-unix-socket-path` to connect to a unix socket (Florian Klink) +* Yandex + * Implement custom user agent to help with upload speeds (Sebastian Bünger) +* Zoho + * Fix inefficiencies uploading with new API to avoid throttling (Nick Craig-Wood) + ## v1.67.0 - 2024-06-14 [See commits](https://github.com/rclone/rclone/compare/v1.66.0...v1.67.0) @@ -58627,9 +60291,11 @@ value, say `export GOGC=20`. This will make the garbage collector work harder, reducing memory size at the expense of CPU usage. The most common cause of rclone using lots of memory is a single -directory with thousands or millions of files in. Rclone has to load -this entirely into memory as rclone objects. Each rclone object takes -0.5k-1k of memory. +directory with millions of files in. Rclone has to load this entirely +into memory as rclone objects. Each rclone object takes 0.5k-1k of +memory. There is +[a workaround for this](https://github.com/rclone/rclone/wiki/Big-syncs-with-millions-of-files) +which involves a bit of scripting. ### Rclone changes fullwidth Unicode punctuation marks in file names @@ -59538,6 +61204,30 @@ put them back in again.` >}} * Michał Dzienisiewicz * Florian Klink * Bill Fraser + * Thearas + * Filipe Herculano + * Russ Bubley + * Paul Collins + * Tomasz Melcer + * itsHenry <2671230065@qq.com> + * Ke Wang + * AThePeanut4 <49614525+AThePeanut4@users.noreply.github.com> + * Tobias Markus + * Ernie Hershey + * Will Miles + * David Seifert <16636962+SoapGentoo@users.noreply.github.com> + * Fornax + * Sam Harrison + * Péter Bozsó <3806723+peterbozso@users.noreply.github.com> + * Georg Welzel + * John Oxley + * Pawel Palucha + * crystalstall + * nipil + * yuval-cloudinary <46710068+yuval-cloudinary@users.noreply.github.com> + * Mathieu Moreau + * fsantagostinobietti <6057026+fsantagostinobietti@users.noreply.github.com> + * Oleg Kunitsyn <114359669+hiddenmarten@users.noreply.github.com> # Contact the rclone project diff --git a/MANUAL.txt b/MANUAL.txt index a401417af..0c2d0d066 100644 --- a/MANUAL.txt +++ b/MANUAL.txt @@ -1,6 +1,6 @@ rclone(1) User Manual Nick Craig-Wood -Jun 14, 2024 +Sep 08, 2024 Rclone syncs your files to cloud storage @@ -108,7 +108,9 @@ S3, that work out of the box.) - Dropbox - Enterprise File Fabric - Fastmail Files +- Files.com - FTP +- Gofile - Google Cloud Storage - Google Drive - Google Photos @@ -147,6 +149,7 @@ S3, that work out of the box.) - pCloud - Petabox - PikPak +- Pixeldrain - premiumize.me - put.io - Proton Drive @@ -617,6 +620,12 @@ resulting executable will be in your GOPATH bin folder go install github.com/rclone/rclone@latest +In some situations, rclone executable size might be too big for +deployment in very restricted environments when all backends with large +SDKs are included. To limit binary size unused backends can be commented +out in backends/all/all.go and unused commands in cmd/all/all.go before +building with go build or make + Ansible installation This can be done with Stefan Weichinger's ansible role. @@ -835,12 +844,15 @@ See the following for detailed instructions for - Digi Storage - Dropbox - Enterprise File Fabric +- Files.com - FTP +- Gofile - Google Cloud Storage - Google Drive - Google Photos - Hasher - to handle checksums for other remotes - HDFS +- Hetzner Storage Box - HiDrive - HTTP - Internet Archive @@ -859,11 +871,13 @@ See the following for detailed instructions for - Oracle Object Storage - Pcloud - PikPak +- Pixeldrain - premiumize.me - put.io - Proton Drive - QingStor - Quatrix by Maytech +- rsync.net - Seafile - SFTP - Sia @@ -937,7 +951,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. - rclone config create - Create a new remote with name, type and @@ -946,6 +960,8 @@ SEE ALSO - rclone config disconnect - Disconnects user from remote - rclone config dump - Dump the config file as JSON. - rclone config edit - Enter an interactive configuration session. +- rclone config encryption - set, remove and check the encryption for + the config file - rclone config file - Show path of configuration file in use. - rclone config password - Update password in an existing remote. - rclone config paths - Show paths used for configuration, cache, temp @@ -1037,13 +1053,16 @@ Options --create-empty-src-dirs Create empty source dirs on destination after copy -h, --help help for copy +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -1076,7 +1095,7 @@ Flags for anything which can Copy a file. Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode @@ -1084,7 +1103,7 @@ Important flags useful for most commands. Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -1111,14 +1130,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -1237,13 +1254,16 @@ Options -s, --separator string Separator for the items in the format (default ";") -t, --timeformat string Specify a custom time format, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -1276,7 +1296,7 @@ Flags for anything which can Copy a file. Sync Options -Flags just used for rclone sync. +Flags used for sync commands --backup-dir string Make backups into hierarchy based in DIR --delete-after When synchronizing, delete files on destination after transferring (default) @@ -1293,7 +1313,7 @@ Flags just used for rclone sync. Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode @@ -1301,7 +1321,7 @@ Important flags useful for most commands. Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -1328,14 +1348,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -1387,13 +1405,16 @@ Options --delete-empty-src-dirs Delete empty source dirs after move -h, --help help for move +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -1426,7 +1447,7 @@ Flags for anything which can Copy a file. Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode @@ -1434,7 +1455,7 @@ Important flags useful for most commands. Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -1461,14 +1482,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -1512,9 +1531,12 @@ Options -h, --help help for delete --rmdirs rmdirs removes empty directories but leaves root intact +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode @@ -1522,7 +1544,7 @@ Important flags useful for most commands. Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -1549,14 +1571,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -1580,17 +1600,18 @@ Options -h, --help help for purge +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -1604,17 +1625,18 @@ Options -h, --help help for mkdir +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -1636,17 +1658,18 @@ Options -h, --help help for rmdir +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -1718,15 +1741,18 @@ Options --missing-on-src string Report all files missing from the source to this file --one-way Check one way only, source files must exist on remote +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Check Options -Flags used for rclone check. +Flags used for check commands --max-backlog int Maximum number of objects in sync or check backlog (default 10000) Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -1753,14 +1779,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -1810,9 +1834,12 @@ Options -h, --help help for ls +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -1839,14 +1866,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -1907,9 +1932,12 @@ Options -h, --help help for lsd -R, --recursive Recurse into the listing +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -1936,14 +1964,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -1994,9 +2020,12 @@ Options -h, --help help for lsl +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -2023,14 +2052,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2067,9 +2094,12 @@ Options -h, --help help for md5sum --output-file string Output hashsums to a file rather than the terminal +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -2096,14 +2126,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2143,9 +2171,12 @@ Options -h, --help help for sha1sum --output-file string Output hashsums to a file rather than the terminal +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -2172,14 +2203,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2211,9 +2240,12 @@ Options -h, --help help for size --json Format output as JSON +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -2240,14 +2272,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2302,7 +2332,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2321,17 +2351,18 @@ Options -h, --help help for cleanup +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2465,17 +2496,18 @@ Options --dedupe-mode string Dedupe mode interactive|skip|first|newest|oldest|largest|smallest|rename (default "interactive") -h, --help help for dedupe +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2485,8 +2517,8 @@ Get quota information from the remote. Synopsis -rclone about prints quota information about a remote to standard output. -The output is typically used, free, quota and trash contents. +Prints quota information about a remote to standard output. The output +is typically used, free, quota and trash contents. E.g. Typical output from rclone about remote: is: @@ -2541,7 +2573,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2571,7 +2603,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2615,17 +2647,18 @@ Options --json Always output in JSON format -o, --option stringArray Option in the form name=value or name +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2680,13 +2713,16 @@ Options --slow-hash-sync-only Ignore slow checksums for listings and deltas, but still consider them during sync calls. --workdir string Use custom working dir - useful for testing. (default: {WORKDIR}) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -2719,7 +2755,7 @@ Flags for anything which can Copy a file. Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode @@ -2727,7 +2763,7 @@ Important flags useful for most commands. Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -2752,9 +2788,7 @@ Flags for filtering directory listings. --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2764,7 +2798,7 @@ Concatenates any files and sends them to stdout. Synopsis -rclone cat sends any files to standard output. +Sends any files to standard output. You can use it like this to output a single file @@ -2807,9 +2841,12 @@ Options --separator string Separator to use between objects when printing multiple files --tail int Only print the last N characters +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -2836,14 +2873,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2910,9 +2945,12 @@ Options --missing-on-src string Report all files missing from the source to this file --one-way Check one way only, source files must exist on remote +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -2939,14 +2977,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -2965,7 +3001,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. - rclone completion bash - Output bash completion script for rclone. @@ -2984,7 +3020,7 @@ Generates a bash shell autocompletion script for rclone. By default, when run without any arguments, - rclone genautocomplete bash + rclone completion bash the generated script will be written to @@ -3016,7 +3052,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone completion - Output completion script for a given shell. @@ -3031,7 +3067,7 @@ Generates a fish autocompletion script for rclone. This writes to /etc/fish/completions/rclone.fish by default so will probably need to be run with sudo or as root, e.g. - sudo rclone genautocomplete fish + sudo rclone completion fish Logout and login again to use the autocompletion scripts, or source them directly @@ -3050,7 +3086,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone completion - Output completion script for a given shell. @@ -3080,7 +3116,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone completion - Output completion script for a given shell. @@ -3095,7 +3131,7 @@ Generates a zsh autocompletion script for rclone. This writes to /usr/share/zsh/vendor-completions/_rclone by default so will probably need to be run with sudo or as root, e.g. - sudo rclone genautocomplete zsh + sudo rclone completion zsh Logout and login again to use the autocompletion scripts, or source them directly @@ -3114,7 +3150,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone completion - Output completion script for a given shell. @@ -3239,7 +3275,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3255,7 +3291,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3279,7 +3315,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3295,7 +3331,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3317,10 +3353,131 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. +rclone config encryption + +set, remove and check the encryption for the config file + +Synopsis + +This command sets, clears and checks the encryption for the config file +using the subcommands below. + +Options + + -h, --help help for encryption + +See the global flags page for global options not listed here. + +See Also + +- rclone config - Enter an interactive configuration session. +- rclone config encryption check - Check that the config file is + encrypted +- rclone config encryption remove - Remove the config file encryption + password +- rclone config encryption set - Set or change the config file + encryption password + +rclone config encryption check + +Check that the config file is encrypted + +Synopsis + +This checks the config file is encrypted and that you can decrypt it. + +It will attempt to decrypt the config using the password you supply. + +If decryption fails it will return a non-zero exit code if using +--password-command, otherwise it will prompt again for the password. + +If the config file is not encrypted it will return a non zero exit code. + + rclone config encryption check [flags] + +Options + + -h, --help help for check + +See the global flags page for global options not listed here. + +See Also + +- rclone config encryption - set, remove and check the encryption for + the config file + +rclone config encryption remove + +Remove the config file encryption password + +Synopsis + +Remove the config file encryption password + +This removes the config file encryption, returning it to un-encrypted. + +If --password-command is in use, this will be called to supply the old +config password. + +If the config was not encrypted then no error will be returned and this +command will do nothing. + + rclone config encryption remove [flags] + +Options + + -h, --help help for remove + +See the global flags page for global options not listed here. + +See Also + +- rclone config encryption - set, remove and check the encryption for + the config file + +rclone config encryption set + +Set or change the config file encryption password + +Synopsis + +This command sets or changes the config file encryption password. + +If there was no config password set then it sets a new one, otherwise it +changes the existing config password. + +Note that if you are changing an encryption password using +--password-command then this will be called once to decrypt the config +using the old password and then again to read the new password to +re-encrypt the config. + +When --password-command is called to change the password then the +environment variable RCLONE_PASSWORD_CHANGE=1 will be set. So if +changing passwords programatically you can use the environment variable +to distinguish which password you must supply. + +Alternatively you can remove the password first (with +rclone config encryption remove), then set it again with this command +which may be easier if you don't mind the unecrypted config file being +on the disk briefly. + + rclone config encryption set [flags] + +Options + + -h, --help help for set + +See the global flags page for global options not listed here. + +See Also + +- rclone config encryption - set, remove and check the encryption for + the config file + rclone config file Show path of configuration file in use. @@ -3333,7 +3490,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3363,7 +3520,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3379,7 +3536,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3395,7 +3552,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3419,7 +3576,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3449,7 +3606,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3465,7 +3622,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3481,7 +3638,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3606,7 +3763,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3628,7 +3785,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone config - Enter an interactive configuration session. @@ -3672,13 +3829,16 @@ Options -h, --help help for copyto +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -3711,7 +3871,7 @@ Flags for anything which can Copy a file. Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode @@ -3719,7 +3879,7 @@ Important flags useful for most commands. Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -3746,14 +3906,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -3804,17 +3962,18 @@ Options -p, --print-filename Print the resulting name from --auto-filename --stdout Write the output to stdout rather than a file +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -3824,9 +3983,9 @@ Cryptcheck checks the integrity of an encrypted remote. Synopsis -rclone cryptcheck checks a remote against a crypted remote. This is the -equivalent of running rclone check, but able to check the checksums of -the encrypted remote. +Checks a remote against a crypted remote. This is the equivalent of +running rclone check, but able to check the checksums of the encrypted +remote. For it to work the underlying remote of the cryptedremote must support some kind of checksum. @@ -3889,15 +4048,18 @@ Options --missing-on-src string Report all files missing from the source to this file --one-way Check one way only, source files must exist on remote +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Check Options -Flags used for rclone check. +Flags used for check commands --max-backlog int Maximum number of objects in sync or check backlog (default 10000) Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -3924,14 +4086,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -3941,8 +4101,8 @@ Cryptdecode returns unencrypted file names. Synopsis -rclone cryptdecode returns unencrypted file names when provided with a -list of encrypted file names. List limit is 10 items. +Returns unencrypted file names when provided with a list of encrypted +file names. List limit is 10 items. If you supply the --reverse flag, it will return encrypted file names. @@ -3965,7 +4125,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -3985,147 +4145,21 @@ Options -h, --help help for deletefile +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. -rclone genautocomplete - -Output completion script for a given shell. - -Synopsis - -Generates a shell completion script for rclone. Run with --help to list -the supported shells. - -Options - - -h, --help help for genautocomplete - -See the global flags page for global options not listed here. - -SEE ALSO - -- rclone - Show help for rclone commands, flags and backends. -- rclone genautocomplete bash - Output bash completion script for - rclone. -- rclone genautocomplete fish - Output fish completion script for - rclone. -- rclone genautocomplete zsh - Output zsh completion script for - rclone. - -rclone genautocomplete bash - -Output bash completion script for rclone. - -Synopsis - -Generates a bash shell autocompletion script for rclone. - -This writes to /etc/bash_completion.d/rclone by default so will probably -need to be run with sudo or as root, e.g. - - sudo rclone genautocomplete bash - -Logout and login again to use the autocompletion scripts, or source them -directly - - . /etc/bash_completion - -If you supply a command line argument the script will be written there. - -If output_file is "-", then the output will be written to stdout. - - rclone genautocomplete bash [output_file] [flags] - -Options - - -h, --help help for bash - -See the global flags page for global options not listed here. - -SEE ALSO - -- rclone genautocomplete - Output completion script for a given shell. - -rclone genautocomplete fish - -Output fish completion script for rclone. - -Synopsis - -Generates a fish autocompletion script for rclone. - -This writes to /etc/fish/completions/rclone.fish by default so will -probably need to be run with sudo or as root, e.g. - - sudo rclone genautocomplete fish - -Logout and login again to use the autocompletion scripts, or source them -directly - - . /etc/fish/completions/rclone.fish - -If you supply a command line argument the script will be written there. - -If output_file is "-", then the output will be written to stdout. - - rclone genautocomplete fish [output_file] [flags] - -Options - - -h, --help help for fish - -See the global flags page for global options not listed here. - -SEE ALSO - -- rclone genautocomplete - Output completion script for a given shell. - -rclone genautocomplete zsh - -Output zsh completion script for rclone. - -Synopsis - -Generates a zsh autocompletion script for rclone. - -This writes to /usr/share/zsh/vendor-completions/_rclone by default so -will probably need to be run with sudo or as root, e.g. - - sudo rclone genautocomplete zsh - -Logout and login again to use the autocompletion scripts, or source them -directly - - autoload -U compinit && compinit - -If you supply a command line argument the script will be written there. - -If output_file is "-", then the output will be written to stdout. - - rclone genautocomplete zsh [output_file] [flags] - -Options - - -h, --help help for zsh - -See the global flags page for global options not listed here. - -SEE ALSO - -- rclone genautocomplete - Output completion script for a given shell. - rclone gendocs Output markdown docs for rclone to the directory supplied. @@ -4144,7 +4178,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -4217,7 +4251,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -4271,9 +4305,12 @@ Options -h, --help help for hashsum --output-file string Output hashsums to a file rather than the terminal +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -4300,14 +4337,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -4317,8 +4352,7 @@ Generate public link to file/folder. Synopsis -rclone link will create, retrieve or remove a public link to the given -file or folder. +Create, retrieve or remove a public link to the given file or folder. rclone link remote:path/to/file rclone link remote:path/to/folder/ @@ -4349,7 +4383,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -4360,21 +4394,35 @@ variables. Synopsis -rclone listremotes lists all the available remotes from the config file. +Lists all the available remotes from the config file, or the remotes +matching an optional filter. -When used with the --long flag it lists the types and the descriptions -too. +Prints the result in human-readable format by default, and as a simple +list of remote names, or if used with flag --long a tabular format +including the remote names, types and descriptions. Using flag --json +produces machine-readable output instead, which always includes all +attributes - including the source (file or environment). - rclone listremotes [flags] +Result can be filtered by a filter argument which applies to all +attributes, and/or filter flags specific for each attribute. The values +must be specified according to regular rclone filtering pattern syntax. + + rclone listremotes [] [flags] Options - -h, --help help for listremotes - --long Show the type and the description as well as names + --description string Filter remotes by description + -h, --help help for listremotes + --json Format output as JSON + --long Show type and description in addition to name + --name string Filter remotes by name + --order-by string Instructions on how to order the result, e.g. 'type,name=descending' + --source string Filter remotes by source, e.g. 'file' or 'environment' + --type string Filter remotes by type See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -4530,9 +4578,12 @@ Options -s, --separator string Separator for the items in the format (default ";") -t, --time-format string Specify a custom time format, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -4559,14 +4610,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -4578,7 +4627,7 @@ Synopsis List directories and objects in the path in JSON format. -The output is an array of Items, where each Item looks like this +The output is an array of Items, where each Item looks like this: { "Hashes" : { @@ -4600,34 +4649,46 @@ The output is an array of Items, where each Item looks like this "Tier" : "hot", } -If --hash is not specified, the Hashes property will be omitted. The -types of hash can be specified with the --hash-type parameter (which may -be repeated). If --hash-type is set then it implies --hash. +The exact set of properties included depends on the backend: -If --no-modtime is specified then ModTime will be blank. This can speed -things up on remotes where reading the ModTime takes an extra request -(e.g. s3, swift). +- The property IsBucket will only be included for bucket-based + remotes, and only for directories that are buckets. It will always + be omitted when value is not true. +- Properties Encrypted and EncryptedPath will only be included for + encrypted remotes, and (as mentioned below) only if the --encrypted + option is set. -If --no-mimetype is specified then MimeType will be blank. This can -speed things up on remotes where reading the MimeType takes an extra -request (e.g. s3, swift). +Different options may also affect which properties are included: -If --encrypted is not specified the Encrypted will be omitted. +- If --hash is not specified, the Hashes property will be omitted. The + types of hash can be specified with the --hash-type parameter (which + may be repeated). If --hash-type is set then it implies --hash. +- If --no-modtime is specified then ModTime will be blank. This can + speed things up on remotes where reading the ModTime takes an extra + request (e.g. s3, swift). +- If --no-mimetype is specified then MimeType will be blank. This can + speed things up on remotes where reading the MimeType takes an extra + request (e.g. s3, swift). +- If --encrypted is not specified the Encrypted and EncryptedPath + properties will be omitted - even for encrypted remotes. +- If --metadata is set then an additional Metadata property will be + returned. This will have metadata in rclone standard format as a + JSON object. -If --dirs-only is not specified files in addition to directories are -returned +The default is to list directories and files/objects, but this can be +changed with the following options: -If --files-only is not specified directories in addition to the files -will be returned. +- If --dirs-only is specified then directories will be returned only, + no files/objects. +- If --files-only is specified then files will be returned only, no + directories. -If --metadata is set then an additional Metadata key will be returned. -This will have metadata in rclone standard format as a JSON object. - -if --stat is set then a single JSON blob will be returned about the item -pointed to. This will return an error if the item isn't found. However -on bucket based backends (like s3, gcs, b2, azureblob etc) if the item -isn't found it will return an empty directory as it isn't possible to -tell empty directories from missing directories there. +If --stat is set then the the output is not an array of items, but +instead a single JSON blob will be returned about the item pointed to. +This will return an error if the item isn't found, however on bucket +based backends (like s3, gcs, b2, azureblob etc) if the item isn't found +it will return an empty directory, as it isn't possible to tell empty +directories from missing directories there. The Path field will only show folders below the remote path being listed. If "remote:path" contains the file "subfolder/file.txt", the @@ -4635,9 +4696,6 @@ Path for "file.txt" will be "subfolder/file.txt", not "remote:path/subfolder/file.txt". When used without --recursive the Path will always be the same as Name. -If the directory is a bucket in a bucket-based backend, then "IsBucket" -will be set to true. This key won't be present unless it is "true". - The time is in RFC3339 format with up to nanosecond precision. The number of decimal digits in the seconds will depend on the precision that the remote can hold the times, so if times are accurate to the @@ -4647,7 +4705,8 @@ accurate to the nearest second (Dropbox, Box, WebDav, etc.) no digits will be shown ("2017-05-31T16:15:57+01:00"). The whole output can be processed as a JSON blob, or alternatively it -can be processed line by line as each item is written one to a line. +can be processed line by line as each item is written on individual +lines (except with --stat). Any of the filtering options can be applied to this command. @@ -4689,9 +4748,12 @@ Options -R, --recursive Recurse into the listing --stat Just return the info for the pointed to file +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -4718,14 +4780,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -5425,6 +5485,12 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the +--vfs-read-chunk-streams parameter. + +--vfs-read-chunk-streams == 0 Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is @@ -5441,6 +5507,30 @@ the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. +The chunks will not be buffered in memory. + +--vfs-read-chunk-streams > 0 + +Rclone reads --vfs-read-chunk-streams chunks of size +--vfs-read-chunk-size concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +--vfs-read-chunk-size and --vfs-read-chunk-streams as these will depend +on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be --vfs-read-chunk-streams 16 and --vfs-read-chunk-size 4M. +In testing with AWS S3 the performance scaled roughly as the +--vfs-read-chunk-streams setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more --vfs-read-chunk-streams in order to get +the throughput. + VFS Performance These flags may be used to enable/disable features of the VFS for @@ -5568,9 +5658,9 @@ Options --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for mount @@ -5586,7 +5676,7 @@ Options --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -5599,6 +5689,7 @@ Options --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -5607,9 +5698,12 @@ Options --volname string Set the volume name (supported on Windows and OSX only) --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -5634,9 +5728,7 @@ Flags for filtering directory listings. --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -5683,13 +5775,16 @@ Options -h, --help help for moveto +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -5722,7 +5817,7 @@ Flags for anything which can Copy a file. Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode @@ -5730,7 +5825,7 @@ Important flags useful for most commands. Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -5757,14 +5852,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -5823,7 +5916,8 @@ flags have the following meaning: ! means an error occurred while reading this directory This an homage to the ncdu tool but for rclone remotes. It is missing -lots of features at the moment but is useful as it stands. +lots of features at the moment but is useful as it stands. Unlike ncdu +it does not show excluded files. Note that it might take some time to delete big files/directories. The UI won't respond in the meantime since the deletion is done @@ -5838,9 +5932,12 @@ Options -h, --help help for ncdu +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -5867,14 +5964,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -6574,6 +6669,12 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the +--vfs-read-chunk-streams parameter. + +--vfs-read-chunk-streams == 0 Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is @@ -6590,6 +6691,30 @@ the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. +The chunks will not be buffered in memory. + +--vfs-read-chunk-streams > 0 + +Rclone reads --vfs-read-chunk-streams chunks of size +--vfs-read-chunk-size concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +--vfs-read-chunk-size and --vfs-read-chunk-streams as these will depend +on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be --vfs-read-chunk-streams 16 and --vfs-read-chunk-size 4M. +In testing with AWS S3 the performance scaled roughly as the +--vfs-read-chunk-streams setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more --vfs-read-chunk-streams in order to get +the throughput. + VFS Performance These flags may be used to enable/disable features of the VFS for @@ -6718,16 +6843,18 @@ Options --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for nfsmount --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki) --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset) --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only) + --nfs-cache-dir string The directory the NFS handle cache will use if set --nfs-cache-handle-limit int max file handles cached simultaneously (min 5) (default 1000000) + --nfs-cache-type memory|disk|symlink Type of NFS handle cache to use (default memory) --no-checksum Don't compare checksums on up/download --no-modtime Don't read/write the modification time (can speed things up) --no-seek Don't allow seeking in files @@ -6736,9 +6863,9 @@ Options -o, --option stringArray Option for libfuse/WinFsp (repeat if required) --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access - --sudo Use sudo to run the mount command as root. + --sudo Use sudo to run the mount/umount commands as root. --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -6751,6 +6878,7 @@ Options --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -6759,9 +6887,12 @@ Options --volname string Set the volume name (supported on Windows and OSX only) --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -6786,9 +6917,7 @@ Flags for filtering directory listings. --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -6828,7 +6957,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -6848,6 +6977,14 @@ A username and password can be passed in with --user and --pass. Note that --rc-addr, --rc-user, --rc-pass will be read also for --url, --user, --pass. +The --unix-socket flag can be used to connect over a unix socket like +this + + # start server on /tmp/my.socket + rclone rcd --rc-addr unix:///tmp/my.socket + # Connect to it + rclone rc --unix-socket /tmp/my.socket core/stats + Arguments should be passed in as parameter=value. The result will be returned as a JSON object by default. @@ -6890,19 +7027,20 @@ Use rclone rc to see a list of all possible commands. Options - -a, --arg stringArray Argument placed in the "arg" array - -h, --help help for rc - --json string Input JSON - use instead of key=value args - --loopback If set connect to this rclone instance not via HTTP - --no-output If set, don't output the JSON result - -o, --opt stringArray Option in the form name=value or name placed in the "opt" array - --pass string Password to use to connect to rclone remote control - --url string URL to connect to rclone remote control (default "http://localhost:5572/") - --user string Username to use to rclone remote control + -a, --arg stringArray Argument placed in the "arg" array + -h, --help help for rc + --json string Input JSON - use instead of key=value args + --loopback If set connect to this rclone instance not via HTTP + --no-output If set, don't output the JSON result + -o, --opt stringArray Option in the form name=value or name placed in the "opt" array + --pass string Password to use to connect to rclone remote control + --unix-socket string Path to a unix domain socket to dial to, instead of opening a TCP connection directly + --url string URL to connect to rclone remote control (default "http://localhost:5572/") + --user string Username to use to rclone remote control See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -6912,8 +7050,7 @@ Copies standard input to file on remote. Synopsis -rclone rcat reads from standard input (stdin) and copies it to a single -remote file. +Reads from standard input (stdin) and copies it to a single remote file. echo "hello world" | rclone rcat remote:path/to/file ffmpeg - | rclone rcat remote:path/to/file @@ -6950,17 +7087,18 @@ Options -h, --help help for rcat --size int File size hint to preallocate (default -1) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -6996,6 +7134,8 @@ the authentication - this is expected to be done with file system permissions. --rc-addr may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to +accomplish the same. --rc-server-read-timeout and --rc-server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for @@ -7027,7 +7167,23 @@ client certificate authority certificate. --rc-min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). -Template +Socket activation + +Instead of the listening addresses specified above, rclone will listen +to all FDs passed by the service manager, if any (and ignore any +arguments passed by --rc-addr`). + +This allows rclone to be a socket-activated service. It can be +configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the +systemd-socket-activatecommand + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 +over TCP. ### Template --rc-template allows a user to specify a custom markup template for HTTP and WebDAV serve functions. The server exports the following markup to @@ -7134,17 +7290,20 @@ Options -h, --help help for rcd +Options shared with other commands are described next. See the global +flags page for global options not listed here. + RC Options -Flags to control the Remote Control API. +Flags to control the Remote Control API --rc Enable the remote control server - --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572]) + --rc-addr stringArray IPaddress:Port or :Port to bind server to (default ["localhost:5572"]) --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from --rc-baseurl string Prefix for URLs - leave blank for root --rc-cert string TLS PEM key (concatenation of certificate and CA certificate) --rc-client-ca string Client certificate authority to verify clients with - --rc-enable-metrics Enable prometheus metrics on /metrics + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server --rc-files string Path to local files to serve on the HTTP server --rc-htpasswd string A htpasswd file - if not provided no authentication is done --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s) @@ -7168,9 +7327,7 @@ Flags to control the Remote Control API. --rc-web-gui-no-open-browser Don't open the browser automatically --rc-web-gui-update Check and update to latest version of web gui -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -7204,17 +7361,18 @@ Options -h, --help help for rmdirs --leave-root Do not remove root directory if empty +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -7287,7 +7445,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -7312,7 +7470,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. - rclone serve dlna - Serve remote:path over DLNA @@ -7343,6 +7501,10 @@ based on media formats or file extensions. Additionally, there is no media transcoding support. This means that some players might show files that they are not able to play back correctly. +Rclone will add external subtitle files (.srt) to videos if they have +the same filename as the video file itself (except the extension), +either in the same directory as the video, or in a "Subs" subdirectory. + Server options Use --addr to specify which IP address and port the server should listen @@ -7580,6 +7742,12 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the +--vfs-read-chunk-streams parameter. + +--vfs-read-chunk-streams == 0 Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is @@ -7596,6 +7764,30 @@ the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. +The chunks will not be buffered in memory. + +--vfs-read-chunk-streams > 0 + +Rclone reads --vfs-read-chunk-streams chunks of size +--vfs-read-chunk-size concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +--vfs-read-chunk-size and --vfs-read-chunk-streams as these will depend +on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be --vfs-read-chunk-streams 16 and --vfs-read-chunk-size 4M. +In testing with AWS S3 the performance scaled roughly as the +--vfs-read-chunk-streams setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more --vfs-read-chunk-streams in order to get +the throughput. + VFS Performance These flags may be used to enable/disable features of the VFS for @@ -7714,8 +7906,8 @@ Options --addr string The ip:port or :port to bind the DLNA http server to (default ":7879") --announce-interval Duration The interval between SSDP announcements (default 12m0s) --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for dlna --interface stringArray The interface to use for SSDP (repeat as necessary) @@ -7727,7 +7919,7 @@ Options --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -7740,15 +7932,19 @@ Options --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -7773,9 +7969,7 @@ Flags for filtering directory listings. --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone serve - Serve a remote over a protocol. @@ -8049,6 +8243,12 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the +--vfs-read-chunk-streams parameter. + +--vfs-read-chunk-streams == 0 Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is @@ -8065,6 +8265,30 @@ the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. +The chunks will not be buffered in memory. + +--vfs-read-chunk-streams > 0 + +Rclone reads --vfs-read-chunk-streams chunks of size +--vfs-read-chunk-size concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +--vfs-read-chunk-size and --vfs-read-chunk-streams as these will depend +on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be --vfs-read-chunk-streams 16 and --vfs-read-chunk-size 4M. +In testing with AWS S3 the performance scaled roughly as the +--vfs-read-chunk-streams setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more --vfs-read-chunk-streams in order to get +the throughput. + VFS Performance These flags may be used to enable/disable features of the VFS for @@ -8193,9 +8417,9 @@ Options --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --forget-state Skip restoring previous state --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) @@ -8215,7 +8439,7 @@ Options --socket-addr string Address or absolute path (default: /run/docker/plugins/rclone.sock) --socket-gid int GID for unix socket (default: current process GID) (default 1000) --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -8228,6 +8452,7 @@ Options --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -8236,9 +8461,12 @@ Options --volname string Set the volume name (supported on Windows and OSX only) --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -8263,9 +8491,7 @@ Flags for filtering directory listings. --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone serve - Serve a remote over a protocol. @@ -8522,6 +8748,12 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the +--vfs-read-chunk-streams parameter. + +--vfs-read-chunk-streams == 0 Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is @@ -8538,6 +8770,30 @@ the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. +The chunks will not be buffered in memory. + +--vfs-read-chunk-streams > 0 + +Rclone reads --vfs-read-chunk-streams chunks of size +--vfs-read-chunk-size concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +--vfs-read-chunk-size and --vfs-read-chunk-streams as these will depend +on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be --vfs-read-chunk-streams 16 and --vfs-read-chunk-size 4M. +In testing with AWS S3 the performance scaled roughly as the +--vfs-read-chunk-streams setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more --vfs-read-chunk-streams in order to get +the throughput. + VFS Performance These flags may be used to enable/disable features of the VFS for @@ -8728,8 +8984,8 @@ Options --auth-proxy string A program to use to create the backend from the auth --cert string TLS PEM key (concatenation of certificate and CA certificate) --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for ftp --key string TLS PEM Private key @@ -8742,7 +8998,7 @@ Options --public-ip string Public IP address to advertise for passive connections --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication (default "anonymous") --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -8756,15 +9012,19 @@ Options --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -8789,9 +9049,7 @@ Flags for filtering directory listings. --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone serve - Serve a remote over a protocol. @@ -8827,7 +9085,9 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. ---addr may be repeated to listen on multiple IPs/ports/sockets. +--addr may be repeated to listen on multiple IPs/ports/sockets. Socket +activation, described further below, can also be used to accomplish the +same. --server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a @@ -8859,7 +9119,23 @@ authority certificate. --min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). -Template +Socket activation + +Instead of the listening addresses specified above, rclone will listen +to all FDs passed by the service manager, if any (and ignore any +arguments passed by --addr`). + +This allows rclone to be a socket-activated service. It can be +configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the +systemd-socket-activatecommand + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 +over TCP. ### Template --template allows a user to specify a custom markup template for HTTP and WebDAV serve functions. The server exports the following markup to @@ -9186,6 +9462,12 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the +--vfs-read-chunk-streams parameter. + +--vfs-read-chunk-streams == 0 Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is @@ -9202,6 +9484,30 @@ the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. +The chunks will not be buffered in memory. + +--vfs-read-chunk-streams > 0 + +Rclone reads --vfs-read-chunk-streams chunks of size +--vfs-read-chunk-size concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +--vfs-read-chunk-size and --vfs-read-chunk-streams as these will depend +on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be --vfs-read-chunk-streams 16 and --vfs-read-chunk-size 4M. +In testing with AWS S3 the performance scaled roughly as the +--vfs-read-chunk-streams setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more --vfs-read-chunk-streams in order to get +the throughput. + VFS Performance These flags may be used to enable/disable features of the VFS for @@ -9388,15 +9694,15 @@ that rclone supports. Options - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for http --htpasswd string A htpasswd file - if not provided no authentication is done @@ -9415,7 +9721,7 @@ Options --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --template string User-specified template --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -9429,15 +9735,19 @@ Options --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -9462,9 +9772,7 @@ Flags for filtering directory listings. --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone serve - Serve a remote over a protocol. @@ -9476,41 +9784,66 @@ Synopsis Create an NFS server that serves the given remote over the network. -The primary purpose for this command is to enable mount command on +This implements an NFSv3 server to serve any rclone remote via NFS. + +The primary purpose for this command is to enable the mount command on recent macOS versions where installing FUSE is very cumbersome. -Since this is running on NFSv3, no authentication method is available. -Any client will be able to access the data. To limit access, you can use -serve NFS on loopback address and rely on secure tunnels (such as SSH). -For this reason, by default, a random TCP port is chosen and loopback -interface is used for the listening address; meaning that it is only -available to the local machine. If you want other machines to access the -NFS mount over local network, you need to specify the listening address -and port using --addr flag. +This server does not implement any authentication so any client will be +able to access the data. To limit access, you can use serve nfs on the +loopback address or rely on secure tunnels (such as SSH) or use +firewalling. -Modifying files through NFS protocol requires VFS caching. Usually you -will need to specify --vfs-cache-mode in order to be able to write to -the mountpoint (full is recommended). If you don't specify VFS cache -mode, the mount will be read-only. Note also that ---nfs-cache-handle-limit controls the maximum number of cached file +For this reason, by default, a random TCP port is chosen and the +loopback interface is used for the listening address by default; meaning +that it is only available to the local machine. If you want other +machines to access the NFS mount over local network, you need to specify +the listening address and port using the --addr flag. + +Modifying files through the NFS protocol requires VFS caching. Usually +you will need to specify --vfs-cache-mode in order to be able to write +to the mountpoint (full is recommended). If you don't specify VFS cache +mode, the mount will be read-only. + +--nfs-cache-type controls the type of the NFS handle cache. By default +this is memory where new handles will be randomly allocated when needed. +These are stored in memory. If the server is restarted the handle cache +will be lost and connected NFS clients will get stale handle errors. + +--nfs-cache-type disk uses an on disk NFS handle cache. Rclone hashes +the path of the object and stores it in a file named after the hash. +These hashes are stored on disk the directory controlled by --cache-dir +or the exact directory may be specified with --nfs-cache-dir. Using this +means that the NFS server can be restarted at will without affecting the +connected clients. + +--nfs-cache-type symlink is similar to --nfs-cache-type disk in that it +uses an on disk cache, but the cache entries are held as symlinks. +Rclone will use the handle of the underlying file as the NFS handle +which improves performance. This sort of cache can't be backed up and +restored as the underlying handles will change. This is Linux only. + +--nfs-cache-handle-limit controls the maximum number of cached NFS handles stored by the caching handler. This should not be set too low or you may experience errors when trying to access files. The default is 1000000, but consider lowering this limit if the server's system -resource usage causes problems. +resource usage causes problems. This is only used by the memory type +cache. To serve NFS over the network use following command: rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full -We specify a specific port that we can use in the mount command: +This specifies a port that can be used in the mount command. To mount +the server under Linux/macOS, use the following command: -To mount the server under Linux/macOS, use the following command: + mount -t nfs -o port=$PORT,mountport=$PORT,tcp $HOSTNAME:/ path/to/mountpoint - mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint +Where $PORT is the same port number used in the serve nfs command and +$HOSTNAME is the network address of the machine that serve nfs was run +on. -Where $PORT is the same port number we used in the serve nfs command. - -This feature is only available on Unix platforms. +This command is only available on Unix platforms. VFS - Virtual File System @@ -9738,6 +10071,12 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the +--vfs-read-chunk-streams parameter. + +--vfs-read-chunk-streams == 0 Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is @@ -9754,6 +10093,30 @@ the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. +The chunks will not be buffered in memory. + +--vfs-read-chunk-streams > 0 + +Rclone reads --vfs-read-chunk-streams chunks of size +--vfs-read-chunk-size concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +--vfs-read-chunk-size and --vfs-read-chunk-streams as these will depend +on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be --vfs-read-chunk-streams 16 and --vfs-read-chunk-size 4M. +In testing with AWS S3 the performance scaled roughly as the +--vfs-read-chunk-streams setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more --vfs-read-chunk-streams in order to get +the throughput. + VFS Performance These flags may be used to enable/disable features of the VFS for @@ -9871,18 +10234,20 @@ Options --addr string IPaddress:Port or :Port to bind server to --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for nfs + --nfs-cache-dir string The directory the NFS handle cache will use if set --nfs-cache-handle-limit int max file handles cached simultaneously (min 5) (default 1000000) + --nfs-cache-type memory|disk|symlink Type of NFS handle cache to use (default memory) --no-checksum Don't compare checksums on up/download --no-modtime Don't read/write the modification time (can speed things up) --no-seek Don't allow seeking in files --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -9895,15 +10260,19 @@ Options --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -9928,9 +10297,7 @@ Flags for filtering directory listings. --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone serve - Serve a remote over a protocol. @@ -10034,7 +10401,9 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. ---addr may be repeated to listen on multiple IPs/ports/sockets. +--addr may be repeated to listen on multiple IPs/ports/sockets. Socket +activation, described further below, can also be used to accomplish the +same. --server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a @@ -10066,7 +10435,23 @@ authority certificate. --min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). -Authentication +Socket activation + +Instead of the listening addresses specified above, rclone will listen +to all FDs passed by the service manager, if any (and ignore any +arguments passed by --addr`). + +This allows rclone to be a socket-activated service. It can be +configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the +systemd-socket-activatecommand + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 +over TCP. ### Authentication By default this will serve files without needing a login. @@ -10098,7 +10483,7 @@ Use --salt to change the password hashing salt from the default. Options - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --append-only Disallow deletion of repository data --baseurl string Prefix for URLs - leave blank for root @@ -10121,7 +10506,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone serve - Serve a remote over a protocol. @@ -10253,6 +10638,34 @@ serve s3 currently supports the following operations. Other operations will return error Unimplemented. +Authentication + +By default this will serve files without needing a login. + +You can either use an htpasswd file which can take lots of users, or set +a single username and password with the --user and --pass flags. + +If no static users are configured by either of the above methods, and +client certificates are required by the --client-ca flag passed to the +server, the client certificate common name will be considered as the +username. + +Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in +standard apache format and supports MD5, SHA1 and BCrypt for basic +authentication. Bcrypt is recommended. + +To create an htpasswd file: + + touch htpasswd + htpasswd -B htpasswd user + htpasswd -B htpasswd anotherUser + +The password file can be updated while rclone is running. + +Use --realm to set the authentication realm. + +Use --salt to change the password hashing salt from the default. + Server options Use --addr to specify which IP address and port the server should listen @@ -10268,7 +10681,9 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. ---addr may be repeated to listen on multiple IPs/ports/sockets. +--addr may be repeated to listen on multiple IPs/ports/sockets. Socket +activation, described further below, can also be used to accomplish the +same. --server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a @@ -10300,7 +10715,23 @@ authority certificate. --min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). -VFS - Virtual File System +Socket activation + +Instead of the listening addresses specified above, rclone will listen +to all FDs passed by the service manager, if any (and ignore any +arguments passed by --addr`). + +This allows rclone to be a socket-activated service. It can be +configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the +systemd-socket-activatecommand + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 +over TCP. ## VFS - Virtual File System This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing @@ -10526,6 +10957,12 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the +--vfs-read-chunk-streams parameter. + +--vfs-read-chunk-streams == 0 Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is @@ -10542,6 +10979,30 @@ the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. +The chunks will not be buffered in memory. + +--vfs-read-chunk-streams > 0 + +Rclone reads --vfs-read-chunk-streams chunks of size +--vfs-read-chunk-size concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +--vfs-read-chunk-size and --vfs-read-chunk-streams as these will depend +on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be --vfs-read-chunk-streams 16 and --vfs-read-chunk-size 4M. +In testing with AWS S3 the performance scaled roughly as the +--vfs-read-chunk-streams setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more --vfs-read-chunk-streams in order to get +the throughput. + VFS Performance These flags may be used to enable/disable features of the VFS for @@ -10657,19 +11118,21 @@ only with caching. Options - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-key stringArray Set key pair for v4 authorization: access_key_id,secret_access_key + --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --etag-hash string Which hash to use for the ETag, or auto or blank for off (default "MD5") - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --force-path-style If true use path style access if false use virtual hosted style (default true) (default true) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for s3 + --htpasswd string A htpasswd file - if not provided no authentication is done --key string TLS PEM Private key --max-header-bytes int Maximum size of request header (default 4096) --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") @@ -10677,12 +11140,16 @@ Options --no-cleanup Not to cleanup empty folder after object is deleted --no-modtime Don't read/write the modification time (can speed things up) --no-seek Don't allow seeking in files + --pass string Password for authentication --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access + --realm string Realm for authentication + --salt string Password hashing salt (default "dlPL2MqE") --server-read-timeout Duration Timeout for server reading data (default 1h0m0s) --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) + --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -10695,15 +11162,19 @@ Options --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -10728,9 +11199,7 @@ Flags for filtering directory listings. --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone serve - Serve a remote over a protocol. @@ -10772,6 +11241,19 @@ directory. By default the server binds to localhost:2022 - if you want it to be reachable externally then supply --addr :2022 for example. +This also supports being run with socket activation, in which case it +will listen on the first passed FD. It can be configured with .socket +and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the +systemd-socket-activatecommand: + + systemd-socket-activate -l 2222 -- rclone serve sftp :local:vfs/ + +This will socket-activate rclone on the first connection to port 2222 +over TCP. + Note that the default of --vfs-cache-mode off is fine for the rclone sftp backend, but it may not be with other SFTP clients. @@ -11018,6 +11500,12 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the +--vfs-read-chunk-streams parameter. + +--vfs-read-chunk-streams == 0 Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is @@ -11034,6 +11522,30 @@ the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. +The chunks will not be buffered in memory. + +--vfs-read-chunk-streams > 0 + +Rclone reads --vfs-read-chunk-streams chunks of size +--vfs-read-chunk-size concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +--vfs-read-chunk-size and --vfs-read-chunk-streams as these will depend +on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be --vfs-read-chunk-streams 16 and --vfs-read-chunk-size 4M. +In testing with AWS S3 the performance scaled roughly as the +--vfs-read-chunk-streams setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more --vfs-read-chunk-streams in order to get +the throughput. + VFS Performance These flags may be used to enable/disable features of the VFS for @@ -11224,8 +11736,8 @@ Options --auth-proxy string A program to use to create the backend from the auth --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys") --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for sftp --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate) @@ -11238,7 +11750,7 @@ Options --read-only Only allow read-only access --stdio Run an sftp server on stdin/stdout --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -11252,15 +11764,19 @@ Options --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -11285,9 +11801,7 @@ Flags for filtering directory listings. --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone serve - Serve a remote over a protocol. @@ -11338,6 +11852,19 @@ Basic authentication enabled for SSL and for non-SSL connections https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint +Serving over a unix socket + +You can serve the webdav on a unix socket like this: + + rclone serve webdav --addr unix:///tmp/my.socket remote:path + +and connect to it like this using rclone and the webdav backend: + + rclone --webdav-unix-socket /tmp/my.socket --webdav-url http://localhost lsf :webdav: + +Note that there is no authentication on http protocol - this is expected +to be done by the permissions on the socket. + Server options Use --addr to specify which IP address and port the server should listen @@ -11353,7 +11880,9 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. ---addr may be repeated to listen on multiple IPs/ports/sockets. +--addr may be repeated to listen on multiple IPs/ports/sockets. Socket +activation, described further below, can also be used to accomplish the +same. --server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a @@ -11385,7 +11914,23 @@ authority certificate. --min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). -Template +Socket activation + +Instead of the listening addresses specified above, rclone will listen +to all FDs passed by the service manager, if any (and ignore any +arguments passed by --addr`). + +This allows rclone to be a socket-activated service. It can be +configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the +systemd-socket-activatecommand + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 +over TCP. ### Template --template allows a user to specify a custom markup template for HTTP and WebDAV serve functions. The server exports the following markup to @@ -11712,6 +12257,12 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the +--vfs-read-chunk-streams parameter. + +--vfs-read-chunk-streams == 0 Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is @@ -11728,6 +12279,30 @@ the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. +The chunks will not be buffered in memory. + +--vfs-read-chunk-streams > 0 + +Rclone reads --vfs-read-chunk-streams chunks of size +--vfs-read-chunk-size concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +--vfs-read-chunk-size and --vfs-read-chunk-streams as these will depend +on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be --vfs-read-chunk-streams 16 and --vfs-read-chunk-size 4M. +In testing with AWS S3 the performance scaled roughly as the +--vfs-read-chunk-streams setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more --vfs-read-chunk-streams in order to get +the throughput. + VFS Performance These flags may be used to enable/disable features of the VFS for @@ -11914,17 +12489,17 @@ that rclone supports. Options - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --disable-dir-list Disable HTML directory list on GET request for a directory --etag-hash string Which hash to use for the ETag, or auto or blank for off - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for webdav --htpasswd string A htpasswd file - if not provided no authentication is done @@ -11943,7 +12518,7 @@ Options --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --template string User-specified template --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -11957,15 +12532,19 @@ Options --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -11990,9 +12569,7 @@ Flags for filtering directory listings. --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone serve - Serve a remote over a protocol. @@ -12002,10 +12579,10 @@ Changes storage class/tier of objects in remote. Synopsis -rclone settier changes storage tier or class at remote if supported. Few -cloud storage services provides different storage classes on objects, -for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and -Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline etc. +Changes storage tier or class at remote if supported. Few cloud storage +services provides different storage classes on objects, for example AWS +S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, Google Cloud +Storage, Regional Storage, Nearline, Coldline etc. Note that, certain tier changes make objects not available to access immediately. For example tiering to archive in azure blob storage makes @@ -12033,7 +12610,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -12060,7 +12637,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. - rclone test changenotify - Log any change notify requests for the @@ -12087,7 +12664,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone test - Run a test command @@ -12111,7 +12688,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone test - Run a test command @@ -12121,10 +12698,10 @@ Discovers file name or other limitations for paths. Synopsis -rclone info discovers what filenames and upload methods are possible to -write to the paths passed in and how long they can be. It can take some -time. It will write test files into the remote:path passed in. It -outputs a bit of go code for each one. +Discovers what filenames and upload methods are possible to write to the +paths passed in and how long they can be. It can take some time. It will +write test files into the remote:path passed in. It outputs a bit of go +code for each one. NB this can create undeletable files and other hazards - use with care @@ -12145,7 +12722,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone test - Run a test command @@ -12167,7 +12744,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone test - Run a test command @@ -12196,7 +12773,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone test - Run a test command @@ -12212,7 +12789,7 @@ Options See the global flags page for global options not listed here. -SEE ALSO +See Also - rclone test - Run a test command @@ -12252,9 +12829,12 @@ Options -R, --recursive Recursively touch all files -t, --timestamp string Use specified time instead of the current time of day +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Important Options -Important flags useful for most commands. +Important flags useful for most commands -n, --dry-run Do a trial run with no permanent changes -i, --interactive Enable interactive mode @@ -12262,7 +12842,7 @@ Important flags useful for most commands. Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -12289,14 +12869,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -12306,8 +12884,8 @@ List the contents of the remote in a tree like fashion. Synopsis -rclone tree lists the contents of a remote in a similar way to the unix -tree command. +Lists the contents of a remote in a similar way to the unix tree +command. For example @@ -12356,9 +12934,12 @@ Options -U, --unsorted Leave files unsorted --version Sort files alphanumerically by version +Options shared with other commands are described next. See the global +flags page for global options not listed here. + Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings --delete-excluded Delete files on dest excluded from sync --exclude stringArray Exclude files matching pattern @@ -12385,14 +12966,12 @@ Flags for filtering directory listings. Listing Options -Flags for listing directories. +Flags for listing directories --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions -See the global flags page for global options not listed here. - -SEE ALSO +See Also - rclone - Show help for rclone commands, flags and backends. @@ -12971,21 +13550,34 @@ optional element. characters. It is optional. - HH:MM is an hour from 00:00 to 23:59. +Entries can be separated by spaces or semicolons. + +Note: Semicolons can be used as separators instead of spaces to avoid +parsing issues in environments like Docker. + An example of a typical timetable to avoid link saturation during daytime working hours could be: +Using spaces as separators: --bwlimit "08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off" -In this example, the transfer bandwidth will be set to 512 KiB/s at 8am -every day. At noon, it will rise to 10 MiB/s, and drop back to 512 +Using semicolons as separators: +--bwlimit "08:00,512k;12:00,10M;13:00,512k;18:00,30M;23:00,off" + +In these examples, the transfer bandwidth will be set to 512 KiB/s at +8am every day. At noon, it will rise to 10 MiB/s, and drop back to 512 KiB/sec at 1pm. At 6pm, the bandwidth limit will be set to 30 MiB/s, and at 11pm it will be completely disabled (full speed). Anything between 11pm and 8am will remain unlimited. An example of timetable with WEEKDAY could be: +Using spaces as separators: --bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off" +Using semicolons as separators: +--bwlimit "Mon-00:00,512;Fri-23:59,10M;Sat-10:00,1M;Sun-20:00,off" + It means that, the transfer bandwidth will be set to 512 KiB/s on Monday. It will rise to 10 MiB/s before the end of Friday. At 10:00 on Saturday it will be set to 1 MiB/s. From 20:00 on Sunday it will be @@ -13573,10 +14165,12 @@ such as: - local - ftp - sftp +- pcloud Without --inplace (the default) rclone will first upload to a temporary -file with an extension like this, where XXXXXX represents a random -string and .partial is --partial-suffix value (.partial by default). +file with an extension like this, where XXXXXX represents a hash of the +source file's fingerprint and .partial is --partial-suffix value +(.partial by default). original-file-name.XXXXXX.partial @@ -14138,7 +14732,8 @@ The default is .partial. This flag supplies a program which should supply the config password when run. This is an alternative to rclone prompting for the password or -setting the RCLONE_CONFIG_PASS variable. +setting the RCLONE_CONFIG_PASS variable. It is also used when setting +the config password for the first time. The argument to this should be a command with a space separated list of arguments. If one of the arguments has a space in then enclose it in ", @@ -14151,6 +14746,10 @@ Eg --password-command 'echo "hello with space"' --password-command 'echo "hello with ""quotes"" and space"' +Note that when changing the configuration password the environment +variable RCLONE_PASSWORD_CHANGE=1 will be set. This can be used to +distinguish initial decryption of the config file from the new password. + See the Configuration Encryption for more info. See a Windows PowerShell example on the Wiki. @@ -14752,6 +15351,13 @@ your configuration. There is no way to recover the configuration if you lose your password. +You can also use + +- rclone config encryption set to set the config encryption directly +- rclone config encryption remove to remove it +- rclone config encryption check to check that it is encrypted + properly. + rclone uses nacl secretbox which in turn uses XSalsa20 and Poly1305 to encrypt and authenticate your configuration with secret-key cryptography. The password is SHA-256 hashed, which produces the key for @@ -14814,11 +15420,60 @@ actually using anything from such a configuration file, you can avoid it being loaded by overriding the location, e.g. with one of the documented special values for memory-only configuration. Since only backend options can be stored in configuration files, this is normally unnecessary for -commands that do not operate on backends, e.g. genautocomplete. However, -it will be relevant for commands that do operate on backends in general, +commands that do not operate on backends, e.g. completion. However, it +will be relevant for commands that do operate on backends in general, but are used without referencing a stored remote, e.g. listing local filesystem paths, or connection strings: rclone --config="" ls . +Configuration Encryption Cheatsheet + +You can quickly apply a configuration encryption without plain-text at +rest or transfer. Detailed instructions for popular OSes: + +Mac + +- Generate and store a password + +security add-generic-password -a rclone -s config -w $(openssl rand -base64 40) + +- Add the retrieval instruction to your .zprofile / .profile + +export RCLONE_PASSWORD_COMMAND="/usr/bin/security find-generic-password -a rclone -s config -w" + +Linux + +- Prerequisite + +Linux doesn't come with a default password manager. Let's install the +"pass" utility using a package manager, e.g. apt install pass, +yum install pass, etc.; then initialize a password store: + +pass init rclone + +- Generate and store a password + +echo $(openssl rand -base64 40) | pass insert -m rclone/config + +- Add the retrieval instruction + +export RCLONE_PASSWORD_COMMAND="/usr/bin/pass rclone/config" + +Windows + +- Generate and store a password + +New-Object -TypeName PSCredential -ArgumentList "rclone", (ConvertTo-SecureString -String ([System.Web.Security.Membership]::GeneratePassword(40, 10)) -AsPlainText -Force) | Export-Clixml -Path "rclone-credential.xml" + +- Add the password retrieval instruction + +[Environment]::SetEnvironmentVariable("RCLONE_PASSWORD_COMMAND", "[System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR((Import-Clixml -Path "rclone-credential.xml").Password))") + +Encrypt the config file (all systems) + +- Execute rclone config -> s + +- Add/update the password from previous steps + Developer options These options are useful when developing or debugging rclone. There are @@ -14966,6 +15621,24 @@ Rclone prefixes all log messages with their level in capitals, e.g. INFO which makes it easy to grep the log file for different kinds of information. +Metrics + +Rclone can publish metrics in the OpenMetrics/Prometheus format. + +To enable the metrics endpoint, use the --metrics-addr flag. Metrics can +also be published on the --rc-addr port if the --rc flag and +--rc-enable-metrics flags are supplied or if using rclone rcd +--rc-enable-metrics + +Rclone provides extensive configuration options for the metrics HTTP +endpoint. These settings are grouped under the Metrics section and have +a prefix --metrics-*. + +When metrics are enabled with --rc-enable-metrics, they will be +published on the same port as the rc API. In this case, the --metrics-* +flags will be ignored, and the HTTP endpoint configuration will be +managed by the --rc-* parameters. + Exit Code If any errors occur during the command execution, rclone will exit with @@ -15139,33 +15812,30 @@ two ways of doing it, described below. Configuring using rclone authorize On the headless box run rclone config but answer N to the -Use web browser to automatically authenticate? question. +Use auto config? question. + + Use auto config? + * Say Y if not sure + * Say N if you are working on a remote or headless machine - ... - Remote config - Use web browser to automatically authenticate rclone with remote? - * Say Y if the machine running rclone has a web browser you can use - * Say N if running rclone on a (remote) machine without web browser access - If not sure try Y. If Y failed, try N. y) Yes (default) n) No y/n> n + + Option config_token. For this to work, you will need rclone available on a machine that has a web browser available. - For more help and alternate methods see: https://rclone.org/remote_setup/ - Execute the following on the machine with the web browser (same rclone version recommended): - - rclone authorize "dropbox" - - Then paste the result below: - result> + rclone authorize "onedrive" + Then paste the result. + Enter a value. + config_token> Then on your main desktop machine - rclone authorize "dropbox" + rclone authorize "onedrive" If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... @@ -15176,7 +15846,7 @@ Then on your main desktop machine Then back to the headless box, paste in the code - result> SECRET_TOKEN + config_token> SECRET_TOKEN -------------------- [acd12] client_id = @@ -15217,14 +15887,12 @@ box port 53682 to local machine by using the following command: ssh -L localhost:53682:localhost:53682 username@remote_server Then on the headless box run rclone config and answer Y to the -Use web browser to automatically authenticate? question. +Use auto config? question. + + Use auto config? + * Say Y if not sure + * Say N if you are working on a remote or headless machine - ... - Remote config - Use web browser to automatically authenticate rclone with remote? - * Say Y if the machine running rclone has a web browser you can use - * Say N if running rclone on a (remote) machine without web browser access - If not sure try Y. If Y failed, try N. y) Yes (default) n) No y/n> y @@ -15896,7 +16564,7 @@ Other filters --min-size - Don't transfer any file smaller than this Controls the minimum size file within the scope of an rclone command. -Default units are KiB but abbreviations K, M, G, T or P are valid. +Default units are KiB but abbreviations B, K, M, G, T or P are valid. E.g. rclone ls remote: --min-size 50k lists files on remote: of 50 KiB size or larger. @@ -15906,7 +16574,7 @@ See the size option docs for more info. --max-size - Don't transfer any file larger than this Controls the maximum size file within the scope of an rclone command. -Default units are KiB but abbreviations K, M, G, T or P are valid. +Default units are KiB but abbreviations B, K, M, G, T or P are valid. E.g. rclone ls remote: --max-size 1G lists files on remote: of 1 GiB size or smaller. @@ -16231,7 +16899,10 @@ Default Off. --rc-enable-metrics -Enable OpenMetrics/Prometheus compatible endpoint at /metrics. +Enable OpenMetrics/Prometheus compatible endpoint at /metrics. If more +control over the metrics is desired (for example running it on a +different port or with different auth) then endpoint can be enabled with +the --metrics-* flags instead. Default Off. @@ -16520,6 +17191,119 @@ parameters. "HARD" for CutoffMode or DEBUG for LogLevel. - BandwidthSpec - this will be set and returned as a string, eg "1M". +Option blocks + +The calls options/info (for the main config) and config/providers (for +the backend config) may be used to get information on the rclone +configuration options. This can be used to build user interfaces for +displaying and setting any rclone option. + +These consist of arrays of Option blocks. These have the following +format. Each block describes a single option. + + ------------------------------------------------------------------------ + Field Type Optional Description + ------------- ------------ ------------------- ------------------------- + Name string N name of the option in + snake_case + + FieldName string N name of the field used in + the rc - if blank use + Name + + Help string N help, started with a + single sentence on a + single line + + Groups string Y groups this option + belongs to - comma + separated string for + options classification + + Provider string Y set to filter on provider + + Default any N default value, if set + (and not to nil or "") + then Required does + nothing + + Value any N value to be set by flags + + Examples Examples Y predefined values that + can be selected from list + (multiple-choice option) + + ShortOpt string Y the short command line + option for this + + Hide Visibility N if non zero, this option + is hidden from the + configurator or the + command line + + Required bool N this option is required, + meaning value cannot be + empty unless there is a + default + + IsPassword bool N set if the option is a + password + + NoPrefix bool N set if the option for + this should not use the + backend prefix + + Advanced bool N set if this is an + advanced config option + + Exclusive bool N set if the answer can + only be one of the + examples (empty string + allowed unless Required + or Default is set) + + Sensitive bool N set if this option should + be redacted when using + rclone config redacted + ------------------------------------------------------------------------ + +An example of this might be the --log-level flag. Note that the Name of +the option becomes the command line flag with _ replaced with -. + + { + "Advanced": false, + "Default": 5, + "DefaultStr": "NOTICE", + "Examples": [ + { + "Help": "", + "Value": "EMERGENCY" + }, + { + "Help": "", + "Value": "ALERT" + }, + ... + ], + "Exclusive": true, + "FieldName": "LogLevel", + "Groups": "Logging", + "Help": "Log level DEBUG|INFO|NOTICE|ERROR", + "Hide": 0, + "IsPassword": false, + "Name": "log_level", + "NoPrefix": true, + "Required": true, + "Sensitive": false, + "Type": "LogLevel", + "Value": null, + "ValueStr": "NOTICE" + }, + +Note that the Help may be multiple lines separated by \n. The first line +will always be a short sentence and this is the sentence shown when +running rclone help flags. + Specifying remotes to work on Remotes are specified with the fs=, srcFs=, dstFs= parameters depending @@ -16545,7 +17329,7 @@ For example this JSON is equivalent to remote:/tmp { "_name": "remote", - "_path": "/tmp" + "_root": "/tmp" } And this is equivalent to :sftp,host='example.com':/tmp @@ -16553,14 +17337,14 @@ And this is equivalent to :sftp,host='example.com':/tmp { "type": "sftp", "host": "example.com", - "_path": "/tmp" + "_root": "/tmp" } And this is equivalent to /tmp/dir { type = "local", - _ path = "/tmp/dir" + _root = "/tmp/dir" } Supported commands @@ -16745,6 +17529,9 @@ Returns a JSON object: - providers - array of objects See the config providers command for more information on the above. +Note that the Options blocks are in the same format as returned by +"options/info". They are described in the option blocks section. + Authentication is required for this call. config/setpath: Set the path of the config file @@ -17777,6 +18564,11 @@ options/get: Get all the global options Returns an object where keys are option block names and values are an object with the current option values in. +Parameters: + +- blocks: optional string of comma separated blocks to include + - all are included if this is missing or "" + Note that these are the global options which are unaffected by use of the _config and _filter parameters. If you wish to read the parameters set in _config then use options/config and for _filter use @@ -17785,6 +18577,19 @@ options/filter. This shows the internal names of the option within rclone which should map to the external options very easily with a few exceptions. +options/info: Get info about all the global options + +Returns an object where keys are option block names and values are an +array of objects with info about each options. + +Parameters: + +- blocks: optional string of comma separated blocks to include + - all are included if this is missing or "" + +These objects are in the same format as returned by "config/providers". +They are described in the option blocks section. + options/local: Get the currently active config for this call Returns an object with the keys "config" and "filter". The "config" key @@ -18080,6 +18885,68 @@ This command takes an "fs" parameter. If this parameter is not supplied and if there is only one VFS in use then that VFS will be used. If there is more than one VFS in use then the "fs" parameter must be supplied. +vfs/queue: Queue info for a VFS. + +This returns info about the upload queue for the selected VFS. + +This is only useful if --vfs-cache-mode > off. If you call it when the +--vfs-cache-mode is off, it will return an empty result. + + { + "queued": // an array of files queued for upload + [ + { + "name": "file", // string: name (full path) of the file, + "id": 123, // integer: id of this item in the queue, + "size": 79, // integer: size of the file in bytes + "expiry": 1.5 // float: time until file is eligible for transfer, lowest goes first + "tries": 1, // integer: number of times we have tried to upload + "delay": 5.0, // float: seconds between upload attempts + "uploading": false, // boolean: true if item is being uploaded + }, + ], + } + +The expiry time is the time until the file is elegible for being +uploaded in floating point seconds. This may go negative. As rclone only +transfers --transfers files at once, only the lowest --transfers expiry +times will have uploading as true. So there may be files with negative +expiry times for which uploading is false. + +This command takes an "fs" parameter. If this parameter is not supplied +and if there is only one VFS in use then that VFS will be used. If there +is more than one VFS in use then the "fs" parameter must be supplied. + +vfs/queue-set-expiry: Set the expiry time for an item queued for upload. + +Use this to adjust the expiry time for an item in the upload queue. You +will need to read the id of the item using vfs/queue before using this +call. + +You can then set expiry to a floating point number of seconds from now +when the item is eligible for upload. If you want the item to be +uploaded as soon as possible then set it to a large negative number (eg +-1000000000). If you want the upload of the item to be delayed for a +long time then set it to a large positive number. + +Setting the expiry of an item which has already has started uploading +will have no effect - the item will carry on being uploaded. + +This will return an error if called with --vfs-cache-mode off or if the +id passed is not found. + +This takes the following parameters + +- fs - select the VFS in use (optional) +- id - a numeric ID as returned from vfs/queue +- expiry - a new expiry time as floating point seconds + +This returns an empty result on success, or an error. + +This command takes an "fs" parameter. If this parameter is not supplied +and if there is only one VFS in use then that VFS will be used. If there +is more than one VFS in use then the "fs" parameter must be supplied. + vfs/refresh: Refresh the directory cache. This reads the directories for the specified paths and freshens the @@ -18351,7 +19218,9 @@ Here is an overview of the major features of each cloud storage system. Citrix ShareFile MD5 R/W Yes No - - Dropbox DBHASH ¹ R Yes No - - Enterprise File Fabric - R/W Yes No R/W - + Files.com MD5, CRC32 DR/W Yes No R - FTP - R/W ¹⁰ No No - - + Gofile MD5 DR/W No Yes R - Google Cloud Storage MD5 R/W No No R/W - Google Drive MD5, SHA1, SHA256 DR/W No Yes R/W DRWU Google Photos - - No Yes R - @@ -18373,6 +19242,7 @@ Here is an overview of the major features of each cloud storage system. Oracle Object Storage MD5 R/W No No R/W - pCloud MD5, SHA1 ⁷ R No No W - PikPak MD5 R No No R - + Pixeldrain SHA256 R/W No No R RW premiumize.me - - Yes No R - put.io CRC-32 R/W No Yes R - Proton Drive SHA1 R/W No No R - @@ -18681,8 +19551,8 @@ that you want to remain as those characters on the remote rather than being translated to regular (halfwidth) *, ? and :. The --backend-encoding flags allow you to change that. You can disable -the encoding completely with --backend-encoding None or set -encoding = None in the config file. +the encoding completely with --backend-encoding Raw or set +encoding = Raw in the config file. Encoding takes a comma separated list of encodings. You can see the list of all possible values by passing an invalid value to this flag, e.g. @@ -18713,6 +19583,8 @@ show you the defaults for the backends. DoubleQuote " ＂ + Exclamation ! ！ + Hash # ＃ InvalidUtf8 An invalid UTF-8 � @@ -18733,8 +19605,7 @@ show you the defaults for the backends. LtGt <, > ＜, ＞ - None No characters are - encoded + None ¹ NUL 0x00 ␀ Percent % ％ @@ -18761,6 +19632,10 @@ show you the defaults for the backends. SquareBracket [, ] ［, ］ ---------------------------------------------------------------------------------- +¹ Encoding from NUL 0x00 to ␀ is always implicit except when using Raw. +It was previously incorrectly documented as disabling encoding, and to +maintain backward compatibility, its behavior has not been changed. + Encoding example: FTP To take a specific example, the FTP backend's default encoding is @@ -18802,7 +19677,7 @@ same as the default value but without Colon,Question,Asterisk: --local-encoding "Slash,LtGt,DoubleQuote,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot" Alternatively, you can disable the conversion of any characters with ---local-encoding None. +--local-encoding Raw. Instead of using command-line argument --local-encoding, you may also set it as environment variable RCLONE_LOCAL_ENCODING, or configure a @@ -18887,8 +19762,12 @@ upon backend-specific capabilities. Enterprise File Yes Yes Yes Yes Yes No No No No No Yes Fabric + Files.com Yes Yes Yes Yes No No Yes No Yes No Yes + FTP No No Yes Yes No No Yes No No No Yes + Gofile Yes Yes Yes Yes No No Yes No Yes Yes Yes + Google Cloud Yes Yes No No No Yes Yes No No No No Storage @@ -18937,6 +19816,8 @@ upon backend-specific capabilities. PikPak Yes Yes Yes Yes Yes No No No Yes Yes Yes + Pixeldrain Yes No Yes Yes No No Yes No Yes Yes Yes + premiumize.me Yes No Yes Yes No No No No Yes Yes Yes put.io Yes No Yes Yes Yes No Yes No No Yes Yes @@ -19076,11 +19957,11 @@ into groups. Copy -Flags for anything which can Copy a file. +Flags for anything which can copy a file. --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -19113,7 +19994,7 @@ Flags for anything which can Copy a file. Sync -Flags just used for rclone sync. +Flags used for sync commands. --backup-dir string Make backups into hierarchy based in DIR --delete-after When synchronizing, delete files on destination after transferring (default) @@ -19138,13 +20019,13 @@ Important flags useful for most commands. Check -Flags used for rclone check. +Flags used for check commands. --max-backlog int Maximum number of objects in sync or check backlog (default 10000) Networking -General networking and HTTP stuff. +Flags for general networking and HTTP stuff. --bind string Local address to bind to for outgoing connections, IPv4, IPv6 or name --bwlimit BwTimetable Bandwidth limit in KiB/s, or use suffix B|K|M|G|T|P or a full timetable @@ -19153,7 +20034,7 @@ General networking and HTTP stuff. --client-cert string Client SSL certificate (PEM) for mutual TLS auth --client-key string Client SSL private key (PEM) for mutual TLS auth --contimeout Duration Connect timeout (default 1m0s) - --disable-http-keep-alives Disable HTTP keep-alives and use each connection once. + --disable-http-keep-alives Disable HTTP keep-alives and use each connection once --disable-http2 Disable HTTP/2 in the global transport --dscp string Set DSCP value to connections, value or name, e.g. CS1, LE, DF, AF21 --expect-continue-timeout Duration Timeout when using expect / 100-continue in HTTP (default 1s) @@ -19166,7 +20047,7 @@ General networking and HTTP stuff. --tpslimit float Limit HTTP transactions per second to this --tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --use-cookies Enable session cookiejar - --user-agent string Set the user-agent to a specified string (default "rclone/v1.67.0") + --user-agent string Set the user-agent to a specified string (default "rclone/v1.68.0") Performance @@ -19178,7 +20059,7 @@ Flags helpful for increasing performance. Config -General configuration of rclone. +Flags for general configuration of rclone. --ask-password Allow prompt for password for encrypted configuration (default true) --auto-confirm If enabled, do not request console confirmation @@ -19250,7 +20131,7 @@ Flags for listing directories. Logging -Logging and statistics. +Flags for logging and statistics. --log-file string Log everything to this file --log-format string Comma separated list of log format options (default "date,time") @@ -19268,7 +20149,7 @@ Logging and statistics. --stats-one-line-date-format string Enable --stats-one-line-date and use custom formatted date: Enclose date string in double quotes ("), see https://golang.org/pkg/time/#Time.Format --stats-unit string Show data rate in stats as either 'bits' or 'bytes' per second (default "bytes") --syslog Use Syslog for logging - --syslog-facility string Facility for syslog, e.g. KERN,USER,... (default "DAEMON") + --syslog-facility string Facility for syslog, e.g. KERN,USER (default "DAEMON") --use-json-log Use json log format -v, --verbose count Print lots more stuff (repeat for more) @@ -19291,12 +20172,12 @@ RC Flags to control the Remote Control API. --rc Enable the remote control server - --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572]) + --rc-addr stringArray IPaddress:Port or :Port to bind server to (default ["localhost:5572"]) --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from --rc-baseurl string Prefix for URLs - leave blank for root --rc-cert string TLS PEM key (concatenation of certificate and CA certificate) --rc-client-ca string Client certificate authority to verify clients with - --rc-enable-metrics Enable prometheus metrics on /metrics + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server --rc-files string Path to local files to serve on the HTTP server --rc-htpasswd string A htpasswd file - if not provided no authentication is done --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s) @@ -19320,9 +20201,31 @@ Flags to control the Remote Control API. --rc-web-gui-no-open-browser Don't open the browser automatically --rc-web-gui-update Check and update to latest version of web gui +Metrics + +Flags to control the Metrics HTTP endpoint.. + + --metrics-addr stringArray IPaddress:Port or :Port to bind metrics server to (default [""]) + --metrics-allow-origin string Origin which cross-domain request (CORS) can be executed from + --metrics-baseurl string Prefix for URLs - leave blank for root + --metrics-cert string TLS PEM key (concatenation of certificate and CA certificate) + --metrics-client-ca string Client certificate authority to verify clients with + --metrics-htpasswd string A htpasswd file - if not provided no authentication is done + --metrics-key string TLS PEM Private key + --metrics-max-header-bytes int Maximum size of request header (default 4096) + --metrics-min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") + --metrics-pass string Password for authentication + --metrics-realm string Realm for authentication + --metrics-salt string Password hashing salt (default "dlPL2MqE") + --metrics-server-read-timeout Duration Timeout for server reading data (default 1h0m0s) + --metrics-server-write-timeout Duration Timeout for server writing data (default 1h0m0s) + --metrics-template string User-specified template + --metrics-user string User name for authentication + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server + Backend -Backend only flags. These can be set in the config file also. +Backend-only flags (these can be set in the config file also). --alias-description string Description of the remote --alias-remote string Remote or path to alias @@ -19545,6 +20448,12 @@ Backend only flags. These can be set in the config file also. --filefabric-token-expiry string Token expiry time --filefabric-url string URL of the Enterprise File Fabric to connect to --filefabric-version string Version read from the file fabric + --filescom-api-key string The API key used to authenticate with Files.com + --filescom-description string Description of the remote + --filescom-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot) + --filescom-password string The password used to authenticate with Files.com (obscured) + --filescom-site string Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com) + --filescom-username string The username used to authenticate with Files.com --ftp-ask-password Allow asking for FTP password when needed --ftp-close-timeout Duration Maximum time to wait for a response to close (default 1m0s) --ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited @@ -19588,6 +20497,12 @@ Backend only flags. These can be set in the config file also. --gcs-token string OAuth Access Token as a JSON blob --gcs-token-url string Token server url --gcs-user-project string User project + --gofile-access-token string API Access token + --gofile-account-id string Account ID + --gofile-description string Description of the remote + --gofile-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftPeriod,RightPeriod,InvalidUtf8,Dot,Exclamation) + --gofile-list-chunk int Number of items to list in each call (default 1000) + --gofile-root-folder-id string ID of the root folder --gphotos-auth-url string Auth server URL --gphotos-batch-commit-timeout Duration Max time to wait for a batch to finish committing (default 10m0s) --gphotos-batch-mode string Upload file batching sync|async|off (default "sync") @@ -19679,6 +20594,7 @@ Backend only flags. These can be set in the config file also. --local-description string Description of the remote --local-encoding Encoding The encoding for the backend (default Slash,Dot) --local-no-check-updated Don't check to see if the files change during upload + --local-no-clone Disable reflink cloning for server-side copies --local-no-preallocate Disable preallocation of disk space for transferred files --local-no-set-modtime Disable setting modtime --local-no-sparse Disable sparse files for multi-thread downloads @@ -19795,6 +20711,10 @@ Backend only flags. These can be set in the config file also. --pikpak-upload-concurrency int Concurrency for multipart uploads (default 5) --pikpak-use-trash Send files to the trash instead of deleting permanently (default true) --pikpak-user string Pikpak username + --pixeldrain-api-key string API key for your pixeldrain account + --pixeldrain-api-url string The API endpoint to connect to. In the vast majority of cases it's fine to leave (default "https://pixeldrain.com/api") + --pixeldrain-description string Description of the remote + --pixeldrain-root-folder-id string Root of the filesystem to use (default "me") --premiumizeme-auth-url string Auth server URL --premiumizeme-client-id string OAuth Client Id --premiumizeme-client-secret string OAuth Client Secret @@ -19869,6 +20789,7 @@ Backend only flags. These can be set in the config file also. --s3-provider string Choose your S3 provider --s3-region string Region to connect to --s3-requester-pays Enables requester pays option when interacting with S3 bucket + --s3-sdk-log-mode Bits Set to debug the SDK (default Off) --s3-secret-access-key string AWS Secret Access Key (password) --s3-server-side-encryption string The server-side encryption algorithm used when storing this object in S3 --s3-session-token string An AWS session token @@ -19879,7 +20800,6 @@ Backend only flags. These can be set in the config file also. --s3-sse-customer-key-md5 string If using SSE-C you may provide the secret encryption key MD5 checksum (optional) --s3-sse-kms-key-id string If using KMS ID you must provide the ARN of Key --s3-storage-class string The storage class to use when storing new objects in S3 - --s3-sts-endpoint string Endpoint for STS --s3-upload-concurrency int Concurrency for multipart uploads and copies (default 4) --s3-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi) --s3-use-accelerate-endpoint If true use the AWS S3 accelerated endpoint @@ -19889,6 +20809,7 @@ Backend only flags. These can be set in the config file also. --s3-use-multipart-etag Tristate Whether to use ETag in multipart uploads for verification (default unset) --s3-use-multipart-uploads Tristate Set if rclone should use multipart uploads (default unset) --s3-use-presigned-request Whether to use a presigned request or PutObject for single part uploads + --s3-use-unsigned-payload Tristate Whether to use an unsigned payload in PutObject (default unset) --s3-v2-auth If true use v2 authentication --s3-version-at Time Show file versions as they were at the specified time (default off) --s3-version-deleted Show deleted file markers when using versions @@ -19997,10 +20918,12 @@ Backend only flags. These can be set in the config file also. --swift-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8) --swift-endpoint-type string Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) (default "public") --swift-env-auth Get swift credentials from environment variables in standard OpenStack form + --swift-fetch-until-empty-page When paginating, always fetch unless we received an empty page --swift-key string API key or password (OS_PASSWORD) --swift-leave-parts-on-error If true avoid calling abort upload on a failure --swift-no-chunk Don't chunk files during streaming upload --swift-no-large-objects Disable support for static and dynamic large objects + --swift-partial-page-fetch-threshold int When paginating, fetch if the current page is within this percentage of the limit --swift-region string Region name - optional (OS_REGION_NAME) --swift-storage-policy string The storage policy to use when creating a new container --swift-storage-url string Storage URL - optional (OS_STORAGE_URL) @@ -20038,6 +20961,7 @@ Backend only flags. These can be set in the config file also. --webdav-owncloud-exclude-shares Exclude ownCloud shares --webdav-pacer-min-sleep Duration Minimum time to sleep between API calls (default 10ms) --webdav-pass string Password (obscured) + --webdav-unix-socket string Path to a unix domain socket to dial to, instead of opening a TCP connection directly --webdav-url string URL of http host to connect to --webdav-user string User name --webdav-vendor string Name of the WebDAV site/service/software you are using @@ -20047,6 +20971,7 @@ Backend only flags. These can be set in the config file also. --yandex-description string Description of the remote --yandex-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot) --yandex-hard-delete Delete files permanently rather than putting them into the trash + --yandex-spoof-ua Set the user agent to match an official version of the yandex disk client. May help with upload performance (default true) --yandex-token string OAuth Access Token as a JSON blob --yandex-token-url string Token server url --zoho-auth-url string Auth server URL @@ -22405,6 +23330,15 @@ Unison and synchronization in general. Changelog +v1.68 + +- Fixed an issue affecting backends that round modtimes to a lower + precision. + +v1.67 + +- Added integration tests against all backends. + v1.66 - Copies and deletes are now handled in one operation instead of two @@ -22606,7 +23540,7 @@ Verify signatures and hashes together You can verify the signatures and hashes in one command line like this: - $ gpg --decrypt SHA256SUMS | sha256sum -c --ignore-missing + $ h=$(gpg --decrypt SHA256SUMS) && echo "$h" | sha256sum - -c --ignore-missing gpg: Signature made Mon 17 Jul 2023 15:03:17 BST gpg: using DSA key FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA gpg: Good signature from "Nick Craig-Wood " [ultimate] @@ -22658,11 +23592,11 @@ This will guide you through an interactive setup process: n) No y/n> Remote config - -------------------- - [remote] - type = fichier - api_key = example_key - -------------------- + Configuration complete. + Options: + - type: fichier + - api_key: example_key + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -22874,10 +23808,11 @@ This will guide you through an interactive setup process: Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path". remote> /mnt/storage/backup Remote config - -------------------- - [remote] - remote = /mnt/storage/backup - -------------------- + Configuration complete. + Options: + - type: alias + - remote: /mnt/storage/backup + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -23192,20 +24127,20 @@ This will guide you through an interactive setup process. \ "GLACIER_IR" storage_class> 1 Remote config - -------------------- - [remote] - type = s3 - provider = AWS - env_auth = false - access_key_id = XXX - secret_access_key = YYY - region = us-east-1 - endpoint = - location_constraint = - acl = private - server_side_encryption = - storage_class = - -------------------- + Configuration complete. + Options: + - type: s3 + - provider: AWS + - env_auth: false + - access_key_id: XXX + - secret_access_key: YYY + - region: us-east-1 + - endpoint: + - location_constraint: + - acl: private + - server_side_encryption: + - storage_class: + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -23502,17 +24437,26 @@ The different authentication methods are tried in this order: - Profile files are standard files used by AWS CLI tools - By default it will use the profile in your home directory (e.g. ~/.aws/credentials on unix based systems) file and the - "default" profile, to change set these environment - variables: - - AWS_SHARED_CREDENTIALS_FILE to control which file. - - AWS_PROFILE to control which profile to use. + "default" profile, to change set these environment variables + or config keys: + - AWS_SHARED_CREDENTIALS_FILE to control which file or the + shared_credentials_file config key. + - AWS_PROFILE to control which profile to use or the + profile config key. - Or, run rclone in an ECS task with an IAM role (AWS only). - Or, run rclone on an EC2 instance with an IAM role (AWS only). - Or, run rclone in an EKS pod with an IAM role that is associated with a service account (AWS only). + - Or, use process credentials to read config from an external + program. + +With env_auth = true rclone (which uses the SDK for Go v2) should +support all authentication methods that the aws CLI tool does and the +other AWS SDKs. If none of these option actually end up providing rclone with AWS -credentials then S3 interaction will be non-authenticated (see below). +credentials then S3 interaction will be non-authenticated (see the +anonymous access section for more info). S3 Permissions @@ -24357,6 +25301,9 @@ Some providers (e.g. AWS, Aliyun OSS, Netease COS, or Tencent COS) require this set to false - rclone will do this automatically based on the provider setting. +Note that if your bucket isn't a valid DNS name, i.e. has '.' or '_' in, +you'll need to set this to true. + Properties: - Config: force_path_style @@ -24644,6 +25591,24 @@ Properties: - Type: Tristate - Default: unset +--s3-use-unsigned-payload + +Whether to use an unsigned payload in PutObject + +Rclone has to avoid the AWS SDK seeking the body when calling PutObject. +The AWS provider can add checksums in the trailer to avoid seeking but +other providers can't. + +This should be true, false or left unset to use the default for the +provider. + +Properties: + +- Config: use_unsigned_payload +- Env Var: RCLONE_S3_USE_UNSIGNED_PAYLOAD +- Type: Tristate +- Default: unset + --s3-use-presigned-request Whether to use a presigned request or PutObject for single part uploads @@ -24797,7 +25762,7 @@ Properties: --s3-sts-endpoint -Endpoint for STS. +Endpoint for STS (deprecated). Leave blank if using AWS to use the default endpoint for the region. @@ -24858,6 +25823,32 @@ Properties: - Type: Tristate - Default: unset +--s3-sdk-log-mode + +Set to debug the SDK + +This can be set to a comma separated list of the following functions: + +- Signing +- Retries +- Request +- RequestWithBody +- Response +- ResponseWithBody +- DeprecatedUsage +- RequestEventMessage +- ResponseEventMessage + +Use Off to disable and All to set all log levels. You will need to use +-vv to see the debug level logs. + +Properties: + +- Config: sdk_log_mode +- Env Var: RCLONE_S3_SDK_LOG_MODE +- Type: Bits +- Default: Off + --s3-description Description of the remote. @@ -24923,18 +25914,20 @@ backend/command. restore -Restore objects from GLACIER to normal storage +Restore objects from GLACIER or INTELLIGENT-TIERING archive tier rclone backend restore remote: [options] [+] This command can be used to restore one or more objects from GLACIER to -normal storage. +normal storage or from INTELLIGENT-TIERING Archive Access / Deep Archive +Access tier to the Frequent Access tier. Usage Examples: rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS + rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY This flag also obeys the filters. Test first with --interactive/-i or --dry-run flags @@ -24962,18 +25955,20 @@ The Status will be OK if it was successful or an error message if not. Options: - "description": The optional description for the job. -- "lifetime": Lifetime of the active copy in days +- "lifetime": Lifetime of the active copy in days, ignored for + INTELLIGENT-TIERING storage - "priority": Priority of restore: Standard|Expedited|Bulk restore-status -Show the restore status for objects being restored from GLACIER to -normal storage +Show the restore status for objects being restored from GLACIER or +INTELLIGENT-TIERING storage rclone backend restore-status remote: [options] [+] This command can be used to show the status for objects being restored -from GLACIER to normal storage. +from GLACIER to normal storage or from INTELLIGENT-TIERING Archive +Access / Deep Archive Access tier to the Frequent Access tier. Usage Examples: @@ -25003,6 +25998,15 @@ It returns a list of status dictionaries. "RestoreExpiryDate": "2023-09-06T12:29:19+01:00" }, "StorageClass": "DEEP_ARCHIVE" + }, + { + "Remote": "test.gz", + "VersionID": null, + "RestoreStatus": { + "IsRestoreInProgress": true, + "RestoreExpiryDate": "null" + }, + "StorageClass": "INTELLIGENT_TIERING" } ] @@ -25132,15 +26136,6 @@ looking like this: [anons3] type = s3 provider = AWS - env_auth = false - access_key_id = - secret_access_key = - region = us-east-1 - endpoint = - location_constraint = - acl = private - server_side_encryption = - storage_class = Then use it as normal with the name of the public bucket, e.g. @@ -25148,6 +26143,10 @@ Then use it as normal with the name of the public bucket, e.g. You will be able to list and copy data but not upload it. +You can also do this entirely on the command line + + rclone lsd :s3,provider=AWS:1000genomes + Providers AWS S3 @@ -25329,6 +26328,11 @@ look within a bucket. For R2 tokens with the "Object Read & Write" permission, you may also need to add no_check_bucket = true for object uploads to work correctly. +Note that Cloudflare decompresses files uploaded with +Content-Encoding: gzip by default which is a deviation from what AWS +does. If this is causing a problem then upload the files with +--header-upload "Cache-Control: no-transform" + Dreamhost Dreamhost DreamObjects is an object storage system based on CEPH. @@ -27867,6 +28871,27 @@ network. For more detailed comparison please check the documentation of the storj backend. +Memory usage {memory} + +The most common cause of rclone using lots of memory is a single +directory with millions of files in. Despite s3 not really having the +concepts of directories, rclone does the sync on a directory by +directory basis to be compatible with normal filing systems. + +Rclone loads each directory into memory as rclone objects. Each rclone +object takes 0.5k-1k of memory, so approximately 1GB per 1,000,000 +files, and the sync for that directory does not begin until it is +entirely loaded in memory. So the sync can take a long time to start for +large directories. + +To sync a directory with 100,000,000 files in you would need +approximately 100 GB of memory. At some point the amount of memory +becomes difficult to provide so there is a workaround for this which +involves a bit of scripting. + +At some point rclone will gain a sync mode which is effectively this +workaround but built in to rclone. + Limitations rclone about is not supported by the S3 backend. Backends without this @@ -28044,12 +29069,13 @@ generating and using an Application Key. Endpoint for the service - leave blank normally. endpoint> Remote config - -------------------- - [remote] - account = 123456789abc - key = 0123456789abcdef0123456789abcdef0123456789 - endpoint = - -------------------- + Configuration complete. + Options: + - type: b2 + - account: 123456789abc + - key: 0123456789abcdef0123456789abcdef0123456789 + - endpoint: + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -28162,11 +29188,18 @@ used. Versions -When rclone uploads a new version of a file it creates a new version of +The default setting of B2 is to keep old versions of files. This means +when rclone uploads a new version of a file it creates a new version of it. Likewise when you delete a file, the old version will be marked -hidden and still be available. Conversely, you may opt in to a "hard -delete" of files with the --b2-hard-delete flag which would permanently -remove the file instead of hiding it. +hidden and still be available. + +Whether B2 keeps old versions of files or not can be adjusted on a per +bucket basis using the "Lifecycle settings" on the B2 control panel or +when creating the bucket using the --b2-lifecycle flag or after creation +using the rclone backend lifecycle command. + +You may opt in to a "hard delete" of files with the --b2-hard-delete +flag which permanently removes files on deletion instead of hiding them. Old versions of files, where available, are visible using the --b2-versions flag. @@ -28802,12 +29835,13 @@ This will guide you through an interactive setup process: Log in and authorize rclone for access Waiting for code... Got code - -------------------- - [remote] - client_id = - client_secret = - token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} - -------------------- + Configuration complete. + Options: + - type: box + - client_id: + - client_secret: + - token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -28888,11 +29922,11 @@ Here is how to do it. Choose a number from below, or type in an existing value 1 > remote remote> remote - -------------------- - [remote] - type = box - token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"} - -------------------- + Configuration complete. + Options: + - type: box + - token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"} + Keep this "remote" remote? Edit remote Value "client_id" = "" Edit? (y/n)> @@ -28920,11 +29954,11 @@ Here is how to do it. Log in and authorize rclone for access Waiting for code... Got code - -------------------- - [remote] - type = box - token = {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"} - -------------------- + Configuration complete. + Options: + - type: box + - token: {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"} + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -30510,12 +31544,12 @@ This will guide you through an interactive setup process: Log in and authorize rclone for access Waiting for code... Got code - -------------------- - [remote] - type = sharefile - endpoint = https://XXX.sharefile.com - token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} - -------------------- + Configuration complete. + Options: + - type: sharefile + - endpoint: https://XXX.sharefile.com + - token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -31816,11 +32850,11 @@ This will guide you through an interactive setup process: "dir=remote:path with space" "dir2=remote2:path with space" Enter a fs.SpaceSepList value. upstreams> images=s3:imagesbucket files=drive:important/files - -------------------- - [remote] - type = combine - upstreams = images=s3:imagesbucket files=drive:important/files - -------------------- + Configuration complete. + Options: + - type: combine + - upstreams: images=s3:imagesbucket files=drive:important/files + Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -31944,12 +32978,13 @@ This will guide you through an interactive setup process: Please visit: https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX - -------------------- - [remote] - app_key = - app_secret = - token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX - -------------------- + Configuration complete. + Options: + - type: dropbox + - app_key: + - app_secret: + - token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -32303,7 +33338,7 @@ Max number of files in upload batch. This sets the batch size of files to upload. It has to be less than 1000. -By default this is 0 which means rclone which calculate the batch size +By default this is 0 which means rclone will calculate the batch size depending on the setting of batch_mode. - batch_mode: async - default batch_size is 100 @@ -32498,12 +33533,12 @@ This will guide you through an interactive setup process: n) No (default) y/n> n Remote config - -------------------- - [remote] - type = filefabric - url = https://yourfabric.smestorage.com/ - permanent_token = xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx - -------------------- + Configuration complete. + Options: + - type: filefabric + - url: https://yourfabric.smestorage.com/ + - permanent_token: xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx + Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -32703,6 +33738,180 @@ Properties: - Type: string - Required: false +Files.com + +Files.com is a cloud storage service that provides a secure and easy way +to store and share files. + +The initial setup for filescom involves authenticating with your +Files.com account. You can do this by providing your site subdomain, +username, and password. Alternatively, you can authenticate using an API +Key from Files.com. rclone config walks you through it. + +Configuration + +Here is an example of how to make a remote called remote. First run: + + rclone config + +This will guide you through an interactive setup process: + + No remotes found, make a new one? + n) New remote + s) Set configuration password + q) Quit config + n/s/q> n + + Enter name for new remote. + name> remote + + Option Storage. + Type of storage to configure. + Choose a number from below, or type in your own value. + [snip] + XX / Files.com + \ "filescom" + [snip] + Storage> filescom + + Option site. + Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com) + Enter a value. Press Enter to leave empty. + site> mysite + + Option username. + The username used to authenticate with Files.com. + Enter a value. Press Enter to leave empty. + username> user + + Option password. + The password used to authenticate with Files.com. + Choose an alternative below. Press Enter for the default (n). + y) Yes, type in my own password + g) Generate random password + n) No, leave this optional password blank (default) + y/g/n> y + Enter the password: + password: + Confirm the password: + password: + + Edit advanced config? + y) Yes + n) No (default) + y/n> n + + Configuration complete. + Options: + - type: filescom + - site: mysite + - username: user + - password: *** ENCRYPTED *** + Keep this "remote" remote? + y) Yes this is OK (default) + e) Edit this remote + d) Delete this remote + y/e/d> y + +Once configured you can use rclone. + +See all files in the top level: + + rclone lsf remote: + +Make a new directory in the root: + + rclone mkdir remote:dir + +Recursively List the contents: + + rclone ls remote: + +Sync /home/local/directory to the remote directory, deleting any excess +files in the directory. + + rclone sync --interactive /home/local/directory remote:dir + +Standard options + +Here are the Standard options specific to filescom (Files.com). + +--filescom-site + +Your site subdomain (e.g. mysite) or custom domain (e.g. +myfiles.customdomain.com). + +Properties: + +- Config: site +- Env Var: RCLONE_FILESCOM_SITE +- Type: string +- Required: false + +--filescom-username + +The username used to authenticate with Files.com. + +Properties: + +- Config: username +- Env Var: RCLONE_FILESCOM_USERNAME +- Type: string +- Required: false + +--filescom-password + +The password used to authenticate with Files.com. + +NB Input to this must be obscured - see rclone obscure. + +Properties: + +- Config: password +- Env Var: RCLONE_FILESCOM_PASSWORD +- Type: string +- Required: false + +Advanced options + +Here are the Advanced options specific to filescom (Files.com). + +--filescom-api-key + +The API key used to authenticate with Files.com. + +Properties: + +- Config: api_key +- Env Var: RCLONE_FILESCOM_API_KEY +- Type: string +- Required: false + +--filescom-encoding + +The encoding for the backend. + +See the encoding section in the overview for more info. + +Properties: + +- Config: encoding +- Env Var: RCLONE_FILESCOM_ENCODING +- Type: Encoding +- Default: + Slash,BackSlash,Del,Ctl,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot + +--filescom-description + +Description of the remote. + +Properties: + +- Config: description +- Env Var: RCLONE_FILESCOM_DESCRIPTION +- Type: string +- Required: false + FTP FTP is the File Transfer Protocol. Rclone FTP support is provided using @@ -32769,12 +33978,12 @@ For an anonymous FTP server, see below. Enter a boolean value (true or false). Press Enter for the default ("false"). explicit_tls> Remote config - -------------------- - [remote] - type = ftp - host = ftp.example.com - pass = *** ENCRYPTED *** - -------------------- + Configuration complete. + Options: + - type: ftp + - host: ftp.example.com + - pass: *** ENCRYPTED *** + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -33208,6 +34417,256 @@ of 1000000000 means that file time precision of 1 second is available. A value of 3153600000000000000 (or another large number) means "unsupported". +Gofile + +Gofile is a content storage and distribution platform. Its aim is to +provide as much service as possible for free or at a very low price. + +The initial setup for Gofile involves logging in to the web interface +and going to the "My Profile" section. Copy the "Account API token" for +use in the config file. + +Note that if you wish to connect rclone to Gofile you will need a +premium account. + +Configuration + +Here is an example of how to make a remote called remote. First run: + + rclone config + +This will guide you through an interactive setup process: + + No remotes found, make a new one? + n) New remote + s) Set configuration password + q) Quit config + n/s/q> n + + Enter name for new remote. + name> remote + + Option Storage. + Type of storage to configure. + Choose a number from below, or type in your own value. + XX / Gofile + \ (gofile) + Storage> gofile + + Option access_token. + API Access token + You can get this from the web control panel. + Enter a value. Press Enter to leave empty. + access_token> YOURACCESSTOKEN + + Edit advanced config? + y) Yes + n) No (default) + y/n> n + + Configuration complete. + Options: + - type: gofile + - access_token: YOURACCESSTOKEN + Keep this "remote" remote? + y) Yes this is OK (default) + e) Edit this remote + d) Delete this remote + y/e/d> y + +Once configured you can then use rclone like this, + +List directories and files in the top level of your Gofile + + rclone lsf remote: + +To copy a local directory to an Gofile directory called backup + + rclone copy /home/source remote:backup + +Modification times and hashes + +Gofile supports modification times with a resolution of 1 second. + +Gofile supports MD5 hashes, so you can use the --checksum flag. + +Restricted filename characters + +In addition to the default restricted characters set the following +characters are also replaced: + + Character Value Replacement + ----------- ------- ------------- + ! 0x21 ！ + " 0x22 ＂ + * 0x2A ＊ + : 0x3A ： + < 0x3C ＜ + > 0x3E ＞ + ? 0x3F ？ + \ 0x5C ＼ + | 0x7C ｜ + +File names can also not start or end with the following characters. +These only get replaced if they are the first or last character in the +name: + + Character Value Replacement + ----------- ------- ------------- + . 0x2E ． + +Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON +strings. + +Public Links + +Gofile supports rclone link to make public links to files or +directories. If you specify a directory it will download as a zip file. +You can use the --expire flag to specify the time the link should be +valid. Note that rclone link --unlink removes all the public links for a +file. + +Root folder ID + +You can set the root_folder_id for rclone. This is the directory +(identified by its Folder ID) that rclone considers to be the root of +your Gofile drive. + +Normally you will leave this blank and rclone will determine the correct +root to use itself and fill in the value in the config file. + +However you can set this to restrict rclone to a specific folder +hierarchy. + +In order to do this you will have to find the Folder ID of the directory +you wish rclone to display. + +You can do this with rclone + + $ rclone lsf -Fip --dirs-only remote: + d6341f53-ee65-4f29-9f59-d11e8070b2a0;Files/ + f4f5c9b8-6ece-478b-b03e-4538edfe5a1c;Photos/ + d50e356c-29ca-4b27-a3a7-494d91026e04;Videos/ + +The ID to use is the part before the ; so you could set + + root_folder_id = d6341f53-ee65-4f29-9f59-d11e8070b2a0 + +To restrict rclone to the Files directory. + +Standard options + +Here are the Standard options specific to gofile (Gofile). + +--gofile-access-token + +API Access token + +You can get this from the web control panel. + +Properties: + +- Config: access_token +- Env Var: RCLONE_GOFILE_ACCESS_TOKEN +- Type: string +- Required: false + +Advanced options + +Here are the Advanced options specific to gofile (Gofile). + +--gofile-root-folder-id + +ID of the root folder + +Leave this blank normally, rclone will fill it in automatically. + +If you want rclone to be restricted to a particular folder you can fill +it in - see the docs for more info. + +Properties: + +- Config: root_folder_id +- Env Var: RCLONE_GOFILE_ROOT_FOLDER_ID +- Type: string +- Required: false + +--gofile-account-id + +Account ID + +Leave this blank normally, rclone will fill it in automatically. + +Properties: + +- Config: account_id +- Env Var: RCLONE_GOFILE_ACCOUNT_ID +- Type: string +- Required: false + +--gofile-list-chunk + +Number of items to list in each call + +Properties: + +- Config: list_chunk +- Env Var: RCLONE_GOFILE_LIST_CHUNK +- Type: int +- Default: 1000 + +--gofile-encoding + +The encoding for the backend. + +See the encoding section in the overview for more info. + +Properties: + +- Config: encoding +- Env Var: RCLONE_GOFILE_ENCODING +- Type: Encoding +- Default: + Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftPeriod,RightPeriod,InvalidUtf8,Dot,Exclamation + +--gofile-description + +Description of the remote. + +Properties: + +- Config: description +- Env Var: RCLONE_GOFILE_DESCRIPTION +- Type: string +- Required: false + +Limitations + +Gofile only supports filenames up to 255 characters in length, where a +character is a unicode character. + +Directories should not be cached for more than 24h otherwise files in +the directory may not be downloadable. In practice this means when using +a VFS based rclone command such as rclone mount you should make sure +--dir-cache-time is less than 24h. + +Note that Gofile is currently limited to a total of 100,000 items. If +you attempt to upload more than that you will get error-limit-100000. +This limit may be lifted in the future. + +Duplicated files + +Gofile is capable of having files with duplicated file names. For +instance two files called hello.txt in the same directory. + +Rclone cannot sync that to a normal file system but it can be fixed with +the rclone dedupe command. + +Duplicated files cause problems with the syncing and you will see +messages in the log about duplicates. + +Use rclone dedupe to fix duplicated files. + Google Cloud Storage Paths are specified as remote:bucket (or remote: for the lsd command.) @@ -33331,16 +34790,16 @@ This will guide you through an interactive setup process: Log in and authorize rclone for access Waiting for code... Got code - -------------------- - [remote] - type = google cloud storage - client_id = - client_secret = - token = {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null} - project_number = 12345678 - object_acl = private - bucket_acl = private - -------------------- + Configuration complete. + Options: + - type: google cloud storage + - client_id: + - client_secret: + - token: {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null} + - project_number: 12345678 + - object_acl: private + - bucket_acl: private + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -33993,15 +35452,16 @@ This will guide you through an interactive setup process: y) Yes n) No y/n> n - -------------------- - [remote] - client_id = - client_secret = - scope = drive - root_folder_id = - service_account_file = - token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"} - -------------------- + Configuration complete. + Options: + type: drive + - client_id: + - client_secret: + - scope: drive + - root_folder_id: + - service_account_file: + - token: {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"} + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -34125,12 +35585,12 @@ credentials file into the rclone config file, you can set service_account_credentials with the actual contents of the file instead, or set the equivalent environment variable. -Use case - Google Apps/G-suite account and individual Drive +Use case - Google Workspace account and individual Drive -Let's say that you are the administrator of a Google Apps (old) or -G-suite account. The goal is to store data on an individual's Drive -account, who IS a member of the domain. We'll call the domain -example.com, and the user foo@example.com. +Let's say that you are the administrator of a Google Workspace. The goal +is to read or write data on an individual's Drive account, who IS a +member of the domain. We'll call the domain example.com, and the user +foo@example.com. There's a few steps we need to go through to accomplish this: @@ -34138,29 +35598,38 @@ There's a few steps we need to go through to accomplish this: - To create a service account and obtain its credentials, go to the Google Developer Console. -- You must have a project - create one if you don't. +- You must have a project - create one if you don't and make sure you + are on the selected project. - Then go to "IAM & admin" -> "Service Accounts". - Use the "Create Service Account" button. Fill in "Service account name" and "Service account ID" with something that identifies your client. - Select "Create And Continue". Step 2 and 3 are optional. -- These credentials are what rclone will use for authentication. If - you ever need to remove access, press the "Delete service account - key" button. +- Click on the newly created service account +- Click "Keys" and then "Add Key" and then "Create new key" +- Choose type "JSON" and click create +- This will download a small JSON file that rclone will use for + authentication. + +If you ever need to remove access, press the "Delete service account +key" button. 2. Allowing API access to example.com Google Drive -- Go to example.com's admin console +- Go to example.com's Workspace Admin Console - Go into "Security" (or use the search bar) -- Select "Show more" and then "Advanced settings" -- Select "Manage API client access" in the "Authentication" section -- In the "Client Name" field enter the service account's "Client ID" - +- Select "Access and data control" and then "API controls" +- Click "Manage domain-wide delegation" +- Click "Add new" +- In the "Client ID" field enter the service account's "Client ID" - this can be found in the Developer Console under "IAM & Admin" -> "Service Accounts", then "View Client ID" for the newly created service account. It is a ~21 character numerical string. -- In the next field, "One or More API Scopes", enter - https://www.googleapis.com/auth/drive to grant access to Google - Drive specifically. +- In the next field, "OAuth Scopes", enter + https://www.googleapis.com/auth/drive to grant read/write access to + Google Drive specifically. You can also use + https://www.googleapis.com/auth/drive.readonly for read only access. +- Click "Authorise" 3. Configure rclone, assuming a new install @@ -34168,12 +35637,12 @@ There's a few steps we need to go through to accomplish this: n/s/q> n # New name>gdrive # Gdrive is an example name - Storage> # Select the number shown for Google Drive + Storage> # Type drive client_id> # Can be left blank client_secret> # Can be left blank - scope> # Select your scope, 1 for example + scope> # Select the scope use used in step 2 root_folder_id> # Can be left blank - service_account_file> /home/foo/myJSONfile.json # This is where the JSON file goes! + service_account_file> /home/foo/myJSONfile.json # Path to the JSON file you downloaded in step 1. y/n> # Auto config, n 4. Verify that it's working @@ -34191,7 +35660,7 @@ Note: in case you configured a specific root folder on gdrive and rclone is unable to access the contents of that folder when using --drive-impersonate, do this instead: - in the gdrive web interface, share your root folder with the user/email of the new Service Account -you created/selected at step #1 - use rclone without specifying the +you created/selected at step 1 - use rclone without specifying the --drive-impersonate option, like this: rclone -v lsf gdrive:backup Shared drives (team drives) @@ -34219,13 +35688,14 @@ For example: 3 / Rclone Test 3 \ "zzzzzzzzzzzzzzzzzzzz" Enter a Shared Drive ID> 1 - -------------------- - [remote] - client_id = - client_secret = - token = {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} - team_drive = xxxxxxxxxxxxxxxxxxxx - -------------------- + Configuration complete. + Options: + - type: drive + - client_id: + - client_secret: + - token: {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} + - team_drive: xxxxxxxxxxxxxxxxxxxx + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -35898,11 +37368,11 @@ This will guide you through an interactive setup process: *** are stored in full resolution at original quality. These uploads *** will count towards storage in your Google Account. - -------------------- - [remote] - type = google photos - token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"} - -------------------- + Configuration complete. + Options: + - type: google photos + - token: {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"} + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -36218,7 +37688,7 @@ Max number of files in upload batch. This sets the batch size of files to upload. It has to be less than 50. -By default this is 0 which means rclone which calculate the batch size +By default this is 0 which means rclone will calculate the batch size depending on the setting of batch_mode. - batch_mode: async - default batch_size is 50 @@ -36694,6 +38164,8 @@ The rclone hashsum (or md5sum or sha1sum) command will: Other operations +- any time a hash is requested, follow the logic from 1-4 from hashsum + above - whenever a file is uploaded or downloaded in full, capture the stream to calculate all supported hashes on the fly and update database @@ -36764,12 +38236,12 @@ This will guide you through an interactive setup process: n) No (default) y/n> n Remote config - -------------------- - [remote] - type = hdfs - namenode = namenode.hadoop:8020 - username = root - -------------------- + Configuration complete. + Options: + - type: hdfs + - namenode: namenode.hadoop:8020 + - username: root + Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -37005,11 +38477,11 @@ This will guide you through an interactive setup process: Log in and authorize rclone for access Waiting for code... Got code - -------------------- - [remote] - type = hidrive - token = {"access_token":"xxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxx","expiry":"xxxxxxxxxxxxxxxxxxxxxxx"} - -------------------- + Configuration complete. + Options: + - type: hidrive + - token: {"access_token":"xxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxx","expiry":"xxxxxxxxxxxxxxxxxxxxxxx"} + Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -37480,10 +38952,11 @@ This will guide you through an interactive setup process: \ "https://example.com" url> https://beta.rclone.org Remote config - -------------------- - [remote] - url = https://beta.rclone.org - -------------------- + Configuration complete. + Options: + - type: http + - url: https://beta.rclone.org + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -38114,12 +39587,12 @@ This will guide you through an interactive setup process. y) Yes n) No (default) y/n> n - -------------------- - [remote] - type = internetarchive - access_key_id = XXXX - secret_access_key = XXXX - -------------------- + Configuration complete. + Options: + - type: internetarchive + - access_key_id: XXXX + - secret_access_key: XXXX + Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -38478,18 +39951,18 @@ This will guide you through an interactive setup process: 2 > Shared 3 > Sync config_mountpoint> 1 - -------------------- - [remote] - type = jottacloud - configVersion = 1 - client_id = jottacli - client_secret = - tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token - token = {........} - username = 2940e57271a93d987d6f8a21 - device = Jotta - mountpoint = Archive - -------------------- + Configuration complete. + Options: + - type: jottacloud + - configVersion: 1 + - client_id: jottacli + - client_secret: + - tokenURL: https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token + - token: {........} + - username: 2940e57271a93d987d6f8a21 + - device: Jotta + - mountpoint: Archive + Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -39366,13 +40839,13 @@ This will guide you through an interactive setup process: n) No y/n> n Remote config - -------------------- - [remote] - type = mailru - user = username@mail.ru - pass = *** ENCRYPTED *** - speedup_enable = true - -------------------- + Configuration complete. + Options: + - type: mailru + - user: username@mail.ru + - pass: *** ENCRYPTED *** + - speedup_enable: true + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -39759,12 +41232,12 @@ This will guide you through an interactive setup process: Confirm the password: password: Remote config - -------------------- - [remote] - type = mega - user = you@example.com - pass = *** ENCRYPTED *** - -------------------- + Configuration complete. + Options: + - type: mega + - user: you@example.com + - pass: *** ENCRYPTED *** + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -40042,10 +41515,10 @@ want to: Remote config - -------------------- - [remote] - type = memory - -------------------- + Configuration complete. + Options: + - type: memory + Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -40449,12 +41922,13 @@ This will guide you through an interactive setup process: Endpoint for the service - leave blank normally. endpoint> Remote config - -------------------- - [remote] - account = account_name - key = base64encodedkey== - endpoint = - -------------------- + Configuration complete. + Options: + - type: azureblob + - account: account_name + - key: base64encodedkey== + - endpoint: + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -40717,6 +42191,13 @@ msi_client_id, or msi_mi_res_id parameters. If none of msi_object_id, msi_client_id, or msi_mi_res_id is set, this is is equivalent to using env_auth. +Anonymous + +If you want to access resources with public anonymous access then set +account only. You can do this without making an rclone config: + + rclone lsf :azureblob,account=ACCOUNT:CONTAINER + Standard options Here are the Standard options specific to azureblob (Microsoft Azure @@ -42112,13 +43593,13 @@ This will guide you through an interactive setup process: y) Yes n) No y/n> y - -------------------- - [remote] - type = onedrive - token = {"access_token":"youraccesstoken","token_type":"Bearer","refresh_token":"yourrefreshtoken","expiry":"2018-08-26T22:39:52.486512262+08:00"} - drive_id = b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk - drive_type = business - -------------------- + Configuration complete. + Options: + - type: onedrive + - token: {"access_token":"youraccesstoken","token_type":"Bearer","refresh_token":"yourrefreshtoken","expiry":"2018-08-26T22:39:52.486512262+08:00"} + - drive_id: b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk + - drive_type: business + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -43273,11 +44754,12 @@ This will guide you through an interactive setup process: password: Confirm the password: password: - -------------------- - [remote] - username = - password = *** ENCRYPTED *** - -------------------- + Configuration complete. + Options: + - type: opendrive + - username: + - password: *** ENCRYPTED *** + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -44341,15 +45823,16 @@ This will guide you through an interactive setup process. Leave blank will use the default value "3". connection_retries> Remote config - -------------------- - [remote] - env_auth = false - access_key_id = access_key - secret_access_key = secret_key - endpoint = - zone = pek3a - connection_retries = - -------------------- + Configuration complete. + Options: + - type: qingstor + - env_auth: false + - access_key_id: access_key + - secret_access_key: secret_key + - endpoint: + - zone: pek3a + - connection_retries: + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -44653,11 +46136,12 @@ This will guide you through an interactive setup process: Host name of Quatrix account. host> example.quatrix.it - -------------------- - [remote] - api_key = your_api_key - host = example.quatrix.it - -------------------- + Configuration complete. + Options: + - type: quatrix + - api_key: your_api_key + - host: example.quatrix.it + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -44703,12 +46187,12 @@ hostname was changed. Choose a number from below, or type in an existing value 1 > remote remote> remote - -------------------- - [remote] - type = quatrix - host = some_host.quatrix.it - api_key = your_api_key - -------------------- + Configuration complete. + Options: + - type: quatrix + - host: some_host.quatrix.it + - api_key: your_api_key + Keep this "remote" remote? Edit remote Option api_key. API key for accessing Quatrix account @@ -44718,12 +46202,12 @@ hostname was changed. Host name of Quatrix account Enter a string value. Press Enter for the default (some_host.quatrix.it). - -------------------- - [remote] - type = quatrix - host = some_host.quatrix.it - api_key = your_api_key - -------------------- + Configuration complete. + Options: + - type: quatrix + - host: some_host.quatrix.it + - api_key: your_api_key + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -45595,6 +47079,49 @@ Properties: - Type: bool - Default: false +--swift-fetch-until-empty-page + +When paginating, always fetch unless we received an empty page. + +Consider using this option if rclone listings show fewer objects than +expected, or if repeated syncs copy unchanged objects. + +It is safe to enable this, but rclone may make more API calls than +necessary. + +This is one of a pair of workarounds to handle implementations of the +Swift API that do not implement pagination as expected. See also +"partial_page_fetch_threshold". + +Properties: + +- Config: fetch_until_empty_page +- Env Var: RCLONE_SWIFT_FETCH_UNTIL_EMPTY_PAGE +- Type: bool +- Default: false + +--swift-partial-page-fetch-threshold + +When paginating, fetch if the current page is within this percentage of +the limit. + +Consider using this option if rclone listings show fewer objects than +expected, or if repeated syncs copy unchanged objects. + +It is safe to enable this, but rclone may make more API calls than +necessary. + +This is one of a pair of workarounds to handle implementations of the +Swift API that do not implement pagination as expected. See also +"fetch_until_empty_page". + +Properties: + +- Config: partial_page_fetch_threshold +- Env Var: RCLONE_SWIFT_PARTIAL_PAGE_FETCH_THRESHOLD +- Type: int +- Default: 0 + --swift-chunk-size Above this size files will be chunked. @@ -45815,12 +47342,13 @@ This will guide you through an interactive setup process: Log in and authorize rclone for access Waiting for code... Got code - -------------------- - [remote] - client_id = - client_secret = - token = {"access_token":"XXX","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"} - -------------------- + Configuration complete. + Options: + - type: pcloud + - client_id: + - client_secret: + - token: {"access_token":"XXX","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"} + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -46423,6 +47951,190 @@ Deleted files still visible with trashed-only Deleted files will still be visible with --pikpak-trashed-only even after the trash emptied. This goes away after few days. +Pixeldrain + +This is the backend for Pixeldrain's premium filesystem feature. This is +not the same as pixeldrain's free file sharing product. The filesystem +requires either a Pro subscription or the Prepaid plan. More information +on subscriptions. + +An overview of the filesystem's features and limitations is available in +the filesystem guide on pixeldrain. + +Usage with account + +To use the personal filesystem you will need a pixeldrain account and +either the Prepaid plan or one of the Patreon-based subscriptions. After +registering and subscribing, your personal filesystem will be available +at this link: https://pixeldrain.com/d/me. + +Go to the API keys page on your account and generate a new API key for +rclone. Then run rclone config and use the API key to create a new +backend. + +Example: + + No remotes found, make a new one? + n) New remote + d) Delete remote + c) Copy remote + s) Set configuration password + q) Quit config + n/d/c/s/q> n + + Enter name for new remote. + name> pixeldrainfs + + Option Storage. + Type of storage to configure. + Choose a number from below, or type in your own value. + ... + XX / Pixeldrain Filesystem + \ (pixeldrain) + ... + Storage> pixeldrain + + Option api_key. + API key for your pixeldrain account. + Found on https://pixeldrain.com/user/api_keys. + Enter a value. Press Enter to leave empty. + api_key> b1bb1e81-9b7b-406b-986a-c9b20be76e15 + + Option directory_id. + Root of the filesystem to use. Set to 'me' to use your personal filesystem. + Set to a shared directory ID to use a shared directory. + Enter a string value. Press Enter for the default (me). + directory_id> + + Edit advanced config? + y) Yes + n) No (default) + y/n> + + Configuration complete. + Options: + - type: pixeldrain + - api_key: b1bb1e81-9b7b-406b-986a-c9b20be76e15 + Keep this "pixeldrainfs" remote? + y) Yes this is OK (default) + e) Edit this remote + d) Delete this remote + y/e/d> + + Current remotes: + + Name Type + ==== ==== + pixeldrainfs pixeldrain + + e) Edit existing remote + n) New remote + d) Delete remote + r) Rename remote + c) Copy remote + s) Set configuration password + q) Quit config + e/n/d/r/c/s/q> q + +Usage without account + +It is possible to gain read-only access to publicly shared directories +through rclone. For this you only need a directory ID. The directory ID +can be found in the URL of a shared directory, the URL will look like +this https://pixeldrain.com/d/abcd1234 where abcd1234 is the directory +ID. Directory IDs in your own filesystem can also be listed with the lsf +command: + +rclone lsf Pixeldrain: --dirs-only -Fpi + +This will print directories in your Pixeldrain home directory and their +public IDs. + +Enter this directory ID in the rclone config and you will be able to +access the directory. + +Standard options + +Here are the Standard options specific to pixeldrain (Pixeldrain +Filesystem). + +--pixeldrain-api-key + +API key for your pixeldrain account. Found on +https://pixeldrain.com/user/api_keys. + +Properties: + +- Config: api_key +- Env Var: RCLONE_PIXELDRAIN_API_KEY +- Type: string +- Required: false + +--pixeldrain-root-folder-id + +Root of the filesystem to use. + +Set to 'me' to use your personal filesystem. Set to a shared directory +ID to use a shared directory. + +Properties: + +- Config: root_folder_id +- Env Var: RCLONE_PIXELDRAIN_ROOT_FOLDER_ID +- Type: string +- Default: "me" + +Advanced options + +Here are the Advanced options specific to pixeldrain (Pixeldrain +Filesystem). + +--pixeldrain-api-url + +The API endpoint to connect to. In the vast majority of cases it's fine +to leave this at default. It is only intended to be changed for testing +purposes. + +Properties: + +- Config: api_url +- Env Var: RCLONE_PIXELDRAIN_API_URL +- Type: string +- Default: "https://pixeldrain.com/api" + +--pixeldrain-description + +Description of the remote. + +Properties: + +- Config: description +- Env Var: RCLONE_PIXELDRAIN_DESCRIPTION +- Type: string +- Required: false + +Metadata + +Pixeldrain supports file modes and creation times. + +Here are the possible system metadata items for the pixeldrain backend. + + ------------------------------------------------------------------------------------------------- + Name Help Type Example Read Only + ----------- -------------- ----------- ------------------------------------- -------------------- + btime Time of file RFC 3339 2006-01-02T15:04:05.999999999Z07:00 N + birth + (creation) + + mode File mode octal, unix 755 N + style + + mtime Time of last RFC 3339 2006-01-02T15:04:05.999999999Z07:00 N + modification + ------------------------------------------------------------------------------------------------- + +See the metadata docs for more info. + premiumize.me Paths are specified as remote:path @@ -46469,11 +48181,11 @@ This will guide you through an interactive setup process: Log in and authorize rclone for access Waiting for code... Got code - -------------------- - [remote] - type = premiumizeme - token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2029-08-07T18:44:15.548915378+01:00"} - -------------------- + Configuration complete. + Options: + - type: premiumizeme + - token: {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2029-08-07T18:44:15.548915378+01:00"} + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -46699,12 +48411,12 @@ This will guide you through an interactive setup process: Enter a value. Press Enter to leave empty. 2fa> 123456 Remote config - -------------------- - [remote] - type = protondrive - user = you@protonmail.com - pass = *** ENCRYPTED *** - -------------------- + Configuration complete. + Options: + - type: protondrive + - user: you@protonmail.com + - pass: *** ENCRYPTED *** + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -47267,12 +48979,12 @@ This will guide you through an interactive setup process: Enter a value. Press Enter to leave empty. 2fa> 123456 Remote config - -------------------- - [remote] - type = protondrive - user = you@protonmail.com - pass = *** ENCRYPTED *** - -------------------- + Configuration complete. + Options: + - type: protondrive + - user: you@protonmail.com + - pass: *** ENCRYPTED *** + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -48023,14 +49735,15 @@ This will guide you through an interactive setup process. Path to unencrypted PEM-encoded private key file, leave blank to use ssh-agent. key_file> Remote config - -------------------- - [remote] - host = example.com - user = sftpuser - port = - pass = - key_file = - -------------------- + Configuration complete. + Options: + - type: sftp + - host: example.com + - user: sftpuser + - port: + - pass: + - key_file: + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -48386,7 +50099,16 @@ Properties: Raw PEM-encoded private key. -If specified, will override key_file parameter. +Note that this should be on a single line with line endings replaced +with '', eg + + key_pem = -----BEGIN RSA PRIVATE KEY-----\nMaMbaIXtE\n0gAMbMbaSsd\nMbaass\n-----END RSA PRIVATE KEY----- + +This will generate the single line correctly: + + awk '{printf "%s\\n", $0}' < ~/.ssh/id_rsa + +If specified, it will override the key_file parameter. Properties: @@ -48849,13 +50571,13 @@ Maximum number of SFTP simultaneous connections, 0 for unlimited. Note that setting this is very likely to cause deadlocks so it should be used with care. -If you are doing a sync or copy then make sure concurrency is one more +If you are doing a sync or copy then make sure connections is one more than the sum of --transfers and --checkers. If you use --check-first then it just needs to be one more than the maximum of --checkers and --transfers. -So for concurrency 3 you'd use --checkers 2 --transfers 2 --check-first +So for connections 3 you'd use --checkers 2 --transfers 2 --check-first or --checkers 1 --transfers 1. Properties: @@ -49459,11 +51181,11 @@ Setup with access grant Enter a string value. Press Enter for the default (""). access_grant> your-access-grant-received-by-someone-else Remote config - -------------------- - [remote] - type = storj - access_grant = your-access-grant-received-by-someone-else - -------------------- + Configuration complete. + Options: + - type: storj + - access_grant: your-access-grant-received-by-someone-else + Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -49512,14 +51234,14 @@ Setup with API key and passphrase Enter a string value. Press Enter for the default (""). passphrase> your-human-readable-encryption-passphrase Remote config - -------------------- - [remote] - type = storj - satellite_address = 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777 - api_key = your-api-key-for-your-storj-project - passphrase = your-human-readable-encryption-passphrase - access_grant = the-access-grant-generated-from-the-api-key-and-passphrase - -------------------- + Configuration complete. + Options: + - type: storj + - satellite_address: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777 + - api_key: your-api-key-for-your-storj-project + - passphrase: your-human-readable-encryption-passphrase + - access_grant: the-access-grant-generated-from-the-api-key-and-passphrase + Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -49825,11 +51547,11 @@ This will guide you through an interactive setup process: Username (email address)> nick@craig-wood.com Your Sugarsync password is only required during setup and will not be stored. password: - -------------------- - [remote] - type = sugarsync - refresh_token = https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX - -------------------- + Configuration complete. + Options: + - type: sugarsync + - refresh_token: https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX + Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -50049,11 +51771,6 @@ policy mfs (most free space) as a member of an rclone union remote. See List of backends that do not support rclone about and rclone about -Tardigrade - -The Tardigrade backend has been renamed to be the Storj backend. Old -configuration files will continue to work. - Uloz.to Paths are specified as remote:path @@ -50544,11 +52261,11 @@ This will guide you through an interactive setup process: Enter a signed integer. Press Enter for the default ("120"). cache_time> Remote config - -------------------- - [remote] - type = union - upstreams = remote1:dir1 remote2:dir2 remote3:dir3 - -------------------- + Configuration complete. + Options: + - type: union + - upstreams: remote1:dir1 remote2:dir2 remote3:dir3 + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -50920,15 +52637,15 @@ This will guide you through an interactive setup process: Bearer token instead of user/pass (e.g. a Macaroon) bearer_token> Remote config - -------------------- - [remote] - type = webdav - url = https://example.com/remote.php/webdav/ - vendor = nextcloud - user = user - pass = *** ENCRYPTED *** - bearer_token = - -------------------- + Configuration complete. + Options: + - type: webdav + - url: https://example.com/remote.php/webdav/ + - vendor: nextcloud + - user: user + - pass: *** ENCRYPTED *** + - bearer_token: + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -51147,6 +52864,18 @@ Properties: - Type: bool - Default: false +--webdav-unix-socket + +Path to a unix domain socket to dial to, instead of opening a TCP +connection directly + +Properties: + +- Config: unix_socket +- Env Var: RCLONE_WEBDAV_UNIX_SOCKET +- Type: string +- Required: false + --webdav-description Description of the remote. @@ -51379,12 +53108,13 @@ This will guide you through an interactive setup process: Log in and authorize rclone for access Waiting for code... Got code - -------------------- - [remote] - client_id = - client_secret = - token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"OAuth","expiry":"2016-12-29T12:27:11.362788025Z"} - -------------------- + Configuration complete. + Options: + - type: yandex + - client_id: + - client_secret: + - token: {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"OAuth","expiry":"2016-12-29T12:27:11.362788025Z"} + Keep this "remote" remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -51541,6 +53271,18 @@ Properties: - Type: Encoding - Default: Slash,Del,Ctl,InvalidUtf8,Dot +--yandex-spoof-ua + +Set the user agent to match an official version of the yandex disk +client. May help with upload performance. + +Properties: + +- Config: spoof_ua +- Env Var: RCLONE_YANDEX_SPOOF_UA +- Type: bool +- Default: true + --yandex-description Description of the remote. @@ -51629,12 +53371,12 @@ This will guide you through an interactive setup process: 1 / General \ "4u2869d2aa6fca04f4f2f896b6539243b85b1" Enter a Workspace ID> 1 - -------------------- - [remote] - type = zoho - token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Zoho-oauthtoken","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2020-10-12T00:54:52.370275223+02:00"} - root_folder_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx - -------------------- + Configuration complete. + Options: + - type: zoho + - token: {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Zoho-oauthtoken","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2020-10-12T00:54:52.370275223+02:00"} + - root_folder_id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -51952,11 +53694,9 @@ requires absolute paths and the use of prefix \\?\, e.g. \\?\D:\some\very\long\path. For convenience rclone will automatically convert regular paths into the corresponding extended-length paths, so in most cases you do not have to worry about this (read more below). - -Note that Windows supports using the same prefix \\?\ to specify path to +Using the same prefix \\?\ it is also possible to specify path to volumes identified by their GUID, e.g. -\\?\Volume{b75e2c83-0000-0000-0000-602f00000000}\some\path. This is not -supported in rclone, due to an issue in go. +\\?\Volume{b75e2c83-0000-0000-0000-602f00000000}\some\path. Long paths @@ -52310,6 +54050,34 @@ Properties: - Type: bool - Default: false +--local-no-clone + +Disable reflink cloning for server-side copies. + +Normally, for local-to-local transfers, rclone will "clone" the file +when possible, and fall back to "copying" only when cloning is not +supported. + +Cloning creates a shallow copy (or "reflink") which initially shares +blocks with the original file. Unlike a "hardlink", the two files are +independent and neither will affect the other if subsequently modified. + +Cloning is usually preferable to copying, as it is much faster and is +deduplicated by default (i.e. having two identical files does not +consume more storage than having just one.) However, for use cases where +data redundancy is preferable, --local-no-clone can be used to disable +cloning and force "deep" copies. + +Currently, cloning is only supported when using APFS on macOS (support +for other platforms may be added in the future.) + +Properties: + +- Config: no_clone +- Env Var: RCLONE_LOCAL_NO_CLONE +- Type: bool +- Default: false + --local-no-preallocate Disable preallocation of disk space for transferred files. @@ -52497,6 +54265,172 @@ Options: Changelog +v1.68.0 - 2024-09-08 + +See commits + +- New backends + - Files.com (Sam Harrison) + - Gofile (Nick Craig-Wood) + - Pixeldrain (Fornax) +- Changed backends + - S3 backend updated to use AWS SDKv2 as v1 is now unsupported. + - The matrix of providers and auth methods is huge and there + could be problems with obscure combinations. + - Please report problems in a new issue on Github. +- New commands + - config encryption: set, remove and check to manage config file + encryption (Nick Craig-Wood) +- New Features + - build + - Update to go1.23 and make go1.21 the minimum required + version (Nick Craig-Wood) + - Update all dependencies (Nick Craig-Wood) + - Disable wasm/js build due to go bug #64856 (Nick Craig-Wood) + - Enable custom linting rules with ruleguard via gocritic + (albertony) + - Update logging statements to make --use-json-log work always + (albertony) + - Adding new code quality tests and fixing the fallout + (albertony) + - config + - Internal config re-organised to be more consistent and make + it available from the rc (Nick Craig-Wood) + - Avoid remotes with empty names from the environment + (albertony) + - Make listing of remotes more consistent (albertony) + - Make getting config values more consistent (albertony) + - Use --password-command to set config file password if + supplied (Nick Craig-Wood) + - doc fixes (albertony, crystalstall, David Seifert, Eng Zer Jun, + Ernie Hershey, Florian Klink, John Oxley, kapitainsky, Mathieu + Moreau, Nick Craig-Wood, nipil, Pétr Bozsó, Russ Bubley, Sam + Harrison, Thearas, URenko, Will Miles, yuval-cloudinary) + - fs: Allow semicolons as well as spaces in --bwlimit timetable + parsing (Kyle Reynolds) + - help + - Global flags help command now takes glob filter (albertony) + - Make help command output less distracting (albertony) + - lib/encoder: Add Raw encoding for use where no encoding at all + is required, eg --local-encoding Raw (URenko) + - listremotes: Added options for filtering, ordering and json + output (albertony) + - nfsmount + - Make the --sudo flag work for umount as well as mount (Nick + Craig-Wood) + - Add -o tcp option to NFS mount options to fix mounting under + Linux (Nick Craig-Wood) + - operations: copy: generate stable partial suffix (Georg Welzel) + - rc + - Add options/info call to enumerate options (Nick Craig-Wood) + - Add option blocks parameter to options/get and options/info + (Nick Craig-Wood) + - Add vfs/queue to show the status of the upload queue (Nick + Craig-Wood) + - Add vfs/queue-set-expiry to adjust expiry of items in the + VFS queue (Nick Craig-Wood) + - Add --unix-socket option to rc command (Florian Klink) + - Prevent unmount rc command from sending a STOPPING=1 + sd-notify message (AThePeanut4) + - rcserver: Implement prometheus metrics on a dedicated port (Oleg + Kunitsyn) + - serve dlna + - Also look at "Subs" subdirectory (Florian Klink) + - Don't swallow video.{idx,sub} (Florian Klink) + - Set more correct mime type (Florian Klink) + - serve nfs + - Implement on disk cache for file handles selected with + --nfs-cache-type (Nick Craig-Wood) + - Add tracing to filesystem calls (Nick Craig-Wood) + - Mask unimplemented error from chmod (Nick Craig-Wood) + - Unify the nfs library logging with rclone's logging better + (Nick Craig-Wood) + - Fix incorrect user id and group id exported to NFS (Nick + Craig-Wood) + - serve s3 + - Implement --auth-proxy (Sawjan Gurung) + - Update to AWS SDKv2 by updating github.com/rclone/gofakes3 + (Nick Craig-Wood) +- Bug Fixes + - bisync: Fix sync time problems with backends that round time (eg + Dropbox) (nielash) + - serve dlna: Fix panic: invalid argument to Int63n (Nick + Craig-Wood) +- VFS + - Add --vfs-read-chunk-streams to parallel read chunks from files + (Nick Craig-Wood) + - This can increase mount performance on high bandwidth or + large latency links + - Fix cache encoding with special characters (URenko) +- Local + - Fix encoding of root path fix (URenko) + - Add server-side copy (using clone) with xattrs on macOS + (nielash) + - --local-no-clone flag to disable cloning for server-side + copies (nielash) + - Support setting custom --metadata during server-side Copy + (nielash) +- Azure Blob + - Allow anonymous access for public resources (Nick Craig-Wood) +- B2 + - Include custom upload headers in large file info (Pat Patterson) +- Drive + - Fix copying Google Docs to a backend which only supports SHA1 + (Nick Craig-Wood) +- Fichier + - Fix detection of Flood Detected error (Nick Craig-Wood) + - Fix server side move (Nick Craig-Wood) +- HTTP + - Reload client certificates on expiry (Saleh Dindar) + - Support listening on passed FDs (Florian Klink) +- Jottacloud + - Fix setting of metadata on server side move (albertony) +- Onedrive + - Fix nil pointer error when uploading small files (Nick + Craig-Wood) +- Pcloud + - Implement SetModTime (Georg Welzel) + - Implement OpenWriterAt feature to enable multipart uploads + (Georg Welzel) +- Pikpak + - Improve data consistency by ensuring async tasks complete + (wiserain) + - Implement custom hash to replace wrong sha1 (wiserain) + - Fix error with copyto command (wiserain) + - Optimize file move by removing unnecessary readMetaData() call + (wiserain) + - Non-buffered hash calculation for local source files (wiserain) + - Optimize upload by pre-fetching gcid from API (wiserain) + - Correct file transfer progress for uploads by hash (wiserain) + - Update to using AWS SDK v2 (wiserain) +- S3 + - Update to using AWS SDK v2 (Nick Craig-Wood) + - Add --s3-sdk-log-mode to control SDKv2 debugging (Nick + Craig-Wood) + - Fix incorrect region for Magalu provider (Filipe Herculano) + - Allow restoring from intelligent-tiering storage class (Pawel + Palucha) +- SFTP + - Use uint32 for mtime to save memory (Tomasz Melcer) + - Ignore useless errors when closing the connection pool (Nick + Craig-Wood) + - Support listening on passed FDs (Florian Klink) +- Swift + - Add workarounds for bad listings in Ceph RGW (Paul Collins) + - Add total/free space info in about command. + (fsantagostinobietti) +- Ulozto + - Fix upload of > 2GB files on 32 bit platforms (Tobias Markus) +- WebDAV + - Add --webdav-unix-socket-path to connect to a unix socket + (Florian Klink) +- Yandex + - Implement custom user agent to help with upload speeds + (Sebastian Bünger) +- Zoho + - Fix inefficiencies uploading with new API to avoid throttling + (Nick Craig-Wood) + v1.67.0 - 2024-06-14 See commits @@ -59765,9 +61699,10 @@ garbage collector work harder, reducing memory size at the expense of CPU usage. The most common cause of rclone using lots of memory is a single -directory with thousands or millions of files in. Rclone has to load -this entirely into memory as rclone objects. Each rclone object takes -0.5k-1k of memory. +directory with millions of files in. Rclone has to load this entirely +into memory as rclone objects. Each rclone object takes 0.5k-1k of +memory. There is a workaround for this which involves a bit of +scripting. Rclone changes fullwidth Unicode punctuation marks in file names @@ -60680,6 +62615,31 @@ email addresses removed from here need to be added to bin/.ignore-emails to make - Michał Dzienisiewicz michal.piotr.dz@gmail.com - Florian Klink flokli@flokli.de - Bill Fraser bill@wfraser.dev +- Thearas thearas850@gmail.com +- Filipe Herculano fifo_@live.com +- Russ Bubley russ.bubley@googlemail.com +- Paul Collins paul.collins@canonical.com +- Tomasz Melcer liori@exroot.org +- itsHenry 2671230065@qq.com +- Ke Wang me@ke.wang +- AThePeanut4 49614525+AThePeanut4@users.noreply.github.com +- Tobias Markus tobbi.bugs@googlemail.com +- Ernie Hershey github@ernie.org +- Will Miles wmiles@sgl.com +- David Seifert 16636962+SoapGentoo@users.noreply.github.com +- Fornax wimbrand96@gmail.com +- Sam Harrison sam.harrison@files.com +- Péter Bozsó 3806723+peterbozso@users.noreply.github.com +- Georg Welzel gwelzel@mailbox.org +- John Oxley john.oxley@gmail.com joxley@meta.com +- Pawel Palucha pawel.palucha@aetion.com +- crystalstall crystalruby@qq.com +- nipil nipil@users.noreply.github.com +- yuval-cloudinary 46710068+yuval-cloudinary@users.noreply.github.com +- Mathieu Moreau mrx23dot@users.noreply.github.com +- fsantagostinobietti + 6057026+fsantagostinobietti@users.noreply.github.com +- Oleg Kunitsyn 114359669+hiddenmarten@users.noreply.github.com Contact the rclone project diff --git a/bin/make_backend_docs.py b/bin/make_backend_docs.py index 1871cc68b..07da64463 100755 --- a/bin/make_backend_docs.py +++ b/bin/make_backend_docs.py @@ -32,6 +32,9 @@ def alter_doc(backend): """Alter the documentation for backend""" rclone_bin_dir = Path(sys.path[0]).parent.absolute() doc_file = "docs/content/"+backend+".md" + doc_file2 = "docs/content/"+backend+"/_index.md" + if not os.path.exists(doc_file) and os.path.exists(doc_file2): + doc_file = doc_file2 if not os.path.exists(doc_file): raise ValueError("Didn't find doc file %s" % (doc_file,)) new_file = doc_file+"~new~" diff --git a/bin/make_manual.py b/bin/make_manual.py index 922bd5688..c28398dfd 100755 --- a/bin/make_manual.py +++ b/bin/make_manual.py @@ -64,7 +64,7 @@ docs = [ "azurefiles.md", "onedrive.md", "opendrive.md", - "oracleobjectstorage.md", + "oracleobjectstorage/_index.md", "qingstor.md", "quatrix.md", "sia.md", @@ -81,7 +81,6 @@ docs = [ "smb.md", "storj.md", "sugarsync.md", - "tardigrade.md", # stub only to redirect to storj.md "ulozto.md", "uptobox.md", "union.md", @@ -159,6 +158,7 @@ def read_doc(doc): def check_docs(docpath): """Check all the docs are in docpath""" files = set(f for f in os.listdir(docpath) if f.endswith(".md")) + files.update(f for f in docs if os.path.exists(os.path.join(docpath,f))) files -= set(ignore_docs) docs_set = set(docs) if files == docs_set: diff --git a/docs/content/changelog.md b/docs/content/changelog.md index 094baf605..4deee4aa3 100644 --- a/docs/content/changelog.md +++ b/docs/content/changelog.md @@ -5,6 +5,127 @@ description: "Rclone Changelog" # Changelog +## v1.68.0 - 2024-09-08 + +[See commits](https://github.com/rclone/rclone/compare/v1.67.0...v1.68.0) + +* New backends + * [Files.com](/filescom) (Sam Harrison) + * [Gofile](/gofile/) (Nick Craig-Wood) + * [Pixeldrain](/pixeldrain/) (Fornax) +* Changed backends + * [S3](/s3/) backend updated to use [AWS SDKv2](https://github.com/aws/aws-sdk-go-v2) as v1 is now unsupported. + * The matrix of providers and auth methods is huge and there could be problems with obscure combinations. + * Please report problems in a [new issue](https://github.com/rclone/rclone/issues/new/choose) on Github. +* New commands + * [config encryption](/commands/rclone_config_encryption/): set, remove and check to manage config file encryption (Nick Craig-Wood) +* New Features + * build + * Update to go1.23 and make go1.21 the minimum required version (Nick Craig-Wood) + * Update all dependencies (Nick Craig-Wood) + * Disable wasm/js build due to [go bug #64856](https://github.com/golang/go/issues/64856) (Nick Craig-Wood) + * Enable custom linting rules with ruleguard via gocritic (albertony) + * Update logging statements to make `--use-json-log` work always (albertony) + * Adding new code quality tests and fixing the fallout (albertony) + * config + * Internal config re-organised to be more consistent and make it available from the rc (Nick Craig-Wood) + * Avoid remotes with empty names from the environment (albertony) + * Make listing of remotes more consistent (albertony) + * Make getting config values more consistent (albertony) + * Use `--password-command` to set config file password if supplied (Nick Craig-Wood) + * doc fixes (albertony, crystalstall, David Seifert, Eng Zer Jun, Ernie Hershey, Florian Klink, John Oxley, kapitainsky, Mathieu Moreau, Nick Craig-Wood, nipil, Pétr Bozsó, Russ Bubley, Sam Harrison, Thearas, URenko, Will Miles, yuval-cloudinary) + * fs: Allow semicolons as well as spaces in `--bwlimit` timetable parsing (Kyle Reynolds) + * help + * Global flags help command now takes glob filter (albertony) + * Make help command output less distracting (albertony) + * lib/encoder: Add Raw encoding for use where no encoding at all is required, eg `--local-encoding Raw` (URenko) + * listremotes: Added options for filtering, ordering and json output (albertony) + * nfsmount + * Make the `--sudo` flag work for umount as well as mount (Nick Craig-Wood) + * Add `-o tcp` option to NFS mount options to fix mounting under Linux (Nick Craig-Wood) + * operations: copy: generate stable partial suffix (Georg Welzel) + * rc + * Add [options/info](/rc/#options-info) call to enumerate options (Nick Craig-Wood) + * Add option blocks parameter to [options/get](/rc/#options-get) and [options/info](/rc/#options-info) (Nick Craig-Wood) + * Add [vfs/queue](/rc/#vfs-queue) to show the status of the upload queue (Nick Craig-Wood) + * Add [vfs/queue-set-expiry](/rc/#vfs-queue-set-expiry) to adjust expiry of items in the VFS queue (Nick Craig-Wood) + * Add `--unix-socket` option to `rc` command (Florian Klink) + * Prevent unmount rc command from sending a `STOPPING=1` sd-notify message (AThePeanut4) + * rcserver: Implement [prometheus metrics](/docs/#metrics) on a dedicated port (Oleg Kunitsyn) + * serve dlna + * Also look at "Subs" subdirectory (Florian Klink) + * Don't swallow `video.{idx,sub}` (Florian Klink) + * Set more correct mime type (Florian Klink) + * serve nfs + * Implement on disk cache for file handles selected with `--nfs-cache-type` (Nick Craig-Wood) + * Add tracing to filesystem calls (Nick Craig-Wood) + * Mask unimplemented error from chmod (Nick Craig-Wood) + * Unify the nfs library logging with rclone's logging better (Nick Craig-Wood) + * Fix incorrect user id and group id exported to NFS (Nick Craig-Wood) + * serve s3 + * Implement `--auth-proxy` (Sawjan Gurung) + * Update to AWS SDKv2 by updating `github.com/rclone/gofakes3` (Nick Craig-Wood) +* Bug Fixes + * bisync: Fix sync time problems with backends that round time (eg Dropbox) (nielash) + * serve dlna: Fix panic: invalid argument to Int63n (Nick Craig-Wood) +* VFS + * Add [--vfs-read-chunk-streams](/commands/rclone_mount/#vfs-read-chunk-streams-0-1) to parallel read chunks from files (Nick Craig-Wood) + * This can increase mount performance on high bandwidth or large latency links + * Fix cache encoding with special characters (URenko) +* Local + * Fix encoding of root path fix (URenko) + * Add server-side copy (using clone) with xattrs on macOS (nielash) + * `--local-no-clone` flag to disable cloning for server-side copies (nielash) + * Support setting custom `--metadata` during server-side Copy (nielash) +* Azure Blob + * Allow anonymous access for public resources (Nick Craig-Wood) +* B2 + * Include custom upload headers in large file info (Pat Patterson) +* Drive + * Fix copying Google Docs to a backend which only supports SHA1 (Nick Craig-Wood) +* Fichier + * Fix detection of Flood Detected error (Nick Craig-Wood) + * Fix server side move (Nick Craig-Wood) +* HTTP + * Reload client certificates on expiry (Saleh Dindar) + * Support listening on passed FDs (Florian Klink) +* Jottacloud + * Fix setting of metadata on server side move (albertony) +* Onedrive + * Fix nil pointer error when uploading small files (Nick Craig-Wood) +* Pcloud + * Implement `SetModTime` (Georg Welzel) + * Implement `OpenWriterAt` feature to enable multipart uploads (Georg Welzel) +* Pikpak + * Improve data consistency by ensuring async tasks complete (wiserain) + * Implement custom hash to replace wrong sha1 (wiserain) + * Fix error with `copyto` command (wiserain) + * Optimize file move by removing unnecessary `readMetaData()` call (wiserain) + * Non-buffered hash calculation for local source files (wiserain) + * Optimize upload by pre-fetching gcid from API (wiserain) + * Correct file transfer progress for uploads by hash (wiserain) + * Update to using AWS SDK v2 (wiserain) +* S3 + * Update to using AWS SDK v2 (Nick Craig-Wood) + * Add `--s3-sdk-log-mode` to control SDKv2 debugging (Nick Craig-Wood) + * Fix incorrect region for Magalu provider (Filipe Herculano) + * Allow restoring from intelligent-tiering storage class (Pawel Palucha) +* SFTP + * Use `uint32` for mtime to save memory (Tomasz Melcer) + * Ignore useless errors when closing the connection pool (Nick Craig-Wood) + * Support listening on passed FDs (Florian Klink) +* Swift + * Add workarounds for bad listings in Ceph RGW (Paul Collins) + * Add total/free space info in `about` command. (fsantagostinobietti) +* Ulozto + * Fix upload of > 2GB files on 32 bit platforms (Tobias Markus) +* WebDAV + * Add `--webdav-unix-socket-path` to connect to a unix socket (Florian Klink) +* Yandex + * Implement custom user agent to help with upload speeds (Sebastian Bünger) +* Zoho + * Fix inefficiencies uploading with new API to avoid throttling (Nick Craig-Wood) + ## v1.67.0 - 2024-06-14 [See commits](https://github.com/rclone/rclone/compare/v1.66.0...v1.67.0) diff --git a/docs/content/commands/rclone.md b/docs/content/commands/rclone.md index c73ff6f5e..2a39a39f8 100644 --- a/docs/content/commands/rclone.md +++ b/docs/content/commands/rclone.md @@ -3,12 +3,11 @@ title: "rclone" description: "Show help for rclone commands, flags and backends." # autogenerated - DO NOT EDIT, instead edit the source code in cmd/ and as part of making a release run "make commanddocs" --- -## rclone +# rclone Show help for rclone commands, flags and backends. -### Synopsis - +## Synopsis Rclone syncs files to and from cloud storage providers as well as mounting them, listing them in lots of different ways. @@ -22,7 +21,7 @@ documentation, changelog and configuration walkthroughs. rclone [flags] ``` -### Options +## Options ``` --alias-description string Description of the remote @@ -123,7 +122,7 @@ rclone [flags] --box-token-url string Token server url --box-upload-cutoff SizeSuffix Cutoff for switching to multipart upload (>= 50 MiB) (default 50Mi) --buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi) - --bwlimit BwTimetable Bandwidth limit in KiB/s, or use suffix B|K|M|G|T|P or a full timetable. + --bwlimit BwTimetable Bandwidth limit in KiB/s, or use suffix B|K|M|G|T|P or a full timetable --bwlimit-file BwTimetable Bandwidth limit per file in KiB/s, or use suffix B|K|M|G|T|P or a full timetable --ca-cert stringArray CA certificate used to verify servers --cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage (default 1m0s) @@ -150,7 +149,7 @@ rclone [flags] --cache-writes Cache file data on writes through the FS --check-first Do all the checks before starting transfers --checkers int Number of checkers to run in parallel (default 8) - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) --chunker-chunk-size SizeSuffix Files larger than chunk size will be split in chunks (default 2Gi) --chunker-description string Description of the remote --chunker-fail-hard Choose how chunker should handle files with missing or invalid chunks @@ -161,7 +160,7 @@ rclone [flags] --color AUTO|NEVER|ALWAYS When to show colors (and other ANSI codes) AUTO|NEVER|ALWAYS (default AUTO) --combine-description string Description of the remote --combine-upstreams SpaceSepList Upstreams for combining - --compare-dest stringArray Include additional comma separated server-side paths during comparison + --compare-dest stringArray Include additional server-side paths during comparison --compress-description string Description of the remote --compress-level int GZIP compression level (-2 to 9) (default -1) --compress-mode string Compression mode (default "gzip") @@ -192,7 +191,7 @@ rclone [flags] --delete-during When synchronizing, delete files during transfer --delete-excluded Delete files on dest excluded from sync --disable string Disable a comma separated list of features (use --disable help to see a list) - --disable-http-keep-alives Disable HTTP keep-alives and use each connection once. + --disable-http-keep-alives Disable HTTP keep-alives and use each connection once --disable-http2 Disable HTTP/2 in the global transport --drive-acknowledge-abuse Set to allow files which return cannotDownloadAbusiveFile to be downloaded --drive-allow-import-name-change Allow the filetype to change when uploading Google docs @@ -288,6 +287,12 @@ rclone [flags] --filefabric-version string Version read from the file fabric --files-from stringArray Read list of source-file names from file (use - to read from stdin) --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) + --filescom-api-key string The API key used to authenticate with Files.com + --filescom-description string Description of the remote + --filescom-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot) + --filescom-password string The password used to authenticate with Files.com (obscured) + --filescom-site string Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com) + --filescom-username string The username used to authenticate with Files.com -f, --filter stringArray Add a file filtering rule --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) --fix-case Force rename of case insensitive dest to match source @@ -336,6 +341,12 @@ rclone [flags] --gcs-token string OAuth Access Token as a JSON blob --gcs-token-url string Token server url --gcs-user-project string User project + --gofile-access-token string API Access token + --gofile-account-id string Account ID + --gofile-description string Description of the remote + --gofile-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftPeriod,RightPeriod,InvalidUtf8,Dot,Exclamation) + --gofile-list-chunk int Number of items to list in each call (default 1000) + --gofile-root-folder-id string ID of the root folder --gphotos-auth-url string Auth server URL --gphotos-batch-commit-timeout Duration Max time to wait for a batch to finish committing (default 10m0s) --gphotos-batch-mode string Upload file batching sync|async|off (default "sync") @@ -445,6 +456,7 @@ rclone [flags] --local-description string Description of the remote --local-encoding Encoding The encoding for the backend (default Slash,Dot) --local-no-check-updated Don't check to see if the files change during upload + --local-no-clone Disable reflink cloning for server-side copies --local-no-preallocate Disable preallocation of disk space for transferred files --local-no-set-modtime Disable setting modtime --local-no-sparse Disable sparse files for multi-thread downloads @@ -498,6 +510,22 @@ rclone [flags] --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) --metadata-mapper SpaceSepList Program to run to transforming metadata before upload --metadata-set stringArray Add metadata key=value when uploading + --metrics-addr stringArray IPaddress:Port or :Port to bind metrics server to (default [""]) + --metrics-allow-origin string Origin which cross-domain request (CORS) can be executed from + --metrics-baseurl string Prefix for URLs - leave blank for root + --metrics-cert string TLS PEM key (concatenation of certificate and CA certificate) + --metrics-client-ca string Client certificate authority to verify clients with + --metrics-htpasswd string A htpasswd file - if not provided no authentication is done + --metrics-key string TLS PEM Private key + --metrics-max-header-bytes int Maximum size of request header (default 4096) + --metrics-min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") + --metrics-pass string Password for authentication + --metrics-realm string Realm for authentication + --metrics-salt string Password hashing salt (default "dlPL2MqE") + --metrics-server-read-timeout Duration Timeout for server reading data (default 1h0m0s) + --metrics-server-write-timeout Duration Timeout for server writing data (default 1h0m0s) + --metrics-template string User-specified template + --metrics-user string User name for authentication --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) --modify-window Duration Max time diff to be considered the same (default 1ns) @@ -603,6 +631,10 @@ rclone [flags] --pikpak-upload-concurrency int Concurrency for multipart uploads (default 5) --pikpak-use-trash Send files to the trash instead of deleting permanently (default true) --pikpak-user string Pikpak username + --pixeldrain-api-key string API key for your pixeldrain account + --pixeldrain-api-url string The API endpoint to connect to. In the vast majority of cases it's fine to leave (default "https://pixeldrain.com/api") + --pixeldrain-description string Description of the remote + --pixeldrain-root-folder-id string Root of the filesystem to use (default "me") --premiumizeme-auth-url string Auth server URL --premiumizeme-client-id string OAuth Client Id --premiumizeme-client-secret string OAuth Client Secret @@ -651,12 +683,12 @@ rclone [flags] --quatrix-skip-project-folders Skip project folders in operations -q, --quiet Print as little stuff as possible --rc Enable the remote control server - --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572]) + --rc-addr stringArray IPaddress:Port or :Port to bind server to (default ["localhost:5572"]) --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from --rc-baseurl string Prefix for URLs - leave blank for root --rc-cert string TLS PEM key (concatenation of certificate and CA certificate) --rc-client-ca string Client certificate authority to verify clients with - --rc-enable-metrics Enable prometheus metrics on /metrics + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server --rc-files string Path to local files to serve on the HTTP server --rc-htpasswd string A htpasswd file - if not provided no authentication is done --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s) @@ -712,6 +744,7 @@ rclone [flags] --s3-provider string Choose your S3 provider --s3-region string Region to connect to --s3-requester-pays Enables requester pays option when interacting with S3 bucket + --s3-sdk-log-mode Bits Set to debug the SDK (default Off) --s3-secret-access-key string AWS Secret Access Key (password) --s3-server-side-encryption string The server-side encryption algorithm used when storing this object in S3 --s3-session-token string An AWS session token @@ -722,7 +755,6 @@ rclone [flags] --s3-sse-customer-key-md5 string If using SSE-C you may provide the secret encryption key MD5 checksum (optional) --s3-sse-kms-key-id string If using KMS ID you must provide the ARN of Key --s3-storage-class string The storage class to use when storing new objects in S3 - --s3-sts-endpoint string Endpoint for STS --s3-upload-concurrency int Concurrency for multipart uploads and copies (default 4) --s3-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi) --s3-use-accelerate-endpoint If true use the AWS S3 accelerated endpoint @@ -732,6 +764,7 @@ rclone [flags] --s3-use-multipart-etag Tristate Whether to use ETag in multipart uploads for verification (default unset) --s3-use-multipart-uploads Tristate Set if rclone should use multipart uploads (default unset) --s3-use-presigned-request Whether to use a presigned request or PutObject for single part uploads + --s3-use-unsigned-payload Tristate Whether to use an unsigned payload in PutObject (default unset) --s3-v2-auth If true use v2 authentication --s3-version-at Time Show file versions as they were at the specified time (default off) --s3-version-deleted Show deleted file markers when using versions @@ -852,10 +885,12 @@ rclone [flags] --swift-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8) --swift-endpoint-type string Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) (default "public") --swift-env-auth Get swift credentials from environment variables in standard OpenStack form + --swift-fetch-until-empty-page When paginating, always fetch unless we received an empty page --swift-key string API key or password (OS_PASSWORD) --swift-leave-parts-on-error If true avoid calling abort upload on a failure --swift-no-chunk Don't chunk files during streaming upload --swift-no-large-objects Disable support for static and dynamic large objects + --swift-partial-page-fetch-threshold int When paginating, fetch if the current page is within this percentage of the limit --swift-region string Region name - optional (OS_REGION_NAME) --swift-storage-policy string The storage policy to use when creating a new container --swift-storage-url string Storage URL - optional (OS_STORAGE_URL) @@ -866,7 +901,7 @@ rclone [flags] --swift-user string User name to log in (OS_USERNAME) --swift-user-id string User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID) --syslog Use Syslog for logging - --syslog-facility string Facility for syslog, e.g. KERN,USER,... (default "DAEMON") + --syslog-facility string Facility for syslog, e.g. KERN,USER (default "DAEMON") --temp-dir string Directory rclone will use for temporary files (default "/tmp") --timeout Duration IO idle timeout (default 5m0s) --tpslimit float Limit HTTP transactions per second to this @@ -897,7 +932,7 @@ rclone [flags] --use-json-log Use json log format --use-mmap Use mmap allocator (see docs) --use-server-modtime Use server modified time instead of object metadata - --user-agent string Set the user-agent to a specified string (default "rclone/v1.67.0") + --user-agent string Set the user-agent to a specified string (default "rclone/v1.68.0") -v, --verbose count Print lots more stuff (repeat for more) -V, --version Print the version number --webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon) @@ -910,6 +945,7 @@ rclone [flags] --webdav-owncloud-exclude-shares Exclude ownCloud shares --webdav-pacer-min-sleep Duration Minimum time to sleep between API calls (default 10ms) --webdav-pass string Password (obscured) + --webdav-unix-socket string Path to a unix domain socket to dial to, instead of opening a TCP connection directly --webdav-url string URL of http host to connect to --webdav-user string User name --webdav-vendor string Name of the WebDAV site/service/software you are using @@ -919,6 +955,7 @@ rclone [flags] --yandex-description string Description of the remote --yandex-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot) --yandex-hard-delete Delete files permanently rather than putting them into the trash + --yandex-spoof-ua Set the user agent to match an official version of the yandex disk client. May help with upload performance (default true) --yandex-token string OAuth Access Token as a JSON blob --yandex-token-url string Token server url --zoho-auth-url string Auth server URL @@ -931,7 +968,7 @@ rclone [flags] --zoho-token-url string Token server url ``` -### SEE ALSO +## See Also * [rclone about](/commands/rclone_about/) - Get quota information from the remote. * [rclone authorize](/commands/rclone_authorize/) - Remote authorization. diff --git a/docs/content/commands/rclone_about.md b/docs/content/commands/rclone_about.md index bee45538d..5ef4d9234 100644 --- a/docs/content/commands/rclone_about.md +++ b/docs/content/commands/rclone_about.md @@ -10,8 +10,7 @@ Get quota information from the remote. ## Synopsis - -`rclone about` prints quota information about a remote to standard +Prints quota information about a remote to standard output. The output is typically used, free, quota and trash contents. E.g. Typical output from `rclone about remote:` is: @@ -70,10 +69,9 @@ rclone about remote: [flags] --json Format output as JSON ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_authorize.md b/docs/content/commands/rclone_authorize.md index f13052c95..abdaff9d2 100644 --- a/docs/content/commands/rclone_authorize.md +++ b/docs/content/commands/rclone_authorize.md @@ -10,7 +10,6 @@ Remote authorization. ## Synopsis - Remote authorization. Used to authorize a remote or headless rclone from a machine with a browser - use as instructed by rclone config. @@ -32,10 +31,9 @@ rclone authorize [flags] --template string The path to a custom Go template for generating HTML responses ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_backend.md b/docs/content/commands/rclone_backend.md index def1f8bba..6708ef634 100644 --- a/docs/content/commands/rclone_backend.md +++ b/docs/content/commands/rclone_backend.md @@ -10,7 +10,6 @@ Run a backend-specific command. ## Synopsis - This runs a backend-specific command. The commands themselves (except for "help" and "features") are defined by the backends and you should see the backend docs for definitions. @@ -50,10 +49,12 @@ rclone backend remote:path [opts] [flags] -o, --option stringArray Option in the form name=value or name ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -61,9 +62,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_bisync.md b/docs/content/commands/rclone_bisync.md index 838d3501c..55fdfdea6 100644 --- a/docs/content/commands/rclone_bisync.md +++ b/docs/content/commands/rclone_bisync.md @@ -63,15 +63,17 @@ rclone bisync remote1:path1 remote2:path2 [flags] --workdir string Use custom working dir - useful for testing. (default: {WORKDIR}) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -103,9 +105,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -113,9 +115,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -142,9 +144,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_cat.md b/docs/content/commands/rclone_cat.md index 3f194f108..6ec5666a0 100644 --- a/docs/content/commands/rclone_cat.md +++ b/docs/content/commands/rclone_cat.md @@ -10,8 +10,7 @@ Concatenates any files and sends them to stdout. ## Synopsis - -rclone cat sends any files to standard output. +Sends any files to standard output. You can use it like this to output a single file @@ -59,10 +58,12 @@ rclone cat remote:path [flags] --tail int Only print the last N characters ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -89,18 +90,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_check.md b/docs/content/commands/rclone_check.md index 66f517ef9..9647280b5 100644 --- a/docs/content/commands/rclone_check.md +++ b/docs/content/commands/rclone_check.md @@ -9,7 +9,6 @@ Checks the files in the source and destination match. ## Synopsis - Checks the files in the source and destination match. It compares sizes and hashes (MD5 or SHA1) and logs a report of files that don't match. It doesn't alter the source or destination. @@ -73,18 +72,20 @@ rclone check source:path dest:path [flags] --one-way Check one way only, source files must exist on remote ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Check Options +### Check Options -Flags used for `rclone check`. +Flags used for check commands ``` --max-backlog int Maximum number of objects in sync or check backlog (default 10000) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -111,18 +112,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_checksum.md b/docs/content/commands/rclone_checksum.md index 0bb9963fe..9f6eaab39 100644 --- a/docs/content/commands/rclone_checksum.md +++ b/docs/content/commands/rclone_checksum.md @@ -10,7 +10,6 @@ Checks the files in the destination against a SUM file. ## Synopsis - Checks that hashsums of destination files match the SUM file. It compares hashes (MD5, SHA1, etc) and logs a report of files which don't match. It doesn't alter the file system. @@ -67,10 +66,12 @@ rclone checksum sumfile dst:path [flags] --one-way Check one way only, source files must exist on remote ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -97,18 +98,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_cleanup.md b/docs/content/commands/rclone_cleanup.md index e4a8b9e8b..8502d3f72 100644 --- a/docs/content/commands/rclone_cleanup.md +++ b/docs/content/commands/rclone_cleanup.md @@ -10,7 +10,6 @@ Clean up the remote if possible. ## Synopsis - Clean up the remote if possible. Empty the trash or delete old file versions. Not supported by all remotes. @@ -25,10 +24,12 @@ rclone cleanup remote:path [flags] -h, --help help for cleanup ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -36,9 +37,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_completion.md b/docs/content/commands/rclone_completion.md index 32394e0a1..d9b7e605d 100644 --- a/docs/content/commands/rclone_completion.md +++ b/docs/content/commands/rclone_completion.md @@ -12,7 +12,6 @@ Output completion script for a given shell. ## Synopsis - Generates a shell completion script for rclone. Run with `--help` to list the supported shells. @@ -23,10 +22,9 @@ Run with `--help` to list the supported shells. -h, --help help for completion ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. * [rclone completion bash](/commands/rclone_completion_bash/) - Output bash completion script for rclone. diff --git a/docs/content/commands/rclone_completion_bash.md b/docs/content/commands/rclone_completion_bash.md index 4b1cf3fb3..54af5149c 100644 --- a/docs/content/commands/rclone_completion_bash.md +++ b/docs/content/commands/rclone_completion_bash.md @@ -11,12 +11,11 @@ Output bash completion script for rclone. ## Synopsis - Generates a bash shell autocompletion script for rclone. By default, when run without any arguments, - rclone genautocomplete bash + rclone completion bash the generated script will be written to @@ -51,10 +50,9 @@ rclone completion bash [output_file] [flags] -h, --help help for bash ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone completion](/commands/rclone_completion/) - Output completion script for a given shell. diff --git a/docs/content/commands/rclone_completion_fish.md b/docs/content/commands/rclone_completion_fish.md index 6b5ed8b64..59dfa52ad 100644 --- a/docs/content/commands/rclone_completion_fish.md +++ b/docs/content/commands/rclone_completion_fish.md @@ -11,13 +11,12 @@ Output fish completion script for rclone. ## Synopsis - Generates a fish autocompletion script for rclone. This writes to /etc/fish/completions/rclone.fish by default so will probably need to be run with sudo or as root, e.g. - sudo rclone genautocomplete fish + sudo rclone completion fish Logout and login again to use the autocompletion scripts, or source them directly @@ -40,10 +39,9 @@ rclone completion fish [output_file] [flags] -h, --help help for fish ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone completion](/commands/rclone_completion/) - Output completion script for a given shell. diff --git a/docs/content/commands/rclone_completion_powershell.md b/docs/content/commands/rclone_completion_powershell.md index 57b5a0615..f872531a3 100644 --- a/docs/content/commands/rclone_completion_powershell.md +++ b/docs/content/commands/rclone_completion_powershell.md @@ -11,7 +11,6 @@ Output powershell completion script for rclone. ## Synopsis - Generate the autocompletion script for powershell. To load completions in your current shell session: @@ -34,10 +33,9 @@ rclone completion powershell [output_file] [flags] -h, --help help for powershell ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone completion](/commands/rclone_completion/) - Output completion script for a given shell. diff --git a/docs/content/commands/rclone_completion_zsh.md b/docs/content/commands/rclone_completion_zsh.md index eb23fa995..a12f3aa84 100644 --- a/docs/content/commands/rclone_completion_zsh.md +++ b/docs/content/commands/rclone_completion_zsh.md @@ -11,13 +11,12 @@ Output zsh completion script for rclone. ## Synopsis - Generates a zsh autocompletion script for rclone. This writes to /usr/share/zsh/vendor-completions/_rclone by default so will probably need to be run with sudo or as root, e.g. - sudo rclone genautocomplete zsh + sudo rclone completion zsh Logout and login again to use the autocompletion scripts, or source them directly @@ -40,10 +39,9 @@ rclone completion zsh [output_file] [flags] -h, --help help for zsh ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone completion](/commands/rclone_completion/) - Output completion script for a given shell. diff --git a/docs/content/commands/rclone_config.md b/docs/content/commands/rclone_config.md index e79798f8b..91b717cbe 100644 --- a/docs/content/commands/rclone_config.md +++ b/docs/content/commands/rclone_config.md @@ -25,10 +25,9 @@ rclone config [flags] -h, --help help for config ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. * [rclone config create](/commands/rclone_config_create/) - Create a new remote with name, type and options. @@ -36,6 +35,7 @@ See the [global flags page](/flags/) for global options not listed here. * [rclone config disconnect](/commands/rclone_config_disconnect/) - Disconnects user from remote * [rclone config dump](/commands/rclone_config_dump/) - Dump the config file as JSON. * [rclone config edit](/commands/rclone_config_edit/) - Enter an interactive configuration session. +* [rclone config encryption](/commands/rclone_config_encryption/) - set, remove and check the encryption for the config file * [rclone config file](/commands/rclone_config_file/) - Show path of configuration file in use. * [rclone config password](/commands/rclone_config_password/) - Update password in an existing remote. * [rclone config paths](/commands/rclone_config_paths/) - Show paths used for configuration, cache, temp etc. diff --git a/docs/content/commands/rclone_config_create.md b/docs/content/commands/rclone_config_create.md index 9a52085da..2832b70a8 100644 --- a/docs/content/commands/rclone_config_create.md +++ b/docs/content/commands/rclone_config_create.md @@ -10,7 +10,6 @@ Create a new remote with name, type and options. ## Synopsis - Create a new remote of `name` with `type` and options. The options should be passed in pairs of `key` `value` or as `key=value`. @@ -130,10 +129,9 @@ rclone config create name type [key value]* [flags] --state string State - use with --continue ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_delete.md b/docs/content/commands/rclone_config_delete.md index a1e740dd1..8ef2f744a 100644 --- a/docs/content/commands/rclone_config_delete.md +++ b/docs/content/commands/rclone_config_delete.md @@ -18,10 +18,9 @@ rclone config delete name [flags] -h, --help help for delete ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_disconnect.md b/docs/content/commands/rclone_config_disconnect.md index 1acdb209a..044842043 100644 --- a/docs/content/commands/rclone_config_disconnect.md +++ b/docs/content/commands/rclone_config_disconnect.md @@ -9,7 +9,6 @@ Disconnects user from remote ## Synopsis - This disconnects the remote: passed in to the cloud storage system. This normally means revoking the oauth token. @@ -27,10 +26,9 @@ rclone config disconnect remote: [flags] -h, --help help for disconnect ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_dump.md b/docs/content/commands/rclone_config_dump.md index d7f8f55d3..7a204b3ee 100644 --- a/docs/content/commands/rclone_config_dump.md +++ b/docs/content/commands/rclone_config_dump.md @@ -18,10 +18,9 @@ rclone config dump [flags] -h, --help help for dump ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_edit.md b/docs/content/commands/rclone_config_edit.md index 4fbd49124..0e988af6d 100644 --- a/docs/content/commands/rclone_config_edit.md +++ b/docs/content/commands/rclone_config_edit.md @@ -25,10 +25,9 @@ rclone config edit [flags] -h, --help help for edit ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_encryption.md b/docs/content/commands/rclone_config_encryption.md new file mode 100644 index 000000000..b7c552ee6 --- /dev/null +++ b/docs/content/commands/rclone_config_encryption.md @@ -0,0 +1,30 @@ +--- +title: "rclone config encryption" +description: "set, remove and check the encryption for the config file" +# autogenerated - DO NOT EDIT, instead edit the source code in cmd/config/encryption/ and as part of making a release run "make commanddocs" +--- +# rclone config encryption + +set, remove and check the encryption for the config file + +## Synopsis + +This command sets, clears and checks the encryption for the config file using +the subcommands below. + + +## Options + +``` + -h, --help help for encryption +``` + +See the [global flags page](/flags/) for global options not listed here. + +## See Also + +* [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. +* [rclone config encryption check](/commands/rclone_config_encryption_check/) - Check that the config file is encrypted +* [rclone config encryption remove](/commands/rclone_config_encryption_remove/) - Remove the config file encryption password +* [rclone config encryption set](/commands/rclone_config_encryption_set/) - Set or change the config file encryption password + diff --git a/docs/content/commands/rclone_config_encryption_check.md b/docs/content/commands/rclone_config_encryption_check.md new file mode 100644 index 000000000..f64c265f6 --- /dev/null +++ b/docs/content/commands/rclone_config_encryption_check.md @@ -0,0 +1,37 @@ +--- +title: "rclone config encryption check" +description: "Check that the config file is encrypted" +# autogenerated - DO NOT EDIT, instead edit the source code in cmd/config/encryption/check/ and as part of making a release run "make commanddocs" +--- +# rclone config encryption check + +Check that the config file is encrypted + +## Synopsis + +This checks the config file is encrypted and that you can decrypt it. + +It will attempt to decrypt the config using the password you supply. + +If decryption fails it will return a non-zero exit code if using +`--password-command`, otherwise it will prompt again for the password. + +If the config file is not encrypted it will return a non zero exit code. + + +``` +rclone config encryption check [flags] +``` + +## Options + +``` + -h, --help help for check +``` + +See the [global flags page](/flags/) for global options not listed here. + +## See Also + +* [rclone config encryption](/commands/rclone_config_encryption/) - set, remove and check the encryption for the config file + diff --git a/docs/content/commands/rclone_config_encryption_remove.md b/docs/content/commands/rclone_config_encryption_remove.md new file mode 100644 index 000000000..fa78458e2 --- /dev/null +++ b/docs/content/commands/rclone_config_encryption_remove.md @@ -0,0 +1,38 @@ +--- +title: "rclone config encryption remove" +description: "Remove the config file encryption password" +# autogenerated - DO NOT EDIT, instead edit the source code in cmd/config/encryption/remove/ and as part of making a release run "make commanddocs" +--- +# rclone config encryption remove + +Remove the config file encryption password + +## Synopsis + +Remove the config file encryption password + +This removes the config file encryption, returning it to un-encrypted. + +If `--password-command` is in use, this will be called to supply the old config +password. + +If the config was not encrypted then no error will be returned and +this command will do nothing. + + +``` +rclone config encryption remove [flags] +``` + +## Options + +``` + -h, --help help for remove +``` + +See the [global flags page](/flags/) for global options not listed here. + +## See Also + +* [rclone config encryption](/commands/rclone_config_encryption/) - set, remove and check the encryption for the config file + diff --git a/docs/content/commands/rclone_config_encryption_set.md b/docs/content/commands/rclone_config_encryption_set.md new file mode 100644 index 000000000..b02dff900 --- /dev/null +++ b/docs/content/commands/rclone_config_encryption_set.md @@ -0,0 +1,48 @@ +--- +title: "rclone config encryption set" +description: "Set or change the config file encryption password" +# autogenerated - DO NOT EDIT, instead edit the source code in cmd/config/encryption/set/ and as part of making a release run "make commanddocs" +--- +# rclone config encryption set + +Set or change the config file encryption password + +## Synopsis + +This command sets or changes the config file encryption password. + +If there was no config password set then it sets a new one, otherwise +it changes the existing config password. + +Note that if you are changing an encryption password using +`--password-command` then this will be called once to decrypt the +config using the old password and then again to read the new +password to re-encrypt the config. + +When `--password-command` is called to change the password then the +environment variable `RCLONE_PASSWORD_CHANGE=1` will be set. So if +changing passwords programatically you can use the environment +variable to distinguish which password you must supply. + +Alternatively you can remove the password first (with `rclone config +encryption remove`), then set it again with this command which may be +easier if you don't mind the unecrypted config file being on the disk +briefly. + + +``` +rclone config encryption set [flags] +``` + +## Options + +``` + -h, --help help for set +``` + +See the [global flags page](/flags/) for global options not listed here. + +## See Also + +* [rclone config encryption](/commands/rclone_config_encryption/) - set, remove and check the encryption for the config file + diff --git a/docs/content/commands/rclone_config_file.md b/docs/content/commands/rclone_config_file.md index 73a8d6e0b..68b4f3158 100644 --- a/docs/content/commands/rclone_config_file.md +++ b/docs/content/commands/rclone_config_file.md @@ -18,10 +18,9 @@ rclone config file [flags] -h, --help help for file ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_password.md b/docs/content/commands/rclone_config_password.md index 7d7cf73fd..6a9909ad1 100644 --- a/docs/content/commands/rclone_config_password.md +++ b/docs/content/commands/rclone_config_password.md @@ -10,7 +10,6 @@ Update password in an existing remote. ## Synopsis - Update an existing remote's password. The password should be passed in pairs of `key` `password` or as `key=password`. The `password` should be passed in in clear (unobscured). @@ -34,10 +33,9 @@ rclone config password name [key value]+ [flags] -h, --help help for password ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_paths.md b/docs/content/commands/rclone_config_paths.md index ef72f97ab..807d40259 100644 --- a/docs/content/commands/rclone_config_paths.md +++ b/docs/content/commands/rclone_config_paths.md @@ -18,10 +18,9 @@ rclone config paths [flags] -h, --help help for paths ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_providers.md b/docs/content/commands/rclone_config_providers.md index d8e636b32..d18c663ad 100644 --- a/docs/content/commands/rclone_config_providers.md +++ b/docs/content/commands/rclone_config_providers.md @@ -18,10 +18,9 @@ rclone config providers [flags] -h, --help help for providers ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_reconnect.md b/docs/content/commands/rclone_config_reconnect.md index 97265f46b..0237850d8 100644 --- a/docs/content/commands/rclone_config_reconnect.md +++ b/docs/content/commands/rclone_config_reconnect.md @@ -9,7 +9,6 @@ Re-authenticates user with remote. ## Synopsis - This reconnects remote: passed in to the cloud storage system. To disconnect the remote use "rclone config disconnect". @@ -27,10 +26,9 @@ rclone config reconnect remote: [flags] -h, --help help for reconnect ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_redacted.md b/docs/content/commands/rclone_config_redacted.md index 58221968e..e37f5d4ef 100644 --- a/docs/content/commands/rclone_config_redacted.md +++ b/docs/content/commands/rclone_config_redacted.md @@ -32,10 +32,9 @@ rclone config redacted [] [flags] -h, --help help for redacted ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_show.md b/docs/content/commands/rclone_config_show.md index 37a6f3c5c..eb1897105 100644 --- a/docs/content/commands/rclone_config_show.md +++ b/docs/content/commands/rclone_config_show.md @@ -18,10 +18,9 @@ rclone config show [] [flags] -h, --help help for show ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_touch.md b/docs/content/commands/rclone_config_touch.md index 913d895f7..8fd7a0028 100644 --- a/docs/content/commands/rclone_config_touch.md +++ b/docs/content/commands/rclone_config_touch.md @@ -18,10 +18,9 @@ rclone config touch [flags] -h, --help help for touch ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_update.md b/docs/content/commands/rclone_config_update.md index 3efc6219a..e4a160b6a 100644 --- a/docs/content/commands/rclone_config_update.md +++ b/docs/content/commands/rclone_config_update.md @@ -10,7 +10,6 @@ Update options in an existing remote. ## Synopsis - Update an existing remote's options. The options should be passed in pairs of `key` `value` or as `key=value`. @@ -130,10 +129,9 @@ rclone config update name [key value]+ [flags] --state string State - use with --continue ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_config_userinfo.md b/docs/content/commands/rclone_config_userinfo.md index 87018428f..cd6a04cdf 100644 --- a/docs/content/commands/rclone_config_userinfo.md +++ b/docs/content/commands/rclone_config_userinfo.md @@ -9,7 +9,6 @@ Prints info about logged in user of remote. ## Synopsis - This prints the details of the person logged in to the cloud storage system. @@ -25,10 +24,9 @@ rclone config userinfo remote: [flags] --json Format output as JSON ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone config](/commands/rclone_config/) - Enter an interactive configuration session. diff --git a/docs/content/commands/rclone_copy.md b/docs/content/commands/rclone_copy.md index ccb85397a..a243c2e16 100644 --- a/docs/content/commands/rclone_copy.md +++ b/docs/content/commands/rclone_copy.md @@ -9,7 +9,6 @@ Copy files from source to dest, skipping identical files. ## Synopsis - Copy the source to the destination. Does not transfer files that are identical on source and destination, testing by size and modification time or MD5SUM. Doesn't delete files from the destination. If you @@ -87,15 +86,17 @@ rclone copy source:path dest:path [flags] -h, --help help for copy ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -127,9 +128,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -137,9 +138,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -166,18 +167,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_copyto.md b/docs/content/commands/rclone_copyto.md index 22479d13f..e2d4b05bf 100644 --- a/docs/content/commands/rclone_copyto.md +++ b/docs/content/commands/rclone_copyto.md @@ -10,7 +10,6 @@ Copy files from source to dest, skipping identical files. ## Synopsis - If source:path is a file or directory then it copies it to a file or directory named dest:path. @@ -50,15 +49,17 @@ rclone copyto source:path dest:path [flags] -h, --help help for copyto ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -90,9 +91,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -100,9 +101,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -129,18 +130,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_copyurl.md b/docs/content/commands/rclone_copyurl.md index 78d4bc6b7..061644fa3 100644 --- a/docs/content/commands/rclone_copyurl.md +++ b/docs/content/commands/rclone_copyurl.md @@ -10,7 +10,6 @@ Copy the contents of the URL supplied content to dest:path. ## Synopsis - Download a URL's content and copy it to the destination without saving it in temporary storage. @@ -56,10 +55,12 @@ rclone copyurl https://example.com dest:path [flags] --stdout Write the output to stdout rather than a file ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -67,9 +68,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_cryptcheck.md b/docs/content/commands/rclone_cryptcheck.md index bbc961449..1f3a916bd 100644 --- a/docs/content/commands/rclone_cryptcheck.md +++ b/docs/content/commands/rclone_cryptcheck.md @@ -10,10 +10,9 @@ Cryptcheck checks the integrity of an encrypted remote. ## Synopsis - -rclone cryptcheck checks a remote against a [crypted](/crypt/) remote. -This is the equivalent of running rclone [check](/commands/rclone_check/), -but able to check the checksums of the encrypted remote. +Checks a remote against a [crypted](/crypt/) remote. This is the equivalent +of running rclone [check](/commands/rclone_check/), but able to check the +checksums of the encrypted remote. For it to work the underlying remote of the cryptedremote must support some kind of checksum. @@ -76,18 +75,20 @@ rclone cryptcheck remote:path cryptedremote:path [flags] --one-way Check one way only, source files must exist on remote ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Check Options +### Check Options -Flags used for `rclone check`. +Flags used for check commands ``` --max-backlog int Maximum number of objects in sync or check backlog (default 10000) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -114,18 +115,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_cryptdecode.md b/docs/content/commands/rclone_cryptdecode.md index 8964268fe..42691cd70 100644 --- a/docs/content/commands/rclone_cryptdecode.md +++ b/docs/content/commands/rclone_cryptdecode.md @@ -10,9 +10,8 @@ Cryptdecode returns unencrypted file names. ## Synopsis - -rclone cryptdecode returns unencrypted file names when provided with -a list of encrypted file names. List limit is 10 items. +Returns unencrypted file names when provided with a list of encrypted file +names. List limit is 10 items. If you supply the `--reverse` flag, it will return encrypted file names. @@ -37,10 +36,9 @@ rclone cryptdecode encryptedremote: encryptedfilename [flags] --reverse Reverse cryptdecode, encrypts filenames ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_dedupe.md b/docs/content/commands/rclone_dedupe.md index 8126fb600..477da82bb 100644 --- a/docs/content/commands/rclone_dedupe.md +++ b/docs/content/commands/rclone_dedupe.md @@ -10,8 +10,6 @@ Interactively find duplicate filenames and delete/rename them. ## Synopsis - - By default `dedupe` interactively finds files with duplicate names and offers to delete all but one or rename them to be different. This is known as deduping by name. @@ -130,10 +128,12 @@ rclone dedupe [mode] remote:path [flags] -h, --help help for dedupe ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -141,9 +141,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_delete.md b/docs/content/commands/rclone_delete.md index 17a326e53..542cfcd16 100644 --- a/docs/content/commands/rclone_delete.md +++ b/docs/content/commands/rclone_delete.md @@ -10,7 +10,6 @@ Remove the files in path. ## Synopsis - Remove the files in path. Unlike [purge](/commands/rclone_purge/) it obeys include/exclude filters so can be used to selectively delete files. @@ -50,10 +49,12 @@ rclone delete remote:path [flags] --rmdirs rmdirs removes empty directories but leaves root intact ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -61,9 +62,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -90,18 +91,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_deletefile.md b/docs/content/commands/rclone_deletefile.md index 37ffc0d11..17fb064d1 100644 --- a/docs/content/commands/rclone_deletefile.md +++ b/docs/content/commands/rclone_deletefile.md @@ -10,7 +10,6 @@ Remove a single file from remote. ## Synopsis - Remove a single file from remote. Unlike `delete` it cannot be used to remove a directory and it doesn't obey include/exclude filters - if the specified file exists, it will always be removed. @@ -26,10 +25,12 @@ rclone deletefile remote:path [flags] -h, --help help for deletefile ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -37,9 +38,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_gendocs.md b/docs/content/commands/rclone_gendocs.md index 43b4ffdf0..3b7bd9aaf 100644 --- a/docs/content/commands/rclone_gendocs.md +++ b/docs/content/commands/rclone_gendocs.md @@ -10,7 +10,6 @@ Output markdown docs for rclone to the directory supplied. ## Synopsis - This produces markdown docs for the rclone commands to the directory supplied. These are in a format suitable for hugo to render into the rclone.org website. @@ -25,10 +24,9 @@ rclone gendocs output_directory [flags] -h, --help help for gendocs ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_gitannex.md b/docs/content/commands/rclone_gitannex.md index bc8d096ec..39410238e 100644 --- a/docs/content/commands/rclone_gitannex.md +++ b/docs/content/commands/rclone_gitannex.md @@ -1,6 +1,8 @@ --- title: "rclone gitannex" description: "Speaks with git-annex over stdin/stdout." +aliases: + - /commands/rclone_git-annex-remote-rclone-builtin/ versionIntroduced: v1.67.0 # autogenerated - DO NOT EDIT, instead edit the source code in cmd/gitannex/ and as part of making a release run "make commanddocs" --- @@ -91,10 +93,9 @@ rclone gitannex [flags] -h, --help help for gitannex ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_hashsum.md b/docs/content/commands/rclone_hashsum.md index 440e7b57d..4ca4f30a2 100644 --- a/docs/content/commands/rclone_hashsum.md +++ b/docs/content/commands/rclone_hashsum.md @@ -10,7 +10,6 @@ Produces a hashsum file for all the objects in the path. ## Synopsis - Produces a hash file for all the objects in the path using the hash named. The output is in the same format as the standard md5sum/sha1sum tool. @@ -59,10 +58,12 @@ rclone hashsum [ remote:path] [flags] --output-file string Output hashsums to a file rather than the terminal ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -89,18 +90,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_link.md b/docs/content/commands/rclone_link.md index 65d0b2fb8..c07e2e221 100644 --- a/docs/content/commands/rclone_link.md +++ b/docs/content/commands/rclone_link.md @@ -10,8 +10,7 @@ Generate public link to file/folder. ## Synopsis -rclone link will create, retrieve or remove a public link to the given -file or folder. +Create, retrieve or remove a public link to the given file or folder. rclone link remote:path/to/file rclone link remote:path/to/folder/ @@ -45,10 +44,9 @@ rclone link remote:path [flags] --unlink Remove existing public link to file/folder ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_listremotes.md b/docs/content/commands/rclone_listremotes.md index 6e14a8115..77361752b 100644 --- a/docs/content/commands/rclone_listremotes.md +++ b/docs/content/commands/rclone_listremotes.md @@ -11,26 +11,40 @@ List all the remotes in the config file and defined in environment variables. ## Synopsis -rclone listremotes lists all the available remotes from the config file. +Lists all the available remotes from the config file, or the remotes matching +an optional filter. -When used with the `--long` flag it lists the types and the descriptions too. +Prints the result in human-readable format by default, and as a simple list of +remote names, or if used with flag `--long` a tabular format including +the remote names, types and descriptions. Using flag `--json` produces +machine-readable output instead, which always includes all attributes - including +the source (file or environment). + +Result can be filtered by a filter argument which applies to all attributes, +and/or filter flags specific for each attribute. The values must be specified +according to regular rclone filtering pattern syntax. ``` -rclone listremotes [flags] +rclone listremotes [] [flags] ``` ## Options ``` - -h, --help help for listremotes - --long Show the type and the description as well as names + --description string Filter remotes by description + -h, --help help for listremotes + --json Format output as JSON + --long Show type and description in addition to name + --name string Filter remotes by name + --order-by string Instructions on how to order the result, e.g. 'type,name=descending' + --source string Filter remotes by source, e.g. 'file' or 'environment' + --type string Filter remotes by type ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_ls.md b/docs/content/commands/rclone_ls.md index cf1ee9de6..54ad257f1 100644 --- a/docs/content/commands/rclone_ls.md +++ b/docs/content/commands/rclone_ls.md @@ -9,7 +9,6 @@ List the objects in the path with size and path. ## Synopsis - Lists the objects in the source path to standard output in a human readable format with size and path. Recurses by default. @@ -55,10 +54,12 @@ rclone ls remote:path [flags] -h, --help help for ls ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -85,18 +86,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_lsd.md b/docs/content/commands/rclone_lsd.md index 60a79b6dd..90fafb28d 100644 --- a/docs/content/commands/rclone_lsd.md +++ b/docs/content/commands/rclone_lsd.md @@ -9,7 +9,6 @@ List all directories/containers/buckets in the path. ## Synopsis - Lists the directories in the source path to standard output. Does not recurse by default. Use the `-R` flag to recurse. @@ -66,10 +65,12 @@ rclone lsd remote:path [flags] -R, --recursive Recurse into the listing ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -96,18 +97,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_lsf.md b/docs/content/commands/rclone_lsf.md index c7311af01..a02014a66 100644 --- a/docs/content/commands/rclone_lsf.md +++ b/docs/content/commands/rclone_lsf.md @@ -10,7 +10,6 @@ List directories and objects in remote:path formatted for parsing. ## Synopsis - List the contents of the source path (directories and objects) to standard output in a form which is easy to parse by scripts. By default this will just be the names of the objects and directories, @@ -163,10 +162,12 @@ rclone lsf remote:path [flags] -t, --time-format string Specify a custom time format, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -193,18 +194,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_lsjson.md b/docs/content/commands/rclone_lsjson.md index f63e76b15..48ea8b155 100644 --- a/docs/content/commands/rclone_lsjson.md +++ b/docs/content/commands/rclone_lsjson.md @@ -12,7 +12,7 @@ List directories and objects in the path in JSON format. List directories and objects in the path in JSON format. -The output is an array of Items, where each Item looks like this +The output is an array of Items, where each Item looks like this: { "Hashes" : { @@ -34,44 +34,50 @@ The output is an array of Items, where each Item looks like this "Tier" : "hot", } -If `--hash` is not specified, the Hashes property will be omitted. The -types of hash can be specified with the `--hash-type` parameter (which -may be repeated). If `--hash-type` is set then it implies `--hash`. +The exact set of properties included depends on the backend: -If `--no-modtime` is specified then ModTime will be blank. This can -speed things up on remotes where reading the ModTime takes an extra -request (e.g. s3, swift). +- The property IsBucket will only be included for bucket-based remotes, and only + for directories that are buckets. It will always be omitted when value is not true. +- Properties Encrypted and EncryptedPath will only be included for encrypted + remotes, and (as mentioned below) only if the `--encrypted` option is set. -If `--no-mimetype` is specified then MimeType will be blank. This can -speed things up on remotes where reading the MimeType takes an extra -request (e.g. s3, swift). +Different options may also affect which properties are included: -If `--encrypted` is not specified the Encrypted will be omitted. +- If `--hash` is not specified, the Hashes property will be omitted. The + types of hash can be specified with the `--hash-type` parameter (which + may be repeated). If `--hash-type` is set then it implies `--hash`. +- If `--no-modtime` is specified then ModTime will be blank. This can + speed things up on remotes where reading the ModTime takes an extra + request (e.g. s3, swift). +- If `--no-mimetype` is specified then MimeType will be blank. This can + speed things up on remotes where reading the MimeType takes an extra + request (e.g. s3, swift). +- If `--encrypted` is not specified the Encrypted and EncryptedPath + properties will be omitted - even for encrypted remotes. +- If `--metadata` is set then an additional Metadata property will be + returned. This will have [metadata](/docs/#metadata) in rclone standard format + as a JSON object. -If `--dirs-only` is not specified files in addition to directories are -returned +The default is to list directories and files/objects, but this can be changed +with the following options: -If `--files-only` is not specified directories in addition to the files -will be returned. +- If `--dirs-only` is specified then directories will be returned + only, no files/objects. +- If `--files-only` is specified then files will be returned only, + no directories. -If `--metadata` is set then an additional Metadata key will be returned. -This will have metadata in rclone standard format as a JSON object. - -if `--stat` is set then a single JSON blob will be returned about the -item pointed to. This will return an error if the item isn't found. -However on bucket based backends (like s3, gcs, b2, azureblob etc) if -the item isn't found it will return an empty directory as it isn't -possible to tell empty directories from missing directories there. +If `--stat` is set then the the output is not an array of items, +but instead a single JSON blob will be returned about the item pointed to. +This will return an error if the item isn't found, however on bucket based +backends (like s3, gcs, b2, azureblob etc) if the item isn't found it will +return an empty directory, as it isn't possible to tell empty directories +from missing directories there. The Path field will only show folders below the remote path being listed. If "remote:path" contains the file "subfolder/file.txt", the Path for "file.txt" will be "subfolder/file.txt", not "remote:path/subfolder/file.txt". When used without `--recursive` the Path will always be the same as Name. -If the directory is a bucket in a bucket-based backend, then -"IsBucket" will be set to true. This key won't be present unless it is -"true". - The time is in RFC3339 format with up to nanosecond precision. The number of decimal digits in the seconds will depend on the precision that the remote can hold the times, so if times are accurate to the @@ -81,7 +87,8 @@ accurate to the nearest second (Dropbox, Box, WebDav, etc.) no digits will be shown ("2017-05-31T16:15:57+01:00"). The whole output can be processed as a JSON blob, or alternatively it -can be processed line by line as each item is written one to a line. +can be processed line by line as each item is written on individual lines +(except with `--stat`). Any of the filtering options can be applied to this command. @@ -127,10 +134,12 @@ rclone lsjson remote:path [flags] --stat Just return the info for the pointed to file ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -157,18 +166,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_lsl.md b/docs/content/commands/rclone_lsl.md index 1fca5a44a..ba2d587ca 100644 --- a/docs/content/commands/rclone_lsl.md +++ b/docs/content/commands/rclone_lsl.md @@ -10,7 +10,6 @@ List the objects in path with modification time, size and path. ## Synopsis - Lists the objects in the source path to standard output in a human readable format with modification time, size and path. Recurses by default. @@ -56,10 +55,12 @@ rclone lsl remote:path [flags] -h, --help help for lsl ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -86,18 +87,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_md5sum.md b/docs/content/commands/rclone_md5sum.md index 76160249b..fe40b7b1e 100644 --- a/docs/content/commands/rclone_md5sum.md +++ b/docs/content/commands/rclone_md5sum.md @@ -10,7 +10,6 @@ Produces an md5sum file for all the objects in the path. ## Synopsis - Produces an md5sum file for all the objects in the path. This is in the same format as the standard md5sum tool produces. @@ -43,10 +42,12 @@ rclone md5sum remote:path [flags] --output-file string Output hashsums to a file rather than the terminal ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -73,18 +74,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_mkdir.md b/docs/content/commands/rclone_mkdir.md index 292fa8a0e..94d6637c8 100644 --- a/docs/content/commands/rclone_mkdir.md +++ b/docs/content/commands/rclone_mkdir.md @@ -17,10 +17,12 @@ rclone mkdir remote:path [flags] -h, --help help for mkdir ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -28,9 +30,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_mount.md b/docs/content/commands/rclone_mount.md index 9da0e7213..67bd27f55 100644 --- a/docs/content/commands/rclone_mount.md +++ b/docs/content/commands/rclone_mount.md @@ -704,6 +704,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -719,6 +724,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -846,9 +875,9 @@ rclone mount remote:path /path/to/mountpoint [flags] --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for mount @@ -864,7 +893,7 @@ rclone mount remote:path /path/to/mountpoint [flags] --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -877,6 +906,7 @@ rclone mount remote:path /path/to/mountpoint [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -886,10 +916,12 @@ rclone mount remote:path /path/to/mountpoint [flags] --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -916,9 +948,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_move.md b/docs/content/commands/rclone_move.md index 79a1d3986..7c7796e06 100644 --- a/docs/content/commands/rclone_move.md +++ b/docs/content/commands/rclone_move.md @@ -10,7 +10,6 @@ Move files from source to dest. ## Synopsis - Moves the contents of the source directory to the destination directory. Rclone will error if the source and destination overlap and the remote does not support a server-side directory move operation. @@ -62,15 +61,17 @@ rclone move source:path dest:path [flags] -h, --help help for move ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -102,9 +103,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -112,9 +113,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -141,18 +142,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_moveto.md b/docs/content/commands/rclone_moveto.md index 4df6e05f3..132efe3d2 100644 --- a/docs/content/commands/rclone_moveto.md +++ b/docs/content/commands/rclone_moveto.md @@ -10,7 +10,6 @@ Move file or directory from source to dest. ## Synopsis - If source:path is a file or directory then it moves it to a file or directory named dest:path. @@ -53,15 +52,17 @@ rclone moveto source:path dest:path [flags] -h, --help help for moveto ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -93,9 +94,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -103,9 +104,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -132,18 +133,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_ncdu.md b/docs/content/commands/rclone_ncdu.md index f182aa82a..62623c59d 100644 --- a/docs/content/commands/rclone_ncdu.md +++ b/docs/content/commands/rclone_ncdu.md @@ -10,7 +10,6 @@ Explore a remote with a text based user interface. ## Synopsis - This displays a text based user interface allowing the navigation of a remote. It is most useful for answering the question - "What is using all my disk space?". @@ -63,7 +62,7 @@ These flags have the following meaning: This an homage to the [ncdu tool](https://dev.yorhel.nl/ncdu) but for rclone remotes. It is missing lots of features at the moment -but is useful as it stands. +but is useful as it stands. Unlike ncdu it does not show excluded files. Note that it might take some time to delete big files/directories. The UI won't respond in the meantime since the deletion is done synchronously. @@ -83,10 +82,12 @@ rclone ncdu remote:path [flags] -h, --help help for ncdu ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -113,18 +114,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_nfsmount.md b/docs/content/commands/rclone_nfsmount.md index 3c53e6fd2..5230b00d9 100644 --- a/docs/content/commands/rclone_nfsmount.md +++ b/docs/content/commands/rclone_nfsmount.md @@ -705,6 +705,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -720,6 +725,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -848,16 +877,18 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for nfsmount --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki) --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset) --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only) + --nfs-cache-dir string The directory the NFS handle cache will use if set --nfs-cache-handle-limit int max file handles cached simultaneously (min 5) (default 1000000) + --nfs-cache-type memory|disk|symlink Type of NFS handle cache to use (default memory) --no-checksum Don't compare checksums on up/download --no-modtime Don't read/write the modification time (can speed things up) --no-seek Don't allow seeking in files @@ -866,9 +897,9 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] -o, --option stringArray Option for libfuse/WinFsp (repeat if required) --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access - --sudo Use sudo to run the mount command as root. + --sudo Use sudo to run the mount/umount commands as root. --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -881,6 +912,7 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -890,10 +922,12 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -920,9 +954,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_obscure.md b/docs/content/commands/rclone_obscure.md index 0498ac79b..07f0ddeff 100644 --- a/docs/content/commands/rclone_obscure.md +++ b/docs/content/commands/rclone_obscure.md @@ -44,10 +44,9 @@ rclone obscure password [flags] -h, --help help for obscure ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_purge.md b/docs/content/commands/rclone_purge.md index 84413f168..ab191f57e 100644 --- a/docs/content/commands/rclone_purge.md +++ b/docs/content/commands/rclone_purge.md @@ -9,7 +9,6 @@ Remove the path and all of its contents. ## Synopsis - Remove the path and all of its contents. Note that this does not obey include/exclude filters - everything will be removed. Use the [delete](/commands/rclone_delete/) command if you want to selectively @@ -30,10 +29,12 @@ rclone purge remote:path [flags] -h, --help help for purge ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -41,9 +42,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_rc.md b/docs/content/commands/rclone_rc.md index a5259dd4a..cf54e8454 100644 --- a/docs/content/commands/rclone_rc.md +++ b/docs/content/commands/rclone_rc.md @@ -10,8 +10,6 @@ Run a command against a running rclone. ## Synopsis - - This runs a command against a running rclone. Use the `--url` flag to specify an non default URL to connect on. This can be either a ":port" which is taken to mean "http://localhost:port" or a @@ -22,6 +20,13 @@ A username and password can be passed in with `--user` and `--pass`. Note that `--rc-addr`, `--rc-user`, `--rc-pass` will be read also for `--url`, `--user`, `--pass`. +The `--unix-socket` flag can be used to connect over a unix socket like this + + # start server on /tmp/my.socket + rclone rcd --rc-addr unix:///tmp/my.socket + # Connect to it + rclone rc --unix-socket /tmp/my.socket core/stats + Arguments should be passed in as parameter=value. The result will be returned as a JSON object by default. @@ -68,21 +73,21 @@ rclone rc commands parameter [flags] ## Options ``` - -a, --arg stringArray Argument placed in the "arg" array - -h, --help help for rc - --json string Input JSON - use instead of key=value args - --loopback If set connect to this rclone instance not via HTTP - --no-output If set, don't output the JSON result - -o, --opt stringArray Option in the form name=value or name placed in the "opt" array - --pass string Password to use to connect to rclone remote control - --url string URL to connect to rclone remote control (default "http://localhost:5572/") - --user string Username to use to rclone remote control + -a, --arg stringArray Argument placed in the "arg" array + -h, --help help for rc + --json string Input JSON - use instead of key=value args + --loopback If set connect to this rclone instance not via HTTP + --no-output If set, don't output the JSON result + -o, --opt stringArray Option in the form name=value or name placed in the "opt" array + --pass string Password to use to connect to rclone remote control + --unix-socket string Path to a unix domain socket to dial to, instead of opening a TCP connection directly + --url string URL to connect to rclone remote control (default "http://localhost:5572/") + --user string Username to use to rclone remote control ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_rcat.md b/docs/content/commands/rclone_rcat.md index 959ca3d14..bc96d3af6 100644 --- a/docs/content/commands/rclone_rcat.md +++ b/docs/content/commands/rclone_rcat.md @@ -10,9 +10,7 @@ Copies standard input to file on remote. ## Synopsis - -rclone rcat reads from standard input (stdin) and copies it to a -single remote file. +Reads from standard input (stdin) and copies it to a single remote file. echo "hello world" | rclone rcat remote:path/to/file ffmpeg - | rclone rcat remote:path/to/file @@ -53,10 +51,12 @@ rclone rcat remote:path [flags] --size int File size hint to preallocate (default -1) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -64,9 +64,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_rcd.md b/docs/content/commands/rclone_rcd.md index 6c64bbb0f..3750bf477 100644 --- a/docs/content/commands/rclone_rcd.md +++ b/docs/content/commands/rclone_rcd.md @@ -35,6 +35,7 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. `--rc-addr` may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to accomplish the same. `--rc-server-read-timeout` and `--rc-server-write-timeout` can be used to control the timeouts on the server. Note that this is the total time @@ -67,6 +68,20 @@ certificate authority certificate. values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). +## Socket activation + +Instead of the listening addresses specified above, rclone will listen to all +FDs passed by the service manager, if any (and ignore any arguments passed by --rc-addr`). + +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 over TCP. ### Template `--rc-template` allows a user to specify a custom markup template for HTTP @@ -142,19 +157,21 @@ rclone rcd * [flags] -h, --help help for rcd ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## RC Options +### RC Options -Flags to control the Remote Control API. +Flags to control the Remote Control API ``` --rc Enable the remote control server - --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572]) + --rc-addr stringArray IPaddress:Port or :Port to bind server to (default ["localhost:5572"]) --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from --rc-baseurl string Prefix for URLs - leave blank for root --rc-cert string TLS PEM key (concatenation of certificate and CA certificate) --rc-client-ca string Client certificate authority to verify clients with - --rc-enable-metrics Enable prometheus metrics on /metrics + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server --rc-files string Path to local files to serve on the HTTP server --rc-htpasswd string A htpasswd file - if not provided no authentication is done --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s) @@ -179,9 +196,7 @@ Flags to control the Remote Control API. --rc-web-gui-update Check and update to latest version of web gui ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_rmdir.md b/docs/content/commands/rclone_rmdir.md index ed5047176..9eb865ee1 100644 --- a/docs/content/commands/rclone_rmdir.md +++ b/docs/content/commands/rclone_rmdir.md @@ -9,7 +9,6 @@ Remove the empty directory at path. ## Synopsis - This removes empty directory given by path. Will not remove the path if it has any objects in it, not even empty subdirectories. Use command [rmdirs](/commands/rclone_rmdirs/) (or [delete](/commands/rclone_delete/) @@ -28,10 +27,12 @@ rclone rmdir remote:path [flags] -h, --help help for rmdir ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -39,9 +40,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_rmdirs.md b/docs/content/commands/rclone_rmdirs.md index d7132f756..b64d3a616 100644 --- a/docs/content/commands/rclone_rmdirs.md +++ b/docs/content/commands/rclone_rmdirs.md @@ -10,7 +10,6 @@ Remove empty directories under the path. ## Synopsis - This recursively removes any empty directories (including directories that only contain empty directories), that it finds under the path. The root path itself will also be removed if it is empty, unless @@ -42,10 +41,12 @@ rclone rmdirs remote:path [flags] --leave-root Do not remove root directory if empty ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -53,9 +54,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_selfupdate.md b/docs/content/commands/rclone_selfupdate.md index e245ed63f..b32a1ed85 100644 --- a/docs/content/commands/rclone_selfupdate.md +++ b/docs/content/commands/rclone_selfupdate.md @@ -1,6 +1,8 @@ --- title: "rclone selfupdate" description: "Update the rclone binary." +aliases: + - /commands/rclone_self-update/ versionIntroduced: v1.55 # autogenerated - DO NOT EDIT, instead edit the source code in cmd/selfupdate/ and as part of making a release run "make commanddocs" --- @@ -75,10 +77,9 @@ rclone selfupdate [flags] --version string Install the given rclone version (default: latest) ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_serve.md b/docs/content/commands/rclone_serve.md index 80d726cfa..df5c9a9de 100644 --- a/docs/content/commands/rclone_serve.md +++ b/docs/content/commands/rclone_serve.md @@ -28,10 +28,9 @@ rclone serve [opts] [flags] -h, --help help for serve ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. * [rclone serve dlna](/commands/rclone_serve_dlna/) - Serve remote:path over DLNA diff --git a/docs/content/commands/rclone_serve_dlna.md b/docs/content/commands/rclone_serve_dlna.md index a121d76ac..e6ed3d6d2 100644 --- a/docs/content/commands/rclone_serve_dlna.md +++ b/docs/content/commands/rclone_serve_dlna.md @@ -21,6 +21,10 @@ based on media formats or file extensions. Additionally, there is no media transcoding support. This means that some players might show files that they are not able to play back correctly. +Rclone will add external subtitle files (.srt) to videos if they have the same +filename as the video file itself (except the extension), either in the same +directory as the video, or in a "Subs" subdirectory. + ## Server options Use `--addr` to specify which IP address and port the server should @@ -265,6 +269,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -280,6 +289,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -398,8 +431,8 @@ rclone serve dlna remote:path [flags] --addr string The ip:port or :port to bind the DLNA http server to (default ":7879") --announce-interval Duration The interval between SSDP announcements (default 12m0s) --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for dlna --interface stringArray The interface to use for SSDP (repeat as necessary) @@ -411,7 +444,7 @@ rclone serve dlna remote:path [flags] --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -424,6 +457,7 @@ rclone serve dlna remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -431,10 +465,12 @@ rclone serve dlna remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -461,9 +497,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](/commands/rclone_serve/) - Serve a remote over a protocol. diff --git a/docs/content/commands/rclone_serve_docker.md b/docs/content/commands/rclone_serve_docker.md index a5147058d..1a2e86e71 100644 --- a/docs/content/commands/rclone_serve_docker.md +++ b/docs/content/commands/rclone_serve_docker.md @@ -281,6 +281,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -296,6 +301,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -424,9 +453,9 @@ rclone serve docker [flags] --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --forget-state Skip restoring previous state --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) @@ -446,7 +475,7 @@ rclone serve docker [flags] --socket-addr string Address or absolute path (default: /run/docker/plugins/rclone.sock) --socket-gid int GID for unix socket (default: current process GID) (default 1000) --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -459,6 +488,7 @@ rclone serve docker [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -468,10 +498,12 @@ rclone serve docker [flags] --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -498,9 +530,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](/commands/rclone_serve/) - Serve a remote over a protocol. diff --git a/docs/content/commands/rclone_serve_ftp.md b/docs/content/commands/rclone_serve_ftp.md index 7c8bfa6f6..9f22e856c 100644 --- a/docs/content/commands/rclone_serve_ftp.md +++ b/docs/content/commands/rclone_serve_ftp.md @@ -262,6 +262,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -277,6 +282,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -477,8 +506,8 @@ rclone serve ftp remote:path [flags] --auth-proxy string A program to use to create the backend from the auth --cert string TLS PEM key (concatenation of certificate and CA certificate) --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for ftp --key string TLS PEM Private key @@ -491,7 +520,7 @@ rclone serve ftp remote:path [flags] --public-ip string Public IP address to advertise for passive connections --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication (default "anonymous") --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -505,6 +534,7 @@ rclone serve ftp remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -512,10 +542,12 @@ rclone serve ftp remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -542,9 +574,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](/commands/rclone_serve/) - Serve a remote over a protocol. diff --git a/docs/content/commands/rclone_serve_http.md b/docs/content/commands/rclone_serve_http.md index dc45363c2..9206d6f8a 100644 --- a/docs/content/commands/rclone_serve_http.md +++ b/docs/content/commands/rclone_serve_http.md @@ -37,6 +37,7 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. `--addr` may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to accomplish the same. `--server-read-timeout` and `--server-write-timeout` can be used to control the timeouts on the server. Note that this is the total time @@ -69,6 +70,20 @@ certificate authority certificate. values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). +## Socket activation + +Instead of the listening addresses specified above, rclone will listen to all +FDs passed by the service manager, if any (and ignore any arguments passed by --addr`). + +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 over TCP. ### Template `--template` allows a user to specify a custom markup template for HTTP @@ -364,6 +379,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -379,6 +399,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -575,15 +619,15 @@ rclone serve http remote:path [flags] ## Options ``` - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for http --htpasswd string A htpasswd file - if not provided no authentication is done @@ -602,7 +646,7 @@ rclone serve http remote:path [flags] --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --template string User-specified template --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -616,6 +660,7 @@ rclone serve http remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -623,10 +668,12 @@ rclone serve http remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -653,9 +700,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](/commands/rclone_serve/) - Serve a remote over a protocol. diff --git a/docs/content/commands/rclone_serve_nfs.md b/docs/content/commands/rclone_serve_nfs.md index 6d3a264dc..f783876b4 100644 --- a/docs/content/commands/rclone_serve_nfs.md +++ b/docs/content/commands/rclone_serve_nfs.md @@ -13,34 +13,69 @@ Serve the remote as an NFS mount Create an NFS server that serves the given remote over the network. -The primary purpose for this command is to enable [mount command](/commands/rclone_mount/) on recent macOS versions where -installing FUSE is very cumbersome. +This implements an NFSv3 server to serve any rclone remote via NFS. -Since this is running on NFSv3, no authentication method is available. Any client -will be able to access the data. To limit access, you can use serve NFS on loopback address -and rely on secure tunnels (such as SSH). For this reason, by default, a random TCP port is chosen and loopback interface is used for the listening address; -meaning that it is only available to the local machine. If you want other machines to access the -NFS mount over local network, you need to specify the listening address and port using `--addr` flag. +The primary purpose for this command is to enable the [mount +command](/commands/rclone_mount/) on recent macOS versions where +installing FUSE is very cumbersome. -Modifying files through NFS protocol requires VFS caching. Usually you will need to specify `--vfs-cache-mode` -in order to be able to write to the mountpoint (full is recommended). If you don't specify VFS cache mode, -the mount will be read-only. Note also that `--nfs-cache-handle-limit` controls the maximum number of cached file handles stored by the caching handler. -This should not be set too low or you may experience errors when trying to access files. The default is `1000000`, but consider lowering this limit if -the server's system resource usage causes problems. +This server does not implement any authentication so any client will be +able to access the data. To limit access, you can use `serve nfs` on +the loopback address or rely on secure tunnels (such as SSH) or use +firewalling. + +For this reason, by default, a random TCP port is chosen and the +loopback interface is used for the listening address by default; +meaning that it is only available to the local machine. If you want +other machines to access the NFS mount over local network, you need to +specify the listening address and port using the `--addr` flag. + +Modifying files through the NFS protocol requires VFS caching. Usually +you will need to specify `--vfs-cache-mode` in order to be able to +write to the mountpoint (`full` is recommended). If you don't specify +VFS cache mode, the mount will be read-only. + +`--nfs-cache-type` controls the type of the NFS handle cache. By +default this is `memory` where new handles will be randomly allocated +when needed. These are stored in memory. If the server is restarted +the handle cache will be lost and connected NFS clients will get stale +handle errors. + +`--nfs-cache-type disk` uses an on disk NFS handle cache. Rclone +hashes the path of the object and stores it in a file named after the +hash. These hashes are stored on disk the directory controlled by +`--cache-dir` or the exact directory may be specified with +`--nfs-cache-dir`. Using this means that the NFS server can be +restarted at will without affecting the connected clients. + +`--nfs-cache-type symlink` is similar to `--nfs-cache-type disk` in +that it uses an on disk cache, but the cache entries are held as +symlinks. Rclone will use the handle of the underlying file as the NFS +handle which improves performance. This sort of cache can't be backed +up and restored as the underlying handles will change. This is Linux +only. + +`--nfs-cache-handle-limit` controls the maximum number of cached NFS +handles stored by the caching handler. This should not be set too low +or you may experience errors when trying to access files. The default +is `1000000`, but consider lowering this limit if the server's system +resource usage causes problems. This is only used by the `memory` type +cache. To serve NFS over the network use following command: rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full -We specify a specific port that we can use in the mount command: - -To mount the server under Linux/macOS, use the following command: +This specifies a port that can be used in the mount command. To mount +the server under Linux/macOS, use the following command: - mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint + mount -t nfs -o port=$PORT,mountport=$PORT,tcp $HOSTNAME:/ path/to/mountpoint -Where `$PORT` is the same port number we used in the serve nfs command. +Where `$PORT` is the same port number used in the `serve nfs` command +and `$HOSTNAME` is the network address of the machine that `serve nfs` +was run on. -This feature is only available on Unix platforms. +This command is only available on Unix platforms. ## VFS - Virtual File System @@ -274,6 +309,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -289,6 +329,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -406,18 +470,20 @@ rclone serve nfs remote:path [flags] ``` --addr string IPaddress:Port or :Port to bind server to --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for nfs + --nfs-cache-dir string The directory the NFS handle cache will use if set --nfs-cache-handle-limit int max file handles cached simultaneously (min 5) (default 1000000) + --nfs-cache-type memory|disk|symlink Type of NFS handle cache to use (default memory) --no-checksum Don't compare checksums on up/download --no-modtime Don't read/write the modification time (can speed things up) --no-seek Don't allow seeking in files --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -430,6 +496,7 @@ rclone serve nfs remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -437,10 +504,12 @@ rclone serve nfs remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -467,9 +536,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](/commands/rclone_serve/) - Serve a remote over a protocol. diff --git a/docs/content/commands/rclone_serve_restic.md b/docs/content/commands/rclone_serve_restic.md index 25c31c9d6..817b406dd 100644 --- a/docs/content/commands/rclone_serve_restic.md +++ b/docs/content/commands/rclone_serve_restic.md @@ -107,6 +107,7 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. `--addr` may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to accomplish the same. `--server-read-timeout` and `--server-write-timeout` can be used to control the timeouts on the server. Note that this is the total time @@ -139,6 +140,20 @@ certificate authority certificate. values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). +## Socket activation + +Instead of the listening addresses specified above, rclone will listen to all +FDs passed by the service manager, if any (and ignore any arguments passed by --addr`). + +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 over TCP. ### Authentication By default this will serve files without needing a login. @@ -175,7 +190,7 @@ rclone serve restic remote:path [flags] ## Options ``` - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --append-only Disallow deletion of repository data --baseurl string Prefix for URLs - leave blank for root @@ -197,10 +212,9 @@ rclone serve restic remote:path [flags] --user string User name for authentication ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone serve](/commands/rclone_serve/) - Serve a remote over a protocol. diff --git a/docs/content/commands/rclone_serve_s3.md b/docs/content/commands/rclone_serve_s3.md index 89f769d07..888affd37 100644 --- a/docs/content/commands/rclone_serve_s3.md +++ b/docs/content/commands/rclone_serve_s3.md @@ -147,6 +147,33 @@ metadata which will be set as the modification time of the file. Other operations will return error `Unimplemented`. +### Authentication + +By default this will serve files without needing a login. + +You can either use an htpasswd file which can take lots of users, or +set a single username and password with the `--user` and `--pass` flags. + +If no static users are configured by either of the above methods, and client +certificates are required by the `--client-ca` flag passed to the server, the +client certificate common name will be considered as the username. + +Use `--htpasswd /path/to/htpasswd` to provide an htpasswd file. This is +in standard apache format and supports MD5, SHA1 and BCrypt for basic +authentication. Bcrypt is recommended. + +To create an htpasswd file: + + touch htpasswd + htpasswd -B htpasswd user + htpasswd -B htpasswd anotherUser + +The password file can be updated while rclone is running. + +Use `--realm` to set the authentication realm. + +Use `--salt` to change the password hashing salt from the default. + ## Server options Use `--addr` to specify which IP address and port the server should @@ -162,6 +189,7 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. `--addr` may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to accomplish the same. `--server-read-timeout` and `--server-write-timeout` can be used to control the timeouts on the server. Note that this is the total time @@ -194,6 +222,20 @@ certificate authority certificate. values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). +## Socket activation + +Instead of the listening addresses specified above, rclone will listen to all +FDs passed by the service manager, if any (and ignore any arguments passed by --addr`). + +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 over TCP. ## VFS - Virtual File System This command uses the VFS layer. This adapts the cloud storage objects @@ -426,6 +468,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -441,6 +488,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -556,19 +627,21 @@ rclone serve s3 remote:path [flags] ## Options ``` - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-key stringArray Set key pair for v4 authorization: access_key_id,secret_access_key + --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --etag-hash string Which hash to use for the ETag, or auto or blank for off (default "MD5") - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --force-path-style If true use path style access if false use virtual hosted style (default true) (default true) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for s3 + --htpasswd string A htpasswd file - if not provided no authentication is done --key string TLS PEM Private key --max-header-bytes int Maximum size of request header (default 4096) --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") @@ -576,12 +649,16 @@ rclone serve s3 remote:path [flags] --no-cleanup Not to cleanup empty folder after object is deleted --no-modtime Don't read/write the modification time (can speed things up) --no-seek Don't allow seeking in files + --pass string Password for authentication --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access + --realm string Realm for authentication + --salt string Password hashing salt (default "dlPL2MqE") --server-read-timeout Duration Timeout for server reading data (default 1h0m0s) --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) + --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -594,6 +671,7 @@ rclone serve s3 remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -601,10 +679,12 @@ rclone serve s3 remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -631,9 +711,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](/commands/rclone_serve/) - Serve a remote over a protocol. diff --git a/docs/content/commands/rclone_serve_sftp.md b/docs/content/commands/rclone_serve_sftp.md index 2dc502dd4..a662547c4 100644 --- a/docs/content/commands/rclone_serve_sftp.md +++ b/docs/content/commands/rclone_serve_sftp.md @@ -43,6 +43,17 @@ directory. By default the server binds to localhost:2022 - if you want it to be reachable externally then supply `--addr :2022` for example. +This also supports being run with socket activation, in which case it will +listen on the first passed FD. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command: + + systemd-socket-activate -l 2222 -- rclone serve sftp :local:vfs/ + +This will socket-activate rclone on the first connection to port 2222 over TCP. + Note that the default of `--vfs-cache-mode off` is fine for the rclone sftp backend, but it may not be with other SFTP clients. @@ -294,6 +305,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -309,6 +325,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -509,8 +549,8 @@ rclone serve sftp remote:path [flags] --auth-proxy string A program to use to create the backend from the auth --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys") --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for sftp --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate) @@ -523,7 +563,7 @@ rclone serve sftp remote:path [flags] --read-only Only allow read-only access --stdio Run an sftp server on stdin/stdout --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -537,6 +577,7 @@ rclone serve sftp remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -544,10 +585,12 @@ rclone serve sftp remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -574,9 +617,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](/commands/rclone_serve/) - Serve a remote over a protocol. diff --git a/docs/content/commands/rclone_serve_webdav.md b/docs/content/commands/rclone_serve_webdav.md index 9ec439e7f..acd1d69ce 100644 --- a/docs/content/commands/rclone_serve_webdav.md +++ b/docs/content/commands/rclone_serve_webdav.md @@ -52,6 +52,19 @@ Create a new DWORD BasicAuthLevel with value 2. https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint +## Serving over a unix socket + +You can serve the webdav on a unix socket like this: + + rclone serve webdav --addr unix:///tmp/my.socket remote:path + +and connect to it like this using rclone and the webdav backend: + + rclone --webdav-unix-socket /tmp/my.socket --webdav-url http://localhost lsf :webdav: + +Note that there is no authentication on http protocol - this is expected to be +done by the permissions on the socket. + ## Server options Use `--addr` to specify which IP address and port the server should @@ -67,6 +80,7 @@ or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions. `--addr` may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to accomplish the same. `--server-read-timeout` and `--server-write-timeout` can be used to control the timeouts on the server. Note that this is the total time @@ -99,6 +113,20 @@ certificate authority certificate. values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). +## Socket activation + +Instead of the listening addresses specified above, rclone will listen to all +FDs passed by the service manager, if any (and ignore any arguments passed by --addr`). + +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html + +Socket activation can be tested ad-hoc with the `systemd-socket-activate`command + + systemd-socket-activate -l 8000 -- rclone serve + +This will socket-activate rclone on the first connection to port 8000 over TCP. ### Template `--template` allows a user to specify a custom markup template for HTTP @@ -394,6 +422,11 @@ These flags control the chunking: --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once + +The chunking behaves differently depending on the `--vfs-read-chunk-streams` parameter. + +### `--vfs-read-chunk-streams` == 0 Rclone will start reading a chunk of size `--vfs-read-chunk-size`, and then double the size for each read. When `--vfs-read-chunk-size-limit` is @@ -409,6 +442,30 @@ When `--vfs-read-chunk-size-limit 500M` is specified, the result would be Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. +The chunks will not be buffered in memory. + +### `--vfs-read-chunk-streams` > 0 + +Rclone reads `--vfs-read-chunk-streams` chunks of size +`--vfs-read-chunk-size` concurrently. The size for each read will stay +constant. + +This improves performance performance massively on high latency links +or very high bandwidth links to high performance object stores. + +Some experimentation will be needed to find the optimum values of +`--vfs-read-chunk-size` and `--vfs-read-chunk-streams` as these will +depend on the backend in use and the latency to the backend. + +For high performance object stores (eg AWS S3) a reasonable place to +start might be `--vfs-read-chunk-streams 16` and +`--vfs-read-chunk-size 4M`. In testing with AWS S3 the performance +scaled roughly as the `--vfs-read-chunk-streams` setting. + +Similar settings should work for high latency links, but depending on +the latency they may need more `--vfs-read-chunk-streams` in order to +get the throughput. + ## VFS Performance These flags may be used to enable/disable features of the VFS for @@ -605,17 +662,17 @@ rclone serve webdav remote:path [flags] ## Options ``` - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --disable-dir-list Disable HTML directory list on GET request for a directory --etag-hash string Which hash to use for the ETag, or auto or blank for off - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for webdav --htpasswd string A htpasswd file - if not provided no authentication is done @@ -634,7 +691,7 @@ rclone serve webdav remote:path [flags] --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --template string User-specified template --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -648,6 +705,7 @@ rclone serve webdav remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -655,10 +713,12 @@ rclone serve webdav remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -685,9 +745,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone serve](/commands/rclone_serve/) - Serve a remote over a protocol. diff --git a/docs/content/commands/rclone_settier.md b/docs/content/commands/rclone_settier.md index 6e1b6d196..268fd61dc 100644 --- a/docs/content/commands/rclone_settier.md +++ b/docs/content/commands/rclone_settier.md @@ -10,10 +10,9 @@ Changes storage class/tier of objects in remote. ## Synopsis - -rclone settier changes storage tier or class at remote if supported. -Few cloud storage services provides different storage classes on objects, -for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, +Changes storage tier or class at remote if supported. Few cloud storage +services provides different storage classes on objects, for example +AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline etc. Note that, certain tier changes make objects not available to access immediately. @@ -44,10 +43,9 @@ rclone settier tier remote:path [flags] -h, --help help for settier ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_sha1sum.md b/docs/content/commands/rclone_sha1sum.md index db05eb2fa..ad6450d9b 100644 --- a/docs/content/commands/rclone_sha1sum.md +++ b/docs/content/commands/rclone_sha1sum.md @@ -10,7 +10,6 @@ Produces an sha1sum file for all the objects in the path. ## Synopsis - Produces an sha1sum file for all the objects in the path. This is in the same format as the standard sha1sum tool produces. @@ -46,10 +45,12 @@ rclone sha1sum remote:path [flags] --output-file string Output hashsums to a file rather than the terminal ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -76,18 +77,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_size.md b/docs/content/commands/rclone_size.md index c853316d9..7e241f557 100644 --- a/docs/content/commands/rclone_size.md +++ b/docs/content/commands/rclone_size.md @@ -10,7 +10,6 @@ Prints the total size and number of objects in remote:path. ## Synopsis - Counts objects in the path and calculates the total size. Prints the result to standard output. @@ -41,10 +40,12 @@ rclone size remote:path [flags] --json Format output as JSON ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -71,18 +72,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_sync.md b/docs/content/commands/rclone_sync.md index 10da45199..82aa00056 100644 --- a/docs/content/commands/rclone_sync.md +++ b/docs/content/commands/rclone_sync.md @@ -9,7 +9,6 @@ Make source and dest identical, modifying destination only. ## Synopsis - Sync the source to the destination, changing the destination only. Doesn't transfer files that are identical on source and destination, testing by size and modification time or MD5SUM. @@ -118,15 +117,17 @@ rclone sync source:path dest:path [flags] -t, --timeformat string Specify a custom time format, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05) ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Copy Options +### Copy Options -Flags for anything which can Copy a file. +Flags for anything which can copy a file ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -158,9 +159,9 @@ Flags for anything which can Copy a file. -u, --update Skip files that are newer on the destination ``` -## Sync Options +### Sync Options -Flags just used for `rclone sync`. +Flags used for sync commands ``` --backup-dir string Make backups into hierarchy based in DIR @@ -177,9 +178,9 @@ Flags just used for `rclone sync`. --track-renames-strategy string Strategies to use when synchronizing using track-renames hash|modtime|leaf (default "hash") ``` -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -187,9 +188,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -216,18 +217,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_test.md b/docs/content/commands/rclone_test.md index 8af7509bf..0aaddb775 100644 --- a/docs/content/commands/rclone_test.md +++ b/docs/content/commands/rclone_test.md @@ -28,10 +28,9 @@ so reading their documentation first is recommended. -h, --help help for test ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. * [rclone test changenotify](/commands/rclone_test_changenotify/) - Log any change notify requests for the remote passed in. diff --git a/docs/content/commands/rclone_test_changenotify.md b/docs/content/commands/rclone_test_changenotify.md index 92c5179ca..1efc25554 100644 --- a/docs/content/commands/rclone_test_changenotify.md +++ b/docs/content/commands/rclone_test_changenotify.md @@ -19,10 +19,9 @@ rclone test changenotify remote: [flags] --poll-interval Duration Time to wait between polling for changes (default 10s) ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](/commands/rclone_test/) - Run a test command diff --git a/docs/content/commands/rclone_test_histogram.md b/docs/content/commands/rclone_test_histogram.md index 6a70b7059..b3b3088ab 100644 --- a/docs/content/commands/rclone_test_histogram.md +++ b/docs/content/commands/rclone_test_histogram.md @@ -27,10 +27,9 @@ rclone test histogram [remote:path] [flags] -h, --help help for histogram ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](/commands/rclone_test/) - Run a test command diff --git a/docs/content/commands/rclone_test_info.md b/docs/content/commands/rclone_test_info.md index 2cfe34855..2a9ccf16f 100644 --- a/docs/content/commands/rclone_test_info.md +++ b/docs/content/commands/rclone_test_info.md @@ -10,10 +10,10 @@ Discovers file name or other limitations for paths. ## Synopsis -rclone info discovers what filenames and upload methods are possible -to write to the paths passed in and how long they can be. It can take some -time. It will write test files into the remote:path passed in. It outputs -a bit of go code for each one. +Discovers what filenames and upload methods are possible to write to the +paths passed in and how long they can be. It can take some time. It will +write test files into the remote:path passed in. It outputs a bit of go +code for each one. **NB** this can create undeletable files and other hazards - use with care @@ -37,10 +37,9 @@ rclone test info [remote:path]+ [flags] --write-json string Write results to file ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](/commands/rclone_test/) - Run a test command diff --git a/docs/content/commands/rclone_test_makefile.md b/docs/content/commands/rclone_test_makefile.md index a0dd91ec6..82e5da0bb 100644 --- a/docs/content/commands/rclone_test_makefile.md +++ b/docs/content/commands/rclone_test_makefile.md @@ -24,10 +24,9 @@ rclone test makefile []+ [flags] --zero Fill files with ASCII 0x00 ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](/commands/rclone_test/) - Run a test command diff --git a/docs/content/commands/rclone_test_makefiles.md b/docs/content/commands/rclone_test_makefiles.md index be1eb0f21..237ba1c9d 100644 --- a/docs/content/commands/rclone_test_makefiles.md +++ b/docs/content/commands/rclone_test_makefiles.md @@ -31,10 +31,9 @@ rclone test makefiles [flags] --zero Fill files with ASCII 0x00 ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](/commands/rclone_test/) - Run a test command diff --git a/docs/content/commands/rclone_test_memory.md b/docs/content/commands/rclone_test_memory.md index c79aaea3d..50b985824 100644 --- a/docs/content/commands/rclone_test_memory.md +++ b/docs/content/commands/rclone_test_memory.md @@ -18,10 +18,9 @@ rclone test memory remote:path [flags] -h, --help help for memory ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone test](/commands/rclone_test/) - Run a test command diff --git a/docs/content/commands/rclone_touch.md b/docs/content/commands/rclone_touch.md index 1368ad294..f1725579f 100644 --- a/docs/content/commands/rclone_touch.md +++ b/docs/content/commands/rclone_touch.md @@ -10,7 +10,6 @@ Create new file or change file modification time. ## Synopsis - Set the modification time on file(s) as specified by remote:path to have the current time. @@ -46,10 +45,12 @@ rclone touch remote:path [flags] -t, --timestamp string Use specified time instead of the current time of day ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Important Options +### Important Options -Important flags useful for most commands. +Important flags useful for most commands ``` -n, --dry-run Do a trial run with no permanent changes @@ -57,9 +58,9 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) ``` -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -86,18 +87,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_tree.md b/docs/content/commands/rclone_tree.md index 43bb3a087..0460b61b3 100644 --- a/docs/content/commands/rclone_tree.md +++ b/docs/content/commands/rclone_tree.md @@ -10,9 +10,7 @@ List the contents of the remote in a tree like fashion. ## Synopsis - -rclone tree lists the contents of a remote in a similar way to the -unix tree command. +Lists the contents of a remote in a similar way to the unix tree command. For example @@ -67,10 +65,12 @@ rclone tree remote:path [flags] --version Sort files alphanumerically by version ``` +Options shared with other commands are described next. +See the [global flags page](/flags/) for global options not listed here. -## Filter Options +### Filter Options -Flags for filtering directory listings. +Flags for filtering directory listings ``` --delete-excluded Delete files on dest excluded from sync @@ -97,18 +97,16 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) ``` -## Listing Options +### Listing Options -Flags for listing directories. +Flags for listing directories ``` --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) --fast-list Use recursive list if available; uses more memory but fewer transactions ``` -See the [global flags page](/flags/) for global options not listed here. - -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/commands/rclone_version.md b/docs/content/commands/rclone_version.md index 0896cda13..127ded349 100644 --- a/docs/content/commands/rclone_version.md +++ b/docs/content/commands/rclone_version.md @@ -10,7 +10,6 @@ Show the version number. ## Synopsis - Show the rclone version number, the go version, the build target OS and architecture, the runtime OS and kernel version and bitness, build tags and the type of executable (static or dynamic). @@ -60,10 +59,9 @@ rclone version [flags] -h, --help help for version ``` - See the [global flags page](/flags/) for global options not listed here. -# SEE ALSO +## See Also * [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. diff --git a/docs/content/flags.md b/docs/content/flags.md index fca12d85c..c2908561a 100644 --- a/docs/content/flags.md +++ b/docs/content/flags.md @@ -1,6 +1,7 @@ --- title: "Global Flags" description: "Rclone Global Flags" +# autogenerated - DO NOT EDIT --- # Global Flags @@ -11,12 +12,12 @@ split into groups. ## Copy -Flags for anything which can Copy a file. +Flags for anything which can copy a file. ``` --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -51,7 +52,7 @@ Flags for anything which can Copy a file. ## Sync -Flags just used for `rclone sync`. +Flags used for sync commands. ``` --backup-dir string Make backups into hierarchy based in DIR @@ -82,7 +83,7 @@ Important flags useful for most commands. ## Check -Flags used for `rclone check`. +Flags used for check commands. ``` --max-backlog int Maximum number of objects in sync or check backlog (default 10000) @@ -91,7 +92,7 @@ Flags used for `rclone check`. ## Networking -General networking and HTTP stuff. +Flags for general networking and HTTP stuff. ``` --bind string Local address to bind to for outgoing connections, IPv4, IPv6 or name @@ -101,7 +102,7 @@ General networking and HTTP stuff. --client-cert string Client SSL certificate (PEM) for mutual TLS auth --client-key string Client SSL private key (PEM) for mutual TLS auth --contimeout Duration Connect timeout (default 1m0s) - --disable-http-keep-alives Disable HTTP keep-alives and use each connection once. + --disable-http-keep-alives Disable HTTP keep-alives and use each connection once --disable-http2 Disable HTTP/2 in the global transport --dscp string Set DSCP value to connections, value or name, e.g. CS1, LE, DF, AF21 --expect-continue-timeout Duration Timeout when using expect / 100-continue in HTTP (default 1s) @@ -114,7 +115,7 @@ General networking and HTTP stuff. --tpslimit float Limit HTTP transactions per second to this --tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --use-cookies Enable session cookiejar - --user-agent string Set the user-agent to a specified string (default "rclone/v1.67.0") + --user-agent string Set the user-agent to a specified string (default "rclone/v1.68.0") ``` @@ -131,7 +132,7 @@ Flags helpful for increasing performance. ## Config -General configuration of rclone. +Flags for general configuration of rclone. ``` --ask-password Allow prompt for password for encrypted configuration (default true) @@ -215,7 +216,7 @@ Flags for listing directories. ## Logging -Logging and statistics. +Flags for logging and statistics. ``` --log-file string Log everything to this file @@ -234,7 +235,7 @@ Logging and statistics. --stats-one-line-date-format string Enable --stats-one-line-date and use custom formatted date: Enclose date string in double quotes ("), see https://golang.org/pkg/time/#Time.Format --stats-unit string Show data rate in stats as either 'bits' or 'bytes' per second (default "bytes") --syslog Use Syslog for logging - --syslog-facility string Facility for syslog, e.g. KERN,USER,... (default "DAEMON") + --syslog-facility string Facility for syslog, e.g. KERN,USER (default "DAEMON") --use-json-log Use json log format -v, --verbose count Print lots more stuff (repeat for more) ``` @@ -263,12 +264,12 @@ Flags to control the Remote Control API. ``` --rc Enable the remote control server - --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572]) + --rc-addr stringArray IPaddress:Port or :Port to bind server to (default ["localhost:5572"]) --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from --rc-baseurl string Prefix for URLs - leave blank for root --rc-cert string TLS PEM key (concatenation of certificate and CA certificate) --rc-client-ca string Client certificate authority to verify clients with - --rc-enable-metrics Enable prometheus metrics on /metrics + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server --rc-files string Path to local files to serve on the HTTP server --rc-htpasswd string A htpasswd file - if not provided no authentication is done --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s) @@ -294,9 +295,34 @@ Flags to control the Remote Control API. ``` +## Metrics + +Flags to control the Metrics HTTP endpoint.. + +``` + --metrics-addr stringArray IPaddress:Port or :Port to bind metrics server to (default [""]) + --metrics-allow-origin string Origin which cross-domain request (CORS) can be executed from + --metrics-baseurl string Prefix for URLs - leave blank for root + --metrics-cert string TLS PEM key (concatenation of certificate and CA certificate) + --metrics-client-ca string Client certificate authority to verify clients with + --metrics-htpasswd string A htpasswd file - if not provided no authentication is done + --metrics-key string TLS PEM Private key + --metrics-max-header-bytes int Maximum size of request header (default 4096) + --metrics-min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") + --metrics-pass string Password for authentication + --metrics-realm string Realm for authentication + --metrics-salt string Password hashing salt (default "dlPL2MqE") + --metrics-server-read-timeout Duration Timeout for server reading data (default 1h0m0s) + --metrics-server-write-timeout Duration Timeout for server writing data (default 1h0m0s) + --metrics-template string User-specified template + --metrics-user string User name for authentication + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server +``` + + ## Backend -Backend only flags. These can be set in the config file also. +Backend-only flags (these can be set in the config file also). ``` --alias-description string Description of the remote @@ -520,6 +546,12 @@ Backend only flags. These can be set in the config file also. --filefabric-token-expiry string Token expiry time --filefabric-url string URL of the Enterprise File Fabric to connect to --filefabric-version string Version read from the file fabric + --filescom-api-key string The API key used to authenticate with Files.com + --filescom-description string Description of the remote + --filescom-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot) + --filescom-password string The password used to authenticate with Files.com (obscured) + --filescom-site string Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com) + --filescom-username string The username used to authenticate with Files.com --ftp-ask-password Allow asking for FTP password when needed --ftp-close-timeout Duration Maximum time to wait for a response to close (default 1m0s) --ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited @@ -563,6 +595,12 @@ Backend only flags. These can be set in the config file also. --gcs-token string OAuth Access Token as a JSON blob --gcs-token-url string Token server url --gcs-user-project string User project + --gofile-access-token string API Access token + --gofile-account-id string Account ID + --gofile-description string Description of the remote + --gofile-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftPeriod,RightPeriod,InvalidUtf8,Dot,Exclamation) + --gofile-list-chunk int Number of items to list in each call (default 1000) + --gofile-root-folder-id string ID of the root folder --gphotos-auth-url string Auth server URL --gphotos-batch-commit-timeout Duration Max time to wait for a batch to finish committing (default 10m0s) --gphotos-batch-mode string Upload file batching sync|async|off (default "sync") @@ -654,6 +692,7 @@ Backend only flags. These can be set in the config file also. --local-description string Description of the remote --local-encoding Encoding The encoding for the backend (default Slash,Dot) --local-no-check-updated Don't check to see if the files change during upload + --local-no-clone Disable reflink cloning for server-side copies --local-no-preallocate Disable preallocation of disk space for transferred files --local-no-set-modtime Disable setting modtime --local-no-sparse Disable sparse files for multi-thread downloads @@ -770,6 +809,10 @@ Backend only flags. These can be set in the config file also. --pikpak-upload-concurrency int Concurrency for multipart uploads (default 5) --pikpak-use-trash Send files to the trash instead of deleting permanently (default true) --pikpak-user string Pikpak username + --pixeldrain-api-key string API key for your pixeldrain account + --pixeldrain-api-url string The API endpoint to connect to. In the vast majority of cases it's fine to leave (default "https://pixeldrain.com/api") + --pixeldrain-description string Description of the remote + --pixeldrain-root-folder-id string Root of the filesystem to use (default "me") --premiumizeme-auth-url string Auth server URL --premiumizeme-client-id string OAuth Client Id --premiumizeme-client-secret string OAuth Client Secret @@ -844,6 +887,7 @@ Backend only flags. These can be set in the config file also. --s3-provider string Choose your S3 provider --s3-region string Region to connect to --s3-requester-pays Enables requester pays option when interacting with S3 bucket + --s3-sdk-log-mode Bits Set to debug the SDK (default Off) --s3-secret-access-key string AWS Secret Access Key (password) --s3-server-side-encryption string The server-side encryption algorithm used when storing this object in S3 --s3-session-token string An AWS session token @@ -854,7 +898,6 @@ Backend only flags. These can be set in the config file also. --s3-sse-customer-key-md5 string If using SSE-C you may provide the secret encryption key MD5 checksum (optional) --s3-sse-kms-key-id string If using KMS ID you must provide the ARN of Key --s3-storage-class string The storage class to use when storing new objects in S3 - --s3-sts-endpoint string Endpoint for STS --s3-upload-concurrency int Concurrency for multipart uploads and copies (default 4) --s3-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi) --s3-use-accelerate-endpoint If true use the AWS S3 accelerated endpoint @@ -864,6 +907,7 @@ Backend only flags. These can be set in the config file also. --s3-use-multipart-etag Tristate Whether to use ETag in multipart uploads for verification (default unset) --s3-use-multipart-uploads Tristate Set if rclone should use multipart uploads (default unset) --s3-use-presigned-request Whether to use a presigned request or PutObject for single part uploads + --s3-use-unsigned-payload Tristate Whether to use an unsigned payload in PutObject (default unset) --s3-v2-auth If true use v2 authentication --s3-version-at Time Show file versions as they were at the specified time (default off) --s3-version-deleted Show deleted file markers when using versions @@ -972,10 +1016,12 @@ Backend only flags. These can be set in the config file also. --swift-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8) --swift-endpoint-type string Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) (default "public") --swift-env-auth Get swift credentials from environment variables in standard OpenStack form + --swift-fetch-until-empty-page When paginating, always fetch unless we received an empty page --swift-key string API key or password (OS_PASSWORD) --swift-leave-parts-on-error If true avoid calling abort upload on a failure --swift-no-chunk Don't chunk files during streaming upload --swift-no-large-objects Disable support for static and dynamic large objects + --swift-partial-page-fetch-threshold int When paginating, fetch if the current page is within this percentage of the limit --swift-region string Region name - optional (OS_REGION_NAME) --swift-storage-policy string The storage policy to use when creating a new container --swift-storage-url string Storage URL - optional (OS_STORAGE_URL) @@ -1013,6 +1059,7 @@ Backend only flags. These can be set in the config file also. --webdav-owncloud-exclude-shares Exclude ownCloud shares --webdav-pacer-min-sleep Duration Minimum time to sleep between API calls (default 10ms) --webdav-pass string Password (obscured) + --webdav-unix-socket string Path to a unix domain socket to dial to, instead of opening a TCP connection directly --webdav-url string URL of http host to connect to --webdav-user string User name --webdav-vendor string Name of the WebDAV site/service/software you are using @@ -1022,6 +1069,7 @@ Backend only flags. These can be set in the config file also. --yandex-description string Description of the remote --yandex-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot) --yandex-hard-delete Delete files permanently rather than putting them into the trash + --yandex-spoof-ua Set the user agent to match an official version of the yandex disk client. May help with upload performance (default true) --yandex-token string OAuth Access Token as a JSON blob --yandex-token-url string Token server url --zoho-auth-url string Auth server URL diff --git a/docs/content/gofile.md b/docs/content/gofile.md index 8ce0d7036..45fe18f7c 100644 --- a/docs/content/gofile.md +++ b/docs/content/gofile.md @@ -202,6 +202,17 @@ Properties: - Type: string - Required: false +#### --gofile-list-chunk + +Number of items to list in each call + +Properties: + +- Config: list_chunk +- Env Var: RCLONE_GOFILE_LIST_CHUNK +- Type: int +- Default: 1000 + #### --gofile-encoding The encoding for the backend. diff --git a/docs/content/local.md b/docs/content/local.md index 8aeb35151..9d5c8363c 100644 --- a/docs/content/local.md +++ b/docs/content/local.md @@ -512,6 +512,32 @@ Properties: - Type: bool - Default: false +#### --local-no-clone + +Disable reflink cloning for server-side copies. + +Normally, for local-to-local transfers, rclone will "clone" the file when +possible, and fall back to "copying" only when cloning is not supported. + +Cloning creates a shallow copy (or "reflink") which initially shares blocks with +the original file. Unlike a "hardlink", the two files are independent and +neither will affect the other if subsequently modified. + +Cloning is usually preferable to copying, as it is much faster and is +deduplicated by default (i.e. having two identical files does not consume more +storage than having just one.) However, for use cases where data redundancy is +preferable, --local-no-clone can be used to disable cloning and force "deep" copies. + +Currently, cloning is only supported when using APFS on macOS (support for other +platforms may be added in the future.) + +Properties: + +- Config: no_clone +- Env Var: RCLONE_LOCAL_NO_CLONE +- Type: bool +- Default: false + #### --local-no-preallocate Disable preallocation of disk space for transferred files. diff --git a/docs/content/rc.md b/docs/content/rc.md index 80c02a591..7a8ef34e4 100644 --- a/docs/content/rc.md +++ b/docs/content/rc.md @@ -1716,6 +1716,11 @@ Returns: Returns an object where keys are option block names and values are an object with the current option values in. +Parameters: + +- blocks: optional string of comma separated blocks to include + - all are included if this is missing or "" + Note that these are the global options which are unaffected by use of the _config and _filter parameters. If you wish to read the parameters set in _config then use options/config and for _filter use options/filter. @@ -1728,6 +1733,11 @@ map to the external options very easily with a few exceptions. Returns an object where keys are option block names and values are an array of objects with info about each options. +Parameters: + +- blocks: optional string of comma separated blocks to include + - all are included if this is missing or "" + These objects are in the same format as returned by "config/providers". They are described in the [option blocks](#option-blocks) section. @@ -2014,6 +2024,73 @@ If poll-interval is updated or disabled temporarily, some changes might not get picked up by the polling function, depending on the used remote. +This command takes an "fs" parameter. If this parameter is not +supplied and if there is only one VFS in use then that VFS will be +used. If there is more than one VFS in use then the "fs" parameter +must be supplied. + +### vfs/queue: Queue info for a VFS. {#vfs-queue} + +This returns info about the upload queue for the selected VFS. + +This is only useful if `--vfs-cache-mode` > off. If you call it when +the `--vfs-cache-mode` is off, it will return an empty result. + + { + "queued": // an array of files queued for upload + [ + { + "name": "file", // string: name (full path) of the file, + "id": 123, // integer: id of this item in the queue, + "size": 79, // integer: size of the file in bytes + "expiry": 1.5 // float: time until file is eligible for transfer, lowest goes first + "tries": 1, // integer: number of times we have tried to upload + "delay": 5.0, // float: seconds between upload attempts + "uploading": false, // boolean: true if item is being uploaded + }, + ], + } + +The `expiry` time is the time until the file is elegible for being +uploaded in floating point seconds. This may go negative. As rclone +only transfers `--transfers` files at once, only the lowest +`--transfers` expiry times will have `uploading` as `true`. So there +may be files with negative expiry times for which `uploading` is +`false`. + + +This command takes an "fs" parameter. If this parameter is not +supplied and if there is only one VFS in use then that VFS will be +used. If there is more than one VFS in use then the "fs" parameter +must be supplied. + +### vfs/queue-set-expiry: Set the expiry time for an item queued for upload. {#vfs-queue-set-expiry} + +Use this to adjust the `expiry` time for an item in the upload queue. +You will need to read the `id` of the item using `vfs/queue` before +using this call. + +You can then set `expiry` to a floating point number of seconds from +now when the item is eligible for upload. If you want the item to be +uploaded as soon as possible then set it to a large negative number (eg +-1000000000). If you want the upload of the item to be delayed +for a long time then set it to a large positive number. + +Setting the `expiry` of an item which has already has started uploading +will have no effect - the item will carry on being uploaded. + +This will return an error if called with `--vfs-cache-mode` off or if +the `id` passed is not found. + +This takes the following parameters + +- `fs` - select the VFS in use (optional) +- `id` - a numeric ID as returned from `vfs/queue` +- `expiry` - a new expiry time as floating point seconds + +This returns an empty result on success, or an error. + + This command takes an "fs" parameter. If this parameter is not supplied and if there is only one VFS in use then that VFS will be used. If there is more than one VFS in use then the "fs" parameter diff --git a/docs/content/s3.md b/docs/content/s3.md index c7a6b946b..db52f5c26 100644 --- a/docs/content/s3.md +++ b/docs/content/s3.md @@ -1409,6 +1409,10 @@ Some providers (e.g. AWS, Aliyun OSS, Netease COS, or Tencent COS) require this false - rclone will do this automatically based on the provider setting. +Note that if your bucket isn't a valid DNS name, i.e. has '.' or '_' in, +you'll need to set this to true. + + Properties: - Config: force_path_style @@ -1544,8 +1548,6 @@ It can also be needed if the user you are using does not have bucket creation permissions. Before v1.52.0 this would have passed silently due to a bug. -We recommend setting `no_check_bucket = true` when the provider is Alibaba, otherwise the bucket ACL may changed after every `rclone mkdir`. See [#7889](https://github.com/rclone/rclone/issues/7889). - Properties: @@ -1700,6 +1702,24 @@ Properties: - Type: Tristate - Default: unset +#### --s3-use-unsigned-payload + +Whether to use an unsigned payload in PutObject + +Rclone has to avoid the AWS SDK seeking the body when calling +PutObject. The AWS provider can add checksums in the trailer to avoid +seeking but other providers can't. + +This should be true, false or left unset to use the default for the provider. + + +Properties: + +- Config: use_unsigned_payload +- Env Var: RCLONE_S3_USE_UNSIGNED_PAYLOAD +- Type: Tristate +- Default: unset + #### --s3-use-presigned-request Whether to use a presigned request or PutObject for single part uploads @@ -1857,7 +1877,7 @@ Properties: #### --s3-sts-endpoint -Endpoint for STS. +Endpoint for STS (deprecated). Leave blank if using AWS to use the default endpoint for the region. @@ -1920,6 +1940,33 @@ Properties: - Type: Tristate - Default: unset +#### --s3-sdk-log-mode + +Set to debug the SDK + +This can be set to a comma separated list of the following functions: + +- `Signing` +- `Retries` +- `Request` +- `RequestWithBody` +- `Response` +- `ResponseWithBody` +- `DeprecatedUsage` +- `RequestEventMessage` +- `ResponseEventMessage` + +Use `Off` to disable and `All` to set all log levels. You will need to +use `-vv` to see the debug level logs. + + +Properties: + +- Config: sdk_log_mode +- Env Var: RCLONE_S3_SDK_LOG_MODE +- Type: Bits +- Default: Off + #### --s3-description Description of the remote. @@ -1968,18 +2015,19 @@ These can be run on a running backend using the rc command ### restore -Restore objects from GLACIER to normal storage +Restore objects from GLACIER or INTELLIGENT-TIERING archive tier rclone backend restore remote: [options] [+] -This command can be used to restore one or more objects from GLACIER -to normal storage. +This command can be used to restore one or more objects from GLACIER to normal storage +or from INTELLIGENT-TIERING Archive Access / Deep Archive Access tier to the Frequent Access tier. Usage Examples: rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS + rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY This flag also obeys the filters. Test first with --interactive/-i or --dry-run flags @@ -2009,17 +2057,17 @@ if not. Options: - "description": The optional description for the job. -- "lifetime": Lifetime of the active copy in days +- "lifetime": Lifetime of the active copy in days, ignored for INTELLIGENT-TIERING storage - "priority": Priority of restore: Standard|Expedited|Bulk ### restore-status -Show the restore status for objects being restored from GLACIER to normal storage +Show the restore status for objects being restored from GLACIER or INTELLIGENT-TIERING storage rclone backend restore-status remote: [options] [+] -This command can be used to show the status for objects being restored from GLACIER -to normal storage. +This command can be used to show the status for objects being restored from GLACIER to normal storage +or from INTELLIGENT-TIERING Archive Access / Deep Archive Access tier to the Frequent Access tier. Usage Examples: @@ -2049,6 +2097,15 @@ It returns a list of status dictionaries. "RestoreExpiryDate": "2023-09-06T12:29:19+01:00" }, "StorageClass": "DEEP_ARCHIVE" + }, + { + "Remote": "test.gz", + "VersionID": null, + "RestoreStatus": { + "IsRestoreInProgress": true, + "RestoreExpiryDate": "null" + }, + "StorageClass": "INTELLIGENT_TIERING" } ] diff --git a/docs/content/sftp.md b/docs/content/sftp.md index 88d2eb76e..2ba4a48a7 100644 --- a/docs/content/sftp.md +++ b/docs/content/sftp.md @@ -446,7 +446,15 @@ Properties: Raw PEM-encoded private key. -If specified, will override key_file parameter. +Note that this should be on a single line with line endings replaced with '\n', eg + + key_pem = -----BEGIN RSA PRIVATE KEY-----\nMaMbaIXtE\n0gAMbMbaSsd\nMbaass\n-----END RSA PRIVATE KEY----- + +This will generate the single line correctly: + + awk '{printf "%s\\n", $0}' < ~/.ssh/id_rsa + +If specified, it will override the key_file parameter. Properties: @@ -903,13 +911,13 @@ Maximum number of SFTP simultaneous connections, 0 for unlimited. Note that setting this is very likely to cause deadlocks so it should be used with care. -If you are doing a sync or copy then make sure concurrency is one more +If you are doing a sync or copy then make sure connections is one more than the sum of `--transfers` and `--checkers`. If you use `--check-first` then it just needs to be one more than the maximum of `--checkers` and `--transfers`. -So for `concurrency 3` you'd use `--checkers 2 --transfers 2 +So for `connections 3` you'd use `--checkers 2 --transfers 2 --check-first` or `--checkers 1 --transfers 1`. diff --git a/docs/content/swift.md b/docs/content/swift.md index f15ba7e81..700263a3c 100644 --- a/docs/content/swift.md +++ b/docs/content/swift.md @@ -508,6 +508,48 @@ Properties: - Type: bool - Default: false +#### --swift-fetch-until-empty-page + +When paginating, always fetch unless we received an empty page. + +Consider using this option if rclone listings show fewer objects +than expected, or if repeated syncs copy unchanged objects. + +It is safe to enable this, but rclone may make more API calls than +necessary. + +This is one of a pair of workarounds to handle implementations +of the Swift API that do not implement pagination as expected. See +also "partial_page_fetch_threshold". + +Properties: + +- Config: fetch_until_empty_page +- Env Var: RCLONE_SWIFT_FETCH_UNTIL_EMPTY_PAGE +- Type: bool +- Default: false + +#### --swift-partial-page-fetch-threshold + +When paginating, fetch if the current page is within this percentage of the limit. + +Consider using this option if rclone listings show fewer objects +than expected, or if repeated syncs copy unchanged objects. + +It is safe to enable this, but rclone may make more API calls than +necessary. + +This is one of a pair of workarounds to handle implementations +of the Swift API that do not implement pagination as expected. See +also "fetch_until_empty_page". + +Properties: + +- Config: partial_page_fetch_threshold +- Env Var: RCLONE_SWIFT_PARTIAL_PAGE_FETCH_THRESHOLD +- Type: int +- Default: 0 + #### --swift-chunk-size Above this size files will be chunked. diff --git a/rclone.1 b/rclone.1 index 4b8022693..12a2f7604 100644 --- a/rclone.1 +++ b/rclone.1 @@ -1,7 +1,7 @@ .\"t .\" Automatically generated by Pandoc 2.9.2.1 .\" -.TH "rclone" "1" "Jun 14, 2024" "User Manual" "" +.TH "rclone" "1" "Sep 08, 2024" "User Manual" "" .hy .SH Rclone syncs your files to cloud storage .PP @@ -178,8 +178,12 @@ Enterprise File Fabric .IP \[bu] 2 Fastmail Files .IP \[bu] 2 +Files.com +.IP \[bu] 2 FTP .IP \[bu] 2 +Gofile +.IP \[bu] 2 Google Cloud Storage .IP \[bu] 2 Google Drive @@ -256,6 +260,8 @@ Petabox .IP \[bu] 2 PikPak .IP \[bu] 2 +Pixeldrain +.IP \[bu] 2 premiumize.me .IP \[bu] 2 put.io @@ -942,6 +948,14 @@ executable will be in your GOPATH bin folder go install github.com/rclone/rclone\[at]latest \f[R] .fi +.PP +In some situations, rclone executable size might be too big for +deployment in very restricted environments when all backends with large +SDKs are included. +To limit binary size unused backends can be commented out in +\f[C]backends/all/all.go\f[R] and unused commands in +\f[C]cmd/all/all.go\f[R] before building with \f[C]go build\f[R] or +\f[C]make\f[R] .SS Ansible installation .PP This can be done with Stefan Weichinger\[aq]s ansible @@ -1216,8 +1230,12 @@ Dropbox (https://rclone.org/dropbox/) .IP \[bu] 2 Enterprise File Fabric (https://rclone.org/filefabric/) .IP \[bu] 2 +Files.com (https://rclone.org/filescom/) +.IP \[bu] 2 FTP (https://rclone.org/ftp/) .IP \[bu] 2 +Gofile (https://rclone.org/gofile/) +.IP \[bu] 2 Google Cloud Storage (https://rclone.org/googlecloudstorage/) .IP \[bu] 2 Google Drive (https://rclone.org/drive/) @@ -1229,6 +1247,8 @@ remotes .IP \[bu] 2 HDFS (https://rclone.org/hdfs/) .IP \[bu] 2 +Hetzner Storage Box (https://rclone.org/sftp/#hetzner-storage-box) +.IP \[bu] 2 HiDrive (https://rclone.org/hidrive/) .IP \[bu] 2 HTTP (https://rclone.org/http/) @@ -1264,6 +1284,8 @@ Pcloud (https://rclone.org/pcloud/) .IP \[bu] 2 PikPak (https://rclone.org/pikpak/) .IP \[bu] 2 +Pixeldrain (https://rclone.org/pixeldrain/) +.IP \[bu] 2 premiumize.me (https://rclone.org/premiumizeme/) .IP \[bu] 2 put.io (https://rclone.org/putio/) @@ -1274,6 +1296,8 @@ QingStor (https://rclone.org/qingstor/) .IP \[bu] 2 Quatrix by Maytech (https://rclone.org/quatrix/) .IP \[bu] 2 +rsync.net (https://rclone.org/sftp/#rsync-net) +.IP \[bu] 2 Seafile (https://rclone.org/seafile/) .IP \[bu] 2 SFTP (https://rclone.org/sftp/) @@ -1380,7 +1404,7 @@ rclone config [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -1401,6 +1425,10 @@ Dump the config file as JSON. rclone config edit (https://rclone.org/commands/rclone_config_edit/) - Enter an interactive configuration session. .IP \[bu] 2 +rclone config +encryption (https://rclone.org/commands/rclone_config_encryption/) - +set, remove and check the encryption for the config file +.IP \[bu] 2 rclone config file (https://rclone.org/commands/rclone_config_file/) - Show path of configuration file in use. .IP \[bu] 2 @@ -1544,15 +1572,19 @@ rclone copy source:path dest:path [flags] -h, --help help for copy \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Copy Options .PP -Flags for anything which can Copy a file. +Flags for anything which can copy a file .IP .nf \f[C] --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -1586,7 +1618,7 @@ Flags for anything which can Copy a file. .fi .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -1597,7 +1629,7 @@ Important flags useful for most commands. .fi .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -1627,7 +1659,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -1635,10 +1667,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -1787,15 +1816,19 @@ rclone sync source:path dest:path [flags] -t, --timeformat string Specify a custom time format, or \[aq]max\[aq] for max precision supported by remote (default: 2006-01-02 15:04:05) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Copy Options .PP -Flags for anything which can Copy a file. +Flags for anything which can copy a file .IP .nf \f[C] --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -1829,7 +1862,7 @@ Flags for anything which can Copy a file. .fi .SS Sync Options .PP -Flags just used for \f[C]rclone sync\f[R]. +Flags used for sync commands .IP .nf \f[C] @@ -1849,7 +1882,7 @@ Flags just used for \f[C]rclone sync\f[R]. .fi .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -1860,7 +1893,7 @@ Important flags useful for most commands. .fi .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -1890,7 +1923,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -1898,10 +1931,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -1963,15 +1993,19 @@ rclone move source:path dest:path [flags] -h, --help help for move \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Copy Options .PP -Flags for anything which can Copy a file. +Flags for anything which can copy a file .IP .nf \f[C] --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -2005,7 +2039,7 @@ Flags for anything which can Copy a file. .fi .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -2016,7 +2050,7 @@ Important flags useful for most commands. .fi .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -2046,7 +2080,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -2054,10 +2088,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2119,9 +2150,13 @@ rclone delete remote:path [flags] --rmdirs rmdirs removes empty directories but leaves root intact \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -2132,7 +2167,7 @@ Important flags useful for most commands. .fi .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -2162,7 +2197,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -2170,10 +2205,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2206,9 +2238,13 @@ rclone purge remote:path [flags] -h, --help help for purge \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -2217,10 +2253,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2240,9 +2273,13 @@ rclone mkdir remote:path [flags] -h, --help help for mkdir \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -2251,10 +2288,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2285,9 +2319,13 @@ rclone rmdir remote:path [flags] -h, --help help for rmdir \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -2296,10 +2334,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2389,9 +2424,13 @@ rclone check source:path dest:path [flags] --one-way Check one way only, source files must exist on remote \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Check Options .PP -Flags used for \f[C]rclone check\f[R]. +Flags used for check commands .IP .nf \f[C] @@ -2400,7 +2439,7 @@ Flags used for \f[C]rclone check\f[R]. .fi .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -2430,7 +2469,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -2438,10 +2477,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2507,9 +2543,13 @@ rclone ls remote:path [flags] -h, --help help for ls \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -2539,7 +2579,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -2547,10 +2587,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2632,9 +2669,13 @@ rclone lsd remote:path [flags] -R, --recursive Recurse into the listing \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -2664,7 +2705,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -2672,10 +2713,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2741,9 +2779,13 @@ rclone lsl remote:path [flags] -h, --help help for lsl \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -2773,7 +2815,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -2781,10 +2823,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2827,9 +2866,13 @@ rclone md5sum remote:path [flags] --output-file string Output hashsums to a file rather than the terminal \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -2859,7 +2902,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -2867,10 +2910,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2916,9 +2956,13 @@ rclone sha1sum remote:path [flags] --output-file string Output hashsums to a file rather than the terminal \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -2948,7 +2992,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -2956,10 +3000,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -2998,9 +3039,13 @@ rclone size remote:path [flags] --json Format output as JSON \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -3030,7 +3075,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -3038,10 +3083,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3115,7 +3157,7 @@ rclone version [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3140,9 +3182,13 @@ rclone cleanup remote:path [flags] -h, --help help for cleanup \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -3151,10 +3197,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3326,9 +3369,13 @@ rclone dedupe [mode] remote:path [flags] -h, --help help for dedupe \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -3337,10 +3384,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3349,8 +3393,7 @@ commands, flags and backends. Get quota information from the remote. .SS Synopsis .PP -\f[C]rclone about\f[R] prints quota information about a remote to -standard output. +Prints quota information about a remote to standard output. The output is typically used, free, quota and trash contents. .PP E.g. @@ -3436,7 +3479,7 @@ rclone about remote: [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3473,7 +3516,7 @@ rclone authorize [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3540,9 +3583,13 @@ rclone backend remote:path [opts] [flags] -o, --option stringArray Option in the form name=value or name \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -3551,10 +3598,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3620,15 +3664,19 @@ rclone bisync remote1:path1 remote2:path2 [flags] --workdir string Use custom working dir - useful for testing. (default: {WORKDIR}) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Copy Options .PP -Flags for anything which can Copy a file. +Flags for anything which can copy a file .IP .nf \f[C] --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -3662,7 +3710,7 @@ Flags for anything which can Copy a file. .fi .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -3673,7 +3721,7 @@ Important flags useful for most commands. .fi .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -3701,10 +3749,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3713,7 +3758,7 @@ commands, flags and backends. Concatenates any files and sends them to stdout. .SS Synopsis .PP -rclone cat sends any files to standard output. +Sends any files to standard output. .PP You can use it like this to output a single file .IP @@ -3788,9 +3833,13 @@ rclone cat remote:path [flags] --tail int Only print the last N characters \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -3820,7 +3869,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -3828,10 +3877,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3913,9 +3959,13 @@ rclone checksum sumfile dst:path [flags] --one-way Check one way only, source files must exist on remote \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -3945,7 +3995,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -3953,10 +4003,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -3977,7 +4024,7 @@ Run with \f[C]--help\f[R] to list the supported shells. .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -4008,7 +4055,7 @@ By default, when run without any arguments, .IP .nf \f[C] -rclone genautocomplete bash +rclone completion bash \f[R] .fi .PP @@ -4058,7 +4105,7 @@ rclone completion bash [output_file] [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone completion (https://rclone.org/commands/rclone_completion/) - Output completion script for a given shell. @@ -4074,7 +4121,7 @@ probably need to be run with sudo or as root, e.g. .IP .nf \f[C] -sudo rclone genautocomplete fish +sudo rclone completion fish \f[R] .fi .PP @@ -4107,7 +4154,7 @@ rclone completion fish [output_file] [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone completion (https://rclone.org/commands/rclone_completion/) - Output completion script for a given shell. @@ -4147,7 +4194,7 @@ rclone completion powershell [output_file] [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone completion (https://rclone.org/commands/rclone_completion/) - Output completion script for a given shell. @@ -4163,7 +4210,7 @@ will probably need to be run with sudo or as root, e.g. .IP .nf \f[C] -sudo rclone genautocomplete zsh +sudo rclone completion zsh \f[R] .fi .PP @@ -4196,7 +4243,7 @@ rclone completion zsh [output_file] [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone completion (https://rclone.org/commands/rclone_completion/) - Output completion script for a given shell. @@ -4361,7 +4408,7 @@ rclone config create name type [key value]* [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4384,7 +4431,7 @@ rclone config delete name [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4414,7 +4461,7 @@ rclone config disconnect remote: [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4437,7 +4484,7 @@ rclone config dump [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4465,10 +4512,158 @@ rclone config edit [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. +.SH rclone config encryption +.PP +set, remove and check the encryption for the config file +.SS Synopsis +.PP +This command sets, clears and checks the encryption for the config file +using the subcommands below. +.SS Options +.IP +.nf +\f[C] + -h, --help help for encryption +\f[R] +.fi +.PP +See the global flags page (https://rclone.org/flags/) for global options +not listed here. +.SS See Also +.IP \[bu] 2 +rclone config (https://rclone.org/commands/rclone_config/) - Enter an +interactive configuration session. +.IP \[bu] 2 +rclone config encryption +check (https://rclone.org/commands/rclone_config_encryption_check/) - +Check that the config file is encrypted +.IP \[bu] 2 +rclone config encryption +remove (https://rclone.org/commands/rclone_config_encryption_remove/) - +Remove the config file encryption password +.IP \[bu] 2 +rclone config encryption +set (https://rclone.org/commands/rclone_config_encryption_set/) - Set or +change the config file encryption password +.SH rclone config encryption check +.PP +Check that the config file is encrypted +.SS Synopsis +.PP +This checks the config file is encrypted and that you can decrypt it. +.PP +It will attempt to decrypt the config using the password you supply. +.PP +If decryption fails it will return a non-zero exit code if using +\f[C]--password-command\f[R], otherwise it will prompt again for the +password. +.PP +If the config file is not encrypted it will return a non zero exit code. +.IP +.nf +\f[C] +rclone config encryption check [flags] +\f[R] +.fi +.SS Options +.IP +.nf +\f[C] + -h, --help help for check +\f[R] +.fi +.PP +See the global flags page (https://rclone.org/flags/) for global options +not listed here. +.SS See Also +.IP \[bu] 2 +rclone config +encryption (https://rclone.org/commands/rclone_config_encryption/) - +set, remove and check the encryption for the config file +.SH rclone config encryption remove +.PP +Remove the config file encryption password +.SS Synopsis +.PP +Remove the config file encryption password +.PP +This removes the config file encryption, returning it to un-encrypted. +.PP +If \f[C]--password-command\f[R] is in use, this will be called to supply +the old config password. +.PP +If the config was not encrypted then no error will be returned and this +command will do nothing. +.IP +.nf +\f[C] +rclone config encryption remove [flags] +\f[R] +.fi +.SS Options +.IP +.nf +\f[C] + -h, --help help for remove +\f[R] +.fi +.PP +See the global flags page (https://rclone.org/flags/) for global options +not listed here. +.SS See Also +.IP \[bu] 2 +rclone config +encryption (https://rclone.org/commands/rclone_config_encryption/) - +set, remove and check the encryption for the config file +.SH rclone config encryption set +.PP +Set or change the config file encryption password +.SS Synopsis +.PP +This command sets or changes the config file encryption password. +.PP +If there was no config password set then it sets a new one, otherwise it +changes the existing config password. +.PP +Note that if you are changing an encryption password using +\f[C]--password-command\f[R] then this will be called once to decrypt +the config using the old password and then again to read the new +password to re-encrypt the config. +.PP +When \f[C]--password-command\f[R] is called to change the password then +the environment variable \f[C]RCLONE_PASSWORD_CHANGE=1\f[R] will be set. +So if changing passwords programatically you can use the environment +variable to distinguish which password you must supply. +.PP +Alternatively you can remove the password first (with +\f[C]rclone config encryption remove\f[R]), then set it again with this +command which may be easier if you don\[aq]t mind the unecrypted config +file being on the disk briefly. +.IP +.nf +\f[C] +rclone config encryption set [flags] +\f[R] +.fi +.SS Options +.IP +.nf +\f[C] + -h, --help help for set +\f[R] +.fi +.PP +See the global flags page (https://rclone.org/flags/) for global options +not listed here. +.SS See Also +.IP \[bu] 2 +rclone config +encryption (https://rclone.org/commands/rclone_config_encryption/) - +set, remove and check the encryption for the config file .SH rclone config file .PP Show path of configuration file in use. @@ -4488,7 +4683,7 @@ rclone config file [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4529,7 +4724,7 @@ rclone config password name [key value]+ [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4552,7 +4747,7 @@ rclone config paths [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4575,7 +4770,7 @@ rclone config providers [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4605,7 +4800,7 @@ rclone config reconnect remote: [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4641,7 +4836,7 @@ rclone config redacted [] [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4664,7 +4859,7 @@ rclone config show [] [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4687,7 +4882,7 @@ rclone config touch [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4852,7 +5047,7 @@ rclone config update name [key value]+ [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4880,7 +5075,7 @@ rclone config userinfo remote: [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone config (https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. @@ -4939,15 +5134,19 @@ rclone copyto source:path dest:path [flags] -h, --help help for copyto \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Copy Options .PP -Flags for anything which can Copy a file. +Flags for anything which can copy a file .IP .nf \f[C] --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -4981,7 +5180,7 @@ Flags for anything which can Copy a file. .fi .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -4992,7 +5191,7 @@ Important flags useful for most commands. .fi .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -5022,7 +5221,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -5030,10 +5229,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -5095,9 +5291,13 @@ rclone copyurl https://example.com dest:path [flags] --stdout Write the output to stdout rather than a file \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -5106,10 +5306,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -5118,8 +5315,7 @@ commands, flags and backends. Cryptcheck checks the integrity of an encrypted remote. .SS Synopsis .PP -rclone cryptcheck checks a remote against a -crypted (https://rclone.org/crypt/) remote. +Checks a remote against a crypted (https://rclone.org/crypt/) remote. This is the equivalent of running rclone check (https://rclone.org/commands/rclone_check/), but able to check the checksums of the encrypted remote. @@ -5208,9 +5404,13 @@ rclone cryptcheck remote:path cryptedremote:path [flags] --one-way Check one way only, source files must exist on remote \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Check Options .PP -Flags used for \f[C]rclone check\f[R]. +Flags used for check commands .IP .nf \f[C] @@ -5219,7 +5419,7 @@ Flags used for \f[C]rclone check\f[R]. .fi .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -5249,7 +5449,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -5257,10 +5457,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -5269,8 +5466,8 @@ commands, flags and backends. Cryptdecode returns unencrypted file names. .SS Synopsis .PP -rclone cryptdecode returns unencrypted file names when provided with a -list of encrypted file names. +Returns unencrypted file names when provided with a list of encrypted +file names. List limit is 10 items. .PP If you supply the \f[C]--reverse\f[R] flag, it will return encrypted @@ -5307,7 +5504,7 @@ rclone cryptdecode encryptedremote: encryptedfilename [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -5333,9 +5530,13 @@ rclone deletefile remote:path [flags] -h, --help help for deletefile \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -5344,196 +5545,10 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. -.SH rclone genautocomplete -.PP -Output completion script for a given shell. -.SH Synopsis -.PP -Generates a shell completion script for rclone. -Run with \f[C]--help\f[R] to list the supported shells. -.SH Options -.IP -.nf -\f[C] - -h, --help help for genautocomplete -\f[R] -.fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO -.IP \[bu] 2 -rclone (https://rclone.org/commands/rclone/) - Show help for rclone -commands, flags and backends. -.IP \[bu] 2 -rclone genautocomplete -bash (https://rclone.org/commands/rclone_genautocomplete_bash/) - Output -bash completion script for rclone. -.IP \[bu] 2 -rclone genautocomplete -fish (https://rclone.org/commands/rclone_genautocomplete_fish/) - Output -fish completion script for rclone. -.IP \[bu] 2 -rclone genautocomplete -zsh (https://rclone.org/commands/rclone_genautocomplete_zsh/) - Output -zsh completion script for rclone. -.SH rclone genautocomplete bash -.PP -Output bash completion script for rclone. -.SH Synopsis -.PP -Generates a bash shell autocompletion script for rclone. -.PP -This writes to /etc/bash_completion.d/rclone by default so will probably -need to be run with sudo or as root, e.g. -.IP -.nf -\f[C] -sudo rclone genautocomplete bash -\f[R] -.fi -.PP -Logout and login again to use the autocompletion scripts, or source them -directly -.IP -.nf -\f[C] -\&. /etc/bash_completion -\f[R] -.fi -.PP -If you supply a command line argument the script will be written there. -.PP -If output_file is \[dq]-\[dq], then the output will be written to -stdout. -.IP -.nf -\f[C] -rclone genautocomplete bash [output_file] [flags] -\f[R] -.fi -.SH Options -.IP -.nf -\f[C] - -h, --help help for bash -\f[R] -.fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO -.IP \[bu] 2 -rclone -genautocomplete (https://rclone.org/commands/rclone_genautocomplete/) - -Output completion script for a given shell. -.SH rclone genautocomplete fish -.PP -Output fish completion script for rclone. -.SH Synopsis -.PP -Generates a fish autocompletion script for rclone. -.PP -This writes to /etc/fish/completions/rclone.fish by default so will -probably need to be run with sudo or as root, e.g. -.IP -.nf -\f[C] -sudo rclone genautocomplete fish -\f[R] -.fi -.PP -Logout and login again to use the autocompletion scripts, or source them -directly -.IP -.nf -\f[C] -\&. /etc/fish/completions/rclone.fish -\f[R] -.fi -.PP -If you supply a command line argument the script will be written there. -.PP -If output_file is \[dq]-\[dq], then the output will be written to -stdout. -.IP -.nf -\f[C] -rclone genautocomplete fish [output_file] [flags] -\f[R] -.fi -.SH Options -.IP -.nf -\f[C] - -h, --help help for fish -\f[R] -.fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO -.IP \[bu] 2 -rclone -genautocomplete (https://rclone.org/commands/rclone_genautocomplete/) - -Output completion script for a given shell. -.SH rclone genautocomplete zsh -.PP -Output zsh completion script for rclone. -.SH Synopsis -.PP -Generates a zsh autocompletion script for rclone. -.PP -This writes to /usr/share/zsh/vendor-completions/_rclone by default so -will probably need to be run with sudo or as root, e.g. -.IP -.nf -\f[C] -sudo rclone genautocomplete zsh -\f[R] -.fi -.PP -Logout and login again to use the autocompletion scripts, or source them -directly -.IP -.nf -\f[C] -autoload -U compinit && compinit -\f[R] -.fi -.PP -If you supply a command line argument the script will be written there. -.PP -If output_file is \[dq]-\[dq], then the output will be written to -stdout. -.IP -.nf -\f[C] -rclone genautocomplete zsh [output_file] [flags] -\f[R] -.fi -.SH Options -.IP -.nf -\f[C] - -h, --help help for zsh -\f[R] -.fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO -.IP \[bu] 2 -rclone -genautocomplete (https://rclone.org/commands/rclone_genautocomplete/) - -Output completion script for a given shell. .SH rclone gendocs .PP Output markdown docs for rclone to the directory supplied. @@ -5559,7 +5574,7 @@ rclone gendocs output_directory [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -5670,7 +5685,7 @@ rclone gitannex [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -5738,9 +5753,13 @@ rclone hashsum [ remote:path] [flags] --output-file string Output hashsums to a file rather than the terminal \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -5770,7 +5789,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -5778,10 +5797,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -5790,8 +5806,7 @@ commands, flags and backends. Generate public link to file/folder. .SS Synopsis .PP -rclone link will create, retrieve or remove a public link to the given -file or folder. +Create, retrieve or remove a public link to the given file or folder. .IP .nf \f[C] @@ -5834,7 +5849,7 @@ rclone link remote:path [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -5844,28 +5859,44 @@ List all the remotes in the config file and defined in environment variables. .SS Synopsis .PP -rclone listremotes lists all the available remotes from the config file. +Lists all the available remotes from the config file, or the remotes +matching an optional filter. .PP -When used with the \f[C]--long\f[R] flag it lists the types and the -descriptions too. +Prints the result in human-readable format by default, and as a simple +list of remote names, or if used with flag \f[C]--long\f[R] a tabular +format including the remote names, types and descriptions. +Using flag \f[C]--json\f[R] produces machine-readable output instead, +which always includes all attributes - including the source (file or +environment). +.PP +Result can be filtered by a filter argument which applies to all +attributes, and/or filter flags specific for each attribute. +The values must be specified according to regular rclone filtering +pattern syntax. .IP .nf \f[C] -rclone listremotes [flags] +rclone listremotes [] [flags] \f[R] .fi .SS Options .IP .nf \f[C] - -h, --help help for listremotes - --long Show the type and the description as well as names + --description string Filter remotes by description + -h, --help help for listremotes + --json Format output as JSON + --long Show type and description in addition to name + --name string Filter remotes by name + --order-by string Instructions on how to order the result, e.g. \[aq]type,name=descending\[aq] + --source string Filter remotes by source, e.g. \[aq]file\[aq] or \[aq]environment\[aq] + --type string Filter remotes by type \f[R] .fi .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -6076,9 +6107,13 @@ rclone lsf remote:path [flags] -t, --time-format string Specify a custom time format, or \[aq]max\[aq] for max precision supported by remote (default: 2006-01-02 15:04:05) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -6108,7 +6143,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -6116,10 +6151,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -6130,7 +6162,7 @@ List directories and objects in the path in JSON format. .PP List directories and objects in the path in JSON format. .PP -The output is an array of Items, where each Item looks like this +The output is an array of Items, where each Item looks like this: .IP .nf \f[C] @@ -6156,41 +6188,58 @@ The output is an array of Items, where each Item looks like this \f[R] .fi .PP +The exact set of properties included depends on the backend: +.IP \[bu] 2 +The property IsBucket will only be included for bucket-based remotes, +and only for directories that are buckets. +It will always be omitted when value is not true. +.IP \[bu] 2 +Properties Encrypted and EncryptedPath will only be included for +encrypted remotes, and (as mentioned below) only if the +\f[C]--encrypted\f[R] option is set. +.PP +Different options may also affect which properties are included: +.IP \[bu] 2 If \f[C]--hash\f[R] is not specified, the Hashes property will be omitted. The types of hash can be specified with the \f[C]--hash-type\f[R] parameter (which may be repeated). If \f[C]--hash-type\f[R] is set then it implies \f[C]--hash\f[R]. -.PP +.IP \[bu] 2 If \f[C]--no-modtime\f[R] is specified then ModTime will be blank. This can speed things up on remotes where reading the ModTime takes an extra request (e.g. s3, swift). -.PP +.IP \[bu] 2 If \f[C]--no-mimetype\f[R] is specified then MimeType will be blank. This can speed things up on remotes where reading the MimeType takes an extra request (e.g. s3, swift). +.IP \[bu] 2 +If \f[C]--encrypted\f[R] is not specified the Encrypted and +EncryptedPath properties will be omitted - even for encrypted remotes. +.IP \[bu] 2 +If \f[C]--metadata\f[R] is set then an additional Metadata property will +be returned. +This will have metadata (https://rclone.org/docs/#metadata) in rclone +standard format as a JSON object. .PP -If \f[C]--encrypted\f[R] is not specified the Encrypted will be omitted. +The default is to list directories and files/objects, but this can be +changed with the following options: +.IP \[bu] 2 +If \f[C]--dirs-only\f[R] is specified then directories will be returned +only, no files/objects. +.IP \[bu] 2 +If \f[C]--files-only\f[R] is specified then files will be returned only, +no directories. .PP -If \f[C]--dirs-only\f[R] is not specified files in addition to -directories are returned -.PP -If \f[C]--files-only\f[R] is not specified directories in addition to -the files will be returned. -.PP -If \f[C]--metadata\f[R] is set then an additional Metadata key will be -returned. -This will have metadata in rclone standard format as a JSON object. -.PP -if \f[C]--stat\f[R] is set then a single JSON blob will be returned -about the item pointed to. -This will return an error if the item isn\[aq]t found. -However on bucket based backends (like s3, gcs, b2, azureblob etc) if -the item isn\[aq]t found it will return an empty directory as it -isn\[aq]t possible to tell empty directories from missing directories -there. +If \f[C]--stat\f[R] is set then the the output is not an array of items, +but instead a single JSON blob will be returned about the item pointed +to. +This will return an error if the item isn\[aq]t found, however on bucket +based backends (like s3, gcs, b2, azureblob etc) if the item isn\[aq]t +found it will return an empty directory, as it isn\[aq]t possible to +tell empty directories from missing directories there. .PP The Path field will only show folders below the remote path being listed. @@ -6200,10 +6249,6 @@ not \[dq]remote:path/subfolder/file.txt\[dq]. When used without \f[C]--recursive\f[R] the Path will always be the same as Name. .PP -If the directory is a bucket in a bucket-based backend, then -\[dq]IsBucket\[dq] will be set to true. -This key won\[aq]t be present unless it is \[dq]true\[dq]. -.PP The time is in RFC3339 format with up to nanosecond precision. The number of decimal digits in the seconds will depend on the precision that the remote can hold the times, so if times are accurate to the @@ -6214,7 +6259,8 @@ accurate to the nearest second (Dropbox, Box, WebDav, etc.) no digits will be shown (\[dq]2017-05-31T16:15:57+01:00\[dq]). .PP The whole output can be processed as a JSON blob, or alternatively it -can be processed line by line as each item is written one to a line. +can be processed line by line as each item is written on individual +lines (except with \f[C]--stat\f[R]). .PP Any of the filtering options can be applied to this command. .PP @@ -6268,9 +6314,13 @@ rclone lsjson remote:path [flags] --stat Just return the info for the pointed to file \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -6300,7 +6350,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -6308,10 +6358,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -7188,9 +7235,14 @@ These flags control the chunking: \f[C] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) +--vfs-read-chunk-streams int The number of parallel streams to read at once \f[R] .fi .PP +The chunking behaves differently depending on the +\f[C]--vfs-read-chunk-streams\f[R] parameter. +.SS \f[C]--vfs-read-chunk-streams\f[R] == 0 +.PP Rclone will start reading a chunk of size \f[C]--vfs-read-chunk-size\f[R], and then double the size for each read. When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater @@ -7208,6 +7260,31 @@ on. .PP Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq] disables chunked reading. +.PP +The chunks will not be buffered in memory. +.SS \f[C]--vfs-read-chunk-streams\f[R] > 0 +.PP +Rclone reads \f[C]--vfs-read-chunk-streams\f[R] chunks of size +\f[C]--vfs-read-chunk-size\f[R] concurrently. +The size for each read will stay constant. +.PP +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. +.PP +Some experimentation will be needed to find the optimum values of +\f[C]--vfs-read-chunk-size\f[R] and \f[C]--vfs-read-chunk-streams\f[R] +as these will depend on the backend in use and the latency to the +backend. +.PP +For high performance object stores (eg AWS S3) a reasonable place to +start might be \f[C]--vfs-read-chunk-streams 16\f[R] and +\f[C]--vfs-read-chunk-size 4M\f[R]. +In testing with AWS S3 the performance scaled roughly as the +\f[C]--vfs-read-chunk-streams\f[R] setting. +.PP +Similar settings should work for high latency links, but depending on +the latency they may need more \f[C]--vfs-read-chunk-streams\f[R] in +order to get the throughput. .SS VFS Performance .PP These flags may be used to enable/disable features of the VFS for @@ -7369,9 +7446,9 @@ rclone mount remote:path /path/to/mountpoint [flags] --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for mount @@ -7387,7 +7464,7 @@ rclone mount remote:path /path/to/mountpoint [flags] --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -7400,6 +7477,7 @@ rclone mount remote:path /path/to/mountpoint [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -7409,9 +7487,13 @@ rclone mount remote:path /path/to/mountpoint [flags] --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -7439,10 +7521,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -7504,15 +7583,19 @@ rclone moveto source:path dest:path [flags] -h, --help help for moveto \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Copy Options .PP -Flags for anything which can Copy a file. +Flags for anything which can copy a file .IP .nf \f[C] --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -7546,7 +7629,7 @@ Flags for anything which can Copy a file. .fi .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -7557,7 +7640,7 @@ Important flags useful for most commands. .fi .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -7587,7 +7670,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -7595,10 +7678,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -7668,6 +7748,7 @@ e means this is an empty directory, i.e. contains no files (but This an homage to the ncdu tool (https://dev.yorhel.nl/ncdu) but for rclone remotes. It is missing lots of features at the moment but is useful as it stands. +Unlike ncdu it does not show excluded files. .PP Note that it might take some time to delete big files/directories. The UI won\[aq]t respond in the meantime since the deletion is done @@ -7690,9 +7771,13 @@ rclone ncdu remote:path [flags] -h, --help help for ncdu \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -7722,7 +7807,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -7730,10 +7815,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -8611,9 +8693,14 @@ These flags control the chunking: \f[C] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) +--vfs-read-chunk-streams int The number of parallel streams to read at once \f[R] .fi .PP +The chunking behaves differently depending on the +\f[C]--vfs-read-chunk-streams\f[R] parameter. +.SS \f[C]--vfs-read-chunk-streams\f[R] == 0 +.PP Rclone will start reading a chunk of size \f[C]--vfs-read-chunk-size\f[R], and then double the size for each read. When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater @@ -8631,6 +8718,31 @@ on. .PP Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq] disables chunked reading. +.PP +The chunks will not be buffered in memory. +.SS \f[C]--vfs-read-chunk-streams\f[R] > 0 +.PP +Rclone reads \f[C]--vfs-read-chunk-streams\f[R] chunks of size +\f[C]--vfs-read-chunk-size\f[R] concurrently. +The size for each read will stay constant. +.PP +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. +.PP +Some experimentation will be needed to find the optimum values of +\f[C]--vfs-read-chunk-size\f[R] and \f[C]--vfs-read-chunk-streams\f[R] +as these will depend on the backend in use and the latency to the +backend. +.PP +For high performance object stores (eg AWS S3) a reasonable place to +start might be \f[C]--vfs-read-chunk-streams 16\f[R] and +\f[C]--vfs-read-chunk-size 4M\f[R]. +In testing with AWS S3 the performance scaled roughly as the +\f[C]--vfs-read-chunk-streams\f[R] setting. +.PP +Similar settings should work for high latency links, but depending on +the latency they may need more \f[C]--vfs-read-chunk-streams\f[R] in +order to get the throughput. .SS VFS Performance .PP These flags may be used to enable/disable features of the VFS for @@ -8793,16 +8905,18 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for nfsmount --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki) --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset) --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only) + --nfs-cache-dir string The directory the NFS handle cache will use if set --nfs-cache-handle-limit int max file handles cached simultaneously (min 5) (default 1000000) + --nfs-cache-type memory|disk|symlink Type of NFS handle cache to use (default memory) --no-checksum Don\[aq]t compare checksums on up/download --no-modtime Don\[aq]t read/write the modification time (can speed things up) --no-seek Don\[aq]t allow seeking in files @@ -8811,9 +8925,9 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] -o, --option stringArray Option for libfuse/WinFsp (repeat if required) --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access - --sudo Use sudo to run the mount command as root. + --sudo Use sudo to run the mount/umount commands as root. --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -8826,6 +8940,7 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -8835,9 +8950,13 @@ rclone nfsmount remote:path /path/to/mountpoint [flags] --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -8865,10 +8984,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -8921,7 +9037,7 @@ rclone obscure password [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -8944,6 +9060,18 @@ Note that \f[C]--rc-addr\f[R], \f[C]--rc-user\f[R], \f[C]--rc-pass\f[R] will be read also for \f[C]--url\f[R], \f[C]--user\f[R], \f[C]--pass\f[R]. .PP +The \f[C]--unix-socket\f[R] flag can be used to connect over a unix +socket like this +.IP +.nf +\f[C] +# start server on /tmp/my.socket +rclone rcd --rc-addr unix:///tmp/my.socket +# Connect to it +rclone rc --unix-socket /tmp/my.socket core/stats +\f[R] +.fi +.PP Arguments should be passed in as parameter=value. .PP The result will be returned as a JSON object by default. @@ -9015,21 +9143,22 @@ rclone rc commands parameter [flags] .IP .nf \f[C] - -a, --arg stringArray Argument placed in the \[dq]arg\[dq] array - -h, --help help for rc - --json string Input JSON - use instead of key=value args - --loopback If set connect to this rclone instance not via HTTP - --no-output If set, don\[aq]t output the JSON result - -o, --opt stringArray Option in the form name=value or name placed in the \[dq]opt\[dq] array - --pass string Password to use to connect to rclone remote control - --url string URL to connect to rclone remote control (default \[dq]http://localhost:5572/\[dq]) - --user string Username to use to rclone remote control + -a, --arg stringArray Argument placed in the \[dq]arg\[dq] array + -h, --help help for rc + --json string Input JSON - use instead of key=value args + --loopback If set connect to this rclone instance not via HTTP + --no-output If set, don\[aq]t output the JSON result + -o, --opt stringArray Option in the form name=value or name placed in the \[dq]opt\[dq] array + --pass string Password to use to connect to rclone remote control + --unix-socket string Path to a unix domain socket to dial to, instead of opening a TCP connection directly + --url string URL to connect to rclone remote control (default \[dq]http://localhost:5572/\[dq]) + --user string Username to use to rclone remote control \f[R] .fi .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -9038,8 +9167,7 @@ commands, flags and backends. Copies standard input to file on remote. .SS Synopsis .PP -rclone rcat reads from standard input (stdin) and copies it to a single -remote file. +Reads from standard input (stdin) and copies it to a single remote file. .IP .nf \f[C] @@ -9092,9 +9220,13 @@ rclone rcat remote:path [flags] --size int File size hint to preallocate (default -1) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -9103,10 +9235,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -9144,6 +9273,8 @@ be done with file system permissions. .PP \f[C]--rc-addr\f[R] may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to +accomplish the same. .PP \f[C]--rc-server-read-timeout\f[R] and \f[C]--rc-server-write-timeout\f[R] can be used to control the timeouts @@ -9181,7 +9312,29 @@ authority certificate. acceptable. Valid values are \[dq]tls1.0\[dq], \[dq]tls1.1\[dq], \[dq]tls1.2\[dq] and \[dq]tls1.3\[dq] (default \[dq]tls1.0\[dq]). -.SS Template +.SS Socket activation +.PP +Instead of the listening addresses specified above, rclone will listen +to all FDs passed by the service manager, if any (and ignore any +arguments passed by --rc-addr\[ga]). +.PP +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described +in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html +.PP +Socket activation can be tested ad-hoc with the +\f[C]systemd-socket-activate\f[R]command +.IP +.nf +\f[C] + systemd-socket-activate -l 8000 -- rclone serve +\f[R] +.fi +.PP +This will socket-activate rclone on the first connection to port 8000 +over TCP. +### Template .PP \f[C]--rc-template\f[R] allows a user to specify a custom markup template for HTTP and WebDAV serve functions. @@ -9363,19 +9516,23 @@ rclone rcd * [flags] -h, --help help for rcd \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS RC Options .PP -Flags to control the Remote Control API. +Flags to control the Remote Control API .IP .nf \f[C] --rc Enable the remote control server - --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572]) + --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [\[dq]localhost:5572\[dq]]) --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from --rc-baseurl string Prefix for URLs - leave blank for root --rc-cert string TLS PEM key (concatenation of certificate and CA certificate) --rc-client-ca string Client certificate authority to verify clients with - --rc-enable-metrics Enable prometheus metrics on /metrics + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server --rc-files string Path to local files to serve on the HTTP server --rc-htpasswd string A htpasswd file - if not provided no authentication is done --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s) @@ -9400,10 +9557,7 @@ Flags to control the Remote Control API. --rc-web-gui-update Check and update to latest version of web gui \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -9445,9 +9599,13 @@ rclone rmdirs remote:path [flags] --leave-root Do not remove root directory if empty \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -9456,10 +9614,7 @@ Important flags useful for most commands. -v, --verbose count Print lots more stuff (repeat for more) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -9554,7 +9709,7 @@ rclone selfupdate [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -9589,7 +9744,7 @@ rclone serve [opts] [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -9637,6 +9792,11 @@ based on media formats or file extensions. Additionally, there is no media transcoding support. This means that some players might show files that they are not able to play back correctly. +.PP +Rclone will add external subtitle files (.srt) to videos if they have +the same filename as the video file itself (except the extension), +either in the same directory as the video, or in a \[dq]Subs\[dq] +subdirectory. .SS Server options .PP Use \f[C]--addr\f[R] to specify which IP address and port the server @@ -9925,9 +10085,14 @@ These flags control the chunking: \f[C] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) +--vfs-read-chunk-streams int The number of parallel streams to read at once \f[R] .fi .PP +The chunking behaves differently depending on the +\f[C]--vfs-read-chunk-streams\f[R] parameter. +.SS \f[C]--vfs-read-chunk-streams\f[R] == 0 +.PP Rclone will start reading a chunk of size \f[C]--vfs-read-chunk-size\f[R], and then double the size for each read. When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater @@ -9945,6 +10110,31 @@ on. .PP Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq] disables chunked reading. +.PP +The chunks will not be buffered in memory. +.SS \f[C]--vfs-read-chunk-streams\f[R] > 0 +.PP +Rclone reads \f[C]--vfs-read-chunk-streams\f[R] chunks of size +\f[C]--vfs-read-chunk-size\f[R] concurrently. +The size for each read will stay constant. +.PP +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. +.PP +Some experimentation will be needed to find the optimum values of +\f[C]--vfs-read-chunk-size\f[R] and \f[C]--vfs-read-chunk-streams\f[R] +as these will depend on the backend in use and the latency to the +backend. +.PP +For high performance object stores (eg AWS S3) a reasonable place to +start might be \f[C]--vfs-read-chunk-streams 16\f[R] and +\f[C]--vfs-read-chunk-size 4M\f[R]. +In testing with AWS S3 the performance scaled roughly as the +\f[C]--vfs-read-chunk-streams\f[R] setting. +.PP +Similar settings should work for high latency links, but depending on +the latency they may need more \f[C]--vfs-read-chunk-streams\f[R] in +order to get the throughput. .SS VFS Performance .PP These flags may be used to enable/disable features of the VFS for @@ -10097,8 +10287,8 @@ rclone serve dlna remote:path [flags] --addr string The ip:port or :port to bind the DLNA http server to (default \[dq]:7879\[dq]) --announce-interval Duration The interval between SSDP announcements (default 12m0s) --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for dlna --interface stringArray The interface to use for SSDP (repeat as necessary) @@ -10110,7 +10300,7 @@ rclone serve dlna remote:path [flags] --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -10123,6 +10313,7 @@ rclone serve dlna remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -10130,9 +10321,13 @@ rclone serve dlna remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -10160,10 +10355,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -10497,9 +10689,14 @@ These flags control the chunking: \f[C] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) +--vfs-read-chunk-streams int The number of parallel streams to read at once \f[R] .fi .PP +The chunking behaves differently depending on the +\f[C]--vfs-read-chunk-streams\f[R] parameter. +.SS \f[C]--vfs-read-chunk-streams\f[R] == 0 +.PP Rclone will start reading a chunk of size \f[C]--vfs-read-chunk-size\f[R], and then double the size for each read. When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater @@ -10517,6 +10714,31 @@ on. .PP Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq] disables chunked reading. +.PP +The chunks will not be buffered in memory. +.SS \f[C]--vfs-read-chunk-streams\f[R] > 0 +.PP +Rclone reads \f[C]--vfs-read-chunk-streams\f[R] chunks of size +\f[C]--vfs-read-chunk-size\f[R] concurrently. +The size for each read will stay constant. +.PP +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. +.PP +Some experimentation will be needed to find the optimum values of +\f[C]--vfs-read-chunk-size\f[R] and \f[C]--vfs-read-chunk-streams\f[R] +as these will depend on the backend in use and the latency to the +backend. +.PP +For high performance object stores (eg AWS S3) a reasonable place to +start might be \f[C]--vfs-read-chunk-streams 16\f[R] and +\f[C]--vfs-read-chunk-size 4M\f[R]. +In testing with AWS S3 the performance scaled roughly as the +\f[C]--vfs-read-chunk-streams\f[R] setting. +.PP +Similar settings should work for high latency links, but depending on +the latency they may need more \f[C]--vfs-read-chunk-streams\f[R] in +order to get the throughput. .SS VFS Performance .PP These flags may be used to enable/disable features of the VFS for @@ -10679,9 +10901,9 @@ rclone serve docker [flags] --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) --devname string Set the device name - default is remote:path --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --direct-io Use Direct IO, disables caching of data - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --forget-state Skip restoring previous state --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) @@ -10701,7 +10923,7 @@ rclone serve docker [flags] --socket-addr string Address or absolute path (default: /run/docker/plugins/rclone.sock) --socket-gid int GID for unix socket (default: current process GID) (default 1000) --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -10714,6 +10936,7 @@ rclone serve docker [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -10723,9 +10946,13 @@ rclone serve docker [flags] --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -10753,10 +10980,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -11060,9 +11284,14 @@ These flags control the chunking: \f[C] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) +--vfs-read-chunk-streams int The number of parallel streams to read at once \f[R] .fi .PP +The chunking behaves differently depending on the +\f[C]--vfs-read-chunk-streams\f[R] parameter. +.SS \f[C]--vfs-read-chunk-streams\f[R] == 0 +.PP Rclone will start reading a chunk of size \f[C]--vfs-read-chunk-size\f[R], and then double the size for each read. When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater @@ -11080,6 +11309,31 @@ on. .PP Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq] disables chunked reading. +.PP +The chunks will not be buffered in memory. +.SS \f[C]--vfs-read-chunk-streams\f[R] > 0 +.PP +Rclone reads \f[C]--vfs-read-chunk-streams\f[R] chunks of size +\f[C]--vfs-read-chunk-size\f[R] concurrently. +The size for each read will stay constant. +.PP +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. +.PP +Some experimentation will be needed to find the optimum values of +\f[C]--vfs-read-chunk-size\f[R] and \f[C]--vfs-read-chunk-streams\f[R] +as these will depend on the backend in use and the latency to the +backend. +.PP +For high performance object stores (eg AWS S3) a reasonable place to +start might be \f[C]--vfs-read-chunk-streams 16\f[R] and +\f[C]--vfs-read-chunk-size 4M\f[R]. +In testing with AWS S3 the performance scaled roughly as the +\f[C]--vfs-read-chunk-streams\f[R] setting. +.PP +Similar settings should work for high latency links, but depending on +the latency they may need more \f[C]--vfs-read-chunk-streams\f[R] in +order to get the throughput. .SS VFS Performance .PP These flags may be used to enable/disable features of the VFS for @@ -11326,8 +11580,8 @@ rclone serve ftp remote:path [flags] --auth-proxy string A program to use to create the backend from the auth --cert string TLS PEM key (concatenation of certificate and CA certificate) --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for ftp --key string TLS PEM Private key @@ -11340,7 +11594,7 @@ rclone serve ftp remote:path [flags] --public-ip string Public IP address to advertise for passive connections --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication (default \[dq]anonymous\[dq]) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -11354,6 +11608,7 @@ rclone serve ftp remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -11361,9 +11616,13 @@ rclone serve ftp remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -11391,10 +11650,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -11434,6 +11690,8 @@ be done with file system permissions. .PP \f[C]--addr\f[R] may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to +accomplish the same. .PP \f[C]--server-read-timeout\f[R] and \f[C]--server-write-timeout\f[R] can be used to control the timeouts on the server. @@ -11468,7 +11726,29 @@ authority certificate. \f[C]--min-tls-version\f[R] is minimum TLS version that is acceptable. Valid values are \[dq]tls1.0\[dq], \[dq]tls1.1\[dq], \[dq]tls1.2\[dq] and \[dq]tls1.3\[dq] (default \[dq]tls1.0\[dq]). -.SS Template +.SS Socket activation +.PP +Instead of the listening addresses specified above, rclone will listen +to all FDs passed by the service manager, if any (and ignore any +arguments passed by --addr\[ga]). +.PP +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described +in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html +.PP +Socket activation can be tested ad-hoc with the +\f[C]systemd-socket-activate\f[R]command +.IP +.nf +\f[C] + systemd-socket-activate -l 8000 -- rclone serve +\f[R] +.fi +.PP +This will socket-activate rclone on the first connection to port 8000 +over TCP. +### Template .PP \f[C]--template\f[R] allows a user to specify a custom markup template for HTTP and WebDAV serve functions. @@ -11912,9 +12192,14 @@ These flags control the chunking: \f[C] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) +--vfs-read-chunk-streams int The number of parallel streams to read at once \f[R] .fi .PP +The chunking behaves differently depending on the +\f[C]--vfs-read-chunk-streams\f[R] parameter. +.SS \f[C]--vfs-read-chunk-streams\f[R] == 0 +.PP Rclone will start reading a chunk of size \f[C]--vfs-read-chunk-size\f[R], and then double the size for each read. When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater @@ -11932,6 +12217,31 @@ on. .PP Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq] disables chunked reading. +.PP +The chunks will not be buffered in memory. +.SS \f[C]--vfs-read-chunk-streams\f[R] > 0 +.PP +Rclone reads \f[C]--vfs-read-chunk-streams\f[R] chunks of size +\f[C]--vfs-read-chunk-size\f[R] concurrently. +The size for each read will stay constant. +.PP +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. +.PP +Some experimentation will be needed to find the optimum values of +\f[C]--vfs-read-chunk-size\f[R] and \f[C]--vfs-read-chunk-streams\f[R] +as these will depend on the backend in use and the latency to the +backend. +.PP +For high performance object stores (eg AWS S3) a reasonable place to +start might be \f[C]--vfs-read-chunk-streams 16\f[R] and +\f[C]--vfs-read-chunk-size 4M\f[R]. +In testing with AWS S3 the performance scaled roughly as the +\f[C]--vfs-read-chunk-streams\f[R] setting. +.PP +Similar settings should work for high latency links, but depending on +the latency they may need more \f[C]--vfs-read-chunk-streams\f[R] in +order to get the throughput. .SS VFS Performance .PP These flags may be used to enable/disable features of the VFS for @@ -12174,15 +12484,15 @@ rclone serve http remote:path [flags] .IP .nf \f[C] - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for http --htpasswd string A htpasswd file - if not provided no authentication is done @@ -12201,7 +12511,7 @@ rclone serve http remote:path [flags] --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --template string User-specified template --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -12215,6 +12525,7 @@ rclone serve http remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -12222,9 +12533,13 @@ rclone serve http remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -12252,10 +12567,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -12266,31 +12578,61 @@ Serve the remote as an NFS mount .PP Create an NFS server that serves the given remote over the network. .PP -The primary purpose for this command is to enable mount +This implements an NFSv3 server to serve any rclone remote via NFS. +.PP +The primary purpose for this command is to enable the mount command (https://rclone.org/commands/rclone_mount/) on recent macOS versions where installing FUSE is very cumbersome. .PP -Since this is running on NFSv3, no authentication method is available. -Any client will be able to access the data. -To limit access, you can use serve NFS on loopback address and rely on -secure tunnels (such as SSH). -For this reason, by default, a random TCP port is chosen and loopback -interface is used for the listening address; meaning that it is only -available to the local machine. +This server does not implement any authentication so any client will be +able to access the data. +To limit access, you can use \f[C]serve nfs\f[R] on the loopback address +or rely on secure tunnels (such as SSH) or use firewalling. +.PP +For this reason, by default, a random TCP port is chosen and the +loopback interface is used for the listening address by default; meaning +that it is only available to the local machine. If you want other machines to access the NFS mount over local network, -you need to specify the listening address and port using +you need to specify the listening address and port using the \f[C]--addr\f[R] flag. .PP -Modifying files through NFS protocol requires VFS caching. +Modifying files through the NFS protocol requires VFS caching. Usually you will need to specify \f[C]--vfs-cache-mode\f[R] in order to -be able to write to the mountpoint (full is recommended). +be able to write to the mountpoint (\f[C]full\f[R] is recommended). If you don\[aq]t specify VFS cache mode, the mount will be read-only. -Note also that \f[C]--nfs-cache-handle-limit\f[R] controls the maximum -number of cached file handles stored by the caching handler. +.PP +\f[C]--nfs-cache-type\f[R] controls the type of the NFS handle cache. +By default this is \f[C]memory\f[R] where new handles will be randomly +allocated when needed. +These are stored in memory. +If the server is restarted the handle cache will be lost and connected +NFS clients will get stale handle errors. +.PP +\f[C]--nfs-cache-type disk\f[R] uses an on disk NFS handle cache. +Rclone hashes the path of the object and stores it in a file named after +the hash. +These hashes are stored on disk the directory controlled by +\f[C]--cache-dir\f[R] or the exact directory may be specified with +\f[C]--nfs-cache-dir\f[R]. +Using this means that the NFS server can be restarted at will without +affecting the connected clients. +.PP +\f[C]--nfs-cache-type symlink\f[R] is similar to +\f[C]--nfs-cache-type disk\f[R] in that it uses an on disk cache, but +the cache entries are held as symlinks. +Rclone will use the handle of the underlying file as the NFS handle +which improves performance. +This sort of cache can\[aq]t be backed up and restored as the underlying +handles will change. +This is Linux only. +.PP +\f[C]--nfs-cache-handle-limit\f[R] controls the maximum number of cached +NFS handles stored by the caching handler. This should not be set too low or you may experience errors when trying to access files. The default is \f[C]1000000\f[R], but consider lowering this limit if the server\[aq]s system resource usage causes problems. +This is only used by the \f[C]memory\f[R] type cache. .PP To serve NFS over the network use following command: .IP @@ -12300,20 +12642,20 @@ rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full \f[R] .fi .PP -We specify a specific port that we can use in the mount command: -.PP +This specifies a port that can be used in the mount command. To mount the server under Linux/macOS, use the following command: .IP .nf \f[C] -mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint +mount -t nfs -o port=$PORT,mountport=$PORT,tcp $HOSTNAME:/ path/to/mountpoint \f[R] .fi .PP -Where \f[C]$PORT\f[R] is the same port number we used in the serve nfs -command. +Where \f[C]$PORT\f[R] is the same port number used in the +\f[C]serve nfs\f[R] command and \f[C]$HOSTNAME\f[R] is the network +address of the machine that \f[C]serve nfs\f[R] was run on. .PP -This feature is only available on Unix platforms. +This command is only available on Unix platforms. .SS VFS - Virtual File System .PP This command uses the VFS layer. @@ -12590,9 +12932,14 @@ These flags control the chunking: \f[C] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) +--vfs-read-chunk-streams int The number of parallel streams to read at once \f[R] .fi .PP +The chunking behaves differently depending on the +\f[C]--vfs-read-chunk-streams\f[R] parameter. +.SS \f[C]--vfs-read-chunk-streams\f[R] == 0 +.PP Rclone will start reading a chunk of size \f[C]--vfs-read-chunk-size\f[R], and then double the size for each read. When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater @@ -12610,6 +12957,31 @@ on. .PP Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq] disables chunked reading. +.PP +The chunks will not be buffered in memory. +.SS \f[C]--vfs-read-chunk-streams\f[R] > 0 +.PP +Rclone reads \f[C]--vfs-read-chunk-streams\f[R] chunks of size +\f[C]--vfs-read-chunk-size\f[R] concurrently. +The size for each read will stay constant. +.PP +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. +.PP +Some experimentation will be needed to find the optimum values of +\f[C]--vfs-read-chunk-size\f[R] and \f[C]--vfs-read-chunk-streams\f[R] +as these will depend on the backend in use and the latency to the +backend. +.PP +For high performance object stores (eg AWS S3) a reasonable place to +start might be \f[C]--vfs-read-chunk-streams 16\f[R] and +\f[C]--vfs-read-chunk-size 4M\f[R]. +In testing with AWS S3 the performance scaled roughly as the +\f[C]--vfs-read-chunk-streams\f[R] setting. +.PP +Similar settings should work for high latency links, but depending on +the latency they may need more \f[C]--vfs-read-chunk-streams\f[R] in +order to get the throughput. .SS VFS Performance .PP These flags may be used to enable/disable features of the VFS for @@ -12761,18 +13133,20 @@ rclone serve nfs remote:path [flags] \f[C] --addr string IPaddress:Port or :Port to bind server to --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for nfs + --nfs-cache-dir string The directory the NFS handle cache will use if set --nfs-cache-handle-limit int max file handles cached simultaneously (min 5) (default 1000000) + --nfs-cache-type memory|disk|symlink Type of NFS handle cache to use (default memory) --no-checksum Don\[aq]t compare checksums on up/download --no-modtime Don\[aq]t read/write the modification time (can speed things up) --no-seek Don\[aq]t allow seeking in files --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -12785,6 +13159,7 @@ rclone serve nfs remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -12792,9 +13167,13 @@ rclone serve nfs remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -12822,10 +13201,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -12947,6 +13323,8 @@ be done with file system permissions. .PP \f[C]--addr\f[R] may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to +accomplish the same. .PP \f[C]--server-read-timeout\f[R] and \f[C]--server-write-timeout\f[R] can be used to control the timeouts on the server. @@ -12981,7 +13359,29 @@ authority certificate. \f[C]--min-tls-version\f[R] is minimum TLS version that is acceptable. Valid values are \[dq]tls1.0\[dq], \[dq]tls1.1\[dq], \[dq]tls1.2\[dq] and \[dq]tls1.3\[dq] (default \[dq]tls1.0\[dq]). -.SS Authentication +.SS Socket activation +.PP +Instead of the listening addresses specified above, rclone will listen +to all FDs passed by the service manager, if any (and ignore any +arguments passed by --addr\[ga]). +.PP +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described +in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html +.PP +Socket activation can be tested ad-hoc with the +\f[C]systemd-socket-activate\f[R]command +.IP +.nf +\f[C] + systemd-socket-activate -l 8000 -- rclone serve +\f[R] +.fi +.PP +This will socket-activate rclone on the first connection to port 8000 +over TCP. +### Authentication .PP By default this will serve files without needing a login. .PP @@ -13025,7 +13425,7 @@ rclone serve restic remote:path [flags] .IP .nf \f[C] - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --append-only Disallow deletion of repository data --baseurl string Prefix for URLs - leave blank for root @@ -13050,7 +13450,7 @@ rclone serve restic remote:path [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -13227,6 +13627,40 @@ Object .RE .PP Other operations will return error \f[C]Unimplemented\f[R]. +.SS Authentication +.PP +By default this will serve files without needing a login. +.PP +You can either use an htpasswd file which can take lots of users, or set +a single username and password with the \f[C]--user\f[R] and +\f[C]--pass\f[R] flags. +.PP +If no static users are configured by either of the above methods, and +client certificates are required by the \f[C]--client-ca\f[R] flag +passed to the server, the client certificate common name will be +considered as the username. +.PP +Use \f[C]--htpasswd /path/to/htpasswd\f[R] to provide an htpasswd file. +This is in standard apache format and supports MD5, SHA1 and BCrypt for +basic authentication. +Bcrypt is recommended. +.PP +To create an htpasswd file: +.IP +.nf +\f[C] +touch htpasswd +htpasswd -B htpasswd user +htpasswd -B htpasswd anotherUser +\f[R] +.fi +.PP +The password file can be updated while rclone is running. +.PP +Use \f[C]--realm\f[R] to set the authentication realm. +.PP +Use \f[C]--salt\f[R] to change the password hashing salt from the +default. .SS Server options .PP Use \f[C]--addr\f[R] to specify which IP address and port the server @@ -13246,6 +13680,8 @@ be done with file system permissions. .PP \f[C]--addr\f[R] may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to +accomplish the same. .PP \f[C]--server-read-timeout\f[R] and \f[C]--server-write-timeout\f[R] can be used to control the timeouts on the server. @@ -13280,7 +13716,29 @@ authority certificate. \f[C]--min-tls-version\f[R] is minimum TLS version that is acceptable. Valid values are \[dq]tls1.0\[dq], \[dq]tls1.1\[dq], \[dq]tls1.2\[dq] and \[dq]tls1.3\[dq] (default \[dq]tls1.0\[dq]). -.SS VFS - Virtual File System +.SS Socket activation +.PP +Instead of the listening addresses specified above, rclone will listen +to all FDs passed by the service manager, if any (and ignore any +arguments passed by --addr\[ga]). +.PP +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described +in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html +.PP +Socket activation can be tested ad-hoc with the +\f[C]systemd-socket-activate\f[R]command +.IP +.nf +\f[C] + systemd-socket-activate -l 8000 -- rclone serve +\f[R] +.fi +.PP +This will socket-activate rclone on the first connection to port 8000 +over TCP. +## VFS - Virtual File System .PP This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something @@ -13556,9 +14014,14 @@ These flags control the chunking: \f[C] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) +--vfs-read-chunk-streams int The number of parallel streams to read at once \f[R] .fi .PP +The chunking behaves differently depending on the +\f[C]--vfs-read-chunk-streams\f[R] parameter. +.SS \f[C]--vfs-read-chunk-streams\f[R] == 0 +.PP Rclone will start reading a chunk of size \f[C]--vfs-read-chunk-size\f[R], and then double the size for each read. When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater @@ -13576,6 +14039,31 @@ on. .PP Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq] disables chunked reading. +.PP +The chunks will not be buffered in memory. +.SS \f[C]--vfs-read-chunk-streams\f[R] > 0 +.PP +Rclone reads \f[C]--vfs-read-chunk-streams\f[R] chunks of size +\f[C]--vfs-read-chunk-size\f[R] concurrently. +The size for each read will stay constant. +.PP +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. +.PP +Some experimentation will be needed to find the optimum values of +\f[C]--vfs-read-chunk-size\f[R] and \f[C]--vfs-read-chunk-streams\f[R] +as these will depend on the backend in use and the latency to the +backend. +.PP +For high performance object stores (eg AWS S3) a reasonable place to +start might be \f[C]--vfs-read-chunk-streams 16\f[R] and +\f[C]--vfs-read-chunk-size 4M\f[R]. +In testing with AWS S3 the performance scaled roughly as the +\f[C]--vfs-read-chunk-streams\f[R] setting. +.PP +Similar settings should work for high latency links, but depending on +the latency they may need more \f[C]--vfs-read-chunk-streams\f[R] in +order to get the throughput. .SS VFS Performance .PP These flags may be used to enable/disable features of the VFS for @@ -13725,19 +14213,21 @@ rclone serve s3 remote:path [flags] .IP .nf \f[C] - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-key stringArray Set key pair for v4 authorization: access_key_id,secret_access_key + --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --etag-hash string Which hash to use for the ETag, or auto or blank for off (default \[dq]MD5\[dq]) - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --force-path-style If true use path style access if false use virtual hosted style (default true) (default true) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for s3 + --htpasswd string A htpasswd file - if not provided no authentication is done --key string TLS PEM Private key --max-header-bytes int Maximum size of request header (default 4096) --min-tls-version string Minimum TLS version that is acceptable (default \[dq]tls1.0\[dq]) @@ -13745,12 +14235,16 @@ rclone serve s3 remote:path [flags] --no-cleanup Not to cleanup empty folder after object is deleted --no-modtime Don\[aq]t read/write the modification time (can speed things up) --no-seek Don\[aq]t allow seeking in files + --pass string Password for authentication --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) --read-only Only allow read-only access + --realm string Realm for authentication + --salt string Password hashing salt (default \[dq]dlPL2MqE\[dq]) --server-read-timeout Duration Timeout for server reading data (default 1h0m0s) --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) + --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) @@ -13763,6 +14257,7 @@ rclone serve s3 remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -13770,9 +14265,13 @@ rclone serve s3 remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -13800,10 +14299,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -13848,6 +14344,24 @@ in the \[dq]serve-sftp\[dq] directory. By default the server binds to localhost:2022 - if you want it to be reachable externally then supply \f[C]--addr :2022\f[R] for example. .PP +This also supports being run with socket activation, in which case it +will listen on the first passed FD. +It can be configured with .socket and .service unit files as described +in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html +.PP +Socket activation can be tested ad-hoc with the +\f[C]systemd-socket-activate\f[R]command: +.IP +.nf +\f[C] +systemd-socket-activate -l 2222 -- rclone serve sftp :local:vfs/ +\f[R] +.fi +.PP +This will socket-activate rclone on the first connection to port 2222 +over TCP. +.PP Note that the default of \f[C]--vfs-cache-mode off\f[R] is fine for the rclone sftp backend, but it may not be with other SFTP clients. .PP @@ -14149,9 +14663,14 @@ These flags control the chunking: \f[C] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) +--vfs-read-chunk-streams int The number of parallel streams to read at once \f[R] .fi .PP +The chunking behaves differently depending on the +\f[C]--vfs-read-chunk-streams\f[R] parameter. +.SS \f[C]--vfs-read-chunk-streams\f[R] == 0 +.PP Rclone will start reading a chunk of size \f[C]--vfs-read-chunk-size\f[R], and then double the size for each read. When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater @@ -14169,6 +14688,31 @@ on. .PP Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq] disables chunked reading. +.PP +The chunks will not be buffered in memory. +.SS \f[C]--vfs-read-chunk-streams\f[R] > 0 +.PP +Rclone reads \f[C]--vfs-read-chunk-streams\f[R] chunks of size +\f[C]--vfs-read-chunk-size\f[R] concurrently. +The size for each read will stay constant. +.PP +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. +.PP +Some experimentation will be needed to find the optimum values of +\f[C]--vfs-read-chunk-size\f[R] and \f[C]--vfs-read-chunk-streams\f[R] +as these will depend on the backend in use and the latency to the +backend. +.PP +For high performance object stores (eg AWS S3) a reasonable place to +start might be \f[C]--vfs-read-chunk-streams 16\f[R] and +\f[C]--vfs-read-chunk-size 4M\f[R]. +In testing with AWS S3 the performance scaled roughly as the +\f[C]--vfs-read-chunk-streams\f[R] setting. +.PP +Similar settings should work for high latency links, but depending on +the latency they may need more \f[C]--vfs-read-chunk-streams\f[R] in +order to get the throughput. .SS VFS Performance .PP These flags may be used to enable/disable features of the VFS for @@ -14415,8 +14959,8 @@ rclone serve sftp remote:path [flags] --auth-proxy string A program to use to create the backend from the auth --authorized-keys string Authorized keys file (default \[dq]\[ti]/.ssh/authorized_keys\[dq]) --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) - --file-perms FileMode File permissions (default 0666) + --dir-perms FileMode Directory permissions (default 777) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for sftp --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate) @@ -14429,7 +14973,7 @@ rclone serve sftp remote:path [flags] --read-only Only allow read-only access --stdio Run an sftp server on stdin/stdout --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -14443,6 +14987,7 @@ rclone serve sftp remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -14450,9 +14995,13 @@ rclone serve sftp remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -14480,10 +15029,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -14537,6 +15083,26 @@ SSL connections only 2 - Basic authentication enabled for SSL and for non-SSL connections .PP https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint +.SS Serving over a unix socket +.PP +You can serve the webdav on a unix socket like this: +.IP +.nf +\f[C] +rclone serve webdav --addr unix:///tmp/my.socket remote:path +\f[R] +.fi +.PP +and connect to it like this using rclone and the webdav backend: +.IP +.nf +\f[C] +rclone --webdav-unix-socket /tmp/my.socket --webdav-url http://localhost lsf :webdav: +\f[R] +.fi +.PP +Note that there is no authentication on http protocol - this is expected +to be done by the permissions on the socket. .SS Server options .PP Use \f[C]--addr\f[R] to specify which IP address and port the server @@ -14556,6 +15122,8 @@ be done with file system permissions. .PP \f[C]--addr\f[R] may be repeated to listen on multiple IPs/ports/sockets. +Socket activation, described further below, can also be used to +accomplish the same. .PP \f[C]--server-read-timeout\f[R] and \f[C]--server-write-timeout\f[R] can be used to control the timeouts on the server. @@ -14590,7 +15158,29 @@ authority certificate. \f[C]--min-tls-version\f[R] is minimum TLS version that is acceptable. Valid values are \[dq]tls1.0\[dq], \[dq]tls1.1\[dq], \[dq]tls1.2\[dq] and \[dq]tls1.3\[dq] (default \[dq]tls1.0\[dq]). -.SS Template +.SS Socket activation +.PP +Instead of the listening addresses specified above, rclone will listen +to all FDs passed by the service manager, if any (and ignore any +arguments passed by --addr\[ga]). +.PP +This allows rclone to be a socket-activated service. +It can be configured with .socket and .service unit files as described +in +https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html +.PP +Socket activation can be tested ad-hoc with the +\f[C]systemd-socket-activate\f[R]command +.IP +.nf +\f[C] + systemd-socket-activate -l 8000 -- rclone serve +\f[R] +.fi +.PP +This will socket-activate rclone on the first connection to port 8000 +over TCP. +### Template .PP \f[C]--template\f[R] allows a user to specify a custom markup template for HTTP and WebDAV serve functions. @@ -15034,9 +15624,14 @@ These flags control the chunking: \f[C] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) +--vfs-read-chunk-streams int The number of parallel streams to read at once \f[R] .fi .PP +The chunking behaves differently depending on the +\f[C]--vfs-read-chunk-streams\f[R] parameter. +.SS \f[C]--vfs-read-chunk-streams\f[R] == 0 +.PP Rclone will start reading a chunk of size \f[C]--vfs-read-chunk-size\f[R], and then double the size for each read. When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater @@ -15054,6 +15649,31 @@ on. .PP Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq] disables chunked reading. +.PP +The chunks will not be buffered in memory. +.SS \f[C]--vfs-read-chunk-streams\f[R] > 0 +.PP +Rclone reads \f[C]--vfs-read-chunk-streams\f[R] chunks of size +\f[C]--vfs-read-chunk-size\f[R] concurrently. +The size for each read will stay constant. +.PP +This improves performance performance massively on high latency links or +very high bandwidth links to high performance object stores. +.PP +Some experimentation will be needed to find the optimum values of +\f[C]--vfs-read-chunk-size\f[R] and \f[C]--vfs-read-chunk-streams\f[R] +as these will depend on the backend in use and the latency to the +backend. +.PP +For high performance object stores (eg AWS S3) a reasonable place to +start might be \f[C]--vfs-read-chunk-streams 16\f[R] and +\f[C]--vfs-read-chunk-size 4M\f[R]. +In testing with AWS S3 the performance scaled roughly as the +\f[C]--vfs-read-chunk-streams\f[R] setting. +.PP +Similar settings should work for high latency links, but depending on +the latency they may need more \f[C]--vfs-read-chunk-streams\f[R] in +order to get the throughput. .SS VFS Performance .PP These flags may be used to enable/disable features of the VFS for @@ -15296,17 +15916,17 @@ rclone serve webdav remote:path [flags] .IP .nf \f[C] - --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) + --addr stringArray IPaddress:Port, :Port or [unix://]/path/to/socket to bind server to (default [127.0.0.1:8080]) --allow-origin string Origin which cross-domain request (CORS) can be executed from --auth-proxy string A program to use to create the backend from the auth --baseurl string Prefix for URLs - leave blank for root --cert string TLS PEM key (concatenation of certificate and CA certificate) --client-ca string Client certificate authority to verify clients with --dir-cache-time Duration Time to cache directory entries for (default 5m0s) - --dir-perms FileMode Directory permissions (default 0777) + --dir-perms FileMode Directory permissions (default 777) --disable-dir-list Disable HTML directory list on GET request for a directory --etag-hash string Which hash to use for the ETag, or auto or blank for off - --file-perms FileMode File permissions (default 0666) + --file-perms FileMode File permissions (default 666) --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) -h, --help help for webdav --htpasswd string A htpasswd file - if not provided no authentication is done @@ -15325,7 +15945,7 @@ rclone serve webdav remote:path [flags] --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) --template string User-specified template --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) - --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --umask FileMode Override the permission bits set by the filesystem (not supported on Windows) (default 002) --user string User name for authentication --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) @@ -15339,6 +15959,7 @@ rclone serve webdav remote:path [flags] --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) + --vfs-read-chunk-streams int The number of parallel streams to read at once --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size @@ -15346,9 +15967,13 @@ rclone serve webdav remote:path [flags] --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -15376,10 +16001,7 @@ Flags for filtering directory listings. --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. @@ -15388,7 +16010,7 @@ remote over a protocol. Changes storage class/tier of objects in remote. .SS Synopsis .PP -rclone settier changes storage tier or class at remote if supported. +Changes storage tier or class at remote if supported. Few cloud storage services provides different storage classes on objects, for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline @@ -15440,7 +16062,7 @@ rclone settier tier remote:path [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -15473,7 +16095,7 @@ things so reading their documentation first is recommended. .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -15518,7 +16140,7 @@ rclone test changenotify remote: [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone test (https://rclone.org/commands/rclone_test/) - Run a test command @@ -15548,7 +16170,7 @@ rclone test histogram [remote:path] [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone test (https://rclone.org/commands/rclone_test/) - Run a test command @@ -15557,8 +16179,8 @@ command Discovers file name or other limitations for paths. .SS Synopsis .PP -rclone info discovers what filenames and upload methods are possible to -write to the paths passed in and how long they can be. +Discovers what filenames and upload methods are possible to write to the +paths passed in and how long they can be. It can take some time. It will write test files into the remote:path passed in. It outputs a bit of go code for each one. @@ -15590,7 +16212,7 @@ rclone test info [remote:path]+ [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone test (https://rclone.org/commands/rclone_test/) - Run a test command @@ -15619,7 +16241,7 @@ rclone test makefile []+ [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone test (https://rclone.org/commands/rclone_test/) - Run a test command @@ -15655,7 +16277,7 @@ rclone test makefiles [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone test (https://rclone.org/commands/rclone_test/) - Run a test command @@ -15678,7 +16300,7 @@ rclone test memory remote:path [flags] .PP See the global flags page (https://rclone.org/flags/) for global options not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone test (https://rclone.org/commands/rclone_test/) - Run a test command @@ -15730,9 +16352,13 @@ rclone touch remote:path [flags] -t, --timestamp string Use specified time instead of the current time of day \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Important Options .PP -Important flags useful for most commands. +Important flags useful for most commands .IP .nf \f[C] @@ -15743,7 +16369,7 @@ Important flags useful for most commands. .fi .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -15773,7 +16399,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -15781,10 +16407,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -15793,8 +16416,8 @@ commands, flags and backends. List the contents of the remote in a tree like fashion. .SS Synopsis .PP -rclone tree lists the contents of a remote in a similar way to the unix -tree command. +Lists the contents of a remote in a similar way to the unix tree +command. .PP For example .IP @@ -15856,9 +16479,13 @@ rclone tree remote:path [flags] --version Sort files alphanumerically by version \f[R] .fi +.PP +Options shared with other commands are described next. +See the global flags page (https://rclone.org/flags/) for global options +not listed here. .SS Filter Options .PP -Flags for filtering directory listings. +Flags for filtering directory listings .IP .nf \f[C] @@ -15888,7 +16515,7 @@ Flags for filtering directory listings. .fi .SS Listing Options .PP -Flags for listing directories. +Flags for listing directories .IP .nf \f[C] @@ -15896,10 +16523,7 @@ Flags for listing directories. --fast-list Use recursive list if available; uses more memory but fewer transactions \f[R] .fi -.PP -See the global flags page (https://rclone.org/flags/) for global options -not listed here. -.SH SEE ALSO +.SS See Also .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. @@ -16730,13 +17354,22 @@ It is optional. .IP \[bu] 2 \f[C]HH:MM\f[R] is an hour from 00:00 to 23:59. .PP +Entries can be separated by spaces or semicolons. +.PP +\f[B]Note:\f[R] Semicolons can be used as separators instead of spaces +to avoid parsing issues in environments like Docker. +.PP An example of a typical timetable to avoid link saturation during daytime working hours could be: .PP +Using spaces as separators: \f[C]--bwlimit \[dq]08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off\[dq]\f[R] .PP -In this example, the transfer bandwidth will be set to 512 KiB/s at 8am -every day. +Using semicolons as separators: +\f[C]--bwlimit \[dq]08:00,512k;12:00,10M;13:00,512k;18:00,30M;23:00,off\[dq]\f[R] +.PP +In these examples, the transfer bandwidth will be set to 512 KiB/s at +8am every day. At noon, it will rise to 10 MiB/s, and drop back to 512 KiB/sec at 1pm. At 6pm, the bandwidth limit will be set to 30 MiB/s, and at 11pm it will be completely disabled (full speed). @@ -16744,8 +17377,12 @@ Anything between 11pm and 8am will remain unlimited. .PP An example of timetable with \f[C]WEEKDAY\f[R] could be: .PP +Using spaces as separators: \f[C]--bwlimit \[dq]Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off\[dq]\f[R] .PP +Using semicolons as separators: +\f[C]--bwlimit \[dq]Mon-00:00,512;Fri-23:59,10M;Sat-10:00,1M;Sun-20:00,off\[dq]\f[R] +.PP It means that, the transfer bandwidth will be set to 512 KiB/s on Monday. It will rise to 10 MiB/s before the end of Friday. @@ -17449,11 +18086,14 @@ local ftp .IP \[bu] 2 sftp +.IP \[bu] 2 +pcloud .PP Without \f[C]--inplace\f[R] (the default) rclone will first upload to a temporary file with an extension like this, where \f[C]XXXXXX\f[R] -represents a random string and \f[C].partial\f[R] is --partial-suffix -value (\f[C].partial\f[R] by default). +represents a hash of the source file\[aq]s fingerprint and +\f[C].partial\f[R] is --partial-suffix value (\f[C].partial\f[R] by +default). .IP .nf \f[C] @@ -18104,6 +18744,7 @@ This flag supplies a program which should supply the config password when run. This is an alternative to rclone prompting for the password or setting the \f[C]RCLONE_CONFIG_PASS\f[R] variable. +It is also used when setting the config password for the first time. .PP The argument to this should be a command with a space separated list of arguments. @@ -18123,6 +18764,11 @@ Eg \f[R] .fi .PP +Note that when changing the configuration password the environment +variable \f[C]RCLONE_PASSWORD_CHANGE=1\f[R] will be set. +This can be used to distinguish initial decryption of the config file +from the new password. +.PP See the Configuration Encryption for more info. .PP See a Windows PowerShell example on the @@ -18774,6 +19420,20 @@ encryption from your configuration. .PP There is no way to recover the configuration if you lose your password. .PP +You can also use +.IP \[bu] 2 +rclone config encryption +set (https://rclone.org/commands/rclone_config_encryption_set/) to set +the config encryption directly +.IP \[bu] 2 +rclone config encryption +remove (https://rclone.org/commands/rclone_config_encryption_remove/) to +remove it +.IP \[bu] 2 +rclone config encryption +check (https://rclone.org/commands/rclone_config_encryption_check/) to +check that it is encrypted properly. +.PP rclone uses nacl secretbox (https://godoc.org/golang.org/x/crypto/nacl/secretbox) which in turn uses XSalsa20 and Poly1305 to encrypt and authenticate your @@ -18855,11 +19515,59 @@ by overriding the location, e.g. with one of the documented special values for memory-only configuration. Since only backend options can be stored in configuration files, this is normally unnecessary for commands that do not operate on backends, e.g. -\f[C]genautocomplete\f[R]. +\f[C]completion\f[R]. However, it will be relevant for commands that do operate on backends in general, but are used without referencing a stored remote, e.g. listing local filesystem paths, or connection strings: \f[C]rclone --config=\[dq]\[dq] ls .\f[R] +.SS Configuration Encryption Cheatsheet +.PP +You can quickly apply a configuration encryption without plain-text at +rest or transfer. +Detailed instructions for popular OSes: +.SS Mac +.IP \[bu] 2 +Generate and store a password +.PP +\f[C]security add-generic-password -a rclone -s config -w $(openssl rand -base64 40)\f[R] +.IP \[bu] 2 +Add the retrieval instruction to your .zprofile / .profile +.PP +\f[C]export RCLONE_PASSWORD_COMMAND=\[dq]/usr/bin/security find-generic-password -a rclone -s config -w\[dq]\f[R] +.SS Linux +.IP \[bu] 2 +Prerequisite +.PP +Linux doesn\[aq]t come with a default password manager. +Let\[aq]s install the \[dq]pass\[dq] utility using a package manager, +e.g. +\f[C]apt install pass\f[R], \f[C]yum install pass\f[R], +etc. (https://www.passwordstore.org/#download); then initialize a +password store: +.PP +\f[C]pass init rclone\f[R] +.IP \[bu] 2 +Generate and store a password +.PP +\f[C]echo $(openssl rand -base64 40) | pass insert -m rclone/config\f[R] +.IP \[bu] 2 +Add the retrieval instruction +.PP +\f[C]export RCLONE_PASSWORD_COMMAND=\[dq]/usr/bin/pass rclone/config\[dq]\f[R] +.SS Windows +.IP \[bu] 2 +Generate and store a password +.PP +\f[C]New-Object -TypeName PSCredential -ArgumentList \[dq]rclone\[dq], (ConvertTo-SecureString -String ([System.Web.Security.Membership]::GeneratePassword(40, 10)) -AsPlainText -Force) | Export-Clixml -Path \[dq]rclone-credential.xml\[dq]\f[R] +.IP \[bu] 2 +Add the password retrieval instruction +.PP +\f[C][Environment]::SetEnvironmentVariable(\[dq]RCLONE_PASSWORD_COMMAND\[dq], \[dq][System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR((Import-Clixml -Path \[dq]rclone-credential.xml\[dq]).Password))\[dq])\f[R] +.SS Encrypt the config file (all systems) +.IP \[bu] 2 +Execute \f[C]rclone config\f[R] -> \f[C]s\f[R] +.IP \[bu] 2 +Add/update the password from previous steps .SS Developer options .PP These options are useful when developing or debugging rclone. @@ -19031,6 +19739,25 @@ and the \f[C]--syslog-facility\f[R] control which facility it uses. Rclone prefixes all log messages with their level in capitals, e.g. INFO which makes it easy to grep the log file for different kinds of information. +.SS Metrics +.PP +Rclone can publish metrics in the OpenMetrics/Prometheus format. +.PP +To enable the metrics endpoint, use the \f[C]--metrics-addr\f[R] flag. +Metrics can also be published on the \f[C]--rc-addr\f[R] port if the +\f[C]--rc\f[R] flag and \f[C]--rc-enable-metrics\f[R] flags are supplied +or if using rclone rcd \f[C]--rc-enable-metrics\f[R] +.PP +Rclone provides extensive configuration options for the metrics HTTP +endpoint. +These settings are grouped under the Metrics section and have a prefix +\f[C]--metrics-*\f[R]. +.PP +When metrics are enabled with \f[C]--rc-enable-metrics\f[R], they will +be published on the same port as the rc API. +In this case, the \f[C]--metrics-*\f[R] flags will be ignored, and the +HTTP endpoint configuration will be managed by the \f[C]--rc-*\f[R] +parameters. .SS Exit Code .PP If any errors occur during the command execution, rclone will exit with @@ -19259,32 +19986,28 @@ There are two ways of doing it, described below. .SS Configuring using rclone authorize .PP On the headless box run \f[C]rclone\f[R] config but answer \f[C]N\f[R] -to the \f[C]Use web browser to automatically authenticate?\f[R] -question. +to the \f[C]Use auto config?\f[R] question. .IP .nf \f[C] -\&... -Remote config -Use web browser to automatically authenticate rclone with remote? - * Say Y if the machine running rclone has a web browser you can use - * Say N if running rclone on a (remote) machine without web browser access -If not sure try Y. If Y failed, try N. +Use auto config? + * Say Y if not sure + * Say N if you are working on a remote or headless machine + y) Yes (default) n) No y/n> n + +Option config_token. For this to work, you will need rclone available on a machine that has a web browser available. - For more help and alternate methods see: https://rclone.org/remote_setup/ - Execute the following on the machine with the web browser (same rclone version recommended): - - rclone authorize \[dq]dropbox\[dq] - -Then paste the result below: -result> + rclone authorize \[dq]onedrive\[dq] +Then paste the result. +Enter a value. +config_token> \f[R] .fi .PP @@ -19292,7 +20015,7 @@ Then on your main desktop machine .IP .nf \f[C] -rclone authorize \[dq]dropbox\[dq] +rclone authorize \[dq]onedrive\[dq] If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... @@ -19307,7 +20030,7 @@ Then back to the headless box, paste in the code .IP .nf \f[C] -result> SECRET_TOKEN +config_token> SECRET_TOKEN -------------------- [acd12] client_id = @@ -19360,18 +20083,15 @@ ssh -L localhost:53682:localhost:53682 username\[at]remote_server \f[R] .fi .PP -Then on the headless box run \f[C]rclone\f[R] config and answer -\f[C]Y\f[R] to the -\f[C]Use web browser to automatically authenticate?\f[R] question. +Then on the headless box run \f[C]rclone config\f[R] and answer +\f[C]Y\f[R] to the \f[C]Use auto config?\f[R] question. .IP .nf \f[C] -\&... -Remote config -Use web browser to automatically authenticate rclone with remote? - * Say Y if the machine running rclone has a web browser you can use - * Say N if running rclone on a (remote) machine without web browser access -If not sure try Y. If Y failed, try N. +Use auto config? + * Say Y if not sure + * Say N if you are working on a remote or headless machine + y) Yes (default) n) No y/n> y @@ -20415,8 +21135,9 @@ The fix then is to quote values containing spaces. .SS \f[C]--min-size\f[R] - Don\[aq]t transfer any file smaller than this .PP Controls the minimum size file within the scope of an rclone command. -Default units are \f[C]KiB\f[R] but abbreviations \f[C]K\f[R], -\f[C]M\f[R], \f[C]G\f[R], \f[C]T\f[R] or \f[C]P\f[R] are valid. +Default units are \f[C]KiB\f[R] but abbreviations \f[C]B\f[R], +\f[C]K\f[R], \f[C]M\f[R], \f[C]G\f[R], \f[C]T\f[R] or \f[C]P\f[R] are +valid. .PP E.g. \f[C]rclone ls remote: --min-size 50k\f[R] lists files on @@ -20427,8 +21148,9 @@ info. .SS \f[C]--max-size\f[R] - Don\[aq]t transfer any file larger than this .PP Controls the maximum size file within the scope of an rclone command. -Default units are \f[C]KiB\f[R] but abbreviations \f[C]K\f[R], -\f[C]M\f[R], \f[C]G\f[R], \f[C]T\f[R] or \f[C]P\f[R] are valid. +Default units are \f[C]KiB\f[R] but abbreviations \f[C]B\f[R], +\f[C]K\f[R], \f[C]M\f[R], \f[C]G\f[R], \f[C]T\f[R] or \f[C]P\f[R] are +valid. .PP E.g. \f[C]rclone ls remote: --max-size 1G\f[R] lists files on @@ -20794,6 +21516,9 @@ Default Off. .SS --rc-enable-metrics .PP Enable OpenMetrics/Prometheus compatible endpoint at \f[C]/metrics\f[R]. +If more control over the metrics is desired (for example running it on a +different port or with different auth) then endpoint can be enabled with +the \f[C]--metrics-*\f[R] flags instead. .PP Default Off. .SS --rc-web-gui @@ -21146,6 +21871,226 @@ be set as a string, eg \[dq]HARD\[dq] for \f[C]CutoffMode\f[R] or .IP \[bu] 2 \f[C]BandwidthSpec\f[R] - this will be set and returned as a string, eg \[dq]1M\[dq]. +.SS Option blocks +.PP +The calls options/info (for the main config) and config/providers (for +the backend config) may be used to get information on the rclone +configuration options. +This can be used to build user interfaces for displaying and setting any +rclone option. +.PP +These consist of arrays of \f[C]Option\f[R] blocks. +These have the following format. +Each block describes a single option. +.PP +.TS +tab(@); +lw(13.6n) lw(11.7n) lw(19.4n) lw(25.3n). +T{ +Field +T}@T{ +Type +T}@T{ +Optional +T}@T{ +Description +T} +_ +T{ +Name +T}@T{ +string +T}@T{ +N +T}@T{ +name of the option in snake_case +T} +T{ +FieldName +T}@T{ +string +T}@T{ +N +T}@T{ +name of the field used in the rc - if blank use Name +T} +T{ +Help +T}@T{ +string +T}@T{ +N +T}@T{ +help, started with a single sentence on a single line +T} +T{ +Groups +T}@T{ +string +T}@T{ +Y +T}@T{ +groups this option belongs to - comma separated string for options +classification +T} +T{ +Provider +T}@T{ +string +T}@T{ +Y +T}@T{ +set to filter on provider +T} +T{ +Default +T}@T{ +any +T}@T{ +N +T}@T{ +default value, if set (and not to nil or \[dq]\[dq]) then Required does +nothing +T} +T{ +Value +T}@T{ +any +T}@T{ +N +T}@T{ +value to be set by flags +T} +T{ +Examples +T}@T{ +Examples +T}@T{ +Y +T}@T{ +predefined values that can be selected from list (multiple-choice +option) +T} +T{ +ShortOpt +T}@T{ +string +T}@T{ +Y +T}@T{ +the short command line option for this +T} +T{ +Hide +T}@T{ +Visibility +T}@T{ +N +T}@T{ +if non zero, this option is hidden from the configurator or the command +line +T} +T{ +Required +T}@T{ +bool +T}@T{ +N +T}@T{ +this option is required, meaning value cannot be empty unless there is a +default +T} +T{ +IsPassword +T}@T{ +bool +T}@T{ +N +T}@T{ +set if the option is a password +T} +T{ +NoPrefix +T}@T{ +bool +T}@T{ +N +T}@T{ +set if the option for this should not use the backend prefix +T} +T{ +Advanced +T}@T{ +bool +T}@T{ +N +T}@T{ +set if this is an advanced config option +T} +T{ +Exclusive +T}@T{ +bool +T}@T{ +N +T}@T{ +set if the answer can only be one of the examples (empty string allowed +unless Required or Default is set) +T} +T{ +Sensitive +T}@T{ +bool +T}@T{ +N +T}@T{ +set if this option should be redacted when using +\f[C]rclone config redacted\f[R] +T} +.TE +.PP +An example of this might be the \f[C]--log-level\f[R] flag. +Note that the \f[C]Name\f[R] of the option becomes the command line flag +with \f[C]_\f[R] replaced with \f[C]-\f[R]. +.IP +.nf +\f[C] +{ + \[dq]Advanced\[dq]: false, + \[dq]Default\[dq]: 5, + \[dq]DefaultStr\[dq]: \[dq]NOTICE\[dq], + \[dq]Examples\[dq]: [ + { + \[dq]Help\[dq]: \[dq]\[dq], + \[dq]Value\[dq]: \[dq]EMERGENCY\[dq] + }, + { + \[dq]Help\[dq]: \[dq]\[dq], + \[dq]Value\[dq]: \[dq]ALERT\[dq] + }, + ... + ], + \[dq]Exclusive\[dq]: true, + \[dq]FieldName\[dq]: \[dq]LogLevel\[dq], + \[dq]Groups\[dq]: \[dq]Logging\[dq], + \[dq]Help\[dq]: \[dq]Log level DEBUG|INFO|NOTICE|ERROR\[dq], + \[dq]Hide\[dq]: 0, + \[dq]IsPassword\[dq]: false, + \[dq]Name\[dq]: \[dq]log_level\[dq], + \[dq]NoPrefix\[dq]: true, + \[dq]Required\[dq]: true, + \[dq]Sensitive\[dq]: false, + \[dq]Type\[dq]: \[dq]LogLevel\[dq], + \[dq]Value\[dq]: null, + \[dq]ValueStr\[dq]: \[dq]NOTICE\[dq] +}, +\f[R] +.fi +.PP +Note that the \f[C]Help\f[R] may be multiple lines separated by +\f[C]\[rs]n\f[R]. +The first line will always be a short sentence and this is the sentence +shown when running \f[C]rclone help flags\f[R]. .SS Specifying remotes to work on .PP Remotes are specified with the \f[C]fs=\f[R], \f[C]srcFs=\f[R], @@ -21180,7 +22125,7 @@ For example this JSON is equivalent to \f[C]remote:/tmp\f[R] \f[C] { \[dq]_name\[dq]: \[dq]remote\[dq], - \[dq]_path\[dq]: \[dq]/tmp\[dq] + \[dq]_root\[dq]: \[dq]/tmp\[dq] } \f[R] .fi @@ -21193,7 +22138,7 @@ And this is equivalent to { \[dq]type\[dq]: \[dq]sftp\[dq], \[dq]host\[dq]: \[dq]example.com\[dq], - \[dq]_path\[dq]: \[dq]/tmp\[dq] + \[dq]_root\[dq]: \[dq]/tmp\[dq] } \f[R] .fi @@ -21204,7 +22149,7 @@ And this is equivalent to \f[C]/tmp/dir\f[R] \f[C] { type = \[dq]local\[dq], - _ path = \[dq]/tmp/dir\[dq] + _root = \[dq]/tmp/dir\[dq] } \f[R] .fi @@ -21438,6 +22383,10 @@ See the config providers (https://rclone.org/commands/rclone_config_providers/) command for more information on the above. .PP +Note that the Options blocks are in the same format as returned by +\[dq]options/info\[dq]. +They are described in the option blocks section. +.PP \f[B]Authentication is required for this call.\f[R] .SS config/setpath: Set the path of the config file .PP @@ -22678,6 +23627,14 @@ Returns: - options - a list of the options block names Returns an object where keys are option block names and values are an object with the current option values in. .PP +Parameters: +.IP \[bu] 2 +blocks: optional string of comma separated blocks to include +.RS 2 +.IP \[bu] 2 +all are included if this is missing or \[dq]\[dq] +.RE +.PP Note that these are the global options which are unaffected by use of the _config and _filter parameters. If you wish to read the parameters set in _config then use @@ -22685,6 +23642,22 @@ options/config and for _filter use options/filter. .PP This shows the internal names of the option within rclone which should map to the external options very easily with a few exceptions. +.SS options/info: Get info about all the global options +.PP +Returns an object where keys are option block names and values are an +array of objects with info about each options. +.PP +Parameters: +.IP \[bu] 2 +blocks: optional string of comma separated blocks to include +.RS 2 +.IP \[bu] 2 +all are included if this is missing or \[dq]\[dq] +.RE +.PP +These objects are in the same format as returned by +\[dq]config/providers\[dq]. +They are described in the option blocks section. .SS options/local: Get the currently active config for this call .PP Returns an object with the keys \[dq]config\[dq] and \[dq]filter\[dq]. @@ -23041,6 +24014,82 @@ If this parameter is not supplied and if there is only one VFS in use then that VFS will be used. If there is more than one VFS in use then the \[dq]fs\[dq] parameter must be supplied. +.SS vfs/queue: Queue info for a VFS. +.PP +This returns info about the upload queue for the selected VFS. +.PP +This is only useful if \f[C]--vfs-cache-mode\f[R] > off. +If you call it when the \f[C]--vfs-cache-mode\f[R] is off, it will +return an empty result. +.IP +.nf +\f[C] +{ + \[dq]queued\[dq]: // an array of files queued for upload + [ + { + \[dq]name\[dq]: \[dq]file\[dq], // string: name (full path) of the file, + \[dq]id\[dq]: 123, // integer: id of this item in the queue, + \[dq]size\[dq]: 79, // integer: size of the file in bytes + \[dq]expiry\[dq]: 1.5 // float: time until file is eligible for transfer, lowest goes first + \[dq]tries\[dq]: 1, // integer: number of times we have tried to upload + \[dq]delay\[dq]: 5.0, // float: seconds between upload attempts + \[dq]uploading\[dq]: false, // boolean: true if item is being uploaded + }, + ], +} +\f[R] +.fi +.PP +The \f[C]expiry\f[R] time is the time until the file is elegible for +being uploaded in floating point seconds. +This may go negative. +As rclone only transfers \f[C]--transfers\f[R] files at once, only the +lowest \f[C]--transfers\f[R] expiry times will have \f[C]uploading\f[R] +as \f[C]true\f[R]. +So there may be files with negative expiry times for which +\f[C]uploading\f[R] is \f[C]false\f[R]. +.PP +This command takes an \[dq]fs\[dq] parameter. +If this parameter is not supplied and if there is only one VFS in use +then that VFS will be used. +If there is more than one VFS in use then the \[dq]fs\[dq] parameter +must be supplied. +.SS vfs/queue-set-expiry: Set the expiry time for an item queued for upload. +.PP +Use this to adjust the \f[C]expiry\f[R] time for an item in the upload +queue. +You will need to read the \f[C]id\f[R] of the item using +\f[C]vfs/queue\f[R] before using this call. +.PP +You can then set \f[C]expiry\f[R] to a floating point number of seconds +from now when the item is eligible for upload. +If you want the item to be uploaded as soon as possible then set it to a +large negative number (eg -1000000000). +If you want the upload of the item to be delayed for a long time then +set it to a large positive number. +.PP +Setting the \f[C]expiry\f[R] of an item which has already has started +uploading will have no effect - the item will carry on being uploaded. +.PP +This will return an error if called with \f[C]--vfs-cache-mode\f[R] off +or if the \f[C]id\f[R] passed is not found. +.PP +This takes the following parameters +.IP \[bu] 2 +\f[C]fs\f[R] - select the VFS in use (optional) +.IP \[bu] 2 +\f[C]id\f[R] - a numeric ID as returned from \f[C]vfs/queue\f[R] +.IP \[bu] 2 +\f[C]expiry\f[R] - a new expiry time as floating point seconds +.PP +This returns an empty result on success, or an error. +.PP +This command takes an \[dq]fs\[dq] parameter. +If this parameter is not supplied and if there is only one VFS in use +then that VFS will be used. +If there is more than one VFS in use then the \[dq]fs\[dq] parameter +must be supplied. .SS vfs/refresh: Refresh the directory cache. .PP This reads the directories for the specified paths and freshens the @@ -23531,6 +24580,21 @@ T}@T{ - T} T{ +Files.com +T}@T{ +MD5, CRC32 +T}@T{ +DR/W +T}@T{ +Yes +T}@T{ +No +T}@T{ +R +T}@T{ +- +T} +T{ FTP T}@T{ - @@ -23546,6 +24610,21 @@ T}@T{ - T} T{ +Gofile +T}@T{ +MD5 +T}@T{ +DR/W +T}@T{ +No +T}@T{ +Yes +T}@T{ +R +T}@T{ +- +T} +T{ Google Cloud Storage T}@T{ MD5 @@ -23861,6 +24940,21 @@ T}@T{ - T} T{ +Pixeldrain +T}@T{ +SHA256 +T}@T{ +R/W +T}@T{ +No +T}@T{ +No +T}@T{ +R +T}@T{ +RW +T} +T{ premiumize.me T}@T{ - @@ -24696,8 +25790,8 @@ translated to regular (halfwidth) \f[C]*\f[R], \f[C]?\f[R] and .PP The \f[C]--backend-encoding\f[R] flags allow you to change that. You can disable the encoding completely with -\f[C]--backend-encoding None\f[R] or set \f[C]encoding = None\f[R] in -the config file. +\f[C]--backend-encoding Raw\f[R] or set \f[C]encoding = Raw\f[R] in the +config file. .PP Encoding takes a comma separated list of encodings. You can see the list of all possible values by passing an invalid value @@ -24788,6 +25882,13 @@ T}@T{ \f[C]\[uFF02]\f[R] T} T{ +Exclamation +T}@T{ +\f[C]!\f[R] +T}@T{ +\f[C]\[uFF01]\f[R] +T} +T{ Hash T}@T{ \f[C]#\f[R] @@ -24839,10 +25940,11 @@ T}@T{ \f[C]\[uFF1C]\f[R], \f[C]\[uFF1E]\f[R] T} T{ -None +None \[S1] T}@T{ -No characters are encoded +NUL 0x00 T}@T{ +\[u2400] T} T{ Percent @@ -24916,6 +26018,11 @@ T}@T{ \f[C]\[uFF3B]\f[R], \f[C]\[uFF3D]\f[R] T} .TE +.PP +\[S1] Encoding from NUL 0x00 to \[u2400] is always implicit except when +using Raw. +It was previously incorrectly documented as disabling encoding, and to +maintain backward compatibility, its behavior has not been changed. .SS Encoding example: FTP .PP To take a specific example, the FTP backend\[aq]s default encoding is @@ -24980,7 +26087,7 @@ the default value but without \f[C]Colon,Question,Asterisk\f[R]: .fi .PP Alternatively, you can disable the conversion of any characters with -\f[C]--local-encoding None\f[R]. +\f[C]--local-encoding Raw\f[R]. .PP Instead of using command-line argument \f[C]--local-encoding\f[R], you may also set it as environment @@ -25302,6 +26409,31 @@ T}@T{ Yes T} T{ +Files.com +T}@T{ +Yes +T}@T{ +Yes +T}@T{ +Yes +T}@T{ +Yes +T}@T{ +No +T}@T{ +No +T}@T{ +Yes +T}@T{ +No +T}@T{ +Yes +T}@T{ +No +T}@T{ +Yes +T} +T{ FTP T}@T{ No @@ -25327,6 +26459,31 @@ T}@T{ Yes T} T{ +Gofile +T}@T{ +Yes +T}@T{ +Yes +T}@T{ +Yes +T}@T{ +Yes +T}@T{ +No +T}@T{ +No +T}@T{ +Yes +T}@T{ +No +T}@T{ +Yes +T}@T{ +Yes +T}@T{ +Yes +T} +T{ Google Cloud Storage T}@T{ Yes @@ -25852,6 +27009,31 @@ T}@T{ Yes T} T{ +Pixeldrain +T}@T{ +Yes +T}@T{ +No +T}@T{ +Yes +T}@T{ +Yes +T}@T{ +No +T}@T{ +No +T}@T{ +Yes +T}@T{ +No +T}@T{ +Yes +T}@T{ +Yes +T}@T{ +Yes +T} +T{ premiumize.me T}@T{ Yes @@ -26382,13 +27564,13 @@ This describes the global flags available to every rclone command split into groups. .SS Copy .PP -Flags for anything which can Copy a file. +Flags for anything which can copy a file. .IP .nf \f[C] --check-first Do all the checks before starting transfers - -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). - --compare-dest stringArray Include additional comma separated server-side paths during comparison + -c, --checksum Check for changes with size & checksum (if available, or fallback to size only) + --compare-dest stringArray Include additional server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing @@ -26422,7 +27604,7 @@ Flags for anything which can Copy a file. .fi .SS Sync .PP -Flags just used for \f[C]rclone sync\f[R]. +Flags used for sync commands. .IP .nf \f[C] @@ -26453,7 +27635,7 @@ Important flags useful for most commands. .fi .SS Check .PP -Flags used for \f[C]rclone check\f[R]. +Flags used for check commands. .IP .nf \f[C] @@ -26462,7 +27644,7 @@ Flags used for \f[C]rclone check\f[R]. .fi .SS Networking .PP -General networking and HTTP stuff. +Flags for general networking and HTTP stuff. .IP .nf \f[C] @@ -26473,7 +27655,7 @@ General networking and HTTP stuff. --client-cert string Client SSL certificate (PEM) for mutual TLS auth --client-key string Client SSL private key (PEM) for mutual TLS auth --contimeout Duration Connect timeout (default 1m0s) - --disable-http-keep-alives Disable HTTP keep-alives and use each connection once. + --disable-http-keep-alives Disable HTTP keep-alives and use each connection once --disable-http2 Disable HTTP/2 in the global transport --dscp string Set DSCP value to connections, value or name, e.g. CS1, LE, DF, AF21 --expect-continue-timeout Duration Timeout when using expect / 100-continue in HTTP (default 1s) @@ -26486,7 +27668,7 @@ General networking and HTTP stuff. --tpslimit float Limit HTTP transactions per second to this --tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --use-cookies Enable session cookiejar - --user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.67.0\[dq]) + --user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.68.0\[dq]) \f[R] .fi .SS Performance @@ -26502,7 +27684,7 @@ Flags helpful for increasing performance. .fi .SS Config .PP -General configuration of rclone. +Flags for general configuration of rclone. .IP .nf \f[C] @@ -26586,7 +27768,7 @@ Flags for listing directories. .fi .SS Logging .PP -Logging and statistics. +Flags for logging and statistics. .IP .nf \f[C] @@ -26606,7 +27788,7 @@ Logging and statistics. --stats-one-line-date-format string Enable --stats-one-line-date and use custom formatted date: Enclose date string in double quotes (\[dq]), see https://golang.org/pkg/time/#Time.Format --stats-unit string Show data rate in stats as either \[aq]bits\[aq] or \[aq]bytes\[aq] per second (default \[dq]bytes\[dq]) --syslog Use Syslog for logging - --syslog-facility string Facility for syslog, e.g. KERN,USER,... (default \[dq]DAEMON\[dq]) + --syslog-facility string Facility for syslog, e.g. KERN,USER (default \[dq]DAEMON\[dq]) --use-json-log Use json log format -v, --verbose count Print lots more stuff (repeat for more) \f[R] @@ -26635,12 +27817,12 @@ Flags to control the Remote Control API. .nf \f[C] --rc Enable the remote control server - --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572]) + --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [\[dq]localhost:5572\[dq]]) --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from --rc-baseurl string Prefix for URLs - leave blank for root --rc-cert string TLS PEM key (concatenation of certificate and CA certificate) --rc-client-ca string Client certificate authority to verify clients with - --rc-enable-metrics Enable prometheus metrics on /metrics + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server --rc-files string Path to local files to serve on the HTTP server --rc-htpasswd string A htpasswd file - if not provided no authentication is done --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s) @@ -26665,10 +27847,34 @@ Flags to control the Remote Control API. --rc-web-gui-update Check and update to latest version of web gui \f[R] .fi +.SS Metrics +.PP +Flags to control the Metrics HTTP endpoint.. +.IP +.nf +\f[C] + --metrics-addr stringArray IPaddress:Port or :Port to bind metrics server to (default [\[dq]\[dq]]) + --metrics-allow-origin string Origin which cross-domain request (CORS) can be executed from + --metrics-baseurl string Prefix for URLs - leave blank for root + --metrics-cert string TLS PEM key (concatenation of certificate and CA certificate) + --metrics-client-ca string Client certificate authority to verify clients with + --metrics-htpasswd string A htpasswd file - if not provided no authentication is done + --metrics-key string TLS PEM Private key + --metrics-max-header-bytes int Maximum size of request header (default 4096) + --metrics-min-tls-version string Minimum TLS version that is acceptable (default \[dq]tls1.0\[dq]) + --metrics-pass string Password for authentication + --metrics-realm string Realm for authentication + --metrics-salt string Password hashing salt (default \[dq]dlPL2MqE\[dq]) + --metrics-server-read-timeout Duration Timeout for server reading data (default 1h0m0s) + --metrics-server-write-timeout Duration Timeout for server writing data (default 1h0m0s) + --metrics-template string User-specified template + --metrics-user string User name for authentication + --rc-enable-metrics Enable the Prometheus metrics path at the remote control server +\f[R] +.fi .SS Backend .PP -Backend only flags. -These can be set in the config file also. +Backend-only flags (these can be set in the config file also). .IP .nf \f[C] @@ -26893,6 +28099,12 @@ These can be set in the config file also. --filefabric-token-expiry string Token expiry time --filefabric-url string URL of the Enterprise File Fabric to connect to --filefabric-version string Version read from the file fabric + --filescom-api-key string The API key used to authenticate with Files.com + --filescom-description string Description of the remote + --filescom-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot) + --filescom-password string The password used to authenticate with Files.com (obscured) + --filescom-site string Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com) + --filescom-username string The username used to authenticate with Files.com --ftp-ask-password Allow asking for FTP password when needed --ftp-close-timeout Duration Maximum time to wait for a response to close (default 1m0s) --ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited @@ -26936,6 +28148,12 @@ These can be set in the config file also. --gcs-token string OAuth Access Token as a JSON blob --gcs-token-url string Token server url --gcs-user-project string User project + --gofile-access-token string API Access token + --gofile-account-id string Account ID + --gofile-description string Description of the remote + --gofile-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftPeriod,RightPeriod,InvalidUtf8,Dot,Exclamation) + --gofile-list-chunk int Number of items to list in each call (default 1000) + --gofile-root-folder-id string ID of the root folder --gphotos-auth-url string Auth server URL --gphotos-batch-commit-timeout Duration Max time to wait for a batch to finish committing (default 10m0s) --gphotos-batch-mode string Upload file batching sync|async|off (default \[dq]sync\[dq]) @@ -27027,6 +28245,7 @@ These can be set in the config file also. --local-description string Description of the remote --local-encoding Encoding The encoding for the backend (default Slash,Dot) --local-no-check-updated Don\[aq]t check to see if the files change during upload + --local-no-clone Disable reflink cloning for server-side copies --local-no-preallocate Disable preallocation of disk space for transferred files --local-no-set-modtime Disable setting modtime --local-no-sparse Disable sparse files for multi-thread downloads @@ -27143,6 +28362,10 @@ These can be set in the config file also. --pikpak-upload-concurrency int Concurrency for multipart uploads (default 5) --pikpak-use-trash Send files to the trash instead of deleting permanently (default true) --pikpak-user string Pikpak username + --pixeldrain-api-key string API key for your pixeldrain account + --pixeldrain-api-url string The API endpoint to connect to. In the vast majority of cases it\[aq]s fine to leave (default \[dq]https://pixeldrain.com/api\[dq]) + --pixeldrain-description string Description of the remote + --pixeldrain-root-folder-id string Root of the filesystem to use (default \[dq]me\[dq]) --premiumizeme-auth-url string Auth server URL --premiumizeme-client-id string OAuth Client Id --premiumizeme-client-secret string OAuth Client Secret @@ -27217,6 +28440,7 @@ These can be set in the config file also. --s3-provider string Choose your S3 provider --s3-region string Region to connect to --s3-requester-pays Enables requester pays option when interacting with S3 bucket + --s3-sdk-log-mode Bits Set to debug the SDK (default Off) --s3-secret-access-key string AWS Secret Access Key (password) --s3-server-side-encryption string The server-side encryption algorithm used when storing this object in S3 --s3-session-token string An AWS session token @@ -27227,7 +28451,6 @@ These can be set in the config file also. --s3-sse-customer-key-md5 string If using SSE-C you may provide the secret encryption key MD5 checksum (optional) --s3-sse-kms-key-id string If using KMS ID you must provide the ARN of Key --s3-storage-class string The storage class to use when storing new objects in S3 - --s3-sts-endpoint string Endpoint for STS --s3-upload-concurrency int Concurrency for multipart uploads and copies (default 4) --s3-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi) --s3-use-accelerate-endpoint If true use the AWS S3 accelerated endpoint @@ -27237,6 +28460,7 @@ These can be set in the config file also. --s3-use-multipart-etag Tristate Whether to use ETag in multipart uploads for verification (default unset) --s3-use-multipart-uploads Tristate Set if rclone should use multipart uploads (default unset) --s3-use-presigned-request Whether to use a presigned request or PutObject for single part uploads + --s3-use-unsigned-payload Tristate Whether to use an unsigned payload in PutObject (default unset) --s3-v2-auth If true use v2 authentication --s3-version-at Time Show file versions as they were at the specified time (default off) --s3-version-deleted Show deleted file markers when using versions @@ -27345,10 +28569,12 @@ These can be set in the config file also. --swift-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8) --swift-endpoint-type string Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) (default \[dq]public\[dq]) --swift-env-auth Get swift credentials from environment variables in standard OpenStack form + --swift-fetch-until-empty-page When paginating, always fetch unless we received an empty page --swift-key string API key or password (OS_PASSWORD) --swift-leave-parts-on-error If true avoid calling abort upload on a failure --swift-no-chunk Don\[aq]t chunk files during streaming upload --swift-no-large-objects Disable support for static and dynamic large objects + --swift-partial-page-fetch-threshold int When paginating, fetch if the current page is within this percentage of the limit --swift-region string Region name - optional (OS_REGION_NAME) --swift-storage-policy string The storage policy to use when creating a new container --swift-storage-url string Storage URL - optional (OS_STORAGE_URL) @@ -27386,6 +28612,7 @@ These can be set in the config file also. --webdav-owncloud-exclude-shares Exclude ownCloud shares --webdav-pacer-min-sleep Duration Minimum time to sleep between API calls (default 10ms) --webdav-pass string Password (obscured) + --webdav-unix-socket string Path to a unix domain socket to dial to, instead of opening a TCP connection directly --webdav-url string URL of http host to connect to --webdav-user string User name --webdav-vendor string Name of the WebDAV site/service/software you are using @@ -27395,6 +28622,7 @@ These can be set in the config file also. --yandex-description string Description of the remote --yandex-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot) --yandex-hard-delete Delete files permanently rather than putting them into the trash + --yandex-spoof-ua Set the user agent to match an official version of the yandex disk client. May help with upload performance (default true) --yandex-token string OAuth Access Token as a JSON blob --yandex-token-url string Token server url --zoho-auth-url string Auth server URL @@ -30522,6 +31750,13 @@ Also note a number of academic publications by Benjamin Pierce (http://www.cis.upenn.edu/%7Ebcpierce/papers/index.shtml#File%20Synchronization) about \f[I]Unison\f[R] and synchronization in general. .SS Changelog +.SS \f[C]v1.68\f[R] +.IP \[bu] 2 +Fixed an issue affecting backends that round modtimes to a lower +precision. +.SS \f[C]v1.67\f[R] +.IP \[bu] 2 +Added integration tests against all backends. .SS \f[C]v1.66\f[R] .IP \[bu] 2 Copies and deletes are now handled in one operation instead of two @@ -30803,7 +32038,7 @@ You can verify the signatures and hashes in one command line like this: .IP .nf \f[C] -$ gpg --decrypt SHA256SUMS | sha256sum -c --ignore-missing +$ h=$(gpg --decrypt SHA256SUMS) && echo \[dq]$h\[dq] | sha256sum - -c --ignore-missing gpg: Signature made Mon 17 Jul 2023 15:03:17 BST gpg: using DSA key FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA gpg: Good signature from \[dq]Nick Craig-Wood \[dq] [ultimate] @@ -30864,11 +32099,11 @@ y) Yes n) No y/n> Remote config --------------------- -[remote] -type = fichier -api_key = example_key --------------------- +Configuration complete. +Options: +- type: fichier +- api_key: example_key +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -31191,10 +32426,11 @@ Remote or path to alias. Can be \[dq]myremote:path/to/dir\[dq], \[dq]myremote:bucket\[dq], \[dq]myremote:\[dq] or \[dq]/local/path\[dq]. remote> /mnt/storage/backup Remote config --------------------- -[remote] -remote = /mnt/storage/backup --------------------- +Configuration complete. +Options: +- type: alias +- remote: /mnt/storage/backup +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -31576,20 +32812,20 @@ Choose a number from below, or type in your own value \[rs] \[dq]GLACIER_IR\[dq] storage_class> 1 Remote config --------------------- -[remote] -type = s3 -provider = AWS -env_auth = false -access_key_id = XXX -secret_access_key = YYY -region = us-east-1 -endpoint = -location_constraint = -acl = private -server_side_encryption = -storage_class = --------------------- +Configuration complete. +Options: +- type: s3 +- provider: AWS +- env_auth: false +- access_key_id: XXX +- secret_access_key: YYY +- region: us-east-1 +- endpoint: +- location_constraint: +- acl: private +- server_side_encryption: +- storage_class: +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -32008,12 +33244,15 @@ Profile files are standard files used by AWS CLI tools .IP \[bu] 2 By default it will use the profile in your home directory (e.g. \f[C]\[ti]/.aws/credentials\f[R] on unix based systems) file and the -\[dq]default\[dq] profile, to change set these environment variables: +\[dq]default\[dq] profile, to change set these environment variables or +config keys: .RS 2 .IP \[bu] 2 -\f[C]AWS_SHARED_CREDENTIALS_FILE\f[R] to control which file. +\f[C]AWS_SHARED_CREDENTIALS_FILE\f[R] to control which file or the +\f[C]shared_credentials_file\f[R] config key. .IP \[bu] 2 -\f[C]AWS_PROFILE\f[R] to control which profile to use. +\f[C]AWS_PROFILE\f[R] to control which profile to use or the +\f[C]profile\f[R] config key. .RE .RE .IP \[bu] 2 @@ -32023,11 +33262,20 @@ Or, run \f[C]rclone\f[R] on an EC2 instance with an IAM role (AWS only). .IP \[bu] 2 Or, run \f[C]rclone\f[R] in an EKS pod with an IAM role that is associated with a service account (AWS only). +.IP \[bu] 2 +Or, use process +credentials (https://docs.aws.amazon.com/sdkref/latest/guide/feature-process-credentials.html) +to read config from an external program. .RE .PP +With \f[C]env_auth = true\f[R] rclone (which uses the SDK for Go v2) +should support all authentication +methods (https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html) +that the \f[C]aws\f[R] CLI tool does and the other AWS SDKs. +.PP If none of these option actually end up providing \f[C]rclone\f[R] with -AWS credentials then S3 interaction will be non-authenticated (see -below). +AWS credentials then S3 interaction will be non-authenticated (see the +anonymous access section for more info). .SS S3 Permissions .PP When using the \f[C]sync\f[R] subcommand of \f[C]rclone\f[R] the @@ -33533,6 +34781,9 @@ Some providers (e.g. AWS, Aliyun OSS, Netease COS, or Tencent COS) require this set to false - rclone will do this automatically based on the provider setting. .PP +Note that if your bucket isn\[aq]t a valid DNS name, i.e. +has \[aq].\[aq] or \[aq]_\[aq] in, you\[aq]ll need to set this to true. +.PP Properties: .IP \[bu] 2 Config: force_path_style @@ -33871,6 +35122,26 @@ Env Var: RCLONE_S3_USE_MULTIPART_ETAG Type: Tristate .IP \[bu] 2 Default: unset +.SS --s3-use-unsigned-payload +.PP +Whether to use an unsigned payload in PutObject +.PP +Rclone has to avoid the AWS SDK seeking the body when calling PutObject. +The AWS provider can add checksums in the trailer to avoid seeking but +other providers can\[aq]t. +.PP +This should be true, false or left unset to use the default for the +provider. +.PP +Properties: +.IP \[bu] 2 +Config: use_unsigned_payload +.IP \[bu] 2 +Env Var: RCLONE_S3_USE_UNSIGNED_PAYLOAD +.IP \[bu] 2 +Type: Tristate +.IP \[bu] 2 +Default: unset .SS --s3-use-presigned-request .PP Whether to use a presigned request or PutObject for single part uploads @@ -34051,7 +35322,7 @@ Type: bool Default: false .SS --s3-sts-endpoint .PP -Endpoint for STS. +Endpoint for STS (deprecated). .PP Leave blank if using AWS to use the default endpoint for the region. .PP @@ -34121,6 +35392,42 @@ Env Var: RCLONE_S3_USE_MULTIPART_UPLOADS Type: Tristate .IP \[bu] 2 Default: unset +.SS --s3-sdk-log-mode +.PP +Set to debug the SDK +.PP +This can be set to a comma separated list of the following functions: +.IP \[bu] 2 +\f[C]Signing\f[R] +.IP \[bu] 2 +\f[C]Retries\f[R] +.IP \[bu] 2 +\f[C]Request\f[R] +.IP \[bu] 2 +\f[C]RequestWithBody\f[R] +.IP \[bu] 2 +\f[C]Response\f[R] +.IP \[bu] 2 +\f[C]ResponseWithBody\f[R] +.IP \[bu] 2 +\f[C]DeprecatedUsage\f[R] +.IP \[bu] 2 +\f[C]RequestEventMessage\f[R] +.IP \[bu] 2 +\f[C]ResponseEventMessage\f[R] +.PP +Use \f[C]Off\f[R] to disable and \f[C]All\f[R] to set all log levels. +You will need to use \f[C]-vv\f[R] to see the debug level logs. +.PP +Properties: +.IP \[bu] 2 +Config: sdk_log_mode +.IP \[bu] 2 +Env Var: RCLONE_S3_SDK_LOG_MODE +.IP \[bu] 2 +Type: Bits +.IP \[bu] 2 +Default: Off .SS --s3-description .PP Description of the remote. @@ -34269,7 +35576,7 @@ These can be run on a running backend using the rc command backend/command (https://rclone.org/rc/#backend-command). .SS restore .PP -Restore objects from GLACIER to normal storage +Restore objects from GLACIER or INTELLIGENT-TIERING archive tier .IP .nf \f[C] @@ -34278,7 +35585,8 @@ rclone backend restore remote: [options] [+] .fi .PP This command can be used to restore one or more objects from GLACIER to -normal storage. +normal storage or from INTELLIGENT-TIERING Archive Access / Deep Archive +Access tier to the Frequent Access tier. .PP Usage Examples: .IP @@ -34287,6 +35595,7 @@ Usage Examples: rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS +rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY \f[R] .fi .PP @@ -34329,13 +35638,14 @@ Options: .IP \[bu] 2 \[dq]description\[dq]: The optional description for the job. .IP \[bu] 2 -\[dq]lifetime\[dq]: Lifetime of the active copy in days +\[dq]lifetime\[dq]: Lifetime of the active copy in days, ignored for +INTELLIGENT-TIERING storage .IP \[bu] 2 \[dq]priority\[dq]: Priority of restore: Standard|Expedited|Bulk .SS restore-status .PP -Show the restore status for objects being restored from GLACIER to -normal storage +Show the restore status for objects being restored from GLACIER or +INTELLIGENT-TIERING storage .IP .nf \f[C] @@ -34344,7 +35654,8 @@ rclone backend restore-status remote: [options] [+] .fi .PP This command can be used to show the status for objects being restored -from GLACIER to normal storage. +from GLACIER to normal storage or from INTELLIGENT-TIERING Archive +Access / Deep Archive Access tier to the Frequent Access tier. .PP Usage Examples: .IP @@ -34380,6 +35691,15 @@ It returns a list of status dictionaries. \[dq]RestoreExpiryDate\[dq]: \[dq]2023-09-06T12:29:19+01:00\[dq] }, \[dq]StorageClass\[dq]: \[dq]DEEP_ARCHIVE\[dq] + }, + { + \[dq]Remote\[dq]: \[dq]test.gz\[dq], + \[dq]VersionID\[dq]: null, + \[dq]RestoreStatus\[dq]: { + \[dq]IsRestoreInProgress\[dq]: true, + \[dq]RestoreExpiryDate\[dq]: \[dq]null\[dq] + }, + \[dq]StorageClass\[dq]: \[dq]INTELLIGENT_TIERING\[dq] } ] \f[R] @@ -34553,15 +35873,6 @@ Your config should end up looking like this: [anons3] type = s3 provider = AWS -env_auth = false -access_key_id = -secret_access_key = -region = us-east-1 -endpoint = -location_constraint = -acl = private -server_side_encryption = -storage_class = \f[R] .fi .PP @@ -34574,6 +35885,14 @@ rclone lsd anons3:1000genomes .fi .PP You will be able to list and copy data but not upload it. +.PP +You can also do this entirely on the command line +.IP +.nf +\f[C] +rclone lsd :s3,provider=AWS:1000genomes +\f[R] +.fi .SS Providers .SS AWS S3 .PP @@ -34782,6 +36101,12 @@ Now run \f[C]rclone lsf r2:\f[R] to see your buckets and For R2 tokens with the \[dq]Object Read & Write\[dq] permission, you may also need to add \f[C]no_check_bucket = true\f[R] for object uploads to work correctly. +.PP +Note that Cloudflare decompresses files uploaded with +\f[C]Content-Encoding: gzip\f[R] by default which is a deviation from +what AWS does. +If this is causing a problem then upload the files with +\f[C]--header-upload \[dq]Cache-Control: no-transform\[dq]\f[R] .SS Dreamhost .PP Dreamhost DreamObjects (https://www.dreamhost.com/cloud/storage/) is an @@ -37762,6 +39087,29 @@ nodes across the network. .PP For more detailed comparison please check the documentation of the storj backend. +.SS Memory usage {memory} +.PP +The most common cause of rclone using lots of memory is a single +directory with millions of files in. +Despite s3 not really having the concepts of directories, rclone does +the sync on a directory by directory basis to be compatible with normal +filing systems. +.PP +Rclone loads each directory into memory as rclone objects. +Each rclone object takes 0.5k-1k of memory, so approximately 1GB per +1,000,000 files, and the sync for that directory does not begin until it +is entirely loaded in memory. +So the sync can take a long time to start for large directories. +.PP +To sync a directory with 100,000,000 files in you would need +approximately 100 GB of memory. +At some point the amount of memory becomes difficult to provide so there +is a workaround for +this (https://github.com/rclone/rclone/wiki/Big-syncs-with-millions-of-files) +which involves a bit of scripting. +.PP +At some point rclone will gain a sync mode which is effectively this +workaround but built in to rclone. .SS Limitations .PP \f[C]rclone about\f[R] is not supported by the S3 backend. @@ -37959,12 +39307,13 @@ key> 0123456789abcdef0123456789abcdef0123456789 Endpoint for the service - leave blank normally. endpoint> Remote config --------------------- -[remote] -account = 123456789abc -key = 0123456789abcdef0123456789abcdef0123456789 -endpoint = --------------------- +Configuration complete. +Options: +- type: b2 +- account: 123456789abc +- key: 0123456789abcdef0123456789abcdef0123456789 +- endpoint: +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -38117,13 +39466,21 @@ There can be at most \f[C]--transfers\f[R] of these in use at any moment, so this sets the upper limit on the memory used. .SS Versions .PP -When rclone uploads a new version of a file it creates a new version of +The default setting of B2 is to keep old versions of files. +This means when rclone uploads a new version of a file it creates a new +version of it (https://www.backblaze.com/docs/cloud-storage-file-versions). Likewise when you delete a file, the old version will be marked hidden and still be available. -Conversely, you may opt in to a \[dq]hard delete\[dq] of files with the -\f[C]--b2-hard-delete\f[R] flag which would permanently remove the file -instead of hiding it. +.PP +Whether B2 keeps old versions of files or not can be adjusted on a per +bucket basis using the \[dq]Lifecycle settings\[dq] on the B2 control +panel or when creating the bucket using the --b2-lifecycle flag or after +creation using the rclone backend lifecycle command. +.PP +You may opt in to a \[dq]hard delete\[dq] of files with the +\f[C]--b2-hard-delete\f[R] flag which permanently removes files on +deletion instead of hiding them. .PP Old versions of files, where available, are visible using the \f[C]--b2-versions\f[R] flag. @@ -38913,12 +40270,13 @@ If your browser doesn\[aq]t open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -client_id = -client_secret = -token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]XXX\[dq]} --------------------- +Configuration complete. +Options: +- type: box +- client_id: +- client_secret: +- token: {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]XXX\[dq]} +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -39022,11 +40380,11 @@ e/n/d/r/c/s/q> e Choose a number from below, or type in an existing value 1 > remote remote> remote --------------------- -[remote] -type = box -token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2017-07-08T23:40:08.059167677+01:00\[dq]} --------------------- +Configuration complete. +Options: +- type: box +- token: {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2017-07-08T23:40:08.059167677+01:00\[dq]} +Keep this \[dq]remote\[dq] remote? Edit remote Value \[dq]client_id\[dq] = \[dq]\[dq] Edit? (y/n)> @@ -39054,11 +40412,11 @@ If your browser doesn\[aq]t open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = box -token = {\[dq]access_token\[dq]:\[dq]YYY\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]YYY\[dq],\[dq]expiry\[dq]:\[dq]2017-07-23T12:22:29.259137901+01:00\[dq]} --------------------- +Configuration complete. +Options: +- type: box +- token: {\[dq]access_token\[dq]:\[dq]YYY\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]YYY\[dq],\[dq]expiry\[dq]:\[dq]2017-07-23T12:22:29.259137901+01:00\[dq]} +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -41014,12 +42372,12 @@ If your browser doesn\[aq]t open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = sharefile -endpoint = https://XXX.sharefile.com -token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2019-09-30T19:41:45.878561877+01:00\[dq]} --------------------- +Configuration complete. +Options: +- type: sharefile +- endpoint: https://XXX.sharefile.com +- token: {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2019-09-30T19:41:45.878561877+01:00\[dq]} +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -42692,11 +44050,11 @@ Embedded spaces can be added using quotes \[dq]dir=remote:path with space\[dq] \[dq]dir2=remote2:path with space\[dq] Enter a fs.SpaceSepList value. upstreams> images=s3:imagesbucket files=drive:important/files --------------------- -[remote] -type = combine -upstreams = images=s3:imagesbucket files=drive:important/files --------------------- +Configuration complete. +Options: +- type: combine +- upstreams: images=s3:imagesbucket files=drive:important/files +Keep this \[dq]remote\[dq] remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -42844,12 +44202,13 @@ Remote config Please visit: https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX --------------------- -[remote] -app_key = -app_secret = -token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX --------------------- +Configuration complete. +Options: +- type: dropbox +- app_key: +- app_secret: +- token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -43308,7 +44667,7 @@ Max number of files in upload batch. This sets the batch size of files to upload. It has to be less than 1000. .PP -By default this is 0 which means rclone which calculate the batch size +By default this is 0 which means rclone will calculate the batch size depending on the setting of batch_mode. .IP \[bu] 2 batch_mode: async - default batch_size is 100 @@ -43540,12 +44899,12 @@ y) Yes n) No (default) y/n> n Remote config --------------------- -[remote] -type = filefabric -url = https://yourfabric.smestorage.com/ -permanent_token = xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx --------------------- +Configuration complete. +Options: +- type: filefabric +- url: https://yourfabric.smestorage.com/ +- permanent_token: xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx +Keep this \[dq]remote\[dq] remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -43792,6 +45151,219 @@ Env Var: RCLONE_FILEFABRIC_DESCRIPTION Type: string .IP \[bu] 2 Required: false +.SH Files.com +.PP +Files.com (https://www.files.com/) is a cloud storage service that +provides a secure and easy way to store and share files. +.PP +The initial setup for filescom involves authenticating with your +Files.com account. +You can do this by providing your site subdomain, username, and +password. +Alternatively, you can authenticate using an API Key from +Files.com (https://www.files.com/docs/sdk-and-apis/api-keys/). +\f[C]rclone config\f[R] walks you through it. +.SS Configuration +.PP +Here is an example of how to make a remote called \f[C]remote\f[R]. +First run: +.IP +.nf +\f[C] +rclone config +\f[R] +.fi +.PP +This will guide you through an interactive setup process: +.IP +.nf +\f[C] +No remotes found, make a new one? +n) New remote +s) Set configuration password +q) Quit config +n/s/q> n + +Enter name for new remote. +name> remote + +Option Storage. +Type of storage to configure. +Choose a number from below, or type in your own value. +[snip] +XX / Files.com + \[rs] \[dq]filescom\[dq] +[snip] +Storage> filescom + +Option site. +Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com) +Enter a value. Press Enter to leave empty. +site> mysite + +Option username. +The username used to authenticate with Files.com. +Enter a value. Press Enter to leave empty. +username> user + +Option password. +The password used to authenticate with Files.com. +Choose an alternative below. Press Enter for the default (n). +y) Yes, type in my own password +g) Generate random password +n) No, leave this optional password blank (default) +y/g/n> y +Enter the password: +password: +Confirm the password: +password: + +Edit advanced config? +y) Yes +n) No (default) +y/n> n + +Configuration complete. +Options: +- type: filescom +- site: mysite +- username: user +- password: *** ENCRYPTED *** +Keep this \[dq]remote\[dq] remote? +y) Yes this is OK (default) +e) Edit this remote +d) Delete this remote +y/e/d> y +\f[R] +.fi +.PP +Once configured you can use rclone. +.PP +See all files in the top level: +.IP +.nf +\f[C] +rclone lsf remote: +\f[R] +.fi +.PP +Make a new directory in the root: +.IP +.nf +\f[C] +rclone mkdir remote:dir +\f[R] +.fi +.PP +Recursively List the contents: +.IP +.nf +\f[C] +rclone ls remote: +\f[R] +.fi +.PP +Sync \f[C]/home/local/directory\f[R] to the remote directory, deleting +any excess files in the directory. +.IP +.nf +\f[C] +rclone sync --interactive /home/local/directory remote:dir +\f[R] +.fi +.SS Standard options +.PP +Here are the Standard options specific to filescom (Files.com). +.SS --filescom-site +.PP +Your site subdomain (e.g. +mysite) or custom domain (e.g. +myfiles.customdomain.com). +.PP +Properties: +.IP \[bu] 2 +Config: site +.IP \[bu] 2 +Env Var: RCLONE_FILESCOM_SITE +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false +.SS --filescom-username +.PP +The username used to authenticate with Files.com. +.PP +Properties: +.IP \[bu] 2 +Config: username +.IP \[bu] 2 +Env Var: RCLONE_FILESCOM_USERNAME +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false +.SS --filescom-password +.PP +The password used to authenticate with Files.com. +.PP +\f[B]NB\f[R] Input to this must be obscured - see rclone +obscure (https://rclone.org/commands/rclone_obscure/). +.PP +Properties: +.IP \[bu] 2 +Config: password +.IP \[bu] 2 +Env Var: RCLONE_FILESCOM_PASSWORD +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false +.SS Advanced options +.PP +Here are the Advanced options specific to filescom (Files.com). +.SS --filescom-api-key +.PP +The API key used to authenticate with Files.com. +.PP +Properties: +.IP \[bu] 2 +Config: api_key +.IP \[bu] 2 +Env Var: RCLONE_FILESCOM_API_KEY +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false +.SS --filescom-encoding +.PP +The encoding for the backend. +.PP +See the encoding section in the +overview (https://rclone.org/overview/#encoding) for more info. +.PP +Properties: +.IP \[bu] 2 +Config: encoding +.IP \[bu] 2 +Env Var: RCLONE_FILESCOM_ENCODING +.IP \[bu] 2 +Type: Encoding +.IP \[bu] 2 +Default: +Slash,BackSlash,Del,Ctl,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot +.SS --filescom-description +.PP +Description of the remote. +.PP +Properties: +.IP \[bu] 2 +Config: description +.IP \[bu] 2 +Env Var: RCLONE_FILESCOM_DESCRIPTION +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false .SH FTP .PP FTP is the File Transfer Protocol. @@ -43867,12 +45439,12 @@ Use FTP over TLS (Explicit) Enter a boolean value (true or false). Press Enter for the default (\[dq]false\[dq]). explicit_tls> Remote config --------------------- -[remote] -type = ftp -host = ftp.example.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: ftp +- host: ftp.example.com +- pass: *** ENCRYPTED *** +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -44447,6 +46019,373 @@ A value of \f[C]1000000000\f[R] means that file time precision of 1 second is available. A value of \f[C]3153600000000000000\f[R] (or another large number) means \[dq]unsupported\[dq]. +.SH Gofile +.PP +Gofile (https://gofile.io) is a content storage and distribution +platform. +Its aim is to provide as much service as possible for free or at a very +low price. +.PP +The initial setup for Gofile involves logging in to the web interface +and going to the \[dq]My Profile\[dq] section. +Copy the \[dq]Account API token\[dq] for use in the config file. +.PP +Note that if you wish to connect rclone to Gofile you will need a +premium account. +.SS Configuration +.PP +Here is an example of how to make a remote called \f[C]remote\f[R]. +First run: +.IP +.nf +\f[C] + rclone config +\f[R] +.fi +.PP +This will guide you through an interactive setup process: +.IP +.nf +\f[C] +No remotes found, make a new one? +n) New remote +s) Set configuration password +q) Quit config +n/s/q> n + +Enter name for new remote. +name> remote + +Option Storage. +Type of storage to configure. +Choose a number from below, or type in your own value. +XX / Gofile + \[rs] (gofile) +Storage> gofile + +Option access_token. +API Access token +You can get this from the web control panel. +Enter a value. Press Enter to leave empty. +access_token> YOURACCESSTOKEN + +Edit advanced config? +y) Yes +n) No (default) +y/n> n + +Configuration complete. +Options: +- type: gofile +- access_token: YOURACCESSTOKEN +Keep this \[dq]remote\[dq] remote? +y) Yes this is OK (default) +e) Edit this remote +d) Delete this remote +y/e/d> y +\f[R] +.fi +.PP +Once configured you can then use \f[C]rclone\f[R] like this, +.PP +List directories and files in the top level of your Gofile +.IP +.nf +\f[C] +rclone lsf remote: +\f[R] +.fi +.PP +To copy a local directory to an Gofile directory called backup +.IP +.nf +\f[C] +rclone copy /home/source remote:backup +\f[R] +.fi +.SS Modification times and hashes +.PP +Gofile supports modification times with a resolution of 1 second. +.PP +Gofile supports MD5 hashes, so you can use the \f[C]--checksum\f[R] +flag. +.SS Restricted filename characters +.PP +In addition to the default restricted characters +set (https://rclone.org/overview/#restricted-characters) the following +characters are also replaced: +.PP +.TS +tab(@); +l c c. +T{ +Character +T}@T{ +Value +T}@T{ +Replacement +T} +_ +T{ +! +T}@T{ +0x21 +T}@T{ +\[uFF01] +T} +T{ +\[dq] +T}@T{ +0x22 +T}@T{ +\[uFF02] +T} +T{ +* +T}@T{ +0x2A +T}@T{ +\[uFF0A] +T} +T{ +: +T}@T{ +0x3A +T}@T{ +\[uFF1A] +T} +T{ +< +T}@T{ +0x3C +T}@T{ +\[uFF1C] +T} +T{ +> +T}@T{ +0x3E +T}@T{ +\[uFF1E] +T} +T{ +? +T}@T{ +0x3F +T}@T{ +\[uFF1F] +T} +T{ +\[rs] +T}@T{ +0x5C +T}@T{ +\[uFF3C] +T} +T{ +| +T}@T{ +0x7C +T}@T{ +\[uFF5C] +T} +.TE +.PP +File names can also not start or end with the following characters. +These only get replaced if they are the first or last character in the +name: +.PP +.TS +tab(@); +l c c. +T{ +Character +T}@T{ +Value +T}@T{ +Replacement +T} +_ +T{ +\&. +T}@T{ +0x2E +T}@T{ +\[uFF0E] +T} +.TE +.PP +Invalid UTF-8 bytes will also be +replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t +be used in JSON strings. +.SS Public Links +.PP +Gofile supports \f[C]rclone link\f[R] to make public links to files or +directories. +If you specify a directory it will download as a \f[C]zip\f[R] file. +You can use the \f[C]--expire\f[R] flag to specify the time the link +should be valid. +Note that \f[C]rclone link --unlink\f[R] removes all the public links +for a file. +.SS Root folder ID +.PP +You can set the \f[C]root_folder_id\f[R] for rclone. +This is the directory (identified by its \f[C]Folder ID\f[R]) that +rclone considers to be the root of your Gofile drive. +.PP +Normally you will leave this blank and rclone will determine the correct +root to use itself and fill in the value in the config file. +.PP +However you can set this to restrict rclone to a specific folder +hierarchy. +.PP +In order to do this you will have to find the \f[C]Folder ID\f[R] of the +directory you wish rclone to display. +.PP +You can do this with rclone +.IP +.nf +\f[C] +$ rclone lsf -Fip --dirs-only remote: +d6341f53-ee65-4f29-9f59-d11e8070b2a0;Files/ +f4f5c9b8-6ece-478b-b03e-4538edfe5a1c;Photos/ +d50e356c-29ca-4b27-a3a7-494d91026e04;Videos/ +\f[R] +.fi +.PP +The ID to use is the part before the \f[C];\f[R] so you could set +.IP +.nf +\f[C] +root_folder_id = d6341f53-ee65-4f29-9f59-d11e8070b2a0 +\f[R] +.fi +.PP +To restrict rclone to the \f[C]Files\f[R] directory. +.SS Standard options +.PP +Here are the Standard options specific to gofile (Gofile). +.SS --gofile-access-token +.PP +API Access token +.PP +You can get this from the web control panel. +.PP +Properties: +.IP \[bu] 2 +Config: access_token +.IP \[bu] 2 +Env Var: RCLONE_GOFILE_ACCESS_TOKEN +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false +.SS Advanced options +.PP +Here are the Advanced options specific to gofile (Gofile). +.SS --gofile-root-folder-id +.PP +ID of the root folder +.PP +Leave this blank normally, rclone will fill it in automatically. +.PP +If you want rclone to be restricted to a particular folder you can fill +it in - see the docs for more info. +.PP +Properties: +.IP \[bu] 2 +Config: root_folder_id +.IP \[bu] 2 +Env Var: RCLONE_GOFILE_ROOT_FOLDER_ID +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false +.SS --gofile-account-id +.PP +Account ID +.PP +Leave this blank normally, rclone will fill it in automatically. +.PP +Properties: +.IP \[bu] 2 +Config: account_id +.IP \[bu] 2 +Env Var: RCLONE_GOFILE_ACCOUNT_ID +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false +.SS --gofile-list-chunk +.PP +Number of items to list in each call +.PP +Properties: +.IP \[bu] 2 +Config: list_chunk +.IP \[bu] 2 +Env Var: RCLONE_GOFILE_LIST_CHUNK +.IP \[bu] 2 +Type: int +.IP \[bu] 2 +Default: 1000 +.SS --gofile-encoding +.PP +The encoding for the backend. +.PP +See the encoding section in the +overview (https://rclone.org/overview/#encoding) for more info. +.PP +Properties: +.IP \[bu] 2 +Config: encoding +.IP \[bu] 2 +Env Var: RCLONE_GOFILE_ENCODING +.IP \[bu] 2 +Type: Encoding +.IP \[bu] 2 +Default: +Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftPeriod,RightPeriod,InvalidUtf8,Dot,Exclamation +.SS --gofile-description +.PP +Description of the remote. +.PP +Properties: +.IP \[bu] 2 +Config: description +.IP \[bu] 2 +Env Var: RCLONE_GOFILE_DESCRIPTION +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false +.SS Limitations +.PP +Gofile only supports filenames up to 255 characters in length, where a +character is a unicode character. +.PP +Directories should not be cached for more than 24h otherwise files in +the directory may not be downloadable. +In practice this means when using a VFS based rclone command such as +\f[C]rclone mount\f[R] you should make sure \f[C]--dir-cache-time\f[R] +is less than \f[C]24h\f[R]. +.PP +Note that Gofile is currently limited to a total of 100,000 items. +If you attempt to upload more than that you will get +\f[C]error-limit-100000\f[R]. +This limit may be lifted in the future. +.SS Duplicated files +.PP +Gofile is capable of having files with duplicated file names. +For instance two files called \f[C]hello.txt\f[R] in the same directory. +.PP +Rclone cannot sync that to a normal file system but it can be fixed with +the \f[C]rclone dedupe\f[R] command. +.PP +Duplicated files cause problems with the syncing and you will see +messages in the log about duplicates. +.PP +Use \f[C]rclone dedupe\f[R] to fix duplicated files. .SH Google Cloud Storage .PP Paths are specified as \f[C]remote:bucket\f[R] (or \f[C]remote:\f[R] for @@ -44577,16 +46516,16 @@ If your browser doesn\[aq]t open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = google cloud storage -client_id = -client_secret = -token = {\[dq]AccessToken\[dq]:\[dq]xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]RefreshToken\[dq]:\[dq]x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx\[dq],\[dq]Expiry\[dq]:\[dq]2014-07-17T20:49:14.929208288+01:00\[dq],\[dq]Extra\[dq]:null} -project_number = 12345678 -object_acl = private -bucket_acl = private --------------------- +Configuration complete. +Options: +- type: google cloud storage +- client_id: +- client_secret: +- token: {\[dq]AccessToken\[dq]:\[dq]xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]RefreshToken\[dq]:\[dq]x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx\[dq],\[dq]Expiry\[dq]:\[dq]2014-07-17T20:49:14.929208288+01:00\[dq],\[dq]Extra\[dq]:null} +- project_number: 12345678 +- object_acl: private +- bucket_acl: private +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -45614,15 +47553,16 @@ Configure this as a Shared Drive (Team Drive)? y) Yes n) No y/n> n --------------------- -[remote] -client_id = -client_secret = -scope = drive -root_folder_id = -service_account_file = -token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2014-03-16T13:57:58.955387075Z\[dq]} --------------------- +Configuration complete. +Options: +type: drive +- client_id: +- client_secret: +- scope: drive +- root_folder_id: +- service_account_file: +- token: {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2014-03-16T13:57:58.955387075Z\[dq]} +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -45760,12 +47700,11 @@ If you\[aq]d rather stuff the contents of the credentials file into the rclone config file, you can set \f[C]service_account_credentials\f[R] with the actual contents of the file instead, or set the equivalent environment variable. -.SS Use case - Google Apps/G-suite account and individual Drive +.SS Use case - Google Workspace account and individual Drive .PP -Let\[aq]s say that you are the administrator of a Google Apps (old) or -G-suite account. -The goal is to store data on an individual\[aq]s Drive account, who IS a -member of the domain. +Let\[aq]s say that you are the administrator of a Google Workspace. +The goal is to read or write data on an individual\[aq]s Drive account, +who IS a member of the domain. We\[aq]ll call the domain \f[B]example.com\f[R], and the user \f[B]foo\[at]example.com\f[R]. .PP @@ -45775,7 +47714,8 @@ There\[aq]s a few steps we need to go through to accomplish this: To create a service account and obtain its credentials, go to the Google Developer Console (https://console.developers.google.com). .IP \[bu] 2 -You must have a project - create one if you don\[aq]t. +You must have a project - create one if you don\[aq]t and make sure you +are on the selected project. .IP \[bu] 2 Then go to \[dq]IAM & admin\[dq] -> \[dq]Service Accounts\[dq]. .IP \[bu] 2 @@ -45786,29 +47726,45 @@ with something that identifies your client. Select \[dq]Create And Continue\[dq]. Step 2 and 3 are optional. .IP \[bu] 2 -These credentials are what rclone will use for authentication. +Click on the newly created service account +.IP \[bu] 2 +Click \[dq]Keys\[dq] and then \[dq]Add Key\[dq] and then \[dq]Create new +key\[dq] +.IP \[bu] 2 +Choose type \[dq]JSON\[dq] and click create +.IP \[bu] 2 +This will download a small JSON file that rclone will use for +authentication. +.PP If you ever need to remove access, press the \[dq]Delete service account key\[dq] button. .SS 2. Allowing API access to example.com Google Drive .IP \[bu] 2 -Go to example.com\[aq]s admin console +Go to example.com\[aq]s Workspace Admin +Console (https://admin.google.com) .IP \[bu] 2 Go into \[dq]Security\[dq] (or use the search bar) .IP \[bu] 2 -Select \[dq]Show more\[dq] and then \[dq]Advanced settings\[dq] +Select \[dq]Access and data control\[dq] and then \[dq]API controls\[dq] .IP \[bu] 2 -Select \[dq]Manage API client access\[dq] in the -\[dq]Authentication\[dq] section +Click \[dq]Manage domain-wide delegation\[dq] .IP \[bu] 2 -In the \[dq]Client Name\[dq] field enter the service account\[aq]s +Click \[dq]Add new\[dq] +.IP \[bu] 2 +In the \[dq]Client ID\[dq] field enter the service account\[aq]s \[dq]Client ID\[dq] - this can be found in the Developer Console under \[dq]IAM & Admin\[dq] -> \[dq]Service Accounts\[dq], then \[dq]View Client ID\[dq] for the newly created service account. It is a \[ti]21 character numerical string. .IP \[bu] 2 -In the next field, \[dq]One or More API Scopes\[dq], enter -\f[C]https://www.googleapis.com/auth/drive\f[R] to grant access to -Google Drive specifically. +In the next field, \[dq]OAuth Scopes\[dq], enter +\f[C]https://www.googleapis.com/auth/drive\f[R] to grant read/write +access to Google Drive specifically. +You can also use +\f[C]https://www.googleapis.com/auth/drive.readonly\f[R] for read only +access. +.IP \[bu] 2 +Click \[dq]Authorise\[dq] .SS 3. Configure rclone, assuming a new install .IP .nf @@ -45817,12 +47773,12 @@ rclone config n/s/q> n # New name>gdrive # Gdrive is an example name -Storage> # Select the number shown for Google Drive +Storage> # Type drive client_id> # Can be left blank client_secret> # Can be left blank -scope> # Select your scope, 1 for example +scope> # Select the scope use used in step 2 root_folder_id> # Can be left blank -service_account_file> /home/foo/myJSONfile.json # This is where the JSON file goes! +service_account_file> /home/foo/myJSONfile.json # Path to the JSON file you downloaded in step 1. y/n> # Auto config, n \f[R] .fi @@ -45848,7 +47804,7 @@ Note: in case you configured a specific root folder on gdrive and rclone is unable to access the contents of that folder when using \f[C]--drive-impersonate\f[R], do this instead: - in the gdrive web interface, share your root folder with the user/email of the new Service -Account you created/selected at step #1 - use rclone without specifying +Account you created/selected at step 1 - use rclone without specifying the \f[C]--drive-impersonate\f[R] option, like this: \f[C]rclone -v lsf gdrive:backup\f[R] .SS Shared drives (team drives) @@ -45878,13 +47834,14 @@ Choose a number from below, or type in your own value 3 / Rclone Test 3 \[rs] \[dq]zzzzzzzzzzzzzzzzzzzz\[dq] Enter a Shared Drive ID> 1 --------------------- -[remote] -client_id = -client_secret = -token = {\[dq]AccessToken\[dq]:\[dq]xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]RefreshToken\[dq]:\[dq]1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]Expiry\[dq]:\[dq]2014-03-16T13:57:58.955387075Z\[dq],\[dq]Extra\[dq]:null} -team_drive = xxxxxxxxxxxxxxxxxxxx --------------------- +Configuration complete. +Options: +- type: drive +- client_id: +- client_secret: +- token: {\[dq]AccessToken\[dq]:\[dq]xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]RefreshToken\[dq]:\[dq]1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]Expiry\[dq]:\[dq]2014-03-16T13:57:58.955387075Z\[dq],\[dq]Extra\[dq]:null} +- team_drive: xxxxxxxxxxxxxxxxxxxx +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -48284,11 +50241,11 @@ Got code *** are stored in full resolution at original quality. These uploads *** will count towards storage in your Google Account. --------------------- -[remote] -type = google photos -token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2019-06-28T17:38:04.644930156+01:00\[dq]} --------------------- +Configuration complete. +Options: +- type: google photos +- token: {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2019-06-28T17:38:04.644930156+01:00\[dq]} +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -48678,7 +50635,7 @@ Max number of files in upload batch. This sets the batch size of files to upload. It has to be less than 50. .PP -By default this is 0 which means rclone which calculate the batch size +By default this is 0 which means rclone will calculate the batch size depending on the setting of batch_mode. .IP \[bu] 2 batch_mode: async - default batch_size is 50 @@ -49239,6 +51196,9 @@ download object, calculate all \f[I]supported\f[R] hashes on the fly and store in cache; return requested hash. .SS Other operations .IP \[bu] 2 +any time a hash is requested, follow the logic from 1-4 from +\f[C]hashsum\f[R] above +.IP \[bu] 2 whenever a file is uploaded or downloaded \f[B]in full\f[R], capture the stream to calculate all supported hashes on the fly and update database .IP \[bu] 2 @@ -49321,12 +51281,12 @@ y) Yes n) No (default) y/n> n Remote config --------------------- -[remote] -type = hdfs -namenode = namenode.hadoop:8020 -username = root --------------------- +Configuration complete. +Options: +- type: hdfs +- namenode: namenode.hadoop:8020 +- username: root +Keep this \[dq]remote\[dq] remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -49639,11 +51599,11 @@ If your browser doesn\[aq]t open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = hidrive -token = {\[dq]access_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxx\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]expiry\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxx\[dq]} --------------------- +Configuration complete. +Options: +- type: hidrive +- token: {\[dq]access_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxx\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]expiry\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxx\[dq]} +Keep this \[dq]remote\[dq] remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -50231,10 +52191,11 @@ Choose a number from below, or type in your own value \[rs] \[dq]https://example.com\[dq] url> https://beta.rclone.org Remote config --------------------- -[remote] -url = https://beta.rclone.org --------------------- +Configuration complete. +Options: +- type: http +- url: https://beta.rclone.org +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -51057,12 +53018,12 @@ Edit advanced config? y) Yes n) No (default) y/n> n --------------------- -[remote] -type = internetarchive -access_key_id = XXXX -secret_access_key = XXXX --------------------- +Configuration complete. +Options: +- type: internetarchive +- access_key_id: XXXX +- secret_access_key: XXXX +Keep this \[dq]remote\[dq] remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -51580,18 +53541,18 @@ Press Enter for the default (Archive). 2 > Shared 3 > Sync config_mountpoint> 1 --------------------- -[remote] -type = jottacloud -configVersion = 1 -client_id = jottacli -client_secret = -tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token -token = {........} -username = 2940e57271a93d987d6f8a21 -device = Jotta -mountpoint = Archive --------------------- +Configuration complete. +Options: +- type: jottacloud +- configVersion: 1 +- client_id: jottacli +- client_secret: +- tokenURL: https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token +- token: {........} +- username: 2940e57271a93d987d6f8a21 +- device: Jotta +- mountpoint: Archive +Keep this \[dq]remote\[dq] remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -52718,13 +54679,13 @@ y) Yes n) No y/n> n Remote config --------------------- -[remote] -type = mailru -user = username\[at]mail.ru -pass = *** ENCRYPTED *** -speedup_enable = true --------------------- +Configuration complete. +Options: +- type: mailru +- user: username\[at]mail.ru +- pass: *** ENCRYPTED *** +- speedup_enable: true +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -53304,12 +55265,12 @@ password: Confirm the password: password: Remote config --------------------- -[remote] -type = mega -user = you\[at]example.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: mega +- user: you\[at]example.com +- pass: *** ENCRYPTED *** +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -53653,10 +55614,10 @@ Storage> memory Remote config --------------------- -[remote] -type = memory --------------------- +Configuration complete. +Options: +- type: memory +Keep this \[dq]remote\[dq] remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -54166,12 +56127,13 @@ key> base64encodedkey== Endpoint for the service - leave blank normally. endpoint> Remote config --------------------- -[remote] -account = account_name -key = base64encodedkey== -endpoint = --------------------- +Configuration complete. +Options: +- type: azureblob +- account: account_name +- key: base64encodedkey== +- endpoint: +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -54564,6 +56526,17 @@ be explicitly specified using exactly one of the If none of \f[C]msi_object_id\f[R], \f[C]msi_client_id\f[R], or \f[C]msi_mi_res_id\f[R] is set, this is is equivalent to using \f[C]env_auth\f[R]. +.SS Anonymous +.PP +If you want to access resources with public anonymous access then set +\f[C]account\f[R] only. +You can do this without making an rclone config: +.IP +.nf +\f[C] +rclone lsf :azureblob,account=ACCOUNT:CONTAINER +\f[R] +.fi .SS Standard options .PP Here are the Standard options specific to azureblob (Microsoft Azure @@ -56326,13 +58299,13 @@ Is that okay? y) Yes n) No y/n> y --------------------- -[remote] -type = onedrive -token = {\[dq]access_token\[dq]:\[dq]youraccesstoken\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]yourrefreshtoken\[dq],\[dq]expiry\[dq]:\[dq]2018-08-26T22:39:52.486512262+08:00\[dq]} -drive_id = b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk -drive_type = business --------------------- +Configuration complete. +Options: +- type: onedrive +- token: {\[dq]access_token\[dq]:\[dq]youraccesstoken\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]yourrefreshtoken\[dq],\[dq]expiry\[dq]:\[dq]2018-08-26T22:39:52.486512262+08:00\[dq]} +- drive_id: b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk +- drive_type: business +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -58066,11 +60039,12 @@ Enter the password: password: Confirm the password: password: --------------------- -[remote] -username = -password = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: opendrive +- username: +- password: *** ENCRYPTED *** +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -59535,15 +61509,16 @@ Number of connection retry. Leave blank will use the default value \[dq]3\[dq]. connection_retries> Remote config --------------------- -[remote] -env_auth = false -access_key_id = access_key -secret_access_key = secret_key -endpoint = -zone = pek3a -connection_retries = --------------------- +Configuration complete. +Options: +- type: qingstor +- env_auth: false +- access_key_id: access_key +- secret_access_key: secret_key +- endpoint: +- zone: pek3a +- connection_retries: +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -59940,11 +61915,12 @@ api_key> your_api_key Host name of Quatrix account. host> example.quatrix.it --------------------- -[remote] -api_key = your_api_key -host = example.quatrix.it --------------------- +Configuration complete. +Options: +- type: quatrix +- api_key: your_api_key +- host: example.quatrix.it +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -60006,12 +61982,12 @@ e/n/d/r/c/s/q> e Choose a number from below, or type in an existing value 1 > remote remote> remote --------------------- -[remote] -type = quatrix -host = some_host.quatrix.it -api_key = your_api_key --------------------- +Configuration complete. +Options: +- type: quatrix +- host: some_host.quatrix.it +- api_key: your_api_key +Keep this \[dq]remote\[dq] remote? Edit remote Option api_key. API key for accessing Quatrix account @@ -60021,12 +61997,12 @@ Option host. Host name of Quatrix account Enter a string value. Press Enter for the default (some_host.quatrix.it). --------------------- -[remote] -type = quatrix -host = some_host.quatrix.it -api_key = your_api_key --------------------- +Configuration complete. +Options: +- type: quatrix +- host: some_host.quatrix.it +- api_key: your_api_key +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -61146,6 +63122,53 @@ Env Var: RCLONE_SWIFT_LEAVE_PARTS_ON_ERROR Type: bool .IP \[bu] 2 Default: false +.SS --swift-fetch-until-empty-page +.PP +When paginating, always fetch unless we received an empty page. +.PP +Consider using this option if rclone listings show fewer objects than +expected, or if repeated syncs copy unchanged objects. +.PP +It is safe to enable this, but rclone may make more API calls than +necessary. +.PP +This is one of a pair of workarounds to handle implementations of the +Swift API that do not implement pagination as expected. +See also \[dq]partial_page_fetch_threshold\[dq]. +.PP +Properties: +.IP \[bu] 2 +Config: fetch_until_empty_page +.IP \[bu] 2 +Env Var: RCLONE_SWIFT_FETCH_UNTIL_EMPTY_PAGE +.IP \[bu] 2 +Type: bool +.IP \[bu] 2 +Default: false +.SS --swift-partial-page-fetch-threshold +.PP +When paginating, fetch if the current page is within this percentage of +the limit. +.PP +Consider using this option if rclone listings show fewer objects than +expected, or if repeated syncs copy unchanged objects. +.PP +It is safe to enable this, but rclone may make more API calls than +necessary. +.PP +This is one of a pair of workarounds to handle implementations of the +Swift API that do not implement pagination as expected. +See also \[dq]fetch_until_empty_page\[dq]. +.PP +Properties: +.IP \[bu] 2 +Config: partial_page_fetch_threshold +.IP \[bu] 2 +Env Var: RCLONE_SWIFT_PARTIAL_PAGE_FETCH_THRESHOLD +.IP \[bu] 2 +Type: int +.IP \[bu] 2 +Default: 0 .SS --swift-chunk-size .PP Above this size files will be chunked. @@ -61392,12 +63415,13 @@ If your browser doesn\[aq]t open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -client_id = -client_secret = -token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]expiry\[dq]:\[dq]0001-01-01T00:00:00Z\[dq]} --------------------- +Configuration complete. +Options: +- type: pcloud +- client_id: +- client_secret: +- token: {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]expiry\[dq]:\[dq]0001-01-01T00:00:00Z\[dq]} +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -62125,6 +64149,237 @@ user-uploaded files. Deleted files will still be visible with \f[C]--pikpak-trashed-only\f[R] even after the trash emptied. This goes away after few days. +.SH Pixeldrain +.PP +This is the backend for Pixeldrain\[aq]s premium filesystem feature. +This is not the same as pixeldrain\[aq]s free file sharing product. +The filesystem requires either a Pro subscription or the Prepaid plan. +More information on subscriptions (https://pixeldrain.com/#pro). +.PP +An overview of the filesystem\[aq]s features and limitations is +available in the filesystem guide (https://pixeldrain.com/filesystem) on +pixeldrain. +.SS Usage with account +.PP +To use the personal filesystem you will need a pixeldrain +account (https://pixeldrain.com/register) and either the Prepaid plan or +one of the Patreon-based subscriptions. +After registering and subscribing, your personal filesystem will be +available at this link: https://pixeldrain.com/d/me. +.PP +Go to the API keys page (https://pixeldrain.com/user/api_keys) on your +account and generate a new API key for rclone. +Then run \f[C]rclone config\f[R] and use the API key to create a new +backend. +.PP +Example: +.IP +.nf +\f[C] +No remotes found, make a new one? +n) New remote +d) Delete remote +c) Copy remote +s) Set configuration password +q) Quit config +n/d/c/s/q> n + +Enter name for new remote. +name> pixeldrainfs + +Option Storage. +Type of storage to configure. +Choose a number from below, or type in your own value. +\&... +XX / Pixeldrain Filesystem + \[rs] (pixeldrain) +\&... +Storage> pixeldrain + +Option api_key. +API key for your pixeldrain account. +Found on https://pixeldrain.com/user/api_keys. +Enter a value. Press Enter to leave empty. +api_key> b1bb1e81-9b7b-406b-986a-c9b20be76e15 + +Option directory_id. +Root of the filesystem to use. Set to \[aq]me\[aq] to use your personal filesystem. +Set to a shared directory ID to use a shared directory. +Enter a string value. Press Enter for the default (me). +directory_id> + +Edit advanced config? +y) Yes +n) No (default) +y/n> + +Configuration complete. +Options: +- type: pixeldrain +- api_key: b1bb1e81-9b7b-406b-986a-c9b20be76e15 +Keep this \[dq]pixeldrainfs\[dq] remote? +y) Yes this is OK (default) +e) Edit this remote +d) Delete this remote +y/e/d> + +Current remotes: + +Name Type +==== ==== +pixeldrainfs pixeldrain + +e) Edit existing remote +n) New remote +d) Delete remote +r) Rename remote +c) Copy remote +s) Set configuration password +q) Quit config +e/n/d/r/c/s/q> q +\f[R] +.fi +.SS Usage without account +.PP +It is possible to gain read-only access to publicly shared directories +through rclone. +For this you only need a directory ID. +The directory ID can be found in the URL of a shared directory, the URL +will look like this \f[C]https://pixeldrain.com/d/abcd1234\f[R] where +\f[C]abcd1234\f[R] is the directory ID. +Directory IDs in your own filesystem can also be listed with the +\f[C]lsf\f[R] command: +.PP +\f[C]rclone lsf Pixeldrain: --dirs-only -Fpi\f[R] +.PP +This will print directories in your \f[C]Pixeldrain\f[R] home directory +and their public IDs. +.PP +Enter this directory ID in the rclone config and you will be able to +access the directory. +.SS Standard options +.PP +Here are the Standard options specific to pixeldrain (Pixeldrain +Filesystem). +.SS --pixeldrain-api-key +.PP +API key for your pixeldrain account. +Found on https://pixeldrain.com/user/api_keys. +.PP +Properties: +.IP \[bu] 2 +Config: api_key +.IP \[bu] 2 +Env Var: RCLONE_PIXELDRAIN_API_KEY +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false +.SS --pixeldrain-root-folder-id +.PP +Root of the filesystem to use. +.PP +Set to \[aq]me\[aq] to use your personal filesystem. +Set to a shared directory ID to use a shared directory. +.PP +Properties: +.IP \[bu] 2 +Config: root_folder_id +.IP \[bu] 2 +Env Var: RCLONE_PIXELDRAIN_ROOT_FOLDER_ID +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Default: \[dq]me\[dq] +.SS Advanced options +.PP +Here are the Advanced options specific to pixeldrain (Pixeldrain +Filesystem). +.SS --pixeldrain-api-url +.PP +The API endpoint to connect to. +In the vast majority of cases it\[aq]s fine to leave this at default. +It is only intended to be changed for testing purposes. +.PP +Properties: +.IP \[bu] 2 +Config: api_url +.IP \[bu] 2 +Env Var: RCLONE_PIXELDRAIN_API_URL +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Default: \[dq]https://pixeldrain.com/api\[dq] +.SS --pixeldrain-description +.PP +Description of the remote. +.PP +Properties: +.IP \[bu] 2 +Config: description +.IP \[bu] 2 +Env Var: RCLONE_PIXELDRAIN_DESCRIPTION +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false +.SS Metadata +.PP +Pixeldrain supports file modes and creation times. +.PP +Here are the possible system metadata items for the pixeldrain backend. +.PP +.TS +tab(@); +lw(11.1n) lw(11.1n) lw(11.1n) lw(16.6n) lw(20.3n). +T{ +Name +T}@T{ +Help +T}@T{ +Type +T}@T{ +Example +T}@T{ +Read Only +T} +_ +T{ +btime +T}@T{ +Time of file birth (creation) +T}@T{ +RFC 3339 +T}@T{ +2006-01-02T15:04:05.999999999Z07:00 +T}@T{ +N +T} +T{ +mode +T}@T{ +File mode +T}@T{ +octal, unix style +T}@T{ +755 +T}@T{ +N +T} +T{ +mtime +T}@T{ +Time of last modification +T}@T{ +RFC 3339 +T}@T{ +2006-01-02T15:04:05.999999999Z07:00 +T}@T{ +N +T} +.TE +.PP +See the metadata (https://rclone.org/docs/#metadata) docs for more info. .SH premiumize.me .PP Paths are specified as \f[C]remote:path\f[R] @@ -62178,11 +64433,11 @@ If your browser doesn\[aq]t open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -type = premiumizeme -token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2029-08-07T18:44:15.548915378+01:00\[dq]} --------------------- +Configuration complete. +Options: +- type: premiumizeme +- token: {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2029-08-07T18:44:15.548915378+01:00\[dq]} +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -62468,12 +64723,12 @@ Option 2fa. Enter a value. Press Enter to leave empty. 2fa> 123456 Remote config --------------------- -[remote] -type = protondrive -user = you\[at]protonmail.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: protondrive +- user: you\[at]protonmail.com +- pass: *** ENCRYPTED *** +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -63146,12 +65401,12 @@ Option 2fa. Enter a value. Press Enter to leave empty. 2fa> 123456 Remote config --------------------- -[remote] -type = protondrive -user = you\[at]protonmail.com -pass = *** ENCRYPTED *** --------------------- +Configuration complete. +Options: +- type: protondrive +- user: you\[at]protonmail.com +- pass: *** ENCRYPTED *** +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -64079,14 +66334,15 @@ y/g/n> n Path to unencrypted PEM-encoded private key file, leave blank to use ssh-agent. key_file> Remote config --------------------- -[remote] -host = example.com -user = sftpuser -port = -pass = -key_file = --------------------- +Configuration complete. +Options: +- type: sftp +- host: example.com +- user: sftpuser +- port: +- pass: +- key_file: +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -64559,7 +66815,24 @@ Required: false .PP Raw PEM-encoded private key. .PP -If specified, will override key_file parameter. +Note that this should be on a single line with line endings replaced +with \[aq]\[aq], eg +.IP +.nf +\f[C] +key_pem = -----BEGIN RSA PRIVATE KEY-----\[rs]nMaMbaIXtE\[rs]n0gAMbMbaSsd\[rs]nMbaass\[rs]n-----END RSA PRIVATE KEY----- +\f[R] +.fi +.PP +This will generate the single line correctly: +.IP +.nf +\f[C] +awk \[aq]{printf \[dq]%s\[rs]\[rs]n\[dq], $0}\[aq] < \[ti]/.ssh/id_rsa +\f[R] +.fi +.PP +If specified, it will override the key_file parameter. .PP Properties: .IP \[bu] 2 @@ -65152,13 +67425,13 @@ Maximum number of SFTP simultaneous connections, 0 for unlimited. Note that setting this is very likely to cause deadlocks so it should be used with care. .PP -If you are doing a sync or copy then make sure concurrency is one more +If you are doing a sync or copy then make sure connections is one more than the sum of \f[C]--transfers\f[R] and \f[C]--checkers\f[R]. .PP If you use \f[C]--check-first\f[R] then it just needs to be one more than the maximum of \f[C]--checkers\f[R] and \f[C]--transfers\f[R]. .PP -So for \f[C]concurrency 3\f[R] you\[aq]d use +So for \f[C]connections 3\f[R] you\[aq]d use \f[C]--checkers 2 --transfers 2 --check-first\f[R] or \f[C]--checkers 1 --transfers 1\f[R]. .PP @@ -65927,11 +68200,11 @@ Access Grant. Enter a string value. Press Enter for the default (\[dq]\[dq]). access_grant> your-access-grant-received-by-someone-else Remote config --------------------- -[remote] -type = storj -access_grant = your-access-grant-received-by-someone-else --------------------- +Configuration complete. +Options: +- type: storj +- access_grant: your-access-grant-received-by-someone-else +Keep this \[dq]remote\[dq] remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -65983,14 +68256,14 @@ Encryption Passphrase. To access existing objects enter passphrase used for uplo Enter a string value. Press Enter for the default (\[dq]\[dq]). passphrase> your-human-readable-encryption-passphrase Remote config --------------------- -[remote] -type = storj -satellite_address = 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\[at]us1.storj.io:7777 -api_key = your-api-key-for-your-storj-project -passphrase = your-human-readable-encryption-passphrase -access_grant = the-access-grant-generated-from-the-api-key-and-passphrase --------------------- +Configuration complete. +Options: +- type: storj +- satellite_address: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\[at]us1.storj.io:7777 +- api_key: your-api-key-for-your-storj-project +- passphrase: your-human-readable-encryption-passphrase +- access_grant: the-access-grant-generated-from-the-api-key-and-passphrase +Keep this \[dq]remote\[dq] remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -66416,11 +68689,11 @@ Remote config Username (email address)> nick\[at]craig-wood.com Your Sugarsync password is only required during setup and will not be stored. password: --------------------- -[remote] -type = sugarsync -refresh_token = https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX --------------------- +Configuration complete. +Options: +- type: sugarsync +- refresh_token: https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX +Keep this \[dq]remote\[dq] remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -66681,11 +68954,6 @@ of an rclone union remote. See List of backends that do not support rclone about (https://rclone.org/overview/#optional-features) and rclone about (https://rclone.org/commands/rclone_about/) -.SH Tardigrade -.PP -The Tardigrade backend has been renamed to be the Storj -backend (https://rclone.org/storj/). -Old configuration files will continue to work. .SH Uloz.to .PP Paths are specified as \f[C]remote:path\f[R] @@ -67295,11 +69563,11 @@ Cache time of usage and free space (in seconds). This option is only useful when Enter a signed integer. Press Enter for the default (\[dq]120\[dq]). cache_time> Remote config --------------------- -[remote] -type = union -upstreams = remote1:dir1 remote2:dir2 remote3:dir3 --------------------- +Configuration complete. +Options: +- type: union +- upstreams: remote1:dir1 remote2:dir2 remote3:dir3 +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -67812,15 +70080,15 @@ password: Bearer token instead of user/pass (e.g. a Macaroon) bearer_token> Remote config --------------------- -[remote] -type = webdav -url = https://example.com/remote.php/webdav/ -vendor = nextcloud -user = user -pass = *** ENCRYPTED *** -bearer_token = --------------------- +Configuration complete. +Options: +- type: webdav +- url: https://example.com/remote.php/webdav/ +- vendor: nextcloud +- user: user +- pass: *** ENCRYPTED *** +- bearer_token: +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -68108,6 +70376,20 @@ Env Var: RCLONE_WEBDAV_OWNCLOUD_EXCLUDE_MOUNTS Type: bool .IP \[bu] 2 Default: false +.SS --webdav-unix-socket +.PP +Path to a unix domain socket to dial to, instead of opening a TCP +connection directly +.PP +Properties: +.IP \[bu] 2 +Config: unix_socket +.IP \[bu] 2 +Env Var: RCLONE_WEBDAV_UNIX_SOCKET +.IP \[bu] 2 +Type: string +.IP \[bu] 2 +Required: false .SS --webdav-description .PP Description of the remote. @@ -68388,12 +70670,13 @@ If your browser doesn\[aq]t open automatically go to the following link: http:// Log in and authorize rclone for access Waiting for code... Got code --------------------- -[remote] -client_id = -client_secret = -token = {\[dq]access_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]token_type\[dq]:\[dq]OAuth\[dq],\[dq]expiry\[dq]:\[dq]2016-12-29T12:27:11.362788025Z\[dq]} --------------------- +Configuration complete. +Options: +- type: yandex +- client_id: +- client_secret: +- token: {\[dq]access_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]token_type\[dq]:\[dq]OAuth\[dq],\[dq]expiry\[dq]:\[dq]2016-12-29T12:27:11.362788025Z\[dq]} +Keep this \[dq]remote\[dq] remote? y) Yes this is OK e) Edit this remote d) Delete this remote @@ -68582,6 +70865,21 @@ Env Var: RCLONE_YANDEX_ENCODING Type: Encoding .IP \[bu] 2 Default: Slash,Del,Ctl,InvalidUtf8,Dot +.SS --yandex-spoof-ua +.PP +Set the user agent to match an official version of the yandex disk +client. +May help with upload performance. +.PP +Properties: +.IP \[bu] 2 +Config: spoof_ua +.IP \[bu] 2 +Env Var: RCLONE_YANDEX_SPOOF_UA +.IP \[bu] 2 +Type: bool +.IP \[bu] 2 +Default: true .SS --yandex-description .PP Description of the remote. @@ -68686,12 +70984,12 @@ Choose a number from below, or type in your own value 1 / General \[rs] \[dq]4u2869d2aa6fca04f4f2f896b6539243b85b1\[dq] Enter a Workspace ID> 1 --------------------- -[remote] -type = zoho -token = {\[dq]access_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]token_type\[dq]:\[dq]Zoho-oauthtoken\[dq],\[dq]refresh_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]expiry\[dq]:\[dq]2020-10-12T00:54:52.370275223+02:00\[dq]} -root_folder_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --------------------- +Configuration complete. +Options: +- type: zoho +- token: {\[dq]access_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]token_type\[dq]:\[dq]Zoho-oauthtoken\[dq],\[dq]refresh_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]expiry\[dq]:\[dq]2020-10-12T00:54:52.370275223+02:00\[dq]} +- root_folder_id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +Keep this \[dq]remote\[dq] remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote @@ -69384,13 +71682,9 @@ This format requires absolute paths and the use of prefix For convenience rclone will automatically convert regular paths into the corresponding extended-length paths, so in most cases you do not have to worry about this (read more below). -.PP -Note that Windows supports using the same prefix -\f[C]\[rs]\[rs]?\[rs]\f[R] to specify path to volumes identified by -their GUID, e.g. +Using the same prefix \f[C]\[rs]\[rs]?\[rs]\f[R] it is also possible to +specify path to volumes identified by their GUID, e.g. \f[C]\[rs]\[rs]?\[rs]Volume{b75e2c83-0000-0000-0000-602f00000000}\[rs]some\[rs]path\f[R]. -This is \f[I]not\f[R] supported in rclone, due to an -issue (https://github.com/golang/go/issues/39785) in go. .SS Long paths .PP Rclone handles long paths automatically, by converting all paths to @@ -69846,6 +72140,38 @@ Env Var: RCLONE_LOCAL_CASE_INSENSITIVE Type: bool .IP \[bu] 2 Default: false +.SS --local-no-clone +.PP +Disable reflink cloning for server-side copies. +.PP +Normally, for local-to-local transfers, rclone will \[dq]clone\[dq] the +file when possible, and fall back to \[dq]copying\[dq] only when cloning +is not supported. +.PP +Cloning creates a shallow copy (or \[dq]reflink\[dq]) which initially +shares blocks with the original file. +Unlike a \[dq]hardlink\[dq], the two files are independent and neither +will affect the other if subsequently modified. +.PP +Cloning is usually preferable to copying, as it is much faster and is +deduplicated by default (i.e. +having two identical files does not consume more storage than having +just one.) However, for use cases where data redundancy is preferable, +--local-no-clone can be used to disable cloning and force \[dq]deep\[dq] +copies. +.PP +Currently, cloning is only supported when using APFS on macOS (support +for other platforms may be added in the future.) +.PP +Properties: +.IP \[bu] 2 +Config: no_clone +.IP \[bu] 2 +Env Var: RCLONE_LOCAL_NO_CLONE +.IP \[bu] 2 +Type: bool +.IP \[bu] 2 +Default: false .SS --local-no-preallocate .PP Disable preallocation of disk space for transferred files. @@ -70146,6 +72472,355 @@ Options: .IP \[bu] 2 \[dq]error\[dq]: return an error based on option value .SH Changelog +.SS v1.68.0 - 2024-09-08 +.PP +See commits (https://github.com/rclone/rclone/compare/v1.67.0...v1.68.0) +.IP \[bu] 2 +New backends +.RS 2 +.IP \[bu] 2 +Files.com (Sam Harrison) +.IP \[bu] 2 +Gofile (https://rclone.org/gofile/) (Nick Craig-Wood) +.IP \[bu] 2 +Pixeldrain (https://rclone.org/pixeldrain/) (Fornax) +.RE +.IP \[bu] 2 +Changed backends +.RS 2 +.IP \[bu] 2 +S3 (https://rclone.org/s3/) backend updated to use AWS +SDKv2 (https://github.com/aws/aws-sdk-go-v2) as v1 is now unsupported. +.RS 2 +.IP \[bu] 2 +The matrix of providers and auth methods is huge and there could be +problems with obscure combinations. +.IP \[bu] 2 +Please report problems in a new +issue (https://github.com/rclone/rclone/issues/new/choose) on Github. +.RE +.RE +.IP \[bu] 2 +New commands +.RS 2 +.IP \[bu] 2 +config +encryption (https://rclone.org/commands/rclone_config_encryption/): set, +remove and check to manage config file encryption (Nick Craig-Wood) +.RE +.IP \[bu] 2 +New Features +.RS 2 +.IP \[bu] 2 +build +.RS 2 +.IP \[bu] 2 +Update to go1.23 and make go1.21 the minimum required version (Nick +Craig-Wood) +.IP \[bu] 2 +Update all dependencies (Nick Craig-Wood) +.IP \[bu] 2 +Disable wasm/js build due to go bug +#64856 (https://github.com/golang/go/issues/64856) (Nick Craig-Wood) +.IP \[bu] 2 +Enable custom linting rules with ruleguard via gocritic (albertony) +.IP \[bu] 2 +Update logging statements to make \f[C]--use-json-log\f[R] work always +(albertony) +.IP \[bu] 2 +Adding new code quality tests and fixing the fallout (albertony) +.RE +.IP \[bu] 2 +config +.RS 2 +.IP \[bu] 2 +Internal config re-organised to be more consistent and make it available +from the rc (Nick Craig-Wood) +.IP \[bu] 2 +Avoid remotes with empty names from the environment (albertony) +.IP \[bu] 2 +Make listing of remotes more consistent (albertony) +.IP \[bu] 2 +Make getting config values more consistent (albertony) +.IP \[bu] 2 +Use \f[C]--password-command\f[R] to set config file password if supplied +(Nick Craig-Wood) +.RE +.IP \[bu] 2 +doc fixes (albertony, crystalstall, David Seifert, Eng Zer Jun, Ernie +Hershey, Florian Klink, John Oxley, kapitainsky, Mathieu Moreau, Nick +Craig-Wood, nipil, P\['e]tr Bozs\['o], Russ Bubley, Sam Harrison, +Thearas, URenko, Will Miles, yuval-cloudinary) +.IP \[bu] 2 +fs: Allow semicolons as well as spaces in \f[C]--bwlimit\f[R] timetable +parsing (Kyle Reynolds) +.IP \[bu] 2 +help +.RS 2 +.IP \[bu] 2 +Global flags help command now takes glob filter (albertony) +.IP \[bu] 2 +Make help command output less distracting (albertony) +.RE +.IP \[bu] 2 +lib/encoder: Add Raw encoding for use where no encoding at all is +required, eg \f[C]--local-encoding Raw\f[R] (URenko) +.IP \[bu] 2 +listremotes: Added options for filtering, ordering and json output +(albertony) +.IP \[bu] 2 +nfsmount +.RS 2 +.IP \[bu] 2 +Make the \f[C]--sudo\f[R] flag work for umount as well as mount (Nick +Craig-Wood) +.IP \[bu] 2 +Add \f[C]-o tcp\f[R] option to NFS mount options to fix mounting under +Linux (Nick Craig-Wood) +.RE +.IP \[bu] 2 +operations: copy: generate stable partial suffix (Georg Welzel) +.IP \[bu] 2 +rc +.RS 2 +.IP \[bu] 2 +Add options/info (https://rclone.org/rc/#options-info) call to enumerate +options (Nick Craig-Wood) +.IP \[bu] 2 +Add option blocks parameter to +options/get (https://rclone.org/rc/#options-get) and options/info (Nick +Craig-Wood) +.IP \[bu] 2 +Add vfs/queue (https://rclone.org/rc/#vfs-queue) to show the status of +the upload queue (Nick Craig-Wood) +.IP \[bu] 2 +Add vfs/queue-set-expiry (https://rclone.org/rc/#vfs-queue-set-expiry) +to adjust expiry of items in the VFS queue (Nick Craig-Wood) +.IP \[bu] 2 +Add \f[C]--unix-socket\f[R] option to \f[C]rc\f[R] command (Florian +Klink) +.IP \[bu] 2 +Prevent unmount rc command from sending a \f[C]STOPPING=1\f[R] sd-notify +message (AThePeanut4) +.RE +.IP \[bu] 2 +rcserver: Implement prometheus +metrics (https://rclone.org/docs/#metrics) on a dedicated port (Oleg +Kunitsyn) +.IP \[bu] 2 +serve dlna +.RS 2 +.IP \[bu] 2 +Also look at \[dq]Subs\[dq] subdirectory (Florian Klink) +.IP \[bu] 2 +Don\[aq]t swallow \f[C]video.{idx,sub}\f[R] (Florian Klink) +.IP \[bu] 2 +Set more correct mime type (Florian Klink) +.RE +.IP \[bu] 2 +serve nfs +.RS 2 +.IP \[bu] 2 +Implement on disk cache for file handles selected with +\f[C]--nfs-cache-type\f[R] (Nick Craig-Wood) +.IP \[bu] 2 +Add tracing to filesystem calls (Nick Craig-Wood) +.IP \[bu] 2 +Mask unimplemented error from chmod (Nick Craig-Wood) +.IP \[bu] 2 +Unify the nfs library logging with rclone\[aq]s logging better (Nick +Craig-Wood) +.IP \[bu] 2 +Fix incorrect user id and group id exported to NFS (Nick Craig-Wood) +.RE +.IP \[bu] 2 +serve s3 +.RS 2 +.IP \[bu] 2 +Implement \f[C]--auth-proxy\f[R] (Sawjan Gurung) +.IP \[bu] 2 +Update to AWS SDKv2 by updating \f[C]github.com/rclone/gofakes3\f[R] +(Nick Craig-Wood) +.RE +.RE +.IP \[bu] 2 +Bug Fixes +.RS 2 +.IP \[bu] 2 +bisync: Fix sync time problems with backends that round time (eg +Dropbox) (nielash) +.IP \[bu] 2 +serve dlna: Fix panic: invalid argument to Int63n (Nick Craig-Wood) +.RE +.IP \[bu] 2 +VFS +.RS 2 +.IP \[bu] 2 +Add +--vfs-read-chunk-streams (https://rclone.org/commands/rclone_mount/#vfs-read-chunk-streams-0-1) +to parallel read chunks from files (Nick Craig-Wood) +.RS 2 +.IP \[bu] 2 +This can increase mount performance on high bandwidth or large latency +links +.RE +.IP \[bu] 2 +Fix cache encoding with special characters (URenko) +.RE +.IP \[bu] 2 +Local +.RS 2 +.IP \[bu] 2 +Fix encoding of root path fix (URenko) +.IP \[bu] 2 +Add server-side copy (using clone) with xattrs on macOS (nielash) +.RS 2 +.IP \[bu] 2 +\f[C]--local-no-clone\f[R] flag to disable cloning for server-side +copies (nielash) +.RE +.IP \[bu] 2 +Support setting custom \f[C]--metadata\f[R] during server-side Copy +(nielash) +.RE +.IP \[bu] 2 +Azure Blob +.RS 2 +.IP \[bu] 2 +Allow anonymous access for public resources (Nick Craig-Wood) +.RE +.IP \[bu] 2 +B2 +.RS 2 +.IP \[bu] 2 +Include custom upload headers in large file info (Pat Patterson) +.RE +.IP \[bu] 2 +Drive +.RS 2 +.IP \[bu] 2 +Fix copying Google Docs to a backend which only supports SHA1 (Nick +Craig-Wood) +.RE +.IP \[bu] 2 +Fichier +.RS 2 +.IP \[bu] 2 +Fix detection of Flood Detected error (Nick Craig-Wood) +.IP \[bu] 2 +Fix server side move (Nick Craig-Wood) +.RE +.IP \[bu] 2 +HTTP +.RS 2 +.IP \[bu] 2 +Reload client certificates on expiry (Saleh Dindar) +.IP \[bu] 2 +Support listening on passed FDs (Florian Klink) +.RE +.IP \[bu] 2 +Jottacloud +.RS 2 +.IP \[bu] 2 +Fix setting of metadata on server side move (albertony) +.RE +.IP \[bu] 2 +Onedrive +.RS 2 +.IP \[bu] 2 +Fix nil pointer error when uploading small files (Nick Craig-Wood) +.RE +.IP \[bu] 2 +Pcloud +.RS 2 +.IP \[bu] 2 +Implement \f[C]SetModTime\f[R] (Georg Welzel) +.IP \[bu] 2 +Implement \f[C]OpenWriterAt\f[R] feature to enable multipart uploads +(Georg Welzel) +.RE +.IP \[bu] 2 +Pikpak +.RS 2 +.IP \[bu] 2 +Improve data consistency by ensuring async tasks complete (wiserain) +.IP \[bu] 2 +Implement custom hash to replace wrong sha1 (wiserain) +.IP \[bu] 2 +Fix error with \f[C]copyto\f[R] command (wiserain) +.IP \[bu] 2 +Optimize file move by removing unnecessary \f[C]readMetaData()\f[R] call +(wiserain) +.IP \[bu] 2 +Non-buffered hash calculation for local source files (wiserain) +.IP \[bu] 2 +Optimize upload by pre-fetching gcid from API (wiserain) +.IP \[bu] 2 +Correct file transfer progress for uploads by hash (wiserain) +.IP \[bu] 2 +Update to using AWS SDK v2 (wiserain) +.RE +.IP \[bu] 2 +S3 +.RS 2 +.IP \[bu] 2 +Update to using AWS SDK v2 (Nick Craig-Wood) +.RS 2 +.IP \[bu] 2 +Add \f[C]--s3-sdk-log-mode\f[R] to control SDKv2 debugging (Nick +Craig-Wood) +.RE +.IP \[bu] 2 +Fix incorrect region for Magalu provider (Filipe Herculano) +.IP \[bu] 2 +Allow restoring from intelligent-tiering storage class (Pawel Palucha) +.RE +.IP \[bu] 2 +SFTP +.RS 2 +.IP \[bu] 2 +Use \f[C]uint32\f[R] for mtime to save memory (Tomasz Melcer) +.IP \[bu] 2 +Ignore useless errors when closing the connection pool (Nick Craig-Wood) +.IP \[bu] 2 +Support listening on passed FDs (Florian Klink) +.RE +.IP \[bu] 2 +Swift +.RS 2 +.IP \[bu] 2 +Add workarounds for bad listings in Ceph RGW (Paul Collins) +.IP \[bu] 2 +Add total/free space info in \f[C]about\f[R] command. +(fsantagostinobietti) +.RE +.IP \[bu] 2 +Ulozto +.RS 2 +.IP \[bu] 2 +Fix upload of > 2GB files on 32 bit platforms (Tobias Markus) +.RE +.IP \[bu] 2 +WebDAV +.RS 2 +.IP \[bu] 2 +Add \f[C]--webdav-unix-socket-path\f[R] to connect to a unix socket +(Florian Klink) +.RE +.IP \[bu] 2 +Yandex +.RS 2 +.IP \[bu] 2 +Implement custom user agent to help with upload speeds (Sebastian +B\[:u]nger) +.RE +.IP \[bu] 2 +Zoho +.RS 2 +.IP \[bu] 2 +Fix inefficiencies uploading with new API to avoid throttling (Nick +Craig-Wood) +.RE .SS v1.67.0 - 2024-06-14 .PP See commits (https://github.com/rclone/rclone/compare/v1.66.0...v1.67.0) @@ -84509,9 +87184,12 @@ This will make the garbage collector work harder, reducing memory size at the expense of CPU usage. .PP The most common cause of rclone using lots of memory is a single -directory with thousands or millions of files in. +directory with millions of files in. Rclone has to load this entirely into memory as rclone objects. Each rclone object takes 0.5k-1k of memory. +There is a workaround for +this (https://github.com/rclone/rclone/wiki/Big-syncs-with-millions-of-files) +which involves a bit of scripting. .SS Rclone changes fullwidth Unicode punctuation marks in file names .PP For example: On a Windows system, you have a file with name @@ -86286,6 +88964,55 @@ Micha\[/l] Dzienisiewicz Florian Klink .IP \[bu] 2 Bill Fraser +.IP \[bu] 2 +Thearas +.IP \[bu] 2 +Filipe Herculano +.IP \[bu] 2 +Russ Bubley +.IP \[bu] 2 +Paul Collins +.IP \[bu] 2 +Tomasz Melcer +.IP \[bu] 2 +itsHenry <2671230065@qq.com> +.IP \[bu] 2 +Ke Wang +.IP \[bu] 2 +AThePeanut4 <49614525+AThePeanut4@users.noreply.github.com> +.IP \[bu] 2 +Tobias Markus +.IP \[bu] 2 +Ernie Hershey +.IP \[bu] 2 +Will Miles +.IP \[bu] 2 +David Seifert <16636962+SoapGentoo@users.noreply.github.com> +.IP \[bu] 2 +Fornax +.IP \[bu] 2 +Sam Harrison +.IP \[bu] 2 +P\['e]ter Bozs\['o] <3806723+peterbozso@users.noreply.github.com> +.IP \[bu] 2 +Georg Welzel +.IP \[bu] 2 +John Oxley +.IP \[bu] 2 +Pawel Palucha +.IP \[bu] 2 +crystalstall +.IP \[bu] 2 +nipil +.IP \[bu] 2 +yuval-cloudinary <46710068+yuval-cloudinary@users.noreply.github.com> +.IP \[bu] 2 +Mathieu Moreau +.IP \[bu] 2 +fsantagostinobietti +<6057026+fsantagostinobietti@users.noreply.github.com> +.IP \[bu] 2 +Oleg Kunitsyn <114359669+hiddenmarten@users.noreply.github.com> .SH Contact the rclone project .SS Forum .PP