From 89f2d43f17e60f38619fda797fb0a349ff819d7e Mon Sep 17 00:00:00 2001 From: Nick Craig-Wood Date: Mon, 26 Oct 2020 13:26:58 +0000 Subject: [PATCH] Version v1.53.2 --- MANUAL.html | 249 +++++++++++--- MANUAL.md | 258 +++++++++----- MANUAL.txt | 270 ++++++++++----- docs/content/changelog.md | 50 +++ docs/content/commands/rclone_mount.md | 3 - docs/content/flags.md | 2 +- docs/content/s3.md | 63 +++- rclone.1 | 462 ++++++++++++++++++++------ 8 files changed, 1036 insertions(+), 321 deletions(-) diff --git a/MANUAL.html b/MANUAL.html index e7192362c..d476d8575 100644 --- a/MANUAL.html +++ b/MANUAL.html @@ -17,7 +17,7 @@

rclone(1) User Manual

Nick Craig-Wood

-

Sep 13, 2020

+

Oct 26, 2020

Rclone syncs your files to cloud storage

rclone logo

@@ -1455,7 +1455,6 @@ umount /path/to/local/mount

--vfs-read-chunk-size will enable reading the source objects in parts. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read at the cost of an increased number of requests.

When --vfs-read-chunk-size-limit is also specified and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled for each chunk read, until the specified value is reached. A value of -1 will disable the limit and the chunk size will grow indefinitely.

With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.

-

Chunked reading will only work with --vfs-cache-mode < full, as the file will always be copied to the vfs cache before opening with --vfs-cache-mode full.

VFS - Virtual File System

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

@@ -3397,8 +3396,10 @@ y/n/s/!/q> n

The remote in use must support server side move or copy and you must use the same remote as the destination of the sync.

This is for use with files to add the suffix in the current directory or with --backup-dir. See --backup-dir for more info.

For example

-
rclone sync -i /path/to/local/file remote:current --suffix .bak
-

will sync /path/to/local to remote:current, but for any files which would have been updated or deleted have .bak added.

+
rclone copy -i /path/to/local/file remote:current --suffix .bak
+

will copy /path/to/local to remote:current, but for any files which would have been updated or deleted have .bak added.

+

If using rclone sync with --suffix and without --backup-dir then it is recommended to put a filter rule in excluding the suffix otherwise the sync will delete the backup files.

+
rclone sync -i /path/to/local/file remote:current --suffix .bak --exclude "*.bak"

--suffix-keep-extension

When using --suffix, setting this causes rclone put the SUFFIX before the extension of the files that it backs up rather than after.

So let's say we had --suffix -2019-01-01, without the flag file.txt would be backed up to file.txt-2019-01-01 and with the flag it would be backed up to file-2019-01-01.txt. This can be helpful to make sure the suffixed files can still be opened.

@@ -6290,7 +6291,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total --use-json-log Use json log format. --use-mmap Use mmap allocator (see docs). --use-server-modtime Use server modified time instead of object metadata - --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.53.1") + --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.53.2") -v, --verbose count Print lots more stuff (repeat for more)

Backend Flags

These flags are available for every command. They control the backends and may be set in the config file.

@@ -7178,7 +7179,7 @@ Choose a number from below, or type in your own value / Asia Pacific (Mumbai) 13 | Needs location constraint ap-south-1. \ "ap-south-1" - / Asia Patific (Hong Kong) Region + / Asia Pacific (Hong Kong) Region 14 | Needs location constraint ap-east-1. \ "ap-east-1" / South America (Sao Paulo) Region @@ -7553,16 +7554,16 @@ y/e/d>
  • US East (Ohio) Region
  • Needs location constraint us-east-2.
    • -
  • "us-west-2" -
  • "us-west-1"
    • +
  • "us-west-2" +
  • "ca-central-1"
    • +
  • "eu-west-3" +
  • "eu-north-1"
    • +
  • "eu-south-1" +
  • "eu-central-1"
    • +
  • "ap-northeast-3" +
  • "ap-south-1"
  • "ap-east-1"
  • "sa-east-1" @@ -7623,6 +7639,36 @@ y/e/d>
  • South America (Sao Paulo) Region
  • Needs location constraint sa-east-1.
    • +
  • "me-south-1" +
    • +
  • "af-south-1" +
    • +
  • "cn-north-1" +
    • +
  • "cn-northwest-1" +
    • +
  • "us-gov-east-1" +
    • +
  • "us-gov-west-1" +

    • --s3-region

    @@ -8202,14 +8248,14 @@ y/e/d> -
  • "us-west-2" -
  • "us-west-1"
    • +
  • "us-west-2" +
  • "ca-central-1"

    • --s3-location-constraint

    @@ -10194,7 +10276,7 @@ y/e/d> y

    Limitations

    Note that Box is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

    -

    Box file names can't have the \ character in. rclone maps this to and from an identical looking unicode equivalent ＼.

    +

    Box file names can't have the \ character in. rclone maps this to and from an identical looking unicode equivalent ＼ (U+FF3C Fullwidth Reverse Solidus).

    Box only supports filenames up to 255 characters in length.

    Cache (BETA)

    The cache remote wraps another existing remote and stores file structure and its data for long running tasks like rclone mount.

    @@ -11058,10 +11140,11 @@ y/e/d> y
  • Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot

    • Crypt

    -

    The crypt remote encrypts and decrypts another remote.

    -

    To use it first set up the underlying remote following the config instructions for that remote. You can also use a local pathname instead of a remote which will encrypt and decrypt from that directory which might be useful for encrypting onto a USB stick for example.

    -

    First check your chosen remote is working - we'll call it remote:path in these docs. Note that anything inside remote:path will be encrypted and anything outside won't. This means that if you are using a bucket based remote (eg S3, B2, swift) then you should probably put the bucket in the remote s3:bucket. If you just use s3: then rclone will make encrypted bucket names too (if using file name encryption) which may or may not be what you want.

    -

    Now configure crypt using rclone config. We will call this one secret to differentiate it from the remote.

    +

    Rclone crypt remotes encrypt and decrypt other remotes.

    +

    To use crypt, first set up the underlying remote. Follow the rclone config instructions for that remote.

    +

    crypt applied to a local pathname instead of a remote will encrypt and decrypt that directory, and can be used to encrypt USB removable drives.

    +

    Before configuring the crypt remote, check the underlying remote is working. In this example the underlying remote is called remote:path. Anything inside remote:path will be encrypted and anything outside will not. In the case of an S3 based underlying remote (eg Amazon S3, B2, Swift) it is generally advisable to define a crypt remote in the underlying remote s3:bucket. If s3: alone is specified alongside file name encryption, rclone will encrypt the bucket name.

    +

    Configure crypt using rclone config. In this example the crypt remote is called secret, to differentiate it from the underlying remote.

    No remotes found - make a new one
 n) New remote
 s) Set configuration password
@@ -11131,21 +11214,20 @@ y) Yes this is OK
 e) Edit this remote
 d) Delete this remote
 y/e/d> y
    -

    Important The password is stored in the config file is lightly obscured so it isn't immediately obvious what it is. It is in no way secure unless you use config file encryption.

    -

    A long passphrase is recommended, or you can use a random one.

    -

    The obscured password is created by using AES-CTR with a static key, with the salt stored verbatim at the beginning of the obscured password. This static key is shared by between all versions of rclone.

    +

    Important The crypt password stored in rclone.conf is lightly obscured. That only protects it from cursory inspection. It is not secure unless encryption of rclone.conf is specified.

    +

    A long passphrase is recommended, or rclone config can generate a random one.

    +

    The obscured password is created using AES-CTR with a static key. The salt is stored verbatim at the beginning of the obscured password. This static key is shared between all versions of rclone.

    If you reconfigure rclone with the same passwords/passphrases elsewhere it will be compatible, but the obscured version will be different due to the different salt.

    -

    Note that rclone does not encrypt

    +

    Rclone does not encrypt

    Specifying the remote

    -

    In normal use, make sure the remote has a : in. If you specify the remote without a : then rclone will use a local directory of that name. So if you use a remote of /path/to/secret/files then rclone will encrypt stuff to that directory. If you use a remote of name then rclone will put files in a directory called name in the current directory.

    -

    If you specify the remote as remote:path/to/dir then rclone will store encrypted files in path/to/dir on the remote. If you are using file name encryption, then when you save files to secret:subdir/subfile this will store them in the unencrypted path path/to/dir but the subdir/subpath bit will be encrypted.

    -

    Note that unless you want encrypted bucket names (which are difficult to manage because you won't know what directory they represent in web interfaces etc), you should probably specify a bucket, eg remote:secretbucket when using bucket based remotes such as S3, Swift, Hubic, B2, GCS.

    +

    In normal use, ensure the remote has a : in. If specified without, rclone uses a local directory of that name. For example if a remote /path/to/secret/files is specified, rclone encrypts content to that directory. If a remote name is specified, rclone targets a directory name in the current directory.

    +

    If remote remote:path/to/dir is specified, rclone stores encrypted files in path/to/dir on the remote. With file name encryption, files saved to secret:subdir/subfile are stored in the unencrypted path path/to/dir but the subdir/subpath element is encrypted.

    Example

    -

    To test I made a little directory of files using "standard" file name encryption.

    +

    Create the following file structure using "standard" file name encryption.

    plaintext/
 ├── file0.txt
 ├── file1.txt
@@ -11154,7 +11236,7 @@ y/e/d> y
    ├── file3.txt └── subsubdir └── file4.txt -

    Copy these to the remote and list them back

    +

    Copy these to the remote, and list them

    $ rclone -q copy plaintext secret:
 $ rclone -q ls secret:
         7 file1.txt
@@ -11162,19 +11244,19 @@ $ rclone -q ls secret:
         8 subdir/file2.txt
        10 subdir/subsubdir/file4.txt
         9 subdir/file3.txt
    -

    Now see what that looked like when encrypted

    +

    The crypt remote looks like

    $ rclone -q ls remote:path
        55 hagjclgavj2mbiqm6u6cnjjqcg
        54 v05749mltvv1tf4onltun46gls
        57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo
        58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc
        56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
    -

    Note that this retains the directory structure which means you can do this

    +

    The directory structure is preserved

    $ rclone -q ls secret:subdir
         8 file2.txt
         9 file3.txt
        10 subsubdir/file4.txt
    -

    If don't use file name encryption then the remote will look like this - note the .bin extensions added to prevent the cloud provider attempting to interpret the data.

    +

    Without file name encryption .bin extensions are added to underlying names. This prevents the cloud provider attempting to interpret file content.

    $ rclone -q ls remote:path
        54 file0.txt.bin
        57 subdir/file3.txt.bin
@@ -11182,7 +11264,6 @@ $ rclone -q ls secret:
        58 subdir/subsubdir/file4.txt.bin
        55 file1.txt.bin

    File name encryption modes

    -

    Here are some of the features of the file name encryption modes

    Off

    Obfuscation

    -

    This is a simple "rotate" of the filename, with each file having a rot distance based on the filename. We store the distance at the beginning of the filename. So a file called "hello" may become "53.jgnnq".

    -

    This is not a strong encryption of filenames, but it may stop automated scanning tools from picking up on filename patterns. As such it's an intermediate between "off" and "standard". The advantage is that it allows for longer path segment names.

    -

    There is a possibility with some unicode based filenames that the obfuscation is weak and may map lower case characters to upper case equivalents. You can not rely on this for strong protection.

    +

    This is a simple "rotate" of the filename, with each file having a rot distance based on the filename. Rclone stores the distance at the beginning of the filename. A file called "hello" may become "53.jgnnq".

    +

    Obfuscation is not a strong encryption of filenames, but hinders automated scanning tools picking up on filename patterns. It is an intermediate between "off" and "standard" which allows for longer path segment names.

    +

    There is a possibility with some unicode based filenames that the obfuscation is weak and may map lower case characters to upper case equivalents.

    +

    Obfuscation cannot be relied upon for strong protection.

    -

    Cloud storage systems have various limits on file name length and total path length which you are more likely to hit using "Standard" file name encryption. If you keep your file names to below 156 characters in length then you should be OK on all providers.

    -

    There may be an even more secure file name encryption mode in the future which will address the long file name problem.

    +

    Cloud storage systems have limits on file name length and total path length which rclone is more likely to breach using "Standard" file name encryption. Where file names are less thn 156 characters in length issues should not be encountered, irrespective of cloud storage provider.

    +

    An alternative, future rclone file name encryption mode may tolerate backend provider path length limits.

    Directory name encryption

    Crypt offers the option of encrypting dir names or leaving them intact. There are two options:

    True

    @@ -11220,7 +11302,7 @@ $ rclone -q ls secret:

    Modified time and hashes

    Crypt stores modification times using the underlying remote so support depends on that.

    Hashes are not stored for crypt. However the data integrity is protected by an extremely strong crypto authenticator.

    -

    Note that you should use the rclone cryptcheck command to check the integrity of a crypted remote instead of rclone check which can't check the checksums properly.

    +

    Use the rclone cryptcheck command to check the integrity of a crypted remote instead of rclone check which can't check the checksums properly.

    Standard Options

    Here are the standard options specific to crypt (Encrypt/Decrypt a remote).

    --crypt-remote

    @@ -18470,6 +18552,87 @@ $ tree /tmp/b
  • "error": return an error based on option value

    • Changelog

    +

    v1.53.2 - 2020-10-26

    +

    See commits

    +

    v1.53.1 - 2020-09-13

    See commits