extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-11-23 00:43:49 +01:00
Code Issues Packages Projects Releases Wiki Activity

crypt: docs: extended description

Browse Source
This commit is contained in:
albertony 2020-11-25 18:50:44 +01:00 committed by Nick Craig-Wood
parent 7c0287b824
commit 3b49440c25
1 changed files with 183 additions and 49 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

232
docs/content/crypt.md
View File

@ -8,59 +8,126 @@ description: "Encryption overlay remote"
Rclone `crypt` remotes encrypt and decrypt other remotes.
To use `crypt`, first set up the underlying remote. Follow the `rclone
config` instructions for that remote.
A remote of type `crypt` does not access a [storage system](https://rclone.org/overview/)
directly, but instead wraps another remote, which in turn accesses
the storage system. This is similar to how [alias](https://rclone.org/alias/),
[union](https://rclone.org/union/), [chunker](https://rclone.org/chunker/)
and a few others work. It makes the usage very flexible, as you can
add a layer, in this case an encryption layer, on top of any other
backend, even in multiple layers. Rclone's functionality
can be used as with any other remote, for example you can
[mount](https://rclone.org/commands/rclone_mount/) a crypt remote.
`crypt` applied to a local pathname instead of a remote will
encrypt and decrypt that directory, and can be used to encrypt USB
removable drives.
Accessing a storage system through a crypt remote realizes client-side
encryption, which makes it safe to keep your data in a location you do
not trust will not get compromised.
When working against the `crypt` remote, rclone will automatically
encrypt (before uploading) and decrypt (after downloading) on your local
system as needed on the fly, leaving the data encrypted at rest in the
wrapped remote. If you access the storage system using an application
other than rclone, or access the wrapped remote directly using rclone,
there will not be any encryption/decryption: Downloading existing content
will just give you the encrypted (scrambled) format, and anything you
upload will *not* become encrypted.
The encryption is a secret-key encryption (also called symmetric key encryption)
algorithm, where a password (or pass phrase) is used to generate real encryption key.
The password can be supplied by user, or you may chose to let rclone
generate one. It will be stored in the configuration file, in a lightly obscured form.
If you are in an environment where you are not able to keep your configuration
secured, you should add
[configuration encryption](https://rclone.org/docs/#configuration-encryption)
as protection. As long as you have this configuration file, you will be able to
decrypt your data. Without the configuration file, as long as you remember
the password (or keep it in a safe place), you can re-create the configuration
and gain access to the existing data. You may also configure a corresponding
remote in a different installation to access the same data.
See below for guidance to [changing password](#changing-password).
Encryption uses [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)),
to permute the encryption key so that the same string may be encrypted in
different ways. When configuring the crypt remote it is optional to enter a salt,
or to let rclone generate a unique salt. If omitted, rclone uses a built-in unique string.
Normally in cryptography, the salt is stored together with the encrypted content,
and do not have to be memorized by the user. This is not the case in rclone,
because rclone does not store any additional information on the remotes. Use of
custom salt is effectively a second password that must be memorized.
[File content](#file-encryption) encryption is performed using
[NaCl SecretBox](https://godoc.org/golang.org/x/crypto/nacl/secretbox),
based on XSalsa20 cipher and Poly1305 for integrity.
[Names](#name-encryption) (file- and directory names) are also encrypted
by default, but this has some implications and is therefore
possible to turned off.
### Configuration
Here is an example of how to make a remote called `secret`.
To use `crypt`, first set up the underlying remote. Follow the
`rclone config` instructions for the specific backend.
Before configuring the crypt remote, check the underlying remote is
working. In this example the underlying remote is called `remote:path`.
Anything inside `remote:path` will be encrypted and anything outside
will not. In the case of an S3 based underlying remote (e.g. Amazon S3,
B2, Swift) it is generally advisable to define a crypt remote in the
underlying remote `s3:bucket`. If `s3:` alone is specified alongside
file name encryption, rclone will encrypt the bucket name.
working. In this example the underlying remote is called `remote`.
We will configure a path `path` within this remote to contain the
encrypted content. Anything inside `remote:path` will be encrypted
and anything outside will not.
Configure `crypt` using `rclone config`. In this example the `crypt`
remote is called `secret`, to differentiate it from the underlying
`remote`.
When you are done you can use the crypt remote named `secret` just
as you would with any other remote, e.g. `rclone copy D:\docs secret:\docs`,
and rclone will encrypt and decrypt as needed on the fly.
If you access the wrapped remote `remote:path` directly you will bypass
the encryption, and anything you read will be in encrypted form, and
anything you write will be undencrypted. To avoid issues it is best to
configure a dedicated path for encrypted content, and access it
exclusively through a crypt remote.
```
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
n/s/q> n
name> secret
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[snip]
XX / Encrypt/Decrypt a remote
\ "crypt"
[snip]
Storage> crypt
** See help for crypt backend at: https://rclone.org/crypt/ **
Remote to encrypt/decrypt.
Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
Normally should contain a ':' and a path, eg "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
Enter a string value. Press Enter for the default ("").
remote> remote:path
How to encrypt the filenames.
Enter a string value. Press Enter for the default ("standard").
Choose a number from below, or type in your own value
1 / Don't encrypt the file names. Adds a ".bin" extension only.
\ "off"
2 / Encrypt the filenames see the docs for the details.
1 / Encrypt the filenames see the docs for the details.
\ "standard"
3 / Very simple filename obfuscation.
2 / Very simple filename obfuscation.
\ "obfuscate"
filename_encryption> 2
3 / Don't encrypt the file names. Adds a ".bin" extension only.
\ "off"
filename_encryption>
Option to either encrypt directory names or leave them intact.
NB If filename_encryption is "off" then this option will do nothing.
Enter a boolean value (true or false). Press Enter for the default ("true").
Choose a number from below, or type in your own value
1 / Encrypt directory names.
\ "true"
2 / Don't encrypt directory names, leave them intact.
\ "false"
directory_name_encryption> 1
directory_name_encryption>
Password or pass phrase for encryption.
y) Yes type in my own password
g) Generate random password
@ -73,7 +140,7 @@ Password or pass phrase for salt. Optional but recommended.
Should be different to the previous password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank
n) No leave this optional password blank (default)
y/g/n> g
Password strength in bits.
64 is just about memorable
@ -81,27 +148,33 @@ Password strength in bits.
1024 is the maximum
Bits> 128
Your password is: JAsJvRcgR-_veXNfy_sGmQ
Use this password?
y) Yes
Use this password? Please note that an obscured version of this
password (and not the password itself) will be stored under your
configuration file, so keep this generated password in a safe place.
y) Yes (default)
n) No
y/n> y
y/n>
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n>
Remote config
--------------------
[secret]
type = crypt
remote = remote:path
filename_encryption = standard
password = *** ENCRYPTED ***
password2 = *** ENCRYPTED ***
--------------------
y) Yes this is OK
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
y/e/d>
```
**Important** The crypt password stored in `rclone.conf` is lightly
obscured. That only protects it from cursory inspection. It is not
secure unless encryption of `rclone.conf` is specified.
secure unless [configuration encryption](https://rclone.org/docs/#configuration-encryption) of `rclone.conf` is specified.
A long passphrase is recommended, or `rclone config` can generate a
random one.
@ -119,20 +192,80 @@ Rclone does not encrypt
* file length - this can be calculated within 16 bytes
* modification time - used for syncing
## Specifying the remote ##
### Specifying the remote
In normal use, ensure the remote has a `:` in. If specified without,
rclone uses a local directory of that name. For example if a remote
`/path/to/secret/files` is specified, rclone encrypts content to that
directory. If a remote `name` is specified, rclone targets a directory
`name` in the current directory.
When configuring the remote to encrypt/decrypt, you may specify any
string that rclone accepts as a source/destination of other commands.
If remote `remote:path/to/dir` is specified, rclone stores encrypted
The primary use case is to specify the path into an already configured
remote (e.g. `remote:path/to/dir` or `remote:bucket`), such that
data in a remote untrusted location can be stored encrypted.
You may also specify a local filesystem path, such as
`/path/to/dir` on Linux, `C:\path\to\dir` on Windows. By creating
a crypt remote pointing to such a local filesystem path, you can
use rclone as a utility for pure local file encryption, for example
to keep encrypted files on a removable USB drive.
**Note**: A string which do not contain a `:` will by rclone be treated
as a relative path in the local filesystem. For example, if you enter
the name `remote` without the trailing `:`, it will be treated as
a subdirectory of the current directory with name "remote".
If a path `remote:path/to/dir` is specified, rclone stores encrypted
files in `path/to/dir` on the remote. With file name encryption, files
saved to `secret:subdir/subfile` are stored in the unencrypted path
`path/to/dir` but the `subdir/subpath` element is encrypted.
## Example ##
The path you specify does not have to exist, rclone will create
it when needed.
If you intend to use the wrapped remote both directly for keeping
unencrypted content, as well as through a crypt remote for encrypted
content, it is recommended to point the crypt remote to a separate
directory within the wrapped remote. If you use a bucket based storage
system (e.g. Swift, S3, Google Compute Storage, B2, Hubic) it is generally
advisable to wrap the crypt remote around a specific bucket (`s3:bucket`).
If wrapping around the entire root of the storage (`s3:`), and use the
optional file name encryption, rclone will encrypt the bucket name.
### Changing password
Should the password, or the configuration file containing a lightly obscured
form of the password, be compromised, you need to re-encrypt your data with
a new password. Since rclone uses secret-key encryption, where the encryption
key is generated directly from the password kept on the client, it is not
possible to change the password/key of already encrypted content. Just changing
the password configured for an existing crypt remote means you will no longer
able to decrypt any of the previously encrypted content. The only possibility
is to re-upload everything via a crypt remote configured with your new password.
Depending on the size of your data, your bandwith, storage quota etc, there are
different approaches you can take:
- If you have everything in a different location, for example on your local system,
you could remove all of the prior encrypted files, change the password for your
configured crypt remote (or delete and re-create the crypt configuration),
and then re-upload everything from the alternative location.
- If you have enough space on the storage system you can create a new crypt
remote pointing to a separate directory on the same backend, and then use
rclone to copy everything from the original crypt remote to the new,
effectively decrypting everything on the fly using the old password and
re-encrypting using the new password. When done, delete the original crypt
remote directory and finally the rclone crypt configuration with the old password.
All data will be streamed from the storage system and back, so you will
get half the bandwith and be charged twice if you have upload and download quota
on the storage system.
**Note**: A security problem related to the random password generator
was fixed in rclone version 1.53.3 (released 2020-11-19). Passwords generated
by rclone config in version 1.49.0 (released 2019-08-26) to 1.53.2
(released 2020-10-26) are not considered secure and should be changed.
If you made up your own password, or used rclone version older than 1.49.0 or
newer than 1.53.2 to generate it, you are *not* affected by this issue.
See [issue #4783](https://github.com/rclone/rclone/issues/4783) for more
details, and a tool you can use to check if you are affected.
### Example
Create the following file structure using "standard" file name
encryption.
@ -193,7 +326,7 @@ $ rclone -q ls remote:path
55 file1.txt.bin
```
### File name encryption modes ###
### File name encryption modes
Off
@ -223,7 +356,7 @@ segment names.
There is a possibility with some unicode based filenames that the
obfuscation is weak and may map lower case characters to upper case
equivalents.
equivalents.
Obfuscation cannot be relied upon for strong protection.
@ -242,7 +375,8 @@ cloud storage provider.
An alternative, future rclone file name encryption mode may tolerate
backend provider path length limits.
### Directory name encryption ###
### Directory name encryption
Crypt offers the option of encrypting dir names or leaving them intact.
There are two options:
@ -261,7 +395,7 @@ Example:
`1/12/qgm4avr35m5loi1th53ato71v0`
### Modified time and hashes ###
### Modified time and hashes
Crypt stores modification times using the underlying remote so support
depends on that.
@ -432,7 +566,7 @@ Usage Example:
{{< rem autogenerated options stop >}}
## Backing up a crypted remote ##
## Backing up a crypted remote
If you wish to backup a crypted remote, it is recommended that you use
`rclone sync` on the encrypted files, and make sure the passwords are
@ -458,14 +592,14 @@ And to check the integrity you would do
rclone check remote:crypt remote2:crypt
## File formats ##
## File formats
### File encryption ###
### File encryption
Files are encrypted 1:1 source file to destination object. The file
has a header and is divided into chunks.
#### Header ####
#### Header
* 8 bytes magic string `RCLONE\x00\x00`
* 24 bytes Nonce (IV)
@ -477,11 +611,11 @@ The chance of a nonce being re-used is minuscule. If you wrote an
exabyte of data (10¹⁸ bytes) you would have a probability of
approximately 2×10⁻³² of re-using a nonce.
#### Chunk ####
#### Chunk
Each chunk will contain 64kB of data, except for the last one which
may have less data. The data chunk is in standard NACL secretbox
format. Secretbox uses XSalsa20 and Poly1305 to encrypt and
may have less data. The data chunk is in standard NaCl SecretBox
format. SecretBox uses XSalsa20 and Poly1305 to encrypt and
authenticate messages.
Each chunk contains:
@ -496,7 +630,7 @@ buffered in memory so they can't be too big.
This uses a 32 byte (256 bit key) key derived from the user password.
#### Examples ####
#### Examples
1 byte file will encrypt to
@ -513,7 +647,7 @@ This uses a 32 byte (256 bit key) key derived from the user password.
1049120 bytes total (a 0.05% overhead). This is the overhead for big
files.
### Name encryption ###
### Name encryption
File names are encrypted segment by segment - the path is broken up
into `/` separated strings and these are encrypted individually.
@ -547,7 +681,7 @@ encoding is modified in two ways:
`base32` is used rather than the more efficient `base64` so rclone can be
used on case insensitive remotes (e.g. Windows, Amazon Drive).
### Key derivation ###
### Key derivation
Rclone uses `scrypt` with parameters `N=16384, r=8, p=1` with an
optional user supplied salt (password2) to derive the 32+32+16 = 80