2015-09-03 00:37:42 +02:00
|
|
|
|
---
|
|
|
|
|
title: "Overview of cloud storage systems"
|
|
|
|
|
description: "Overview of cloud storage systems"
|
|
|
|
|
type: page
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# Overview of cloud storage systems #
|
|
|
|
|
|
2017-10-28 08:03:51 +02:00
|
|
|
|
Each cloud storage system is slightly different. Rclone attempts to
|
2015-09-03 00:37:42 +02:00
|
|
|
|
provide a unified interface to them, but some underlying differences
|
|
|
|
|
show through.
|
|
|
|
|
|
|
|
|
|
## Features ##
|
|
|
|
|
|
|
|
|
|
Here is an overview of the major features of each cloud storage system.
|
|
|
|
|
|
2022-05-24 12:00:00 +02:00
|
|
|
|
| Name | Hash | ModTime | Case Insensitive | Duplicate Files | MIME Type | Metadata |
|
|
|
|
|
| ---------------------------- |:----------------:|:-------:|:----------------:|:---------------:|:---------:|:--------:|
|
|
|
|
|
| 1Fichier | Whirlpool | - | No | Yes | R | - |
|
|
|
|
|
| Akamai Netstorage | MD5, SHA256 | R/W | No | No | R | - |
|
|
|
|
|
| Amazon Drive | MD5 | - | Yes | No | R | - |
|
|
|
|
|
| Amazon S3 (or S3 compatible) | MD5 | R/W | No | No | R/W | RWU |
|
|
|
|
|
| Backblaze B2 | SHA1 | R/W | No | No | R/W | - |
|
|
|
|
|
| Box | SHA1 | R/W | Yes | No | - | - |
|
|
|
|
|
| Citrix ShareFile | MD5 | R/W | Yes | No | - | - |
|
|
|
|
|
| Dropbox | DBHASH ¹ | R | Yes | No | - | - |
|
|
|
|
|
| Enterprise File Fabric | - | R/W | Yes | No | R/W | - |
|
|
|
|
|
| FTP | - | R/W ¹⁰ | No | No | - | - |
|
|
|
|
|
| Google Cloud Storage | MD5 | R/W | No | No | R/W | - |
|
|
|
|
|
| Google Drive | MD5 | R/W | No | Yes | R/W | - |
|
|
|
|
|
| Google Photos | - | - | No | Yes | R | - |
|
|
|
|
|
| HDFS | - | R/W | No | No | - | - |
|
2022-07-07 19:58:22 +02:00
|
|
|
|
| HiDrive | HiDrive ¹² | R/W | No | No | - | - |
|
2022-05-24 12:00:00 +02:00
|
|
|
|
| HTTP | - | R | No | No | R | - |
|
2022-07-09 00:47:50 +02:00
|
|
|
|
| Internet Archive | MD5, SHA1, CRC32 | R/W ¹¹ | No | No | - | RWU |
|
2022-05-24 12:00:00 +02:00
|
|
|
|
| Jottacloud | MD5 | R/W | Yes | No | R | - |
|
|
|
|
|
| Koofr | MD5 | - | Yes | No | - | - |
|
|
|
|
|
| Mail.ru Cloud | Mailru ⁶ | R/W | Yes | No | - | - |
|
|
|
|
|
| Mega | - | - | No | Yes | - | - |
|
|
|
|
|
| Memory | MD5 | R/W | No | No | - | - |
|
|
|
|
|
| Microsoft Azure Blob Storage | MD5 | R/W | No | No | R/W | - |
|
|
|
|
|
| Microsoft OneDrive | SHA1 ⁵ | R/W | Yes | No | R | - |
|
|
|
|
|
| OpenDrive | MD5 | R/W | Yes | Partial ⁸ | - | - |
|
|
|
|
|
| OpenStack Swift | MD5 | R/W | No | No | R/W | - |
|
2022-10-20 21:24:19 +02:00
|
|
|
|
| Oracle Object Storage | MD5 | R/W | No | No | R/W | - |
|
2022-05-24 12:00:00 +02:00
|
|
|
|
| pCloud | MD5, SHA1 ⁷ | R | No | No | W | - |
|
|
|
|
|
| premiumize.me | - | - | Yes | No | R | - |
|
|
|
|
|
| put.io | CRC-32 | R/W | No | Yes | R | - |
|
|
|
|
|
| QingStor | MD5 | - ⁹ | No | No | R/W | - |
|
|
|
|
|
| Seafile | - | - | No | No | - | - |
|
|
|
|
|
| SFTP | MD5, SHA1 ² | R/W | Depends | No | - | - |
|
|
|
|
|
| Sia | - | - | No | No | - | - |
|
2022-09-30 17:10:57 +02:00
|
|
|
|
| SMB | - | - | Yes | No | - | - |
|
2022-05-24 12:00:00 +02:00
|
|
|
|
| SugarSync | - | - | No | No | - | - |
|
|
|
|
|
| Storj | - | R | No | No | - | - |
|
|
|
|
|
| Uptobox | - | - | No | Yes | - | - |
|
|
|
|
|
| WebDAV | MD5, SHA1 ³ | R ⁴ | Depends | No | - | - |
|
|
|
|
|
| Yandex Disk | MD5 | R/W | No | No | R | - |
|
|
|
|
|
| Zoho WorkDrive | - | - | No | No | - | - |
|
|
|
|
|
| The local filesystem | All | R/W | Depends | No | - | RWU |
|
2016-01-13 11:29:43 +01:00
|
|
|
|
|
2020-10-08 19:14:53 +02:00
|
|
|
|
### Notes
|
2015-09-03 00:37:42 +02:00
|
|
|
|
|
2020-10-08 19:14:53 +02:00
|
|
|
|
¹ Dropbox supports [its own custom
|
2017-05-26 16:09:31 +02:00
|
|
|
|
hash](https://www.dropbox.com/developers/reference/content-hash).
|
2021-03-02 20:11:57 +01:00
|
|
|
|
This is an SHA256 sum of all the 4 MiB block SHA256s.
|
2017-05-26 16:09:31 +02:00
|
|
|
|
|
2020-10-08 19:14:53 +02:00
|
|
|
|
² SFTP supports checksums if the same login has shell access and
|
|
|
|
|
`md5sum` or `sha1sum` as well as `echo` are in the remote's PATH.
|
2017-08-06 12:49:52 +02:00
|
|
|
|
|
2020-10-08 19:14:53 +02:00
|
|
|
|
³ WebDAV supports hashes when used with Owncloud and Nextcloud only.
|
2019-01-27 14:33:21 +01:00
|
|
|
|
|
2020-10-08 19:14:53 +02:00
|
|
|
|
⁴ WebDAV supports modtimes when used with Owncloud and Nextcloud only.
|
2017-10-02 21:29:23 +02:00
|
|
|
|
|
2020-10-08 19:14:53 +02:00
|
|
|
|
⁵ Microsoft OneDrive Personal supports SHA1 hashes, whereas OneDrive
|
2018-04-20 13:55:49 +02:00
|
|
|
|
for business and SharePoint server support Microsoft's own
|
|
|
|
|
[QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash).
|
|
|
|
|
|
2020-10-08 19:14:53 +02:00
|
|
|
|
⁶ Mail.ru uses its own modified SHA1 hash
|
|
|
|
|
|
|
|
|
|
⁷ pCloud only supports SHA1 (not MD5) in its EU region
|
|
|
|
|
|
|
|
|
|
⁸ Opendrive does not support creation of duplicate files using
|
|
|
|
|
their web client interface or other stock clients, but the underlying
|
|
|
|
|
storage platform has been determined to allow duplicate files, and it
|
|
|
|
|
is possible to create them with `rclone`. It may be that this is a
|
|
|
|
|
mistake or an unsupported feature.
|
|
|
|
|
|
2022-06-08 16:29:35 +02:00
|
|
|
|
⁹ QingStor does not support SetModTime for objects bigger than 5 GiB.
|
|
|
|
|
|
|
|
|
|
¹⁰ FTP supports modtimes for the major FTP servers, and also others
|
|
|
|
|
if they advertised required protocol extensions. See [this](/ftp/#modified-time)
|
|
|
|
|
for more details.
|
|
|
|
|
|
|
|
|
|
¹¹ Internet Archive requires option `wait_archive` to be set to a non-zero value
|
|
|
|
|
for full modtime support.
|
|
|
|
|
|
2022-07-07 19:58:22 +02:00
|
|
|
|
¹² HiDrive supports [its own custom
|
|
|
|
|
hash](https://static.hidrive.com/dev/0001).
|
|
|
|
|
It combines SHA1 sums for each 4 KiB block hierarchically to a single
|
|
|
|
|
top-level sum.
|
|
|
|
|
|
2020-10-08 19:14:53 +02:00
|
|
|
|
### Hash ###
|
|
|
|
|
|
|
|
|
|
The cloud storage system supports various hash types of the objects.
|
|
|
|
|
The hashes are used when transferring data as an integrity check and
|
|
|
|
|
can be specifically used with the `--checksum` flag in syncs and in
|
|
|
|
|
the `check` command.
|
|
|
|
|
|
|
|
|
|
To use the verify checksums when transferring between cloud storage
|
|
|
|
|
systems they must support a common hash type.
|
2019-09-09 22:56:16 +02:00
|
|
|
|
|
2015-09-03 00:37:42 +02:00
|
|
|
|
### ModTime ###
|
|
|
|
|
|
2022-08-14 04:56:32 +02:00
|
|
|
|
Almost all cloud storage systems store some sort of timestamp
|
2022-06-08 16:29:35 +02:00
|
|
|
|
on objects, but several of them not something that is appropriate
|
|
|
|
|
to use for syncing. E.g. some backends will only write a timestamp
|
|
|
|
|
that represent the time of the upload. To be relevant for syncing
|
|
|
|
|
it should be able to store the modification time of the source
|
|
|
|
|
object. If this is not the case, rclone will only check the file
|
|
|
|
|
size by default, though can be configured to check the file hash
|
|
|
|
|
(with the `--checksum` flag). Ideally it should also be possible to
|
|
|
|
|
change the timestamp of an existing file without having to re-upload it.
|
|
|
|
|
|
|
|
|
|
Storage systems with a `-` in the ModTime column, means the
|
|
|
|
|
modification read on objects is not the modification time of the
|
|
|
|
|
file when uploaded. It is most likely the time the file was uploaded,
|
|
|
|
|
or possibly something else (like the time the picture was taken in
|
|
|
|
|
Google Photos).
|
|
|
|
|
|
|
|
|
|
Storage systems with a `R` (for read-only) in the ModTime column,
|
|
|
|
|
means the it keeps modification times on objects, and updates them
|
|
|
|
|
when uploading objects, but it does not support changing only the
|
|
|
|
|
modification time (`SetModTime` operation) without re-uploading,
|
|
|
|
|
possibly not even without deleting existing first. Some operations
|
|
|
|
|
in rclone, such as `copy` and `sync` commands, will automatically
|
|
|
|
|
check for `SetModTime` support and re-upload if necessary to keep
|
|
|
|
|
the modification times in sync. Other commands will not work
|
|
|
|
|
without `SetModTime` support, e.g. `touch` command on an existing
|
|
|
|
|
file will fail, and changes to modification time only on a files
|
|
|
|
|
in a `mount` will be silently ignored.
|
|
|
|
|
|
|
|
|
|
Storage systems with `R/W` (for read/write) in the ModTime column,
|
|
|
|
|
means they do also support modtime-only operations.
|
2015-09-03 00:37:42 +02:00
|
|
|
|
|
2015-11-14 12:20:04 +01:00
|
|
|
|
### Case Insensitive ###
|
2015-09-03 00:37:42 +02:00
|
|
|
|
|
|
|
|
|
If a cloud storage systems is case sensitive then it is possible to
|
2020-10-13 23:49:58 +02:00
|
|
|
|
have two files which differ only in case, e.g. `file.txt` and
|
2015-09-03 00:37:42 +02:00
|
|
|
|
`FILE.txt`. If a cloud storage system is case insensitive then that
|
|
|
|
|
isn't possible.
|
|
|
|
|
|
|
|
|
|
This can cause problems when syncing between a case insensitive
|
|
|
|
|
system and a case sensitive system. The symptom of this is that no
|
|
|
|
|
matter how many times you run the sync it never completes fully.
|
|
|
|
|
|
2017-01-31 21:34:11 +01:00
|
|
|
|
The local filesystem and SFTP may or may not be case sensitive
|
|
|
|
|
depending on OS.
|
2015-09-03 00:37:42 +02:00
|
|
|
|
|
2016-01-13 11:29:43 +01:00
|
|
|
|
* Windows - usually case insensitive, though case is preserved
|
2015-09-03 00:37:42 +02:00
|
|
|
|
* OSX - usually case insensitive, though it is possible to format case sensitive
|
2020-10-13 23:49:58 +02:00
|
|
|
|
* Linux - usually case sensitive, but there are case insensitive file systems (e.g. FAT formatted USB keys)
|
2015-09-03 00:37:42 +02:00
|
|
|
|
|
|
|
|
|
Most of the time this doesn't cause any problems as people tend to
|
|
|
|
|
avoid files whose name differs only by case even on case sensitive
|
|
|
|
|
systems.
|
|
|
|
|
|
|
|
|
|
### Duplicate files ###
|
|
|
|
|
|
|
|
|
|
If a cloud storage system allows duplicate files then it can have two
|
|
|
|
|
objects with the same name.
|
|
|
|
|
|
2016-03-22 16:23:37 +01:00
|
|
|
|
This confuses rclone greatly when syncing - use the `rclone dedupe`
|
|
|
|
|
command to rename or remove duplicates.
|
2016-09-12 18:30:37 +02:00
|
|
|
|
|
2019-04-26 15:33:59 +02:00
|
|
|
|
### Restricted filenames ###
|
|
|
|
|
|
|
|
|
|
Some cloud storage systems might have restrictions on the characters
|
|
|
|
|
that are usable in file or directory names.
|
|
|
|
|
When `rclone` detects such a name during a file upload, it will
|
|
|
|
|
transparently replace the restricted characters with similar looking
|
2021-11-01 13:46:23 +01:00
|
|
|
|
Unicode characters. To handle the different sets of restricted characters
|
|
|
|
|
for different backends, rclone uses something it calls [encoding](#encoding).
|
2019-04-26 15:33:59 +02:00
|
|
|
|
|
|
|
|
|
This process is designed to avoid ambiguous file names as much as
|
|
|
|
|
possible and allow to move files between many cloud storage systems
|
|
|
|
|
transparently.
|
|
|
|
|
|
|
|
|
|
The name shown by `rclone` to the user or during log output will only
|
|
|
|
|
contain a minimal set of [replaced characters](#restricted-characters)
|
|
|
|
|
to ensure correct formatting and not necessarily the actual name used
|
|
|
|
|
on the cloud storage.
|
|
|
|
|
|
|
|
|
|
This transformation is reversed when downloading a file or parsing
|
2021-11-01 13:46:23 +01:00
|
|
|
|
`rclone` arguments. For example, when uploading a file named `my file?.txt`
|
|
|
|
|
to Onedrive, it will be displayed as `my file?.txt` on the console, but
|
|
|
|
|
stored as `my file?.txt` to Onedrive (the `?` gets replaced by the similar
|
|
|
|
|
looking `?` character, the so-called "fullwidth question mark").
|
|
|
|
|
The reverse transformation allows to read a file `unusual/name.txt`
|
|
|
|
|
from Google Drive, by passing the name `unusual/name.txt` on the command line
|
|
|
|
|
(the `/` needs to be replaced by the similar looking `/` character).
|
|
|
|
|
|
|
|
|
|
#### Caveats {#restricted-filenames-caveats}
|
|
|
|
|
|
|
|
|
|
The filename encoding system works well in most cases, at least
|
|
|
|
|
where file names are written in English or similar languages.
|
|
|
|
|
You might not even notice it: It just works. In some cases it may
|
|
|
|
|
lead to issues, though. E.g. when file names are written in Chinese,
|
|
|
|
|
or Japanese, where it is always the Unicode fullwidth variants of the
|
|
|
|
|
punctuation marks that are used.
|
|
|
|
|
|
|
|
|
|
On Windows, the characters `:`, `*` and `?` are examples of restricted
|
|
|
|
|
characters. If these are used in filenames on a remote that supports it,
|
|
|
|
|
Rclone will transparently convert them to their fullwidth Unicode
|
|
|
|
|
variants `*`, `?` and `:` when downloading to Windows, and back again
|
|
|
|
|
when uploading. This way files with names that are not allowed on Windows
|
|
|
|
|
can still be stored.
|
|
|
|
|
|
|
|
|
|
However, if you have files on your Windows system originally with these same
|
|
|
|
|
Unicode characters in their names, they will be included in the same conversion
|
|
|
|
|
process. E.g. if you create a file in your Windows filesystem with name
|
|
|
|
|
`Test:1.jpg`, where `:` is the Unicode fullwidth colon symbol, and use
|
|
|
|
|
rclone to upload it to Google Drive, which supports regular `:` (halfwidth
|
|
|
|
|
question mark), rclone will replace the fullwidth `:` with the
|
|
|
|
|
halfwidth `:` and store the file as `Test:1.jpg` in Google Drive. Since
|
|
|
|
|
both Windows and Google Drive allows the name `Test:1.jpg`, it would
|
|
|
|
|
probably be better if rclone just kept the name as is in this case.
|
|
|
|
|
|
|
|
|
|
With the opposite situation; if you have a file named `Test:1.jpg`,
|
|
|
|
|
in your Google Drive, e.g. uploaded from a Linux system where `:` is valid
|
|
|
|
|
in file names. Then later use rclone to copy this file to your Windows
|
|
|
|
|
computer you will notice that on your local disk it gets renamed
|
|
|
|
|
to `Test:1.jpg`. The original filename is not legal on Windows, due to
|
|
|
|
|
the `:`, and rclone therefore renames it to make the copy possible.
|
|
|
|
|
That is all good. However, this can also lead to an issue: If you already
|
|
|
|
|
had a *different* file named `Test:1.jpg` on Windows, and then use rclone
|
|
|
|
|
to copy either way. Rclone will then treat the file originally named
|
|
|
|
|
`Test:1.jpg` on Google Drive and the file originally named `Test:1.jpg`
|
|
|
|
|
on Windows as the same file, and replace the contents from one with the other.
|
|
|
|
|
|
|
|
|
|
Its virtually impossible to handle all cases like these correctly in all
|
|
|
|
|
situations, but by customizing the [encoding option](#encoding), changing the
|
|
|
|
|
set of characters that rclone should convert, you should be able to
|
|
|
|
|
create a configuration that works well for your specific situation.
|
|
|
|
|
See also the [example](/overview/#encoding-example-windows) below.
|
|
|
|
|
|
|
|
|
|
(Windows was used as an example of a file system with many restricted
|
|
|
|
|
characters, and Google drive a storage system with few.)
|
2019-04-26 15:33:59 +02:00
|
|
|
|
|
|
|
|
|
#### Default restricted characters {#restricted-characters}
|
|
|
|
|
|
|
|
|
|
The table below shows the characters that are replaced by default.
|
|
|
|
|
|
|
|
|
|
When a replacement character is found in a filename, this character
|
|
|
|
|
will be escaped with the `‛` character to avoid ambiguous file names.
|
|
|
|
|
(e.g. a file named `␀.txt` would shown as `‛␀.txt`)
|
|
|
|
|
|
|
|
|
|
Each cloud storage backend can use a different set of characters,
|
|
|
|
|
which will be specified in the documentation for each backend.
|
|
|
|
|
|
|
|
|
|
| Character | Value | Replacement |
|
|
|
|
|
| --------- |:-----:|:-----------:|
|
|
|
|
|
| NUL | 0x00 | ␀ |
|
|
|
|
|
| SOH | 0x01 | ␁ |
|
|
|
|
|
| STX | 0x02 | ␂ |
|
|
|
|
|
| ETX | 0x03 | ␃ |
|
|
|
|
|
| EOT | 0x04 | ␄ |
|
|
|
|
|
| ENQ | 0x05 | ␅ |
|
|
|
|
|
| ACK | 0x06 | ␆ |
|
|
|
|
|
| BEL | 0x07 | ␇ |
|
|
|
|
|
| BS | 0x08 | ␈ |
|
|
|
|
|
| HT | 0x09 | ␉ |
|
|
|
|
|
| LF | 0x0A | ␊ |
|
|
|
|
|
| VT | 0x0B | ␋ |
|
|
|
|
|
| FF | 0x0C | ␌ |
|
|
|
|
|
| CR | 0x0D | ␍ |
|
|
|
|
|
| SO | 0x0E | ␎ |
|
|
|
|
|
| SI | 0x0F | ␏ |
|
|
|
|
|
| DLE | 0x10 | ␐ |
|
|
|
|
|
| DC1 | 0x11 | ␑ |
|
|
|
|
|
| DC2 | 0x12 | ␒ |
|
|
|
|
|
| DC3 | 0x13 | ␓ |
|
|
|
|
|
| DC4 | 0x14 | ␔ |
|
|
|
|
|
| NAK | 0x15 | ␕ |
|
|
|
|
|
| SYN | 0x16 | ␖ |
|
|
|
|
|
| ETB | 0x17 | ␗ |
|
|
|
|
|
| CAN | 0x18 | ␘ |
|
|
|
|
|
| EM | 0x19 | ␙ |
|
|
|
|
|
| SUB | 0x1A | ␚ |
|
|
|
|
|
| ESC | 0x1B | ␛ |
|
|
|
|
|
| FS | 0x1C | ␜ |
|
|
|
|
|
| GS | 0x1D | ␝ |
|
|
|
|
|
| RS | 0x1E | ␞ |
|
|
|
|
|
| US | 0x1F | ␟ |
|
|
|
|
|
| / | 0x2F | / |
|
|
|
|
|
| DEL | 0x7F | ␡ |
|
|
|
|
|
|
2019-04-30 13:01:19 +02:00
|
|
|
|
The default encoding will also encode these file names as they are
|
|
|
|
|
problematic with many cloud storage systems.
|
|
|
|
|
|
|
|
|
|
| File name | Replacement |
|
|
|
|
|
| --------- |:-----------:|
|
|
|
|
|
| . | . |
|
|
|
|
|
| .. | .. |
|
|
|
|
|
|
2019-04-26 15:33:59 +02:00
|
|
|
|
#### Invalid UTF-8 bytes {#invalid-utf8}
|
|
|
|
|
|
|
|
|
|
Some backends only support a sequence of well formed UTF-8 bytes
|
|
|
|
|
as file or directory names.
|
|
|
|
|
|
|
|
|
|
In this case all invalid UTF-8 bytes will be replaced with a quoted
|
|
|
|
|
representation of the byte value to allow uploading a file to such a
|
|
|
|
|
backend. For example, the invalid byte `0xFE` will be encoded as `‛FE`.
|
|
|
|
|
|
|
|
|
|
A common source of invalid UTF-8 bytes are local filesystems, that store
|
|
|
|
|
names in a different encoding than UTF-8 or UTF-16, like latin1. See the
|
|
|
|
|
[local filenames](/local/#filenames) section for details.
|
|
|
|
|
|
2020-01-15 14:08:50 +01:00
|
|
|
|
#### Encoding option {#encoding}
|
|
|
|
|
|
2021-11-01 13:46:23 +01:00
|
|
|
|
Most backends have an encoding option, specified as a flag
|
2020-01-15 14:08:50 +01:00
|
|
|
|
`--backend-encoding` where `backend` is the name of the backend, or as
|
|
|
|
|
a config parameter `encoding` (you'll need to select the Advanced
|
|
|
|
|
config in `rclone config` to see it).
|
|
|
|
|
|
|
|
|
|
This will have default value which encodes and decodes characters in
|
|
|
|
|
such a way as to preserve the maximum number of characters (see
|
|
|
|
|
above).
|
|
|
|
|
|
|
|
|
|
However this can be incorrect in some scenarios, for example if you
|
2021-11-01 13:46:23 +01:00
|
|
|
|
have a Windows file system with Unicode fullwidth characters
|
|
|
|
|
`*`, `?` or `:`, that you want to remain as those characters on the
|
|
|
|
|
remote rather than being translated to regular (halfwidth) `*`, `?` and `:`.
|
2020-01-15 14:08:50 +01:00
|
|
|
|
|
|
|
|
|
The `--backend-encoding` flags allow you to change that. You can
|
|
|
|
|
disable the encoding completely with `--backend-encoding None` or set
|
|
|
|
|
`encoding = None` in the config file.
|
|
|
|
|
|
|
|
|
|
Encoding takes a comma separated list of encodings. You can see the
|
2021-11-01 13:46:23 +01:00
|
|
|
|
list of all possible values by passing an invalid value to this
|
|
|
|
|
flag, e.g. `--local-encoding "help"`. The command `rclone help flags encoding`
|
2020-01-15 14:08:50 +01:00
|
|
|
|
will show you the defaults for the backends.
|
|
|
|
|
|
2022-04-21 13:22:04 +02:00
|
|
|
|
| Encoding | Characters | Encoded as |
|
|
|
|
|
| --------- | ---------- | ---------- |
|
|
|
|
|
| Asterisk | `*` | `*` |
|
|
|
|
|
| BackQuote | `` ` `` | ``` |
|
|
|
|
|
| BackSlash | `\` | `\` |
|
|
|
|
|
| Colon | `:` | `:` |
|
|
|
|
|
| CrLf | CR 0x0D, LF 0x0A | `␍`, `␊` |
|
|
|
|
|
| Ctl | All control characters 0x00-0x1F | `␀␁␂␃␄␅␆␇␈␉␊␋␌␍␎␏␐␑␒␓␔␕␖␗␘␙␚␛␜␝␞␟` |
|
|
|
|
|
| Del | DEL 0x7F | `␡` |
|
|
|
|
|
| Dollar | `$` | `$` |
|
|
|
|
|
| Dot | `.` or `..` as entire string | `.`, `..` |
|
|
|
|
|
| DoubleQuote | `"` | `"` |
|
|
|
|
|
| Hash | `#` | `#` |
|
|
|
|
|
| InvalidUtf8 | An invalid UTF-8 character (e.g. latin1) | `<60>` |
|
|
|
|
|
| LeftCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the left of a string | `␍`, `␊`, `␉`, `␋` |
|
|
|
|
|
| LeftPeriod | `.` on the left of a string | `.` |
|
|
|
|
|
| LeftSpace | SPACE on the left of a string | `␠` |
|
|
|
|
|
| LeftTilde | `~` on the left of a string | `~` |
|
|
|
|
|
| LtGt | `<`, `>` | `<`, `>` |
|
|
|
|
|
| None | No characters are encoded | |
|
|
|
|
|
| Percent | `%` | `%` |
|
|
|
|
|
| Pipe | \| | `|` |
|
|
|
|
|
| Question | `?` | `?` |
|
|
|
|
|
| RightCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string | `␍`, `␊`, `␉`, `␋` |
|
|
|
|
|
| RightPeriod | `.` on the right of a string | `.` |
|
|
|
|
|
| RightSpace | SPACE on the right of a string | `␠` |
|
|
|
|
|
| Semicolon | `;` | `;` |
|
|
|
|
|
| SingleQuote | `'` | `'` |
|
|
|
|
|
| Slash | `/` | `/` |
|
|
|
|
|
| SquareBracket | `[`, `]` | `[`, `]` |
|
2020-01-15 14:08:50 +01:00
|
|
|
|
|
2021-11-01 13:46:23 +01:00
|
|
|
|
##### Encoding example: FTP
|
|
|
|
|
|
2020-01-15 14:08:50 +01:00
|
|
|
|
To take a specific example, the FTP backend's default encoding is
|
|
|
|
|
|
|
|
|
|
--ftp-encoding "Slash,Del,Ctl,RightSpace,Dot"
|
|
|
|
|
|
|
|
|
|
However, let's say the FTP server is running on Windows and can't have
|
|
|
|
|
any of the invalid Windows characters in file names. You are backing
|
|
|
|
|
up Linux servers to this FTP server which do have those characters in
|
|
|
|
|
file names. So you would add the Windows set which are
|
|
|
|
|
|
|
|
|
|
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot
|
|
|
|
|
|
|
|
|
|
to the existing ones, giving:
|
|
|
|
|
|
|
|
|
|
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace
|
|
|
|
|
|
|
|
|
|
This can be specified using the `--ftp-encoding` flag or using an `encoding` parameter in the config file.
|
|
|
|
|
|
2021-11-01 13:46:23 +01:00
|
|
|
|
##### Encoding example: Windows
|
|
|
|
|
|
|
|
|
|
As a nother example, take a Windows system where there is a file with
|
|
|
|
|
name `Test:1.jpg`, where `:` is the Unicode fullwidth colon symbol.
|
|
|
|
|
When using rclone to copy this to a remote which supports `:`,
|
|
|
|
|
the regular (halfwidth) colon (such as Google Drive), you will notice
|
|
|
|
|
that the file gets renamed to `Test:1.jpg`.
|
|
|
|
|
|
|
|
|
|
To avoid this you can change the set of characters rclone should convert
|
|
|
|
|
for the local filesystem, using command-line argument `--local-encoding`.
|
|
|
|
|
Rclone's default behavior on Windows corresponds to
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
--local-encoding "Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
If you want to use fullwidth characters `:`, `*` and `?` in your filenames
|
|
|
|
|
without rclone changing them when uploading to a remote, then set the same as
|
|
|
|
|
the default value but without `Colon,Question,Asterisk`:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
--local-encoding "Slash,LtGt,DoubleQuote,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Alternatively, you can disable the conversion of any characters with `--local-encoding None`.
|
2020-01-15 14:08:50 +01:00
|
|
|
|
|
2021-11-01 13:46:23 +01:00
|
|
|
|
Instead of using command-line argument `--local-encoding`, you may also set it
|
|
|
|
|
as [environment variable](/docs/#environment-variables) `RCLONE_LOCAL_ENCODING`,
|
|
|
|
|
or [configure](/docs/#configure) a remote of type `local` in your config,
|
|
|
|
|
and set the `encoding` option there.
|
2020-01-15 14:08:50 +01:00
|
|
|
|
|
2021-11-01 13:46:23 +01:00
|
|
|
|
The risk by doing this is that if you have a filename with the regular (halfwidth)
|
|
|
|
|
`:`, `*` and `?` in your cloud storage, and you try to download
|
|
|
|
|
it to your Windows filesystem, this will fail. These characters are not
|
|
|
|
|
valid in filenames on Windows, and you have told rclone not to work around
|
|
|
|
|
this by converting them to valid fullwidth variants.
|
2020-01-15 14:08:50 +01:00
|
|
|
|
|
2016-09-21 23:13:24 +02:00
|
|
|
|
### MIME Type ###
|
|
|
|
|
|
|
|
|
|
MIME types (also known as media types) classify types of documents
|
2020-10-13 23:49:58 +02:00
|
|
|
|
using a simple text classification, e.g. `text/html` or
|
2016-09-21 23:13:24 +02:00
|
|
|
|
`application/pdf`.
|
|
|
|
|
|
|
|
|
|
Some cloud storage systems support reading (`R`) the MIME type of
|
|
|
|
|
objects and some support writing (`W`) the MIME type of objects.
|
|
|
|
|
|
|
|
|
|
The MIME type can be important if you are serving files directly to
|
|
|
|
|
HTTP from the storage system.
|
|
|
|
|
|
|
|
|
|
If you are copying from a remote which supports reading (`R`) to a
|
|
|
|
|
remote which supports writing (`W`) then rclone will preserve the MIME
|
|
|
|
|
types. Otherwise they will be guessed from the extension, or the
|
|
|
|
|
remote itself may assign the MIME type.
|
|
|
|
|
|
2022-05-24 12:00:00 +02:00
|
|
|
|
### Metadata
|
|
|
|
|
|
|
|
|
|
Backends may or may support reading or writing metadata. They may
|
|
|
|
|
support reading and writing system metadata (metadata intrinsic to
|
|
|
|
|
that backend) and/or user metadata (general purpose metadata).
|
|
|
|
|
|
|
|
|
|
The levels of metadata support are
|
|
|
|
|
|
|
|
|
|
| Key | Explanation |
|
|
|
|
|
|-----|-------------|
|
|
|
|
|
| `R` | Read only System Metadata |
|
|
|
|
|
| `RW` | Read and write System Metadata |
|
|
|
|
|
| `RWU` | Read and write System Metadata and read and write User Metadata |
|
|
|
|
|
|
|
|
|
|
See [the metadata docs](/docs/#metadata) for more info.
|
|
|
|
|
|
2016-09-12 18:30:37 +02:00
|
|
|
|
## Optional Features ##
|
|
|
|
|
|
2020-11-23 17:34:27 +01:00
|
|
|
|
All rclone remotes support a base command set. Other features depend
|
2021-11-04 12:50:43 +01:00
|
|
|
|
upon backend-specific capabilities.
|
2016-09-12 18:30:37 +02:00
|
|
|
|
|
2021-03-07 14:12:01 +01:00
|
|
|
|
| Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | LinkSharing | About | EmptyDir |
|
|
|
|
|
| ---------------------------- |:-----:|:----:|:----:|:-------:|:-------:|:-----:|:------------:|:------------:|:-----:|:--------:|
|
|
|
|
|
| 1Fichier | No | Yes | Yes | No | No | No | No | Yes | No | Yes |
|
2022-06-08 20:47:58 +02:00
|
|
|
|
| Akamai Netstorage | Yes | No | No | No | No | Yes | Yes | No | No | Yes |
|
2021-03-07 14:12:01 +01:00
|
|
|
|
| Amazon Drive | Yes | No | Yes | Yes | No | No | No | No | No | Yes |
|
2022-06-08 20:47:58 +02:00
|
|
|
|
| Amazon S3 (or S3 compatible) | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No |
|
2021-03-07 14:12:01 +01:00
|
|
|
|
| Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No |
|
|
|
|
|
| Box | Yes | Yes | Yes | Yes | Yes ‡‡ | No | Yes | Yes | Yes | Yes |
|
|
|
|
|
| Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes |
|
|
|
|
|
| Dropbox | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | Yes |
|
|
|
|
|
| Enterprise File Fabric | Yes | Yes | Yes | Yes | Yes | No | No | No | No | Yes |
|
|
|
|
|
| FTP | No | No | Yes | Yes | No | No | Yes | No | No | Yes |
|
|
|
|
|
| Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No | No | No |
|
|
|
|
|
| Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
|
|
|
|
|
| Google Photos | No | No | No | No | No | No | No | No | No | No |
|
2021-11-11 17:41:43 +01:00
|
|
|
|
| HDFS | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes |
|
2022-07-07 19:58:22 +02:00
|
|
|
|
| HiDrive | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes |
|
2021-03-07 14:12:01 +01:00
|
|
|
|
| HTTP | No | No | No | No | No | No | No | No | No | Yes |
|
2022-04-12 11:38:44 +02:00
|
|
|
|
| Internet Archive | No | Yes | No | No | Yes | Yes | No | Yes | Yes | No |
|
2021-03-07 14:12:01 +01:00
|
|
|
|
| Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |
|
2022-06-08 20:47:58 +02:00
|
|
|
|
| Koofr | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | Yes |
|
2021-03-07 14:12:01 +01:00
|
|
|
|
| Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
|
|
|
|
|
| Mega | Yes | No | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
|
2021-11-04 12:50:43 +01:00
|
|
|
|
| Memory | No | Yes | No | No | No | Yes | Yes | No | No | No |
|
2021-03-07 14:12:01 +01:00
|
|
|
|
| Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | No | No | No |
|
|
|
|
|
| Microsoft OneDrive | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
|
|
|
|
|
| OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes |
|
|
|
|
|
| OpenStack Swift | Yes † | Yes | No | No | No | Yes | Yes | No | Yes | No |
|
2022-10-28 20:32:17 +02:00
|
|
|
|
| Oracle Object Storage | No | Yes | No | No | Yes | Yes | Yes | No | No | No |
|
2021-03-07 14:12:01 +01:00
|
|
|
|
| pCloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
|
|
|
|
|
| premiumize.me | Yes | No | Yes | Yes | No | No | No | Yes | Yes | Yes |
|
|
|
|
|
| put.io | Yes | No | Yes | Yes | Yes | No | Yes | No | Yes | Yes |
|
|
|
|
|
| QingStor | No | Yes | No | No | Yes | Yes | No | No | No | No |
|
|
|
|
|
| Seafile | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
|
|
|
|
|
| SFTP | No | No | Yes | Yes | No | No | Yes | No | Yes | Yes |
|
2022-06-08 20:47:58 +02:00
|
|
|
|
| Sia | No | No | No | No | No | No | Yes | No | No | Yes |
|
2022-09-30 17:10:57 +02:00
|
|
|
|
| SMB | No | No | Yes | Yes | No | No | Yes | No | No | Yes |
|
2021-03-07 14:12:01 +01:00
|
|
|
|
| SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes |
|
2022-03-14 16:44:44 +01:00
|
|
|
|
| Storj | Yes † | No | Yes | No | No | Yes | Yes | No | No | No |
|
2021-04-09 16:57:24 +02:00
|
|
|
|
| Uptobox | No | Yes | Yes | Yes | No | No | No | No | No | No |
|
2021-03-07 14:12:01 +01:00
|
|
|
|
| WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ‡ | No | Yes | Yes |
|
|
|
|
|
| Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes |
|
|
|
|
|
| Zoho WorkDrive | Yes | Yes | Yes | Yes | No | No | No | No | Yes | Yes |
|
|
|
|
|
| The local filesystem | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes |
|
2016-09-12 18:30:37 +02:00
|
|
|
|
|
|
|
|
|
### Purge ###
|
|
|
|
|
|
|
|
|
|
This deletes a directory quicker than just deleting all the files in
|
|
|
|
|
the directory.
|
|
|
|
|
|
2022-10-04 18:10:00 +02:00
|
|
|
|
† Note Swift and Storj implement this in order to delete
|
2020-05-11 20:56:41 +02:00
|
|
|
|
directory markers but they don't actually have a quicker way of deleting
|
|
|
|
|
files other than deleting them individually.
|
2016-09-12 18:30:37 +02:00
|
|
|
|
|
2017-10-02 21:29:23 +02:00
|
|
|
|
‡ StreamUpload is not supported with Nextcloud
|
|
|
|
|
|
2016-09-12 18:30:37 +02:00
|
|
|
|
### Copy ###
|
|
|
|
|
|
|
|
|
|
Used when copying an object to and from the same remote. This known
|
2020-10-13 23:43:40 +02:00
|
|
|
|
as a server-side copy so you can copy a file without downloading it
|
2016-09-12 18:30:37 +02:00
|
|
|
|
and uploading it again. It is used if you use `rclone copy` or
|
|
|
|
|
`rclone move` if the remote doesn't support `Move` directly.
|
|
|
|
|
|
|
|
|
|
If the server doesn't support `Copy` directly then for copy operations
|
|
|
|
|
the file is downloaded then re-uploaded.
|
|
|
|
|
|
|
|
|
|
### Move ###
|
|
|
|
|
|
|
|
|
|
Used when moving/renaming an object on the same remote. This is known
|
2020-10-13 23:43:40 +02:00
|
|
|
|
as a server-side move of a file. This is used in `rclone move` if the
|
2016-09-12 18:30:37 +02:00
|
|
|
|
server doesn't support `DirMove`.
|
|
|
|
|
|
|
|
|
|
If the server isn't capable of `Move` then rclone simulates it with
|
|
|
|
|
`Copy` then delete. If the server doesn't support `Copy` then rclone
|
|
|
|
|
will download the file and re-upload it.
|
|
|
|
|
|
|
|
|
|
### DirMove ###
|
|
|
|
|
|
|
|
|
|
This is used to implement `rclone move` to move a directory if
|
|
|
|
|
possible. If it isn't then it will use `Move` on each file (which
|
|
|
|
|
falls back to `Copy` then download and upload - see `Move` section).
|
|
|
|
|
|
|
|
|
|
### CleanUp ###
|
|
|
|
|
|
|
|
|
|
This is used for emptying the trash for a remote by `rclone cleanup`.
|
|
|
|
|
|
|
|
|
|
If the server can't do `CleanUp` then `rclone cleanup` will return an
|
|
|
|
|
error.
|
2017-06-06 17:40:00 +02:00
|
|
|
|
|
2020-06-17 14:13:50 +02:00
|
|
|
|
‡‡ Note that while Box implements this it has to delete every file
|
Spelling fixes
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-09 02:17:24 +02:00
|
|
|
|
individually so it will be slower than emptying the trash via the WebUI
|
2020-06-17 14:13:50 +02:00
|
|
|
|
|
2017-06-06 17:40:00 +02:00
|
|
|
|
### ListR ###
|
|
|
|
|
|
|
|
|
|
The remote supports a recursive list to list all the contents beneath
|
|
|
|
|
a directory quickly. This enables the `--fast-list` flag to work.
|
|
|
|
|
See the [rclone docs](/docs/#fast-list) for more details.
|
2017-08-19 14:35:17 +02:00
|
|
|
|
|
|
|
|
|
### StreamUpload ###
|
|
|
|
|
|
|
|
|
|
Some remotes allow files to be uploaded without knowing the file size
|
|
|
|
|
in advance. This allows certain operations to work without spooling the
|
|
|
|
|
file to local disk first, e.g. `rclone rcat`.
|
2018-03-29 09:10:19 +02:00
|
|
|
|
|
|
|
|
|
### LinkSharing ###
|
|
|
|
|
|
|
|
|
|
Sets the necessary permissions on a file or folder and prints a link
|
|
|
|
|
that allows others to access them, even if they don't have an account
|
|
|
|
|
on the particular cloud provider.
|
2018-02-09 21:48:32 +01:00
|
|
|
|
|
|
|
|
|
### About ###
|
|
|
|
|
|
2020-11-23 17:34:27 +01:00
|
|
|
|
Rclone `about` prints quota information for a remote. Typical output
|
|
|
|
|
includes bytes used, free, quota and in trash.
|
2018-02-09 21:48:32 +01:00
|
|
|
|
|
2020-11-23 17:34:27 +01:00
|
|
|
|
If a remote lacks about capability `rclone about remote:`returns
|
|
|
|
|
an error.
|
2019-01-27 17:01:42 +01:00
|
|
|
|
|
2020-11-23 17:34:27 +01:00
|
|
|
|
Backends without about capability cannot determine free space for an
|
|
|
|
|
rclone mount, or use policy `mfs` (most free space) as a member of an
|
|
|
|
|
rclone union remote.
|
2019-08-13 22:25:12 +02:00
|
|
|
|
|
2020-11-27 15:08:52 +01:00
|
|
|
|
See [rclone about command](https://rclone.org/commands/rclone_about/)
|
|
|
|
|
|
2019-08-13 22:25:12 +02:00
|
|
|
|
### EmptyDir ###
|
|
|
|
|
|
|
|
|
|
The remote supports empty directories. See [Limitations](/bugs/#limitations)
|
2021-11-04 12:50:43 +01:00
|
|
|
|
for details. Most Object/Bucket-based remotes do not support this.
|