mirror of
https://github.com/rclone/rclone.git
synced 2024-11-26 02:14:42 +01:00
60236 lines
1.6 MiB
Generated
60236 lines
1.6 MiB
Generated
.\"t
|
|
.\" Automatically generated by Pandoc 2.9.2.1
|
|
.\"
|
|
.TH "rclone" "1" "Mar 18, 2022" "User Manual" ""
|
|
.hy
|
|
.SH Rclone syncs your files to cloud storage
|
|
.PP
|
|
.IP \[bu] 2
|
|
About rclone
|
|
.IP \[bu] 2
|
|
What can rclone do for you?
|
|
.IP \[bu] 2
|
|
What features does rclone have?
|
|
.IP \[bu] 2
|
|
What providers does rclone support?
|
|
.IP \[bu] 2
|
|
Download (https://rclone.org/downloads/)
|
|
.IP \[bu] 2
|
|
Install (https://rclone.org/install/)
|
|
.IP \[bu] 2
|
|
Donate. (https://rclone.org/donate/)
|
|
.SS About rclone
|
|
.PP
|
|
Rclone is a command-line program to manage files on cloud storage.
|
|
It is a feature-rich alternative to cloud vendors\[aq] web storage
|
|
interfaces.
|
|
Over 40 cloud storage products support rclone including S3 object
|
|
stores, business & consumer file storage services, as well as standard
|
|
transfer protocols.
|
|
.PP
|
|
Rclone has powerful cloud equivalents to the unix commands rsync, cp,
|
|
mv, mount, ls, ncdu, tree, rm, and cat.
|
|
Rclone\[aq]s familiar syntax includes shell pipeline support, and
|
|
\f[C]--dry-run\f[R] protection.
|
|
It is used at the command line, in scripts or via its API.
|
|
.PP
|
|
Users call rclone \f[I]\[dq]The Swiss army knife of cloud
|
|
storage\[dq]\f[R], and \f[I]\[dq]Technology indistinguishable from
|
|
magic\[dq]\f[R].
|
|
.PP
|
|
Rclone really looks after your data.
|
|
It preserves timestamps and verifies checksums at all times.
|
|
Transfers over limited bandwidth; intermittent connections, or subject
|
|
to quota can be restarted, from the last good file transferred.
|
|
You can check (https://rclone.org/commands/rclone_check/) the integrity
|
|
of your files.
|
|
Where possible, rclone employs server-side transfers to minimise local
|
|
bandwidth use and transfers from one provider to another without using
|
|
local disk.
|
|
.PP
|
|
Virtual backends wrap local and cloud file systems to apply
|
|
encryption (https://rclone.org/crypt/),
|
|
compression (https://rclone.org/compress/),
|
|
chunking (https://rclone.org/chunker/),
|
|
hashing (https://rclone.org/hasher/) and
|
|
joining (https://rclone.org/union/).
|
|
.PP
|
|
Rclone mounts (https://rclone.org/commands/rclone_mount/) any local,
|
|
cloud or virtual filesystem as a disk on Windows, macOS, linux and
|
|
FreeBSD, and also serves these over
|
|
SFTP (https://rclone.org/commands/rclone_serve_sftp/),
|
|
HTTP (https://rclone.org/commands/rclone_serve_http/),
|
|
WebDAV (https://rclone.org/commands/rclone_serve_webdav/),
|
|
FTP (https://rclone.org/commands/rclone_serve_ftp/) and
|
|
DLNA (https://rclone.org/commands/rclone_serve_dlna/).
|
|
.PP
|
|
Rclone is mature, open-source software originally inspired by rsync and
|
|
written in Go (https://golang.org).
|
|
The friendly support community is familiar with varied use cases.
|
|
Official Ubuntu, Debian, Fedora, Brew and Chocolatey repos.
|
|
include rclone.
|
|
For the latest version downloading from
|
|
rclone.org (https://rclone.org/downloads/) is recommended.
|
|
.PP
|
|
Rclone is widely used on Linux, Windows and Mac.
|
|
Third-party developers create innovative backup, restore, GUI and
|
|
business process solutions using the rclone command line or API.
|
|
.PP
|
|
Rclone does the heavy lifting of communicating with cloud storage.
|
|
.SS What can rclone do for you?
|
|
.PP
|
|
Rclone helps you:
|
|
.IP \[bu] 2
|
|
Backup (and encrypt) files to cloud storage
|
|
.IP \[bu] 2
|
|
Restore (and decrypt) files from cloud storage
|
|
.IP \[bu] 2
|
|
Mirror cloud data to other cloud services or locally
|
|
.IP \[bu] 2
|
|
Migrate data to the cloud, or between cloud storage vendors
|
|
.IP \[bu] 2
|
|
Mount multiple, encrypted, cached or diverse cloud storage as a disk
|
|
.IP \[bu] 2
|
|
Analyse and account for data held on cloud storage using
|
|
lsf (https://rclone.org/commands/rclone_lsf/),
|
|
ljson (https://rclone.org/commands/rclone_lsjson/),
|
|
size (https://rclone.org/commands/rclone_size/),
|
|
ncdu (https://rclone.org/commands/rclone_ncdu/)
|
|
.IP \[bu] 2
|
|
Union (https://rclone.org/union/) file systems together to present
|
|
multiple local and/or cloud file systems as one
|
|
.SS Features
|
|
.IP \[bu] 2
|
|
Transfers
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
MD5, SHA1 hashes are checked at all times for file integrity
|
|
.IP \[bu] 2
|
|
Timestamps are preserved on files
|
|
.IP \[bu] 2
|
|
Operations can be restarted at any time
|
|
.IP \[bu] 2
|
|
Can be to and from network, e.g.
|
|
two different cloud providers
|
|
.IP \[bu] 2
|
|
Can use multi-threaded downloads to local disk
|
|
.RE
|
|
.IP \[bu] 2
|
|
Copy (https://rclone.org/commands/rclone_copy/) new or changed files to
|
|
cloud storage
|
|
.IP \[bu] 2
|
|
Sync (https://rclone.org/commands/rclone_sync/) (one way) to make a
|
|
directory identical
|
|
.IP \[bu] 2
|
|
Move (https://rclone.org/commands/rclone_move/) files to cloud storage
|
|
deleting the local after verification
|
|
.IP \[bu] 2
|
|
Check (https://rclone.org/commands/rclone_check/) hashes and for
|
|
missing/extra files
|
|
.IP \[bu] 2
|
|
Mount (https://rclone.org/commands/rclone_mount/) your cloud storage as
|
|
a network disk
|
|
.IP \[bu] 2
|
|
Serve (https://rclone.org/commands/rclone_serve/) local or remote files
|
|
over
|
|
HTTP (https://rclone.org/commands/rclone_serve_http/)/WebDav (https://rclone.org/commands/rclone_serve_webdav/)/FTP (https://rclone.org/commands/rclone_serve_ftp/)/SFTP (https://rclone.org/commands/rclone_serve_sftp/)/dlna (https://rclone.org/commands/rclone_serve_dlna/)
|
|
.IP \[bu] 2
|
|
Experimental Web based GUI (https://rclone.org/gui/)
|
|
.SS Supported providers
|
|
.PP
|
|
(There are many others, built on standard protocols such as WebDAV or
|
|
S3, that work out of the box.)
|
|
.IP \[bu] 2
|
|
1Fichier
|
|
.IP \[bu] 2
|
|
Akamai Netstorage
|
|
.IP \[bu] 2
|
|
Alibaba Cloud (Aliyun) Object Storage System (OSS)
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Amazon S3
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Box
|
|
.IP \[bu] 2
|
|
Ceph
|
|
.IP \[bu] 2
|
|
Citrix ShareFile
|
|
.IP \[bu] 2
|
|
C14
|
|
.IP \[bu] 2
|
|
DigitalOcean Spaces
|
|
.IP \[bu] 2
|
|
Digi Storage
|
|
.IP \[bu] 2
|
|
Dreamhost
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Enterprise File Fabric
|
|
.IP \[bu] 2
|
|
FTP
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Google Photos
|
|
.IP \[bu] 2
|
|
HDFS
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.IP \[bu] 2
|
|
Hubic
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.IP \[bu] 2
|
|
IBM COS S3
|
|
.IP \[bu] 2
|
|
Koofr
|
|
.IP \[bu] 2
|
|
Mail.ru Cloud
|
|
.IP \[bu] 2
|
|
Memset Memstore
|
|
.IP \[bu] 2
|
|
Mega
|
|
.IP \[bu] 2
|
|
Memory
|
|
.IP \[bu] 2
|
|
Microsoft Azure Blob Storage
|
|
.IP \[bu] 2
|
|
Microsoft OneDrive
|
|
.IP \[bu] 2
|
|
Minio
|
|
.IP \[bu] 2
|
|
Nextcloud
|
|
.IP \[bu] 2
|
|
OVH
|
|
.IP \[bu] 2
|
|
OpenDrive
|
|
.IP \[bu] 2
|
|
OpenStack Swift
|
|
.IP \[bu] 2
|
|
Oracle Cloud Storage
|
|
.IP \[bu] 2
|
|
ownCloud
|
|
.IP \[bu] 2
|
|
pCloud
|
|
.IP \[bu] 2
|
|
premiumize.me
|
|
.IP \[bu] 2
|
|
put.io
|
|
.IP \[bu] 2
|
|
QingStor
|
|
.IP \[bu] 2
|
|
Rackspace Cloud Files
|
|
.IP \[bu] 2
|
|
rsync.net
|
|
.IP \[bu] 2
|
|
Scaleway
|
|
.IP \[bu] 2
|
|
Seafile
|
|
.IP \[bu] 2
|
|
Seagate Lyve Cloud
|
|
.IP \[bu] 2
|
|
SeaweedFS
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.IP \[bu] 2
|
|
Sia
|
|
.IP \[bu] 2
|
|
StackPath
|
|
.IP \[bu] 2
|
|
Storj
|
|
.IP \[bu] 2
|
|
SugarSync
|
|
.IP \[bu] 2
|
|
Tencent Cloud Object Storage (COS)
|
|
.IP \[bu] 2
|
|
Uptobox
|
|
.IP \[bu] 2
|
|
Wasabi
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.IP \[bu] 2
|
|
Yandex Disk
|
|
.IP \[bu] 2
|
|
Zoho WorkDrive
|
|
.IP \[bu] 2
|
|
The local filesystem
|
|
.PP
|
|
Links
|
|
.IP \[bu] 2
|
|
Home page (https://rclone.org/)
|
|
.IP \[bu] 2
|
|
GitHub project page for source and bug
|
|
tracker (https://github.com/rclone/rclone)
|
|
.IP \[bu] 2
|
|
Rclone Forum (https://forum.rclone.org)
|
|
.IP \[bu] 2
|
|
Downloads (https://rclone.org/downloads/)
|
|
.SH Install
|
|
.PP
|
|
Rclone is a Go program and comes as a single binary file.
|
|
.SS Quickstart
|
|
.IP \[bu] 2
|
|
Download (https://rclone.org/downloads/) the relevant binary.
|
|
.IP \[bu] 2
|
|
Extract the \f[C]rclone\f[R] executable, \f[C]rclone.exe\f[R] on
|
|
Windows, from the archive.
|
|
.IP \[bu] 2
|
|
Run \f[C]rclone config\f[R] to setup.
|
|
See rclone config docs (https://rclone.org/docs/) for more details.
|
|
.IP \[bu] 2
|
|
Optionally configure automatic execution.
|
|
.PP
|
|
See below for some expanded Linux / macOS instructions.
|
|
.PP
|
|
See the usage (https://rclone.org/docs/) docs for how to use rclone, or
|
|
run \f[C]rclone -h\f[R].
|
|
.PP
|
|
Already installed rclone can be easily updated to the latest version
|
|
using the rclone
|
|
selfupdate (https://rclone.org/commands/rclone_selfupdate/) command.
|
|
.SS Script installation
|
|
.PP
|
|
To install rclone on Linux/macOS/BSD systems, run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl https://rclone.org/install.sh | sudo bash
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
For beta installation, run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl https://rclone.org/install.sh | sudo bash -s beta
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that this script checks the version of rclone installed first and
|
|
won\[aq]t re-download if not needed.
|
|
.SS Linux installation from precompiled binary
|
|
.PP
|
|
Fetch and unpack
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
|
|
unzip rclone-current-linux-amd64.zip
|
|
cd rclone-*-linux-amd64
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Copy binary file
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo cp rclone /usr/bin/
|
|
sudo chown root:root /usr/bin/rclone
|
|
sudo chmod 755 /usr/bin/rclone
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Install manpage
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo mkdir -p /usr/local/share/man/man1
|
|
sudo cp rclone.1 /usr/local/share/man/man1/
|
|
sudo mandb
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Run \f[C]rclone config\f[R] to setup.
|
|
See rclone config docs (https://rclone.org/docs/) for more details.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.SS macOS installation with brew
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
brew install rclone
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
NOTE: This version of rclone will not support \f[C]mount\f[R] any more
|
|
(see #5373 (https://github.com/rclone/rclone/issues/5373)).
|
|
If mounting is wanted on macOS, either install a precompiled binary or
|
|
enable the relevant option when installing from source.
|
|
.SS macOS installation from precompiled binary, using curl
|
|
.PP
|
|
To avoid problems with macOS gatekeeper enforcing the binary to be
|
|
signed and notarized it is enough to download with \f[C]curl\f[R].
|
|
.PP
|
|
Download the latest version of rclone.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
cd && curl -O https://downloads.rclone.org/rclone-current-osx-amd64.zip
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Unzip the download and cd to the extracted folder.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
unzip -a rclone-current-osx-amd64.zip && cd rclone-*-osx-amd64
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Move rclone to your $PATH.
|
|
You will be prompted for your password.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo mkdir -p /usr/local/bin
|
|
sudo mv rclone /usr/local/bin/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
(the \f[C]mkdir\f[R] command is safe to run, even if the directory
|
|
already exists).
|
|
.PP
|
|
Remove the leftover files.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
cd .. && rm -rf rclone-*-osx-amd64 rclone-current-osx-amd64.zip
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Run \f[C]rclone config\f[R] to setup.
|
|
See rclone config docs (https://rclone.org/docs/) for more details.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.SS macOS installation from precompiled binary, using a web browser
|
|
.PP
|
|
When downloading a binary with a web browser, the browser will set the
|
|
macOS gatekeeper quarantine attribute.
|
|
Starting from Catalina, when attempting to run \f[C]rclone\f[R], a
|
|
pop-up will appear saying:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\[lq]rclone\[rq] cannot be opened because the developer cannot be verified.
|
|
macOS cannot verify that this app is free from malware.
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The simplest fix is to run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
xattr -d com.apple.quarantine rclone
|
|
\f[R]
|
|
.fi
|
|
.SS Install with docker
|
|
.PP
|
|
The rclone maintains a docker image for
|
|
rclone (https://hub.docker.com/r/rclone/rclone).
|
|
These images are autobuilt by docker hub from the rclone source based on
|
|
a minimal Alpine linux image.
|
|
.PP
|
|
The \f[C]:latest\f[R] tag will always point to the latest stable
|
|
release.
|
|
You can use the \f[C]:beta\f[R] tag to get the latest build from master.
|
|
You can also use version tags, e.g.
|
|
\f[C]:1.49.1\f[R], \f[C]:1.49\f[R] or \f[C]:1\f[R].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ docker pull rclone/rclone:latest
|
|
latest: Pulling from rclone/rclone
|
|
Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11
|
|
\&...
|
|
$ docker run --rm rclone/rclone:latest version
|
|
rclone v1.49.1
|
|
- os/arch: linux/amd64
|
|
- go version: go1.12.9
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
There are a few command line options to consider when starting an rclone
|
|
Docker container from the rclone image.
|
|
.IP \[bu] 2
|
|
You need to mount the host rclone config dir at \f[C]/config/rclone\f[R]
|
|
into the Docker container.
|
|
Due to the fact that rclone updates tokens inside its config file, and
|
|
that the update process involves a file rename, you need to mount the
|
|
whole host rclone config dir, not just the single host rclone config
|
|
file.
|
|
.IP \[bu] 2
|
|
You need to mount a host data dir at \f[C]/data\f[R] into the Docker
|
|
container.
|
|
.IP \[bu] 2
|
|
By default, the rclone binary inside a Docker container runs with UID=0
|
|
(root).
|
|
As a result, all files created in a run will have UID=0.
|
|
If your config and data files reside on the host with a non-root
|
|
UID:GID, you need to pass these on the container start command line.
|
|
.IP \[bu] 2
|
|
If you want to access the RC interface (either via the API or the Web
|
|
UI), it is required to set the \f[C]--rc-addr\f[R] to \f[C]:5572\f[R] in
|
|
order to connect to it from outside the container.
|
|
An explanation about why this is necessary is present
|
|
here (https://web.archive.org/web/20200808071950/https://pythonspeed.com/articles/docker-connection-refused/).
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
NOTE: Users running this container with the docker network set to
|
|
\f[C]host\f[R] should probably set it to listen to localhost only, with
|
|
\f[C]127.0.0.1:5572\f[R] as the value for \f[C]--rc-addr\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
It is possible to use \f[C]rclone mount\f[R] inside a userspace Docker
|
|
container, and expose the resulting fuse mount to the host.
|
|
The exact \f[C]docker run\f[R] options to do that might vary slightly
|
|
between hosts.
|
|
See, e.g.
|
|
the discussion in this
|
|
thread (https://github.com/moby/moby/issues/9448).
|
|
.RS 2
|
|
.PP
|
|
You also need to mount the host \f[C]/etc/passwd\f[R] and
|
|
\f[C]/etc/group\f[R] for fuse to work inside the container.
|
|
.RE
|
|
.PP
|
|
Here are some commands tested on an Ubuntu 18.04.3 host:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
# config on host at \[ti]/.config/rclone/rclone.conf
|
|
# data on host at \[ti]/data
|
|
|
|
# make sure the config is ok by listing the remotes
|
|
docker run --rm \[rs]
|
|
--volume \[ti]/.config/rclone:/config/rclone \[rs]
|
|
--volume \[ti]/data:/data:shared \[rs]
|
|
--user $(id -u):$(id -g) \[rs]
|
|
rclone/rclone \[rs]
|
|
listremotes
|
|
|
|
# perform mount inside Docker container, expose result to host
|
|
mkdir -p \[ti]/data/mount
|
|
docker run --rm \[rs]
|
|
--volume \[ti]/.config/rclone:/config/rclone \[rs]
|
|
--volume \[ti]/data:/data:shared \[rs]
|
|
--user $(id -u):$(id -g) \[rs]
|
|
--volume /etc/passwd:/etc/passwd:ro --volume /etc/group:/etc/group:ro \[rs]
|
|
--device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor:unconfined \[rs]
|
|
rclone/rclone \[rs]
|
|
mount dropbox:Photos /data/mount &
|
|
ls \[ti]/data/mount
|
|
kill %1
|
|
\f[R]
|
|
.fi
|
|
.SS Install from source
|
|
.PP
|
|
Make sure you have at least Go (https://golang.org/) go1.15 installed.
|
|
Download go (https://golang.org/dl/) if necessary.
|
|
The latest release is recommended.
|
|
Then
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
git clone https://github.com/rclone/rclone.git
|
|
cd rclone
|
|
go build
|
|
# If on macOS and mount is wanted, instead run: make GOTAGS=cmount
|
|
\&./rclone version
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will leave you a checked out version of rclone you can modify and
|
|
send pull requests with.
|
|
If you use \f[C]make\f[R] instead of \f[C]go build\f[R] then the rclone
|
|
build will have the correct version information in it.
|
|
.PP
|
|
You can also build the latest stable rclone with:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
go get github.com/rclone/rclone
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
or the latest version (equivalent to the beta) with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
go get github.com/rclone/rclone\[at]master
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
These will build the binary in \f[C]$(go env GOPATH)/bin\f[R]
|
|
(\f[C]\[ti]/go/bin/rclone\f[R] by default) after downloading the source
|
|
to the go module cache.
|
|
Note - do \f[B]not\f[R] use the \f[C]-u\f[R] flag here.
|
|
This causes go to try to update the dependencies that rclone uses and
|
|
sometimes these don\[aq]t work with the current version of rclone.
|
|
.SS Installation with Ansible
|
|
.PP
|
|
This can be done with Stefan Weichinger\[aq]s ansible
|
|
role (https://github.com/stefangweichinger/ansible-rclone).
|
|
.PP
|
|
Instructions
|
|
.IP "1." 3
|
|
\f[C]git clone https://github.com/stefangweichinger/ansible-rclone.git\f[R]
|
|
into your local roles-directory
|
|
.IP "2." 3
|
|
add the role to the hosts you want rclone installed to:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
- hosts: rclone-hosts
|
|
roles:
|
|
- rclone
|
|
\f[R]
|
|
.fi
|
|
.SS Portable installation
|
|
.PP
|
|
As mentioned above (https://rclone.org/install/#quickstart), rclone is
|
|
single executable (\f[C]rclone\f[R], or \f[C]rclone.exe\f[R] on Windows)
|
|
that you can download as a zip archive and extract into a location of
|
|
your choosing.
|
|
When executing different commands, it may create files in different
|
|
locations, such as a configuration file and various temporary files.
|
|
By default the locations for these are according to your operating
|
|
system, e.g.
|
|
configuration file in your user profile directory and temporary files in
|
|
the standard temporary directory, but you can customize all of them,
|
|
e.g.
|
|
to make a completely self-contained, portable installation.
|
|
.PP
|
|
Run the config paths (https://rclone.org/commands/rclone_config_paths/)
|
|
command to see the locations that rclone will use.
|
|
.PP
|
|
To override them set the corresponding options (as command-line
|
|
arguments, or as environment
|
|
variables (https://rclone.org/docs/#environment-variables)): -
|
|
--config (https://rclone.org/docs/#config-config-file) -
|
|
--cache-dir (https://rclone.org/docs/#cache-dir-dir) -
|
|
--temp-dir (https://rclone.org/docs/#temp-dir-dir)
|
|
.SS Autostart
|
|
.PP
|
|
After installing and configuring rclone, as described above, you are
|
|
ready to use rclone as an interactive command line utility.
|
|
If your goal is to perform \f[I]periodic\f[R] operations, such as a
|
|
regular sync (https://rclone.org/commands/rclone_sync/), you will
|
|
probably want to configure your rclone command in your operating
|
|
system\[aq]s scheduler.
|
|
If you need to expose \f[I]service\f[R]-like features, such as remote
|
|
control (https://rclone.org/rc/), GUI (https://rclone.org/gui/),
|
|
serve (https://rclone.org/commands/rclone_serve/) or
|
|
mount (https://rclone.org/commands/rclone_move/), you will often want an
|
|
rclone command always running in the background, and configuring it to
|
|
run in a service infrastructure may be a better option.
|
|
Below are some alternatives on how to achieve this on different
|
|
operating systems.
|
|
.PP
|
|
NOTE: Before setting up autorun it is highly recommended that you have
|
|
tested your command manually from a Command Prompt first.
|
|
.SS Autostart on Windows
|
|
.PP
|
|
The most relevant alternatives for autostart on Windows are: - Run at
|
|
user log on using the Startup folder - Run at user log on, at system
|
|
startup or at schedule using Task Scheduler - Run at system startup
|
|
using Windows service
|
|
.SS Running in background
|
|
.PP
|
|
Rclone is a console application, so if not starting from an existing
|
|
Command Prompt, e.g.
|
|
when starting rclone.exe from a shortcut, it will open a Command Prompt
|
|
window.
|
|
When configuring rclone to run from task scheduler and windows service
|
|
you are able to set it to run hidden in background.
|
|
From rclone version 1.54 you can also make it run hidden from anywhere
|
|
by adding option \f[C]--no-console\f[R] (it may still flash briefly when
|
|
the program starts).
|
|
Since rclone normally writes information and any error messages to the
|
|
console, you must redirect this to a file to be able to see it.
|
|
Rclone has a built-in option \f[C]--log-file\f[R] for that.
|
|
.PP
|
|
Example command to run a sync in background:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
c:\[rs]rclone\[rs]rclone.exe sync c:\[rs]files remote:/files --no-console --log-file c:\[rs]rclone\[rs]logs\[rs]sync_files.txt
|
|
\f[R]
|
|
.fi
|
|
.SS User account
|
|
.PP
|
|
As mentioned in the mount (https://rclone.org/commands/rclone_move/)
|
|
documentation, mounted drives created as Administrator are not visible
|
|
to other accounts, not even the account that was elevated as
|
|
Administrator.
|
|
By running the mount command as the built-in \f[C]SYSTEM\f[R] user
|
|
account, it will create drives accessible for everyone on the system.
|
|
Both scheduled task and Windows service can be used to achieve this.
|
|
.PP
|
|
NOTE: Remember that when rclone runs as the \f[C]SYSTEM\f[R] user, the
|
|
user profile that it sees will not be yours.
|
|
This means that if you normally run rclone with configuration file in
|
|
the default location, to be able to use the same configuration when
|
|
running as the system user you must explicitely tell rclone where to
|
|
find it with the
|
|
\f[C]--config\f[R] (https://rclone.org/docs/#config-config-file) option,
|
|
or else it will look in the system users profile path
|
|
(\f[C]C:\[rs]Windows\[rs]System32\[rs]config\[rs]systemprofile\f[R]).
|
|
To test your command manually from a Command Prompt, you can run it with
|
|
the
|
|
PsExec (https://docs.microsoft.com/en-us/sysinternals/downloads/psexec)
|
|
utility from Microsoft\[aq]s Sysinternals suite, which takes option
|
|
\f[C]-s\f[R] to execute commands as the \f[C]SYSTEM\f[R] user.
|
|
.SS Start from Startup folder
|
|
.PP
|
|
To quickly execute an rclone command you can simply create a standard
|
|
Windows Explorer shortcut for the complete rclone command you want to
|
|
run.
|
|
If you store this shortcut in the special \[dq]Startup\[dq] start-menu
|
|
folder, Windows will automatically run it at login.
|
|
To open this folder in Windows Explorer, enter path
|
|
\f[C]%APPDATA%\[rs]Microsoft\[rs]Windows\[rs]Start Menu\[rs]Programs\[rs]Startup\f[R],
|
|
or
|
|
\f[C]C:\[rs]ProgramData\[rs]Microsoft\[rs]Windows\[rs]Start Menu\[rs]Programs\[rs]StartUp\f[R]
|
|
if you want the command to start for \f[I]every\f[R] user that logs in.
|
|
.PP
|
|
This is the easiest approach to autostarting of rclone, but it offers no
|
|
functionality to set it to run as different user, or to set conditions
|
|
or actions on certain events.
|
|
Setting up a scheduled task as described below will often give you
|
|
better results.
|
|
.SS Start from Task Scheduler
|
|
.PP
|
|
Task Scheduler is an administrative tool built into Windows, and it can
|
|
be used to configure rclone to be started automatically in a highly
|
|
configurable way, e.g.
|
|
periodically on a schedule, on user log on, or at system startup.
|
|
It can run be configured to run as the current user, or for a mount
|
|
command that needs to be available to all users it can run as the
|
|
\f[C]SYSTEM\f[R] user.
|
|
For technical information, see
|
|
https://docs.microsoft.com/windows/win32/taskschd/task-scheduler-start-page.
|
|
.SS Run as service
|
|
.PP
|
|
For running rclone at system startup, you can create a Windows service
|
|
that executes your rclone command, as an alternative to scheduled task
|
|
configured to run at startup.
|
|
.SS Mount command built-in service integration
|
|
.PP
|
|
For mount commands, Rclone has a built-in Windows service integration
|
|
via the third-party WinFsp library it uses.
|
|
Registering as a regular Windows service easy, as you just have to
|
|
execute the built-in PowerShell command \f[C]New-Service\f[R] (requires
|
|
administrative privileges).
|
|
.PP
|
|
Example of a PowerShell command that creates a Windows service for
|
|
mounting some \f[C]remote:/files\f[R] as drive letter \f[C]X:\f[R], for
|
|
\f[I]all\f[R] users (service will be running as the local system
|
|
account):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
New-Service -Name Rclone -BinaryPathName \[aq]c:\[rs]rclone\[rs]rclone.exe mount remote:/files X: --config c:\[rs]rclone\[rs]config\[rs]rclone.conf --log-file c:\[rs]rclone\[rs]logs\[rs]mount.txt\[aq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The WinFsp service
|
|
infrastructure (https://github.com/billziss-gh/winfsp/wiki/WinFsp-Service-Architecture)
|
|
supports incorporating services for file system implementations, such as
|
|
rclone, into its own launcher service, as kind of \[dq]child
|
|
services\[dq].
|
|
This has the additional advantage that it also implements a network
|
|
provider that integrates into Windows standard methods for managing
|
|
network drives.
|
|
This is currently not officially supported by Rclone, but with WinFsp
|
|
version 2019.3 B2 / v1.5B2 or later it should be possible through path
|
|
rewriting as described
|
|
here (https://github.com/rclone/rclone/issues/3340).
|
|
.SS Third-party service integration
|
|
.PP
|
|
To Windows service running any rclone command, the excellent third-party
|
|
utility NSSM (http://nssm.cc), the \[dq]Non-Sucking Service
|
|
Manager\[dq], can be used.
|
|
It includes some advanced features such as adjusting process periority,
|
|
defining process environment variables, redirect to file anything
|
|
written to stdout, and customized response to different exit codes, with
|
|
a GUI to configure everything from (although it can also be used from
|
|
command line ).
|
|
.PP
|
|
There are also several other alternatives.
|
|
To mention one more, WinSW (https://github.com/winsw/winsw),
|
|
\[dq]Windows Service Wrapper\[dq], is worth checking out.
|
|
It requires .NET Framework, but it is preinstalled on newer versions of
|
|
Windows, and it also provides alternative standalone distributions which
|
|
includes necessary runtime (.NET 5).
|
|
WinSW is a command-line only utility, where you have to manually create
|
|
an XML file with service configuration.
|
|
This may be a drawback for some, but it can also be an advantage as it
|
|
is easy to back up and re-use the configuration settings, without having
|
|
go through manual steps in a GUI.
|
|
One thing to note is that by default it does not restart the service on
|
|
error, one have to explicit enable this in the configuration file (via
|
|
the \[dq]onfailure\[dq] parameter).
|
|
.SS Autostart on Linux
|
|
.SS Start as a service
|
|
.PP
|
|
To always run rclone in background, relevant for mount commands etc, you
|
|
can use systemd to set up rclone as a system or user service.
|
|
Running as a system service ensures that it is run at startup even if
|
|
the user it is running as has no active session.
|
|
Running rclone as a user service ensures that it only starts after the
|
|
configured user has logged into the system.
|
|
.SS Run periodically from cron
|
|
.PP
|
|
To run a periodic command, such as a copy/sync, you can set up a cron
|
|
job.
|
|
.SH Usage
|
|
.PP
|
|
Rclone is a command line program to manage files on cloud storage.
|
|
After download (https://rclone.org/downloads/) and install, continue
|
|
here to learn how to use it: Initial configuration, what the basic
|
|
syntax looks like, describes the various subcommands, the various
|
|
options, and more.
|
|
.SS Configure
|
|
.PP
|
|
First, you\[aq]ll need to configure rclone.
|
|
As the object storage systems have quite complicated authentication
|
|
these are kept in a config file.
|
|
(See the \f[C]--config\f[R] entry for how to find the config file and
|
|
choose its location.)
|
|
.PP
|
|
The easiest way to make the config is to run rclone with the config
|
|
option:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the following for detailed instructions for
|
|
.IP \[bu] 2
|
|
1Fichier (https://rclone.org/fichier/)
|
|
.IP \[bu] 2
|
|
Akamai Netstorage (https://rclone.org/netstorage/)
|
|
.IP \[bu] 2
|
|
Alias (https://rclone.org/alias/)
|
|
.IP \[bu] 2
|
|
Amazon Drive (https://rclone.org/amazonclouddrive/)
|
|
.IP \[bu] 2
|
|
Amazon S3 (https://rclone.org/s3/)
|
|
.IP \[bu] 2
|
|
Backblaze B2 (https://rclone.org/b2/)
|
|
.IP \[bu] 2
|
|
Box (https://rclone.org/box/)
|
|
.IP \[bu] 2
|
|
Chunker (https://rclone.org/chunker/) - transparently splits large files
|
|
for other remotes
|
|
.IP \[bu] 2
|
|
Citrix ShareFile (https://rclone.org/sharefile/)
|
|
.IP \[bu] 2
|
|
Compress (https://rclone.org/compress/)
|
|
.IP \[bu] 2
|
|
Crypt (https://rclone.org/crypt/) - to encrypt other remotes
|
|
.IP \[bu] 2
|
|
DigitalOcean Spaces (https://rclone.org/s3/#digitalocean-spaces)
|
|
.IP \[bu] 2
|
|
Digi Storage (https://rclone.org/koofr/#digi-storage)
|
|
.IP \[bu] 2
|
|
Dropbox (https://rclone.org/dropbox/)
|
|
.IP \[bu] 2
|
|
Enterprise File Fabric (https://rclone.org/filefabric/)
|
|
.IP \[bu] 2
|
|
FTP (https://rclone.org/ftp/)
|
|
.IP \[bu] 2
|
|
Google Cloud Storage (https://rclone.org/googlecloudstorage/)
|
|
.IP \[bu] 2
|
|
Google Drive (https://rclone.org/drive/)
|
|
.IP \[bu] 2
|
|
Google Photos (https://rclone.org/googlephotos/)
|
|
.IP \[bu] 2
|
|
Hasher (https://rclone.org/hasher/) - to handle checksums for other
|
|
remotes
|
|
.IP \[bu] 2
|
|
HDFS (https://rclone.org/hdfs/)
|
|
.IP \[bu] 2
|
|
HTTP (https://rclone.org/http/)
|
|
.IP \[bu] 2
|
|
Hubic (https://rclone.org/hubic/)
|
|
.IP \[bu] 2
|
|
Jottacloud (https://rclone.org/jottacloud/)
|
|
.IP \[bu] 2
|
|
Koofr (https://rclone.org/koofr/)
|
|
.IP \[bu] 2
|
|
Mail.ru Cloud (https://rclone.org/mailru/)
|
|
.IP \[bu] 2
|
|
Mega (https://rclone.org/mega/)
|
|
.IP \[bu] 2
|
|
Memory (https://rclone.org/memory/)
|
|
.IP \[bu] 2
|
|
Microsoft Azure Blob Storage (https://rclone.org/azureblob/)
|
|
.IP \[bu] 2
|
|
Microsoft OneDrive (https://rclone.org/onedrive/)
|
|
.IP \[bu] 2
|
|
OpenStack Swift / Rackspace Cloudfiles / Memset
|
|
Memstore (https://rclone.org/swift/)
|
|
.IP \[bu] 2
|
|
OpenDrive (https://rclone.org/opendrive/)
|
|
.IP \[bu] 2
|
|
Pcloud (https://rclone.org/pcloud/)
|
|
.IP \[bu] 2
|
|
premiumize.me (https://rclone.org/premiumizeme/)
|
|
.IP \[bu] 2
|
|
put.io (https://rclone.org/putio/)
|
|
.IP \[bu] 2
|
|
QingStor (https://rclone.org/qingstor/)
|
|
.IP \[bu] 2
|
|
Seafile (https://rclone.org/seafile/)
|
|
.IP \[bu] 2
|
|
SFTP (https://rclone.org/sftp/)
|
|
.IP \[bu] 2
|
|
Sia (https://rclone.org/sia/)
|
|
.IP \[bu] 2
|
|
Storj (https://rclone.org/storj/)
|
|
.IP \[bu] 2
|
|
SugarSync (https://rclone.org/sugarsync/)
|
|
.IP \[bu] 2
|
|
Union (https://rclone.org/union/)
|
|
.IP \[bu] 2
|
|
Uptobox (https://rclone.org/uptobox/)
|
|
.IP \[bu] 2
|
|
WebDAV (https://rclone.org/webdav/)
|
|
.IP \[bu] 2
|
|
Yandex Disk (https://rclone.org/yandex/)
|
|
.IP \[bu] 2
|
|
Zoho WorkDrive (https://rclone.org/zoho/)
|
|
.IP \[bu] 2
|
|
The local filesystem (https://rclone.org/local/)
|
|
.SS Basic syntax
|
|
.PP
|
|
Rclone syncs a directory tree from one storage system to another.
|
|
.PP
|
|
Its syntax is like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Syntax: [options] subcommand <parameters> <parameters...>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Source and destination paths are specified by the name you gave the
|
|
storage system in the config file then the sub path, e.g.
|
|
\[dq]drive:myfolder\[dq] to look at \[dq]myfolder\[dq] in Google drive.
|
|
.PP
|
|
You can define as many storage paths as you like in the config file.
|
|
.PP
|
|
Please use the \f[C]-i\f[R] / \f[C]--interactive\f[R] flag while
|
|
learning rclone to avoid accidental data loss.
|
|
.SS Subcommands
|
|
.PP
|
|
rclone uses a system of subcommands.
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:path # lists a remote
|
|
rclone copy /local/path remote:path # copies /local/path to the remote
|
|
rclone sync -i /local/path remote:path # syncs /local/path to the remote
|
|
\f[R]
|
|
.fi
|
|
.SH rclone config
|
|
.PP
|
|
Enter an interactive configuration session.
|
|
.SS Synopsis
|
|
.PP
|
|
Enter an interactive configuration session where you can setup new
|
|
remotes and manage existing ones.
|
|
You may also set or remove a password to protect your configuration.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.IP \[bu] 2
|
|
rclone config create (https://rclone.org/commands/rclone_config_create/)
|
|
- Create a new remote with name, type and options.
|
|
.IP \[bu] 2
|
|
rclone config delete (https://rclone.org/commands/rclone_config_delete/)
|
|
- Delete an existing remote.
|
|
.IP \[bu] 2
|
|
rclone config
|
|
disconnect (https://rclone.org/commands/rclone_config_disconnect/) -
|
|
Disconnects user from remote
|
|
.IP \[bu] 2
|
|
rclone config dump (https://rclone.org/commands/rclone_config_dump/) -
|
|
Dump the config file as JSON.
|
|
.IP \[bu] 2
|
|
rclone config file (https://rclone.org/commands/rclone_config_file/) -
|
|
Show path of configuration file in use.
|
|
.IP \[bu] 2
|
|
rclone config
|
|
password (https://rclone.org/commands/rclone_config_password/) - Update
|
|
password in an existing remote.
|
|
.IP \[bu] 2
|
|
rclone config paths (https://rclone.org/commands/rclone_config_paths/) -
|
|
Show paths used for configuration, cache, temp etc.
|
|
.IP \[bu] 2
|
|
rclone config
|
|
providers (https://rclone.org/commands/rclone_config_providers/) - List
|
|
in JSON format all the providers and options.
|
|
.IP \[bu] 2
|
|
rclone config
|
|
reconnect (https://rclone.org/commands/rclone_config_reconnect/) -
|
|
Re-authenticates user with remote.
|
|
.IP \[bu] 2
|
|
rclone config show (https://rclone.org/commands/rclone_config_show/) -
|
|
Print (decrypted) config file, or the config for a single remote.
|
|
.IP \[bu] 2
|
|
rclone config touch (https://rclone.org/commands/rclone_config_touch/) -
|
|
Ensure configuration file exists.
|
|
.IP \[bu] 2
|
|
rclone config update (https://rclone.org/commands/rclone_config_update/)
|
|
- Update options in an existing remote.
|
|
.IP \[bu] 2
|
|
rclone config
|
|
userinfo (https://rclone.org/commands/rclone_config_userinfo/) - Prints
|
|
info about logged in user of remote.
|
|
.SH rclone copy
|
|
.PP
|
|
Copy files from source to dest, skipping identical files.
|
|
.SS Synopsis
|
|
.PP
|
|
Copy the source to the destination.
|
|
Does not transfer files that are identical on source and destination,
|
|
testing by size and modification time or MD5SUM.
|
|
Doesn\[aq]t delete files from the destination.
|
|
.PP
|
|
Note that it is always the contents of the directory that is synced, not
|
|
the directory so when source:path is a directory, it\[aq]s the contents
|
|
of source:path that are copied, not the directory name and contents.
|
|
.PP
|
|
If dest:path doesn\[aq]t exist, it is created and the source:path
|
|
contents go there.
|
|
.PP
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy source:sourcepath dest:destpath
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Let\[aq]s say there are two files in sourcepath
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sourcepath/one.txt
|
|
sourcepath/two.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This copies them to
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
destpath/one.txt
|
|
destpath/two.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Not to
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
destpath/sourcepath/one.txt
|
|
destpath/sourcepath/two.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you are familiar with \f[C]rsync\f[R], rclone always works as if you
|
|
had written a trailing \f[C]/\f[R] - meaning \[dq]copy the contents of
|
|
this directory\[dq].
|
|
This applies to all commands and whether you are talking about the
|
|
source or destination.
|
|
.PP
|
|
See the --no-traverse (https://rclone.org/docs/#no-traverse) option for
|
|
controlling whether rclone lists the destination directory or not.
|
|
Supplying this option when copying a small number of files into a large
|
|
destination can speed transfers up greatly.
|
|
.PP
|
|
For example, if you have many files in /path/to/src but only a few of
|
|
them change every day, you can copy all the files which have changed
|
|
recently very efficiently like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --max-age 24h --no-traverse /path/to/src remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]Note\f[R]: Use the \f[C]-P\f[R]/\f[C]--progress\f[R] flag to view
|
|
real-time transfer statistics.
|
|
.PP
|
|
\f[B]Note\f[R]: Use the \f[C]--dry-run\f[R] or the
|
|
\f[C]--interactive\f[R]/\f[C]-i\f[R] flag to test without copying
|
|
anything.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy source:path dest:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--create-empty-src-dirs Create empty source dirs on destination after copy
|
|
-h, --help help for copy
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone sync
|
|
.PP
|
|
Make source and dest identical, modifying destination only.
|
|
.SS Synopsis
|
|
.PP
|
|
Sync the source to the destination, changing the destination only.
|
|
Doesn\[aq]t transfer files that are identical on source and destination,
|
|
testing by size and modification time or MD5SUM.
|
|
Destination is updated to match source, including deleting files if
|
|
necessary (except duplicate objects, see below).
|
|
.PP
|
|
\f[B]Important\f[R]: Since this can cause data loss, test first with the
|
|
\f[C]--dry-run\f[R] or the \f[C]--interactive\f[R]/\f[C]-i\f[R] flag.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i SOURCE remote:DESTINATION
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that files in the destination won\[aq]t be deleted if there were
|
|
any errors at any point.
|
|
Duplicate objects (files with the same name, on those providers that
|
|
support it) are also not yet handled.
|
|
.PP
|
|
It is always the contents of the directory that is synced, not the
|
|
directory so when source:path is a directory, it\[aq]s the contents of
|
|
source:path that are copied, not the directory name and contents.
|
|
See extended explanation in the \f[C]copy\f[R] command above if unsure.
|
|
.PP
|
|
If dest:path doesn\[aq]t exist, it is created and the source:path
|
|
contents go there.
|
|
.PP
|
|
\f[B]Note\f[R]: Use the \f[C]-P\f[R]/\f[C]--progress\f[R] flag to view
|
|
real-time transfer statistics
|
|
.PP
|
|
\f[B]Note\f[R]: Use the \f[C]rclone dedupe\f[R] command to deal with
|
|
\[dq]Duplicate object/directory found in source/destination -
|
|
ignoring\[dq] errors.
|
|
See this forum
|
|
post (https://forum.rclone.org/t/sync-not-clearing-duplicates/14372) for
|
|
more info.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync source:path dest:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--create-empty-src-dirs Create empty source dirs on destination after sync
|
|
-h, --help help for sync
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone move
|
|
.PP
|
|
Move files from source to dest.
|
|
.SS Synopsis
|
|
.PP
|
|
Moves the contents of the source directory to the destination directory.
|
|
Rclone will error if the source and destination overlap and the remote
|
|
does not support a server-side directory move operation.
|
|
.PP
|
|
If no filters are in use and if possible this will server-side move
|
|
\f[C]source:path\f[R] into \f[C]dest:path\f[R].
|
|
After this \f[C]source:path\f[R] will no longer exist.
|
|
.PP
|
|
Otherwise for each file in \f[C]source:path\f[R] selected by the filters
|
|
(if any) this will move it into \f[C]dest:path\f[R].
|
|
If possible a server-side move will be used, otherwise it will copy it
|
|
(server-side if possible) into \f[C]dest:path\f[R] then delete the
|
|
original (if no errors on copy) in \f[C]source:path\f[R].
|
|
.PP
|
|
If you want to delete empty source directories after move, use the
|
|
--delete-empty-src-dirs flag.
|
|
.PP
|
|
See the --no-traverse (https://rclone.org/docs/#no-traverse) option for
|
|
controlling whether rclone lists the destination directory or not.
|
|
Supplying this option when moving a small number of files into a large
|
|
destination can speed transfers up greatly.
|
|
.PP
|
|
\f[B]Important\f[R]: Since this can cause data loss, test first with the
|
|
\f[C]--dry-run\f[R] or the \f[C]--interactive\f[R]/\f[C]-i\f[R] flag.
|
|
.PP
|
|
\f[B]Note\f[R]: Use the \f[C]-P\f[R]/\f[C]--progress\f[R] flag to view
|
|
real-time transfer statistics.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone move source:path dest:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--create-empty-src-dirs Create empty source dirs on destination after move
|
|
--delete-empty-src-dirs Delete empty source dirs after move
|
|
-h, --help help for move
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone delete
|
|
.PP
|
|
Remove the files in path.
|
|
.SS Synopsis
|
|
.PP
|
|
Remove the files in path.
|
|
Unlike \f[C]purge\f[R] it obeys include/exclude filters so can be used
|
|
to selectively delete files.
|
|
.PP
|
|
\f[C]rclone delete\f[R] only deletes files but leaves the directory
|
|
structure alone.
|
|
If you want to delete a directory and all of its contents use the
|
|
\f[C]purge\f[R] command.
|
|
.PP
|
|
If you supply the \f[C]--rmdirs\f[R] flag, it will remove all empty
|
|
directories along with it.
|
|
You can also use the separate command \f[C]rmdir\f[R] or
|
|
\f[C]rmdirs\f[R] to delete empty directories only.
|
|
.PP
|
|
For example, to delete all files bigger than 100 MiB, you may first want
|
|
to check what would be deleted (use either):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone --min-size 100M lsl remote:path
|
|
rclone --dry-run --min-size 100M delete remote:path
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Then proceed with the actual delete:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone --min-size 100M delete remote:path
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
That reads \[dq]delete everything with a minimum size of 100 MiB\[dq],
|
|
hence delete all files bigger than 100 MiB.
|
|
.PP
|
|
\f[B]Important\f[R]: Since this can cause data loss, test first with the
|
|
\f[C]--dry-run\f[R] or the \f[C]--interactive\f[R]/\f[C]-i\f[R] flag.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone delete remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for delete
|
|
--rmdirs rmdirs removes empty directories but leaves root intact
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone purge
|
|
.PP
|
|
Remove the path and all of its contents.
|
|
.SS Synopsis
|
|
.PP
|
|
Remove the path and all of its contents.
|
|
Note that this does not obey include/exclude filters - everything will
|
|
be removed.
|
|
Use the \f[C]delete\f[R] command if you want to selectively delete
|
|
files.
|
|
To delete empty directories only, use command \f[C]rmdir\f[R] or
|
|
\f[C]rmdirs\f[R].
|
|
.PP
|
|
\f[B]Important\f[R]: Since this can cause data loss, test first with the
|
|
\f[C]--dry-run\f[R] or the \f[C]--interactive\f[R]/\f[C]-i\f[R] flag.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone purge remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for purge
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone mkdir
|
|
.PP
|
|
Make the path if it doesn\[aq]t already exist.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for mkdir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone rmdir
|
|
.PP
|
|
Remove the empty directory at path.
|
|
.SS Synopsis
|
|
.PP
|
|
This removes empty directory given by path.
|
|
Will not remove the path if it has any objects in it, not even empty
|
|
subdirectories.
|
|
Use command \f[C]rmdirs\f[R] (or \f[C]delete\f[R] with option
|
|
\f[C]--rmdirs\f[R]) to do that.
|
|
.PP
|
|
To delete a path and any objects in it, use \f[C]purge\f[R] command.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rmdir remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for rmdir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone check
|
|
.PP
|
|
Checks the files in the source and destination match.
|
|
.SS Synopsis
|
|
.PP
|
|
Checks the files in the source and destination match.
|
|
It compares sizes and hashes (MD5 or SHA1) and logs a report of files
|
|
that don\[aq]t match.
|
|
It doesn\[aq]t alter the source or destination.
|
|
.PP
|
|
If you supply the \f[C]--size-only\f[R] flag, it will only compare the
|
|
sizes not the hashes as well.
|
|
Use this for a quick check.
|
|
.PP
|
|
If you supply the \f[C]--download\f[R] flag, it will download the data
|
|
from both remotes and check them against each other on the fly.
|
|
This can be useful for remotes that don\[aq]t support hashes or if you
|
|
really want to check all the data.
|
|
.PP
|
|
If you supply the \f[C]--checkfile HASH\f[R] flag with a valid hash
|
|
name, the \f[C]source:path\f[R] must point to a text file in the SUM
|
|
format.
|
|
.PP
|
|
If you supply the \f[C]--one-way\f[R] flag, it will only check that
|
|
files in the source match the files in the destination, not the other
|
|
way around.
|
|
This means that extra files in the destination that are not in the
|
|
source will not be detected.
|
|
.PP
|
|
The \f[C]--differ\f[R], \f[C]--missing-on-dst\f[R],
|
|
\f[C]--missing-on-src\f[R], \f[C]--match\f[R] and \f[C]--error\f[R]
|
|
flags write paths, one per line, to the file name (or stdout if it is
|
|
\f[C]-\f[R]) supplied.
|
|
What they write is described in the help below.
|
|
For example \f[C]--differ\f[R] will write all paths which are present on
|
|
both the source and destination but different.
|
|
.PP
|
|
The \f[C]--combined\f[R] flag will write a file (or stdout) which
|
|
contains all file paths with a symbol and then a space and then the path
|
|
to tell you what happened to it.
|
|
These are reminiscent of diff files.
|
|
.IP \[bu] 2
|
|
\f[C]= path\f[R] means path was found in source and destination and was
|
|
identical
|
|
.IP \[bu] 2
|
|
\[ga]- path\[ga] means path was missing on the source, so only in the
|
|
destination
|
|
.IP \[bu] 2
|
|
\[ga]+ path\[ga] means path was missing on the destination, so only in
|
|
the source
|
|
.IP \[bu] 2
|
|
\[ga]* path\[ga] means path was present in source and destination but
|
|
different.
|
|
.IP \[bu] 2
|
|
\f[C]! path\f[R] means there was an error reading or hashing the source
|
|
or dest.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone check source:path dest:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-C, --checkfile string Treat source:path as a SUM file with hashes of given type
|
|
--combined string Make a combined report of changes to this file
|
|
--differ string Report all non-matching files to this file
|
|
--download Check by downloading rather than with hash
|
|
--error string Report all files with errors (hashing or reading) to this file
|
|
-h, --help help for check
|
|
--match string Report all matching files to this file
|
|
--missing-on-dst string Report all files missing from the destination to this file
|
|
--missing-on-src string Report all files missing from the source to this file
|
|
--one-way Check one way only, source files must exist on remote
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone ls
|
|
.PP
|
|
List the objects in the path with size and path.
|
|
.SS Synopsis
|
|
.PP
|
|
Lists the objects in the source path to standard output in a human
|
|
readable format with size and path.
|
|
Recurses by default.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone ls swift:bucket
|
|
60295 bevajer5jef
|
|
90613 canole
|
|
94467 diwogej7
|
|
37600 fubuwic
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Any of the filtering options can be applied to this command.
|
|
.PP
|
|
There are several related list commands
|
|
.IP \[bu] 2
|
|
\f[C]ls\f[R] to list size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsl\f[R] to list modification time, size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsd\f[R] to list directories only
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[R] to list objects and directories in easy to parse format
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[R] to list objects and directories in JSON format
|
|
.PP
|
|
\f[C]ls\f[R],\f[C]lsl\f[R],\f[C]lsd\f[R] are designed to be
|
|
human-readable.
|
|
\f[C]lsf\f[R] is designed to be human and machine-readable.
|
|
\f[C]lsjson\f[R] is designed to be machine-readable.
|
|
.PP
|
|
Note that \f[C]ls\f[R] and \f[C]lsl\f[R] recurse by default - use
|
|
\f[C]--max-depth 1\f[R] to stop the recursion.
|
|
.PP
|
|
The other list commands \f[C]lsd\f[R],\f[C]lsf\f[R],\f[C]lsjson\f[R] do
|
|
not recurse by default - use \f[C]-R\f[R] to make them recurse.
|
|
.PP
|
|
Listing a non-existent directory will produce an error except for
|
|
remotes which can\[aq]t have empty directories (e.g.
|
|
s3, swift, or gcs - the bucket-based remotes).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for ls
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone lsd
|
|
.PP
|
|
List all directories/containers/buckets in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
Lists the directories in the source path to standard output.
|
|
Does not recurse by default.
|
|
Use the -R flag to recurse.
|
|
.PP
|
|
This command lists the total size of the directory (if known, -1 if
|
|
not), the modification time (if known, the current time if not), the
|
|
number of objects in the directory (if known, -1 if not) and the name of
|
|
the directory, Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsd swift:
|
|
494000 2018-04-26 08:43:20 10000 10000files
|
|
65 2018-04-26 08:43:20 1 1File
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsd drive:test
|
|
-1 2016-10-17 17:41:53 -1 1000files
|
|
-1 2017-01-03 14:40:54 -1 2500files
|
|
-1 2017-07-08 14:39:28 -1 4000files
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you just want the directory names use \[dq]rclone lsf
|
|
--dirs-only\[dq].
|
|
.PP
|
|
Any of the filtering options can be applied to this command.
|
|
.PP
|
|
There are several related list commands
|
|
.IP \[bu] 2
|
|
\f[C]ls\f[R] to list size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsl\f[R] to list modification time, size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsd\f[R] to list directories only
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[R] to list objects and directories in easy to parse format
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[R] to list objects and directories in JSON format
|
|
.PP
|
|
\f[C]ls\f[R],\f[C]lsl\f[R],\f[C]lsd\f[R] are designed to be
|
|
human-readable.
|
|
\f[C]lsf\f[R] is designed to be human and machine-readable.
|
|
\f[C]lsjson\f[R] is designed to be machine-readable.
|
|
.PP
|
|
Note that \f[C]ls\f[R] and \f[C]lsl\f[R] recurse by default - use
|
|
\f[C]--max-depth 1\f[R] to stop the recursion.
|
|
.PP
|
|
The other list commands \f[C]lsd\f[R],\f[C]lsf\f[R],\f[C]lsjson\f[R] do
|
|
not recurse by default - use \f[C]-R\f[R] to make them recurse.
|
|
.PP
|
|
Listing a non-existent directory will produce an error except for
|
|
remotes which can\[aq]t have empty directories (e.g.
|
|
s3, swift, or gcs - the bucket-based remotes).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for lsd
|
|
-R, --recursive Recurse into the listing
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone lsl
|
|
.PP
|
|
List the objects in path with modification time, size and path.
|
|
.SS Synopsis
|
|
.PP
|
|
Lists the objects in the source path to standard output in a human
|
|
readable format with modification time, size and path.
|
|
Recurses by default.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsl swift:bucket
|
|
60295 2016-06-25 18:55:41.062626927 bevajer5jef
|
|
90613 2016-06-25 18:55:43.302607074 canole
|
|
94467 2016-06-25 18:55:43.046609333 diwogej7
|
|
37600 2016-06-25 18:55:40.814629136 fubuwic
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Any of the filtering options can be applied to this command.
|
|
.PP
|
|
There are several related list commands
|
|
.IP \[bu] 2
|
|
\f[C]ls\f[R] to list size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsl\f[R] to list modification time, size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsd\f[R] to list directories only
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[R] to list objects and directories in easy to parse format
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[R] to list objects and directories in JSON format
|
|
.PP
|
|
\f[C]ls\f[R],\f[C]lsl\f[R],\f[C]lsd\f[R] are designed to be
|
|
human-readable.
|
|
\f[C]lsf\f[R] is designed to be human and machine-readable.
|
|
\f[C]lsjson\f[R] is designed to be machine-readable.
|
|
.PP
|
|
Note that \f[C]ls\f[R] and \f[C]lsl\f[R] recurse by default - use
|
|
\f[C]--max-depth 1\f[R] to stop the recursion.
|
|
.PP
|
|
The other list commands \f[C]lsd\f[R],\f[C]lsf\f[R],\f[C]lsjson\f[R] do
|
|
not recurse by default - use \f[C]-R\f[R] to make them recurse.
|
|
.PP
|
|
Listing a non-existent directory will produce an error except for
|
|
remotes which can\[aq]t have empty directories (e.g.
|
|
s3, swift, or gcs - the bucket-based remotes).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsl remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for lsl
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone md5sum
|
|
.PP
|
|
Produces an md5sum file for all the objects in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
Produces an md5sum file for all the objects in the path.
|
|
This is in the same format as the standard md5sum tool produces.
|
|
.PP
|
|
By default, the hash is requested from the remote.
|
|
If MD5 is not supported by the remote, no hash will be returned.
|
|
With the download flag, the file will be downloaded from the remote and
|
|
hashed locally enabling MD5 for any remote.
|
|
.PP
|
|
This command can also hash data received on standard input (stdin), by
|
|
not passing a remote:path, or by passing a hyphen as remote:path when
|
|
there is data to read (if not, the hypen will be treated literaly, as a
|
|
relative path).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone md5sum remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--base64 Output base64 encoded hashsum
|
|
-C, --checkfile string Validate hashes against a given SUM file instead of printing them
|
|
--download Download the file and hash it locally; if this flag is not specified, the hash is requested from the remote
|
|
-h, --help help for md5sum
|
|
--output-file string Output hashsums to a file rather than the terminal
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone sha1sum
|
|
.PP
|
|
Produces an sha1sum file for all the objects in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
Produces an sha1sum file for all the objects in the path.
|
|
This is in the same format as the standard sha1sum tool produces.
|
|
.PP
|
|
By default, the hash is requested from the remote.
|
|
If SHA-1 is not supported by the remote, no hash will be returned.
|
|
With the download flag, the file will be downloaded from the remote and
|
|
hashed locally enabling SHA-1 for any remote.
|
|
.PP
|
|
This command can also hash data received on standard input (stdin), by
|
|
not passing a remote:path, or by passing a hyphen as remote:path when
|
|
there is data to read (if not, the hypen will be treated literaly, as a
|
|
relative path).
|
|
.PP
|
|
This command can also hash data received on STDIN, if not passing a
|
|
remote:path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sha1sum remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--base64 Output base64 encoded hashsum
|
|
-C, --checkfile string Validate hashes against a given SUM file instead of printing them
|
|
--download Download the file and hash it locally; if this flag is not specified, the hash is requested from the remote
|
|
-h, --help help for sha1sum
|
|
--output-file string Output hashsums to a file rather than the terminal
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone size
|
|
.PP
|
|
Prints the total size and number of objects in remote:path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone size remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for size
|
|
--json Format output as JSON
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone version
|
|
.PP
|
|
Show the version number.
|
|
.SS Synopsis
|
|
.PP
|
|
Show the rclone version number, the go version, the build target OS and
|
|
architecture, the runtime OS and kernel version and bitness, build tags
|
|
and the type of executable (static or dynamic).
|
|
.PP
|
|
For example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone version
|
|
rclone v1.55.0
|
|
- os/version: ubuntu 18.04 (64 bit)
|
|
- os/kernel: 4.15.0-136-generic (x86_64)
|
|
- os/type: linux
|
|
- os/arch: amd64
|
|
- go/version: go1.16
|
|
- go/linking: static
|
|
- go/tags: none
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note: before rclone version 1.55 the os/type and os/arch lines were
|
|
merged, and the \[dq]go/version\[dq] line was tagged as \[dq]go
|
|
version\[dq].
|
|
.PP
|
|
If you supply the --check flag, then it will do an online check to
|
|
compare your version with the latest release and the latest beta.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone version --check
|
|
yours: 1.42.0.6
|
|
latest: 1.42 (released 2018-06-16)
|
|
beta: 1.42.0.5 (released 2018-06-17)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone version --check
|
|
yours: 1.41
|
|
latest: 1.42 (released 2018-06-16)
|
|
upgrade: https://downloads.rclone.org/v1.42
|
|
beta: 1.42.0.5 (released 2018-06-17)
|
|
upgrade: https://beta.rclone.org/v1.42-005-g56e1e820
|
|
\f[R]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone version [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--check Check for new version
|
|
-h, --help help for version
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone cleanup
|
|
.PP
|
|
Clean up the remote if possible.
|
|
.SS Synopsis
|
|
.PP
|
|
Clean up the remote if possible.
|
|
Empty the trash or delete old file versions.
|
|
Not supported by all remotes.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone cleanup remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for cleanup
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone dedupe
|
|
.PP
|
|
Interactively find duplicate filenames and delete/rename them.
|
|
.SS Synopsis
|
|
.PP
|
|
By default \f[C]dedupe\f[R] interactively finds files with duplicate
|
|
names and offers to delete all but one or rename them to be different.
|
|
This is known as deduping by name.
|
|
.PP
|
|
Deduping by name is only useful with a small group of backends (e.g.
|
|
Google Drive, Opendrive) that can have duplicate file names.
|
|
It can be run on wrapping backends (e.g.
|
|
crypt) if they wrap a backend which supports duplicate file names.
|
|
.PP
|
|
However if --by-hash is passed in then dedupe will find files with
|
|
duplicate hashes instead which will work on any backend which supports
|
|
at least one hash.
|
|
This can be used to find files with duplicate content.
|
|
This is known as deduping by hash.
|
|
.PP
|
|
If deduping by name, first rclone will merge directories with the same
|
|
name.
|
|
It will do this iteratively until all the identically named directories
|
|
have been merged.
|
|
.PP
|
|
Next, if deduping by name, for every group of duplicate file names /
|
|
hashes, it will delete all but one identical file it finds without
|
|
confirmation.
|
|
This means that for most duplicated files the \f[C]dedupe\f[R] command
|
|
will not be interactive.
|
|
.PP
|
|
\f[C]dedupe\f[R] considers files to be identical if they have the same
|
|
file path and the same hash.
|
|
If the backend does not support hashes (e.g.
|
|
crypt wrapping Google Drive) then they will never be found to be
|
|
identical.
|
|
If you use the \f[C]--size-only\f[R] flag then files will be considered
|
|
identical if they have the same size (any hash will be ignored).
|
|
This can be useful on crypt backends which do not support hashes.
|
|
.PP
|
|
Next rclone will resolve the remaining duplicates.
|
|
Exactly which action is taken depends on the dedupe mode.
|
|
By default, rclone will interactively query the user for each one.
|
|
.PP
|
|
\f[B]Important\f[R]: Since this can cause data loss, test first with the
|
|
\f[C]--dry-run\f[R] or the \f[C]--interactive\f[R]/\f[C]-i\f[R] flag.
|
|
.PP
|
|
Here is an example run.
|
|
.PP
|
|
Before - with duplicates
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsl drive:dupes
|
|
6048320 2016-03-05 16:23:16.798000000 one.txt
|
|
6048320 2016-03-05 16:23:11.775000000 one.txt
|
|
564374 2016-03-05 16:23:06.731000000 one.txt
|
|
6048320 2016-03-05 16:18:26.092000000 one.txt
|
|
6048320 2016-03-05 16:22:46.185000000 two.txt
|
|
1744073 2016-03-05 16:22:38.104000000 two.txt
|
|
564374 2016-03-05 16:22:52.118000000 two.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Now the \f[C]dedupe\f[R] session
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone dedupe drive:dupes
|
|
2016/03/05 16:24:37 Google drive root \[aq]dupes\[aq]: Looking for duplicates using interactive mode.
|
|
one.txt: Found 4 files with duplicate names
|
|
one.txt: Deleting 2/3 identical duplicates (MD5 \[dq]1eedaa9fe86fd4b8632e2ac549403b36\[dq])
|
|
one.txt: 2 duplicates remain
|
|
1: 6048320 bytes, 2016-03-05 16:23:16.798000000, MD5 1eedaa9fe86fd4b8632e2ac549403b36
|
|
2: 564374 bytes, 2016-03-05 16:23:06.731000000, MD5 7594e7dc9fc28f727c42ee3e0749de81
|
|
s) Skip and do nothing
|
|
k) Keep just one (choose which in next step)
|
|
r) Rename all to be different (by changing file.jpg to file-1.jpg)
|
|
s/k/r> k
|
|
Enter the number of the file to keep> 1
|
|
one.txt: Deleted 1 extra copies
|
|
two.txt: Found 3 files with duplicate names
|
|
two.txt: 3 duplicates remain
|
|
1: 564374 bytes, 2016-03-05 16:22:52.118000000, MD5 7594e7dc9fc28f727c42ee3e0749de81
|
|
2: 6048320 bytes, 2016-03-05 16:22:46.185000000, MD5 1eedaa9fe86fd4b8632e2ac549403b36
|
|
3: 1744073 bytes, 2016-03-05 16:22:38.104000000, MD5 851957f7fb6f0bc4ce76be966d336802
|
|
s) Skip and do nothing
|
|
k) Keep just one (choose which in next step)
|
|
r) Rename all to be different (by changing file.jpg to file-1.jpg)
|
|
s/k/r> r
|
|
two-1.txt: renamed from: two.txt
|
|
two-2.txt: renamed from: two.txt
|
|
two-3.txt: renamed from: two.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The result being
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsl drive:dupes
|
|
6048320 2016-03-05 16:23:16.798000000 one.txt
|
|
564374 2016-03-05 16:22:52.118000000 two-1.txt
|
|
6048320 2016-03-05 16:22:46.185000000 two-2.txt
|
|
1744073 2016-03-05 16:22:38.104000000 two-3.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Dedupe can be run non interactively using the \f[C]--dedupe-mode\f[R]
|
|
flag or by using an extra parameter with the same value
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode interactive\f[R] - interactive as above.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode skip\f[R] - removes identical files then skips
|
|
anything left.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode first\f[R] - removes identical files then keeps the
|
|
first one.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode newest\f[R] - removes identical files then keeps the
|
|
newest one.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode oldest\f[R] - removes identical files then keeps the
|
|
oldest one.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode largest\f[R] - removes identical files then keeps the
|
|
largest one.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode smallest\f[R] - removes identical files then keeps
|
|
the smallest one.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode rename\f[R] - removes identical files then renames
|
|
the rest to be different.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode list\f[R] - lists duplicate dirs and files only and
|
|
changes nothing.
|
|
.PP
|
|
For example, to rename all the identically named photos in your Google
|
|
Photos directory, do
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone dedupe --dedupe-mode rename \[dq]drive:Google Photos\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone dedupe rename \[dq]drive:Google Photos\[dq]
|
|
\f[R]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone dedupe [mode] remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--by-hash Find identical hashes rather than names
|
|
--dedupe-mode string Dedupe mode interactive|skip|first|newest|oldest|largest|smallest|rename (default \[dq]interactive\[dq])
|
|
-h, --help help for dedupe
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone about
|
|
.PP
|
|
Get quota information from the remote.
|
|
.SS Synopsis
|
|
.PP
|
|
\f[C]rclone about\f[R] prints quota information about a remote to
|
|
standard output.
|
|
The output is typically used, free, quota and trash contents.
|
|
.PP
|
|
E.g.
|
|
Typical output from \f[C]rclone about remote:\f[R] is:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Total: 17 GiB
|
|
Used: 7.444 GiB
|
|
Free: 1.315 GiB
|
|
Trashed: 100.000 MiB
|
|
Other: 8.241 GiB
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Where the fields are:
|
|
.IP \[bu] 2
|
|
Total: Total size available.
|
|
.IP \[bu] 2
|
|
Used: Total size used.
|
|
.IP \[bu] 2
|
|
Free: Total space available to this user.
|
|
.IP \[bu] 2
|
|
Trashed: Total space used by trash.
|
|
.IP \[bu] 2
|
|
Other: Total amount in other storage (e.g.
|
|
Gmail, Google Photos).
|
|
.IP \[bu] 2
|
|
Objects: Total number of objects in the storage.
|
|
.PP
|
|
All sizes are in number of bytes.
|
|
.PP
|
|
Applying a \f[C]--full\f[R] flag to the command prints the bytes in
|
|
full, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Total: 18253611008
|
|
Used: 7993453766
|
|
Free: 1411001220
|
|
Trashed: 104857602
|
|
Other: 8849156022
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
A \f[C]--json\f[R] flag generates conveniently machine-readable output,
|
|
e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]total\[dq]: 18253611008,
|
|
\[dq]used\[dq]: 7993453766,
|
|
\[dq]trashed\[dq]: 104857602,
|
|
\[dq]other\[dq]: 8849156022,
|
|
\[dq]free\[dq]: 1411001220
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Not all backends print all fields.
|
|
Information is not included if it is not provided by a backend.
|
|
Where the value is unlimited it is omitted.
|
|
.PP
|
|
Some backends does not support the \f[C]rclone about\f[R] command at
|
|
all, see complete list in
|
|
documentation (https://rclone.org/overview/#optional-features).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone about remote: [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--full Full numbers instead of human-readable
|
|
-h, --help help for about
|
|
--json Format output as JSON
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone authorize
|
|
.PP
|
|
Remote authorization.
|
|
.SS Synopsis
|
|
.PP
|
|
Remote authorization.
|
|
Used to authorize a remote or headless rclone from a machine with a
|
|
browser - use as instructed by rclone config.
|
|
.PP
|
|
Use the --auth-no-open-browser to prevent rclone to open auth link in
|
|
default browser automatically.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone authorize [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--auth-no-open-browser Do not automatically open auth link in default browser
|
|
-h, --help help for authorize
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone backend
|
|
.PP
|
|
Run a backend-specific command.
|
|
.SS Synopsis
|
|
.PP
|
|
This runs a backend-specific command.
|
|
The commands themselves (except for \[dq]help\[dq] and
|
|
\[dq]features\[dq]) are defined by the backends and you should see the
|
|
backend docs for definitions.
|
|
.PP
|
|
You can discover what commands a backend implements by using
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend help remote:
|
|
rclone backend help <backendname>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can also discover information about the backend using (see
|
|
operations/fsinfo (https://rclone.org/rc/#operations-fsinfo) in the
|
|
remote control docs for more info).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend features remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Pass options to the backend command with -o.
|
|
This should be key=value or key, e.g.:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend stats remote:path stats -o format=json -o long
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Pass arguments to the backend by placing them on the end of the line
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend cleanup remote:path file1 file2 file3
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note to run these commands on a running backend then see
|
|
backend/command (https://rclone.org/rc/#backend-command) in the rc docs.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend <command> remote:path [opts] <args> [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for backend
|
|
--json Always output in JSON format
|
|
-o, --option stringArray Option in the form name=value or name
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone bisync
|
|
.PP
|
|
Perform bidirectonal synchronization between two paths.
|
|
.SS Synopsis
|
|
.PP
|
|
Perform bidirectonal synchronization between two paths.
|
|
.PP
|
|
Bisync (https://rclone.org/bisync/) provides a bidirectional cloud sync
|
|
solution in rclone.
|
|
It retains the Path1 and Path2 filesystem listings from the prior run.
|
|
On each successive run it will: - list files on Path1 and Path2, and
|
|
check for changes on each side.
|
|
Changes include \f[C]New\f[R], \f[C]Newer\f[R], \f[C]Older\f[R], and
|
|
\f[C]Deleted\f[R] files.
|
|
- Propagate changes on Path1 to Path2, and vice-versa.
|
|
.PP
|
|
See full bisync description (https://rclone.org/bisync/) for details.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone bisync remote1:path1 remote2:path2 [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--check-access Ensure expected RCLONE_TEST files are found on both Path1 and Path2 filesystems, else abort.
|
|
--check-filename string Filename for --check-access (default: RCLONE_TEST)
|
|
--check-sync string Controls comparison of final listings: true|false|only (default: true) (default \[dq]true\[dq])
|
|
--filters-file string Read filtering patterns from a file
|
|
--force Bypass --max-delete safety check and run the sync. Consider using with --verbose
|
|
-h, --help help for bisync
|
|
--localtime Use local time in listings (default: UTC)
|
|
--no-cleanup Retain working files (useful for troubleshooting and testing).
|
|
--remove-empty-dirs Remove empty directories at the final cleanup step.
|
|
-1, --resync Performs the resync run. Path1 files may overwrite Path2 versions. Consider using --verbose or --dry-run first.
|
|
--workdir string Use custom working dir - useful for testing. (default: $HOME/.cache/rclone/bisync)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone cat
|
|
.PP
|
|
Concatenates any files and sends them to stdout.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone cat sends any files to standard output.
|
|
.PP
|
|
You can use it like this to output a single file
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone cat remote:path/to/file
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or like this to output any file in dir or its subdirectories.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone cat remote:path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or like this to output any .txt files in dir or its subdirectories.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone --include \[dq]*.txt\[dq] cat remote:path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Use the \f[C]--head\f[R] flag to print characters only at the start,
|
|
\f[C]--tail\f[R] for the end and \f[C]--offset\f[R] and
|
|
\f[C]--count\f[R] to print a section in the middle.
|
|
Note that if offset is negative it will count from the end, so
|
|
\f[C]--offset -1 --count 1\f[R] is equivalent to \f[C]--tail 1\f[R].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone cat remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--count int Only print N characters (default -1)
|
|
--discard Discard the output instead of printing
|
|
--head int Only print the first N characters
|
|
-h, --help help for cat
|
|
--offset int Start printing at offset N (or from end if -ve)
|
|
--tail int Only print the last N characters
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone checksum
|
|
.PP
|
|
Checks the files in the source against a SUM file.
|
|
.SS Synopsis
|
|
.PP
|
|
Checks that hashsums of source files match the SUM file.
|
|
It compares hashes (MD5, SHA1, etc) and logs a report of files which
|
|
don\[aq]t match.
|
|
It doesn\[aq]t alter the file system.
|
|
.PP
|
|
If you supply the \f[C]--download\f[R] flag, it will download the data
|
|
from remote and calculate the contents hash on the fly.
|
|
This can be useful for remotes that don\[aq]t support hashes or if you
|
|
really want to check all the data.
|
|
.PP
|
|
Note that hash values in the SUM file are treated as case insensitive.
|
|
.PP
|
|
If you supply the \f[C]--one-way\f[R] flag, it will only check that
|
|
files in the source match the files in the destination, not the other
|
|
way around.
|
|
This means that extra files in the destination that are not in the
|
|
source will not be detected.
|
|
.PP
|
|
The \f[C]--differ\f[R], \f[C]--missing-on-dst\f[R],
|
|
\f[C]--missing-on-src\f[R], \f[C]--match\f[R] and \f[C]--error\f[R]
|
|
flags write paths, one per line, to the file name (or stdout if it is
|
|
\f[C]-\f[R]) supplied.
|
|
What they write is described in the help below.
|
|
For example \f[C]--differ\f[R] will write all paths which are present on
|
|
both the source and destination but different.
|
|
.PP
|
|
The \f[C]--combined\f[R] flag will write a file (or stdout) which
|
|
contains all file paths with a symbol and then a space and then the path
|
|
to tell you what happened to it.
|
|
These are reminiscent of diff files.
|
|
.IP \[bu] 2
|
|
\f[C]= path\f[R] means path was found in source and destination and was
|
|
identical
|
|
.IP \[bu] 2
|
|
\[ga]- path\[ga] means path was missing on the source, so only in the
|
|
destination
|
|
.IP \[bu] 2
|
|
\[ga]+ path\[ga] means path was missing on the destination, so only in
|
|
the source
|
|
.IP \[bu] 2
|
|
\[ga]* path\[ga] means path was present in source and destination but
|
|
different.
|
|
.IP \[bu] 2
|
|
\f[C]! path\f[R] means there was an error reading or hashing the source
|
|
or dest.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone checksum <hash> sumfile src:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--combined string Make a combined report of changes to this file
|
|
--differ string Report all non-matching files to this file
|
|
--download Check by hashing the contents
|
|
--error string Report all files with errors (hashing or reading) to this file
|
|
-h, --help help for checksum
|
|
--match string Report all matching files to this file
|
|
--missing-on-dst string Report all files missing from the destination to this file
|
|
--missing-on-src string Report all files missing from the source to this file
|
|
--one-way Check one way only, source files must exist on remote
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone completion
|
|
.PP
|
|
generate the autocompletion script for the specified shell
|
|
.SS Synopsis
|
|
.PP
|
|
Generate the autocompletion script for rclone for the specified shell.
|
|
See each sub-command\[aq]s help for details on how to use the generated
|
|
script.
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for completion
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.IP \[bu] 2
|
|
rclone completion
|
|
bash (https://rclone.org/commands/rclone_completion_bash/) - generate
|
|
the autocompletion script for bash
|
|
.IP \[bu] 2
|
|
rclone completion
|
|
fish (https://rclone.org/commands/rclone_completion_fish/) - generate
|
|
the autocompletion script for fish
|
|
.IP \[bu] 2
|
|
rclone completion
|
|
powershell (https://rclone.org/commands/rclone_completion_powershell/) -
|
|
generate the autocompletion script for powershell
|
|
.IP \[bu] 2
|
|
rclone completion
|
|
zsh (https://rclone.org/commands/rclone_completion_zsh/) - generate the
|
|
autocompletion script for zsh
|
|
.SH rclone completion bash
|
|
.PP
|
|
generate the autocompletion script for bash
|
|
.SS Synopsis
|
|
.PP
|
|
Generate the autocompletion script for the bash shell.
|
|
.PP
|
|
This script depends on the \[aq]bash-completion\[aq] package.
|
|
If it is not installed already, you can install it via your OS\[aq]s
|
|
package manager.
|
|
.PP
|
|
To load completions in your current shell session: $ source <(rclone
|
|
completion bash)
|
|
.PP
|
|
To load completions for every new session, execute once: Linux: $ rclone
|
|
completion bash > /etc/bash_completion.d/rclone MacOS: $ rclone
|
|
completion bash > /usr/local/etc/bash_completion.d/rclone
|
|
.PP
|
|
You will need to start a new shell for this setup to take effect.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone completion bash
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for bash
|
|
--no-descriptions disable completion descriptions
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone completion (https://rclone.org/commands/rclone_completion/) -
|
|
generate the autocompletion script for the specified shell
|
|
.SH rclone completion fish
|
|
.PP
|
|
generate the autocompletion script for fish
|
|
.SS Synopsis
|
|
.PP
|
|
Generate the autocompletion script for the fish shell.
|
|
.PP
|
|
To load completions in your current shell session: $ rclone completion
|
|
fish | source
|
|
.PP
|
|
To load completions for every new session, execute once: $ rclone
|
|
completion fish > \[ti]/.config/fish/completions/rclone.fish
|
|
.PP
|
|
You will need to start a new shell for this setup to take effect.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone completion fish [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for fish
|
|
--no-descriptions disable completion descriptions
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone completion (https://rclone.org/commands/rclone_completion/) -
|
|
generate the autocompletion script for the specified shell
|
|
.SH rclone completion powershell
|
|
.PP
|
|
generate the autocompletion script for powershell
|
|
.SS Synopsis
|
|
.PP
|
|
Generate the autocompletion script for powershell.
|
|
.PP
|
|
To load completions in your current shell session: PS C:> rclone
|
|
completion powershell | Out-String | Invoke-Expression
|
|
.PP
|
|
To load completions for every new session, add the output of the above
|
|
command to your powershell profile.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone completion powershell [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for powershell
|
|
--no-descriptions disable completion descriptions
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone completion (https://rclone.org/commands/rclone_completion/) -
|
|
generate the autocompletion script for the specified shell
|
|
.SH rclone completion zsh
|
|
.PP
|
|
generate the autocompletion script for zsh
|
|
.SS Synopsis
|
|
.PP
|
|
Generate the autocompletion script for the zsh shell.
|
|
.PP
|
|
If shell completion is not already enabled in your environment you will
|
|
need to enable it.
|
|
You can execute the following once:
|
|
.PP
|
|
$ echo \[dq]autoload -U compinit; compinit\[dq] >> \[ti]/.zshrc
|
|
.PP
|
|
To load completions for every new session, execute once: # Linux: $
|
|
rclone completion zsh > \[dq]${fpath[1]}/_rclone\[dq] # macOS: $ rclone
|
|
completion zsh > /usr/local/share/zsh/site-functions/_rclone
|
|
.PP
|
|
You will need to start a new shell for this setup to take effect.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone completion zsh [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for zsh
|
|
--no-descriptions disable completion descriptions
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone completion (https://rclone.org/commands/rclone_completion/) -
|
|
generate the autocompletion script for the specified shell
|
|
.SH rclone config create
|
|
.PP
|
|
Create a new remote with name, type and options.
|
|
.SS Synopsis
|
|
.PP
|
|
Create a new remote of \f[C]name\f[R] with \f[C]type\f[R] and options.
|
|
The options should be passed in pairs of \f[C]key\f[R] \f[C]value\f[R]
|
|
or as \f[C]key=value\f[R].
|
|
.PP
|
|
For example, to make a swift remote of name myremote using auto config
|
|
you would do:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config create myremote swift env_auth true
|
|
rclone config create myremote swift env_auth=true
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
So for example if you wanted to configure a Google Drive remote but
|
|
using remote authorization you would do this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config create mydrive drive config_is_local=false
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that if the config process would normally ask a question the
|
|
default is taken (unless \f[C]--non-interactive\f[R] is used).
|
|
Each time that happens rclone will print or DEBUG a message saying how
|
|
to affect the value taken.
|
|
.PP
|
|
If any of the parameters passed is a password field, then rclone will
|
|
automatically obscure them if they aren\[aq]t already obscured before
|
|
putting them in the config file.
|
|
.PP
|
|
\f[B]NB\f[R] If the password parameter is 22 characters or longer and
|
|
consists only of base64 characters then rclone can get confused about
|
|
whether the password is already obscured or not and put unobscured
|
|
passwords into the config file.
|
|
If you want to be 100% certain that the passwords get obscured then use
|
|
the \f[C]--obscure\f[R] flag, or if you are 100% certain you are already
|
|
passing obscured passwords then use \f[C]--no-obscure\f[R].
|
|
You can also set obscured passwords using the
|
|
\f[C]rclone config password\f[R] command.
|
|
.PP
|
|
The flag \f[C]--non-interactive\f[R] is for use by applications that
|
|
wish to configure rclone themeselves, rather than using rclone\[aq]s
|
|
text based configuration questions.
|
|
If this flag is set, and rclone needs to ask the user a question, a JSON
|
|
blob will be returned with the question in it.
|
|
.PP
|
|
This will look something like (some irrelevant detail removed):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]State\[dq]: \[dq]*oauth-islocal,teamdrive,,\[dq],
|
|
\[dq]Option\[dq]: {
|
|
\[dq]Name\[dq]: \[dq]config_is_local\[dq],
|
|
\[dq]Help\[dq]: \[dq]Use auto config?\[rs]n * Say Y if not sure\[rs]n * Say N if you are working on a remote or headless machine\[rs]n\[dq],
|
|
\[dq]Default\[dq]: true,
|
|
\[dq]Examples\[dq]: [
|
|
{
|
|
\[dq]Value\[dq]: \[dq]true\[dq],
|
|
\[dq]Help\[dq]: \[dq]Yes\[dq]
|
|
},
|
|
{
|
|
\[dq]Value\[dq]: \[dq]false\[dq],
|
|
\[dq]Help\[dq]: \[dq]No\[dq]
|
|
}
|
|
],
|
|
\[dq]Required\[dq]: false,
|
|
\[dq]IsPassword\[dq]: false,
|
|
\[dq]Type\[dq]: \[dq]bool\[dq],
|
|
\[dq]Exclusive\[dq]: true,
|
|
},
|
|
\[dq]Error\[dq]: \[dq]\[dq],
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The format of \f[C]Option\f[R] is the same as returned by
|
|
\f[C]rclone config providers\f[R].
|
|
The question should be asked to the user and returned to rclone as the
|
|
\f[C]--result\f[R] option along with the \f[C]--state\f[R] parameter.
|
|
.PP
|
|
The keys of \f[C]Option\f[R] are used as follows:
|
|
.IP \[bu] 2
|
|
\f[C]Name\f[R] - name of variable - show to user
|
|
.IP \[bu] 2
|
|
\f[C]Help\f[R] - help text.
|
|
Hard wrapped at 80 chars.
|
|
Any URLs should be clicky.
|
|
.IP \[bu] 2
|
|
\f[C]Default\f[R] - default value - return this if the user just wants
|
|
the default.
|
|
.IP \[bu] 2
|
|
\f[C]Examples\f[R] - the user should be able to choose one of these
|
|
.IP \[bu] 2
|
|
\f[C]Required\f[R] - the value should be non-empty
|
|
.IP \[bu] 2
|
|
\f[C]IsPassword\f[R] - the value is a password and should be edited as
|
|
such
|
|
.IP \[bu] 2
|
|
\f[C]Type\f[R] - type of value, eg \f[C]bool\f[R], \f[C]string\f[R],
|
|
\f[C]int\f[R] and others
|
|
.IP \[bu] 2
|
|
\f[C]Exclusive\f[R] - if set no free-form entry allowed only the
|
|
\f[C]Examples\f[R]
|
|
.IP \[bu] 2
|
|
Irrelevant keys \f[C]Provider\f[R], \f[C]ShortOpt\f[R], \f[C]Hide\f[R],
|
|
\f[C]NoPrefix\f[R], \f[C]Advanced\f[R]
|
|
.PP
|
|
If \f[C]Error\f[R] is set then it should be shown to the user at the
|
|
same time as the question.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config update name --continue --state \[dq]*oauth-islocal,teamdrive,,\[dq] --result \[dq]true\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that when using \f[C]--continue\f[R] all passwords should be passed
|
|
in the clear (not obscured).
|
|
Any default config values should be passed in with each invocation of
|
|
\f[C]--continue\f[R].
|
|
.PP
|
|
At the end of the non interactive process, rclone will return a result
|
|
with \f[C]State\f[R] as empty string.
|
|
.PP
|
|
If \f[C]--all\f[R] is passed then rclone will ask all the config
|
|
questions, not just the post config questions.
|
|
Any parameters are used as defaults for questions as usual.
|
|
.PP
|
|
Note that \f[C]bin/config.py\f[R] in the rclone source implements this
|
|
protocol as a readable demonstration.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config create name type [key value]* [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--all Ask the full set of config questions
|
|
--continue Continue the configuration process with an answer
|
|
-h, --help help for create
|
|
--no-obscure Force any passwords not to be obscured
|
|
--non-interactive Don\[aq]t interact with user and return questions
|
|
--obscure Force any passwords to be obscured
|
|
--result string Result - use with --continue
|
|
--state string State - use with --continue
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config delete
|
|
.PP
|
|
Delete an existing remote.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config delete name [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for delete
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config disconnect
|
|
.PP
|
|
Disconnects user from remote
|
|
.SS Synopsis
|
|
.PP
|
|
This disconnects the remote: passed in to the cloud storage system.
|
|
.PP
|
|
This normally means revoking the oauth token.
|
|
.PP
|
|
To reconnect use \[dq]rclone config reconnect\[dq].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config disconnect remote: [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for disconnect
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config dump
|
|
.PP
|
|
Dump the config file as JSON.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config dump [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for dump
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config edit
|
|
.PP
|
|
Enter an interactive configuration session.
|
|
.SH Synopsis
|
|
.PP
|
|
Enter an interactive configuration session where you can setup new
|
|
remotes and manage existing ones.
|
|
You may also set or remove a password to protect your configuration.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config edit [flags]
|
|
\f[R]
|
|
.fi
|
|
.SH Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for edit
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SH SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config file
|
|
.PP
|
|
Show path of configuration file in use.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config file [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for file
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config password
|
|
.PP
|
|
Update password in an existing remote.
|
|
.SS Synopsis
|
|
.PP
|
|
Update an existing remote\[aq]s password.
|
|
The password should be passed in pairs of \f[C]key\f[R]
|
|
\f[C]password\f[R] or as \f[C]key=password\f[R].
|
|
The \f[C]password\f[R] should be passed in in clear (unobscured).
|
|
.PP
|
|
For example, to set password of a remote of name myremote you would do:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config password myremote fieldname mypassword
|
|
rclone config password myremote fieldname=mypassword
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This command is obsolete now that \[dq]config update\[dq] and
|
|
\[dq]config create\[dq] both support obscuring passwords directly.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config password name [key value]+ [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for password
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config paths
|
|
.PP
|
|
Show paths used for configuration, cache, temp etc.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config paths [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for paths
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config providers
|
|
.PP
|
|
List in JSON format all the providers and options.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config providers [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for providers
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config reconnect
|
|
.PP
|
|
Re-authenticates user with remote.
|
|
.SS Synopsis
|
|
.PP
|
|
This reconnects remote: passed in to the cloud storage system.
|
|
.PP
|
|
To disconnect the remote use \[dq]rclone config disconnect\[dq].
|
|
.PP
|
|
This normally means going through the interactive oauth flow again.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config reconnect remote: [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for reconnect
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config show
|
|
.PP
|
|
Print (decrypted) config file, or the config for a single remote.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config show [<remote>] [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for show
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config touch
|
|
.PP
|
|
Ensure configuration file exists.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config touch [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for touch
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config update
|
|
.PP
|
|
Update options in an existing remote.
|
|
.SS Synopsis
|
|
.PP
|
|
Update an existing remote\[aq]s options.
|
|
The options should be passed in pairs of \f[C]key\f[R] \f[C]value\f[R]
|
|
or as \f[C]key=value\f[R].
|
|
.PP
|
|
For example, to update the env_auth field of a remote of name myremote
|
|
you would do:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config update myremote env_auth true
|
|
rclone config update myremote env_auth=true
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If the remote uses OAuth the token will be updated, if you don\[aq]t
|
|
require this add an extra parameter thus:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config update myremote env_auth=true config_refresh_token=false
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that if the config process would normally ask a question the
|
|
default is taken (unless \f[C]--non-interactive\f[R] is used).
|
|
Each time that happens rclone will print or DEBUG a message saying how
|
|
to affect the value taken.
|
|
.PP
|
|
If any of the parameters passed is a password field, then rclone will
|
|
automatically obscure them if they aren\[aq]t already obscured before
|
|
putting them in the config file.
|
|
.PP
|
|
\f[B]NB\f[R] If the password parameter is 22 characters or longer and
|
|
consists only of base64 characters then rclone can get confused about
|
|
whether the password is already obscured or not and put unobscured
|
|
passwords into the config file.
|
|
If you want to be 100% certain that the passwords get obscured then use
|
|
the \f[C]--obscure\f[R] flag, or if you are 100% certain you are already
|
|
passing obscured passwords then use \f[C]--no-obscure\f[R].
|
|
You can also set obscured passwords using the
|
|
\f[C]rclone config password\f[R] command.
|
|
.PP
|
|
The flag \f[C]--non-interactive\f[R] is for use by applications that
|
|
wish to configure rclone themeselves, rather than using rclone\[aq]s
|
|
text based configuration questions.
|
|
If this flag is set, and rclone needs to ask the user a question, a JSON
|
|
blob will be returned with the question in it.
|
|
.PP
|
|
This will look something like (some irrelevant detail removed):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]State\[dq]: \[dq]*oauth-islocal,teamdrive,,\[dq],
|
|
\[dq]Option\[dq]: {
|
|
\[dq]Name\[dq]: \[dq]config_is_local\[dq],
|
|
\[dq]Help\[dq]: \[dq]Use auto config?\[rs]n * Say Y if not sure\[rs]n * Say N if you are working on a remote or headless machine\[rs]n\[dq],
|
|
\[dq]Default\[dq]: true,
|
|
\[dq]Examples\[dq]: [
|
|
{
|
|
\[dq]Value\[dq]: \[dq]true\[dq],
|
|
\[dq]Help\[dq]: \[dq]Yes\[dq]
|
|
},
|
|
{
|
|
\[dq]Value\[dq]: \[dq]false\[dq],
|
|
\[dq]Help\[dq]: \[dq]No\[dq]
|
|
}
|
|
],
|
|
\[dq]Required\[dq]: false,
|
|
\[dq]IsPassword\[dq]: false,
|
|
\[dq]Type\[dq]: \[dq]bool\[dq],
|
|
\[dq]Exclusive\[dq]: true,
|
|
},
|
|
\[dq]Error\[dq]: \[dq]\[dq],
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The format of \f[C]Option\f[R] is the same as returned by
|
|
\f[C]rclone config providers\f[R].
|
|
The question should be asked to the user and returned to rclone as the
|
|
\f[C]--result\f[R] option along with the \f[C]--state\f[R] parameter.
|
|
.PP
|
|
The keys of \f[C]Option\f[R] are used as follows:
|
|
.IP \[bu] 2
|
|
\f[C]Name\f[R] - name of variable - show to user
|
|
.IP \[bu] 2
|
|
\f[C]Help\f[R] - help text.
|
|
Hard wrapped at 80 chars.
|
|
Any URLs should be clicky.
|
|
.IP \[bu] 2
|
|
\f[C]Default\f[R] - default value - return this if the user just wants
|
|
the default.
|
|
.IP \[bu] 2
|
|
\f[C]Examples\f[R] - the user should be able to choose one of these
|
|
.IP \[bu] 2
|
|
\f[C]Required\f[R] - the value should be non-empty
|
|
.IP \[bu] 2
|
|
\f[C]IsPassword\f[R] - the value is a password and should be edited as
|
|
such
|
|
.IP \[bu] 2
|
|
\f[C]Type\f[R] - type of value, eg \f[C]bool\f[R], \f[C]string\f[R],
|
|
\f[C]int\f[R] and others
|
|
.IP \[bu] 2
|
|
\f[C]Exclusive\f[R] - if set no free-form entry allowed only the
|
|
\f[C]Examples\f[R]
|
|
.IP \[bu] 2
|
|
Irrelevant keys \f[C]Provider\f[R], \f[C]ShortOpt\f[R], \f[C]Hide\f[R],
|
|
\f[C]NoPrefix\f[R], \f[C]Advanced\f[R]
|
|
.PP
|
|
If \f[C]Error\f[R] is set then it should be shown to the user at the
|
|
same time as the question.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config update name --continue --state \[dq]*oauth-islocal,teamdrive,,\[dq] --result \[dq]true\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that when using \f[C]--continue\f[R] all passwords should be passed
|
|
in the clear (not obscured).
|
|
Any default config values should be passed in with each invocation of
|
|
\f[C]--continue\f[R].
|
|
.PP
|
|
At the end of the non interactive process, rclone will return a result
|
|
with \f[C]State\f[R] as empty string.
|
|
.PP
|
|
If \f[C]--all\f[R] is passed then rclone will ask all the config
|
|
questions, not just the post config questions.
|
|
Any parameters are used as defaults for questions as usual.
|
|
.PP
|
|
Note that \f[C]bin/config.py\f[R] in the rclone source implements this
|
|
protocol as a readable demonstration.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config update name [key value]+ [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--all Ask the full set of config questions
|
|
--continue Continue the configuration process with an answer
|
|
-h, --help help for update
|
|
--no-obscure Force any passwords not to be obscured
|
|
--non-interactive Don\[aq]t interact with user and return questions
|
|
--obscure Force any passwords to be obscured
|
|
--result string Result - use with --continue
|
|
--state string State - use with --continue
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone config userinfo
|
|
.PP
|
|
Prints info about logged in user of remote.
|
|
.SS Synopsis
|
|
.PP
|
|
This prints the details of the person logged in to the cloud storage
|
|
system.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config userinfo remote: [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for userinfo
|
|
--json Format output as JSON
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone config (https://rclone.org/commands/rclone_config/) - Enter an
|
|
interactive configuration session.
|
|
.SH rclone copyto
|
|
.PP
|
|
Copy files from source to dest, skipping identical files.
|
|
.SS Synopsis
|
|
.PP
|
|
If source:path is a file or directory then it copies it to a file or
|
|
directory named dest:path.
|
|
.PP
|
|
This can be used to upload single files to other than their current
|
|
name.
|
|
If the source is a directory then it acts exactly like the copy command.
|
|
.PP
|
|
So
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copyto src dst
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
where src and dst are rclone paths, either remote:path or /path/to/local
|
|
or C:.
|
|
.PP
|
|
This will:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
if src is file
|
|
copy it to dst, overwriting an existing file if it exists
|
|
if src is directory
|
|
copy it to dst, overwriting existing files if they exist
|
|
see copy command for full details
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This doesn\[aq]t transfer files that are identical on src and dst,
|
|
testing by size and modification time or MD5SUM.
|
|
It doesn\[aq]t delete files from the destination.
|
|
.PP
|
|
\f[B]Note\f[R]: Use the \f[C]-P\f[R]/\f[C]--progress\f[R] flag to view
|
|
real-time transfer statistics
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copyto source:path dest:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for copyto
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone copyurl
|
|
.PP
|
|
Copy url content to dest.
|
|
.SS Synopsis
|
|
.PP
|
|
Download a URL\[aq]s content and copy it to the destination without
|
|
saving it in temporary storage.
|
|
.PP
|
|
Setting \f[C]--auto-filename\f[R] will cause the file name to be
|
|
retrieved from the URL (after any redirections) and used in the
|
|
destination path.
|
|
With \f[C]--print-filename\f[R] in addition, the resulting file name
|
|
will be printed.
|
|
.PP
|
|
Setting \f[C]--no-clobber\f[R] will prevent overwriting file on the
|
|
destination if there is one with the same name.
|
|
.PP
|
|
Setting \f[C]--stdout\f[R] or making the output file name \f[C]-\f[R]
|
|
will cause the output to be written to standard output.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copyurl https://example.com dest:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-a, --auto-filename Get the file name from the URL and use it for destination file path
|
|
-h, --help help for copyurl
|
|
--no-clobber Prevent overwriting file with same name
|
|
-p, --print-filename Print the resulting name from --auto-filename
|
|
--stdout Write the output to stdout rather than a file
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone cryptcheck
|
|
.PP
|
|
Cryptcheck checks the integrity of a crypted remote.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone cryptcheck checks a remote against a crypted remote.
|
|
This is the equivalent of running rclone check, but able to check the
|
|
checksums of the crypted remote.
|
|
.PP
|
|
For it to work the underlying remote of the cryptedremote must support
|
|
some kind of checksum.
|
|
.PP
|
|
It works by reading the nonce from each file on the cryptedremote: and
|
|
using that to encrypt each file on the remote:.
|
|
It then checks the checksum of the underlying file on the cryptedremote:
|
|
against the checksum of the file it has just encrypted.
|
|
.PP
|
|
Use it like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone cryptcheck /path/to/files encryptedremote:path
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can use it like this also, but that will involve downloading all the
|
|
files in remote:path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone cryptcheck remote:path encryptedremote:path
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
After it has run it will log the status of the encryptedremote:.
|
|
.PP
|
|
If you supply the \f[C]--one-way\f[R] flag, it will only check that
|
|
files in the source match the files in the destination, not the other
|
|
way around.
|
|
This means that extra files in the destination that are not in the
|
|
source will not be detected.
|
|
.PP
|
|
The \f[C]--differ\f[R], \f[C]--missing-on-dst\f[R],
|
|
\f[C]--missing-on-src\f[R], \f[C]--match\f[R] and \f[C]--error\f[R]
|
|
flags write paths, one per line, to the file name (or stdout if it is
|
|
\f[C]-\f[R]) supplied.
|
|
What they write is described in the help below.
|
|
For example \f[C]--differ\f[R] will write all paths which are present on
|
|
both the source and destination but different.
|
|
.PP
|
|
The \f[C]--combined\f[R] flag will write a file (or stdout) which
|
|
contains all file paths with a symbol and then a space and then the path
|
|
to tell you what happened to it.
|
|
These are reminiscent of diff files.
|
|
.IP \[bu] 2
|
|
\f[C]= path\f[R] means path was found in source and destination and was
|
|
identical
|
|
.IP \[bu] 2
|
|
\[ga]- path\[ga] means path was missing on the source, so only in the
|
|
destination
|
|
.IP \[bu] 2
|
|
\[ga]+ path\[ga] means path was missing on the destination, so only in
|
|
the source
|
|
.IP \[bu] 2
|
|
\[ga]* path\[ga] means path was present in source and destination but
|
|
different.
|
|
.IP \[bu] 2
|
|
\f[C]! path\f[R] means there was an error reading or hashing the source
|
|
or dest.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone cryptcheck remote:path cryptedremote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--combined string Make a combined report of changes to this file
|
|
--differ string Report all non-matching files to this file
|
|
--error string Report all files with errors (hashing or reading) to this file
|
|
-h, --help help for cryptcheck
|
|
--match string Report all matching files to this file
|
|
--missing-on-dst string Report all files missing from the destination to this file
|
|
--missing-on-src string Report all files missing from the source to this file
|
|
--one-way Check one way only, source files must exist on remote
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone cryptdecode
|
|
.PP
|
|
Cryptdecode returns unencrypted file names.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone cryptdecode returns unencrypted file names when provided with a
|
|
list of encrypted file names.
|
|
List limit is 10 items.
|
|
.PP
|
|
If you supply the --reverse flag, it will return encrypted file names.
|
|
.PP
|
|
use it like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone cryptdecode encryptedremote: encryptedfilename1 encryptedfilename2
|
|
|
|
rclone cryptdecode --reverse encryptedremote: filename1 filename2
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Another way to accomplish this is by using the
|
|
\f[C]rclone backend encode\f[R] (or \f[C]decode\f[R])command.
|
|
See the documentation on the \f[C]crypt\f[R] overlay for more info.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone cryptdecode encryptedremote: encryptedfilename [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for cryptdecode
|
|
--reverse Reverse cryptdecode, encrypts filenames
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone deletefile
|
|
.PP
|
|
Remove a single file from remote.
|
|
.SS Synopsis
|
|
.PP
|
|
Remove a single file from remote.
|
|
Unlike \f[C]delete\f[R] it cannot be used to remove a directory and it
|
|
doesn\[aq]t obey include/exclude filters - if the specified file exists,
|
|
it will always be removed.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone deletefile remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for deletefile
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone genautocomplete
|
|
.PP
|
|
Output completion script for a given shell.
|
|
.SS Synopsis
|
|
.PP
|
|
Generates a shell completion script for rclone.
|
|
Run with --help to list the supported shells.
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for genautocomplete
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.IP \[bu] 2
|
|
rclone genautocomplete
|
|
bash (https://rclone.org/commands/rclone_genautocomplete_bash/) - Output
|
|
bash completion script for rclone.
|
|
.IP \[bu] 2
|
|
rclone genautocomplete
|
|
fish (https://rclone.org/commands/rclone_genautocomplete_fish/) - Output
|
|
fish completion script for rclone.
|
|
.IP \[bu] 2
|
|
rclone genautocomplete
|
|
zsh (https://rclone.org/commands/rclone_genautocomplete_zsh/) - Output
|
|
zsh completion script for rclone.
|
|
.SH rclone genautocomplete bash
|
|
.PP
|
|
Output bash completion script for rclone.
|
|
.SS Synopsis
|
|
.PP
|
|
Generates a bash shell autocompletion script for rclone.
|
|
.PP
|
|
This writes to /etc/bash_completion.d/rclone by default so will probably
|
|
need to be run with sudo or as root, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo rclone genautocomplete bash
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Logout and login again to use the autocompletion scripts, or source them
|
|
directly
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\&. /etc/bash_completion
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you supply a command line argument the script will be written there.
|
|
.PP
|
|
If output_file is \[dq]-\[dq], then the output will be written to
|
|
stdout.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone genautocomplete bash [output_file] [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for bash
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone
|
|
genautocomplete (https://rclone.org/commands/rclone_genautocomplete/) -
|
|
Output completion script for a given shell.
|
|
.SH rclone genautocomplete fish
|
|
.PP
|
|
Output fish completion script for rclone.
|
|
.SS Synopsis
|
|
.PP
|
|
Generates a fish autocompletion script for rclone.
|
|
.PP
|
|
This writes to /etc/fish/completions/rclone.fish by default so will
|
|
probably need to be run with sudo or as root, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo rclone genautocomplete fish
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Logout and login again to use the autocompletion scripts, or source them
|
|
directly
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\&. /etc/fish/completions/rclone.fish
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you supply a command line argument the script will be written there.
|
|
.PP
|
|
If output_file is \[dq]-\[dq], then the output will be written to
|
|
stdout.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone genautocomplete fish [output_file] [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for fish
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone
|
|
genautocomplete (https://rclone.org/commands/rclone_genautocomplete/) -
|
|
Output completion script for a given shell.
|
|
.SH rclone genautocomplete zsh
|
|
.PP
|
|
Output zsh completion script for rclone.
|
|
.SS Synopsis
|
|
.PP
|
|
Generates a zsh autocompletion script for rclone.
|
|
.PP
|
|
This writes to /usr/share/zsh/vendor-completions/_rclone by default so
|
|
will probably need to be run with sudo or as root, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo rclone genautocomplete zsh
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Logout and login again to use the autocompletion scripts, or source them
|
|
directly
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
autoload -U compinit && compinit
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you supply a command line argument the script will be written there.
|
|
.PP
|
|
If output_file is \[dq]-\[dq], then the output will be written to
|
|
stdout.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone genautocomplete zsh [output_file] [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for zsh
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone
|
|
genautocomplete (https://rclone.org/commands/rclone_genautocomplete/) -
|
|
Output completion script for a given shell.
|
|
.SH rclone gendocs
|
|
.PP
|
|
Output markdown docs for rclone to the directory supplied.
|
|
.SS Synopsis
|
|
.PP
|
|
This produces markdown docs for the rclone commands to the directory
|
|
supplied.
|
|
These are in a format suitable for hugo to render into the rclone.org
|
|
website.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone gendocs output_directory [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for gendocs
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone hashsum
|
|
.PP
|
|
Produces a hashsum file for all the objects in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
Produces a hash file for all the objects in the path using the hash
|
|
named.
|
|
The output is in the same format as the standard md5sum/sha1sum tool.
|
|
.PP
|
|
By default, the hash is requested from the remote.
|
|
If the hash is not supported by the remote, no hash will be returned.
|
|
With the download flag, the file will be downloaded from the remote and
|
|
hashed locally enabling any hash for any remote.
|
|
.PP
|
|
This command can also hash data received on standard input (stdin), by
|
|
not passing a remote:path, or by passing a hyphen as remote:path when
|
|
there is data to read (if not, the hypen will be treated literaly, as a
|
|
relative path).
|
|
.PP
|
|
Run without a hash to see the list of all supported hashes, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone hashsum
|
|
Supported hashes are:
|
|
* md5
|
|
* sha1
|
|
* whirlpool
|
|
* crc32
|
|
* sha256
|
|
* dropbox
|
|
* mailru
|
|
* quickxor
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Then
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone hashsum MD5 remote:path
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that hash names are case insensitive and values are output in lower
|
|
case.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone hashsum <hash> remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--base64 Output base64 encoded hashsum
|
|
-C, --checkfile string Validate hashes against a given SUM file instead of printing them
|
|
--download Download the file and hash it locally; if this flag is not specified, the hash is requested from the remote
|
|
-h, --help help for hashsum
|
|
--output-file string Output hashsums to a file rather than the terminal
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone link
|
|
.PP
|
|
Generate public link to file/folder.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone link will create, retrieve or remove a public link to the given
|
|
file or folder.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone link remote:path/to/file
|
|
rclone link remote:path/to/folder/
|
|
rclone link --unlink remote:path/to/folder/
|
|
rclone link --expire 1d remote:path/to/file
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you supply the --expire flag, it will set the expiration time
|
|
otherwise it will use the default (100 years).
|
|
\f[B]Note\f[R] not all backends support the --expire flag - if the
|
|
backend doesn\[aq]t support it then the link returned won\[aq]t expire.
|
|
.PP
|
|
Use the --unlink flag to remove existing public links to the file or
|
|
folder.
|
|
\f[B]Note\f[R] not all backends support \[dq]--unlink\[dq] flag - those
|
|
that don\[aq]t will just ignore it.
|
|
.PP
|
|
If successful, the last line of the output will contain the link.
|
|
Exact capabilities depend on the remote, but the link will always by
|
|
default be created with the least constraints \[en] e.g.
|
|
no expiry, no password protection, accessible without account.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone link remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--expire Duration The amount of time that the link will be valid (default off)
|
|
-h, --help help for link
|
|
--unlink Remove existing public link to file/folder
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone listremotes
|
|
.PP
|
|
List all the remotes in the config file.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone listremotes lists all the available remotes from the config file.
|
|
.PP
|
|
When uses with the -l flag it lists the types too.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone listremotes [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for listremotes
|
|
--long Show the type as well as names
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone lsf
|
|
.PP
|
|
List directories and objects in remote:path formatted for parsing.
|
|
.SS Synopsis
|
|
.PP
|
|
List the contents of the source path (directories and objects) to
|
|
standard output in a form which is easy to parse by scripts.
|
|
By default this will just be the names of the objects and directories,
|
|
one per line.
|
|
The directories will have a / suffix.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsf swift:bucket
|
|
bevajer5jef
|
|
canole
|
|
diwogej7
|
|
ferejej3gux/
|
|
fubuwic
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Use the --format option to control what gets listed.
|
|
By default this is just the path, but you can use these parameters to
|
|
control the output:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
p - path
|
|
s - size
|
|
t - modification time
|
|
h - hash
|
|
i - ID of object
|
|
o - Original ID of underlying object
|
|
m - MimeType of object if known
|
|
e - encrypted name
|
|
T - tier of storage if known, e.g. \[dq]Hot\[dq] or \[dq]Cool\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
So if you wanted the path, size and modification time, you would use
|
|
--format \[dq]pst\[dq], or maybe --format \[dq]tsp\[dq] to put the path
|
|
last.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsf --format \[dq]tsp\[dq] swift:bucket
|
|
2016-06-25 18:55:41;60295;bevajer5jef
|
|
2016-06-25 18:55:43;90613;canole
|
|
2016-06-25 18:55:43;94467;diwogej7
|
|
2018-04-26 08:50:45;0;ferejej3gux/
|
|
2016-06-25 18:55:40;37600;fubuwic
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you specify \[dq]h\[dq] in the format you will get the MD5 hash by
|
|
default, use the \[dq]--hash\[dq] flag to change which hash you want.
|
|
Note that this can be returned as an empty string if it isn\[aq]t
|
|
available on the object (and for directories), \[dq]ERROR\[dq] if there
|
|
was an error reading it from the object and \[dq]UNSUPPORTED\[dq] if
|
|
that object does not support that hash type.
|
|
.PP
|
|
For example, to emulate the md5sum command you can use
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsf -R --hash MD5 --format hp --separator \[dq] \[dq] --files-only .
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsf -R --hash MD5 --format hp --separator \[dq] \[dq] --files-only swift:bucket
|
|
7908e352297f0f530b84a756f188baa3 bevajer5jef
|
|
cd65ac234e6fea5925974a51cdd865cc canole
|
|
03b5341b4f234b9d984d03ad076bae91 diwogej7
|
|
8fd37c3810dd660778137ac3a66cc06d fubuwic
|
|
99713e14a4c4ff553acaf1930fad985b gixacuh7ku
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
(Though \[dq]rclone md5sum .\[dq] is an easier way of typing this.)
|
|
.PP
|
|
By default the separator is \[dq];\[dq] this can be changed with the
|
|
--separator flag.
|
|
Note that separators aren\[aq]t escaped in the path so putting it last
|
|
is a good strategy.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsf --separator \[dq],\[dq] --format \[dq]tshp\[dq] swift:bucket
|
|
2016-06-25 18:55:41,60295,7908e352297f0f530b84a756f188baa3,bevajer5jef
|
|
2016-06-25 18:55:43,90613,cd65ac234e6fea5925974a51cdd865cc,canole
|
|
2016-06-25 18:55:43,94467,03b5341b4f234b9d984d03ad076bae91,diwogej7
|
|
2018-04-26 08:52:53,0,,ferejej3gux/
|
|
2016-06-25 18:55:40,37600,8fd37c3810dd660778137ac3a66cc06d,fubuwic
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can output in CSV standard format.
|
|
This will escape things in \[dq] if they contain ,
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsf --csv --files-only --format ps remote:path
|
|
test.log,22355
|
|
test.sh,449
|
|
\[dq]this file contains a comma, in the file name.txt\[dq],6
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that the --absolute parameter is useful for making lists of files
|
|
to pass to an rclone copy with the --files-from-raw flag.
|
|
.PP
|
|
For example, to find all the files modified within one day and copy
|
|
those only (without traversing the whole directory structure):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsf --absolute --files-only --max-age 1d /path/to/local > new_files
|
|
rclone copy --files-from-raw new_files /path/to/local remote:path
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Any of the filtering options can be applied to this command.
|
|
.PP
|
|
There are several related list commands
|
|
.IP \[bu] 2
|
|
\f[C]ls\f[R] to list size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsl\f[R] to list modification time, size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsd\f[R] to list directories only
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[R] to list objects and directories in easy to parse format
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[R] to list objects and directories in JSON format
|
|
.PP
|
|
\f[C]ls\f[R],\f[C]lsl\f[R],\f[C]lsd\f[R] are designed to be
|
|
human-readable.
|
|
\f[C]lsf\f[R] is designed to be human and machine-readable.
|
|
\f[C]lsjson\f[R] is designed to be machine-readable.
|
|
.PP
|
|
Note that \f[C]ls\f[R] and \f[C]lsl\f[R] recurse by default - use
|
|
\f[C]--max-depth 1\f[R] to stop the recursion.
|
|
.PP
|
|
The other list commands \f[C]lsd\f[R],\f[C]lsf\f[R],\f[C]lsjson\f[R] do
|
|
not recurse by default - use \f[C]-R\f[R] to make them recurse.
|
|
.PP
|
|
Listing a non-existent directory will produce an error except for
|
|
remotes which can\[aq]t have empty directories (e.g.
|
|
s3, swift, or gcs - the bucket-based remotes).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsf remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--absolute Put a leading / in front of path names
|
|
--csv Output in CSV format
|
|
-d, --dir-slash Append a slash to directory names (default true)
|
|
--dirs-only Only list directories
|
|
--files-only Only list files
|
|
-F, --format string Output format - see help for details (default \[dq]p\[dq])
|
|
--hash h Use this hash when h is used in the format MD5|SHA-1|DropboxHash (default \[dq]md5\[dq])
|
|
-h, --help help for lsf
|
|
-R, --recursive Recurse into the listing
|
|
-s, --separator string Separator for the items in the format (default \[dq];\[dq])
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone lsjson
|
|
.PP
|
|
List directories and objects in the path in JSON format.
|
|
.SS Synopsis
|
|
.PP
|
|
List directories and objects in the path in JSON format.
|
|
.PP
|
|
The output is an array of Items, where each Item looks like this
|
|
.PP
|
|
{ \[dq]Hashes\[dq] : { \[dq]SHA-1\[dq] :
|
|
\[dq]f572d396fae9206628714fb2ce00f72e94f2258f\[dq], \[dq]MD5\[dq] :
|
|
\[dq]b1946ac92492d2347c6235b4d2611184\[dq], \[dq]DropboxHash\[dq] :
|
|
\[dq]ecb65bb98f9d905b70458986c39fcbad7715e5f2fcc3b1f07767d7c83e2438cc\[dq]
|
|
}, \[dq]ID\[dq]: \[dq]y2djkhiujf83u33\[dq], \[dq]OrigID\[dq]:
|
|
\[dq]UYOJVTUW00Q1RzTDA\[dq], \[dq]IsBucket\[dq] : false, \[dq]IsDir\[dq]
|
|
: false, \[dq]MimeType\[dq] : \[dq]application/octet-stream\[dq],
|
|
\[dq]ModTime\[dq] : \[dq]2017-05-31T16:15:57.034468261+01:00\[dq],
|
|
\[dq]Name\[dq] : \[dq]file.txt\[dq], \[dq]Encrypted\[dq] :
|
|
\[dq]v0qpsdq8anpci8n929v3uu9338\[dq], \[dq]EncryptedPath\[dq] :
|
|
\[dq]kja9098349023498/v0qpsdq8anpci8n929v3uu9338\[dq], \[dq]Path\[dq] :
|
|
\[dq]full/path/goes/here/file.txt\[dq], \[dq]Size\[dq] : 6,
|
|
\[dq]Tier\[dq] : \[dq]hot\[dq], }
|
|
.PP
|
|
If --hash is not specified the Hashes property won\[aq]t be emitted.
|
|
The types of hash can be specified with the --hash-type parameter (which
|
|
may be repeated).
|
|
If --hash-type is set then it implies --hash.
|
|
.PP
|
|
If --no-modtime is specified then ModTime will be blank.
|
|
This can speed things up on remotes where reading the ModTime takes an
|
|
extra request (e.g.
|
|
s3, swift).
|
|
.PP
|
|
If --no-mimetype is specified then MimeType will be blank.
|
|
This can speed things up on remotes where reading the MimeType takes an
|
|
extra request (e.g.
|
|
s3, swift).
|
|
.PP
|
|
If --encrypted is not specified the Encrypted won\[aq]t be emitted.
|
|
.PP
|
|
If --dirs-only is not specified files in addition to directories are
|
|
returned
|
|
.PP
|
|
If --files-only is not specified directories in addition to the files
|
|
will be returned.
|
|
.PP
|
|
if --stat is set then a single JSON blob will be returned about the item
|
|
pointed to.
|
|
This will return an error if the item isn\[aq]t found.
|
|
However on bucket based backends (like s3, gcs, b2, azureblob etc) if
|
|
the item isn\[aq]t found it will return an empty directory as it
|
|
isn\[aq]t possible to tell empty directories from missing directories
|
|
there.
|
|
.PP
|
|
The Path field will only show folders below the remote path being
|
|
listed.
|
|
If \[dq]remote:path\[dq] contains the file \[dq]subfolder/file.txt\[dq],
|
|
the Path for \[dq]file.txt\[dq] will be \[dq]subfolder/file.txt\[dq],
|
|
not \[dq]remote:path/subfolder/file.txt\[dq].
|
|
When used without --recursive the Path will always be the same as Name.
|
|
.PP
|
|
If the directory is a bucket in a bucket-based backend, then
|
|
\[dq]IsBucket\[dq] will be set to true.
|
|
This key won\[aq]t be present unless it is \[dq]true\[dq].
|
|
.PP
|
|
The time is in RFC3339 format with up to nanosecond precision.
|
|
The number of decimal digits in the seconds will depend on the precision
|
|
that the remote can hold the times, so if times are accurate to the
|
|
nearest millisecond (e.g.
|
|
Google Drive) then 3 digits will always be shown
|
|
(\[dq]2017-05-31T16:15:57.034+01:00\[dq]) whereas if the times are
|
|
accurate to the nearest second (Dropbox, Box, WebDav, etc.) no digits
|
|
will be shown (\[dq]2017-05-31T16:15:57+01:00\[dq]).
|
|
.PP
|
|
The whole output can be processed as a JSON blob, or alternatively it
|
|
can be processed line by line as each item is written one to a line.
|
|
.PP
|
|
Any of the filtering options can be applied to this command.
|
|
.PP
|
|
There are several related list commands
|
|
.IP \[bu] 2
|
|
\f[C]ls\f[R] to list size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsl\f[R] to list modification time, size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsd\f[R] to list directories only
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[R] to list objects and directories in easy to parse format
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[R] to list objects and directories in JSON format
|
|
.PP
|
|
\f[C]ls\f[R],\f[C]lsl\f[R],\f[C]lsd\f[R] are designed to be
|
|
human-readable.
|
|
\f[C]lsf\f[R] is designed to be human and machine-readable.
|
|
\f[C]lsjson\f[R] is designed to be machine-readable.
|
|
.PP
|
|
Note that \f[C]ls\f[R] and \f[C]lsl\f[R] recurse by default - use
|
|
\f[C]--max-depth 1\f[R] to stop the recursion.
|
|
.PP
|
|
The other list commands \f[C]lsd\f[R],\f[C]lsf\f[R],\f[C]lsjson\f[R] do
|
|
not recurse by default - use \f[C]-R\f[R] to make them recurse.
|
|
.PP
|
|
Listing a non-existent directory will produce an error except for
|
|
remotes which can\[aq]t have empty directories (e.g.
|
|
s3, swift, or gcs - the bucket-based remotes).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsjson remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--dirs-only Show only directories in the listing
|
|
-M, --encrypted Show the encrypted names
|
|
--files-only Show only files in the listing
|
|
--hash Include hashes in the output (may take longer)
|
|
--hash-type stringArray Show only this hash type (may be repeated)
|
|
-h, --help help for lsjson
|
|
--no-mimetype Don\[aq]t read the mime type (can speed things up)
|
|
--no-modtime Don\[aq]t read the modification time (can speed things up)
|
|
--original Show the ID of the underlying Object
|
|
-R, --recursive Recurse into the listing
|
|
--stat Just return the info for the pointed to file
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone mount
|
|
.PP
|
|
Mount the remote as file system on a mountpoint.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone mount allows Linux, FreeBSD, macOS and Windows to mount any of
|
|
Rclone\[aq]s cloud storage systems as a file system with FUSE.
|
|
.PP
|
|
First set up your remote using \f[C]rclone config\f[R].
|
|
Check it works with \f[C]rclone ls\f[R] etc.
|
|
.PP
|
|
On Linux and macOS, you can run mount in either foreground or background
|
|
(aka daemon) mode.
|
|
Mount runs in foreground mode by default.
|
|
Use the \f[C]--daemon\f[R] flag to force background mode.
|
|
On Windows you can run mount in foreground only, the flag is ignored.
|
|
.PP
|
|
In background mode rclone acts as a generic Unix mount program: the main
|
|
program starts, spawns background rclone process to setup and maintain
|
|
the mount, waits until success or timeout and exits with appropriate
|
|
code (killing the child process if it fails).
|
|
.PP
|
|
On Linux/macOS/FreeBSD start the mount like this, where
|
|
\f[C]/path/to/local/mount\f[R] is an \f[B]empty\f[R] \f[B]existing\f[R]
|
|
directory:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mount remote:path/to/files /path/to/local/mount
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
On Windows you can start a mount in different ways.
|
|
See below for details.
|
|
If foreground mount is used interactively from a console window, rclone
|
|
will serve the mount and occupy the console so another window should be
|
|
used to work with the mount until rclone is interrupted e.g.
|
|
by pressing Ctrl-C.
|
|
.PP
|
|
The following examples will mount to an automatically assigned drive, to
|
|
specific drive letter \f[C]X:\f[R], to path
|
|
\f[C]C:\[rs]path\[rs]parent\[rs]mount\f[R] (where parent directory or
|
|
drive must exist, and mount must \f[B]not\f[R] exist, and is not
|
|
supported when mounting as a network drive), and the last example will
|
|
mount as network share \f[C]\[rs]\[rs]cloud\[rs]remote\f[R] and map it
|
|
to an automatically assigned drive:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mount remote:path/to/files *
|
|
rclone mount remote:path/to/files X:
|
|
rclone mount remote:path/to/files C:\[rs]path\[rs]parent\[rs]mount
|
|
rclone mount remote:path/to/files \[rs]\[rs]cloud\[rs]remote
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
When the program ends while in foreground mode, either via Ctrl+C or
|
|
receiving a SIGINT or SIGTERM signal, the mount should be automatically
|
|
stopped.
|
|
.PP
|
|
When running in background mode the user will have to stop the mount
|
|
manually:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
# Linux
|
|
fusermount -u /path/to/local/mount
|
|
# OS X
|
|
umount /path/to/local/mount
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The umount operation can fail, for example when the mountpoint is busy.
|
|
When that happens, it is the user\[aq]s responsibility to stop the mount
|
|
manually.
|
|
.PP
|
|
The size of the mounted file system will be set according to information
|
|
retrieved from the remote, the same as returned by the rclone
|
|
about (https://rclone.org/commands/rclone_about/) command.
|
|
Remotes with unlimited storage may report the used size only, then an
|
|
additional 1 PiB of free space is assumed.
|
|
If the remote does not
|
|
support (https://rclone.org/overview/#optional-features) the about
|
|
feature at all, then 1 PiB is set as both the total and the free size.
|
|
.SS Installing on Windows
|
|
.PP
|
|
To run rclone mount on Windows, you will need to download and install
|
|
WinFsp (http://www.secfs.net/winfsp/).
|
|
.PP
|
|
WinFsp (https://github.com/billziss-gh/winfsp) is an open-source Windows
|
|
File System Proxy which makes it easy to write user space file systems
|
|
for Windows.
|
|
It provides a FUSE emulation layer which rclone uses combination with
|
|
cgofuse (https://github.com/billziss-gh/cgofuse).
|
|
Both of these packages are by Bill Zissimopoulos who was very helpful
|
|
during the implementation of rclone mount for Windows.
|
|
.SS Mounting modes on windows
|
|
.PP
|
|
Unlike other operating systems, Microsoft Windows provides a different
|
|
filesystem type for network and fixed drives.
|
|
It optimises access on the assumption fixed disk drives are fast and
|
|
reliable, while network drives have relatively high latency and less
|
|
reliability.
|
|
Some settings can also be differentiated between the two types, for
|
|
example that Windows Explorer should just display icons and not create
|
|
preview thumbnails for image and video files on network drives.
|
|
.PP
|
|
In most cases, rclone will mount the remote as a normal, fixed disk
|
|
drive by default.
|
|
However, you can also choose to mount it as a remote network drive,
|
|
often described as a network share.
|
|
If you mount an rclone remote using the default, fixed drive mode and
|
|
experience unexpected program errors, freezes or other issues, consider
|
|
mounting as a network drive instead.
|
|
.PP
|
|
When mounting as a fixed disk drive you can either mount to an unused
|
|
drive letter, or to a path representing a \f[B]non-existent\f[R]
|
|
subdirectory of an \f[B]existing\f[R] parent directory or drive.
|
|
Using the special value \f[C]*\f[R] will tell rclone to automatically
|
|
assign the next available drive letter, starting with Z: and moving
|
|
backward.
|
|
Examples:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mount remote:path/to/files *
|
|
rclone mount remote:path/to/files X:
|
|
rclone mount remote:path/to/files C:\[rs]path\[rs]parent\[rs]mount
|
|
rclone mount remote:path/to/files X:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Option \f[C]--volname\f[R] can be used to set a custom volume name for
|
|
the mounted file system.
|
|
The default is to use the remote name and path.
|
|
.PP
|
|
To mount as network drive, you can add option \f[C]--network-mode\f[R]
|
|
to your mount command.
|
|
Mounting to a directory path is not supported in this mode, it is a
|
|
limitation Windows imposes on junctions, so the remote must always be
|
|
mounted to a drive letter.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mount remote:path/to/files X: --network-mode
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
A volume name specified with \f[C]--volname\f[R] will be used to create
|
|
the network share path.
|
|
A complete UNC path, such as \f[C]\[rs]\[rs]cloud\[rs]remote\f[R],
|
|
optionally with path
|
|
\f[C]\[rs]\[rs]cloud\[rs]remote\[rs]madeup\[rs]path\f[R], will be used
|
|
as is.
|
|
Any other string will be used as the share part, after a default prefix
|
|
\f[C]\[rs]\[rs]server\[rs]\f[R].
|
|
If no volume name is specified then \f[C]\[rs]\[rs]server\[rs]share\f[R]
|
|
will be used.
|
|
You must make sure the volume name is unique when you are mounting more
|
|
than one drive, or else the mount command will fail.
|
|
The share name will treated as the volume label for the mapped drive,
|
|
shown in Windows Explorer etc, while the complete
|
|
\f[C]\[rs]\[rs]server\[rs]share\f[R] will be reported as the remote UNC
|
|
path by \f[C]net use\f[R] etc, just like a normal network drive mapping.
|
|
.PP
|
|
If you specify a full network share UNC path with \f[C]--volname\f[R],
|
|
this will implicitely set the \f[C]--network-mode\f[R] option, so the
|
|
following two examples have same result:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mount remote:path/to/files X: --network-mode
|
|
rclone mount remote:path/to/files X: --volname \[rs]\[rs]server\[rs]share
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You may also specify the network share UNC path as the mountpoint
|
|
itself.
|
|
Then rclone will automatically assign a drive letter, same as with
|
|
\f[C]*\f[R] and use that as mountpoint, and instead use the UNC path
|
|
specified as the volume name, as if it were specified with the
|
|
\f[C]--volname\f[R] option.
|
|
This will also implicitely set the \f[C]--network-mode\f[R] option.
|
|
This means the following two examples have same result:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mount remote:path/to/files \[rs]\[rs]cloud\[rs]remote
|
|
rclone mount remote:path/to/files * --volname \[rs]\[rs]cloud\[rs]remote
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
There is yet another way to enable network mode, and to set the share
|
|
path, and that is to pass the \[dq]native\[dq] libfuse/WinFsp option
|
|
directly: \f[C]--fuse-flag --VolumePrefix=\[rs]server\[rs]share\f[R].
|
|
Note that the path must be with just a single backslash prefix in this
|
|
case.
|
|
.PP
|
|
\f[I]Note:\f[R] In previous versions of rclone this was the only
|
|
supported method.
|
|
.PP
|
|
Read more about drive
|
|
mapping (https://en.wikipedia.org/wiki/Drive_mapping)
|
|
.PP
|
|
See also Limitations section below.
|
|
.SS Windows filesystem permissions
|
|
.PP
|
|
The FUSE emulation layer on Windows must convert between the POSIX-based
|
|
permission model used in FUSE, and the permission model used in Windows,
|
|
based on access-control lists (ACL).
|
|
.PP
|
|
The mounted filesystem will normally get three entries in its
|
|
access-control list (ACL), representing permissions for the POSIX
|
|
permission scopes: Owner, group and others.
|
|
By default, the owner and group will be taken from the current user, and
|
|
the built-in group \[dq]Everyone\[dq] will be used to represent others.
|
|
The user/group can be customized with FUSE options \[dq]UserName\[dq]
|
|
and \[dq]GroupName\[dq], e.g.
|
|
\f[C]-o UserName=user123 -o GroupName=\[dq]Authenticated Users\[dq]\f[R].
|
|
The permissions on each entry will be set according to options
|
|
\f[C]--dir-perms\f[R] and \f[C]--file-perms\f[R], which takes a value in
|
|
traditional numeric
|
|
notation (https://en.wikipedia.org/wiki/File-system_permissions#Numeric_notation).
|
|
.PP
|
|
The default permissions corresponds to
|
|
\f[C]--file-perms 0666 --dir-perms 0777\f[R], i.e.
|
|
read and write permissions to everyone.
|
|
This means you will not be able to start any programs from the the
|
|
mount.
|
|
To be able to do that you must add execute permissions, e.g.
|
|
\f[C]--file-perms 0777 --dir-perms 0777\f[R] to add it to everyone.
|
|
If the program needs to write files, chances are you will have to enable
|
|
VFS File Caching as well (see also limitations).
|
|
.PP
|
|
Note that the mapping of permissions is not always trivial, and the
|
|
result you see in Windows Explorer may not be exactly like you expected.
|
|
For example, when setting a value that includes write access, this will
|
|
be mapped to individual permissions \[dq]write attributes\[dq],
|
|
\[dq]write data\[dq] and \[dq]append data\[dq], but not \[dq]write
|
|
extended attributes\[dq].
|
|
Windows will then show this as basic permission \[dq]Special\[dq]
|
|
instead of \[dq]Write\[dq], because \[dq]Write\[dq] includes the
|
|
\[dq]write extended attributes\[dq] permission.
|
|
.PP
|
|
If you set POSIX permissions for only allowing access to the owner,
|
|
using \f[C]--file-perms 0600 --dir-perms 0700\f[R], the user group and
|
|
the built-in \[dq]Everyone\[dq] group will still be given some special
|
|
permissions, such as \[dq]read attributes\[dq] and \[dq]read
|
|
permissions\[dq], in Windows.
|
|
This is done for compatibility reasons, e.g.
|
|
to allow users without additional permissions to be able to read basic
|
|
metadata about files like in UNIX.
|
|
One case that may arise is that other programs (incorrectly) interprets
|
|
this as the file being accessible by everyone.
|
|
For example an SSH client may warn about \[dq]unprotected private key
|
|
file\[dq].
|
|
.PP
|
|
WinFsp 2021 (version 1.9) introduces a new FUSE option
|
|
\[dq]FileSecurity\[dq], that allows the complete specification of file
|
|
security descriptors using
|
|
SDDL (https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format).
|
|
With this you can work around issues such as the mentioned
|
|
\[dq]unprotected private key file\[dq] by specifying
|
|
\f[C]-o FileSecurity=\[dq]D:P(A;;FA;;;OW)\[dq]\f[R], for file all access
|
|
(FA) to the owner (OW).
|
|
.SS Windows caveats
|
|
.PP
|
|
Drives created as Administrator are not visible to other accounts, not
|
|
even an account that was elevated to Administrator with the User Account
|
|
Control (UAC) feature.
|
|
A result of this is that if you mount to a drive letter from a Command
|
|
Prompt run as Administrator, and then try to access the same drive from
|
|
Windows Explorer (which does not run as Administrator), you will not be
|
|
able to see the mounted drive.
|
|
.PP
|
|
If you don\[aq]t need to access the drive from applications running with
|
|
administrative privileges, the easiest way around this is to always
|
|
create the mount from a non-elevated command prompt.
|
|
.PP
|
|
To make mapped drives available to the user account that created them
|
|
regardless if elevated or not, there is a special Windows setting called
|
|
linked
|
|
connections (https://docs.microsoft.com/en-us/troubleshoot/windows-client/networking/mapped-drives-not-available-from-elevated-command#detail-to-configure-the-enablelinkedconnections-registry-entry)
|
|
that can be enabled.
|
|
.PP
|
|
It is also possible to make a drive mount available to everyone on the
|
|
system, by running the process creating it as the built-in SYSTEM
|
|
account.
|
|
There are several ways to do this: One is to use the command-line
|
|
utility
|
|
PsExec (https://docs.microsoft.com/en-us/sysinternals/downloads/psexec),
|
|
from Microsoft\[aq]s Sysinternals suite, which has option \f[C]-s\f[R]
|
|
to start processes as the SYSTEM account.
|
|
Another alternative is to run the mount command from a Windows Scheduled
|
|
Task, or a Windows Service, configured to run as the SYSTEM account.
|
|
A third alternative is to use the WinFsp.Launcher
|
|
infrastructure (https://github.com/billziss-gh/winfsp/wiki/WinFsp-Service-Architecture)).
|
|
Note that when running rclone as another user, it will not use the
|
|
configuration file from your profile unless you tell it to with the
|
|
\f[C]--config\f[R] (https://rclone.org/docs/#config-config-file) option.
|
|
Read more in the install documentation (https://rclone.org/install/).
|
|
.PP
|
|
Note that mapping to a directory path, instead of a drive letter, does
|
|
not suffer from the same limitations.
|
|
.SS Limitations
|
|
.PP
|
|
Without the use of \f[C]--vfs-cache-mode\f[R] this can only write files
|
|
sequentially, it can only seek when reading.
|
|
This means that many applications won\[aq]t work with their files on an
|
|
rclone mount without \f[C]--vfs-cache-mode writes\f[R] or
|
|
\f[C]--vfs-cache-mode full\f[R].
|
|
See the VFS File Caching section for more info.
|
|
.PP
|
|
The bucket-based remotes (e.g.
|
|
Swift, S3, Google Compute Storage, B2, Hubic) do not support the concept
|
|
of empty directories, so empty directories will have a tendency to
|
|
disappear once they fall out of the directory cache.
|
|
.PP
|
|
When \f[C]rclone mount\f[R] is invoked on Unix with \f[C]--daemon\f[R]
|
|
flag, the main rclone program will wait for the background mount to
|
|
become ready or until the timeout specified by the
|
|
\f[C]--daemon-wait\f[R] flag.
|
|
On Linux it can check mount status using ProcFS so the flag in fact sets
|
|
\f[B]maximum\f[R] time to wait, while the real wait can be less.
|
|
On macOS / BSD the time to wait is constant and the check is performed
|
|
only at the end.
|
|
We advise you to set wait time on macOS reasonably.
|
|
.PP
|
|
Only supported on Linux, FreeBSD, OS X and Windows at the moment.
|
|
.SS rclone mount vs rclone sync/copy
|
|
.PP
|
|
File systems expect things to be 100% reliable, whereas cloud storage
|
|
systems are a long way from 100% reliable.
|
|
The rclone sync/copy commands cope with this with lots of retries.
|
|
However rclone mount can\[aq]t use retries in the same way without
|
|
making local copies of the uploads.
|
|
Look at the VFS File Caching for solutions to make mount more reliable.
|
|
.SS Attribute caching
|
|
.PP
|
|
You can use the flag \f[C]--attr-timeout\f[R] to set the time the kernel
|
|
caches the attributes (size, modification time, etc.) for directory
|
|
entries.
|
|
.PP
|
|
The default is \f[C]1s\f[R] which caches files just long enough to avoid
|
|
too many callbacks to rclone from the kernel.
|
|
.PP
|
|
In theory 0s should be the correct value for filesystems which can
|
|
change outside the control of the kernel.
|
|
However this causes quite a few problems such as rclone using too much
|
|
memory (https://github.com/rclone/rclone/issues/2157), rclone not
|
|
serving files to
|
|
samba (https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112)
|
|
and excessive time listing
|
|
directories (https://github.com/rclone/rclone/issues/2095#issuecomment-371141147).
|
|
.PP
|
|
The kernel can cache the info about a file for the time given by
|
|
\f[C]--attr-timeout\f[R].
|
|
You may see corruption if the remote file changes length during this
|
|
window.
|
|
It will show up as either a truncated file or a file with garbage on the
|
|
end.
|
|
With \f[C]--attr-timeout 1s\f[R] this is very unlikely but not
|
|
impossible.
|
|
The higher you set \f[C]--attr-timeout\f[R] the more likely it is.
|
|
The default setting of \[dq]1s\[dq] is the lowest setting which
|
|
mitigates the problems above.
|
|
.PP
|
|
If you set it higher (\f[C]10s\f[R] or \f[C]1m\f[R] say) then the kernel
|
|
will call back to rclone less often making it more efficient, however
|
|
there is more chance of the corruption issue above.
|
|
.PP
|
|
If files don\[aq]t change on the remote outside of the control of rclone
|
|
then there is no chance of corruption.
|
|
.PP
|
|
This is the same as setting the attr_timeout option in mount.fuse.
|
|
.SS Filters
|
|
.PP
|
|
Note that all the rclone filters can be used to select a subset of the
|
|
files to be visible in the mount.
|
|
.SS systemd
|
|
.PP
|
|
When running rclone mount as a systemd service, it is possible to use
|
|
Type=notify.
|
|
In this case the service will enter the started state after the
|
|
mountpoint has been successfully set up.
|
|
Units having the rclone mount service specified as a requirement will
|
|
see all files and folders immediately in this mode.
|
|
.PP
|
|
Note that systemd runs mount units without any environment variables
|
|
including \f[C]PATH\f[R] or \f[C]HOME\f[R].
|
|
This means that tilde (\f[C]\[ti]\f[R]) expansion will not work and you
|
|
should provide \f[C]--config\f[R] and \f[C]--cache-dir\f[R] explicitly
|
|
as absolute paths via rclone arguments.
|
|
Since mounting requires the \f[C]fusermount\f[R] program, rclone will
|
|
use the fallback PATH of \f[C]/bin:/usr/bin\f[R] in this scenario.
|
|
Please ensure that \f[C]fusermount\f[R] is present on this PATH.
|
|
.SS Rclone as Unix mount helper
|
|
.PP
|
|
The core Unix program \f[C]/bin/mount\f[R] normally takes the
|
|
\f[C]-t FSTYPE\f[R] argument then runs the \f[C]/sbin/mount.FSTYPE\f[R]
|
|
helper program passing it mount options as \f[C]-o key=val,...\f[R] or
|
|
\f[C]--opt=...\f[R].
|
|
Automount (classic or systemd) behaves in a similar way.
|
|
.PP
|
|
rclone by default expects GNU-style flags \f[C]--key val\f[R].
|
|
To run it as a mount helper you should symlink rclone binary to
|
|
\f[C]/sbin/mount.rclone\f[R] and optionally \f[C]/usr/bin/rclonefs\f[R],
|
|
e.g.
|
|
\f[C]ln -s /usr/bin/rclone /sbin/mount.rclone\f[R].
|
|
rclone will detect it and translate command-line arguments
|
|
appropriately.
|
|
.PP
|
|
Now you can run classic mounts like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
mount sftp1:subdir /mnt/data -t rclone -o vfs_cache_mode=writes,sftp_key_file=/path/to/pem
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
or create systemd mount units:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
# /etc/systemd/system/mnt-data.mount
|
|
[Unit]
|
|
After=network-online.target
|
|
[Mount]
|
|
Type=rclone
|
|
What=sftp1:subdir
|
|
Where=/mnt/data
|
|
Options=rw,allow_other,args2env,vfs-cache-mode=writes,config=/etc/rclone.conf,cache-dir=/var/rclone
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
optionally accompanied by systemd automount unit
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
# /etc/systemd/system/mnt-data.automount
|
|
[Unit]
|
|
After=network-online.target
|
|
Before=remote-fs.target
|
|
[Automount]
|
|
Where=/mnt/data
|
|
TimeoutIdleSec=600
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
or add in \f[C]/etc/fstab\f[R] a line like
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sftp1:subdir /mnt/data rclone rw,noauto,nofail,_netdev,x-systemd.automount,args2env,vfs_cache_mode=writes,config=/etc/rclone.conf,cache_dir=/var/cache/rclone 0 0
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
or use classic Automountd.
|
|
Remember to provide explicit \f[C]config=...,cache-dir=...\f[R] as a
|
|
workaround for mount units being run without \f[C]HOME\f[R].
|
|
.PP
|
|
Rclone in the mount helper mode will split \f[C]-o\f[R] argument(s) by
|
|
comma, replace \f[C]_\f[R] by \f[C]-\f[R] and prepend \f[C]--\f[R] to
|
|
get the command-line flags.
|
|
Options containing commas or spaces can be wrapped in single or double
|
|
quotes.
|
|
Any inner quotes inside outer quotes of the same type should be doubled.
|
|
.PP
|
|
Mount option syntax includes a few extra options treated specially:
|
|
.IP \[bu] 2
|
|
\f[C]env.NAME=VALUE\f[R] will set an environment variable for the mount
|
|
process.
|
|
This helps with Automountd and Systemd.mount which don\[aq]t allow
|
|
setting custom environment for mount helpers.
|
|
Typically you will use \f[C]env.HTTPS_PROXY=proxy.host:3128\f[R] or
|
|
\f[C]env.HOME=/root\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]command=cmount\f[R] can be used to run \f[C]cmount\f[R] or any
|
|
other rclone command rather than the default \f[C]mount\f[R].
|
|
.IP \[bu] 2
|
|
\f[C]args2env\f[R] will pass mount options to the mount helper running
|
|
in background via environment variables instead of command line
|
|
arguments.
|
|
This allows to hide secrets from such commands as \f[C]ps\f[R] or
|
|
\f[C]pgrep\f[R].
|
|
.IP \[bu] 2
|
|
\f[C]vv...\f[R] will be transformed into appropriate
|
|
\f[C]--verbose=N\f[R]
|
|
.IP \[bu] 2
|
|
standard mount options like \f[C]x-systemd.automount\f[R],
|
|
\f[C]_netdev\f[R], \f[C]nosuid\f[R] and alike are intended only for
|
|
Automountd and ignored by rclone.
|
|
.SS VFS - Virtual File System
|
|
.PP
|
|
This command uses the VFS layer.
|
|
This adapts the cloud storage objects that rclone uses into something
|
|
which looks much more like a disk filing system.
|
|
.PP
|
|
Cloud storage objects have lots of properties which aren\[aq]t like disk
|
|
files - you can\[aq]t extend them or write to the middle of them, so the
|
|
VFS layer has to deal with that.
|
|
Because there is no one right way of doing this there are various
|
|
options explained below.
|
|
.PP
|
|
The VFS layer also implements a directory cache - this caches info about
|
|
files and directories (but not the data) in memory.
|
|
.SS VFS Directory Cache
|
|
.PP
|
|
Using the \f[C]--dir-cache-time\f[R] flag, you can control how long a
|
|
directory should be considered up to date and not refreshed from the
|
|
backend.
|
|
Changes made through the mount will appear immediately or invalidate the
|
|
cache.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
However, changes made directly on the cloud storage by the web interface
|
|
or a different copy of rclone will only be picked up once the directory
|
|
cache expires if the backend configured does not support polling for
|
|
changes.
|
|
If the backend supports polling, changes will be picked up within the
|
|
polling interval.
|
|
.PP
|
|
You can send a \f[C]SIGHUP\f[R] signal to rclone for it to flush all
|
|
directory caches, regardless of how old they are.
|
|
Assuming only one rclone instance is running, you can reset the cache
|
|
like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill -SIGHUP $(pidof rclone)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control then you can use rclone rc
|
|
to flush the whole directory cache:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or individual files or directories:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget file=path/to/file dir=path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.SS VFS File Buffering
|
|
.PP
|
|
The \f[C]--buffer-size\f[R] flag determines the amount of memory, that
|
|
will be used to buffer data in advance.
|
|
.PP
|
|
Each open file will try to keep the specified amount of data in memory
|
|
at all times.
|
|
The buffered data is bound to one open file and won\[aq]t be shared.
|
|
.PP
|
|
This flag is a upper limit for the used memory per open file.
|
|
The buffer will only use memory for data that is downloaded but not not
|
|
yet read.
|
|
If the buffer is empty, only a small amount of memory will be used.
|
|
.PP
|
|
The maximum memory used by rclone for buffering can be up to
|
|
\f[C]--buffer-size * open files\f[R].
|
|
.SS VFS File Caching
|
|
.PP
|
|
These flags control the VFS file caching options.
|
|
File caching is necessary to make the VFS layer appear compatible with a
|
|
normal file system.
|
|
It can be disabled at the cost of some compatibility.
|
|
.PP
|
|
For example you\[aq]ll need to enable VFS caching if you want to read
|
|
and write simultaneously to a file.
|
|
See below for more details.
|
|
.PP
|
|
Note that the VFS cache is separate from the cache backend and you may
|
|
find that you need one or the other or both.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--cache-dir string Directory rclone will use for caching.
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If run with \f[C]-vv\f[R] rclone will print the location of the file
|
|
cache.
|
|
The files are stored in the user cache file area which is OS dependent
|
|
but can be controlled with \f[C]--cache-dir\f[R] or setting the
|
|
appropriate environment variable.
|
|
.PP
|
|
The cache has 4 different modes selected by \f[C]--vfs-cache-mode\f[R].
|
|
The higher the cache mode the more compatible rclone becomes at the cost
|
|
of using disk space.
|
|
.PP
|
|
Note that files are written back to the remote only when they are closed
|
|
and if they haven\[aq]t been accessed for \f[C]--vfs-write-back\f[R]
|
|
seconds.
|
|
If rclone is quit or dies with files that haven\[aq]t been uploaded,
|
|
these will be uploaded next time rclone is run with the same flags.
|
|
.PP
|
|
If using \f[C]--vfs-cache-max-size\f[R] note that the cache may exceed
|
|
this size for two reasons.
|
|
Firstly because it is only checked every
|
|
\f[C]--vfs-cache-poll-interval\f[R].
|
|
Secondly because open files cannot be evicted from the cache.
|
|
.PP
|
|
You \f[B]should not\f[R] run two copies of rclone using the same VFS
|
|
cache with the same or overlapping remotes if using
|
|
\f[C]--vfs-cache-mode > off\f[R].
|
|
This can potentially cause data corruption if you do.
|
|
You can work around this by giving each rclone its own cache hierarchy
|
|
with \f[C]--cache-dir\f[R].
|
|
You don\[aq]t need to worry about this if the remotes in use don\[aq]t
|
|
overlap.
|
|
.SS --vfs-cache-mode off
|
|
.PP
|
|
In this mode (the default) the cache will read directly from the remote
|
|
and write directly to the remote without caching anything on disk.
|
|
.PP
|
|
This will mean some operations are not possible
|
|
.IP \[bu] 2
|
|
Files can\[aq]t be opened for both read AND write
|
|
.IP \[bu] 2
|
|
Files opened for write can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files open for read with O_TRUNC will be opened write only
|
|
.IP \[bu] 2
|
|
Files open for write only will behave as if O_TRUNC was supplied
|
|
.IP \[bu] 2
|
|
Open modes O_APPEND, O_TRUNC are ignored
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode minimal
|
|
.PP
|
|
This is very similar to \[dq]off\[dq] except that files opened for read
|
|
AND write will be buffered to disk.
|
|
This means that files opened for write will be a lot more compatible,
|
|
but uses the minimal disk space.
|
|
.PP
|
|
These operations are not possible
|
|
.IP \[bu] 2
|
|
Files opened for write only can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files opened for write only will ignore O_APPEND, O_TRUNC
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode writes
|
|
.PP
|
|
In this mode files opened for read only are still read directly from the
|
|
remote, write only and read/write files are buffered to disk first.
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload fails it will be retried at exponentially increasing
|
|
intervals up to 1 minute.
|
|
.SS --vfs-cache-mode full
|
|
.PP
|
|
In this mode all reads and writes are buffered to and from disk.
|
|
When data is read from the remote this is buffered to disk as well.
|
|
.PP
|
|
In this mode the files in the cache will be sparse files and rclone will
|
|
keep track of which bits of the files it has downloaded.
|
|
.PP
|
|
So if an application only reads the starts of each file, then rclone
|
|
will only buffer the start of the file.
|
|
These files will appear to be their full size in the cache, but they
|
|
will be sparse files with only the data that has been downloaded present
|
|
in them.
|
|
.PP
|
|
This mode should support all normal file system operations and is
|
|
otherwise identical to \f[C]--vfs-cache-mode\f[R] writes.
|
|
.PP
|
|
When reading a file rclone will read \f[C]--buffer-size\f[R] plus
|
|
\f[C]--vfs-read-ahead\f[R] bytes ahead.
|
|
The \f[C]--buffer-size\f[R] is buffered in memory whereas the
|
|
\f[C]--vfs-read-ahead\f[R] is buffered on disk.
|
|
.PP
|
|
When using this mode it is recommended that \f[C]--buffer-size\f[R] is
|
|
not set too large and \f[C]--vfs-read-ahead\f[R] is set large if
|
|
required.
|
|
.PP
|
|
\f[B]IMPORTANT\f[R] not all file systems support sparse files.
|
|
In particular FAT/exFAT do not.
|
|
Rclone will perform very badly if the cache directory is on a filesystem
|
|
which doesn\[aq]t support sparse files and it will log an ERROR message
|
|
if one is detected.
|
|
.SS VFS Chunked Reading
|
|
.PP
|
|
When rclone reads files from a remote it reads them in chunks.
|
|
This means that rather than requesting the whole file rclone reads the
|
|
chunk specified.
|
|
This can reduce the used download quota for some remotes by requesting
|
|
only chunks from the remote that are actually read, at the cost of an
|
|
increased number of requests.
|
|
.PP
|
|
These flags control the chunking:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
|
|
--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Rclone will start reading a chunk of size
|
|
\f[C]--vfs-read-chunk-size\f[R], and then double the size for each read.
|
|
When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater
|
|
than \f[C]--vfs-read-chunk-size\f[R], the chunk size for each open file
|
|
will get doubled only until the specified value is reached.
|
|
If the value is \[dq]off\[dq], which is the default, the limit is
|
|
disabled and the chunk size will grow indefinitely.
|
|
.PP
|
|
With \f[C]--vfs-read-chunk-size 100M\f[R] and
|
|
\f[C]--vfs-read-chunk-size-limit 0\f[R] the following parts will be
|
|
downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
|
|
When \f[C]--vfs-read-chunk-size-limit 500M\f[R] is specified, the result
|
|
would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so
|
|
on.
|
|
.PP
|
|
Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq]
|
|
disables chunked reading.
|
|
.SS VFS Performance
|
|
.PP
|
|
These flags may be used to enable/disable features of the VFS for
|
|
performance or other reasons.
|
|
See also the chunked reading feature.
|
|
.PP
|
|
In particular S3 and Swift benefit hugely from the
|
|
\f[C]--no-modtime\f[R] flag (or use \f[C]--use-server-modtime\f[R] for a
|
|
slightly different effect) as each read of the modification time takes a
|
|
transaction.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--no-checksum Don\[aq]t compare checksums on up/download.
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up).
|
|
--no-seek Don\[aq]t allow seeking in files.
|
|
--read-only Mount read-only.
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sometimes rclone is delivered reads or writes out of order.
|
|
Rather than seeking rclone will wait a short time for the in sequence
|
|
read or write to come in.
|
|
These flags only come into effect when not using an on disk cache file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
When using VFS write caching (\f[C]--vfs-cache-mode\f[R] with value
|
|
writes or full), the global flag \f[C]--transfers\f[R] can be set to
|
|
adjust the number of parallel uploads of modified files from cache (the
|
|
related global flag \f[C]--checkers\f[R] have no effect on mount).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--transfers int Number of file transfers to run in parallel (default 4)
|
|
\f[R]
|
|
.fi
|
|
.SS VFS Case Sensitivity
|
|
.PP
|
|
Linux file systems are case-sensitive: two files can differ only by
|
|
case, and the exact case must be used when opening a file.
|
|
.PP
|
|
File systems in modern Windows are case-insensitive but case-preserving:
|
|
although existing files can be opened using any case, the exact case
|
|
used to create the file is preserved and available for programs to
|
|
query.
|
|
It is not allowed for two files in the same directory to differ only by
|
|
case.
|
|
.PP
|
|
Usually file systems on macOS are case-insensitive.
|
|
It is possible to make macOS file systems case-sensitive but that is not
|
|
the default.
|
|
.PP
|
|
The \f[C]--vfs-case-insensitive\f[R] mount flag controls how rclone
|
|
handles these two cases.
|
|
If its value is \[dq]false\[dq], rclone passes file names to the mounted
|
|
file system as-is.
|
|
If the flag is \[dq]true\[dq] (or appears without a value on command
|
|
line), rclone may perform a \[dq]fixup\[dq] as explained below.
|
|
.PP
|
|
The user may specify a file name to open/delete/rename/etc with a case
|
|
different than what is stored on mounted file system.
|
|
If an argument refers to an existing file with exactly the same name,
|
|
then the case of the existing file on the disk will be used.
|
|
However, if a file name with exactly the same name is not found but a
|
|
name differing only by case exists, rclone will transparently fixup the
|
|
name.
|
|
This fixup happens only when an existing file is requested.
|
|
Case sensitivity of file names created anew by rclone is controlled by
|
|
an underlying mounted file system.
|
|
.PP
|
|
Note that case sensitivity of the operating system running rclone (the
|
|
target) may differ from case sensitivity of a file system mounted by
|
|
rclone (the source).
|
|
The flag controls whether \[dq]fixup\[dq] is performed to satisfy the
|
|
target.
|
|
.PP
|
|
If the flag is not provided on the command line, then its default value
|
|
depends on the operating system where rclone runs: \[dq]true\[dq] on
|
|
Windows and macOS, \[dq]false\[dq] otherwise.
|
|
If the flag is provided without a value, then it is \[dq]true\[dq].
|
|
.SS Alternate report of used bytes
|
|
.PP
|
|
Some backends, most notably S3, do not report the amount of bytes used.
|
|
If you need this information to be available when running \f[C]df\f[R]
|
|
on the filesystem, then pass the flag \f[C]--vfs-used-is-size\f[R] to
|
|
rclone.
|
|
With this flag set, instead of relying on the backend to report this
|
|
information, rclone will scan the whole remote similar to
|
|
\f[C]rclone size\f[R] and compute the total used space itself.
|
|
.PP
|
|
\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
|
|
filters so that the result is accurate.
|
|
However, this is very inefficient and may cost lots of API calls
|
|
resulting in extra charges.
|
|
Use it as a last resort and only with caching.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mount remote:path /path/to/mountpoint [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--allow-non-empty Allow mounting over a non-empty directory (not supported on Windows)
|
|
--allow-other Allow access to other users (not supported on Windows)
|
|
--allow-root Allow access to root user (not supported on Windows)
|
|
--async-read Use asynchronous reads (not supported on Windows) (default true)
|
|
--attr-timeout duration Time for which file/directory attributes are cached (default 1s)
|
|
--daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows)
|
|
--daemon-timeout duration Time limit for rclone to respond to kernel (not supported on Windows)
|
|
--daemon-wait duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s)
|
|
--debug-fuse Debug the FUSE internals - needs -v
|
|
--default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows)
|
|
--devname string Set the device name - default is remote:path
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--dir-perms FileMode Directory permissions (default 0777)
|
|
--file-perms FileMode File permissions (default 0666)
|
|
--fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required)
|
|
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
|
|
-h, --help help for mount
|
|
--max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki)
|
|
--network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only)
|
|
--no-checksum Don\[aq]t compare checksums on up/download
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up)
|
|
--no-seek Don\[aq]t allow seeking in files
|
|
--noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true)
|
|
--noapplexattr Ignore all \[dq]com.apple.*\[dq] extended attributes (supported on OSX only)
|
|
-o, --option stringArray Option for libfuse/WinFsp (repeat if required)
|
|
--poll-interval duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
|
|
--read-only Mount read-only
|
|
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
|
|
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-case-insensitive If a file name not found, find a case insensitive match
|
|
--vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
|
|
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off)
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
--volname string Set the volume name (supported on Windows and OSX only)
|
|
--write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone moveto
|
|
.PP
|
|
Move file or directory from source to dest.
|
|
.SS Synopsis
|
|
.PP
|
|
If source:path is a file or directory then it moves it to a file or
|
|
directory named dest:path.
|
|
.PP
|
|
This can be used to rename files or upload single files to other than
|
|
their existing name.
|
|
If the source is a directory then it acts exactly like the move command.
|
|
.PP
|
|
So
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone moveto src dst
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
where src and dst are rclone paths, either remote:path or /path/to/local
|
|
or C:.
|
|
.PP
|
|
This will:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
if src is file
|
|
move it to dst, overwriting an existing file if it exists
|
|
if src is directory
|
|
move it to dst, overwriting existing files if they exist
|
|
see move command for full details
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This doesn\[aq]t transfer files that are identical on src and dst,
|
|
testing by size and modification time or MD5SUM.
|
|
src will be deleted on successful transfer.
|
|
.PP
|
|
\f[B]Important\f[R]: Since this can cause data loss, test first with the
|
|
\f[C]--dry-run\f[R] or the \f[C]--interactive\f[R]/\f[C]-i\f[R] flag.
|
|
.PP
|
|
\f[B]Note\f[R]: Use the \f[C]-P\f[R]/\f[C]--progress\f[R] flag to view
|
|
real-time transfer statistics.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone moveto source:path dest:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for moveto
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone ncdu
|
|
.PP
|
|
Explore a remote with a text based user interface.
|
|
.SS Synopsis
|
|
.PP
|
|
This displays a text based user interface allowing the navigation of a
|
|
remote.
|
|
It is most useful for answering the question - \[dq]What is using all my
|
|
disk space?\[dq].
|
|
.PP
|
|
To make the user interface it first scans the entire remote given and
|
|
builds an in memory representation.
|
|
rclone ncdu can be used during this scanning phase and you will see it
|
|
building up the directory structure as it goes along.
|
|
.PP
|
|
Here are the keys - press \[aq]?\[aq] to toggle the help on and off
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\[ua],\[da] or k,j to Move
|
|
\[->],l to enter
|
|
\[<-],h to return
|
|
c toggle counts
|
|
g toggle graph
|
|
a toggle average size in directory
|
|
u toggle human-readable format
|
|
n,s,C,A sort by name,size,count,average size
|
|
d delete file/directory
|
|
y copy current path to clipboard
|
|
Y display current path
|
|
\[ha]L refresh screen
|
|
? to toggle help on and off
|
|
q/ESC/c-C to quit
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This an homage to the ncdu tool (https://dev.yorhel.nl/ncdu) but for
|
|
rclone remotes.
|
|
It is missing lots of features at the moment but is useful as it stands.
|
|
.PP
|
|
Note that it might take some time to delete big files/folders.
|
|
The UI won\[aq]t respond in the meantime since the deletion is done
|
|
synchronously.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ncdu remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for ncdu
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone obscure
|
|
.PP
|
|
Obscure password for use in the rclone config file.
|
|
.SS Synopsis
|
|
.PP
|
|
In the rclone config file, human-readable passwords are obscured.
|
|
Obscuring them is done by encrypting them and writing them out in
|
|
base64.
|
|
This is \f[B]not\f[R] a secure way of encrypting these passwords as
|
|
rclone can decrypt them - it is to prevent \[dq]eyedropping\[dq] -
|
|
namely someone seeing a password in the rclone config file by accident.
|
|
.PP
|
|
Many equally important things (like access tokens) are not obscured in
|
|
the config file.
|
|
However it is very hard to shoulder surf a 64 character hex token.
|
|
.PP
|
|
This command can also accept a password through STDIN instead of an
|
|
argument by passing a hyphen as an argument.
|
|
This will use the first line of STDIN as the password not including the
|
|
trailing newline.
|
|
.PP
|
|
echo \[dq]secretpassword\[dq] | rclone obscure -
|
|
.PP
|
|
If there is no data on STDIN to read, rclone obscure will default to
|
|
obfuscating the hyphen itself.
|
|
.PP
|
|
If you want to encrypt the config file then please use config file
|
|
encryption - see rclone
|
|
config (https://rclone.org/commands/rclone_config/) for more info.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone obscure password [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for obscure
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone rc
|
|
.PP
|
|
Run a command against a running rclone.
|
|
.SS Synopsis
|
|
.PP
|
|
This runs a command against a running rclone.
|
|
Use the --url flag to specify an non default URL to connect on.
|
|
This can be either a \[dq]:port\[dq] which is taken to mean
|
|
\[dq]http://localhost:port\[dq] or a \[dq]host:port\[dq] which is taken
|
|
to mean \[dq]http://host:port\[dq]
|
|
.PP
|
|
A username and password can be passed in with --user and --pass.
|
|
.PP
|
|
Note that --rc-addr, --rc-user, --rc-pass will be read also for --url,
|
|
--user, --pass.
|
|
.PP
|
|
Arguments should be passed in as parameter=value.
|
|
.PP
|
|
The result will be returned as a JSON object by default.
|
|
.PP
|
|
The --json parameter can be used to pass in a JSON blob as an input
|
|
instead of key=value arguments.
|
|
This is the only way of passing in more complicated values.
|
|
.PP
|
|
The -o/--opt option can be used to set a key \[dq]opt\[dq] with key,
|
|
value options in the form \[dq]-o key=value\[dq] or \[dq]-o key\[dq].
|
|
It can be repeated as many times as required.
|
|
This is useful for rc commands which take the \[dq]opt\[dq] parameter
|
|
which by convention is a dictionary of strings.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-o key=value -o key2
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Will place this in the \[dq]opt\[dq] value
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{\[dq]key\[dq]:\[dq]value\[dq], \[dq]key2\[dq],\[dq]\[dq])
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The -a/--arg option can be used to set strings in the \[dq]arg\[dq]
|
|
value.
|
|
It can be repeated as many times as required.
|
|
This is useful for rc commands which take the \[dq]arg\[dq] parameter
|
|
which by convention is a list of strings.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-a value -a value2
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Will place this in the \[dq]arg\[dq] value
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[\[dq]value\[dq], \[dq]value2\[dq]]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Use --loopback to connect to the rclone instance running \[dq]rclone
|
|
rc\[dq].
|
|
This is very useful for testing commands without having to run an rclone
|
|
rc server, e.g.:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc --loopback operations/about fs=/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Use \[dq]rclone rc\[dq] to see a list of all possible commands.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc commands parameter [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-a, --arg stringArray Argument placed in the \[dq]arg\[dq] array
|
|
-h, --help help for rc
|
|
--json string Input JSON - use instead of key=value args
|
|
--loopback If set connect to this rclone instance not via HTTP
|
|
--no-output If set, don\[aq]t output the JSON result
|
|
-o, --opt stringArray Option in the form name=value or name placed in the \[dq]opt\[dq] array
|
|
--pass string Password to use to connect to rclone remote control
|
|
--url string URL to connect to rclone remote control (default \[dq]http://localhost:5572/\[dq])
|
|
--user string Username to use to rclone remote control
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone rcat
|
|
.PP
|
|
Copies standard input to file on remote.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone rcat reads from standard input (stdin) and copies it to a single
|
|
remote file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
echo \[dq]hello world\[dq] | rclone rcat remote:path/to/file
|
|
ffmpeg - | rclone rcat remote:path/to/file
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If the remote file already exists, it will be overwritten.
|
|
.PP
|
|
rcat will try to upload small files in a single request, which is
|
|
usually more efficient than the streaming/chunked upload endpoints,
|
|
which use multiple requests.
|
|
Exact behaviour depends on the remote.
|
|
What is considered a small file may be set through
|
|
\f[C]--streaming-upload-cutoff\f[R].
|
|
Uploading only starts after the cutoff is reached or if the file ends
|
|
before that.
|
|
The data must fit into RAM.
|
|
The cutoff needs to be small enough to adhere the limits of your remote,
|
|
please see there.
|
|
Generally speaking, setting this cutoff too high will decrease your
|
|
performance.
|
|
.PP
|
|
Use the |--size| flag to preallocate the file in advance at the remote
|
|
end and actually stream it, even if remote backend doesn\[aq]t support
|
|
streaming.
|
|
.PP
|
|
|--size| should be the exact size of the input stream in bytes.
|
|
If the size of the stream is different in length to the |--size| passed
|
|
in then the transfer will likely fail.
|
|
.PP
|
|
Note that the upload can also not be retried because the data is not
|
|
kept around until the upload succeeds.
|
|
If you need to transfer a lot of data, you\[aq]re better off caching
|
|
locally and then \f[C]rclone move\f[R] it to the destination.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rcat remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for rcat
|
|
--size int File size hint to preallocate (default -1)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone rcd
|
|
.PP
|
|
Run rclone listening to remote control commands only.
|
|
.SS Synopsis
|
|
.PP
|
|
This runs rclone so that it only listens to remote control commands.
|
|
.PP
|
|
This is useful if you are controlling rclone via the rc API.
|
|
.PP
|
|
If you pass in a path to a directory, rclone will serve that directory
|
|
for GET requests on the URL passed in.
|
|
It will also open the URL in the browser when rclone is run.
|
|
.PP
|
|
See the rc documentation (https://rclone.org/rc/) for more info on the
|
|
rc flags.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rcd <path to files to serve>* [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for rcd
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone rmdirs
|
|
.PP
|
|
Remove empty directories under the path.
|
|
.SS Synopsis
|
|
.PP
|
|
This recursively removes any empty directories (including directories
|
|
that only contain empty directories), that it finds under the path.
|
|
The root path itself will also be removed if it is empty, unless you
|
|
supply the \f[C]--leave-root\f[R] flag.
|
|
.PP
|
|
Use command \f[C]rmdir\f[R] to delete just the empty directory given by
|
|
path, not recurse.
|
|
.PP
|
|
This is useful for tidying up remotes that rclone has left a lot of
|
|
empty directories in.
|
|
For example the \f[C]delete\f[R] command will delete files but leave the
|
|
directory structure (unless used with option \f[C]--rmdirs\f[R]).
|
|
.PP
|
|
To delete a path and any objects in it, use \f[C]purge\f[R] command.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rmdirs remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for rmdirs
|
|
--leave-root Do not remove root directory if empty
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone selfupdate
|
|
.PP
|
|
Update the rclone binary.
|
|
.SS Synopsis
|
|
.PP
|
|
This command downloads the latest release of rclone and replaces the
|
|
currently running binary.
|
|
The download is verified with a hashsum and cryptographically signed
|
|
signature.
|
|
.PP
|
|
If used without flags (or with implied \f[C]--stable\f[R] flag), this
|
|
command will install the latest stable release.
|
|
However, some issues may be fixed (or features added) only in the latest
|
|
beta release.
|
|
In such cases you should run the command with the \f[C]--beta\f[R] flag,
|
|
i.e.
|
|
\f[C]rclone selfupdate --beta\f[R].
|
|
You can check in advance what version would be installed by adding the
|
|
\f[C]--check\f[R] flag, then repeat the command without it when you are
|
|
satisfied.
|
|
.PP
|
|
Sometimes the rclone team may recommend you a concrete beta or stable
|
|
rclone release to troubleshoot your issue or add a bleeding edge
|
|
feature.
|
|
The \f[C]--version VER\f[R] flag, if given, will update to the concrete
|
|
version instead of the latest one.
|
|
If you omit micro version from \f[C]VER\f[R] (for example
|
|
\f[C]1.53\f[R]), the latest matching micro version will be used.
|
|
.PP
|
|
Upon successful update rclone will print a message that contains a
|
|
previous version number.
|
|
You will need it if you later decide to revert your update for some
|
|
reason.
|
|
Then you\[aq]ll have to note the previous version and run the following
|
|
command: \f[C]rclone selfupdate [--beta] OLDVER\f[R].
|
|
If the old version contains only dots and digits (for example
|
|
\f[C]v1.54.0\f[R]) then it\[aq]s a stable release so you won\[aq]t need
|
|
the \f[C]--beta\f[R] flag.
|
|
Beta releases have an additional information similar to
|
|
\f[C]v1.54.0-beta.5111.06f1c0c61\f[R].
|
|
(if you are a developer and use a locally built rclone, the version
|
|
number will end with \f[C]-DEV\f[R], you will have to rebuild it as it
|
|
obviously can\[aq]t be distributed).
|
|
.PP
|
|
If you previously installed rclone via a package manager, the package
|
|
may include local documentation or configure services.
|
|
You may wish to update with the flag \f[C]--package deb\f[R] or
|
|
\f[C]--package rpm\f[R] (whichever is correct for your OS) to update
|
|
these too.
|
|
This command with the default \f[C]--package zip\f[R] will update only
|
|
the rclone executable so the local manual may become inaccurate after
|
|
it.
|
|
.PP
|
|
The \f[C]rclone mount\f[R] command
|
|
(https://rclone.org/commands/rclone_mount/) may or may not support
|
|
extended FUSE options depending on the build and OS.
|
|
\f[C]selfupdate\f[R] will refuse to update if the capability would be
|
|
discarded.
|
|
.PP
|
|
Note: Windows forbids deletion of a currently running executable so this
|
|
command will rename the old executable to \[aq]rclone.old.exe\[aq] upon
|
|
success.
|
|
.PP
|
|
Please note that this command was not available before rclone version
|
|
1.55.
|
|
If it fails for you with the message
|
|
\f[C]unknown command \[dq]selfupdate\[dq]\f[R] then you will need to
|
|
update manually following the install instructions located at
|
|
https://rclone.org/install/
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone selfupdate [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--beta Install beta release
|
|
--check Check for latest release, do not download
|
|
-h, --help help for selfupdate
|
|
--output string Save the downloaded binary at a given path (default: replace running binary)
|
|
--package string Package format: zip|deb|rpm (default: zip)
|
|
--stable Install stable release (this is the default)
|
|
--version string Install the given rclone version (default: latest)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone serve
|
|
.PP
|
|
Serve a remote over a protocol.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone serve is used to serve a remote over a given protocol.
|
|
This command requires the use of a subcommand to specify the protocol,
|
|
e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone serve http remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Each subcommand has its own options which you can see in their help.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone serve <protocol> [opts] <remote> [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for serve
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.IP \[bu] 2
|
|
rclone serve dlna (https://rclone.org/commands/rclone_serve_dlna/) -
|
|
Serve remote:path over DLNA
|
|
.IP \[bu] 2
|
|
rclone serve docker (https://rclone.org/commands/rclone_serve_docker/) -
|
|
Serve any remote on docker\[aq]s volume plugin API.
|
|
.IP \[bu] 2
|
|
rclone serve ftp (https://rclone.org/commands/rclone_serve_ftp/) - Serve
|
|
remote:path over FTP.
|
|
.IP \[bu] 2
|
|
rclone serve http (https://rclone.org/commands/rclone_serve_http/) -
|
|
Serve the remote over HTTP.
|
|
.IP \[bu] 2
|
|
rclone serve restic (https://rclone.org/commands/rclone_serve_restic/) -
|
|
Serve the remote for restic\[aq]s REST API.
|
|
.IP \[bu] 2
|
|
rclone serve sftp (https://rclone.org/commands/rclone_serve_sftp/) -
|
|
Serve the remote over SFTP.
|
|
.IP \[bu] 2
|
|
rclone serve webdav (https://rclone.org/commands/rclone_serve_webdav/) -
|
|
Serve remote:path over webdav.
|
|
.SH rclone serve dlna
|
|
.PP
|
|
Serve remote:path over DLNA
|
|
.SS Synopsis
|
|
.PP
|
|
rclone serve dlna is a DLNA media server for media stored in an rclone
|
|
remote.
|
|
Many devices, such as the Xbox and PlayStation, can automatically
|
|
discover this server in the LAN and play audio/video from it.
|
|
VLC is also supported.
|
|
Service discovery uses UDP multicast packets (SSDP) and will thus only
|
|
work on LANs.
|
|
.PP
|
|
Rclone will list all files present in the remote, without filtering
|
|
based on media formats or file extensions.
|
|
Additionally, there is no media transcoding support.
|
|
This means that some players might show files that they are not able to
|
|
play back correctly.
|
|
.SS Server options
|
|
.PP
|
|
Use \f[C]--addr\f[R] to specify which IP address and port the server
|
|
should listen on, e.g.
|
|
\f[C]--addr 1.2.3.4:8000\f[R] or \f[C]--addr :8080\f[R] to listen to all
|
|
IPs.
|
|
.PP
|
|
Use \f[C]--name\f[R] to choose the friendly server name, which is by
|
|
default \[dq]rclone (hostname)\[dq].
|
|
.PP
|
|
Use \f[C]--log-trace\f[R] in conjunction with \f[C]-vv\f[R] to enable
|
|
additional debug logging of all UPNP traffic.
|
|
.SS VFS - Virtual File System
|
|
.PP
|
|
This command uses the VFS layer.
|
|
This adapts the cloud storage objects that rclone uses into something
|
|
which looks much more like a disk filing system.
|
|
.PP
|
|
Cloud storage objects have lots of properties which aren\[aq]t like disk
|
|
files - you can\[aq]t extend them or write to the middle of them, so the
|
|
VFS layer has to deal with that.
|
|
Because there is no one right way of doing this there are various
|
|
options explained below.
|
|
.PP
|
|
The VFS layer also implements a directory cache - this caches info about
|
|
files and directories (but not the data) in memory.
|
|
.SS VFS Directory Cache
|
|
.PP
|
|
Using the \f[C]--dir-cache-time\f[R] flag, you can control how long a
|
|
directory should be considered up to date and not refreshed from the
|
|
backend.
|
|
Changes made through the mount will appear immediately or invalidate the
|
|
cache.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
However, changes made directly on the cloud storage by the web interface
|
|
or a different copy of rclone will only be picked up once the directory
|
|
cache expires if the backend configured does not support polling for
|
|
changes.
|
|
If the backend supports polling, changes will be picked up within the
|
|
polling interval.
|
|
.PP
|
|
You can send a \f[C]SIGHUP\f[R] signal to rclone for it to flush all
|
|
directory caches, regardless of how old they are.
|
|
Assuming only one rclone instance is running, you can reset the cache
|
|
like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill -SIGHUP $(pidof rclone)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control then you can use rclone rc
|
|
to flush the whole directory cache:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or individual files or directories:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget file=path/to/file dir=path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.SS VFS File Buffering
|
|
.PP
|
|
The \f[C]--buffer-size\f[R] flag determines the amount of memory, that
|
|
will be used to buffer data in advance.
|
|
.PP
|
|
Each open file will try to keep the specified amount of data in memory
|
|
at all times.
|
|
The buffered data is bound to one open file and won\[aq]t be shared.
|
|
.PP
|
|
This flag is a upper limit for the used memory per open file.
|
|
The buffer will only use memory for data that is downloaded but not not
|
|
yet read.
|
|
If the buffer is empty, only a small amount of memory will be used.
|
|
.PP
|
|
The maximum memory used by rclone for buffering can be up to
|
|
\f[C]--buffer-size * open files\f[R].
|
|
.SS VFS File Caching
|
|
.PP
|
|
These flags control the VFS file caching options.
|
|
File caching is necessary to make the VFS layer appear compatible with a
|
|
normal file system.
|
|
It can be disabled at the cost of some compatibility.
|
|
.PP
|
|
For example you\[aq]ll need to enable VFS caching if you want to read
|
|
and write simultaneously to a file.
|
|
See below for more details.
|
|
.PP
|
|
Note that the VFS cache is separate from the cache backend and you may
|
|
find that you need one or the other or both.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--cache-dir string Directory rclone will use for caching.
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If run with \f[C]-vv\f[R] rclone will print the location of the file
|
|
cache.
|
|
The files are stored in the user cache file area which is OS dependent
|
|
but can be controlled with \f[C]--cache-dir\f[R] or setting the
|
|
appropriate environment variable.
|
|
.PP
|
|
The cache has 4 different modes selected by \f[C]--vfs-cache-mode\f[R].
|
|
The higher the cache mode the more compatible rclone becomes at the cost
|
|
of using disk space.
|
|
.PP
|
|
Note that files are written back to the remote only when they are closed
|
|
and if they haven\[aq]t been accessed for \f[C]--vfs-write-back\f[R]
|
|
seconds.
|
|
If rclone is quit or dies with files that haven\[aq]t been uploaded,
|
|
these will be uploaded next time rclone is run with the same flags.
|
|
.PP
|
|
If using \f[C]--vfs-cache-max-size\f[R] note that the cache may exceed
|
|
this size for two reasons.
|
|
Firstly because it is only checked every
|
|
\f[C]--vfs-cache-poll-interval\f[R].
|
|
Secondly because open files cannot be evicted from the cache.
|
|
.PP
|
|
You \f[B]should not\f[R] run two copies of rclone using the same VFS
|
|
cache with the same or overlapping remotes if using
|
|
\f[C]--vfs-cache-mode > off\f[R].
|
|
This can potentially cause data corruption if you do.
|
|
You can work around this by giving each rclone its own cache hierarchy
|
|
with \f[C]--cache-dir\f[R].
|
|
You don\[aq]t need to worry about this if the remotes in use don\[aq]t
|
|
overlap.
|
|
.SS --vfs-cache-mode off
|
|
.PP
|
|
In this mode (the default) the cache will read directly from the remote
|
|
and write directly to the remote without caching anything on disk.
|
|
.PP
|
|
This will mean some operations are not possible
|
|
.IP \[bu] 2
|
|
Files can\[aq]t be opened for both read AND write
|
|
.IP \[bu] 2
|
|
Files opened for write can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files open for read with O_TRUNC will be opened write only
|
|
.IP \[bu] 2
|
|
Files open for write only will behave as if O_TRUNC was supplied
|
|
.IP \[bu] 2
|
|
Open modes O_APPEND, O_TRUNC are ignored
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode minimal
|
|
.PP
|
|
This is very similar to \[dq]off\[dq] except that files opened for read
|
|
AND write will be buffered to disk.
|
|
This means that files opened for write will be a lot more compatible,
|
|
but uses the minimal disk space.
|
|
.PP
|
|
These operations are not possible
|
|
.IP \[bu] 2
|
|
Files opened for write only can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files opened for write only will ignore O_APPEND, O_TRUNC
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode writes
|
|
.PP
|
|
In this mode files opened for read only are still read directly from the
|
|
remote, write only and read/write files are buffered to disk first.
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload fails it will be retried at exponentially increasing
|
|
intervals up to 1 minute.
|
|
.SS --vfs-cache-mode full
|
|
.PP
|
|
In this mode all reads and writes are buffered to and from disk.
|
|
When data is read from the remote this is buffered to disk as well.
|
|
.PP
|
|
In this mode the files in the cache will be sparse files and rclone will
|
|
keep track of which bits of the files it has downloaded.
|
|
.PP
|
|
So if an application only reads the starts of each file, then rclone
|
|
will only buffer the start of the file.
|
|
These files will appear to be their full size in the cache, but they
|
|
will be sparse files with only the data that has been downloaded present
|
|
in them.
|
|
.PP
|
|
This mode should support all normal file system operations and is
|
|
otherwise identical to \f[C]--vfs-cache-mode\f[R] writes.
|
|
.PP
|
|
When reading a file rclone will read \f[C]--buffer-size\f[R] plus
|
|
\f[C]--vfs-read-ahead\f[R] bytes ahead.
|
|
The \f[C]--buffer-size\f[R] is buffered in memory whereas the
|
|
\f[C]--vfs-read-ahead\f[R] is buffered on disk.
|
|
.PP
|
|
When using this mode it is recommended that \f[C]--buffer-size\f[R] is
|
|
not set too large and \f[C]--vfs-read-ahead\f[R] is set large if
|
|
required.
|
|
.PP
|
|
\f[B]IMPORTANT\f[R] not all file systems support sparse files.
|
|
In particular FAT/exFAT do not.
|
|
Rclone will perform very badly if the cache directory is on a filesystem
|
|
which doesn\[aq]t support sparse files and it will log an ERROR message
|
|
if one is detected.
|
|
.SS VFS Chunked Reading
|
|
.PP
|
|
When rclone reads files from a remote it reads them in chunks.
|
|
This means that rather than requesting the whole file rclone reads the
|
|
chunk specified.
|
|
This can reduce the used download quota for some remotes by requesting
|
|
only chunks from the remote that are actually read, at the cost of an
|
|
increased number of requests.
|
|
.PP
|
|
These flags control the chunking:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
|
|
--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Rclone will start reading a chunk of size
|
|
\f[C]--vfs-read-chunk-size\f[R], and then double the size for each read.
|
|
When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater
|
|
than \f[C]--vfs-read-chunk-size\f[R], the chunk size for each open file
|
|
will get doubled only until the specified value is reached.
|
|
If the value is \[dq]off\[dq], which is the default, the limit is
|
|
disabled and the chunk size will grow indefinitely.
|
|
.PP
|
|
With \f[C]--vfs-read-chunk-size 100M\f[R] and
|
|
\f[C]--vfs-read-chunk-size-limit 0\f[R] the following parts will be
|
|
downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
|
|
When \f[C]--vfs-read-chunk-size-limit 500M\f[R] is specified, the result
|
|
would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so
|
|
on.
|
|
.PP
|
|
Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq]
|
|
disables chunked reading.
|
|
.SS VFS Performance
|
|
.PP
|
|
These flags may be used to enable/disable features of the VFS for
|
|
performance or other reasons.
|
|
See also the chunked reading feature.
|
|
.PP
|
|
In particular S3 and Swift benefit hugely from the
|
|
\f[C]--no-modtime\f[R] flag (or use \f[C]--use-server-modtime\f[R] for a
|
|
slightly different effect) as each read of the modification time takes a
|
|
transaction.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--no-checksum Don\[aq]t compare checksums on up/download.
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up).
|
|
--no-seek Don\[aq]t allow seeking in files.
|
|
--read-only Mount read-only.
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sometimes rclone is delivered reads or writes out of order.
|
|
Rather than seeking rclone will wait a short time for the in sequence
|
|
read or write to come in.
|
|
These flags only come into effect when not using an on disk cache file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
When using VFS write caching (\f[C]--vfs-cache-mode\f[R] with value
|
|
writes or full), the global flag \f[C]--transfers\f[R] can be set to
|
|
adjust the number of parallel uploads of modified files from cache (the
|
|
related global flag \f[C]--checkers\f[R] have no effect on mount).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--transfers int Number of file transfers to run in parallel (default 4)
|
|
\f[R]
|
|
.fi
|
|
.SS VFS Case Sensitivity
|
|
.PP
|
|
Linux file systems are case-sensitive: two files can differ only by
|
|
case, and the exact case must be used when opening a file.
|
|
.PP
|
|
File systems in modern Windows are case-insensitive but case-preserving:
|
|
although existing files can be opened using any case, the exact case
|
|
used to create the file is preserved and available for programs to
|
|
query.
|
|
It is not allowed for two files in the same directory to differ only by
|
|
case.
|
|
.PP
|
|
Usually file systems on macOS are case-insensitive.
|
|
It is possible to make macOS file systems case-sensitive but that is not
|
|
the default.
|
|
.PP
|
|
The \f[C]--vfs-case-insensitive\f[R] mount flag controls how rclone
|
|
handles these two cases.
|
|
If its value is \[dq]false\[dq], rclone passes file names to the mounted
|
|
file system as-is.
|
|
If the flag is \[dq]true\[dq] (or appears without a value on command
|
|
line), rclone may perform a \[dq]fixup\[dq] as explained below.
|
|
.PP
|
|
The user may specify a file name to open/delete/rename/etc with a case
|
|
different than what is stored on mounted file system.
|
|
If an argument refers to an existing file with exactly the same name,
|
|
then the case of the existing file on the disk will be used.
|
|
However, if a file name with exactly the same name is not found but a
|
|
name differing only by case exists, rclone will transparently fixup the
|
|
name.
|
|
This fixup happens only when an existing file is requested.
|
|
Case sensitivity of file names created anew by rclone is controlled by
|
|
an underlying mounted file system.
|
|
.PP
|
|
Note that case sensitivity of the operating system running rclone (the
|
|
target) may differ from case sensitivity of a file system mounted by
|
|
rclone (the source).
|
|
The flag controls whether \[dq]fixup\[dq] is performed to satisfy the
|
|
target.
|
|
.PP
|
|
If the flag is not provided on the command line, then its default value
|
|
depends on the operating system where rclone runs: \[dq]true\[dq] on
|
|
Windows and macOS, \[dq]false\[dq] otherwise.
|
|
If the flag is provided without a value, then it is \[dq]true\[dq].
|
|
.SS Alternate report of used bytes
|
|
.PP
|
|
Some backends, most notably S3, do not report the amount of bytes used.
|
|
If you need this information to be available when running \f[C]df\f[R]
|
|
on the filesystem, then pass the flag \f[C]--vfs-used-is-size\f[R] to
|
|
rclone.
|
|
With this flag set, instead of relying on the backend to report this
|
|
information, rclone will scan the whole remote similar to
|
|
\f[C]rclone size\f[R] and compute the total used space itself.
|
|
.PP
|
|
\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
|
|
filters so that the result is accurate.
|
|
However, this is very inefficient and may cost lots of API calls
|
|
resulting in extra charges.
|
|
Use it as a last resort and only with caching.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone serve dlna remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--addr string The ip:port or :port to bind the DLNA http server to (default \[dq]:7879\[dq])
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--dir-perms FileMode Directory permissions (default 0777)
|
|
--file-perms FileMode File permissions (default 0666)
|
|
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
|
|
-h, --help help for dlna
|
|
--log-trace Enable trace logging of SOAP traffic
|
|
--name string Name of DLNA server
|
|
--no-checksum Don\[aq]t compare checksums on up/download
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up)
|
|
--no-seek Don\[aq]t allow seeking in files
|
|
--poll-interval duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
|
|
--read-only Mount read-only
|
|
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
|
|
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-case-insensitive If a file name not found, find a case insensitive match
|
|
--vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
|
|
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off)
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a
|
|
remote over a protocol.
|
|
.SH rclone serve docker
|
|
.PP
|
|
Serve any remote on docker\[aq]s volume plugin API.
|
|
.SS Synopsis
|
|
.PP
|
|
This command implements the Docker volume plugin API allowing docker to
|
|
use rclone as a data storage mechanism for various cloud providers.
|
|
rclone provides docker volume plugin based on it.
|
|
.PP
|
|
To create a docker plugin, one must create a Unix or TCP socket that
|
|
Docker will look for when you use the plugin and then it listens for
|
|
commands from docker daemon and runs the corresponding code when
|
|
necessary.
|
|
Docker plugins can run as a managed plugin under control of the docker
|
|
daemon or as an independent native service.
|
|
For testing, you can just run it directly from the command line, for
|
|
example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo rclone serve docker --base-dir /tmp/rclone-volumes --socket-addr localhost:8787 -vv
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Running \f[C]rclone serve docker\f[R] will create the said socket,
|
|
listening for commands from Docker to create the necessary Volumes.
|
|
Normally you need not give the \f[C]--socket-addr\f[R] flag.
|
|
The API will listen on the unix domain socket at
|
|
\f[C]/run/docker/plugins/rclone.sock\f[R].
|
|
In the example above rclone will create a TCP socket and a small file
|
|
\f[C]/etc/docker/plugins/rclone.spec\f[R] containing the socket address.
|
|
We use \f[C]sudo\f[R] because both paths are writeable only by the root
|
|
user.
|
|
.PP
|
|
If you later decide to change listening socket, the docker daemon must
|
|
be restarted to reconnect to \f[C]/run/docker/plugins/rclone.sock\f[R]
|
|
or parse new \f[C]/etc/docker/plugins/rclone.spec\f[R].
|
|
Until you restart, any volume related docker commands will timeout
|
|
trying to access the old socket.
|
|
Running directly is supported on \f[B]Linux only\f[R], not on Windows or
|
|
MacOS.
|
|
This is not a problem with managed plugin mode described in details in
|
|
the full documentation (https://rclone.org/docker).
|
|
.PP
|
|
The command will create volume mounts under the path given by
|
|
\f[C]--base-dir\f[R] (by default
|
|
\f[C]/var/lib/docker-volumes/rclone\f[R] available only to root) and
|
|
maintain the JSON formatted file \f[C]docker-plugin.state\f[R] in the
|
|
rclone cache directory with book-keeping records of created and mounted
|
|
volumes.
|
|
.PP
|
|
All mount and VFS options are submitted by the docker daemon via API,
|
|
but you can also provide defaults on the command line as well as set
|
|
path to the config file and cache directory or adjust logging verbosity.
|
|
.SS VFS - Virtual File System
|
|
.PP
|
|
This command uses the VFS layer.
|
|
This adapts the cloud storage objects that rclone uses into something
|
|
which looks much more like a disk filing system.
|
|
.PP
|
|
Cloud storage objects have lots of properties which aren\[aq]t like disk
|
|
files - you can\[aq]t extend them or write to the middle of them, so the
|
|
VFS layer has to deal with that.
|
|
Because there is no one right way of doing this there are various
|
|
options explained below.
|
|
.PP
|
|
The VFS layer also implements a directory cache - this caches info about
|
|
files and directories (but not the data) in memory.
|
|
.SS VFS Directory Cache
|
|
.PP
|
|
Using the \f[C]--dir-cache-time\f[R] flag, you can control how long a
|
|
directory should be considered up to date and not refreshed from the
|
|
backend.
|
|
Changes made through the mount will appear immediately or invalidate the
|
|
cache.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
However, changes made directly on the cloud storage by the web interface
|
|
or a different copy of rclone will only be picked up once the directory
|
|
cache expires if the backend configured does not support polling for
|
|
changes.
|
|
If the backend supports polling, changes will be picked up within the
|
|
polling interval.
|
|
.PP
|
|
You can send a \f[C]SIGHUP\f[R] signal to rclone for it to flush all
|
|
directory caches, regardless of how old they are.
|
|
Assuming only one rclone instance is running, you can reset the cache
|
|
like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill -SIGHUP $(pidof rclone)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control then you can use rclone rc
|
|
to flush the whole directory cache:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or individual files or directories:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget file=path/to/file dir=path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.SS VFS File Buffering
|
|
.PP
|
|
The \f[C]--buffer-size\f[R] flag determines the amount of memory, that
|
|
will be used to buffer data in advance.
|
|
.PP
|
|
Each open file will try to keep the specified amount of data in memory
|
|
at all times.
|
|
The buffered data is bound to one open file and won\[aq]t be shared.
|
|
.PP
|
|
This flag is a upper limit for the used memory per open file.
|
|
The buffer will only use memory for data that is downloaded but not not
|
|
yet read.
|
|
If the buffer is empty, only a small amount of memory will be used.
|
|
.PP
|
|
The maximum memory used by rclone for buffering can be up to
|
|
\f[C]--buffer-size * open files\f[R].
|
|
.SS VFS File Caching
|
|
.PP
|
|
These flags control the VFS file caching options.
|
|
File caching is necessary to make the VFS layer appear compatible with a
|
|
normal file system.
|
|
It can be disabled at the cost of some compatibility.
|
|
.PP
|
|
For example you\[aq]ll need to enable VFS caching if you want to read
|
|
and write simultaneously to a file.
|
|
See below for more details.
|
|
.PP
|
|
Note that the VFS cache is separate from the cache backend and you may
|
|
find that you need one or the other or both.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--cache-dir string Directory rclone will use for caching.
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If run with \f[C]-vv\f[R] rclone will print the location of the file
|
|
cache.
|
|
The files are stored in the user cache file area which is OS dependent
|
|
but can be controlled with \f[C]--cache-dir\f[R] or setting the
|
|
appropriate environment variable.
|
|
.PP
|
|
The cache has 4 different modes selected by \f[C]--vfs-cache-mode\f[R].
|
|
The higher the cache mode the more compatible rclone becomes at the cost
|
|
of using disk space.
|
|
.PP
|
|
Note that files are written back to the remote only when they are closed
|
|
and if they haven\[aq]t been accessed for \f[C]--vfs-write-back\f[R]
|
|
seconds.
|
|
If rclone is quit or dies with files that haven\[aq]t been uploaded,
|
|
these will be uploaded next time rclone is run with the same flags.
|
|
.PP
|
|
If using \f[C]--vfs-cache-max-size\f[R] note that the cache may exceed
|
|
this size for two reasons.
|
|
Firstly because it is only checked every
|
|
\f[C]--vfs-cache-poll-interval\f[R].
|
|
Secondly because open files cannot be evicted from the cache.
|
|
.PP
|
|
You \f[B]should not\f[R] run two copies of rclone using the same VFS
|
|
cache with the same or overlapping remotes if using
|
|
\f[C]--vfs-cache-mode > off\f[R].
|
|
This can potentially cause data corruption if you do.
|
|
You can work around this by giving each rclone its own cache hierarchy
|
|
with \f[C]--cache-dir\f[R].
|
|
You don\[aq]t need to worry about this if the remotes in use don\[aq]t
|
|
overlap.
|
|
.SS --vfs-cache-mode off
|
|
.PP
|
|
In this mode (the default) the cache will read directly from the remote
|
|
and write directly to the remote without caching anything on disk.
|
|
.PP
|
|
This will mean some operations are not possible
|
|
.IP \[bu] 2
|
|
Files can\[aq]t be opened for both read AND write
|
|
.IP \[bu] 2
|
|
Files opened for write can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files open for read with O_TRUNC will be opened write only
|
|
.IP \[bu] 2
|
|
Files open for write only will behave as if O_TRUNC was supplied
|
|
.IP \[bu] 2
|
|
Open modes O_APPEND, O_TRUNC are ignored
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode minimal
|
|
.PP
|
|
This is very similar to \[dq]off\[dq] except that files opened for read
|
|
AND write will be buffered to disk.
|
|
This means that files opened for write will be a lot more compatible,
|
|
but uses the minimal disk space.
|
|
.PP
|
|
These operations are not possible
|
|
.IP \[bu] 2
|
|
Files opened for write only can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files opened for write only will ignore O_APPEND, O_TRUNC
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode writes
|
|
.PP
|
|
In this mode files opened for read only are still read directly from the
|
|
remote, write only and read/write files are buffered to disk first.
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload fails it will be retried at exponentially increasing
|
|
intervals up to 1 minute.
|
|
.SS --vfs-cache-mode full
|
|
.PP
|
|
In this mode all reads and writes are buffered to and from disk.
|
|
When data is read from the remote this is buffered to disk as well.
|
|
.PP
|
|
In this mode the files in the cache will be sparse files and rclone will
|
|
keep track of which bits of the files it has downloaded.
|
|
.PP
|
|
So if an application only reads the starts of each file, then rclone
|
|
will only buffer the start of the file.
|
|
These files will appear to be their full size in the cache, but they
|
|
will be sparse files with only the data that has been downloaded present
|
|
in them.
|
|
.PP
|
|
This mode should support all normal file system operations and is
|
|
otherwise identical to \f[C]--vfs-cache-mode\f[R] writes.
|
|
.PP
|
|
When reading a file rclone will read \f[C]--buffer-size\f[R] plus
|
|
\f[C]--vfs-read-ahead\f[R] bytes ahead.
|
|
The \f[C]--buffer-size\f[R] is buffered in memory whereas the
|
|
\f[C]--vfs-read-ahead\f[R] is buffered on disk.
|
|
.PP
|
|
When using this mode it is recommended that \f[C]--buffer-size\f[R] is
|
|
not set too large and \f[C]--vfs-read-ahead\f[R] is set large if
|
|
required.
|
|
.PP
|
|
\f[B]IMPORTANT\f[R] not all file systems support sparse files.
|
|
In particular FAT/exFAT do not.
|
|
Rclone will perform very badly if the cache directory is on a filesystem
|
|
which doesn\[aq]t support sparse files and it will log an ERROR message
|
|
if one is detected.
|
|
.SS VFS Chunked Reading
|
|
.PP
|
|
When rclone reads files from a remote it reads them in chunks.
|
|
This means that rather than requesting the whole file rclone reads the
|
|
chunk specified.
|
|
This can reduce the used download quota for some remotes by requesting
|
|
only chunks from the remote that are actually read, at the cost of an
|
|
increased number of requests.
|
|
.PP
|
|
These flags control the chunking:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
|
|
--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Rclone will start reading a chunk of size
|
|
\f[C]--vfs-read-chunk-size\f[R], and then double the size for each read.
|
|
When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater
|
|
than \f[C]--vfs-read-chunk-size\f[R], the chunk size for each open file
|
|
will get doubled only until the specified value is reached.
|
|
If the value is \[dq]off\[dq], which is the default, the limit is
|
|
disabled and the chunk size will grow indefinitely.
|
|
.PP
|
|
With \f[C]--vfs-read-chunk-size 100M\f[R] and
|
|
\f[C]--vfs-read-chunk-size-limit 0\f[R] the following parts will be
|
|
downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
|
|
When \f[C]--vfs-read-chunk-size-limit 500M\f[R] is specified, the result
|
|
would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so
|
|
on.
|
|
.PP
|
|
Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq]
|
|
disables chunked reading.
|
|
.SS VFS Performance
|
|
.PP
|
|
These flags may be used to enable/disable features of the VFS for
|
|
performance or other reasons.
|
|
See also the chunked reading feature.
|
|
.PP
|
|
In particular S3 and Swift benefit hugely from the
|
|
\f[C]--no-modtime\f[R] flag (or use \f[C]--use-server-modtime\f[R] for a
|
|
slightly different effect) as each read of the modification time takes a
|
|
transaction.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--no-checksum Don\[aq]t compare checksums on up/download.
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up).
|
|
--no-seek Don\[aq]t allow seeking in files.
|
|
--read-only Mount read-only.
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sometimes rclone is delivered reads or writes out of order.
|
|
Rather than seeking rclone will wait a short time for the in sequence
|
|
read or write to come in.
|
|
These flags only come into effect when not using an on disk cache file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
When using VFS write caching (\f[C]--vfs-cache-mode\f[R] with value
|
|
writes or full), the global flag \f[C]--transfers\f[R] can be set to
|
|
adjust the number of parallel uploads of modified files from cache (the
|
|
related global flag \f[C]--checkers\f[R] have no effect on mount).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--transfers int Number of file transfers to run in parallel (default 4)
|
|
\f[R]
|
|
.fi
|
|
.SS VFS Case Sensitivity
|
|
.PP
|
|
Linux file systems are case-sensitive: two files can differ only by
|
|
case, and the exact case must be used when opening a file.
|
|
.PP
|
|
File systems in modern Windows are case-insensitive but case-preserving:
|
|
although existing files can be opened using any case, the exact case
|
|
used to create the file is preserved and available for programs to
|
|
query.
|
|
It is not allowed for two files in the same directory to differ only by
|
|
case.
|
|
.PP
|
|
Usually file systems on macOS are case-insensitive.
|
|
It is possible to make macOS file systems case-sensitive but that is not
|
|
the default.
|
|
.PP
|
|
The \f[C]--vfs-case-insensitive\f[R] mount flag controls how rclone
|
|
handles these two cases.
|
|
If its value is \[dq]false\[dq], rclone passes file names to the mounted
|
|
file system as-is.
|
|
If the flag is \[dq]true\[dq] (or appears without a value on command
|
|
line), rclone may perform a \[dq]fixup\[dq] as explained below.
|
|
.PP
|
|
The user may specify a file name to open/delete/rename/etc with a case
|
|
different than what is stored on mounted file system.
|
|
If an argument refers to an existing file with exactly the same name,
|
|
then the case of the existing file on the disk will be used.
|
|
However, if a file name with exactly the same name is not found but a
|
|
name differing only by case exists, rclone will transparently fixup the
|
|
name.
|
|
This fixup happens only when an existing file is requested.
|
|
Case sensitivity of file names created anew by rclone is controlled by
|
|
an underlying mounted file system.
|
|
.PP
|
|
Note that case sensitivity of the operating system running rclone (the
|
|
target) may differ from case sensitivity of a file system mounted by
|
|
rclone (the source).
|
|
The flag controls whether \[dq]fixup\[dq] is performed to satisfy the
|
|
target.
|
|
.PP
|
|
If the flag is not provided on the command line, then its default value
|
|
depends on the operating system where rclone runs: \[dq]true\[dq] on
|
|
Windows and macOS, \[dq]false\[dq] otherwise.
|
|
If the flag is provided without a value, then it is \[dq]true\[dq].
|
|
.SS Alternate report of used bytes
|
|
.PP
|
|
Some backends, most notably S3, do not report the amount of bytes used.
|
|
If you need this information to be available when running \f[C]df\f[R]
|
|
on the filesystem, then pass the flag \f[C]--vfs-used-is-size\f[R] to
|
|
rclone.
|
|
With this flag set, instead of relying on the backend to report this
|
|
information, rclone will scan the whole remote similar to
|
|
\f[C]rclone size\f[R] and compute the total used space itself.
|
|
.PP
|
|
\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
|
|
filters so that the result is accurate.
|
|
However, this is very inefficient and may cost lots of API calls
|
|
resulting in extra charges.
|
|
Use it as a last resort and only with caching.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone serve docker [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--allow-non-empty Allow mounting over a non-empty directory (not supported on Windows)
|
|
--allow-other Allow access to other users (not supported on Windows)
|
|
--allow-root Allow access to root user (not supported on Windows)
|
|
--async-read Use asynchronous reads (not supported on Windows) (default true)
|
|
--attr-timeout duration Time for which file/directory attributes are cached (default 1s)
|
|
--base-dir string Base directory for volumes (default \[dq]/var/lib/docker-volumes/rclone\[dq])
|
|
--daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows)
|
|
--daemon-timeout duration Time limit for rclone to respond to kernel (not supported on Windows)
|
|
--daemon-wait duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s)
|
|
--debug-fuse Debug the FUSE internals - needs -v
|
|
--default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows)
|
|
--devname string Set the device name - default is remote:path
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--dir-perms FileMode Directory permissions (default 0777)
|
|
--file-perms FileMode File permissions (default 0666)
|
|
--forget-state Skip restoring previous state
|
|
--fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required)
|
|
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
|
|
-h, --help help for docker
|
|
--max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki)
|
|
--network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only)
|
|
--no-checksum Don\[aq]t compare checksums on up/download
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up)
|
|
--no-seek Don\[aq]t allow seeking in files
|
|
--no-spec Do not write spec file
|
|
--noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true)
|
|
--noapplexattr Ignore all \[dq]com.apple.*\[dq] extended attributes (supported on OSX only)
|
|
-o, --option stringArray Option for libfuse/WinFsp (repeat if required)
|
|
--poll-interval duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
|
|
--read-only Mount read-only
|
|
--socket-addr string Address <host:port> or absolute path (default: /run/docker/plugins/rclone.sock)
|
|
--socket-gid int GID for unix socket (default: current process GID) (default 1000)
|
|
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
|
|
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-case-insensitive If a file name not found, find a case insensitive match
|
|
--vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
|
|
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off)
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
--volname string Set the volume name (supported on Windows and OSX only)
|
|
--write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a
|
|
remote over a protocol.
|
|
.SH rclone serve ftp
|
|
.PP
|
|
Serve remote:path over FTP.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone serve ftp implements a basic ftp server to serve the remote over
|
|
FTP protocol.
|
|
This can be viewed with a ftp client or you can make a remote of type
|
|
ftp to read and write it.
|
|
.SS Server options
|
|
.PP
|
|
Use --addr to specify which IP address and port the server should listen
|
|
on, e.g.
|
|
--addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs.
|
|
By default it only listens on localhost.
|
|
You can use port :0 to let the OS choose an available port.
|
|
.PP
|
|
If you set --addr to listen on a public or LAN accessible IP address
|
|
then using Authentication is advised - see the next section for info.
|
|
.SS Authentication
|
|
.PP
|
|
By default this will serve files without needing a login.
|
|
.PP
|
|
You can set a single username and password with the --user and --pass
|
|
flags.
|
|
.SS VFS - Virtual File System
|
|
.PP
|
|
This command uses the VFS layer.
|
|
This adapts the cloud storage objects that rclone uses into something
|
|
which looks much more like a disk filing system.
|
|
.PP
|
|
Cloud storage objects have lots of properties which aren\[aq]t like disk
|
|
files - you can\[aq]t extend them or write to the middle of them, so the
|
|
VFS layer has to deal with that.
|
|
Because there is no one right way of doing this there are various
|
|
options explained below.
|
|
.PP
|
|
The VFS layer also implements a directory cache - this caches info about
|
|
files and directories (but not the data) in memory.
|
|
.SS VFS Directory Cache
|
|
.PP
|
|
Using the \f[C]--dir-cache-time\f[R] flag, you can control how long a
|
|
directory should be considered up to date and not refreshed from the
|
|
backend.
|
|
Changes made through the mount will appear immediately or invalidate the
|
|
cache.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
However, changes made directly on the cloud storage by the web interface
|
|
or a different copy of rclone will only be picked up once the directory
|
|
cache expires if the backend configured does not support polling for
|
|
changes.
|
|
If the backend supports polling, changes will be picked up within the
|
|
polling interval.
|
|
.PP
|
|
You can send a \f[C]SIGHUP\f[R] signal to rclone for it to flush all
|
|
directory caches, regardless of how old they are.
|
|
Assuming only one rclone instance is running, you can reset the cache
|
|
like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill -SIGHUP $(pidof rclone)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control then you can use rclone rc
|
|
to flush the whole directory cache:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or individual files or directories:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget file=path/to/file dir=path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.SS VFS File Buffering
|
|
.PP
|
|
The \f[C]--buffer-size\f[R] flag determines the amount of memory, that
|
|
will be used to buffer data in advance.
|
|
.PP
|
|
Each open file will try to keep the specified amount of data in memory
|
|
at all times.
|
|
The buffered data is bound to one open file and won\[aq]t be shared.
|
|
.PP
|
|
This flag is a upper limit for the used memory per open file.
|
|
The buffer will only use memory for data that is downloaded but not not
|
|
yet read.
|
|
If the buffer is empty, only a small amount of memory will be used.
|
|
.PP
|
|
The maximum memory used by rclone for buffering can be up to
|
|
\f[C]--buffer-size * open files\f[R].
|
|
.SS VFS File Caching
|
|
.PP
|
|
These flags control the VFS file caching options.
|
|
File caching is necessary to make the VFS layer appear compatible with a
|
|
normal file system.
|
|
It can be disabled at the cost of some compatibility.
|
|
.PP
|
|
For example you\[aq]ll need to enable VFS caching if you want to read
|
|
and write simultaneously to a file.
|
|
See below for more details.
|
|
.PP
|
|
Note that the VFS cache is separate from the cache backend and you may
|
|
find that you need one or the other or both.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--cache-dir string Directory rclone will use for caching.
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If run with \f[C]-vv\f[R] rclone will print the location of the file
|
|
cache.
|
|
The files are stored in the user cache file area which is OS dependent
|
|
but can be controlled with \f[C]--cache-dir\f[R] or setting the
|
|
appropriate environment variable.
|
|
.PP
|
|
The cache has 4 different modes selected by \f[C]--vfs-cache-mode\f[R].
|
|
The higher the cache mode the more compatible rclone becomes at the cost
|
|
of using disk space.
|
|
.PP
|
|
Note that files are written back to the remote only when they are closed
|
|
and if they haven\[aq]t been accessed for \f[C]--vfs-write-back\f[R]
|
|
seconds.
|
|
If rclone is quit or dies with files that haven\[aq]t been uploaded,
|
|
these will be uploaded next time rclone is run with the same flags.
|
|
.PP
|
|
If using \f[C]--vfs-cache-max-size\f[R] note that the cache may exceed
|
|
this size for two reasons.
|
|
Firstly because it is only checked every
|
|
\f[C]--vfs-cache-poll-interval\f[R].
|
|
Secondly because open files cannot be evicted from the cache.
|
|
.PP
|
|
You \f[B]should not\f[R] run two copies of rclone using the same VFS
|
|
cache with the same or overlapping remotes if using
|
|
\f[C]--vfs-cache-mode > off\f[R].
|
|
This can potentially cause data corruption if you do.
|
|
You can work around this by giving each rclone its own cache hierarchy
|
|
with \f[C]--cache-dir\f[R].
|
|
You don\[aq]t need to worry about this if the remotes in use don\[aq]t
|
|
overlap.
|
|
.SS --vfs-cache-mode off
|
|
.PP
|
|
In this mode (the default) the cache will read directly from the remote
|
|
and write directly to the remote without caching anything on disk.
|
|
.PP
|
|
This will mean some operations are not possible
|
|
.IP \[bu] 2
|
|
Files can\[aq]t be opened for both read AND write
|
|
.IP \[bu] 2
|
|
Files opened for write can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files open for read with O_TRUNC will be opened write only
|
|
.IP \[bu] 2
|
|
Files open for write only will behave as if O_TRUNC was supplied
|
|
.IP \[bu] 2
|
|
Open modes O_APPEND, O_TRUNC are ignored
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode minimal
|
|
.PP
|
|
This is very similar to \[dq]off\[dq] except that files opened for read
|
|
AND write will be buffered to disk.
|
|
This means that files opened for write will be a lot more compatible,
|
|
but uses the minimal disk space.
|
|
.PP
|
|
These operations are not possible
|
|
.IP \[bu] 2
|
|
Files opened for write only can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files opened for write only will ignore O_APPEND, O_TRUNC
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode writes
|
|
.PP
|
|
In this mode files opened for read only are still read directly from the
|
|
remote, write only and read/write files are buffered to disk first.
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload fails it will be retried at exponentially increasing
|
|
intervals up to 1 minute.
|
|
.SS --vfs-cache-mode full
|
|
.PP
|
|
In this mode all reads and writes are buffered to and from disk.
|
|
When data is read from the remote this is buffered to disk as well.
|
|
.PP
|
|
In this mode the files in the cache will be sparse files and rclone will
|
|
keep track of which bits of the files it has downloaded.
|
|
.PP
|
|
So if an application only reads the starts of each file, then rclone
|
|
will only buffer the start of the file.
|
|
These files will appear to be their full size in the cache, but they
|
|
will be sparse files with only the data that has been downloaded present
|
|
in them.
|
|
.PP
|
|
This mode should support all normal file system operations and is
|
|
otherwise identical to \f[C]--vfs-cache-mode\f[R] writes.
|
|
.PP
|
|
When reading a file rclone will read \f[C]--buffer-size\f[R] plus
|
|
\f[C]--vfs-read-ahead\f[R] bytes ahead.
|
|
The \f[C]--buffer-size\f[R] is buffered in memory whereas the
|
|
\f[C]--vfs-read-ahead\f[R] is buffered on disk.
|
|
.PP
|
|
When using this mode it is recommended that \f[C]--buffer-size\f[R] is
|
|
not set too large and \f[C]--vfs-read-ahead\f[R] is set large if
|
|
required.
|
|
.PP
|
|
\f[B]IMPORTANT\f[R] not all file systems support sparse files.
|
|
In particular FAT/exFAT do not.
|
|
Rclone will perform very badly if the cache directory is on a filesystem
|
|
which doesn\[aq]t support sparse files and it will log an ERROR message
|
|
if one is detected.
|
|
.SS VFS Chunked Reading
|
|
.PP
|
|
When rclone reads files from a remote it reads them in chunks.
|
|
This means that rather than requesting the whole file rclone reads the
|
|
chunk specified.
|
|
This can reduce the used download quota for some remotes by requesting
|
|
only chunks from the remote that are actually read, at the cost of an
|
|
increased number of requests.
|
|
.PP
|
|
These flags control the chunking:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
|
|
--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Rclone will start reading a chunk of size
|
|
\f[C]--vfs-read-chunk-size\f[R], and then double the size for each read.
|
|
When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater
|
|
than \f[C]--vfs-read-chunk-size\f[R], the chunk size for each open file
|
|
will get doubled only until the specified value is reached.
|
|
If the value is \[dq]off\[dq], which is the default, the limit is
|
|
disabled and the chunk size will grow indefinitely.
|
|
.PP
|
|
With \f[C]--vfs-read-chunk-size 100M\f[R] and
|
|
\f[C]--vfs-read-chunk-size-limit 0\f[R] the following parts will be
|
|
downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
|
|
When \f[C]--vfs-read-chunk-size-limit 500M\f[R] is specified, the result
|
|
would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so
|
|
on.
|
|
.PP
|
|
Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq]
|
|
disables chunked reading.
|
|
.SS VFS Performance
|
|
.PP
|
|
These flags may be used to enable/disable features of the VFS for
|
|
performance or other reasons.
|
|
See also the chunked reading feature.
|
|
.PP
|
|
In particular S3 and Swift benefit hugely from the
|
|
\f[C]--no-modtime\f[R] flag (or use \f[C]--use-server-modtime\f[R] for a
|
|
slightly different effect) as each read of the modification time takes a
|
|
transaction.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--no-checksum Don\[aq]t compare checksums on up/download.
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up).
|
|
--no-seek Don\[aq]t allow seeking in files.
|
|
--read-only Mount read-only.
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sometimes rclone is delivered reads or writes out of order.
|
|
Rather than seeking rclone will wait a short time for the in sequence
|
|
read or write to come in.
|
|
These flags only come into effect when not using an on disk cache file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
When using VFS write caching (\f[C]--vfs-cache-mode\f[R] with value
|
|
writes or full), the global flag \f[C]--transfers\f[R] can be set to
|
|
adjust the number of parallel uploads of modified files from cache (the
|
|
related global flag \f[C]--checkers\f[R] have no effect on mount).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--transfers int Number of file transfers to run in parallel (default 4)
|
|
\f[R]
|
|
.fi
|
|
.SS VFS Case Sensitivity
|
|
.PP
|
|
Linux file systems are case-sensitive: two files can differ only by
|
|
case, and the exact case must be used when opening a file.
|
|
.PP
|
|
File systems in modern Windows are case-insensitive but case-preserving:
|
|
although existing files can be opened using any case, the exact case
|
|
used to create the file is preserved and available for programs to
|
|
query.
|
|
It is not allowed for two files in the same directory to differ only by
|
|
case.
|
|
.PP
|
|
Usually file systems on macOS are case-insensitive.
|
|
It is possible to make macOS file systems case-sensitive but that is not
|
|
the default.
|
|
.PP
|
|
The \f[C]--vfs-case-insensitive\f[R] mount flag controls how rclone
|
|
handles these two cases.
|
|
If its value is \[dq]false\[dq], rclone passes file names to the mounted
|
|
file system as-is.
|
|
If the flag is \[dq]true\[dq] (or appears without a value on command
|
|
line), rclone may perform a \[dq]fixup\[dq] as explained below.
|
|
.PP
|
|
The user may specify a file name to open/delete/rename/etc with a case
|
|
different than what is stored on mounted file system.
|
|
If an argument refers to an existing file with exactly the same name,
|
|
then the case of the existing file on the disk will be used.
|
|
However, if a file name with exactly the same name is not found but a
|
|
name differing only by case exists, rclone will transparently fixup the
|
|
name.
|
|
This fixup happens only when an existing file is requested.
|
|
Case sensitivity of file names created anew by rclone is controlled by
|
|
an underlying mounted file system.
|
|
.PP
|
|
Note that case sensitivity of the operating system running rclone (the
|
|
target) may differ from case sensitivity of a file system mounted by
|
|
rclone (the source).
|
|
The flag controls whether \[dq]fixup\[dq] is performed to satisfy the
|
|
target.
|
|
.PP
|
|
If the flag is not provided on the command line, then its default value
|
|
depends on the operating system where rclone runs: \[dq]true\[dq] on
|
|
Windows and macOS, \[dq]false\[dq] otherwise.
|
|
If the flag is provided without a value, then it is \[dq]true\[dq].
|
|
.SS Alternate report of used bytes
|
|
.PP
|
|
Some backends, most notably S3, do not report the amount of bytes used.
|
|
If you need this information to be available when running \f[C]df\f[R]
|
|
on the filesystem, then pass the flag \f[C]--vfs-used-is-size\f[R] to
|
|
rclone.
|
|
With this flag set, instead of relying on the backend to report this
|
|
information, rclone will scan the whole remote similar to
|
|
\f[C]rclone size\f[R] and compute the total used space itself.
|
|
.PP
|
|
\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
|
|
filters so that the result is accurate.
|
|
However, this is very inefficient and may cost lots of API calls
|
|
resulting in extra charges.
|
|
Use it as a last resort and only with caching.
|
|
.SS Auth Proxy
|
|
.PP
|
|
If you supply the parameter \f[C]--auth-proxy /path/to/program\f[R] then
|
|
rclone will use that program to generate backends on the fly which then
|
|
are used to authenticate incoming requests.
|
|
This uses a simple JSON based protocol with input on STDIN and output on
|
|
STDOUT.
|
|
.PP
|
|
\f[B]PLEASE NOTE:\f[R] \f[C]--auth-proxy\f[R] and
|
|
\f[C]--authorized-keys\f[R] cannot be used together, if
|
|
\f[C]--auth-proxy\f[R] is set the authorized keys option will be
|
|
ignored.
|
|
.PP
|
|
There is an example program
|
|
bin/test_proxy.py (https://github.com/rclone/rclone/blob/master/test_proxy.py)
|
|
in the rclone source code.
|
|
.PP
|
|
The program\[aq]s job is to take a \f[C]user\f[R] and \f[C]pass\f[R] on
|
|
the input and turn those into the config for a backend on STDOUT in JSON
|
|
format.
|
|
This config will have any default parameters for the backend added, but
|
|
it won\[aq]t use configuration from environment variables or command
|
|
line options - it is the job of the proxy program to make a complete
|
|
config.
|
|
.PP
|
|
This config generated must have this extra parameter - \f[C]_root\f[R] -
|
|
root to use for the backend
|
|
.PP
|
|
And it may have this parameter - \f[C]_obscure\f[R] - comma separated
|
|
strings for parameters to obscure
|
|
.PP
|
|
If password authentication was used by the client, input to the proxy
|
|
process (on STDIN) would look similar to this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]user\[dq]: \[dq]me\[dq],
|
|
\[dq]pass\[dq]: \[dq]mypassword\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If public-key authentication was used by the client, input to the proxy
|
|
process (on STDIN) would look similar to this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]user\[dq]: \[dq]me\[dq],
|
|
\[dq]public_key\[dq]: \[dq]AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And as an example return this on STDOUT
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]type\[dq]: \[dq]sftp\[dq],
|
|
\[dq]_root\[dq]: \[dq]\[dq],
|
|
\[dq]_obscure\[dq]: \[dq]pass\[dq],
|
|
\[dq]user\[dq]: \[dq]me\[dq],
|
|
\[dq]pass\[dq]: \[dq]mypassword\[dq],
|
|
\[dq]host\[dq]: \[dq]sftp.example.com\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This would mean that an SFTP backend would be created on the fly for the
|
|
\f[C]user\f[R] and \f[C]pass\f[R]/\f[C]public_key\f[R] returned in the
|
|
output to the host given.
|
|
Note that since \f[C]_obscure\f[R] is set to \f[C]pass\f[R], rclone will
|
|
obscure the \f[C]pass\f[R] parameter before creating the backend (which
|
|
is required for sftp backends).
|
|
.PP
|
|
The program can manipulate the supplied \f[C]user\f[R] in any way, for
|
|
example to make proxy to many different sftp backends, you could make
|
|
the \f[C]user\f[R] be \f[C]user\[at]example.com\f[R] and then set the
|
|
\f[C]host\f[R] to \f[C]example.com\f[R] in the output and the user to
|
|
\f[C]user\f[R].
|
|
For security you\[aq]d probably want to restrict the \f[C]host\f[R] to a
|
|
limited list.
|
|
.PP
|
|
Note that an internal cache is keyed on \f[C]user\f[R] so only use that
|
|
for configuration, don\[aq]t use \f[C]pass\f[R] or \f[C]public_key\f[R].
|
|
This also means that if a user\[aq]s password or public-key is changed
|
|
the cache will need to expire (which takes 5 mins) before it takes
|
|
effect.
|
|
.PP
|
|
This can be used to build general purpose proxies to any kind of backend
|
|
that rclone supports.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone serve ftp remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--addr string IPaddress:Port or :Port to bind server to (default \[dq]localhost:2121\[dq])
|
|
--auth-proxy string A program to use to create the backend from the auth
|
|
--cert string TLS PEM key (concatenation of certificate and CA certificate)
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--dir-perms FileMode Directory permissions (default 0777)
|
|
--file-perms FileMode File permissions (default 0666)
|
|
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
|
|
-h, --help help for ftp
|
|
--key string TLS PEM Private key
|
|
--no-checksum Don\[aq]t compare checksums on up/download
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up)
|
|
--no-seek Don\[aq]t allow seeking in files
|
|
--pass string Password for authentication (empty value allow every password)
|
|
--passive-port string Passive port range to use (default \[dq]30000-32000\[dq])
|
|
--poll-interval duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
|
|
--public-ip string Public IP address to advertise for passive connections
|
|
--read-only Mount read-only
|
|
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
|
|
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
|
|
--user string User name for authentication (default \[dq]anonymous\[dq])
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-case-insensitive If a file name not found, find a case insensitive match
|
|
--vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
|
|
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off)
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a
|
|
remote over a protocol.
|
|
.SH rclone serve http
|
|
.PP
|
|
Serve the remote over HTTP.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone serve http implements a basic web server to serve the remote over
|
|
HTTP.
|
|
This can be viewed in a web browser or you can make a remote of type
|
|
http read from it.
|
|
.PP
|
|
You can use the filter flags (e.g.
|
|
--include, --exclude) to control what is served.
|
|
.PP
|
|
The server will log errors.
|
|
Use -v to see access logs.
|
|
.PP
|
|
--bwlimit will be respected for file transfers.
|
|
Use --stats to control the stats printing.
|
|
.SS Server options
|
|
.PP
|
|
Use --addr to specify which IP address and port the server should listen
|
|
on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs.
|
|
By default it only listens on localhost.
|
|
You can use port :0 to let the OS choose an available port.
|
|
.PP
|
|
If you set --addr to listen on a public or LAN accessible IP address
|
|
then using Authentication is advised - see the next section for info.
|
|
.PP
|
|
--server-read-timeout and --server-write-timeout can be used to control
|
|
the timeouts on the server.
|
|
Note that this is the total time for a transfer.
|
|
.PP
|
|
--max-header-bytes controls the maximum number of bytes the server will
|
|
accept in the HTTP header.
|
|
.PP
|
|
--baseurl controls the URL prefix that rclone serves from.
|
|
By default rclone will serve from the root.
|
|
If you used --baseurl \[dq]/rclone\[dq] then rclone would serve from a
|
|
URL starting with \[dq]/rclone/\[dq].
|
|
This is useful if you wish to proxy rclone serve.
|
|
Rclone automatically inserts leading and trailing \[dq]/\[dq] on
|
|
--baseurl, so --baseurl \[dq]rclone\[dq], --baseurl \[dq]/rclone\[dq]
|
|
and --baseurl \[dq]/rclone/\[dq] are all treated identically.
|
|
.SS SSL/TLS
|
|
.PP
|
|
By default this will serve over http.
|
|
If you want you can serve over https.
|
|
You will need to supply the --cert and --key flags.
|
|
If you wish to do client side certificate validation then you will need
|
|
to supply --client-ca also.
|
|
.PP
|
|
--cert should be a either a PEM encoded certificate or a concatenation
|
|
of that with the CA certificate.
|
|
--key should be the PEM encoded private key and --client-ca should be
|
|
the PEM encoded client certificate authority certificate.
|
|
.SS Template
|
|
.PP
|
|
--template allows a user to specify a custom markup template for http
|
|
and webdav serve functions.
|
|
The server exports the following markup to be used within the template
|
|
to server pages:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
lw(35.0n) lw(35.0n).
|
|
T{
|
|
Parameter
|
|
T}@T{
|
|
Description
|
|
T}
|
|
_
|
|
T{
|
|
\&.Name
|
|
T}@T{
|
|
The full path of a file/directory.
|
|
T}
|
|
T{
|
|
\&.Title
|
|
T}@T{
|
|
Directory listing of .Name
|
|
T}
|
|
T{
|
|
\&.Sort
|
|
T}@T{
|
|
The current sort used.
|
|
This is changeable via ?sort= parameter
|
|
T}
|
|
T{
|
|
T}@T{
|
|
Sort Options: namedirfirst,name,size,time (default namedirfirst)
|
|
T}
|
|
T{
|
|
\&.Order
|
|
T}@T{
|
|
The current ordering used.
|
|
This is changeable via ?order= parameter
|
|
T}
|
|
T{
|
|
T}@T{
|
|
Order Options: asc,desc (default asc)
|
|
T}
|
|
T{
|
|
\&.Query
|
|
T}@T{
|
|
Currently unused.
|
|
T}
|
|
T{
|
|
\&.Breadcrumb
|
|
T}@T{
|
|
Allows for creating a relative navigation
|
|
T}
|
|
T{
|
|
-- .Link
|
|
T}@T{
|
|
The relative to the root link of the Text.
|
|
T}
|
|
T{
|
|
-- .Text
|
|
T}@T{
|
|
The Name of the directory.
|
|
T}
|
|
T{
|
|
\&.Entries
|
|
T}@T{
|
|
Information about a specific file/directory.
|
|
T}
|
|
T{
|
|
-- .URL
|
|
T}@T{
|
|
The \[aq]url\[aq] of an entry.
|
|
T}
|
|
T{
|
|
-- .Leaf
|
|
T}@T{
|
|
Currently same as \[aq]URL\[aq] but intended to be \[aq]just\[aq] the
|
|
name.
|
|
T}
|
|
T{
|
|
-- .IsDir
|
|
T}@T{
|
|
Boolean for if an entry is a directory or not.
|
|
T}
|
|
T{
|
|
-- .Size
|
|
T}@T{
|
|
Size in Bytes of the entry.
|
|
T}
|
|
T{
|
|
-- .ModTime
|
|
T}@T{
|
|
The UTC timestamp of an entry.
|
|
T}
|
|
.TE
|
|
.SS Authentication
|
|
.PP
|
|
By default this will serve files without needing a login.
|
|
.PP
|
|
You can either use an htpasswd file which can take lots of users, or set
|
|
a single username and password with the --user and --pass flags.
|
|
.PP
|
|
Use --htpasswd /path/to/htpasswd to provide an htpasswd file.
|
|
This is in standard apache format and supports MD5, SHA1 and BCrypt for
|
|
basic authentication.
|
|
Bcrypt is recommended.
|
|
.PP
|
|
To create an htpasswd file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
touch htpasswd
|
|
htpasswd -B htpasswd user
|
|
htpasswd -B htpasswd anotherUser
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The password file can be updated while rclone is running.
|
|
.PP
|
|
Use --realm to set the authentication realm.
|
|
.PP
|
|
Use --salt to change the password hashing salt from the default.
|
|
.SS VFS - Virtual File System
|
|
.PP
|
|
This command uses the VFS layer.
|
|
This adapts the cloud storage objects that rclone uses into something
|
|
which looks much more like a disk filing system.
|
|
.PP
|
|
Cloud storage objects have lots of properties which aren\[aq]t like disk
|
|
files - you can\[aq]t extend them or write to the middle of them, so the
|
|
VFS layer has to deal with that.
|
|
Because there is no one right way of doing this there are various
|
|
options explained below.
|
|
.PP
|
|
The VFS layer also implements a directory cache - this caches info about
|
|
files and directories (but not the data) in memory.
|
|
.SS VFS Directory Cache
|
|
.PP
|
|
Using the \f[C]--dir-cache-time\f[R] flag, you can control how long a
|
|
directory should be considered up to date and not refreshed from the
|
|
backend.
|
|
Changes made through the mount will appear immediately or invalidate the
|
|
cache.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
However, changes made directly on the cloud storage by the web interface
|
|
or a different copy of rclone will only be picked up once the directory
|
|
cache expires if the backend configured does not support polling for
|
|
changes.
|
|
If the backend supports polling, changes will be picked up within the
|
|
polling interval.
|
|
.PP
|
|
You can send a \f[C]SIGHUP\f[R] signal to rclone for it to flush all
|
|
directory caches, regardless of how old they are.
|
|
Assuming only one rclone instance is running, you can reset the cache
|
|
like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill -SIGHUP $(pidof rclone)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control then you can use rclone rc
|
|
to flush the whole directory cache:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or individual files or directories:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget file=path/to/file dir=path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.SS VFS File Buffering
|
|
.PP
|
|
The \f[C]--buffer-size\f[R] flag determines the amount of memory, that
|
|
will be used to buffer data in advance.
|
|
.PP
|
|
Each open file will try to keep the specified amount of data in memory
|
|
at all times.
|
|
The buffered data is bound to one open file and won\[aq]t be shared.
|
|
.PP
|
|
This flag is a upper limit for the used memory per open file.
|
|
The buffer will only use memory for data that is downloaded but not not
|
|
yet read.
|
|
If the buffer is empty, only a small amount of memory will be used.
|
|
.PP
|
|
The maximum memory used by rclone for buffering can be up to
|
|
\f[C]--buffer-size * open files\f[R].
|
|
.SS VFS File Caching
|
|
.PP
|
|
These flags control the VFS file caching options.
|
|
File caching is necessary to make the VFS layer appear compatible with a
|
|
normal file system.
|
|
It can be disabled at the cost of some compatibility.
|
|
.PP
|
|
For example you\[aq]ll need to enable VFS caching if you want to read
|
|
and write simultaneously to a file.
|
|
See below for more details.
|
|
.PP
|
|
Note that the VFS cache is separate from the cache backend and you may
|
|
find that you need one or the other or both.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--cache-dir string Directory rclone will use for caching.
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If run with \f[C]-vv\f[R] rclone will print the location of the file
|
|
cache.
|
|
The files are stored in the user cache file area which is OS dependent
|
|
but can be controlled with \f[C]--cache-dir\f[R] or setting the
|
|
appropriate environment variable.
|
|
.PP
|
|
The cache has 4 different modes selected by \f[C]--vfs-cache-mode\f[R].
|
|
The higher the cache mode the more compatible rclone becomes at the cost
|
|
of using disk space.
|
|
.PP
|
|
Note that files are written back to the remote only when they are closed
|
|
and if they haven\[aq]t been accessed for \f[C]--vfs-write-back\f[R]
|
|
seconds.
|
|
If rclone is quit or dies with files that haven\[aq]t been uploaded,
|
|
these will be uploaded next time rclone is run with the same flags.
|
|
.PP
|
|
If using \f[C]--vfs-cache-max-size\f[R] note that the cache may exceed
|
|
this size for two reasons.
|
|
Firstly because it is only checked every
|
|
\f[C]--vfs-cache-poll-interval\f[R].
|
|
Secondly because open files cannot be evicted from the cache.
|
|
.PP
|
|
You \f[B]should not\f[R] run two copies of rclone using the same VFS
|
|
cache with the same or overlapping remotes if using
|
|
\f[C]--vfs-cache-mode > off\f[R].
|
|
This can potentially cause data corruption if you do.
|
|
You can work around this by giving each rclone its own cache hierarchy
|
|
with \f[C]--cache-dir\f[R].
|
|
You don\[aq]t need to worry about this if the remotes in use don\[aq]t
|
|
overlap.
|
|
.SS --vfs-cache-mode off
|
|
.PP
|
|
In this mode (the default) the cache will read directly from the remote
|
|
and write directly to the remote without caching anything on disk.
|
|
.PP
|
|
This will mean some operations are not possible
|
|
.IP \[bu] 2
|
|
Files can\[aq]t be opened for both read AND write
|
|
.IP \[bu] 2
|
|
Files opened for write can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files open for read with O_TRUNC will be opened write only
|
|
.IP \[bu] 2
|
|
Files open for write only will behave as if O_TRUNC was supplied
|
|
.IP \[bu] 2
|
|
Open modes O_APPEND, O_TRUNC are ignored
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode minimal
|
|
.PP
|
|
This is very similar to \[dq]off\[dq] except that files opened for read
|
|
AND write will be buffered to disk.
|
|
This means that files opened for write will be a lot more compatible,
|
|
but uses the minimal disk space.
|
|
.PP
|
|
These operations are not possible
|
|
.IP \[bu] 2
|
|
Files opened for write only can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files opened for write only will ignore O_APPEND, O_TRUNC
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode writes
|
|
.PP
|
|
In this mode files opened for read only are still read directly from the
|
|
remote, write only and read/write files are buffered to disk first.
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload fails it will be retried at exponentially increasing
|
|
intervals up to 1 minute.
|
|
.SS --vfs-cache-mode full
|
|
.PP
|
|
In this mode all reads and writes are buffered to and from disk.
|
|
When data is read from the remote this is buffered to disk as well.
|
|
.PP
|
|
In this mode the files in the cache will be sparse files and rclone will
|
|
keep track of which bits of the files it has downloaded.
|
|
.PP
|
|
So if an application only reads the starts of each file, then rclone
|
|
will only buffer the start of the file.
|
|
These files will appear to be their full size in the cache, but they
|
|
will be sparse files with only the data that has been downloaded present
|
|
in them.
|
|
.PP
|
|
This mode should support all normal file system operations and is
|
|
otherwise identical to \f[C]--vfs-cache-mode\f[R] writes.
|
|
.PP
|
|
When reading a file rclone will read \f[C]--buffer-size\f[R] plus
|
|
\f[C]--vfs-read-ahead\f[R] bytes ahead.
|
|
The \f[C]--buffer-size\f[R] is buffered in memory whereas the
|
|
\f[C]--vfs-read-ahead\f[R] is buffered on disk.
|
|
.PP
|
|
When using this mode it is recommended that \f[C]--buffer-size\f[R] is
|
|
not set too large and \f[C]--vfs-read-ahead\f[R] is set large if
|
|
required.
|
|
.PP
|
|
\f[B]IMPORTANT\f[R] not all file systems support sparse files.
|
|
In particular FAT/exFAT do not.
|
|
Rclone will perform very badly if the cache directory is on a filesystem
|
|
which doesn\[aq]t support sparse files and it will log an ERROR message
|
|
if one is detected.
|
|
.SS VFS Chunked Reading
|
|
.PP
|
|
When rclone reads files from a remote it reads them in chunks.
|
|
This means that rather than requesting the whole file rclone reads the
|
|
chunk specified.
|
|
This can reduce the used download quota for some remotes by requesting
|
|
only chunks from the remote that are actually read, at the cost of an
|
|
increased number of requests.
|
|
.PP
|
|
These flags control the chunking:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
|
|
--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Rclone will start reading a chunk of size
|
|
\f[C]--vfs-read-chunk-size\f[R], and then double the size for each read.
|
|
When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater
|
|
than \f[C]--vfs-read-chunk-size\f[R], the chunk size for each open file
|
|
will get doubled only until the specified value is reached.
|
|
If the value is \[dq]off\[dq], which is the default, the limit is
|
|
disabled and the chunk size will grow indefinitely.
|
|
.PP
|
|
With \f[C]--vfs-read-chunk-size 100M\f[R] and
|
|
\f[C]--vfs-read-chunk-size-limit 0\f[R] the following parts will be
|
|
downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
|
|
When \f[C]--vfs-read-chunk-size-limit 500M\f[R] is specified, the result
|
|
would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so
|
|
on.
|
|
.PP
|
|
Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq]
|
|
disables chunked reading.
|
|
.SS VFS Performance
|
|
.PP
|
|
These flags may be used to enable/disable features of the VFS for
|
|
performance or other reasons.
|
|
See also the chunked reading feature.
|
|
.PP
|
|
In particular S3 and Swift benefit hugely from the
|
|
\f[C]--no-modtime\f[R] flag (or use \f[C]--use-server-modtime\f[R] for a
|
|
slightly different effect) as each read of the modification time takes a
|
|
transaction.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--no-checksum Don\[aq]t compare checksums on up/download.
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up).
|
|
--no-seek Don\[aq]t allow seeking in files.
|
|
--read-only Mount read-only.
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sometimes rclone is delivered reads or writes out of order.
|
|
Rather than seeking rclone will wait a short time for the in sequence
|
|
read or write to come in.
|
|
These flags only come into effect when not using an on disk cache file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
When using VFS write caching (\f[C]--vfs-cache-mode\f[R] with value
|
|
writes or full), the global flag \f[C]--transfers\f[R] can be set to
|
|
adjust the number of parallel uploads of modified files from cache (the
|
|
related global flag \f[C]--checkers\f[R] have no effect on mount).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--transfers int Number of file transfers to run in parallel (default 4)
|
|
\f[R]
|
|
.fi
|
|
.SS VFS Case Sensitivity
|
|
.PP
|
|
Linux file systems are case-sensitive: two files can differ only by
|
|
case, and the exact case must be used when opening a file.
|
|
.PP
|
|
File systems in modern Windows are case-insensitive but case-preserving:
|
|
although existing files can be opened using any case, the exact case
|
|
used to create the file is preserved and available for programs to
|
|
query.
|
|
It is not allowed for two files in the same directory to differ only by
|
|
case.
|
|
.PP
|
|
Usually file systems on macOS are case-insensitive.
|
|
It is possible to make macOS file systems case-sensitive but that is not
|
|
the default.
|
|
.PP
|
|
The \f[C]--vfs-case-insensitive\f[R] mount flag controls how rclone
|
|
handles these two cases.
|
|
If its value is \[dq]false\[dq], rclone passes file names to the mounted
|
|
file system as-is.
|
|
If the flag is \[dq]true\[dq] (or appears without a value on command
|
|
line), rclone may perform a \[dq]fixup\[dq] as explained below.
|
|
.PP
|
|
The user may specify a file name to open/delete/rename/etc with a case
|
|
different than what is stored on mounted file system.
|
|
If an argument refers to an existing file with exactly the same name,
|
|
then the case of the existing file on the disk will be used.
|
|
However, if a file name with exactly the same name is not found but a
|
|
name differing only by case exists, rclone will transparently fixup the
|
|
name.
|
|
This fixup happens only when an existing file is requested.
|
|
Case sensitivity of file names created anew by rclone is controlled by
|
|
an underlying mounted file system.
|
|
.PP
|
|
Note that case sensitivity of the operating system running rclone (the
|
|
target) may differ from case sensitivity of a file system mounted by
|
|
rclone (the source).
|
|
The flag controls whether \[dq]fixup\[dq] is performed to satisfy the
|
|
target.
|
|
.PP
|
|
If the flag is not provided on the command line, then its default value
|
|
depends on the operating system where rclone runs: \[dq]true\[dq] on
|
|
Windows and macOS, \[dq]false\[dq] otherwise.
|
|
If the flag is provided without a value, then it is \[dq]true\[dq].
|
|
.SS Alternate report of used bytes
|
|
.PP
|
|
Some backends, most notably S3, do not report the amount of bytes used.
|
|
If you need this information to be available when running \f[C]df\f[R]
|
|
on the filesystem, then pass the flag \f[C]--vfs-used-is-size\f[R] to
|
|
rclone.
|
|
With this flag set, instead of relying on the backend to report this
|
|
information, rclone will scan the whole remote similar to
|
|
\f[C]rclone size\f[R] and compute the total used space itself.
|
|
.PP
|
|
\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
|
|
filters so that the result is accurate.
|
|
However, this is very inefficient and may cost lots of API calls
|
|
resulting in extra charges.
|
|
Use it as a last resort and only with caching.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone serve http remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--addr string IPaddress:Port or :Port to bind server to (default \[dq]127.0.0.1:8080\[dq])
|
|
--baseurl string Prefix for URLs - leave blank for root
|
|
--cert string SSL PEM key (concatenation of certificate and CA certificate)
|
|
--client-ca string Client certificate authority to verify clients with
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--dir-perms FileMode Directory permissions (default 0777)
|
|
--file-perms FileMode File permissions (default 0666)
|
|
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
|
|
-h, --help help for http
|
|
--htpasswd string A htpasswd file - if not provided no authentication is done
|
|
--key string SSL PEM Private key
|
|
--max-header-bytes int Maximum size of request header (default 4096)
|
|
--no-checksum Don\[aq]t compare checksums on up/download
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up)
|
|
--no-seek Don\[aq]t allow seeking in files
|
|
--pass string Password for authentication
|
|
--poll-interval duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
|
|
--read-only Mount read-only
|
|
--realm string Realm for authentication
|
|
--salt string Password hashing salt (default \[dq]dlPL2MqE\[dq])
|
|
--server-read-timeout duration Timeout for server reading data (default 1h0m0s)
|
|
--server-write-timeout duration Timeout for server writing data (default 1h0m0s)
|
|
--template string User-specified template
|
|
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
|
|
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
|
|
--user string User name for authentication
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-case-insensitive If a file name not found, find a case insensitive match
|
|
--vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
|
|
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off)
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a
|
|
remote over a protocol.
|
|
.SH rclone serve restic
|
|
.PP
|
|
Serve the remote for restic\[aq]s REST API.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone serve restic implements restic\[aq]s REST backend API over HTTP.
|
|
This allows restic to use rclone as a data storage mechanism for cloud
|
|
providers that restic does not support directly.
|
|
.PP
|
|
Restic (https://restic.net/) is a command-line program for doing
|
|
backups.
|
|
.PP
|
|
The server will log errors.
|
|
Use -v to see access logs.
|
|
.PP
|
|
--bwlimit will be respected for file transfers.
|
|
Use --stats to control the stats printing.
|
|
.SS Setting up rclone for use by restic
|
|
.PP
|
|
First set up a remote for your chosen cloud
|
|
provider (https://rclone.org/docs/#configure).
|
|
.PP
|
|
Once you have set up the remote, check it is working with, for example
|
|
\[dq]rclone lsd remote:\[dq].
|
|
You may have called the remote something other than \[dq]remote:\[dq] -
|
|
just substitute whatever you called it in the following instructions.
|
|
.PP
|
|
Now start the rclone restic server
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone serve restic -v remote:backup
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Where you can replace \[dq]backup\[dq] in the above by whatever path in
|
|
the remote you wish to use.
|
|
.PP
|
|
By default this will serve on \[dq]localhost:8080\[dq] you can change
|
|
this with use of the \[dq]--addr\[dq] flag.
|
|
.PP
|
|
You might wish to start this server on boot.
|
|
.PP
|
|
Adding --cache-objects=false will cause rclone to stop caching objects
|
|
returned from the List call.
|
|
Caching is normally desirable as it speeds up downloading objects, saves
|
|
transactions and uses very little memory.
|
|
.SS Setting up restic to use rclone
|
|
.PP
|
|
Now you can follow the restic
|
|
instructions (http://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server)
|
|
on setting up restic.
|
|
.PP
|
|
Note that you will need restic 0.8.2 or later to interoperate with
|
|
rclone.
|
|
.PP
|
|
For the example above you will want to use
|
|
\[dq]http://localhost:8080/\[dq] as the URL for the REST server.
|
|
.PP
|
|
For example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ export RESTIC_REPOSITORY=rest:http://localhost:8080/
|
|
$ export RESTIC_PASSWORD=yourpassword
|
|
$ restic init
|
|
created restic backend 8b1a4b56ae at rest:http://localhost:8080/
|
|
|
|
Please note that knowledge of your password is required to access
|
|
the repository. Losing your password means that your data is
|
|
irrecoverably lost.
|
|
$ restic backup /path/to/files/to/backup
|
|
scan [/path/to/files/to/backup]
|
|
scanned 189 directories, 312 files in 0:00
|
|
[0:00] 100.00% 38.128 MiB / 38.128 MiB 501 / 501 items 0 errors ETA 0:00
|
|
duration: 0:00
|
|
snapshot 45c8fdd8 saved
|
|
\f[R]
|
|
.fi
|
|
.SS Multiple repositories
|
|
.PP
|
|
Note that you can use the endpoint to host multiple repositories.
|
|
Do this by adding a directory name or path after the URL.
|
|
Note that these \f[B]must\f[R] end with /.
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
|
|
# backup user1 stuff
|
|
$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
|
|
# backup user2 stuff
|
|
\f[R]
|
|
.fi
|
|
.SS Private repositories
|
|
.PP
|
|
The \[dq]--private-repos\[dq] flag can be used to limit users to
|
|
repositories starting with a path of \f[C]/<username>/\f[R].
|
|
.SS Server options
|
|
.PP
|
|
Use --addr to specify which IP address and port the server should listen
|
|
on, e.g.
|
|
--addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs.
|
|
By default it only listens on localhost.
|
|
You can use port :0 to let the OS choose an available port.
|
|
.PP
|
|
If you set --addr to listen on a public or LAN accessible IP address
|
|
then using Authentication is advised - see the next section for info.
|
|
.PP
|
|
--server-read-timeout and --server-write-timeout can be used to control
|
|
the timeouts on the server.
|
|
Note that this is the total time for a transfer.
|
|
.PP
|
|
--max-header-bytes controls the maximum number of bytes the server will
|
|
accept in the HTTP header.
|
|
.PP
|
|
--baseurl controls the URL prefix that rclone serves from.
|
|
By default rclone will serve from the root.
|
|
If you used --baseurl \[dq]/rclone\[dq] then rclone would serve from a
|
|
URL starting with \[dq]/rclone/\[dq].
|
|
This is useful if you wish to proxy rclone serve.
|
|
Rclone automatically inserts leading and trailing \[dq]/\[dq] on
|
|
--baseurl, so --baseurl \[dq]rclone\[dq], --baseurl \[dq]/rclone\[dq]
|
|
and --baseurl \[dq]/rclone/\[dq] are all treated identically.
|
|
.PP
|
|
--template allows a user to specify a custom markup template for http
|
|
and webdav serve functions.
|
|
The server exports the following markup to be used within the template
|
|
to server pages:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
lw(35.0n) lw(35.0n).
|
|
T{
|
|
Parameter
|
|
T}@T{
|
|
Description
|
|
T}
|
|
_
|
|
T{
|
|
\&.Name
|
|
T}@T{
|
|
The full path of a file/directory.
|
|
T}
|
|
T{
|
|
\&.Title
|
|
T}@T{
|
|
Directory listing of .Name
|
|
T}
|
|
T{
|
|
\&.Sort
|
|
T}@T{
|
|
The current sort used.
|
|
This is changeable via ?sort= parameter
|
|
T}
|
|
T{
|
|
T}@T{
|
|
Sort Options: namedirfirst,name,size,time (default namedirfirst)
|
|
T}
|
|
T{
|
|
\&.Order
|
|
T}@T{
|
|
The current ordering used.
|
|
This is changeable via ?order= parameter
|
|
T}
|
|
T{
|
|
T}@T{
|
|
Order Options: asc,desc (default asc)
|
|
T}
|
|
T{
|
|
\&.Query
|
|
T}@T{
|
|
Currently unused.
|
|
T}
|
|
T{
|
|
\&.Breadcrumb
|
|
T}@T{
|
|
Allows for creating a relative navigation
|
|
T}
|
|
T{
|
|
-- .Link
|
|
T}@T{
|
|
The relative to the root link of the Text.
|
|
T}
|
|
T{
|
|
-- .Text
|
|
T}@T{
|
|
The Name of the directory.
|
|
T}
|
|
T{
|
|
\&.Entries
|
|
T}@T{
|
|
Information about a specific file/directory.
|
|
T}
|
|
T{
|
|
-- .URL
|
|
T}@T{
|
|
The \[aq]url\[aq] of an entry.
|
|
T}
|
|
T{
|
|
-- .Leaf
|
|
T}@T{
|
|
Currently same as \[aq]URL\[aq] but intended to be \[aq]just\[aq] the
|
|
name.
|
|
T}
|
|
T{
|
|
-- .IsDir
|
|
T}@T{
|
|
Boolean for if an entry is a directory or not.
|
|
T}
|
|
T{
|
|
-- .Size
|
|
T}@T{
|
|
Size in Bytes of the entry.
|
|
T}
|
|
T{
|
|
-- .ModTime
|
|
T}@T{
|
|
The UTC timestamp of an entry.
|
|
T}
|
|
.TE
|
|
.SS Authentication
|
|
.PP
|
|
By default this will serve files without needing a login.
|
|
.PP
|
|
You can either use an htpasswd file which can take lots of users, or set
|
|
a single username and password with the --user and --pass flags.
|
|
.PP
|
|
Use --htpasswd /path/to/htpasswd to provide an htpasswd file.
|
|
This is in standard apache format and supports MD5, SHA1 and BCrypt for
|
|
basic authentication.
|
|
Bcrypt is recommended.
|
|
.PP
|
|
To create an htpasswd file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
touch htpasswd
|
|
htpasswd -B htpasswd user
|
|
htpasswd -B htpasswd anotherUser
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The password file can be updated while rclone is running.
|
|
.PP
|
|
Use --realm to set the authentication realm.
|
|
.SS SSL/TLS
|
|
.PP
|
|
By default this will serve over http.
|
|
If you want you can serve over https.
|
|
You will need to supply the --cert and --key flags.
|
|
If you wish to do client side certificate validation then you will need
|
|
to supply --client-ca also.
|
|
.PP
|
|
--cert should be either a PEM encoded certificate or a concatenation of
|
|
that with the CA certificate.
|
|
--key should be the PEM encoded private key and --client-ca should be
|
|
the PEM encoded client certificate authority certificate.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone serve restic remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--addr string IPaddress:Port or :Port to bind server to (default \[dq]localhost:8080\[dq])
|
|
--append-only Disallow deletion of repository data
|
|
--baseurl string Prefix for URLs - leave blank for root
|
|
--cache-objects Cache listed objects (default true)
|
|
--cert string SSL PEM key (concatenation of certificate and CA certificate)
|
|
--client-ca string Client certificate authority to verify clients with
|
|
-h, --help help for restic
|
|
--htpasswd string htpasswd file - if not provided no authentication is done
|
|
--key string SSL PEM Private key
|
|
--max-header-bytes int Maximum size of request header (default 4096)
|
|
--pass string Password for authentication
|
|
--private-repos Users can only access their private repo
|
|
--realm string Realm for authentication (default \[dq]rclone\[dq])
|
|
--server-read-timeout duration Timeout for server reading data (default 1h0m0s)
|
|
--server-write-timeout duration Timeout for server writing data (default 1h0m0s)
|
|
--stdio Run an HTTP2 server on stdin/stdout
|
|
--template string User-specified template
|
|
--user string User name for authentication
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a
|
|
remote over a protocol.
|
|
.SH rclone serve sftp
|
|
.PP
|
|
Serve the remote over SFTP.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone serve sftp implements an SFTP server to serve the remote over
|
|
SFTP.
|
|
This can be used with an SFTP client or you can make a remote of type
|
|
sftp to use with it.
|
|
.PP
|
|
You can use the filter flags (e.g.
|
|
--include, --exclude) to control what is served.
|
|
.PP
|
|
The server will log errors.
|
|
Use -v to see access logs.
|
|
.PP
|
|
--bwlimit will be respected for file transfers.
|
|
Use --stats to control the stats printing.
|
|
.PP
|
|
You must provide some means of authentication, either with
|
|
--user/--pass, an authorized keys file (specify location with
|
|
--authorized-keys - the default is the same as ssh), an --auth-proxy, or
|
|
set the --no-auth flag for no authentication when logging in.
|
|
.PP
|
|
Note that this also implements a small number of shell commands so that
|
|
it can provide md5sum/sha1sum/df information for the rclone sftp
|
|
backend.
|
|
This means that is can support SHA1SUMs, MD5SUMs and the about command
|
|
when paired with the rclone sftp backend.
|
|
.PP
|
|
If you don\[aq]t supply a host --key then rclone will generate rsa,
|
|
ecdsa and ed25519 variants, and cache them for later use in rclone\[aq]s
|
|
cache directory (see \[dq]rclone help flags cache-dir\[dq]) in the
|
|
\[dq]serve-sftp\[dq] directory.
|
|
.PP
|
|
By default the server binds to localhost:2022 - if you want it to be
|
|
reachable externally then supply \[dq]--addr :2022\[dq] for example.
|
|
.PP
|
|
Note that the default of \[dq]--vfs-cache-mode off\[dq] is fine for the
|
|
rclone sftp backend, but it may not be with other SFTP clients.
|
|
.PP
|
|
If --stdio is specified, rclone will serve SFTP over stdio, which can be
|
|
used with sshd via \[ti]/.ssh/authorized_keys, for example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
restrict,command=\[dq]rclone serve sftp --stdio ./photos\[dq] ssh-rsa ...
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
On the client you need to set \[dq]--transfers 1\[dq] when using
|
|
--stdio.
|
|
Otherwise multiple instances of the rclone server are started by OpenSSH
|
|
which can lead to \[dq]corrupted on transfer\[dq] errors.
|
|
This is the case because the client chooses indiscriminately which
|
|
server to send commands to while the servers all have different views of
|
|
the state of the filing system.
|
|
.PP
|
|
The \[dq]restrict\[dq] in authorized_keys prevents SHA1SUMs and MD5SUMs
|
|
from beeing used.
|
|
Omitting \[dq]restrict\[dq] and using --sftp-path-override to enable
|
|
checksumming is possible but less secure and you could use the SFTP
|
|
server provided by OpenSSH in this case.
|
|
.SS VFS - Virtual File System
|
|
.PP
|
|
This command uses the VFS layer.
|
|
This adapts the cloud storage objects that rclone uses into something
|
|
which looks much more like a disk filing system.
|
|
.PP
|
|
Cloud storage objects have lots of properties which aren\[aq]t like disk
|
|
files - you can\[aq]t extend them or write to the middle of them, so the
|
|
VFS layer has to deal with that.
|
|
Because there is no one right way of doing this there are various
|
|
options explained below.
|
|
.PP
|
|
The VFS layer also implements a directory cache - this caches info about
|
|
files and directories (but not the data) in memory.
|
|
.SS VFS Directory Cache
|
|
.PP
|
|
Using the \f[C]--dir-cache-time\f[R] flag, you can control how long a
|
|
directory should be considered up to date and not refreshed from the
|
|
backend.
|
|
Changes made through the mount will appear immediately or invalidate the
|
|
cache.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
However, changes made directly on the cloud storage by the web interface
|
|
or a different copy of rclone will only be picked up once the directory
|
|
cache expires if the backend configured does not support polling for
|
|
changes.
|
|
If the backend supports polling, changes will be picked up within the
|
|
polling interval.
|
|
.PP
|
|
You can send a \f[C]SIGHUP\f[R] signal to rclone for it to flush all
|
|
directory caches, regardless of how old they are.
|
|
Assuming only one rclone instance is running, you can reset the cache
|
|
like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill -SIGHUP $(pidof rclone)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control then you can use rclone rc
|
|
to flush the whole directory cache:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or individual files or directories:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget file=path/to/file dir=path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.SS VFS File Buffering
|
|
.PP
|
|
The \f[C]--buffer-size\f[R] flag determines the amount of memory, that
|
|
will be used to buffer data in advance.
|
|
.PP
|
|
Each open file will try to keep the specified amount of data in memory
|
|
at all times.
|
|
The buffered data is bound to one open file and won\[aq]t be shared.
|
|
.PP
|
|
This flag is a upper limit for the used memory per open file.
|
|
The buffer will only use memory for data that is downloaded but not not
|
|
yet read.
|
|
If the buffer is empty, only a small amount of memory will be used.
|
|
.PP
|
|
The maximum memory used by rclone for buffering can be up to
|
|
\f[C]--buffer-size * open files\f[R].
|
|
.SS VFS File Caching
|
|
.PP
|
|
These flags control the VFS file caching options.
|
|
File caching is necessary to make the VFS layer appear compatible with a
|
|
normal file system.
|
|
It can be disabled at the cost of some compatibility.
|
|
.PP
|
|
For example you\[aq]ll need to enable VFS caching if you want to read
|
|
and write simultaneously to a file.
|
|
See below for more details.
|
|
.PP
|
|
Note that the VFS cache is separate from the cache backend and you may
|
|
find that you need one or the other or both.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--cache-dir string Directory rclone will use for caching.
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If run with \f[C]-vv\f[R] rclone will print the location of the file
|
|
cache.
|
|
The files are stored in the user cache file area which is OS dependent
|
|
but can be controlled with \f[C]--cache-dir\f[R] or setting the
|
|
appropriate environment variable.
|
|
.PP
|
|
The cache has 4 different modes selected by \f[C]--vfs-cache-mode\f[R].
|
|
The higher the cache mode the more compatible rclone becomes at the cost
|
|
of using disk space.
|
|
.PP
|
|
Note that files are written back to the remote only when they are closed
|
|
and if they haven\[aq]t been accessed for \f[C]--vfs-write-back\f[R]
|
|
seconds.
|
|
If rclone is quit or dies with files that haven\[aq]t been uploaded,
|
|
these will be uploaded next time rclone is run with the same flags.
|
|
.PP
|
|
If using \f[C]--vfs-cache-max-size\f[R] note that the cache may exceed
|
|
this size for two reasons.
|
|
Firstly because it is only checked every
|
|
\f[C]--vfs-cache-poll-interval\f[R].
|
|
Secondly because open files cannot be evicted from the cache.
|
|
.PP
|
|
You \f[B]should not\f[R] run two copies of rclone using the same VFS
|
|
cache with the same or overlapping remotes if using
|
|
\f[C]--vfs-cache-mode > off\f[R].
|
|
This can potentially cause data corruption if you do.
|
|
You can work around this by giving each rclone its own cache hierarchy
|
|
with \f[C]--cache-dir\f[R].
|
|
You don\[aq]t need to worry about this if the remotes in use don\[aq]t
|
|
overlap.
|
|
.SS --vfs-cache-mode off
|
|
.PP
|
|
In this mode (the default) the cache will read directly from the remote
|
|
and write directly to the remote without caching anything on disk.
|
|
.PP
|
|
This will mean some operations are not possible
|
|
.IP \[bu] 2
|
|
Files can\[aq]t be opened for both read AND write
|
|
.IP \[bu] 2
|
|
Files opened for write can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files open for read with O_TRUNC will be opened write only
|
|
.IP \[bu] 2
|
|
Files open for write only will behave as if O_TRUNC was supplied
|
|
.IP \[bu] 2
|
|
Open modes O_APPEND, O_TRUNC are ignored
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode minimal
|
|
.PP
|
|
This is very similar to \[dq]off\[dq] except that files opened for read
|
|
AND write will be buffered to disk.
|
|
This means that files opened for write will be a lot more compatible,
|
|
but uses the minimal disk space.
|
|
.PP
|
|
These operations are not possible
|
|
.IP \[bu] 2
|
|
Files opened for write only can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files opened for write only will ignore O_APPEND, O_TRUNC
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode writes
|
|
.PP
|
|
In this mode files opened for read only are still read directly from the
|
|
remote, write only and read/write files are buffered to disk first.
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload fails it will be retried at exponentially increasing
|
|
intervals up to 1 minute.
|
|
.SS --vfs-cache-mode full
|
|
.PP
|
|
In this mode all reads and writes are buffered to and from disk.
|
|
When data is read from the remote this is buffered to disk as well.
|
|
.PP
|
|
In this mode the files in the cache will be sparse files and rclone will
|
|
keep track of which bits of the files it has downloaded.
|
|
.PP
|
|
So if an application only reads the starts of each file, then rclone
|
|
will only buffer the start of the file.
|
|
These files will appear to be their full size in the cache, but they
|
|
will be sparse files with only the data that has been downloaded present
|
|
in them.
|
|
.PP
|
|
This mode should support all normal file system operations and is
|
|
otherwise identical to \f[C]--vfs-cache-mode\f[R] writes.
|
|
.PP
|
|
When reading a file rclone will read \f[C]--buffer-size\f[R] plus
|
|
\f[C]--vfs-read-ahead\f[R] bytes ahead.
|
|
The \f[C]--buffer-size\f[R] is buffered in memory whereas the
|
|
\f[C]--vfs-read-ahead\f[R] is buffered on disk.
|
|
.PP
|
|
When using this mode it is recommended that \f[C]--buffer-size\f[R] is
|
|
not set too large and \f[C]--vfs-read-ahead\f[R] is set large if
|
|
required.
|
|
.PP
|
|
\f[B]IMPORTANT\f[R] not all file systems support sparse files.
|
|
In particular FAT/exFAT do not.
|
|
Rclone will perform very badly if the cache directory is on a filesystem
|
|
which doesn\[aq]t support sparse files and it will log an ERROR message
|
|
if one is detected.
|
|
.SS VFS Chunked Reading
|
|
.PP
|
|
When rclone reads files from a remote it reads them in chunks.
|
|
This means that rather than requesting the whole file rclone reads the
|
|
chunk specified.
|
|
This can reduce the used download quota for some remotes by requesting
|
|
only chunks from the remote that are actually read, at the cost of an
|
|
increased number of requests.
|
|
.PP
|
|
These flags control the chunking:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
|
|
--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Rclone will start reading a chunk of size
|
|
\f[C]--vfs-read-chunk-size\f[R], and then double the size for each read.
|
|
When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater
|
|
than \f[C]--vfs-read-chunk-size\f[R], the chunk size for each open file
|
|
will get doubled only until the specified value is reached.
|
|
If the value is \[dq]off\[dq], which is the default, the limit is
|
|
disabled and the chunk size will grow indefinitely.
|
|
.PP
|
|
With \f[C]--vfs-read-chunk-size 100M\f[R] and
|
|
\f[C]--vfs-read-chunk-size-limit 0\f[R] the following parts will be
|
|
downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
|
|
When \f[C]--vfs-read-chunk-size-limit 500M\f[R] is specified, the result
|
|
would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so
|
|
on.
|
|
.PP
|
|
Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq]
|
|
disables chunked reading.
|
|
.SS VFS Performance
|
|
.PP
|
|
These flags may be used to enable/disable features of the VFS for
|
|
performance or other reasons.
|
|
See also the chunked reading feature.
|
|
.PP
|
|
In particular S3 and Swift benefit hugely from the
|
|
\f[C]--no-modtime\f[R] flag (or use \f[C]--use-server-modtime\f[R] for a
|
|
slightly different effect) as each read of the modification time takes a
|
|
transaction.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--no-checksum Don\[aq]t compare checksums on up/download.
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up).
|
|
--no-seek Don\[aq]t allow seeking in files.
|
|
--read-only Mount read-only.
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sometimes rclone is delivered reads or writes out of order.
|
|
Rather than seeking rclone will wait a short time for the in sequence
|
|
read or write to come in.
|
|
These flags only come into effect when not using an on disk cache file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
When using VFS write caching (\f[C]--vfs-cache-mode\f[R] with value
|
|
writes or full), the global flag \f[C]--transfers\f[R] can be set to
|
|
adjust the number of parallel uploads of modified files from cache (the
|
|
related global flag \f[C]--checkers\f[R] have no effect on mount).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--transfers int Number of file transfers to run in parallel (default 4)
|
|
\f[R]
|
|
.fi
|
|
.SS VFS Case Sensitivity
|
|
.PP
|
|
Linux file systems are case-sensitive: two files can differ only by
|
|
case, and the exact case must be used when opening a file.
|
|
.PP
|
|
File systems in modern Windows are case-insensitive but case-preserving:
|
|
although existing files can be opened using any case, the exact case
|
|
used to create the file is preserved and available for programs to
|
|
query.
|
|
It is not allowed for two files in the same directory to differ only by
|
|
case.
|
|
.PP
|
|
Usually file systems on macOS are case-insensitive.
|
|
It is possible to make macOS file systems case-sensitive but that is not
|
|
the default.
|
|
.PP
|
|
The \f[C]--vfs-case-insensitive\f[R] mount flag controls how rclone
|
|
handles these two cases.
|
|
If its value is \[dq]false\[dq], rclone passes file names to the mounted
|
|
file system as-is.
|
|
If the flag is \[dq]true\[dq] (or appears without a value on command
|
|
line), rclone may perform a \[dq]fixup\[dq] as explained below.
|
|
.PP
|
|
The user may specify a file name to open/delete/rename/etc with a case
|
|
different than what is stored on mounted file system.
|
|
If an argument refers to an existing file with exactly the same name,
|
|
then the case of the existing file on the disk will be used.
|
|
However, if a file name with exactly the same name is not found but a
|
|
name differing only by case exists, rclone will transparently fixup the
|
|
name.
|
|
This fixup happens only when an existing file is requested.
|
|
Case sensitivity of file names created anew by rclone is controlled by
|
|
an underlying mounted file system.
|
|
.PP
|
|
Note that case sensitivity of the operating system running rclone (the
|
|
target) may differ from case sensitivity of a file system mounted by
|
|
rclone (the source).
|
|
The flag controls whether \[dq]fixup\[dq] is performed to satisfy the
|
|
target.
|
|
.PP
|
|
If the flag is not provided on the command line, then its default value
|
|
depends on the operating system where rclone runs: \[dq]true\[dq] on
|
|
Windows and macOS, \[dq]false\[dq] otherwise.
|
|
If the flag is provided without a value, then it is \[dq]true\[dq].
|
|
.SS Alternate report of used bytes
|
|
.PP
|
|
Some backends, most notably S3, do not report the amount of bytes used.
|
|
If you need this information to be available when running \f[C]df\f[R]
|
|
on the filesystem, then pass the flag \f[C]--vfs-used-is-size\f[R] to
|
|
rclone.
|
|
With this flag set, instead of relying on the backend to report this
|
|
information, rclone will scan the whole remote similar to
|
|
\f[C]rclone size\f[R] and compute the total used space itself.
|
|
.PP
|
|
\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
|
|
filters so that the result is accurate.
|
|
However, this is very inefficient and may cost lots of API calls
|
|
resulting in extra charges.
|
|
Use it as a last resort and only with caching.
|
|
.SS Auth Proxy
|
|
.PP
|
|
If you supply the parameter \f[C]--auth-proxy /path/to/program\f[R] then
|
|
rclone will use that program to generate backends on the fly which then
|
|
are used to authenticate incoming requests.
|
|
This uses a simple JSON based protocol with input on STDIN and output on
|
|
STDOUT.
|
|
.PP
|
|
\f[B]PLEASE NOTE:\f[R] \f[C]--auth-proxy\f[R] and
|
|
\f[C]--authorized-keys\f[R] cannot be used together, if
|
|
\f[C]--auth-proxy\f[R] is set the authorized keys option will be
|
|
ignored.
|
|
.PP
|
|
There is an example program
|
|
bin/test_proxy.py (https://github.com/rclone/rclone/blob/master/test_proxy.py)
|
|
in the rclone source code.
|
|
.PP
|
|
The program\[aq]s job is to take a \f[C]user\f[R] and \f[C]pass\f[R] on
|
|
the input and turn those into the config for a backend on STDOUT in JSON
|
|
format.
|
|
This config will have any default parameters for the backend added, but
|
|
it won\[aq]t use configuration from environment variables or command
|
|
line options - it is the job of the proxy program to make a complete
|
|
config.
|
|
.PP
|
|
This config generated must have this extra parameter - \f[C]_root\f[R] -
|
|
root to use for the backend
|
|
.PP
|
|
And it may have this parameter - \f[C]_obscure\f[R] - comma separated
|
|
strings for parameters to obscure
|
|
.PP
|
|
If password authentication was used by the client, input to the proxy
|
|
process (on STDIN) would look similar to this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]user\[dq]: \[dq]me\[dq],
|
|
\[dq]pass\[dq]: \[dq]mypassword\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If public-key authentication was used by the client, input to the proxy
|
|
process (on STDIN) would look similar to this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]user\[dq]: \[dq]me\[dq],
|
|
\[dq]public_key\[dq]: \[dq]AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And as an example return this on STDOUT
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]type\[dq]: \[dq]sftp\[dq],
|
|
\[dq]_root\[dq]: \[dq]\[dq],
|
|
\[dq]_obscure\[dq]: \[dq]pass\[dq],
|
|
\[dq]user\[dq]: \[dq]me\[dq],
|
|
\[dq]pass\[dq]: \[dq]mypassword\[dq],
|
|
\[dq]host\[dq]: \[dq]sftp.example.com\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This would mean that an SFTP backend would be created on the fly for the
|
|
\f[C]user\f[R] and \f[C]pass\f[R]/\f[C]public_key\f[R] returned in the
|
|
output to the host given.
|
|
Note that since \f[C]_obscure\f[R] is set to \f[C]pass\f[R], rclone will
|
|
obscure the \f[C]pass\f[R] parameter before creating the backend (which
|
|
is required for sftp backends).
|
|
.PP
|
|
The program can manipulate the supplied \f[C]user\f[R] in any way, for
|
|
example to make proxy to many different sftp backends, you could make
|
|
the \f[C]user\f[R] be \f[C]user\[at]example.com\f[R] and then set the
|
|
\f[C]host\f[R] to \f[C]example.com\f[R] in the output and the user to
|
|
\f[C]user\f[R].
|
|
For security you\[aq]d probably want to restrict the \f[C]host\f[R] to a
|
|
limited list.
|
|
.PP
|
|
Note that an internal cache is keyed on \f[C]user\f[R] so only use that
|
|
for configuration, don\[aq]t use \f[C]pass\f[R] or \f[C]public_key\f[R].
|
|
This also means that if a user\[aq]s password or public-key is changed
|
|
the cache will need to expire (which takes 5 mins) before it takes
|
|
effect.
|
|
.PP
|
|
This can be used to build general purpose proxies to any kind of backend
|
|
that rclone supports.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone serve sftp remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--addr string IPaddress:Port or :Port to bind server to (default \[dq]localhost:2022\[dq])
|
|
--auth-proxy string A program to use to create the backend from the auth
|
|
--authorized-keys string Authorized keys file (default \[dq]\[ti]/.ssh/authorized_keys\[dq])
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--dir-perms FileMode Directory permissions (default 0777)
|
|
--file-perms FileMode File permissions (default 0666)
|
|
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
|
|
-h, --help help for sftp
|
|
--key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate)
|
|
--no-auth Allow connections with no authentication if set
|
|
--no-checksum Don\[aq]t compare checksums on up/download
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up)
|
|
--no-seek Don\[aq]t allow seeking in files
|
|
--pass string Password for authentication
|
|
--poll-interval duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
|
|
--read-only Mount read-only
|
|
--stdio Run an sftp server on run stdin/stdout
|
|
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
|
|
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
|
|
--user string User name for authentication
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-case-insensitive If a file name not found, find a case insensitive match
|
|
--vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
|
|
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off)
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a
|
|
remote over a protocol.
|
|
.SH rclone serve webdav
|
|
.PP
|
|
Serve remote:path over webdav.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone serve webdav implements a basic webdav server to serve the remote
|
|
over HTTP via the webdav protocol.
|
|
This can be viewed with a webdav client, through a web browser, or you
|
|
can make a remote of type webdav to read and write it.
|
|
.SS Webdav options
|
|
.SS --etag-hash
|
|
.PP
|
|
This controls the ETag header.
|
|
Without this flag the ETag will be based on the ModTime and Size of the
|
|
object.
|
|
.PP
|
|
If this flag is set to \[dq]auto\[dq] then rclone will choose the first
|
|
supported hash on the backend or you can use a named hash such as
|
|
\[dq]MD5\[dq] or \[dq]SHA-1\[dq].
|
|
.PP
|
|
Use \[dq]rclone hashsum\[dq] to see the full list.
|
|
.SS Server options
|
|
.PP
|
|
Use --addr to specify which IP address and port the server should listen
|
|
on, e.g.
|
|
--addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs.
|
|
By default it only listens on localhost.
|
|
You can use port :0 to let the OS choose an available port.
|
|
.PP
|
|
If you set --addr to listen on a public or LAN accessible IP address
|
|
then using Authentication is advised - see the next section for info.
|
|
.PP
|
|
--server-read-timeout and --server-write-timeout can be used to control
|
|
the timeouts on the server.
|
|
Note that this is the total time for a transfer.
|
|
.PP
|
|
--max-header-bytes controls the maximum number of bytes the server will
|
|
accept in the HTTP header.
|
|
.PP
|
|
--baseurl controls the URL prefix that rclone serves from.
|
|
By default rclone will serve from the root.
|
|
If you used --baseurl \[dq]/rclone\[dq] then rclone would serve from a
|
|
URL starting with \[dq]/rclone/\[dq].
|
|
This is useful if you wish to proxy rclone serve.
|
|
Rclone automatically inserts leading and trailing \[dq]/\[dq] on
|
|
--baseurl, so --baseurl \[dq]rclone\[dq], --baseurl \[dq]/rclone\[dq]
|
|
and --baseurl \[dq]/rclone/\[dq] are all treated identically.
|
|
.PP
|
|
--template allows a user to specify a custom markup template for http
|
|
and webdav serve functions.
|
|
The server exports the following markup to be used within the template
|
|
to server pages:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
lw(35.0n) lw(35.0n).
|
|
T{
|
|
Parameter
|
|
T}@T{
|
|
Description
|
|
T}
|
|
_
|
|
T{
|
|
\&.Name
|
|
T}@T{
|
|
The full path of a file/directory.
|
|
T}
|
|
T{
|
|
\&.Title
|
|
T}@T{
|
|
Directory listing of .Name
|
|
T}
|
|
T{
|
|
\&.Sort
|
|
T}@T{
|
|
The current sort used.
|
|
This is changeable via ?sort= parameter
|
|
T}
|
|
T{
|
|
T}@T{
|
|
Sort Options: namedirfirst,name,size,time (default namedirfirst)
|
|
T}
|
|
T{
|
|
\&.Order
|
|
T}@T{
|
|
The current ordering used.
|
|
This is changeable via ?order= parameter
|
|
T}
|
|
T{
|
|
T}@T{
|
|
Order Options: asc,desc (default asc)
|
|
T}
|
|
T{
|
|
\&.Query
|
|
T}@T{
|
|
Currently unused.
|
|
T}
|
|
T{
|
|
\&.Breadcrumb
|
|
T}@T{
|
|
Allows for creating a relative navigation
|
|
T}
|
|
T{
|
|
-- .Link
|
|
T}@T{
|
|
The relative to the root link of the Text.
|
|
T}
|
|
T{
|
|
-- .Text
|
|
T}@T{
|
|
The Name of the directory.
|
|
T}
|
|
T{
|
|
\&.Entries
|
|
T}@T{
|
|
Information about a specific file/directory.
|
|
T}
|
|
T{
|
|
-- .URL
|
|
T}@T{
|
|
The \[aq]url\[aq] of an entry.
|
|
T}
|
|
T{
|
|
-- .Leaf
|
|
T}@T{
|
|
Currently same as \[aq]URL\[aq] but intended to be \[aq]just\[aq] the
|
|
name.
|
|
T}
|
|
T{
|
|
-- .IsDir
|
|
T}@T{
|
|
Boolean for if an entry is a directory or not.
|
|
T}
|
|
T{
|
|
-- .Size
|
|
T}@T{
|
|
Size in Bytes of the entry.
|
|
T}
|
|
T{
|
|
-- .ModTime
|
|
T}@T{
|
|
The UTC timestamp of an entry.
|
|
T}
|
|
.TE
|
|
.SS Authentication
|
|
.PP
|
|
By default this will serve files without needing a login.
|
|
.PP
|
|
You can either use an htpasswd file which can take lots of users, or set
|
|
a single username and password with the --user and --pass flags.
|
|
.PP
|
|
Use --htpasswd /path/to/htpasswd to provide an htpasswd file.
|
|
This is in standard apache format and supports MD5, SHA1 and BCrypt for
|
|
basic authentication.
|
|
Bcrypt is recommended.
|
|
.PP
|
|
To create an htpasswd file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
touch htpasswd
|
|
htpasswd -B htpasswd user
|
|
htpasswd -B htpasswd anotherUser
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The password file can be updated while rclone is running.
|
|
.PP
|
|
Use --realm to set the authentication realm.
|
|
.SS SSL/TLS
|
|
.PP
|
|
By default this will serve over http.
|
|
If you want you can serve over https.
|
|
You will need to supply the --cert and --key flags.
|
|
If you wish to do client side certificate validation then you will need
|
|
to supply --client-ca also.
|
|
.PP
|
|
--cert should be either a PEM encoded certificate or a concatenation of
|
|
that with the CA certificate.
|
|
--key should be the PEM encoded private key and --client-ca should be
|
|
the PEM encoded client certificate authority certificate.
|
|
.SS VFS - Virtual File System
|
|
.PP
|
|
This command uses the VFS layer.
|
|
This adapts the cloud storage objects that rclone uses into something
|
|
which looks much more like a disk filing system.
|
|
.PP
|
|
Cloud storage objects have lots of properties which aren\[aq]t like disk
|
|
files - you can\[aq]t extend them or write to the middle of them, so the
|
|
VFS layer has to deal with that.
|
|
Because there is no one right way of doing this there are various
|
|
options explained below.
|
|
.PP
|
|
The VFS layer also implements a directory cache - this caches info about
|
|
files and directories (but not the data) in memory.
|
|
.SS VFS Directory Cache
|
|
.PP
|
|
Using the \f[C]--dir-cache-time\f[R] flag, you can control how long a
|
|
directory should be considered up to date and not refreshed from the
|
|
backend.
|
|
Changes made through the mount will appear immediately or invalidate the
|
|
cache.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
However, changes made directly on the cloud storage by the web interface
|
|
or a different copy of rclone will only be picked up once the directory
|
|
cache expires if the backend configured does not support polling for
|
|
changes.
|
|
If the backend supports polling, changes will be picked up within the
|
|
polling interval.
|
|
.PP
|
|
You can send a \f[C]SIGHUP\f[R] signal to rclone for it to flush all
|
|
directory caches, regardless of how old they are.
|
|
Assuming only one rclone instance is running, you can reset the cache
|
|
like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill -SIGHUP $(pidof rclone)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control then you can use rclone rc
|
|
to flush the whole directory cache:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or individual files or directories:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget file=path/to/file dir=path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.SS VFS File Buffering
|
|
.PP
|
|
The \f[C]--buffer-size\f[R] flag determines the amount of memory, that
|
|
will be used to buffer data in advance.
|
|
.PP
|
|
Each open file will try to keep the specified amount of data in memory
|
|
at all times.
|
|
The buffered data is bound to one open file and won\[aq]t be shared.
|
|
.PP
|
|
This flag is a upper limit for the used memory per open file.
|
|
The buffer will only use memory for data that is downloaded but not not
|
|
yet read.
|
|
If the buffer is empty, only a small amount of memory will be used.
|
|
.PP
|
|
The maximum memory used by rclone for buffering can be up to
|
|
\f[C]--buffer-size * open files\f[R].
|
|
.SS VFS File Caching
|
|
.PP
|
|
These flags control the VFS file caching options.
|
|
File caching is necessary to make the VFS layer appear compatible with a
|
|
normal file system.
|
|
It can be disabled at the cost of some compatibility.
|
|
.PP
|
|
For example you\[aq]ll need to enable VFS caching if you want to read
|
|
and write simultaneously to a file.
|
|
See below for more details.
|
|
.PP
|
|
Note that the VFS cache is separate from the cache backend and you may
|
|
find that you need one or the other or both.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--cache-dir string Directory rclone will use for caching.
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If run with \f[C]-vv\f[R] rclone will print the location of the file
|
|
cache.
|
|
The files are stored in the user cache file area which is OS dependent
|
|
but can be controlled with \f[C]--cache-dir\f[R] or setting the
|
|
appropriate environment variable.
|
|
.PP
|
|
The cache has 4 different modes selected by \f[C]--vfs-cache-mode\f[R].
|
|
The higher the cache mode the more compatible rclone becomes at the cost
|
|
of using disk space.
|
|
.PP
|
|
Note that files are written back to the remote only when they are closed
|
|
and if they haven\[aq]t been accessed for \f[C]--vfs-write-back\f[R]
|
|
seconds.
|
|
If rclone is quit or dies with files that haven\[aq]t been uploaded,
|
|
these will be uploaded next time rclone is run with the same flags.
|
|
.PP
|
|
If using \f[C]--vfs-cache-max-size\f[R] note that the cache may exceed
|
|
this size for two reasons.
|
|
Firstly because it is only checked every
|
|
\f[C]--vfs-cache-poll-interval\f[R].
|
|
Secondly because open files cannot be evicted from the cache.
|
|
.PP
|
|
You \f[B]should not\f[R] run two copies of rclone using the same VFS
|
|
cache with the same or overlapping remotes if using
|
|
\f[C]--vfs-cache-mode > off\f[R].
|
|
This can potentially cause data corruption if you do.
|
|
You can work around this by giving each rclone its own cache hierarchy
|
|
with \f[C]--cache-dir\f[R].
|
|
You don\[aq]t need to worry about this if the remotes in use don\[aq]t
|
|
overlap.
|
|
.SS --vfs-cache-mode off
|
|
.PP
|
|
In this mode (the default) the cache will read directly from the remote
|
|
and write directly to the remote without caching anything on disk.
|
|
.PP
|
|
This will mean some operations are not possible
|
|
.IP \[bu] 2
|
|
Files can\[aq]t be opened for both read AND write
|
|
.IP \[bu] 2
|
|
Files opened for write can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files open for read with O_TRUNC will be opened write only
|
|
.IP \[bu] 2
|
|
Files open for write only will behave as if O_TRUNC was supplied
|
|
.IP \[bu] 2
|
|
Open modes O_APPEND, O_TRUNC are ignored
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode minimal
|
|
.PP
|
|
This is very similar to \[dq]off\[dq] except that files opened for read
|
|
AND write will be buffered to disk.
|
|
This means that files opened for write will be a lot more compatible,
|
|
but uses the minimal disk space.
|
|
.PP
|
|
These operations are not possible
|
|
.IP \[bu] 2
|
|
Files opened for write only can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files opened for write only will ignore O_APPEND, O_TRUNC
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS --vfs-cache-mode writes
|
|
.PP
|
|
In this mode files opened for read only are still read directly from the
|
|
remote, write only and read/write files are buffered to disk first.
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload fails it will be retried at exponentially increasing
|
|
intervals up to 1 minute.
|
|
.SS --vfs-cache-mode full
|
|
.PP
|
|
In this mode all reads and writes are buffered to and from disk.
|
|
When data is read from the remote this is buffered to disk as well.
|
|
.PP
|
|
In this mode the files in the cache will be sparse files and rclone will
|
|
keep track of which bits of the files it has downloaded.
|
|
.PP
|
|
So if an application only reads the starts of each file, then rclone
|
|
will only buffer the start of the file.
|
|
These files will appear to be their full size in the cache, but they
|
|
will be sparse files with only the data that has been downloaded present
|
|
in them.
|
|
.PP
|
|
This mode should support all normal file system operations and is
|
|
otherwise identical to \f[C]--vfs-cache-mode\f[R] writes.
|
|
.PP
|
|
When reading a file rclone will read \f[C]--buffer-size\f[R] plus
|
|
\f[C]--vfs-read-ahead\f[R] bytes ahead.
|
|
The \f[C]--buffer-size\f[R] is buffered in memory whereas the
|
|
\f[C]--vfs-read-ahead\f[R] is buffered on disk.
|
|
.PP
|
|
When using this mode it is recommended that \f[C]--buffer-size\f[R] is
|
|
not set too large and \f[C]--vfs-read-ahead\f[R] is set large if
|
|
required.
|
|
.PP
|
|
\f[B]IMPORTANT\f[R] not all file systems support sparse files.
|
|
In particular FAT/exFAT do not.
|
|
Rclone will perform very badly if the cache directory is on a filesystem
|
|
which doesn\[aq]t support sparse files and it will log an ERROR message
|
|
if one is detected.
|
|
.SS VFS Chunked Reading
|
|
.PP
|
|
When rclone reads files from a remote it reads them in chunks.
|
|
This means that rather than requesting the whole file rclone reads the
|
|
chunk specified.
|
|
This can reduce the used download quota for some remotes by requesting
|
|
only chunks from the remote that are actually read, at the cost of an
|
|
increased number of requests.
|
|
.PP
|
|
These flags control the chunking:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
|
|
--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Rclone will start reading a chunk of size
|
|
\f[C]--vfs-read-chunk-size\f[R], and then double the size for each read.
|
|
When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater
|
|
than \f[C]--vfs-read-chunk-size\f[R], the chunk size for each open file
|
|
will get doubled only until the specified value is reached.
|
|
If the value is \[dq]off\[dq], which is the default, the limit is
|
|
disabled and the chunk size will grow indefinitely.
|
|
.PP
|
|
With \f[C]--vfs-read-chunk-size 100M\f[R] and
|
|
\f[C]--vfs-read-chunk-size-limit 0\f[R] the following parts will be
|
|
downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
|
|
When \f[C]--vfs-read-chunk-size-limit 500M\f[R] is specified, the result
|
|
would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so
|
|
on.
|
|
.PP
|
|
Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq]
|
|
disables chunked reading.
|
|
.SS VFS Performance
|
|
.PP
|
|
These flags may be used to enable/disable features of the VFS for
|
|
performance or other reasons.
|
|
See also the chunked reading feature.
|
|
.PP
|
|
In particular S3 and Swift benefit hugely from the
|
|
\f[C]--no-modtime\f[R] flag (or use \f[C]--use-server-modtime\f[R] for a
|
|
slightly different effect) as each read of the modification time takes a
|
|
transaction.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--no-checksum Don\[aq]t compare checksums on up/download.
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up).
|
|
--no-seek Don\[aq]t allow seeking in files.
|
|
--read-only Mount read-only.
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sometimes rclone is delivered reads or writes out of order.
|
|
Rather than seeking rclone will wait a short time for the in sequence
|
|
read or write to come in.
|
|
These flags only come into effect when not using an on disk cache file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
When using VFS write caching (\f[C]--vfs-cache-mode\f[R] with value
|
|
writes or full), the global flag \f[C]--transfers\f[R] can be set to
|
|
adjust the number of parallel uploads of modified files from cache (the
|
|
related global flag \f[C]--checkers\f[R] have no effect on mount).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--transfers int Number of file transfers to run in parallel (default 4)
|
|
\f[R]
|
|
.fi
|
|
.SS VFS Case Sensitivity
|
|
.PP
|
|
Linux file systems are case-sensitive: two files can differ only by
|
|
case, and the exact case must be used when opening a file.
|
|
.PP
|
|
File systems in modern Windows are case-insensitive but case-preserving:
|
|
although existing files can be opened using any case, the exact case
|
|
used to create the file is preserved and available for programs to
|
|
query.
|
|
It is not allowed for two files in the same directory to differ only by
|
|
case.
|
|
.PP
|
|
Usually file systems on macOS are case-insensitive.
|
|
It is possible to make macOS file systems case-sensitive but that is not
|
|
the default.
|
|
.PP
|
|
The \f[C]--vfs-case-insensitive\f[R] mount flag controls how rclone
|
|
handles these two cases.
|
|
If its value is \[dq]false\[dq], rclone passes file names to the mounted
|
|
file system as-is.
|
|
If the flag is \[dq]true\[dq] (or appears without a value on command
|
|
line), rclone may perform a \[dq]fixup\[dq] as explained below.
|
|
.PP
|
|
The user may specify a file name to open/delete/rename/etc with a case
|
|
different than what is stored on mounted file system.
|
|
If an argument refers to an existing file with exactly the same name,
|
|
then the case of the existing file on the disk will be used.
|
|
However, if a file name with exactly the same name is not found but a
|
|
name differing only by case exists, rclone will transparently fixup the
|
|
name.
|
|
This fixup happens only when an existing file is requested.
|
|
Case sensitivity of file names created anew by rclone is controlled by
|
|
an underlying mounted file system.
|
|
.PP
|
|
Note that case sensitivity of the operating system running rclone (the
|
|
target) may differ from case sensitivity of a file system mounted by
|
|
rclone (the source).
|
|
The flag controls whether \[dq]fixup\[dq] is performed to satisfy the
|
|
target.
|
|
.PP
|
|
If the flag is not provided on the command line, then its default value
|
|
depends on the operating system where rclone runs: \[dq]true\[dq] on
|
|
Windows and macOS, \[dq]false\[dq] otherwise.
|
|
If the flag is provided without a value, then it is \[dq]true\[dq].
|
|
.SS Alternate report of used bytes
|
|
.PP
|
|
Some backends, most notably S3, do not report the amount of bytes used.
|
|
If you need this information to be available when running \f[C]df\f[R]
|
|
on the filesystem, then pass the flag \f[C]--vfs-used-is-size\f[R] to
|
|
rclone.
|
|
With this flag set, instead of relying on the backend to report this
|
|
information, rclone will scan the whole remote similar to
|
|
\f[C]rclone size\f[R] and compute the total used space itself.
|
|
.PP
|
|
\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
|
|
filters so that the result is accurate.
|
|
However, this is very inefficient and may cost lots of API calls
|
|
resulting in extra charges.
|
|
Use it as a last resort and only with caching.
|
|
.SS Auth Proxy
|
|
.PP
|
|
If you supply the parameter \f[C]--auth-proxy /path/to/program\f[R] then
|
|
rclone will use that program to generate backends on the fly which then
|
|
are used to authenticate incoming requests.
|
|
This uses a simple JSON based protocol with input on STDIN and output on
|
|
STDOUT.
|
|
.PP
|
|
\f[B]PLEASE NOTE:\f[R] \f[C]--auth-proxy\f[R] and
|
|
\f[C]--authorized-keys\f[R] cannot be used together, if
|
|
\f[C]--auth-proxy\f[R] is set the authorized keys option will be
|
|
ignored.
|
|
.PP
|
|
There is an example program
|
|
bin/test_proxy.py (https://github.com/rclone/rclone/blob/master/test_proxy.py)
|
|
in the rclone source code.
|
|
.PP
|
|
The program\[aq]s job is to take a \f[C]user\f[R] and \f[C]pass\f[R] on
|
|
the input and turn those into the config for a backend on STDOUT in JSON
|
|
format.
|
|
This config will have any default parameters for the backend added, but
|
|
it won\[aq]t use configuration from environment variables or command
|
|
line options - it is the job of the proxy program to make a complete
|
|
config.
|
|
.PP
|
|
This config generated must have this extra parameter - \f[C]_root\f[R] -
|
|
root to use for the backend
|
|
.PP
|
|
And it may have this parameter - \f[C]_obscure\f[R] - comma separated
|
|
strings for parameters to obscure
|
|
.PP
|
|
If password authentication was used by the client, input to the proxy
|
|
process (on STDIN) would look similar to this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]user\[dq]: \[dq]me\[dq],
|
|
\[dq]pass\[dq]: \[dq]mypassword\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If public-key authentication was used by the client, input to the proxy
|
|
process (on STDIN) would look similar to this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]user\[dq]: \[dq]me\[dq],
|
|
\[dq]public_key\[dq]: \[dq]AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And as an example return this on STDOUT
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]type\[dq]: \[dq]sftp\[dq],
|
|
\[dq]_root\[dq]: \[dq]\[dq],
|
|
\[dq]_obscure\[dq]: \[dq]pass\[dq],
|
|
\[dq]user\[dq]: \[dq]me\[dq],
|
|
\[dq]pass\[dq]: \[dq]mypassword\[dq],
|
|
\[dq]host\[dq]: \[dq]sftp.example.com\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This would mean that an SFTP backend would be created on the fly for the
|
|
\f[C]user\f[R] and \f[C]pass\f[R]/\f[C]public_key\f[R] returned in the
|
|
output to the host given.
|
|
Note that since \f[C]_obscure\f[R] is set to \f[C]pass\f[R], rclone will
|
|
obscure the \f[C]pass\f[R] parameter before creating the backend (which
|
|
is required for sftp backends).
|
|
.PP
|
|
The program can manipulate the supplied \f[C]user\f[R] in any way, for
|
|
example to make proxy to many different sftp backends, you could make
|
|
the \f[C]user\f[R] be \f[C]user\[at]example.com\f[R] and then set the
|
|
\f[C]host\f[R] to \f[C]example.com\f[R] in the output and the user to
|
|
\f[C]user\f[R].
|
|
For security you\[aq]d probably want to restrict the \f[C]host\f[R] to a
|
|
limited list.
|
|
.PP
|
|
Note that an internal cache is keyed on \f[C]user\f[R] so only use that
|
|
for configuration, don\[aq]t use \f[C]pass\f[R] or \f[C]public_key\f[R].
|
|
This also means that if a user\[aq]s password or public-key is changed
|
|
the cache will need to expire (which takes 5 mins) before it takes
|
|
effect.
|
|
.PP
|
|
This can be used to build general purpose proxies to any kind of backend
|
|
that rclone supports.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone serve webdav remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--addr string IPaddress:Port or :Port to bind server to (default \[dq]localhost:8080\[dq])
|
|
--auth-proxy string A program to use to create the backend from the auth
|
|
--baseurl string Prefix for URLs - leave blank for root
|
|
--cert string SSL PEM key (concatenation of certificate and CA certificate)
|
|
--client-ca string Client certificate authority to verify clients with
|
|
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
|
|
--dir-perms FileMode Directory permissions (default 0777)
|
|
--disable-dir-list Disable HTML directory list on GET request for a directory
|
|
--etag-hash string Which hash to use for the ETag, or auto or blank for off
|
|
--file-perms FileMode File permissions (default 0666)
|
|
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
|
|
-h, --help help for webdav
|
|
--htpasswd string htpasswd file - if not provided no authentication is done
|
|
--key string SSL PEM Private key
|
|
--max-header-bytes int Maximum size of request header (default 4096)
|
|
--no-checksum Don\[aq]t compare checksums on up/download
|
|
--no-modtime Don\[aq]t read/write the modification time (can speed things up)
|
|
--no-seek Don\[aq]t allow seeking in files
|
|
--pass string Password for authentication
|
|
--poll-interval duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
|
|
--read-only Mount read-only
|
|
--realm string Realm for authentication (default \[dq]rclone\[dq])
|
|
--server-read-timeout duration Timeout for server reading data (default 1h0m0s)
|
|
--server-write-timeout duration Timeout for server writing data (default 1h0m0s)
|
|
--template string User-specified template
|
|
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
|
|
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
|
|
--user string User name for authentication
|
|
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
|
|
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
|
|
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
|
|
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
|
|
--vfs-case-insensitive If a file name not found, find a case insensitive match
|
|
--vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
|
|
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
|
|
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off)
|
|
--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
|
|
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
|
|
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
|
|
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone serve (https://rclone.org/commands/rclone_serve/) - Serve a
|
|
remote over a protocol.
|
|
.SH rclone settier
|
|
.PP
|
|
Changes storage class/tier of objects in remote.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone settier changes storage tier or class at remote if supported.
|
|
Few cloud storage services provides different storage classes on
|
|
objects, for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool
|
|
and Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline
|
|
etc.
|
|
.PP
|
|
Note that, certain tier changes make objects not available to access
|
|
immediately.
|
|
For example tiering to archive in azure blob storage makes objects in
|
|
frozen state, user can restore by setting tier to Hot/Cool, similarly S3
|
|
to Glacier makes object inaccessible.true
|
|
.PP
|
|
You can use it to tier single object
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone settier Cool remote:path/file
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or use rclone filters to set tier on only specific files
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone --include \[dq]*.txt\[dq] settier Hot remote:path/dir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or just provide remote directory and all files in directory will be
|
|
tiered
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone settier tier remote:path/dir
|
|
\f[R]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone settier tier remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for settier
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone test
|
|
.PP
|
|
Run a test command
|
|
.SS Synopsis
|
|
.PP
|
|
Rclone test is used to run test commands.
|
|
.PP
|
|
Select which test comand you want with the subcommand, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone test memory remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Each subcommand has its own options which you can see in their help.
|
|
.PP
|
|
\f[B]NB\f[R] Be careful running these commands, they may do strange
|
|
things so reading their documentation first is recommended.
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for test
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.IP \[bu] 2
|
|
rclone test
|
|
changenotify (https://rclone.org/commands/rclone_test_changenotify/) -
|
|
Log any change notify requests for the remote passed in.
|
|
.IP \[bu] 2
|
|
rclone test
|
|
histogram (https://rclone.org/commands/rclone_test_histogram/) - Makes a
|
|
histogram of file name characters.
|
|
.IP \[bu] 2
|
|
rclone test info (https://rclone.org/commands/rclone_test_info/) -
|
|
Discovers file name or other limitations for paths.
|
|
.IP \[bu] 2
|
|
rclone test
|
|
makefiles (https://rclone.org/commands/rclone_test_makefiles/) - Make a
|
|
random file hierarchy in a directory
|
|
.IP \[bu] 2
|
|
rclone test memory (https://rclone.org/commands/rclone_test_memory/) -
|
|
Load all the objects at remote:path into memory and report memory stats.
|
|
.SH rclone test changenotify
|
|
.PP
|
|
Log any change notify requests for the remote passed in.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone test changenotify remote: [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for changenotify
|
|
--poll-interval duration Time to wait between polling for changes (default 10s)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone test (https://rclone.org/commands/rclone_test/) - Run a test
|
|
command
|
|
.SH rclone test histogram
|
|
.PP
|
|
Makes a histogram of file name characters.
|
|
.SS Synopsis
|
|
.PP
|
|
This command outputs JSON which shows the histogram of characters used
|
|
in filenames in the remote:path specified.
|
|
.PP
|
|
The data doesn\[aq]t contain any identifying information but is useful
|
|
for the rclone developers when developing filename compression.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone test histogram [remote:path] [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for histogram
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone test (https://rclone.org/commands/rclone_test/) - Run a test
|
|
command
|
|
.SH rclone test info
|
|
.PP
|
|
Discovers file name or other limitations for paths.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone info discovers what filenames and upload methods are possible to
|
|
write to the paths passed in and how long they can be.
|
|
It can take some time.
|
|
It will write test files into the remote:path passed in.
|
|
It outputs a bit of go code for each one.
|
|
.PP
|
|
\f[B]NB\f[R] this can create undeletable files and other hazards - use
|
|
with care
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone test info [remote:path]+ [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--all Run all tests
|
|
--check-control Check control characters
|
|
--check-length Check max filename length
|
|
--check-normalization Check UTF-8 Normalization
|
|
--check-streaming Check uploads with indeterminate file size
|
|
-h, --help help for info
|
|
--upload-wait duration Wait after writing a file
|
|
--write-json string Write results to file
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone test (https://rclone.org/commands/rclone_test/) - Run a test
|
|
command
|
|
.SH rclone test makefiles
|
|
.PP
|
|
Make a random file hierarchy in a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone test makefiles <dir> [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--files int Number of files to create (default 1000)
|
|
--files-per-directory int Average number of files per directory (default 10)
|
|
-h, --help help for makefiles
|
|
--max-file-size SizeSuffix Maximum size of files to create (default 100)
|
|
--max-name-length int Maximum size of file names (default 12)
|
|
--min-file-size SizeSuffix Minimum size of file to create
|
|
--min-name-length int Minimum size of file names (default 4)
|
|
--seed int Seed for the random number generator (0 for random) (default 1)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone test (https://rclone.org/commands/rclone_test/) - Run a test
|
|
command
|
|
.SH rclone test memory
|
|
.PP
|
|
Load all the objects at remote:path into memory and report memory stats.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone test memory remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for memory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone test (https://rclone.org/commands/rclone_test/) - Run a test
|
|
command
|
|
.SH rclone touch
|
|
.PP
|
|
Create new file or change file modification time.
|
|
.SS Synopsis
|
|
.PP
|
|
Set the modification time on file(s) as specified by remote:path to have
|
|
the current time.
|
|
.PP
|
|
If remote:path does not exist then a zero sized file will be created,
|
|
unless \f[C]--no-create\f[R] or \f[C]--recursive\f[R] is provided.
|
|
.PP
|
|
If \f[C]--recursive\f[R] is used then recursively sets the modification
|
|
time on all existing files that is found under the path.
|
|
Filters are supported, and you can test with the \f[C]--dry-run\f[R] or
|
|
the \f[C]--interactive\f[R] flag.
|
|
.PP
|
|
If \f[C]--timestamp\f[R] is used then sets the modification time to that
|
|
time instead of the current time.
|
|
Times may be specified as one of:
|
|
.IP \[bu] 2
|
|
\[aq]YYMMDD\[aq] - e.g.
|
|
17.10.30
|
|
.IP \[bu] 2
|
|
\[aq]YYYY-MM-DDTHH:MM:SS\[aq] - e.g.
|
|
2006-01-02T15:04:05
|
|
.IP \[bu] 2
|
|
\[aq]YYYY-MM-DDTHH:MM:SS.SSS\[aq] - e.g.
|
|
2006-01-02T15:04:05.123456789
|
|
.PP
|
|
Note that value of \f[C]--timestamp\f[R] is in UTC.
|
|
If you want local time then add the \f[C]--localtime\f[R] flag.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone touch remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-h, --help help for touch
|
|
--localtime Use localtime for timestamp, not UTC
|
|
-C, --no-create Do not create the file if it does not exist (implied with --recursive)
|
|
-R, --recursive Recursively touch all files
|
|
-t, --timestamp string Use specified time instead of the current time of day
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SH rclone tree
|
|
.PP
|
|
List the contents of the remote in a tree like fashion.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone tree lists the contents of a remote in a similar way to the unix
|
|
tree command.
|
|
.PP
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone tree remote:path
|
|
/
|
|
\[u251C]\[u2500]\[u2500] file1
|
|
\[u251C]\[u2500]\[u2500] file2
|
|
\[u251C]\[u2500]\[u2500] file3
|
|
\[u2514]\[u2500]\[u2500] subdir
|
|
\[u251C]\[u2500]\[u2500] file4
|
|
\[u2514]\[u2500]\[u2500] file5
|
|
|
|
1 directories, 5 files
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can use any of the filtering options with the tree command (e.g.
|
|
--include and --exclude).
|
|
You can also use --fast-list.
|
|
.PP
|
|
The tree command has many options for controlling the listing which are
|
|
compatible with the tree command.
|
|
Note that not all of them have short options as they conflict with
|
|
rclone\[aq]s short options.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone tree remote:path [flags]
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-a, --all All files are listed (list . files too)
|
|
-C, --color Turn colorization on always
|
|
-d, --dirs-only List directories only
|
|
--dirsfirst List directories before files (-U disables)
|
|
--full-path Print the full path prefix for each file
|
|
-h, --help help for tree
|
|
--level int Descend only level directories deep
|
|
-D, --modtime Print the date of last modification.
|
|
--noindent Don\[aq]t print indentation lines
|
|
--noreport Turn off file/directory count at end of tree listing
|
|
-o, --output string Output to file instead of stdout
|
|
-p, --protections Print the protections for each file.
|
|
-Q, --quote Quote filenames with double quotes.
|
|
-s, --size Print the size in bytes of each file.
|
|
--sort string Select sort: name,version,size,mtime,ctime
|
|
--sort-ctime Sort files by last status change time
|
|
-t, --sort-modtime Sort files by last modification time
|
|
-r, --sort-reverse Reverse the order of the sort
|
|
-U, --unsorted Leave files unsorted
|
|
--version Sort files alphanumerically by version
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the global flags page (https://rclone.org/flags/) for global options
|
|
not listed here.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone (https://rclone.org/commands/rclone/) - Show help for rclone
|
|
commands, flags and backends.
|
|
.SS Copying single files
|
|
.PP
|
|
rclone normally syncs or copies directories.
|
|
However, if the source remote points to a file, rclone will just copy
|
|
that file.
|
|
The destination remote must point to a directory - rclone will give the
|
|
error
|
|
\f[C]Failed to create file system for \[dq]remote:file\[dq]: is a file not a directory\f[R]
|
|
if it isn\[aq]t.
|
|
.PP
|
|
For example, suppose you have a remote with a file in called
|
|
\f[C]test.jpg\f[R], then you could copy just that file like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy remote:test.jpg /tmp/download
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The file \f[C]test.jpg\f[R] will be placed inside
|
|
\f[C]/tmp/download\f[R].
|
|
.PP
|
|
This is equivalent to specifying
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --files-from /tmp/files remote: /tmp/download
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Where \f[C]/tmp/files\f[R] contains the single line
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
test.jpg
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
It is recommended to use \f[C]copy\f[R] when copying individual files,
|
|
not \f[C]sync\f[R].
|
|
They have pretty much the same effect but \f[C]copy\f[R] will use a lot
|
|
less memory.
|
|
.SS Syntax of remote paths
|
|
.PP
|
|
The syntax of the paths passed to the rclone command are as follows.
|
|
.SS /path/to/dir
|
|
.PP
|
|
This refers to the local file system.
|
|
.PP
|
|
On Windows \f[C]\[rs]\f[R] may be used instead of \f[C]/\f[R] in local
|
|
paths \f[B]only\f[R], non local paths must use \f[C]/\f[R].
|
|
See local filesystem (https://rclone.org/local/#paths-on-windows)
|
|
documentation for more about Windows-specific paths.
|
|
.PP
|
|
These paths needn\[aq]t start with a leading \f[C]/\f[R] - if they
|
|
don\[aq]t then they will be relative to the current directory.
|
|
.SS remote:path/to/dir
|
|
.PP
|
|
This refers to a directory \f[C]path/to/dir\f[R] on \f[C]remote:\f[R] as
|
|
defined in the config file (configured with \f[C]rclone config\f[R]).
|
|
.SS remote:/path/to/dir
|
|
.PP
|
|
On most backends this is refers to the same directory as
|
|
\f[C]remote:path/to/dir\f[R] and that format should be preferred.
|
|
On a very small number of remotes (FTP, SFTP, Dropbox for business) this
|
|
will refer to a different directory.
|
|
On these, paths without a leading \f[C]/\f[R] will refer to your
|
|
\[dq]home\[dq] directory and paths with a leading \f[C]/\f[R] will refer
|
|
to the root.
|
|
.SS :backend:path/to/dir
|
|
.PP
|
|
This is an advanced form for creating remotes on the fly.
|
|
\f[C]backend\f[R] should be the name or prefix of a backend (the
|
|
\f[C]type\f[R] in the config file) and all the configuration for the
|
|
backend should be provided on the command line (or in environment
|
|
variables).
|
|
.PP
|
|
Here are some examples:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd --http-url https://pub.rclone.org :http:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To list all the directories in the root of
|
|
\f[C]https://pub.rclone.org/\f[R].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsf --http-url https://example.com :http:path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To list files and directories in
|
|
\f[C]https://example.com/path/to/dir/\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --http-url https://example.com :http:path/to/dir /tmp/dir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy files and directories in
|
|
\f[C]https://example.com/path/to/dir\f[R] to \f[C]/tmp/dir\f[R].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --sftp-host example.com :sftp:path/to/dir /tmp/dir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy files and directories from \f[C]example.com\f[R] in the relative
|
|
directory \f[C]path/to/dir\f[R] to \f[C]/tmp/dir\f[R] using sftp.
|
|
.SS Connection strings
|
|
.PP
|
|
The above examples can also be written using a connection string syntax,
|
|
so instead of providing the arguments as command line parameters
|
|
\f[C]--http-url https://pub.rclone.org\f[R] they are provided as part of
|
|
the remote specification as a kind of connection string.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd \[dq]:http,url=\[aq]https://pub.rclone.org\[aq]:\[dq]
|
|
rclone lsf \[dq]:http,url=\[aq]https://example.com\[aq]:path/to/dir\[dq]
|
|
rclone copy \[dq]:http,url=\[aq]https://example.com\[aq]:path/to/dir\[dq] /tmp/dir
|
|
rclone copy :sftp,host=example.com:path/to/dir /tmp/dir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
These can apply to modify existing remotes as well as create new remotes
|
|
with the on the fly syntax.
|
|
This example is equivalent to adding the
|
|
\f[C]--drive-shared-with-me\f[R] parameter to the remote
|
|
\f[C]gdrive:\f[R].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsf \[dq]gdrive,shared_with_me:path/to/dir\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The major advantage to using the connection string style syntax is that
|
|
it only applies to the remote, not to all the remotes of that type of
|
|
the command line.
|
|
A common confusion is this attempt to copy a file shared on google drive
|
|
to the normal drive which \f[B]does not work\f[R] because the
|
|
\f[C]--drive-shared-with-me\f[R] flag applies to both the source and the
|
|
destination.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --drive-shared-with-me gdrive:shared-file.txt gdrive:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
However using the connection string syntax, this does work.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy \[dq]gdrive,shared_with_me:shared-file.txt\[dq] gdrive:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that the connection string only affects the options of the
|
|
immediate backend.
|
|
If for example gdriveCrypt is a crypt based on gdrive, then the
|
|
following command \f[B]will not work\f[R] as intended, because
|
|
\f[C]shared_with_me\f[R] is ignored by the crypt backend:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy \[dq]gdriveCrypt,shared_with_me:shared-file.txt\[dq] gdriveCrypt:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The connection strings have the following syntax
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
remote,parameter=value,parameter2=value2:path/to/dir
|
|
:backend,parameter=value,parameter2=value2:path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If the \f[C]parameter\f[R] has a \f[C]:\f[R] or \f[C],\f[R] then it must
|
|
be placed in quotes \f[C]\[dq]\f[R] or \f[C]\[aq]\f[R], so
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
remote,parameter=\[dq]colon:value\[dq],parameter2=\[dq]comma,value\[dq]:path/to/dir
|
|
:backend,parameter=\[aq]colon:value\[aq],parameter2=\[aq]comma,value\[aq]:path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If a quoted value needs to include that quote, then it should be
|
|
doubled, so
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
remote,parameter=\[dq]with\[dq]\[dq]quote\[dq],parameter2=\[aq]with\[aq]\[aq]quote\[aq]:path/to/dir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will make \f[C]parameter\f[R] be \f[C]with\[dq]quote\f[R] and
|
|
\f[C]parameter2\f[R] be \f[C]with\[aq]quote\f[R].
|
|
.PP
|
|
If you leave off the \f[C]=parameter\f[R] then rclone will substitute
|
|
\f[C]=true\f[R] which works very well with flags.
|
|
For example, to use s3 configured in the environment you could use:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd :s3,env_auth:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Which is equivalent to
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd :s3,env_auth=true:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that on the command line you might need to surround these
|
|
connection strings with \f[C]\[dq]\f[R] or \f[C]\[aq]\f[R] to stop the
|
|
shell interpreting any special characters within them.
|
|
.PP
|
|
If you are a shell master then you\[aq]ll know which strings are OK and
|
|
which aren\[aq]t, but if you aren\[aq]t sure then enclose them in
|
|
\f[C]\[dq]\f[R] and use \f[C]\[aq]\f[R] as the inside quote.
|
|
This syntax works on all OSes.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy \[dq]:http,url=\[aq]https://example.com\[aq]:path/to/dir\[dq] /tmp/dir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
On Linux/macOS some characters are still interpreted inside
|
|
\f[C]\[dq]\f[R] strings in the shell (notably \f[C]\[rs]\f[R] and
|
|
\f[C]$\f[R] and \f[C]\[dq]\f[R]) so if your strings contain those you
|
|
can swap the roles of \f[C]\[dq]\f[R] and \f[C]\[aq]\f[R] thus.
|
|
(This syntax does not work on Windows.)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy \[aq]:http,url=\[dq]https://example.com\[dq]:path/to/dir\[aq] /tmp/dir
|
|
\f[R]
|
|
.fi
|
|
.SS Connection strings, config and logging
|
|
.PP
|
|
If you supply extra configuration to a backend by command line flag,
|
|
environment variable or connection string then rclone will add a suffix
|
|
based on the hash of the config to the name of the remote, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone -vv lsf --s3-chunk-size 20M s3:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Has the log message
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
DEBUG : s3: detected overridden config - adding \[dq]{Srj1p}\[dq] suffix to name
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This is so rclone can tell the modified remote apart from the unmodified
|
|
remote when caching the backends.
|
|
.PP
|
|
This should only be noticeable in the logs.
|
|
.PP
|
|
This means that on the fly backends such as
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone -vv lsf :s3,env_auth:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Will get their own names
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
DEBUG : :s3: detected overridden config - adding \[dq]{YTu53}\[dq] suffix to name
|
|
\f[R]
|
|
.fi
|
|
.SS Valid remote names
|
|
.PP
|
|
Remote names are case sensitive, and must adhere to the following rules:
|
|
- May only contain \f[C]0\f[R]-\f[C]9\f[R], \f[C]A\f[R]-\f[C]Z\f[R],
|
|
\f[C]a\f[R]-\f[C]z\f[R], \f[C]_\f[R], \f[C]-\f[R], \f[C].\f[R] and
|
|
space.
|
|
- May not start with \f[C]-\f[R] or space.
|
|
.SS Quoting and the shell
|
|
.PP
|
|
When you are typing commands to your computer you are using something
|
|
called the command line shell.
|
|
This interprets various characters in an OS specific way.
|
|
.PP
|
|
Here are some gotchas which may help users unfamiliar with the shell
|
|
rules
|
|
.SS Linux / OSX
|
|
.PP
|
|
If your names have spaces or shell metacharacters (e.g.
|
|
\f[C]*\f[R], \f[C]?\f[R], \f[C]$\f[R], \f[C]\[aq]\f[R], \f[C]\[dq]\f[R],
|
|
etc.) then you must quote them.
|
|
Use single quotes \f[C]\[aq]\f[R] by default.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy \[aq]Important files?\[aq] remote:backup
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you want to send a \f[C]\[aq]\f[R] you will need to use
|
|
\f[C]\[dq]\f[R], e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy \[dq]O\[aq]Reilly Reviews\[dq] remote:backup
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The rules for quoting metacharacters are complicated and if you want the
|
|
full details you\[aq]ll have to consult the manual page for your shell.
|
|
.SS Windows
|
|
.PP
|
|
If your names have spaces in you need to put them in \f[C]\[dq]\f[R],
|
|
e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy \[dq]E:\[rs]folder name\[rs]folder name\[rs]folder name\[dq] remote:backup
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you are using the root directory on its own then don\[aq]t quote it
|
|
(see #464 (https://github.com/rclone/rclone/issues/464) for why), e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy E:\[rs] remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Copying files or directories with \f[C]:\f[R] in the names
|
|
.PP
|
|
rclone uses \f[C]:\f[R] to mark a remote name.
|
|
This is, however, a valid filename component in non-Windows OSes.
|
|
The remote name parser will only search for a \f[C]:\f[R] up to the
|
|
first \f[C]/\f[R] so if you need to act on a file or directory like this
|
|
then use the full path starting with a \f[C]/\f[R], or use \f[C]./\f[R]
|
|
as a current directory prefix.
|
|
.PP
|
|
So to sync a directory called \f[C]sync:me\f[R] to a remote called
|
|
\f[C]remote:\f[R] use
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i ./sync:me remote:path
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
or
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /full/path/to/sync:me remote:path
|
|
\f[R]
|
|
.fi
|
|
.SS Server Side Copy
|
|
.PP
|
|
Most remotes (but not all - see the
|
|
overview (https://rclone.org/overview/#optional-features)) support
|
|
server-side copy.
|
|
.PP
|
|
This means if you want to copy one folder to another then rclone
|
|
won\[aq]t download all the files and re-upload them; it will instruct
|
|
the server to copy them in place.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy s3:oldbucket s3:newbucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Will copy the contents of \f[C]oldbucket\f[R] to \f[C]newbucket\f[R]
|
|
without downloading and re-uploading.
|
|
.PP
|
|
Remotes which don\[aq]t support server-side copy \f[B]will\f[R] download
|
|
and re-upload in this case.
|
|
.PP
|
|
Server side copies are used with \f[C]sync\f[R] and \f[C]copy\f[R] and
|
|
will be identified in the log when using the \f[C]-v\f[R] flag.
|
|
The \f[C]move\f[R] command may also use them if remote doesn\[aq]t
|
|
support server-side move directly.
|
|
This is done by issuing a server-side copy then a delete which is much
|
|
quicker than a download and re-upload.
|
|
.PP
|
|
Server side copies will only be attempted if the remote names are the
|
|
same.
|
|
.PP
|
|
This can be used when scripting to make aged backups efficiently, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i remote:current-backup remote:previous-backup
|
|
rclone sync -i /path/to/files remote:current-backup
|
|
\f[R]
|
|
.fi
|
|
.SS Options
|
|
.PP
|
|
Rclone has a number of options to control its behaviour.
|
|
.PP
|
|
Options that take parameters can have the values passed in two ways,
|
|
\f[C]--option=value\f[R] or \f[C]--option value\f[R].
|
|
However boolean (true/false) options behave slightly differently to the
|
|
other options in that \f[C]--boolean\f[R] sets the option to
|
|
\f[C]true\f[R] and the absence of the flag sets it to \f[C]false\f[R].
|
|
It is also possible to specify \f[C]--boolean=false\f[R] or
|
|
\f[C]--boolean=true\f[R].
|
|
Note that \f[C]--boolean false\f[R] is not valid - this is parsed as
|
|
\f[C]--boolean\f[R] and the \f[C]false\f[R] is parsed as an extra
|
|
command line argument for rclone.
|
|
.PP
|
|
Options which use TIME use the go time parser.
|
|
A duration string is a possibly signed sequence of decimal numbers, each
|
|
with optional fraction and a unit suffix, such as \[dq]300ms\[dq],
|
|
\[dq]-1.5h\[dq] or \[dq]2h45m\[dq].
|
|
Valid time units are \[dq]ns\[dq], \[dq]us\[dq] (or \[dq]\[mc]s\[dq]),
|
|
\[dq]ms\[dq], \[dq]s\[dq], \[dq]m\[dq], \[dq]h\[dq].
|
|
.PP
|
|
Options which use SIZE use KiB (multiples of 1024 bytes) by default.
|
|
However, a suffix of \f[C]B\f[R] for Byte, \f[C]K\f[R] for KiB,
|
|
\f[C]M\f[R] for MiB, \f[C]G\f[R] for GiB, \f[C]T\f[R] for TiB and
|
|
\f[C]P\f[R] for PiB may be used.
|
|
These are the binary units, e.g.
|
|
1, 2**10, 2**20, 2**30 respectively.
|
|
.SS --backup-dir=DIR
|
|
.PP
|
|
When using \f[C]sync\f[R], \f[C]copy\f[R] or \f[C]move\f[R] any files
|
|
which would have been overwritten or deleted are moved in their original
|
|
hierarchy into this directory.
|
|
.PP
|
|
If \f[C]--suffix\f[R] is set, then the moved files will have the suffix
|
|
added to them.
|
|
If there is a file with the same path (after the suffix has been added)
|
|
in DIR, then it will be overwritten.
|
|
.PP
|
|
The remote in use must support server-side move or copy and you must use
|
|
the same remote as the destination of the sync.
|
|
The backup directory must not overlap the destination directory.
|
|
.PP
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /path/to/local remote:current --backup-dir remote:old
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
will sync \f[C]/path/to/local\f[R] to \f[C]remote:current\f[R], but for
|
|
any files which would have been updated or deleted will be stored in
|
|
\f[C]remote:old\f[R].
|
|
.PP
|
|
If running rclone from a script you might want to use today\[aq]s date
|
|
as the directory name passed to \f[C]--backup-dir\f[R] to store the old
|
|
files, or you might want to pass \f[C]--suffix\f[R] with today\[aq]s
|
|
date.
|
|
.PP
|
|
See \f[C]--compare-dest\f[R] and \f[C]--copy-dest\f[R].
|
|
.SS --bind string
|
|
.PP
|
|
Local address to bind to for outgoing connections.
|
|
This can be an IPv4 address (1.2.3.4), an IPv6 address (1234::789A) or
|
|
host name.
|
|
If the host name doesn\[aq]t resolve or resolves to more than one IP
|
|
address it will give an error.
|
|
.SS --bwlimit=BANDWIDTH_SPEC
|
|
.PP
|
|
This option controls the bandwidth limit.
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--bwlimit 10M
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
would mean limit the upload and download bandwidth to 10 MiB/s.
|
|
\f[B]NB\f[R] this is \f[B]bytes\f[R] per second not \f[B]bits\f[R] per
|
|
second.
|
|
To use a single limit, specify the desired bandwidth in KiB/s, or use a
|
|
suffix B|K|M|G|T|P.
|
|
The default is \f[C]0\f[R] which means to not limit bandwidth.
|
|
.PP
|
|
The upload and download bandwidth can be specified seperately, as
|
|
\f[C]--bwlimit UP:DOWN\f[R], so
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--bwlimit 10M:100k
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
would mean limit the upload bandwidth to 10 MiB/s and the download
|
|
bandwidth to 100 KiB/s.
|
|
Either limit can be \[dq]off\[dq] meaning no limit, so to just limit the
|
|
upload bandwidth you would use
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--bwlimit 10M:off
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
this would limit the upload bandwidth to 10 MiB/s but the download
|
|
bandwidth would be unlimited.
|
|
.PP
|
|
When specified as above the bandwidth limits last for the duration of
|
|
run of the rclone binary.
|
|
.PP
|
|
It is also possible to specify a \[dq]timetable\[dq] of limits, which
|
|
will cause certain limits to be applied at certain times.
|
|
To specify a timetable, format your entries as
|
|
\f[C]WEEKDAY-HH:MM,BANDWIDTH WEEKDAY-HH:MM,BANDWIDTH...\f[R] where:
|
|
\f[C]WEEKDAY\f[R] is optional element.
|
|
.IP \[bu] 2
|
|
\f[C]BANDWIDTH\f[R] can be a single number, e.g.\f[C]100k\f[R] or a pair
|
|
of numbers for upload:download, e.g.\f[C]10M:1M\f[R].
|
|
.IP \[bu] 2
|
|
\f[C]WEEKDAY\f[R] can be written as the whole word or only using the
|
|
first 3 characters.
|
|
It is optional.
|
|
.IP \[bu] 2
|
|
\f[C]HH:MM\f[R] is an hour from 00:00 to 23:59.
|
|
.PP
|
|
An example of a typical timetable to avoid link saturation during
|
|
daytime working hours could be:
|
|
.PP
|
|
\f[C]--bwlimit \[dq]08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off\[dq]\f[R]
|
|
.PP
|
|
In this example, the transfer bandwidth will be set to 512 KiB/s at 8am
|
|
every day.
|
|
At noon, it will rise to 10 MiB/s, and drop back to 512 KiB/sec at 1pm.
|
|
At 6pm, the bandwidth limit will be set to 30 MiB/s, and at 11pm it will
|
|
be completely disabled (full speed).
|
|
Anything between 11pm and 8am will remain unlimited.
|
|
.PP
|
|
An example of timetable with \f[C]WEEKDAY\f[R] could be:
|
|
.PP
|
|
\f[C]--bwlimit \[dq]Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off\[dq]\f[R]
|
|
.PP
|
|
It means that, the transfer bandwidth will be set to 512 KiB/s on
|
|
Monday.
|
|
It will rise to 10 MiB/s before the end of Friday.
|
|
At 10:00 on Saturday it will be set to 1 MiB/s.
|
|
From 20:00 on Sunday it will be unlimited.
|
|
.PP
|
|
Timeslots without \f[C]WEEKDAY\f[R] are extended to the whole week.
|
|
So this example:
|
|
.PP
|
|
\f[C]--bwlimit \[dq]Mon-00:00,512 12:00,1M Sun-20:00,off\[dq]\f[R]
|
|
.PP
|
|
Is equivalent to this:
|
|
.PP
|
|
\f[C]--bwlimit \[dq]Mon-00:00,512Mon-12:00,1M Tue-12:00,1M Wed-12:00,1M Thu-12:00,1M Fri-12:00,1M Sat-12:00,1M Sun-12:00,1M Sun-20:00,off\[dq]\f[R]
|
|
.PP
|
|
Bandwidth limit apply to the data transfer for all backends.
|
|
For most backends the directory listing bandwidth is also included
|
|
(exceptions being the non HTTP backends, \f[C]ftp\f[R], \f[C]sftp\f[R]
|
|
and \f[C]storj\f[R]).
|
|
.PP
|
|
Note that the units are \f[B]Byte/s\f[R], not \f[B]bit/s\f[R].
|
|
Typically connections are measured in bit/s - to convert divide by 8.
|
|
For example, let\[aq]s say you have a 10 Mbit/s connection and you wish
|
|
rclone to use half of it - 5 Mbit/s.
|
|
This is 5/8 = 0.625 MiB/s so you would use a \f[C]--bwlimit 0.625M\f[R]
|
|
parameter for rclone.
|
|
.PP
|
|
On Unix systems (Linux, macOS, \&...) the bandwidth limiter can be
|
|
toggled by sending a \f[C]SIGUSR2\f[R] signal to rclone.
|
|
This allows to remove the limitations of a long running rclone transfer
|
|
and to restore it back to the value specified with \f[C]--bwlimit\f[R]
|
|
quickly when needed.
|
|
Assuming there is only one rclone instance running, you can toggle the
|
|
limiter like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill -SIGUSR2 $(pidof rclone)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control then you can use change
|
|
the bwlimit dynamically:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc core/bwlimit rate=1M
|
|
\f[R]
|
|
.fi
|
|
.SS --bwlimit-file=BANDWIDTH_SPEC
|
|
.PP
|
|
This option controls per file bandwidth limit.
|
|
For the options see the \f[C]--bwlimit\f[R] flag.
|
|
.PP
|
|
For example use this to allow no transfers to be faster than 1 MiB/s
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--bwlimit-file 1M
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This can be used in conjunction with \f[C]--bwlimit\f[R].
|
|
.PP
|
|
Note that if a schedule is provided the file will use the schedule in
|
|
effect at the start of the transfer.
|
|
.SS --buffer-size=SIZE
|
|
.PP
|
|
Use this sized buffer to speed up file transfers.
|
|
Each \f[C]--transfer\f[R] will use this much memory for buffering.
|
|
.PP
|
|
When using \f[C]mount\f[R] or \f[C]cmount\f[R] each open file descriptor
|
|
will use this much memory for buffering.
|
|
See the mount (https://rclone.org/commands/rclone_mount/#file-buffering)
|
|
documentation for more details.
|
|
.PP
|
|
Set to \f[C]0\f[R] to disable the buffering for the minimum memory
|
|
usage.
|
|
.PP
|
|
Note that the memory allocation of the buffers is influenced by the
|
|
--use-mmap flag.
|
|
.SS --cache-dir=DIR
|
|
.PP
|
|
Specify the directory rclone will use for caching, to override the
|
|
default.
|
|
.PP
|
|
Default value is depending on operating system: - Windows
|
|
\f[C]%LocalAppData%\[rs]rclone\f[R], if \f[C]LocalAppData\f[R] is
|
|
defined.
|
|
- macOS \f[C]$HOME/Library/Caches/rclone\f[R] if \f[C]HOME\f[R] is
|
|
defined.
|
|
- Unix \f[C]$XDG_CACHE_HOME/rclone\f[R] if \f[C]XDG_CACHE_HOME\f[R] is
|
|
defined, else \f[C]$HOME/.cache/rclone\f[R] if \f[C]HOME\f[R] is
|
|
defined.
|
|
- Fallback (on all OS) to \f[C]$TMPDIR/rclone\f[R], where
|
|
\f[C]TMPDIR\f[R] is the value from --temp-dir.
|
|
.PP
|
|
You can use the config
|
|
paths (https://rclone.org/commands/rclone_config_paths/) command to see
|
|
the current value.
|
|
.PP
|
|
Cache directory is heavily used by the VFS File
|
|
Caching (https://rclone.org/commands/rclone_mount/#vfs-file-caching)
|
|
mount feature, but also by
|
|
serve (https://rclone.org/commands/rclone_serve/), GUI and other parts
|
|
of rclone.
|
|
.SS --check-first
|
|
.PP
|
|
If this flag is set then in a \f[C]sync\f[R], \f[C]copy\f[R] or
|
|
\f[C]move\f[R], rclone will do all the checks to see whether files need
|
|
to be transferred before doing any of the transfers.
|
|
Normally rclone would start running transfers as soon as possible.
|
|
.PP
|
|
This flag can be useful on IO limited systems where transfers interfere
|
|
with checking.
|
|
.PP
|
|
It can also be useful to ensure perfect ordering when using
|
|
\f[C]--order-by\f[R].
|
|
.PP
|
|
Using this flag can use more memory as it effectively sets
|
|
\f[C]--max-backlog\f[R] to infinite.
|
|
This means that all the info on the objects to transfer is held in
|
|
memory before the transfers start.
|
|
.SS --checkers=N
|
|
.PP
|
|
The number of checkers to run in parallel.
|
|
Checkers do the equality checking of files during a sync.
|
|
For some storage systems (e.g.
|
|
S3, Swift, Dropbox) this can take a significant amount of time so they
|
|
are run in parallel.
|
|
.PP
|
|
The default is to run 8 checkers in parallel.
|
|
.SS -c, --checksum
|
|
.PP
|
|
Normally rclone will look at modification time and size of files to see
|
|
if they are equal.
|
|
If you set this flag then rclone will check the file hash and size to
|
|
determine if files are equal.
|
|
.PP
|
|
This is useful when the remote doesn\[aq]t support setting modified time
|
|
and a more accurate sync is desired than just checking the file size.
|
|
.PP
|
|
This is very useful when transferring between remotes which store the
|
|
same hash type on the object, e.g.
|
|
Drive and Swift.
|
|
For details of which remotes support which hash type see the table in
|
|
the overview section (https://rclone.org/overview/).
|
|
.PP
|
|
Eg \f[C]rclone --checksum sync s3:/bucket swift:/bucket\f[R] would run
|
|
much quicker than without the \f[C]--checksum\f[R] flag.
|
|
.PP
|
|
When using this flag, rclone won\[aq]t update mtimes of remote files if
|
|
they are incorrect as it would normally.
|
|
.SS --compare-dest=DIR
|
|
.PP
|
|
When using \f[C]sync\f[R], \f[C]copy\f[R] or \f[C]move\f[R] DIR is
|
|
checked in addition to the destination for files.
|
|
If a file identical to the source is found that file is NOT copied from
|
|
source.
|
|
This is useful to copy just files that have changed since the last
|
|
backup.
|
|
.PP
|
|
You must use the same remote as the destination of the sync.
|
|
The compare directory must not overlap the destination directory.
|
|
.PP
|
|
See \f[C]--copy-dest\f[R] and \f[C]--backup-dir\f[R].
|
|
.SS --config=CONFIG_FILE
|
|
.PP
|
|
Specify the location of the rclone configuration file, to override the
|
|
default.
|
|
E.g.
|
|
\f[C]rclone config --config=\[dq]rclone.conf\[dq]\f[R].
|
|
.PP
|
|
The exact default is a bit complex to describe, due to changes
|
|
introduced through different versions of rclone while preserving
|
|
backwards compatibility, but in most cases it is as simple as:
|
|
.IP \[bu] 2
|
|
\f[C]%APPDATA%/rclone/rclone.conf\f[R] on Windows
|
|
.IP \[bu] 2
|
|
\f[C]\[ti]/.config/rclone/rclone.conf\f[R] on other
|
|
.PP
|
|
The complete logic is as follows: Rclone will look for an existing
|
|
configuration file in any of the following locations, in priority order:
|
|
.IP "1." 3
|
|
\f[C]rclone.conf\f[R] (in program directory, where rclone executable is)
|
|
.IP "2." 3
|
|
\f[C]%APPDATA%/rclone/rclone.conf\f[R] (only on Windows)
|
|
.IP "3." 3
|
|
\f[C]$XDG_CONFIG_HOME/rclone/rclone.conf\f[R] (on all systems, including
|
|
Windows)
|
|
.IP "4." 3
|
|
\f[C]\[ti]/.config/rclone/rclone.conf\f[R] (see below for explanation of
|
|
\[ti] symbol)
|
|
.IP "5." 3
|
|
\f[C]\[ti]/.rclone.conf\f[R]
|
|
.PP
|
|
If no existing configuration file is found, then a new one will be
|
|
created in the following location:
|
|
.IP \[bu] 2
|
|
On Windows: Location 2 listed above, except in the unlikely event that
|
|
\f[C]APPDATA\f[R] is not defined, then location 4 is used instead.
|
|
.IP \[bu] 2
|
|
On Unix: Location 3 if \f[C]XDG_CONFIG_HOME\f[R] is defined, else
|
|
location 4.
|
|
.IP \[bu] 2
|
|
Fallback to location 5 (on all OS), when the rclone directory cannot be
|
|
created, but if also a home directory was not found then path
|
|
\f[C].rclone.conf\f[R] relative to current working directory will be
|
|
used as a final resort.
|
|
.PP
|
|
The \f[C]\[ti]\f[R] symbol in paths above represent the home directory
|
|
of the current user on any OS, and the value is defined as following:
|
|
.IP \[bu] 2
|
|
On Windows: \f[C]%HOME%\f[R] if defined, else \f[C]%USERPROFILE%\f[R],
|
|
or else \f[C]%HOMEDRIVE%\[rs]%HOMEPATH%\f[R].
|
|
.IP \[bu] 2
|
|
On Unix: \f[C]$HOME\f[R] if defined, else by looking up current user in
|
|
OS-specific user database (e.g.
|
|
passwd file), or else use the result from shell command
|
|
\f[C]cd && pwd\f[R].
|
|
.PP
|
|
If you run \f[C]rclone config file\f[R] you will see where the default
|
|
location is for you.
|
|
.PP
|
|
The fact that an existing file \f[C]rclone.conf\f[R] in the same
|
|
directory as the rclone executable is always preferred, means that it is
|
|
easy to run in \[dq]portable\[dq] mode by downloading rclone executable
|
|
to a writable directory and then create an empty file
|
|
\f[C]rclone.conf\f[R] in the same directory.
|
|
.PP
|
|
If the location is set to empty string \f[C]\[dq]\[dq]\f[R] or path to a
|
|
file with name \f[C]notfound\f[R], or the os null device represented by
|
|
value \f[C]NUL\f[R] on Windows and \f[C]/dev/null\f[R] on Unix systems,
|
|
then rclone will keep the config file in memory only.
|
|
.PP
|
|
The file format is basic
|
|
INI (https://en.wikipedia.org/wiki/INI_file#Format): Sections of text,
|
|
led by a \f[C][section]\f[R] header and followed by \f[C]key=value\f[R]
|
|
entries on separate lines.
|
|
In rclone each remote is represented by its own section, where the
|
|
section name defines the name of the remote.
|
|
Options are specified as the \f[C]key=value\f[R] entries, where the key
|
|
is the option name without the \f[C]--backend-\f[R] prefix, in lowercase
|
|
and with \f[C]_\f[R] instead of \f[C]-\f[R].
|
|
E.g.
|
|
option \f[C]--mega-hard-delete\f[R] corresponds to key
|
|
\f[C]hard_delete\f[R].
|
|
Only backend options can be specified.
|
|
A special, and required, key \f[C]type\f[R] identifies the storage
|
|
system (https://rclone.org/overview/), where the value is the internal
|
|
lowercase name as returned by command \f[C]rclone help backends\f[R].
|
|
Comments are indicated by \f[C];\f[R] or \f[C]#\f[R] at the beginning of
|
|
a line.
|
|
.PP
|
|
Example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[megaremote]
|
|
type = mega
|
|
user = you\[at]example.com
|
|
pass = PDPcQVVjVtzFY-GTdDFozqBhTdsPg3qH
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that passwords are in
|
|
obscured (https://rclone.org/commands/rclone_obscure/) form.
|
|
Also, many storage systems uses token-based authentication instead of
|
|
passwords, and this requires additional steps.
|
|
It is easier, and safer, to use the interactive command
|
|
\f[C]rclone config\f[R] instead of manually editing the configuration
|
|
file.
|
|
.PP
|
|
The configuration file will typically contain login information, and
|
|
should therefore have restricted permissions so that only the current
|
|
user can read it.
|
|
Rclone tries to ensure this when it writes the file.
|
|
You may also choose to encrypt the file.
|
|
.PP
|
|
When token-based authentication are used, the configuration file must be
|
|
writable, because rclone needs to update the tokens inside it.
|
|
.SS --contimeout=TIME
|
|
.PP
|
|
Set the connection timeout.
|
|
This should be in go time format which looks like \f[C]5s\f[R] for 5
|
|
seconds, \f[C]10m\f[R] for 10 minutes, or \f[C]3h30m\f[R].
|
|
.PP
|
|
The connection timeout is the amount of time rclone will wait for a
|
|
connection to go through to a remote object storage system.
|
|
It is \f[C]1m\f[R] by default.
|
|
.SS --copy-dest=DIR
|
|
.PP
|
|
When using \f[C]sync\f[R], \f[C]copy\f[R] or \f[C]move\f[R] DIR is
|
|
checked in addition to the destination for files.
|
|
If a file identical to the source is found that file is server-side
|
|
copied from DIR to the destination.
|
|
This is useful for incremental backup.
|
|
.PP
|
|
The remote in use must support server-side copy and you must use the
|
|
same remote as the destination of the sync.
|
|
The compare directory must not overlap the destination directory.
|
|
.PP
|
|
See \f[C]--compare-dest\f[R] and \f[C]--backup-dir\f[R].
|
|
.SS --dedupe-mode MODE
|
|
.PP
|
|
Mode to run dedupe command in.
|
|
One of \f[C]interactive\f[R], \f[C]skip\f[R], \f[C]first\f[R],
|
|
\f[C]newest\f[R], \f[C]oldest\f[R], \f[C]rename\f[R].
|
|
The default is \f[C]interactive\f[R].
|
|
See the dedupe command for more information as to what these options
|
|
mean.
|
|
.SS --disable FEATURE,FEATURE,...
|
|
.PP
|
|
This disables a comma separated list of optional features.
|
|
For example to disable server-side move and server-side copy use:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--disable move,copy
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The features can be put in any case.
|
|
.PP
|
|
To see a list of which features can be disabled use:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--disable help
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the overview features (https://rclone.org/overview/#features) and
|
|
optional features (https://rclone.org/overview/#optional-features) to
|
|
get an idea of which feature does what.
|
|
.PP
|
|
This flag can be useful for debugging and in exceptional circumstances
|
|
(e.g.
|
|
Google Drive limiting the total volume of Server Side Copies to 100
|
|
GiB/day).
|
|
.SS --disable-http2
|
|
.PP
|
|
This stops rclone from trying to use HTTP/2 if available.
|
|
This can sometimes speed up transfers due to a problem in the Go
|
|
standard library (https://github.com/golang/go/issues/37373).
|
|
.SS --dscp VALUE
|
|
.PP
|
|
Specify a DSCP value or name to use in connections.
|
|
This could help QoS system to identify traffic class.
|
|
BE, EF, DF, LE, CSx and AFxx are allowed.
|
|
.PP
|
|
See the description of differentiated
|
|
services (https://en.wikipedia.org/wiki/Differentiated_services) to get
|
|
an idea of this field.
|
|
Setting this to 1 (LE) to identify the flow to SCAVENGER class can avoid
|
|
occupying too much bandwidth in a network with DiffServ support (RFC
|
|
8622 (https://tools.ietf.org/html/rfc8622)).
|
|
.PP
|
|
For example, if you configured QoS on router to handle LE properly.
|
|
Running:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --dscp LE from:/from to:/to
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
would make the priority lower than usual internet flows.
|
|
.PP
|
|
This option has no effect on Windows (see
|
|
golang/go#42728 (https://github.com/golang/go/issues/42728)).
|
|
.SS -n, --dry-run
|
|
.PP
|
|
Do a trial run with no permanent changes.
|
|
Use this to see what rclone would do without actually doing it.
|
|
Useful when setting up the \f[C]sync\f[R] command which deletes files in
|
|
the destination.
|
|
.SS --expect-continue-timeout=TIME
|
|
.PP
|
|
This specifies the amount of time to wait for a server\[aq]s first
|
|
response headers after fully writing the request headers if the request
|
|
has an \[dq]Expect: 100-continue\[dq] header.
|
|
Not all backends support using this.
|
|
.PP
|
|
Zero means no timeout and causes the body to be sent immediately,
|
|
without waiting for the server to approve.
|
|
This time does not include the time to send the request header.
|
|
.PP
|
|
The default is \f[C]1s\f[R].
|
|
Set to \f[C]0\f[R] to disable.
|
|
.SS --error-on-no-transfer
|
|
.PP
|
|
By default, rclone will exit with return code 0 if there were no errors.
|
|
.PP
|
|
This option allows rclone to return exit code 9 if no files were
|
|
transferred between the source and destination.
|
|
This allows using rclone in scripts, and triggering follow-on actions if
|
|
data was copied, or skipping if not.
|
|
.PP
|
|
NB: Enabling this option turns a usually non-fatal error into a
|
|
potentially fatal one - please check and adjust your scripts
|
|
accordingly!
|
|
.SS --fs-cache-expire-duration=TIME
|
|
.PP
|
|
When using rclone via the API rclone caches created remotes for 5
|
|
minutes by default in the \[dq]fs cache\[dq].
|
|
This means that if you do repeated actions on the same remote then
|
|
rclone won\[aq]t have to build it again from scratch, which makes it
|
|
more efficient.
|
|
.PP
|
|
This flag sets the time that the remotes are cached for.
|
|
If you set it to \f[C]0\f[R] (or negative) then rclone won\[aq]t cache
|
|
the remotes at all.
|
|
.PP
|
|
Note that if you use some flags, eg \f[C]--backup-dir\f[R] and if this
|
|
is set to \f[C]0\f[R] rclone may build two remotes (one for the source
|
|
or destination and one for the \f[C]--backup-dir\f[R] where it may have
|
|
only built one before.
|
|
.SS --fs-cache-expire-interval=TIME
|
|
.PP
|
|
This controls how often rclone checks for cached remotes to expire.
|
|
See the \f[C]--fs-cache-expire-duration\f[R] documentation above for
|
|
more info.
|
|
The default is 60s, set to 0 to disable expiry.
|
|
.SS --header
|
|
.PP
|
|
Add an HTTP header for all transactions.
|
|
The flag can be repeated to add multiple headers.
|
|
.PP
|
|
If you want to add headers only for uploads use
|
|
\f[C]--header-upload\f[R] and if you want to add headers only for
|
|
downloads use \f[C]--header-download\f[R].
|
|
.PP
|
|
This flag is supported for all HTTP based backends even those not
|
|
supported by \f[C]--header-upload\f[R] and \f[C]--header-download\f[R]
|
|
so may be used as a workaround for those with care.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:test --header \[dq]X-Rclone: Foo\[dq] --header \[dq]X-LetMeIn: Yes\[dq]
|
|
\f[R]
|
|
.fi
|
|
.SS --header-download
|
|
.PP
|
|
Add an HTTP header for all download transactions.
|
|
The flag can be repeated to add multiple headers.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i s3:test/src \[ti]/dst --header-download \[dq]X-Amz-Meta-Test: Foo\[dq] --header-download \[dq]X-Amz-Meta-Test2: Bar\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the GitHub issue here (https://github.com/rclone/rclone/issues/59)
|
|
for currently supported backends.
|
|
.SS --header-upload
|
|
.PP
|
|
Add an HTTP header for all upload transactions.
|
|
The flag can be repeated to add multiple headers.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i \[ti]/src s3:test/dst --header-upload \[dq]Content-Disposition: attachment; filename=\[aq]cool.html\[aq]\[dq] --header-upload \[dq]X-Amz-Meta-Test: FooBar\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the GitHub issue here (https://github.com/rclone/rclone/issues/59)
|
|
for currently supported backends.
|
|
.SS --human-readable
|
|
.PP
|
|
Rclone commands output values for sizes (e.g.
|
|
number of bytes) and counts (e.g.
|
|
number of files) either as \f[I]raw\f[R] numbers, or in
|
|
\f[I]human-readable\f[R] format.
|
|
.PP
|
|
In human-readable format the values are scaled to larger units,
|
|
indicated with a suffix shown after the value, and rounded to three
|
|
decimals.
|
|
Rclone consistently uses binary units (powers of 2) for sizes and
|
|
decimal units (powers of 10) for counts.
|
|
The unit prefix for size is according to IEC standard notation, e.g.
|
|
\f[C]Ki\f[R] for kibi.
|
|
Used with byte unit, \f[C]1 KiB\f[R] means 1024 Byte.
|
|
In list type of output, only the unit prefix appended to the value (e.g.
|
|
\f[C]9.762Ki\f[R]), while in more textual output the full unit is shown
|
|
(e.g.
|
|
\f[C]9.762 KiB\f[R]).
|
|
For counts the SI standard notation is used, e.g.
|
|
prefix \f[C]k\f[R] for kilo.
|
|
Used with file counts, \f[C]1k\f[R] means 1000 files.
|
|
.PP
|
|
The various list commands output raw numbers by default.
|
|
Option \f[C]--human-readable\f[R] will make them output values in
|
|
human-readable format instead (with the short unit prefix).
|
|
.PP
|
|
The about command outputs human-readable by default, with a
|
|
command-specific option \f[C]--full\f[R] to output the raw numbers
|
|
instead.
|
|
.PP
|
|
Command size outputs both human-readable and raw numbers in the same
|
|
output.
|
|
.PP
|
|
The tree command also considers \f[C]--human-readable\f[R], but it will
|
|
not use the exact same notation as the other commands: It rounds to one
|
|
decimal, and uses single letter suffix, e.g.
|
|
\f[C]K\f[R] instead of \f[C]Ki\f[R].
|
|
The reason for this is that it relies on an external library.
|
|
.PP
|
|
The interactive command ncdu shows human-readable by default, and
|
|
responds to key \f[C]u\f[R] for toggling human-readable format.
|
|
.SS --ignore-case-sync
|
|
.PP
|
|
Using this option will cause rclone to ignore the case of the files when
|
|
synchronizing so files will not be copied/synced when the existing
|
|
filenames are the same, even if the casing is different.
|
|
.SS --ignore-checksum
|
|
.PP
|
|
Normally rclone will check that the checksums of transferred files
|
|
match, and give an error \[dq]corrupted on transfer\[dq] if they
|
|
don\[aq]t.
|
|
.PP
|
|
You can use this option to skip that check.
|
|
You should only use it if you have had the \[dq]corrupted on
|
|
transfer\[dq] error message and you are sure you might want to transfer
|
|
potentially corrupted data.
|
|
.SS --ignore-existing
|
|
.PP
|
|
Using this option will make rclone unconditionally skip all files that
|
|
exist on the destination, no matter the content of these files.
|
|
.PP
|
|
While this isn\[aq]t a generally recommended option, it can be useful in
|
|
cases where your files change due to encryption.
|
|
However, it cannot correct partial transfers in case a transfer was
|
|
interrupted.
|
|
.PP
|
|
When performing a \f[C]move\f[R]/\f[C]moveto\f[R] command, this flag
|
|
will leave skipped files in the source location unchanged when a file
|
|
with the same name exists on the destination.
|
|
.SS --ignore-size
|
|
.PP
|
|
Normally rclone will look at modification time and size of files to see
|
|
if they are equal.
|
|
If you set this flag then rclone will check only the modification time.
|
|
If \f[C]--checksum\f[R] is set then it only checks the checksum.
|
|
.PP
|
|
It will also cause rclone to skip verifying the sizes are the same after
|
|
transfer.
|
|
.PP
|
|
This can be useful for transferring files to and from OneDrive which
|
|
occasionally misreports the size of image files (see
|
|
#399 (https://github.com/rclone/rclone/issues/399) for more info).
|
|
.SS -I, --ignore-times
|
|
.PP
|
|
Using this option will cause rclone to unconditionally upload all files
|
|
regardless of the state of files on the destination.
|
|
.PP
|
|
Normally rclone would skip any files that have the same modification
|
|
time and are the same size (or have the same checksum if using
|
|
\f[C]--checksum\f[R]).
|
|
.SS --immutable
|
|
.PP
|
|
Treat source and destination files as immutable and disallow
|
|
modification.
|
|
.PP
|
|
With this option set, files will be created and deleted as requested,
|
|
but existing files will never be updated.
|
|
If an existing file does not match between the source and destination,
|
|
rclone will give the error
|
|
\f[C]Source and destination exist but do not match: immutable file modified\f[R].
|
|
.PP
|
|
Note that only commands which transfer files (e.g.
|
|
\f[C]sync\f[R], \f[C]copy\f[R], \f[C]move\f[R]) are affected by this
|
|
behavior, and only modification is disallowed.
|
|
Files may still be deleted explicitly (e.g.
|
|
\f[C]delete\f[R], \f[C]purge\f[R]) or implicitly (e.g.
|
|
\f[C]sync\f[R], \f[C]move\f[R]).
|
|
Use \f[C]copy --immutable\f[R] if it is desired to avoid deletion as
|
|
well as modification.
|
|
.PP
|
|
This can be useful as an additional layer of protection for immutable or
|
|
append-only data sets (notably backup archives), where modification
|
|
implies corruption and should not be propagated.
|
|
.SS -i / --interactive
|
|
.PP
|
|
This flag can be used to tell rclone that you wish a manual confirmation
|
|
before destructive operations.
|
|
.PP
|
|
It is \f[B]recommended\f[R] that you use this flag while learning rclone
|
|
especially with \f[C]rclone sync\f[R].
|
|
.PP
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone delete -i /tmp/dir
|
|
rclone: delete \[dq]important-file.txt\[dq]?
|
|
y) Yes, this is OK (default)
|
|
n) No, skip this
|
|
s) Skip all delete operations with no more questions
|
|
!) Do all delete operations with no more questions
|
|
q) Exit rclone now.
|
|
y/n/s/!/q> n
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The options mean
|
|
.IP \[bu] 2
|
|
\f[C]y\f[R]: \f[B]Yes\f[R], this operation should go ahead.
|
|
You can also press Return for this to happen.
|
|
You\[aq]ll be asked every time unless you choose \f[C]s\f[R] or
|
|
\f[C]!\f[R].
|
|
.IP \[bu] 2
|
|
\f[C]n\f[R]: \f[B]No\f[R], do not do this operation.
|
|
You\[aq]ll be asked every time unless you choose \f[C]s\f[R] or
|
|
\f[C]!\f[R].
|
|
.IP \[bu] 2
|
|
\f[C]s\f[R]: \f[B]Skip\f[R] all the following operations of this type
|
|
with no more questions.
|
|
This takes effect until rclone exits.
|
|
If there are any different kind of operations you\[aq]ll be prompted for
|
|
them.
|
|
.IP \[bu] 2
|
|
\f[C]!\f[R]: \f[B]Do all\f[R] the following operations with no more
|
|
questions.
|
|
Useful if you\[aq]ve decided that you don\[aq]t mind rclone doing that
|
|
kind of operation.
|
|
This takes effect until rclone exits .
|
|
If there are any different kind of operations you\[aq]ll be prompted for
|
|
them.
|
|
.IP \[bu] 2
|
|
\f[C]q\f[R]: \f[B]Quit\f[R] rclone now, just in case!
|
|
.SS --leave-root
|
|
.PP
|
|
During rmdirs it will not remove root directory, even if it\[aq]s empty.
|
|
.SS --log-file=FILE
|
|
.PP
|
|
Log all of rclone\[aq]s output to FILE.
|
|
This is not active by default.
|
|
This can be useful for tracking down problems with syncs in combination
|
|
with the \f[C]-v\f[R] flag.
|
|
See the Logging section for more info.
|
|
.PP
|
|
If FILE exists then rclone will append to it.
|
|
.PP
|
|
Note that if you are using the \f[C]logrotate\f[R] program to manage
|
|
rclone\[aq]s logs, then you should use the \f[C]copytruncate\f[R] option
|
|
as rclone doesn\[aq]t have a signal to rotate logs.
|
|
.SS --log-format LIST
|
|
.PP
|
|
Comma separated list of log format options.
|
|
Accepted options are \f[C]date\f[R], \f[C]time\f[R],
|
|
\f[C]microseconds\f[R], \f[C]pid\f[R], \f[C]longfile\f[R],
|
|
\f[C]shortfile\f[R], \f[C]UTC\f[R].
|
|
Any other keywords will be silently ignored.
|
|
\f[C]pid\f[R] will tag log messages with process identifier which useful
|
|
with \f[C]rclone mount --daemon\f[R].
|
|
Other accepted options are explained in the go
|
|
documentation (https://pkg.go.dev/log#pkg-constants).
|
|
The default log format is \[dq]\f[C]date\f[R],\f[C]time\f[R]\[dq].
|
|
.SS --log-level LEVEL
|
|
.PP
|
|
This sets the log level for rclone.
|
|
The default log level is \f[C]NOTICE\f[R].
|
|
.PP
|
|
\f[C]DEBUG\f[R] is equivalent to \f[C]-vv\f[R].
|
|
It outputs lots of debug info - useful for bug reports and really
|
|
finding out what rclone is doing.
|
|
.PP
|
|
\f[C]INFO\f[R] is equivalent to \f[C]-v\f[R].
|
|
It outputs information about each transfer and prints stats once a
|
|
minute by default.
|
|
.PP
|
|
\f[C]NOTICE\f[R] is the default log level if no logging flags are
|
|
supplied.
|
|
It outputs very little when things are working normally.
|
|
It outputs warnings and significant events.
|
|
.PP
|
|
\f[C]ERROR\f[R] is equivalent to \f[C]-q\f[R].
|
|
It only outputs error messages.
|
|
.SS --use-json-log
|
|
.PP
|
|
This switches the log format to JSON for rclone.
|
|
The fields of json log are level, msg, source, time.
|
|
.SS --low-level-retries NUMBER
|
|
.PP
|
|
This controls the number of low level retries rclone does.
|
|
.PP
|
|
A low level retry is used to retry a failing operation - typically one
|
|
HTTP request.
|
|
This might be uploading a chunk of a big file for example.
|
|
You will see low level retries in the log with the \f[C]-v\f[R] flag.
|
|
.PP
|
|
This shouldn\[aq]t need to be changed from the default in normal
|
|
operations.
|
|
However, if you get a lot of low level retries you may wish to reduce
|
|
the value so rclone moves on to a high level retry (see the
|
|
\f[C]--retries\f[R] flag) quicker.
|
|
.PP
|
|
Disable low level retries with \f[C]--low-level-retries 1\f[R].
|
|
.SS --max-backlog=N
|
|
.PP
|
|
This is the maximum allowable backlog of files in a sync/copy/move
|
|
queued for being checked or transferred.
|
|
.PP
|
|
This can be set arbitrarily large.
|
|
It will only use memory when the queue is in use.
|
|
Note that it will use in the order of N KiB of memory when the backlog
|
|
is in use.
|
|
.PP
|
|
Setting this large allows rclone to calculate how many files are pending
|
|
more accurately, give a more accurate estimated finish time and make
|
|
\f[C]--order-by\f[R] work more accurately.
|
|
.PP
|
|
Setting this small will make rclone more synchronous to the listings of
|
|
the remote which may be desirable.
|
|
.PP
|
|
Setting this to a negative number will make the backlog as large as
|
|
possible.
|
|
.SS --max-delete=N
|
|
.PP
|
|
This tells rclone not to delete more than N files.
|
|
If that limit is exceeded then a fatal error will be generated and
|
|
rclone will stop the operation in progress.
|
|
.SS --max-depth=N
|
|
.PP
|
|
This modifies the recursion depth for all the commands except purge.
|
|
.PP
|
|
So if you do \f[C]rclone --max-depth 1 ls remote:path\f[R] you will see
|
|
only the files in the top level directory.
|
|
Using \f[C]--max-depth 2\f[R] means you will see all the files in first
|
|
two directory levels and so on.
|
|
.PP
|
|
For historical reasons the \f[C]lsd\f[R] command defaults to using a
|
|
\f[C]--max-depth\f[R] of 1 - you can override this with the command line
|
|
flag.
|
|
.PP
|
|
You can use this command to disable recursion (with
|
|
\f[C]--max-depth 1\f[R]).
|
|
.PP
|
|
Note that if you use this with \f[C]sync\f[R] and
|
|
\f[C]--delete-excluded\f[R] the files not recursed through are
|
|
considered excluded and will be deleted on the destination.
|
|
Test first with \f[C]--dry-run\f[R] if you are not sure what will
|
|
happen.
|
|
.SS --max-duration=TIME
|
|
.PP
|
|
Rclone will stop scheduling new transfers when it has run for the
|
|
duration specified.
|
|
.PP
|
|
Defaults to off.
|
|
.PP
|
|
When the limit is reached any existing transfers will complete.
|
|
.PP
|
|
Rclone won\[aq]t exit with an error if the transfer limit is reached.
|
|
.SS --max-transfer=SIZE
|
|
.PP
|
|
Rclone will stop transferring when it has reached the size specified.
|
|
Defaults to off.
|
|
.PP
|
|
When the limit is reached all transfers will stop immediately.
|
|
.PP
|
|
Rclone will exit with exit code 8 if the transfer limit is reached.
|
|
.SS --cutoff-mode=hard|soft|cautious
|
|
.PP
|
|
This modifies the behavior of \f[C]--max-transfer\f[R] Defaults to
|
|
\f[C]--cutoff-mode=hard\f[R].
|
|
.PP
|
|
Specifying \f[C]--cutoff-mode=hard\f[R] will stop transferring
|
|
immediately when Rclone reaches the limit.
|
|
.PP
|
|
Specifying \f[C]--cutoff-mode=soft\f[R] will stop starting new transfers
|
|
when Rclone reaches the limit.
|
|
.PP
|
|
Specifying \f[C]--cutoff-mode=cautious\f[R] will try to prevent Rclone
|
|
from reaching the limit.
|
|
.SS --modify-window=TIME
|
|
.PP
|
|
When checking whether a file has been modified, this is the maximum
|
|
allowed time difference that a file can have and still be considered
|
|
equivalent.
|
|
.PP
|
|
The default is \f[C]1ns\f[R] unless this is overridden by a remote.
|
|
For example OS X only stores modification times to the nearest second so
|
|
if you are reading and writing to an OS X filing system this will be
|
|
\f[C]1s\f[R] by default.
|
|
.PP
|
|
This command line flag allows you to override that computed default.
|
|
.SS --multi-thread-cutoff=SIZE
|
|
.PP
|
|
When downloading files to the local backend above this size, rclone will
|
|
use multiple threads to download the file (default 250M).
|
|
.PP
|
|
Rclone preallocates the file (using
|
|
\f[C]fallocate(FALLOC_FL_KEEP_SIZE)\f[R] on unix or
|
|
\f[C]NTSetInformationFile\f[R] on Windows both of which takes no time)
|
|
then each thread writes directly into the file at the correct place.
|
|
This means that rclone won\[aq]t create fragmented or sparse files and
|
|
there won\[aq]t be any assembly time at the end of the transfer.
|
|
.PP
|
|
The number of threads used to download is controlled by
|
|
\f[C]--multi-thread-streams\f[R].
|
|
.PP
|
|
Use \f[C]-vv\f[R] if you wish to see info about the threads.
|
|
.PP
|
|
This will work with the \f[C]sync\f[R]/\f[C]copy\f[R]/\f[C]move\f[R]
|
|
commands and friends \f[C]copyto\f[R]/\f[C]moveto\f[R].
|
|
Multi thread downloads will be used with \f[C]rclone mount\f[R] and
|
|
\f[C]rclone serve\f[R] if \f[C]--vfs-cache-mode\f[R] is set to
|
|
\f[C]writes\f[R] or above.
|
|
.PP
|
|
\f[B]NB\f[R] that this \f[B]only\f[R] works for a local destination but
|
|
will work with any source.
|
|
.PP
|
|
\f[B]NB\f[R] that multi thread copies are disabled for local to local
|
|
copies as they are faster without unless
|
|
\f[C]--multi-thread-streams\f[R] is set explicitly.
|
|
.PP
|
|
\f[B]NB\f[R] on Windows using multi-thread downloads will cause the
|
|
resulting files to be
|
|
sparse (https://en.wikipedia.org/wiki/Sparse_file).
|
|
Use \f[C]--local-no-sparse\f[R] to disable sparse files (which may cause
|
|
long delays at the start of downloads) or disable multi-thread downloads
|
|
with \f[C]--multi-thread-streams 0\f[R]
|
|
.SS --multi-thread-streams=N
|
|
.PP
|
|
When using multi thread downloads (see above
|
|
\f[C]--multi-thread-cutoff\f[R]) this sets the maximum number of streams
|
|
to use.
|
|
Set to \f[C]0\f[R] to disable multi thread downloads (Default 4).
|
|
.PP
|
|
Exactly how many streams rclone uses for the download depends on the
|
|
size of the file.
|
|
To calculate the number of download streams Rclone divides the size of
|
|
the file by the \f[C]--multi-thread-cutoff\f[R] and rounds up, up to the
|
|
maximum set with \f[C]--multi-thread-streams\f[R].
|
|
.PP
|
|
So if \f[C]--multi-thread-cutoff 250M\f[R] and
|
|
\f[C]--multi-thread-streams 4\f[R] are in effect (the defaults):
|
|
.IP \[bu] 2
|
|
0..250 MiB files will be downloaded with 1 stream
|
|
.IP \[bu] 2
|
|
250..500 MiB files will be downloaded with 2 streams
|
|
.IP \[bu] 2
|
|
500..750 MiB files will be downloaded with 3 streams
|
|
.IP \[bu] 2
|
|
750+ MiB files will be downloaded with 4 streams
|
|
.SS --no-check-dest
|
|
.PP
|
|
The \f[C]--no-check-dest\f[R] can be used with \f[C]move\f[R] or
|
|
\f[C]copy\f[R] and it causes rclone not to check the destination at all
|
|
when copying files.
|
|
.PP
|
|
This means that:
|
|
.IP \[bu] 2
|
|
the destination is not listed minimising the API calls
|
|
.IP \[bu] 2
|
|
files are always transferred
|
|
.IP \[bu] 2
|
|
this can cause duplicates on remotes which allow it (e.g.
|
|
Google Drive)
|
|
.IP \[bu] 2
|
|
\f[C]--retries 1\f[R] is recommended otherwise you\[aq]ll transfer
|
|
everything again on a retry
|
|
.PP
|
|
This flag is useful to minimise the transactions if you know that none
|
|
of the files are on the destination.
|
|
.PP
|
|
This is a specialized flag which should be ignored by most users!
|
|
.SS --no-gzip-encoding
|
|
.PP
|
|
Don\[aq]t set \f[C]Accept-Encoding: gzip\f[R].
|
|
This means that rclone won\[aq]t ask the server for compressed files
|
|
automatically.
|
|
Useful if you\[aq]ve set the server to return files with
|
|
\f[C]Content-Encoding: gzip\f[R] but you uploaded compressed files.
|
|
.PP
|
|
There is no need to set this in normal operation, and doing so will
|
|
decrease the network transfer efficiency of rclone.
|
|
.SS --no-traverse
|
|
.PP
|
|
The \f[C]--no-traverse\f[R] flag controls whether the destination file
|
|
system is traversed when using the \f[C]copy\f[R] or \f[C]move\f[R]
|
|
commands.
|
|
\f[C]--no-traverse\f[R] is not compatible with \f[C]sync\f[R] and will
|
|
be ignored if you supply it with \f[C]sync\f[R].
|
|
.PP
|
|
If you are only copying a small number of files (or are filtering most
|
|
of the files) and/or have a large number of files on the destination
|
|
then \f[C]--no-traverse\f[R] will stop rclone listing the destination
|
|
and save time.
|
|
.PP
|
|
However, if you are copying a large number of files, especially if you
|
|
are doing a copy where lots of the files under consideration haven\[aq]t
|
|
changed and won\[aq]t need copying then you shouldn\[aq]t use
|
|
\f[C]--no-traverse\f[R].
|
|
.PP
|
|
See rclone copy (https://rclone.org/commands/rclone_copy/) for an
|
|
example of how to use it.
|
|
.SS --no-unicode-normalization
|
|
.PP
|
|
Don\[aq]t normalize unicode characters in filenames during the sync
|
|
routine.
|
|
.PP
|
|
Sometimes, an operating system will store filenames containing unicode
|
|
parts in their decomposed form (particularly macOS).
|
|
Some cloud storage systems will then recompose the unicode, resulting in
|
|
duplicate files if the data is ever copied back to a local filesystem.
|
|
.PP
|
|
Using this flag will disable that functionality, treating each unicode
|
|
character as unique.
|
|
For example, by default e\[u0301] and \['e] will be normalized into the
|
|
same character.
|
|
With \f[C]--no-unicode-normalization\f[R] they will be treated as unique
|
|
characters.
|
|
.SS --no-update-modtime
|
|
.PP
|
|
When using this flag, rclone won\[aq]t update modification times of
|
|
remote files if they are incorrect as it would normally.
|
|
.PP
|
|
This can be used if the remote is being synced with another tool also
|
|
(e.g.
|
|
the Google Drive client).
|
|
.SS --order-by string
|
|
.PP
|
|
The \f[C]--order-by\f[R] flag controls the order in which files in the
|
|
backlog are processed in \f[C]rclone sync\f[R], \f[C]rclone copy\f[R]
|
|
and \f[C]rclone move\f[R].
|
|
.PP
|
|
The order by string is constructed like this.
|
|
The first part describes what aspect is being measured:
|
|
.IP \[bu] 2
|
|
\f[C]size\f[R] - order by the size of the files
|
|
.IP \[bu] 2
|
|
\f[C]name\f[R] - order by the full path of the files
|
|
.IP \[bu] 2
|
|
\f[C]modtime\f[R] - order by the modification date of the files
|
|
.PP
|
|
This can have a modifier appended with a comma:
|
|
.IP \[bu] 2
|
|
\f[C]ascending\f[R] or \f[C]asc\f[R] - order so that the smallest (or
|
|
oldest) is processed first
|
|
.IP \[bu] 2
|
|
\f[C]descending\f[R] or \f[C]desc\f[R] - order so that the largest (or
|
|
newest) is processed first
|
|
.IP \[bu] 2
|
|
\f[C]mixed\f[R] - order so that the smallest is processed first for some
|
|
threads and the largest for others
|
|
.PP
|
|
If the modifier is \f[C]mixed\f[R] then it can have an optional
|
|
percentage (which defaults to \f[C]50\f[R]), e.g.
|
|
\f[C]size,mixed,25\f[R] which means that 25% of the threads should be
|
|
taking the smallest items and 75% the largest.
|
|
The threads which take the smallest first will always take the smallest
|
|
first and likewise the largest first threads.
|
|
The \f[C]mixed\f[R] mode can be useful to minimise the transfer time
|
|
when you are transferring a mixture of large and small files - the large
|
|
files are guaranteed upload threads and bandwidth and the small files
|
|
will be processed continuously.
|
|
.PP
|
|
If no modifier is supplied then the order is \f[C]ascending\f[R].
|
|
.PP
|
|
For example
|
|
.IP \[bu] 2
|
|
\f[C]--order-by size,desc\f[R] - send the largest files first
|
|
.IP \[bu] 2
|
|
\f[C]--order-by modtime,ascending\f[R] - send the oldest files first
|
|
.IP \[bu] 2
|
|
\f[C]--order-by name\f[R] - send the files with alphabetically by path
|
|
first
|
|
.PP
|
|
If the \f[C]--order-by\f[R] flag is not supplied or it is supplied with
|
|
an empty string then the default ordering will be used which is as
|
|
scanned.
|
|
With \f[C]--checkers 1\f[R] this is mostly alphabetical, however with
|
|
the default \f[C]--checkers 8\f[R] it is somewhat random.
|
|
.SS Limitations
|
|
.PP
|
|
The \f[C]--order-by\f[R] flag does not do a separate pass over the data.
|
|
This means that it may transfer some files out of the order specified if
|
|
.IP \[bu] 2
|
|
there are no files in the backlog or the source has not been fully
|
|
scanned yet
|
|
.IP \[bu] 2
|
|
there are more than --max-backlog files in the backlog
|
|
.PP
|
|
Rclone will do its best to transfer the best file it has so in practice
|
|
this should not cause a problem.
|
|
Think of \f[C]--order-by\f[R] as being more of a best efforts flag
|
|
rather than a perfect ordering.
|
|
.PP
|
|
If you want perfect ordering then you will need to specify --check-first
|
|
which will find all the files which need transferring first before
|
|
transferring any.
|
|
.SS --password-command SpaceSepList
|
|
.PP
|
|
This flag supplies a program which should supply the config password
|
|
when run.
|
|
This is an alternative to rclone prompting for the password or setting
|
|
the \f[C]RCLONE_CONFIG_PASS\f[R] variable.
|
|
.PP
|
|
The argument to this should be a command with a space separated list of
|
|
arguments.
|
|
If one of the arguments has a space in then enclose it in
|
|
\f[C]\[dq]\f[R], if you want a literal \f[C]\[dq]\f[R] in an argument
|
|
then enclose the argument in \f[C]\[dq]\f[R] and double the
|
|
\f[C]\[dq]\f[R].
|
|
See CSV encoding (https://godoc.org/encoding/csv) for more info.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--password-command echo hello
|
|
--password-command echo \[dq]hello with space\[dq]
|
|
--password-command echo \[dq]hello with \[dq]\[dq]quotes\[dq]\[dq] and space\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the Configuration Encryption for more info.
|
|
.PP
|
|
See a Windows PowerShell example on the
|
|
Wiki (https://github.com/rclone/rclone/wiki/Windows-Powershell-use-rclone-password-command-for-Config-file-password).
|
|
.SS -P, --progress
|
|
.PP
|
|
This flag makes rclone update the stats in a static block in the
|
|
terminal providing a realtime overview of the transfer.
|
|
.PP
|
|
Any log messages will scroll above the static block.
|
|
Log messages will push the static block down to the bottom of the
|
|
terminal where it will stay.
|
|
.PP
|
|
Normally this is updated every 500mS but this period can be overridden
|
|
with the \f[C]--stats\f[R] flag.
|
|
.PP
|
|
This can be used with the \f[C]--stats-one-line\f[R] flag for a simpler
|
|
display.
|
|
.PP
|
|
Note: On Windows until this
|
|
bug (https://github.com/Azure/go-ansiterm/issues/26) is fixed all
|
|
non-ASCII characters will be replaced with \f[C].\f[R] when
|
|
\f[C]--progress\f[R] is in use.
|
|
.SS --progress-terminal-title
|
|
.PP
|
|
This flag, when used with \f[C]-P/--progress\f[R], will print the string
|
|
\f[C]ETA: %s\f[R] to the terminal title.
|
|
.SS -q, --quiet
|
|
.PP
|
|
This flag will limit rclone\[aq]s output to error messages only.
|
|
.SS --refresh-times
|
|
.PP
|
|
The \f[C]--refresh-times\f[R] flag can be used to update modification
|
|
times of existing files when they are out of sync on backends which
|
|
don\[aq]t support hashes.
|
|
.PP
|
|
This is useful if you uploaded files with the incorrect timestamps and
|
|
you now wish to correct them.
|
|
.PP
|
|
This flag is \f[B]only\f[R] useful for destinations which don\[aq]t
|
|
support hashes (e.g.
|
|
\f[C]crypt\f[R]).
|
|
.PP
|
|
This can be used any of the sync commands \f[C]sync\f[R], \f[C]copy\f[R]
|
|
or \f[C]move\f[R].
|
|
.PP
|
|
To use this flag you will need to be doing a modification time sync (so
|
|
not using \f[C]--size-only\f[R] or \f[C]--checksum\f[R]).
|
|
The flag will have no effect when using \f[C]--size-only\f[R] or
|
|
\f[C]--checksum\f[R].
|
|
.PP
|
|
If this flag is used when rclone comes to upload a file it will check to
|
|
see if there is an existing file on the destination.
|
|
If this file matches the source with size (and checksum if available)
|
|
but has a differing timestamp then instead of re-uploading it, rclone
|
|
will update the timestamp on the destination file.
|
|
If the checksum does not match rclone will upload the new file.
|
|
If the checksum is absent (e.g.
|
|
on a \f[C]crypt\f[R] backend) then rclone will update the timestamp.
|
|
.PP
|
|
Note that some remotes can\[aq]t set the modification time without
|
|
re-uploading the file so this flag is less useful on them.
|
|
.PP
|
|
Normally if you are doing a modification time sync rclone will update
|
|
modification times without \f[C]--refresh-times\f[R] provided that the
|
|
remote supports checksums \f[B]and\f[R] the checksums match on the file.
|
|
However if the checksums are absent then rclone will upload the file
|
|
rather than setting the timestamp as this is the safe behaviour.
|
|
.SS --retries int
|
|
.PP
|
|
Retry the entire sync if it fails this many times it fails (default 3).
|
|
.PP
|
|
Some remotes can be unreliable and a few retries help pick up the files
|
|
which didn\[aq]t get transferred because of errors.
|
|
.PP
|
|
Disable retries with \f[C]--retries 1\f[R].
|
|
.SS --retries-sleep=TIME
|
|
.PP
|
|
This sets the interval between each retry specified by
|
|
\f[C]--retries\f[R]
|
|
.PP
|
|
The default is \f[C]0\f[R].
|
|
Use \f[C]0\f[R] to disable.
|
|
.SS --size-only
|
|
.PP
|
|
Normally rclone will look at modification time and size of files to see
|
|
if they are equal.
|
|
If you set this flag then rclone will check only the size.
|
|
.PP
|
|
This can be useful transferring files from Dropbox which have been
|
|
modified by the desktop sync client which doesn\[aq]t set checksums of
|
|
modification times in the same way as rclone.
|
|
.SS --stats=TIME
|
|
.PP
|
|
Commands which transfer data (\f[C]sync\f[R], \f[C]copy\f[R],
|
|
\f[C]copyto\f[R], \f[C]move\f[R], \f[C]moveto\f[R]) will print data
|
|
transfer stats at regular intervals to show their progress.
|
|
.PP
|
|
This sets the interval.
|
|
.PP
|
|
The default is \f[C]1m\f[R].
|
|
Use \f[C]0\f[R] to disable.
|
|
.PP
|
|
If you set the stats interval then all commands can show stats.
|
|
This can be useful when running other commands, \f[C]check\f[R] or
|
|
\f[C]mount\f[R] for example.
|
|
.PP
|
|
Stats are logged at \f[C]INFO\f[R] level by default which means they
|
|
won\[aq]t show at default log level \f[C]NOTICE\f[R].
|
|
Use \f[C]--stats-log-level NOTICE\f[R] or \f[C]-v\f[R] to make them
|
|
show.
|
|
See the Logging section for more info on log levels.
|
|
.PP
|
|
Note that on macOS you can send a SIGINFO (which is normally ctrl-T in
|
|
the terminal) to make the stats print immediately.
|
|
.SS --stats-file-name-length integer
|
|
.PP
|
|
By default, the \f[C]--stats\f[R] output will truncate file names and
|
|
paths longer than 40 characters.
|
|
This is equivalent to providing \f[C]--stats-file-name-length 40\f[R].
|
|
Use \f[C]--stats-file-name-length 0\f[R] to disable any truncation of
|
|
file names printed by stats.
|
|
.SS --stats-log-level string
|
|
.PP
|
|
Log level to show \f[C]--stats\f[R] output at.
|
|
This can be \f[C]DEBUG\f[R], \f[C]INFO\f[R], \f[C]NOTICE\f[R], or
|
|
\f[C]ERROR\f[R].
|
|
The default is \f[C]INFO\f[R].
|
|
This means at the default level of logging which is \f[C]NOTICE\f[R] the
|
|
stats won\[aq]t show - if you want them to then use
|
|
\f[C]--stats-log-level NOTICE\f[R].
|
|
See the Logging section for more info on log levels.
|
|
.SS --stats-one-line
|
|
.PP
|
|
When this is specified, rclone condenses the stats into a single line
|
|
showing the most important stats only.
|
|
.SS --stats-one-line-date
|
|
.PP
|
|
When this is specified, rclone enables the single-line stats and
|
|
prepends the display with a date string.
|
|
The default is \f[C]2006/01/02 15:04:05 -\f[R]
|
|
.SS --stats-one-line-date-format
|
|
.PP
|
|
When this is specified, rclone enables the single-line stats and
|
|
prepends the display with a user-supplied date string.
|
|
The date string MUST be enclosed in quotes.
|
|
Follow golang specs (https://golang.org/pkg/time/#Time.Format) for date
|
|
formatting syntax.
|
|
.SS --stats-unit=bits|bytes
|
|
.PP
|
|
By default, data transfer rates will be printed in bytes per second.
|
|
.PP
|
|
This option allows the data rate to be printed in bits per second.
|
|
.PP
|
|
Data transfer volume will still be reported in bytes.
|
|
.PP
|
|
The rate is reported as a binary unit, not SI unit.
|
|
So 1 Mbit/s equals 1,048,576 bit/s and not 1,000,000 bit/s.
|
|
.PP
|
|
The default is \f[C]bytes\f[R].
|
|
.SS --suffix=SUFFIX
|
|
.PP
|
|
When using \f[C]sync\f[R], \f[C]copy\f[R] or \f[C]move\f[R] any files
|
|
which would have been overwritten or deleted will have the suffix added
|
|
to them.
|
|
If there is a file with the same path (after the suffix has been added),
|
|
then it will be overwritten.
|
|
.PP
|
|
The remote in use must support server-side move or copy and you must use
|
|
the same remote as the destination of the sync.
|
|
.PP
|
|
This is for use with files to add the suffix in the current directory or
|
|
with \f[C]--backup-dir\f[R].
|
|
See \f[C]--backup-dir\f[R] for more info.
|
|
.PP
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy -i /path/to/local/file remote:current --suffix .bak
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
will copy \f[C]/path/to/local\f[R] to \f[C]remote:current\f[R], but for
|
|
any files which would have been updated or deleted have .bak added.
|
|
.PP
|
|
If using \f[C]rclone sync\f[R] with \f[C]--suffix\f[R] and without
|
|
\f[C]--backup-dir\f[R] then it is recommended to put a filter rule in
|
|
excluding the suffix otherwise the \f[C]sync\f[R] will delete the backup
|
|
files.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /path/to/local/file remote:current --suffix .bak --exclude \[dq]*.bak\[dq]
|
|
\f[R]
|
|
.fi
|
|
.SS --suffix-keep-extension
|
|
.PP
|
|
When using \f[C]--suffix\f[R], setting this causes rclone put the SUFFIX
|
|
before the extension of the files that it backs up rather than after.
|
|
.PP
|
|
So let\[aq]s say we had \f[C]--suffix -2019-01-01\f[R], without the flag
|
|
\f[C]file.txt\f[R] would be backed up to \f[C]file.txt-2019-01-01\f[R]
|
|
and with the flag it would be backed up to
|
|
\f[C]file-2019-01-01.txt\f[R].
|
|
This can be helpful to make sure the suffixed files can still be opened.
|
|
.SS --syslog
|
|
.PP
|
|
On capable OSes (not Windows or Plan9) send all log output to syslog.
|
|
.PP
|
|
This can be useful for running rclone in a script or
|
|
\f[C]rclone mount\f[R].
|
|
.SS --syslog-facility string
|
|
.PP
|
|
If using \f[C]--syslog\f[R] this sets the syslog facility (e.g.
|
|
\f[C]KERN\f[R], \f[C]USER\f[R]).
|
|
See \f[C]man syslog\f[R] for a list of possible facilities.
|
|
The default facility is \f[C]DAEMON\f[R].
|
|
.SS --temp-dir=DIR
|
|
.PP
|
|
Specify the directory rclone will use for temporary files, to override
|
|
the default.
|
|
Make sure the directory exists and have accessible permissions.
|
|
.PP
|
|
By default the operating system\[aq]s temp directory will be used: - On
|
|
Unix systems, \f[C]$TMPDIR\f[R] if non-empty, else \f[C]/tmp\f[R].
|
|
- On Windows, the first non-empty value from \f[C]%TMP%\f[R],
|
|
\f[C]%TEMP%\f[R], \f[C]%USERPROFILE%\f[R], or the Windows directory.
|
|
.PP
|
|
When overriding the default with this option, the specified path will be
|
|
set as value of environment variable \f[C]TMPDIR\f[R] on Unix systems
|
|
and \f[C]TMP\f[R] and \f[C]TEMP\f[R] on Windows.
|
|
.PP
|
|
You can use the config
|
|
paths (https://rclone.org/commands/rclone_config_paths/) command to see
|
|
the current value.
|
|
.SS --tpslimit float
|
|
.PP
|
|
Limit transactions per second to this number.
|
|
Default is 0 which is used to mean unlimited transactions per second.
|
|
.PP
|
|
A transaction is roughly defined as an API call; its exact meaning will
|
|
depend on the backend.
|
|
For HTTP based backends it is an HTTP PUT/GET/POST/etc and its response.
|
|
For FTP/SFTP it is a round trip transaction over TCP.
|
|
.PP
|
|
For example, to limit rclone to 10 transactions per second use
|
|
\f[C]--tpslimit 10\f[R], or to 1 transaction every 2 seconds use
|
|
\f[C]--tpslimit 0.5\f[R].
|
|
.PP
|
|
Use this when the number of transactions per second from rclone is
|
|
causing a problem with the cloud storage provider (e.g.
|
|
getting you banned or rate limited).
|
|
.PP
|
|
This can be very useful for \f[C]rclone mount\f[R] to control the
|
|
behaviour of applications using it.
|
|
.PP
|
|
This limit applies to all HTTP based backends and to the FTP and SFTP
|
|
backends.
|
|
It does not apply to the local backend or the Storj backend.
|
|
.PP
|
|
See also \f[C]--tpslimit-burst\f[R].
|
|
.SS --tpslimit-burst int
|
|
.PP
|
|
Max burst of transactions for \f[C]--tpslimit\f[R] (default
|
|
\f[C]1\f[R]).
|
|
.PP
|
|
Normally \f[C]--tpslimit\f[R] will do exactly the number of transaction
|
|
per second specified.
|
|
However if you supply \f[C]--tps-burst\f[R] then rclone can save up some
|
|
transactions from when it was idle giving a burst of up to the parameter
|
|
supplied.
|
|
.PP
|
|
For example if you provide \f[C]--tpslimit-burst 10\f[R] then if rclone
|
|
has been idle for more than 10*\f[C]--tpslimit\f[R] then it can do 10
|
|
transactions very quickly before they are limited again.
|
|
.PP
|
|
This may be used to increase performance of \f[C]--tpslimit\f[R] without
|
|
changing the long term average number of transactions per second.
|
|
.SS --track-renames
|
|
.PP
|
|
By default, rclone doesn\[aq]t keep track of renamed files, so if you
|
|
rename a file locally then sync it to a remote, rclone will delete the
|
|
old file on the remote and upload a new copy.
|
|
.PP
|
|
If you use this flag, and the remote supports server-side copy or
|
|
server-side move, and the source and destination have a compatible hash,
|
|
then this will track renames during \f[C]sync\f[R] operations and
|
|
perform renaming server-side.
|
|
.PP
|
|
Files will be matched by size and hash - if both match then a rename
|
|
will be considered.
|
|
.PP
|
|
If the destination does not support server-side copy or move, rclone
|
|
will fall back to the default behaviour and log an error level message
|
|
to the console.
|
|
.PP
|
|
Encrypted destinations are not currently supported by
|
|
\f[C]--track-renames\f[R] if \f[C]--track-renames-strategy\f[R] includes
|
|
\f[C]hash\f[R].
|
|
.PP
|
|
Note that \f[C]--track-renames\f[R] is incompatible with
|
|
\f[C]--no-traverse\f[R] and that it uses extra memory to keep track of
|
|
all the rename candidates.
|
|
.PP
|
|
Note also that \f[C]--track-renames\f[R] is incompatible with
|
|
\f[C]--delete-before\f[R] and will select \f[C]--delete-after\f[R]
|
|
instead of \f[C]--delete-during\f[R].
|
|
.SS --track-renames-strategy (hash,modtime,leaf,size)
|
|
.PP
|
|
This option changes the matching criteria for \f[C]--track-renames\f[R].
|
|
.PP
|
|
The matching is controlled by a comma separated selection of these
|
|
tokens:
|
|
.IP \[bu] 2
|
|
\f[C]modtime\f[R] - the modification time of the file - not supported on
|
|
all backends
|
|
.IP \[bu] 2
|
|
\f[C]hash\f[R] - the hash of the file contents - not supported on all
|
|
backends
|
|
.IP \[bu] 2
|
|
\f[C]leaf\f[R] - the name of the file not including its directory name
|
|
.IP \[bu] 2
|
|
\f[C]size\f[R] - the size of the file (this is always enabled)
|
|
.PP
|
|
So using \f[C]--track-renames-strategy modtime,leaf\f[R] would match
|
|
files based on modification time, the leaf of the file name and the size
|
|
only.
|
|
.PP
|
|
Using \f[C]--track-renames-strategy modtime\f[R] or \f[C]leaf\f[R] can
|
|
enable \f[C]--track-renames\f[R] support for encrypted destinations.
|
|
.PP
|
|
If nothing is specified, the default option is matching by
|
|
\f[C]hash\f[R]es.
|
|
.PP
|
|
Note that the \f[C]hash\f[R] strategy is not supported with encrypted
|
|
destinations.
|
|
.SS --delete-(before,during,after)
|
|
.PP
|
|
This option allows you to specify when files on your destination are
|
|
deleted when you sync folders.
|
|
.PP
|
|
Specifying the value \f[C]--delete-before\f[R] will delete all files
|
|
present on the destination, but not on the source \f[I]before\f[R]
|
|
starting the transfer of any new or updated files.
|
|
This uses two passes through the file systems, one for the deletions and
|
|
one for the copies.
|
|
.PP
|
|
Specifying \f[C]--delete-during\f[R] will delete files while checking
|
|
and uploading files.
|
|
This is the fastest option and uses the least memory.
|
|
.PP
|
|
Specifying \f[C]--delete-after\f[R] (the default value) will delay
|
|
deletion of files until all new/updated files have been successfully
|
|
transferred.
|
|
The files to be deleted are collected in the copy pass then deleted
|
|
after the copy pass has completed successfully.
|
|
The files to be deleted are held in memory so this mode may use more
|
|
memory.
|
|
This is the safest mode as it will only delete files if there have been
|
|
no errors subsequent to that.
|
|
If there have been errors before the deletions start then you will get
|
|
the message \f[C]not deleting files as there were IO errors\f[R].
|
|
.SS --fast-list
|
|
.PP
|
|
When doing anything which involves a directory listing (e.g.
|
|
\f[C]sync\f[R], \f[C]copy\f[R], \f[C]ls\f[R] - in fact nearly every
|
|
command), rclone normally lists a directory and processes it before
|
|
using more directory lists to process any subdirectories.
|
|
This can be parallelised and works very quickly using the least amount
|
|
of memory.
|
|
.PP
|
|
However, some remotes have a way of listing all files beneath a
|
|
directory in one (or a small number) of transactions.
|
|
These tend to be the bucket-based remotes (e.g.
|
|
S3, B2, GCS, Swift, Hubic).
|
|
.PP
|
|
If you use the \f[C]--fast-list\f[R] flag then rclone will use this
|
|
method for listing directories.
|
|
This will have the following consequences for the listing:
|
|
.IP \[bu] 2
|
|
It \f[B]will\f[R] use fewer transactions (important if you pay for them)
|
|
.IP \[bu] 2
|
|
It \f[B]will\f[R] use more memory.
|
|
Rclone has to load the whole listing into memory.
|
|
.IP \[bu] 2
|
|
It \f[I]may\f[R] be faster because it uses fewer transactions
|
|
.IP \[bu] 2
|
|
It \f[I]may\f[R] be slower because it can\[aq]t be parallelized
|
|
.PP
|
|
rclone should always give identical results with and without
|
|
\f[C]--fast-list\f[R].
|
|
.PP
|
|
If you pay for transactions and can fit your entire sync listing into
|
|
memory then \f[C]--fast-list\f[R] is recommended.
|
|
If you have a very big sync to do then don\[aq]t use
|
|
\f[C]--fast-list\f[R] otherwise you will run out of memory.
|
|
.PP
|
|
If you use \f[C]--fast-list\f[R] on a remote which doesn\[aq]t support
|
|
it, then rclone will just ignore it.
|
|
.SS --timeout=TIME
|
|
.PP
|
|
This sets the IO idle timeout.
|
|
If a transfer has started but then becomes idle for this long it is
|
|
considered broken and disconnected.
|
|
.PP
|
|
The default is \f[C]5m\f[R].
|
|
Set to \f[C]0\f[R] to disable.
|
|
.SS --transfers=N
|
|
.PP
|
|
The number of file transfers to run in parallel.
|
|
It can sometimes be useful to set this to a smaller number if the remote
|
|
is giving a lot of timeouts or bigger if you have lots of bandwidth and
|
|
a fast remote.
|
|
.PP
|
|
The default is to run 4 file transfers in parallel.
|
|
.SS -u, --update
|
|
.PP
|
|
This forces rclone to skip any files which exist on the destination and
|
|
have a modified time that is newer than the source file.
|
|
.PP
|
|
This can be useful in avoiding needless transfers when transferring to a
|
|
remote which doesn\[aq]t support modification times directly (or when
|
|
using \f[C]--use-server-modtime\f[R] to avoid extra API calls) as it is
|
|
more accurate than a \f[C]--size-only\f[R] check and faster than using
|
|
\f[C]--checksum\f[R].
|
|
On such remotes (or when using \f[C]--use-server-modtime\f[R]) the time
|
|
checked will be the uploaded time.
|
|
.PP
|
|
If an existing destination file has a modification time older than the
|
|
source file\[aq]s, it will be updated if the sizes are different.
|
|
If the sizes are the same, it will be updated if the checksum is
|
|
different or not available.
|
|
.PP
|
|
If an existing destination file has a modification time equal (within
|
|
the computed modify window) to the source file\[aq]s, it will be updated
|
|
if the sizes are different.
|
|
The checksum will not be checked in this case unless the
|
|
\f[C]--checksum\f[R] flag is provided.
|
|
.PP
|
|
In all other cases the file will not be updated.
|
|
.PP
|
|
Consider using the \f[C]--modify-window\f[R] flag to compensate for time
|
|
skews between the source and the backend, for backends that do not
|
|
support mod times, and instead use uploaded times.
|
|
However, if the backend does not support checksums, note that
|
|
sync\[aq]ing or copying within the time skew window may still result in
|
|
additional transfers for safety.
|
|
.SS --use-mmap
|
|
.PP
|
|
If this flag is set then rclone will use anonymous memory allocated by
|
|
mmap on Unix based platforms and VirtualAlloc on Windows for its
|
|
transfer buffers (size controlled by \f[C]--buffer-size\f[R]).
|
|
Memory allocated like this does not go on the Go heap and can be
|
|
returned to the OS immediately when it is finished with.
|
|
.PP
|
|
If this flag is not set then rclone will allocate and free the buffers
|
|
using the Go memory allocator which may use more memory as memory pages
|
|
are returned less aggressively to the OS.
|
|
.PP
|
|
It is possible this does not work well on all platforms so it is
|
|
disabled by default; in the future it may be enabled by default.
|
|
.SS --use-server-modtime
|
|
.PP
|
|
Some object-store backends (e.g, Swift, S3) do not preserve file
|
|
modification times (modtime).
|
|
On these backends, rclone stores the original modtime as additional
|
|
metadata on the object.
|
|
By default it will make an API call to retrieve the metadata when the
|
|
modtime is needed by an operation.
|
|
.PP
|
|
Use this flag to disable the extra API call and rely instead on the
|
|
server\[aq]s modified time.
|
|
In cases such as a local to remote sync using \f[C]--update\f[R],
|
|
knowing the local file is newer than the time it was last uploaded to
|
|
the remote is sufficient.
|
|
In those cases, this flag can speed up the process and reduce the number
|
|
of API calls necessary.
|
|
.PP
|
|
Using this flag on a sync operation without also using
|
|
\f[C]--update\f[R] would cause all files modified at any time other than
|
|
the last upload time to be uploaded again, which is probably not what
|
|
you want.
|
|
.SS -v, -vv, --verbose
|
|
.PP
|
|
With \f[C]-v\f[R] rclone will tell you about each file that is
|
|
transferred and a small number of significant events.
|
|
.PP
|
|
With \f[C]-vv\f[R] rclone will become very verbose telling you about
|
|
every file it considers and transfers.
|
|
Please send bug reports with a log with this setting.
|
|
.SS -V, --version
|
|
.PP
|
|
Prints the version number
|
|
.SS SSL/TLS options
|
|
.PP
|
|
The outgoing SSL/TLS connections rclone makes can be controlled with
|
|
these options.
|
|
For example this can be very useful with the HTTP or WebDAV backends.
|
|
Rclone HTTP servers have their own set of configuration for SSL/TLS
|
|
which you can find in their documentation.
|
|
.SS --ca-cert string
|
|
.PP
|
|
This loads the PEM encoded certificate authority certificate and uses it
|
|
to verify the certificates of the servers rclone connects to.
|
|
.PP
|
|
If you have generated certificates signed with a local CA then you will
|
|
need this flag to connect to servers using those certificates.
|
|
.SS --client-cert string
|
|
.PP
|
|
This loads the PEM encoded client side certificate.
|
|
.PP
|
|
This is used for mutual TLS
|
|
authentication (https://en.wikipedia.org/wiki/Mutual_authentication).
|
|
.PP
|
|
The \f[C]--client-key\f[R] flag is required too when using this.
|
|
.SS --client-key string
|
|
.PP
|
|
This loads the PEM encoded client side private key used for mutual TLS
|
|
authentication.
|
|
Used in conjunction with \f[C]--client-cert\f[R].
|
|
.SS --no-check-certificate=true/false
|
|
.PP
|
|
\f[C]--no-check-certificate\f[R] controls whether a client verifies the
|
|
server\[aq]s certificate chain and host name.
|
|
If \f[C]--no-check-certificate\f[R] is true, TLS accepts any certificate
|
|
presented by the server and any host name in that certificate.
|
|
In this mode, TLS is susceptible to man-in-the-middle attacks.
|
|
.PP
|
|
This option defaults to \f[C]false\f[R].
|
|
.PP
|
|
\f[B]This should be used only for testing.\f[R]
|
|
.SS Configuration Encryption
|
|
.PP
|
|
Your configuration file contains information for logging in to your
|
|
cloud services.
|
|
This means that you should keep your \f[C]rclone.conf\f[R] file in a
|
|
secure location.
|
|
.PP
|
|
If you are in an environment where that isn\[aq]t possible, you can add
|
|
a password to your configuration.
|
|
This means that you will have to supply the password every time you
|
|
start rclone.
|
|
.PP
|
|
To add a password to your rclone configuration, execute
|
|
\f[C]rclone config\f[R].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
>rclone config
|
|
Current remotes:
|
|
|
|
e) Edit existing remote
|
|
n) New remote
|
|
d) Delete remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/s/q>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Go into \f[C]s\f[R], Set configuration password:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
e/n/d/s/q> s
|
|
Your configuration is not encrypted.
|
|
If you add a password, you will protect your login information to cloud services.
|
|
a) Add Password
|
|
q) Quit to main menu
|
|
a/q> a
|
|
Enter NEW configuration password:
|
|
password:
|
|
Confirm NEW password:
|
|
password:
|
|
Password set
|
|
Your configuration is encrypted.
|
|
c) Change Password
|
|
u) Unencrypt configuration
|
|
q) Quit to main menu
|
|
c/u/q>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Your configuration is now encrypted, and every time you start rclone you
|
|
will have to supply the password.
|
|
See below for details.
|
|
In the same menu, you can change the password or completely remove
|
|
encryption from your configuration.
|
|
.PP
|
|
There is no way to recover the configuration if you lose your password.
|
|
.PP
|
|
rclone uses nacl
|
|
secretbox (https://godoc.org/golang.org/x/crypto/nacl/secretbox) which
|
|
in turn uses XSalsa20 and Poly1305 to encrypt and authenticate your
|
|
configuration with secret-key cryptography.
|
|
The password is SHA-256 hashed, which produces the key for secretbox.
|
|
The hashed password is not stored.
|
|
.PP
|
|
While this provides very good security, we do not recommend storing your
|
|
encrypted rclone configuration in public if it contains sensitive
|
|
information, maybe except if you use a very strong password.
|
|
.PP
|
|
If it is safe in your environment, you can set the
|
|
\f[C]RCLONE_CONFIG_PASS\f[R] environment variable to contain your
|
|
password, in which case it will be used for decrypting the
|
|
configuration.
|
|
.PP
|
|
You can set this for a session from a script.
|
|
For unix like systems save this to a file called
|
|
\f[C]set-rclone-password\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#!/bin/echo Source this file don\[aq]t run it
|
|
|
|
read -s RCLONE_CONFIG_PASS
|
|
export RCLONE_CONFIG_PASS
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Then source the file when you want to use it.
|
|
From the shell you would do \f[C]source set-rclone-password\f[R].
|
|
It will then ask you for the password and set it in the environment
|
|
variable.
|
|
.PP
|
|
An alternate means of supplying the password is to provide a script
|
|
which will retrieve the password and print on standard output.
|
|
This script should have a fully specified path name and not rely on any
|
|
environment variables.
|
|
The script is supplied either via
|
|
\f[C]--password-command=\[dq]...\[dq]\f[R] command line argument or via
|
|
the \f[C]RCLONE_PASSWORD_COMMAND\f[R] environment variable.
|
|
.PP
|
|
One useful example of this is using the \f[C]passwordstore\f[R]
|
|
application to retrieve the password:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
export RCLONE_PASSWORD_COMMAND=\[dq]pass rclone/config\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If the \f[C]passwordstore\f[R] password manager holds the password for
|
|
the rclone configuration, using the script method means the password is
|
|
primarily protected by the \f[C]passwordstore\f[R] system, and is never
|
|
embedded in the clear in scripts, nor available for examination using
|
|
the standard commands available.
|
|
It is quite possible with long running rclone sessions for copies of
|
|
passwords to be innocently captured in log files or terminal scroll
|
|
buffers, etc.
|
|
Using the script method of supplying the password enhances the security
|
|
of the config password considerably.
|
|
.PP
|
|
If you are running rclone inside a script, unless you are using the
|
|
\f[C]--password-command\f[R] method, you might want to disable password
|
|
prompts.
|
|
To do that, pass the parameter \f[C]--ask-password=false\f[R] to rclone.
|
|
This will make rclone fail instead of asking for a password if
|
|
\f[C]RCLONE_CONFIG_PASS\f[R] doesn\[aq]t contain a valid password, and
|
|
\f[C]--password-command\f[R] has not been supplied.
|
|
.PP
|
|
Whenever running commands that may be affected by options in a
|
|
configuration file, rclone will look for an existing file according to
|
|
the rules described above, and load any it finds.
|
|
If an encrypted file is found, this includes decrypting it, with the
|
|
possible consequence of a password prompt.
|
|
When executing a command line that you know are not actually using
|
|
anything from such a configuration file, you can avoid it being loaded
|
|
by overriding the location, e.g.
|
|
with one of the documented special values for memory-only configuration.
|
|
Since only backend options can be stored in configuration files, this is
|
|
normally unnecessary for commands that do not operate on backends, e.g.
|
|
\f[C]genautocomplete\f[R].
|
|
However, it will be relevant for commands that do operate on backends in
|
|
general, but are used without referencing a stored remote, e.g.
|
|
listing local filesystem paths, or connection strings:
|
|
\f[C]rclone --config=\[dq]\[dq] ls .\f[R]
|
|
.SS Developer options
|
|
.PP
|
|
These options are useful when developing or debugging rclone.
|
|
There are also some more remote specific options which aren\[aq]t
|
|
documented here which are used for testing.
|
|
These start with remote name e.g.
|
|
\f[C]--drive-test-option\f[R] - see the docs for the remote in question.
|
|
.SS --cpuprofile=FILE
|
|
.PP
|
|
Write CPU profile to file.
|
|
This can be analysed with \f[C]go tool pprof\f[R].
|
|
.SS --dump flag,flag,flag
|
|
.PP
|
|
The \f[C]--dump\f[R] flag takes a comma separated list of flags to dump
|
|
info about.
|
|
.PP
|
|
Note that some headers including \f[C]Accept-Encoding\f[R] as shown may
|
|
not be correct in the request and the response may not show
|
|
\f[C]Content-Encoding\f[R] if the go standard libraries auto gzip
|
|
encoding was in effect.
|
|
In this case the body of the request will be gunzipped before showing
|
|
it.
|
|
.PP
|
|
The available flags are:
|
|
.SS --dump headers
|
|
.PP
|
|
Dump HTTP headers with \f[C]Authorization:\f[R] lines removed.
|
|
May still contain sensitive info.
|
|
Can be very verbose.
|
|
Useful for debugging only.
|
|
.PP
|
|
Use \f[C]--dump auth\f[R] if you do want the \f[C]Authorization:\f[R]
|
|
headers.
|
|
.SS --dump bodies
|
|
.PP
|
|
Dump HTTP headers and bodies - may contain sensitive info.
|
|
Can be very verbose.
|
|
Useful for debugging only.
|
|
.PP
|
|
Note that the bodies are buffered in memory so don\[aq]t use this for
|
|
enormous files.
|
|
.SS --dump requests
|
|
.PP
|
|
Like \f[C]--dump bodies\f[R] but dumps the request bodies and the
|
|
response headers.
|
|
Useful for debugging download problems.
|
|
.SS --dump responses
|
|
.PP
|
|
Like \f[C]--dump bodies\f[R] but dumps the response bodies and the
|
|
request headers.
|
|
Useful for debugging upload problems.
|
|
.SS --dump auth
|
|
.PP
|
|
Dump HTTP headers - will contain sensitive info such as
|
|
\f[C]Authorization:\f[R] headers - use \f[C]--dump headers\f[R] to dump
|
|
without \f[C]Authorization:\f[R] headers.
|
|
Can be very verbose.
|
|
Useful for debugging only.
|
|
.SS --dump filters
|
|
.PP
|
|
Dump the filters to the output.
|
|
Useful to see exactly what include and exclude options are filtering on.
|
|
.SS --dump goroutines
|
|
.PP
|
|
This dumps a list of the running go-routines at the end of the command
|
|
to standard output.
|
|
.SS --dump openfiles
|
|
.PP
|
|
This dumps a list of the open files at the end of the command.
|
|
It uses the \f[C]lsof\f[R] command to do that so you\[aq]ll need that
|
|
installed to use it.
|
|
.SS --memprofile=FILE
|
|
.PP
|
|
Write memory profile to file.
|
|
This can be analysed with \f[C]go tool pprof\f[R].
|
|
.SS Filtering
|
|
.PP
|
|
For the filtering options
|
|
.IP \[bu] 2
|
|
\f[C]--delete-excluded\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--filter\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--filter-from\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--exclude\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--exclude-from\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--include\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--include-from\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--files-from\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--files-from-raw\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--min-size\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--max-size\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--min-age\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--max-age\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--dump filters\f[R]
|
|
.PP
|
|
See the filtering section (https://rclone.org/filtering/).
|
|
.SS Remote control
|
|
.PP
|
|
For the remote control options and for instructions on how to remote
|
|
control rclone
|
|
.IP \[bu] 2
|
|
\f[C]--rc\f[R]
|
|
.IP \[bu] 2
|
|
and anything starting with \f[C]--rc-\f[R]
|
|
.PP
|
|
See the remote control section (https://rclone.org/rc/).
|
|
.SS Logging
|
|
.PP
|
|
rclone has 4 levels of logging, \f[C]ERROR\f[R], \f[C]NOTICE\f[R],
|
|
\f[C]INFO\f[R] and \f[C]DEBUG\f[R].
|
|
.PP
|
|
By default, rclone logs to standard error.
|
|
This means you can redirect standard error and still see the normal
|
|
output of rclone commands (e.g.
|
|
\f[C]rclone ls\f[R]).
|
|
.PP
|
|
By default, rclone will produce \f[C]Error\f[R] and \f[C]Notice\f[R]
|
|
level messages.
|
|
.PP
|
|
If you use the \f[C]-q\f[R] flag, rclone will only produce
|
|
\f[C]Error\f[R] messages.
|
|
.PP
|
|
If you use the \f[C]-v\f[R] flag, rclone will produce \f[C]Error\f[R],
|
|
\f[C]Notice\f[R] and \f[C]Info\f[R] messages.
|
|
.PP
|
|
If you use the \f[C]-vv\f[R] flag, rclone will produce \f[C]Error\f[R],
|
|
\f[C]Notice\f[R], \f[C]Info\f[R] and \f[C]Debug\f[R] messages.
|
|
.PP
|
|
You can also control the log levels with the \f[C]--log-level\f[R] flag.
|
|
.PP
|
|
If you use the \f[C]--log-file=FILE\f[R] option, rclone will redirect
|
|
\f[C]Error\f[R], \f[C]Info\f[R] and \f[C]Debug\f[R] messages along with
|
|
standard error to FILE.
|
|
.PP
|
|
If you use the \f[C]--syslog\f[R] flag then rclone will log to syslog
|
|
and the \f[C]--syslog-facility\f[R] control which facility it uses.
|
|
.PP
|
|
Rclone prefixes all log messages with their level in capitals, e.g.
|
|
INFO which makes it easy to grep the log file for different kinds of
|
|
information.
|
|
.SS Exit Code
|
|
.PP
|
|
If any errors occur during the command execution, rclone will exit with
|
|
a non-zero exit code.
|
|
This allows scripts to detect when rclone operations have failed.
|
|
.PP
|
|
During the startup phase, rclone will exit immediately if an error is
|
|
detected in the configuration.
|
|
There will always be a log message immediately before exiting.
|
|
.PP
|
|
When rclone is running it will accumulate errors as it goes along, and
|
|
only exit with a non-zero exit code if (after retries) there were still
|
|
failed transfers.
|
|
For every error counted there will be a high priority log message
|
|
(visible with \f[C]-q\f[R]) showing the message and which file caused
|
|
the problem.
|
|
A high priority message is also shown when starting a retry so the user
|
|
can see that any previous error messages may not be valid after the
|
|
retry.
|
|
If rclone has done a retry it will log a high priority message if the
|
|
retry was successful.
|
|
.SS List of exit codes
|
|
.IP \[bu] 2
|
|
\f[C]0\f[R] - success
|
|
.IP \[bu] 2
|
|
\f[C]1\f[R] - Syntax or usage error
|
|
.IP \[bu] 2
|
|
\f[C]2\f[R] - Error not otherwise categorised
|
|
.IP \[bu] 2
|
|
\f[C]3\f[R] - Directory not found
|
|
.IP \[bu] 2
|
|
\f[C]4\f[R] - File not found
|
|
.IP \[bu] 2
|
|
\f[C]5\f[R] - Temporary error (one that more retries might fix) (Retry
|
|
errors)
|
|
.IP \[bu] 2
|
|
\f[C]6\f[R] - Less serious errors (like 461 errors from dropbox)
|
|
(NoRetry errors)
|
|
.IP \[bu] 2
|
|
\f[C]7\f[R] - Fatal error (one that more retries won\[aq]t fix, like
|
|
account suspended) (Fatal errors)
|
|
.IP \[bu] 2
|
|
\f[C]8\f[R] - Transfer exceeded - limit set by --max-transfer reached
|
|
.IP \[bu] 2
|
|
\f[C]9\f[R] - Operation successful, but no files transferred
|
|
.SS Environment Variables
|
|
.PP
|
|
Rclone can be configured entirely using environment variables.
|
|
These can be used to set defaults for options or config file entries.
|
|
.SS Options
|
|
.PP
|
|
Every option in rclone can have its default set by environment variable.
|
|
.PP
|
|
To find the name of the environment variable, first, take the long
|
|
option name, strip the leading \f[C]--\f[R], change \f[C]-\f[R] to
|
|
\f[C]_\f[R], make upper case and prepend \f[C]RCLONE_\f[R].
|
|
.PP
|
|
For example, to always set \f[C]--stats 5s\f[R], set the environment
|
|
variable \f[C]RCLONE_STATS=5s\f[R].
|
|
If you set stats on the command line this will override the environment
|
|
variable setting.
|
|
.PP
|
|
Or to always use the trash in drive \f[C]--drive-use-trash\f[R], set
|
|
\f[C]RCLONE_DRIVE_USE_TRASH=true\f[R].
|
|
.PP
|
|
The same parser is used for the options and the environment variables so
|
|
they take exactly the same form.
|
|
.PP
|
|
The options set by environment variables can be seen with the
|
|
\f[C]-vv\f[R] flag, e.g.
|
|
\f[C]rclone version -vv\f[R].
|
|
.SS Config file
|
|
.PP
|
|
You can set defaults for values in the config file on an individual
|
|
remote basis.
|
|
The names of the config items are documented in the page for each
|
|
backend.
|
|
.PP
|
|
To find the name of the environment variable, you need to set, take
|
|
\f[C]RCLONE_CONFIG_\f[R] + name of remote + \f[C]_\f[R] + name of config
|
|
file option and make it all uppercase.
|
|
.PP
|
|
For example, to configure an S3 remote named \f[C]mys3:\f[R] without a
|
|
config file (using unix ways of setting environment variables):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ export RCLONE_CONFIG_MYS3_TYPE=s3
|
|
$ export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=XXX
|
|
$ export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX
|
|
$ rclone lsd mys3:
|
|
-1 2016-09-21 12:54:21 -1 my-bucket
|
|
$ rclone listremotes | grep mys3
|
|
mys3:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that if you want to create a remote using environment variables you
|
|
must create the \f[C]..._TYPE\f[R] variable as above.
|
|
.PP
|
|
Note that the name of a remote created using environment variable is
|
|
case insensitive, in contrast to regular remotes stored in config file
|
|
as documented above.
|
|
You must write the name in uppercase in the environment variable, but as
|
|
seen from example above it will be listed and can be accessed in
|
|
lowercase, while you can also refer to the same remote in uppercase:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsd mys3:
|
|
-1 2016-09-21 12:54:21 -1 my-bucket
|
|
$ rclone lsd MYS3:
|
|
-1 2016-09-21 12:54:21 -1 my-bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that you can only set the options of the immediate backend, so
|
|
RCLONE_CONFIG_MYS3CRYPT_ACCESS_KEY_ID has no effect, if myS3Crypt is a
|
|
crypt remote based on an S3 remote.
|
|
However RCLONE_S3_ACCESS_KEY_ID will set the access key of all remotes
|
|
using S3, including myS3Crypt.
|
|
.PP
|
|
Note also that now rclone has connection strings, it is probably easier
|
|
to use those instead which makes the above example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd :s3,access_key_id=XXX,secret_access_key=XXX:
|
|
\f[R]
|
|
.fi
|
|
.SS Precedence
|
|
.PP
|
|
The various different methods of backend configuration are read in this
|
|
order and the first one with a value is used.
|
|
.IP \[bu] 2
|
|
Parameters in connection strings, e.g.
|
|
\f[C]myRemote,skip_links:\f[R]
|
|
.IP \[bu] 2
|
|
Flag values as supplied on the command line, e.g.
|
|
\f[C]--skip-links\f[R]
|
|
.IP \[bu] 2
|
|
Remote specific environment vars, e.g.
|
|
\f[C]RCLONE_CONFIG_MYREMOTE_SKIP_LINKS\f[R] (see above).
|
|
.IP \[bu] 2
|
|
Backend-specific environment vars, e.g.
|
|
\f[C]RCLONE_LOCAL_SKIP_LINKS\f[R].
|
|
.IP \[bu] 2
|
|
Backend generic environment vars, e.g.
|
|
\f[C]RCLONE_SKIP_LINKS\f[R].
|
|
.IP \[bu] 2
|
|
Config file, e.g.
|
|
\f[C]skip_links = true\f[R].
|
|
.IP \[bu] 2
|
|
Default values, e.g.
|
|
\f[C]false\f[R] - these can\[aq]t be changed.
|
|
.PP
|
|
So if both \f[C]--skip-links\f[R] is supplied on the command line and an
|
|
environment variable \f[C]RCLONE_LOCAL_SKIP_LINKS\f[R] is set, the
|
|
command line flag will take preference.
|
|
.PP
|
|
The backend configurations set by environment variables can be seen with
|
|
the \f[C]-vv\f[R] flag, e.g.
|
|
\f[C]rclone about myRemote: -vv\f[R].
|
|
.PP
|
|
For non backend configuration the order is as follows:
|
|
.IP \[bu] 2
|
|
Flag values as supplied on the command line, e.g.
|
|
\f[C]--stats 5s\f[R].
|
|
.IP \[bu] 2
|
|
Environment vars, e.g.
|
|
\f[C]RCLONE_STATS=5s\f[R].
|
|
.IP \[bu] 2
|
|
Default values, e.g.
|
|
\f[C]1m\f[R] - these can\[aq]t be changed.
|
|
.SS Other environment variables
|
|
.IP \[bu] 2
|
|
\f[C]RCLONE_CONFIG_PASS\f[R] set to contain your config file password
|
|
(see Configuration Encryption section)
|
|
.IP \[bu] 2
|
|
\f[C]HTTP_PROXY\f[R], \f[C]HTTPS_PROXY\f[R] and \f[C]NO_PROXY\f[R] (or
|
|
the lowercase versions thereof).
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]HTTPS_PROXY\f[R] takes precedence over \f[C]HTTP_PROXY\f[R] for
|
|
https requests.
|
|
.IP \[bu] 2
|
|
The environment values may be either a complete URL or a
|
|
\[dq]host[:port]\[dq] for, in which case the \[dq]http\[dq] scheme is
|
|
assumed.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]USER\f[R] and \f[C]LOGNAME\f[R] values are used as fallbacks for
|
|
current username.
|
|
The primary method for looking up username is OS-specific: Windows API
|
|
on Windows, real user ID in /etc/passwd on Unix systems.
|
|
In the documentation the current username is simply referred to as
|
|
\f[C]$USER\f[R].
|
|
.IP \[bu] 2
|
|
\f[C]RCLONE_CONFIG_DIR\f[R] - rclone \f[B]sets\f[R] this variable for
|
|
use in config files and sub processes to point to the directory holding
|
|
the config file.
|
|
.PP
|
|
The options set by environment variables can be seen with the
|
|
\f[C]-vv\f[R] and \f[C]--log-level=DEBUG\f[R] flags, e.g.
|
|
\f[C]rclone version -vv\f[R].
|
|
.SH Configuring rclone on a remote / headless machine
|
|
.PP
|
|
Some of the configurations (those involving oauth2) require an Internet
|
|
connected web browser.
|
|
.PP
|
|
If you are trying to set rclone up on a remote or headless box with no
|
|
browser available on it (e.g.
|
|
a NAS or a server in a datacenter) then you will need to use an
|
|
alternative means of configuration.
|
|
There are two ways of doing it, described below.
|
|
.SS Configuring using rclone authorize
|
|
.PP
|
|
On the headless box run \f[C]rclone\f[R] config but answer \f[C]N\f[R]
|
|
to the \f[C]Use auto config?\f[R] question.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\&...
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes (default)
|
|
n) No
|
|
y/n> n
|
|
For this to work, you will need rclone available on a machine that has
|
|
a web browser available.
|
|
|
|
For more help and alternate methods see: https://rclone.org/remote_setup/
|
|
|
|
Execute the following on the machine with the web browser (same rclone
|
|
version recommended):
|
|
|
|
rclone authorize \[dq]amazon cloud drive\[dq]
|
|
|
|
Then paste the result below:
|
|
result>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Then on your main desktop machine
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone authorize \[dq]amazon cloud drive\[dq]
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
Paste the following into your remote machine --->
|
|
SECRET_TOKEN
|
|
<---End paste
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Then back to the headless box, paste in the code
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
result> SECRET_TOKEN
|
|
--------------------
|
|
[acd12]
|
|
client_id =
|
|
client_secret =
|
|
token = SECRET_TOKEN
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d>
|
|
\f[R]
|
|
.fi
|
|
.SS Configuring by copying the config file
|
|
.PP
|
|
Rclone stores all of its config in a single configuration file.
|
|
This can easily be copied to configure a remote rclone.
|
|
.PP
|
|
So first configure rclone on your desktop machine with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
to set up the config file.
|
|
.PP
|
|
Find the config file by running \f[C]rclone config file\f[R], for
|
|
example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone config file
|
|
Configuration file is stored at:
|
|
/home/user/.rclone.conf
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Now transfer it to the remote box (scp, cut paste, ftp, sftp, etc.) and
|
|
place it in the correct place (use \f[C]rclone config file\f[R] on the
|
|
remote box to find out where).
|
|
.SH Filtering, includes and excludes
|
|
.PP
|
|
Filter flags determine which files rclone \f[C]sync\f[R],
|
|
\f[C]move\f[R], \f[C]ls\f[R], \f[C]lsl\f[R], \f[C]md5sum\f[R],
|
|
\f[C]sha1sum\f[R], \f[C]size\f[R], \f[C]delete\f[R], \f[C]check\f[R] and
|
|
similar commands apply to.
|
|
.PP
|
|
They are specified in terms of path/file name patterns; path/file lists;
|
|
file age and size, or presence of a file in a directory.
|
|
Bucket based remotes without the concept of directory apply filters to
|
|
object key, age and size in an analogous way.
|
|
.PP
|
|
Rclone \f[C]purge\f[R] does not obey filters.
|
|
.PP
|
|
To test filters without risk of damage to data, apply them to
|
|
\f[C]rclone ls\f[R], or with the \f[C]--dry-run\f[R] and \f[C]-vv\f[R]
|
|
flags.
|
|
.PP
|
|
Rclone filter patterns can only be used in filter command line options,
|
|
not in the specification of a remote.
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone copy \[dq]remote:dir*.jpg\[dq] /path/to/dir\f[R] does not
|
|
have a filter effect.
|
|
\f[C]rclone copy remote:dir /path/to/dir --include \[dq]*.jpg\[dq]\f[R]
|
|
does.
|
|
.PP
|
|
\f[B]Important\f[R] Avoid mixing any two of \f[C]--include...\f[R],
|
|
\f[C]--exclude...\f[R] or \f[C]--filter...\f[R] flags in an rclone
|
|
command.
|
|
The results may not be what you expect.
|
|
Instead use a \f[C]--filter...\f[R] flag.
|
|
.SS Patterns for matching path/file names
|
|
.SS Pattern syntax
|
|
.PP
|
|
Here is a formal definition of the pattern syntax, examples are below.
|
|
.PP
|
|
Rclone matching rules follow a glob style:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
* matches any sequence of non-separator (/) characters
|
|
** matches any sequence of characters including / separators
|
|
? matches any single non-separator (/) character
|
|
[ [ ! ] { character-range } ]
|
|
character class (must be non-empty)
|
|
{ pattern-list }
|
|
pattern alternatives
|
|
{{ regexp }}
|
|
regular expression to match
|
|
c matches character c (c != *, **, ?, \[rs], [, {, })
|
|
\[rs]c matches reserved character c (c = *, **, ?, \[rs], [, {, }) or character class
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
character-range:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
c matches character c (c != \[rs], -, ])
|
|
\[rs]c matches reserved character c (c = \[rs], -, ])
|
|
lo - hi matches character c for lo <= c <= hi
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
pattern-list:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
pattern { , pattern }
|
|
comma-separated (without spaces) patterns
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
character classes (see Go regular expression
|
|
reference (https://golang.org/pkg/regexp/syntax/)) include:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Named character classes (e.g. [\[rs]d], [\[ha]\[rs]d], [\[rs]D], [\[ha]\[rs]D])
|
|
Perl character classes (e.g. \[rs]s, \[rs]S, \[rs]w, \[rs]W)
|
|
ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]])
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
regexp for advanced users to insert a regular expression - see below for
|
|
more info:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Any re2 regular expression not containing \[ga]}}\[ga]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If the filter pattern starts with a \f[C]/\f[R] then it only matches at
|
|
the top level of the directory tree, \f[B]relative to the root of the
|
|
remote\f[R] (not necessarily the root of the drive).
|
|
If it does not start with \f[C]/\f[R] then it is matched starting at the
|
|
\f[B]end of the path/file name\f[R] but it only matches a complete path
|
|
element - it must match from a \f[C]/\f[R] separator or the beginning of
|
|
the path/file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
file.jpg - matches \[dq]file.jpg\[dq]
|
|
- matches \[dq]directory/file.jpg\[dq]
|
|
- doesn\[aq]t match \[dq]afile.jpg\[dq]
|
|
- doesn\[aq]t match \[dq]directory/afile.jpg\[dq]
|
|
/file.jpg - matches \[dq]file.jpg\[dq] in the root directory of the remote
|
|
- doesn\[aq]t match \[dq]afile.jpg\[dq]
|
|
- doesn\[aq]t match \[dq]directory/file.jpg\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The top level of the remote may not be the top level of the drive.
|
|
.PP
|
|
E.g.
|
|
for a Microsoft Windows local directory structure
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
F:
|
|
\[u251C]\[u2500]\[u2500] bkp
|
|
\[u251C]\[u2500]\[u2500] data
|
|
\[br] \[u251C]\[u2500]\[u2500] excl
|
|
\[br] \[br] \[u251C]\[u2500]\[u2500] 123.jpg
|
|
\[br] \[br] \[u2514]\[u2500]\[u2500] 456.jpg
|
|
\[br] \[u251C]\[u2500]\[u2500] incl
|
|
\[br] \[br] \[u2514]\[u2500]\[u2500] document.pdf
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy the contents of folder \f[C]data\f[R] into folder \f[C]bkp\f[R]
|
|
excluding the contents of subfolder \f[C]excl\f[R]the following command
|
|
treats \f[C]F:\[rs]data\f[R] and \f[C]F:\[rs]bkp\f[R] as top level for
|
|
filtering.
|
|
.PP
|
|
\f[C]rclone copy F:\[rs]data\[rs] F:\[rs]bkp\[rs] --exclude=/excl/**\f[R]
|
|
.PP
|
|
\f[B]Important\f[R] Use \f[C]/\f[R] in path/file name patterns and not
|
|
\f[C]\[rs]\f[R] even if running on Microsoft Windows.
|
|
.PP
|
|
Simple patterns are case sensitive unless the \f[C]--ignore-case\f[R]
|
|
flag is used.
|
|
.PP
|
|
Without \f[C]--ignore-case\f[R] (default)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
potato - matches \[dq]potato\[dq]
|
|
- doesn\[aq]t match \[dq]POTATO\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
With \f[C]--ignore-case\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
potato - matches \[dq]potato\[dq]
|
|
- matches \[dq]POTATO\[dq]
|
|
\f[R]
|
|
.fi
|
|
.SS Using regular expressions in filter patterns
|
|
.PP
|
|
The syntax of filter patterns is glob style matching (like
|
|
\f[C]bash\f[R] uses) to make things easy for users.
|
|
However this does not provide absolute control over the matching, so for
|
|
advanced users rclone also provides a regular expression syntax.
|
|
.PP
|
|
The regular expressions used are as defined in the Go regular expression
|
|
reference (https://golang.org/pkg/regexp/syntax/).
|
|
Regular expressions should be enclosed in \f[C]{{\f[R] \f[C]}}\f[R].
|
|
They will match only the last path segment if the glob doesn\[aq]t start
|
|
with \f[C]/\f[R] or the whole path name if it does.
|
|
.PP
|
|
Here is how the \f[C]{{regexp}}\f[R] is transformed into an full regular
|
|
expression to match the entire path:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{{regexp}} becomes (\[ha]|/)(regexp)$
|
|
/{{regexp}} becomes \[ha](regexp)$
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Regexp syntax can be mixed with glob syntax, for example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
*.{{jpe?g}} to match file.jpg, file.jpeg but not file.png
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can also use regexp flags - to set case insensitive, for example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
*.{{(?i)jpg}} to match file.jpg, file.JPG but not file.png
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Be careful with wildcards in regular expressions - you don\[aq]t want
|
|
them to match path separators normally.
|
|
To match any file name starting with \f[C]start\f[R] and ending with
|
|
\f[C]end\f[R] write
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{{start[\[ha]/]*end\[rs].jpg}}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Not
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{{start.*end\[rs].jpg}}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Which will match a directory called \f[C]start\f[R] with a file called
|
|
\f[C]end.jpg\f[R] in it as the \f[C].*\f[R] will match \f[C]/\f[R]
|
|
characters.
|
|
.PP
|
|
Note that you can use \f[C]-vv --dump filters\f[R] to show the filter
|
|
patterns in regexp format - rclone implements the glob patters by
|
|
transforming them into regular expressions.
|
|
.SS Filter pattern examples
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l l l l.
|
|
T{
|
|
Description
|
|
T}@T{
|
|
Pattern
|
|
T}@T{
|
|
Matches
|
|
T}@T{
|
|
Does not match
|
|
T}
|
|
_
|
|
T{
|
|
Wildcard
|
|
T}@T{
|
|
\f[C]*.jpg\f[R]
|
|
T}@T{
|
|
\f[C]/file.jpg\f[R]
|
|
T}@T{
|
|
\f[C]/file.png\f[R]
|
|
T}
|
|
T{
|
|
T}@T{
|
|
T}@T{
|
|
\f[C]/dir/file.jpg\f[R]
|
|
T}@T{
|
|
\f[C]/dir/file.png\f[R]
|
|
T}
|
|
T{
|
|
Rooted
|
|
T}@T{
|
|
\f[C]/*.jpg\f[R]
|
|
T}@T{
|
|
\f[C]/file.jpg\f[R]
|
|
T}@T{
|
|
\f[C]/file.png\f[R]
|
|
T}
|
|
T{
|
|
T}@T{
|
|
T}@T{
|
|
\f[C]/file2.jpg\f[R]
|
|
T}@T{
|
|
\f[C]/dir/file.jpg\f[R]
|
|
T}
|
|
T{
|
|
Alternates
|
|
T}@T{
|
|
\f[C]*.{jpg,png}\f[R]
|
|
T}@T{
|
|
\f[C]/file.jpg\f[R]
|
|
T}@T{
|
|
\f[C]/file.gif\f[R]
|
|
T}
|
|
T{
|
|
T}@T{
|
|
T}@T{
|
|
\f[C]/dir/file.gif\f[R]
|
|
T}@T{
|
|
\f[C]/dir/file.gif\f[R]
|
|
T}
|
|
T{
|
|
Path Wildcard
|
|
T}@T{
|
|
\f[C]dir/**\f[R]
|
|
T}@T{
|
|
\f[C]/dir/anyfile\f[R]
|
|
T}@T{
|
|
\f[C]file.png\f[R]
|
|
T}
|
|
T{
|
|
T}@T{
|
|
T}@T{
|
|
\f[C]/subdir/dir/subsubdir/anyfile\f[R]
|
|
T}@T{
|
|
\f[C]/subdir/file.png\f[R]
|
|
T}
|
|
T{
|
|
Any Char
|
|
T}@T{
|
|
\f[C]*.t?t\f[R]
|
|
T}@T{
|
|
\f[C]/file.txt\f[R]
|
|
T}@T{
|
|
\f[C]/file.qxt\f[R]
|
|
T}
|
|
T{
|
|
T}@T{
|
|
T}@T{
|
|
\f[C]/dir/file.tzt\f[R]
|
|
T}@T{
|
|
\f[C]/dir/file.png\f[R]
|
|
T}
|
|
T{
|
|
Range
|
|
T}@T{
|
|
\f[C]*.[a-z]\f[R]
|
|
T}@T{
|
|
\f[C]/file.a\f[R]
|
|
T}@T{
|
|
\f[C]/file.0\f[R]
|
|
T}
|
|
T{
|
|
T}@T{
|
|
T}@T{
|
|
\f[C]/dir/file.b\f[R]
|
|
T}@T{
|
|
\f[C]/dir/file.1\f[R]
|
|
T}
|
|
T{
|
|
Escape
|
|
T}@T{
|
|
\f[C]*.\[rs]?\[rs]?\[rs]?\f[R]
|
|
T}@T{
|
|
\f[C]/file.???\f[R]
|
|
T}@T{
|
|
\f[C]/file.abc\f[R]
|
|
T}
|
|
T{
|
|
T}@T{
|
|
T}@T{
|
|
\f[C]/dir/file.???\f[R]
|
|
T}@T{
|
|
\f[C]/dir/file.def\f[R]
|
|
T}
|
|
T{
|
|
Class
|
|
T}@T{
|
|
\f[C]*.\[rs]d\[rs]d\[rs]d\f[R]
|
|
T}@T{
|
|
\f[C]/file.012\f[R]
|
|
T}@T{
|
|
\f[C]/file.abc\f[R]
|
|
T}
|
|
T{
|
|
T}@T{
|
|
T}@T{
|
|
\f[C]/dir/file.345\f[R]
|
|
T}@T{
|
|
\f[C]/dir/file.def\f[R]
|
|
T}
|
|
T{
|
|
Regexp
|
|
T}@T{
|
|
\f[C]*.{{jpe?g}}\f[R]
|
|
T}@T{
|
|
\f[C]/file.jpeg\f[R]
|
|
T}@T{
|
|
\f[C]/file.png\f[R]
|
|
T}
|
|
T{
|
|
T}@T{
|
|
T}@T{
|
|
\f[C]/dir/file.jpg\f[R]
|
|
T}@T{
|
|
\f[C]/dir/file.jpeeg\f[R]
|
|
T}
|
|
T{
|
|
Rooted Regexp
|
|
T}@T{
|
|
\f[C]/{{.*\[rs].jpe?g}}\f[R]
|
|
T}@T{
|
|
\f[C]/file.jpeg\f[R]
|
|
T}@T{
|
|
\f[C]/file.png\f[R]
|
|
T}
|
|
T{
|
|
T}@T{
|
|
T}@T{
|
|
\f[C]/file.jpg\f[R]
|
|
T}@T{
|
|
\f[C]/dir/file.jpg\f[R]
|
|
T}
|
|
.TE
|
|
.SS How filter rules are applied to files
|
|
.PP
|
|
Rclone path/file name filters are made up of one or more of the
|
|
following flags:
|
|
.IP \[bu] 2
|
|
\f[C]--include\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--include-from\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--exclude\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--exclude-from\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--filter\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--filter-from\f[R]
|
|
.PP
|
|
There can be more than one instance of individual flags.
|
|
.PP
|
|
Rclone internally uses a combined list of all the include and exclude
|
|
rules.
|
|
The order in which rules are processed can influence the result of the
|
|
filter.
|
|
.PP
|
|
All flags of the same type are processed together in the order above,
|
|
regardless of what order the different types of flags are included on
|
|
the command line.
|
|
.PP
|
|
Multiple instances of the same flag are processed from left to right
|
|
according to their position in the command line.
|
|
.PP
|
|
To mix up the order of processing includes and excludes use
|
|
\f[C]--filter...\f[R] flags.
|
|
.PP
|
|
Within \f[C]--include-from\f[R], \f[C]--exclude-from\f[R] and
|
|
\f[C]--filter-from\f[R] flags rules are processed from top to bottom of
|
|
the referenced file.
|
|
.PP
|
|
If there is an \f[C]--include\f[R] or \f[C]--include-from\f[R] flag
|
|
specified, rclone implies a \f[C]- **\f[R] rule which it adds to the
|
|
bottom of the internal rule list.
|
|
Specifying a \f[C]+\f[R] rule with a \f[C]--filter...\f[R] flag does not
|
|
imply that rule.
|
|
.PP
|
|
Each path/file name passed through rclone is matched against the
|
|
combined filter list.
|
|
At first match to a rule the path/file name is included or excluded and
|
|
no further filter rules are processed for that path/file.
|
|
.PP
|
|
If rclone does not find a match, after testing against all rules
|
|
(including the implied rule if appropriate), the path/file name is
|
|
included.
|
|
.PP
|
|
Any path/file included at that stage is processed by the rclone command.
|
|
.PP
|
|
\f[C]--files-from\f[R] and \f[C]--files-from-raw\f[R] flags over-ride
|
|
and cannot be combined with other filter options.
|
|
.PP
|
|
To see the internal combined rule list, in regular expression form, for
|
|
a command add the \f[C]--dump filters\f[R] flag.
|
|
Running an rclone command with \f[C]--dump filters\f[R] and
|
|
\f[C]-vv\f[R] flags lists the internal filter elements and shows how
|
|
they are applied to each source path/file.
|
|
There is not currently a means provided to pass regular expression
|
|
filter options into rclone directly though character class filter rules
|
|
contain character classes.
|
|
Go regular expression reference (https://golang.org/pkg/regexp/syntax/)
|
|
.SS How filter rules are applied to directories
|
|
.PP
|
|
Rclone commands are applied to path/file names not directories.
|
|
The entire contents of a directory can be matched to a filter by the
|
|
pattern \f[C]directory/*\f[R] or recursively by \f[C]directory/**\f[R].
|
|
.PP
|
|
Directory filter rules are defined with a closing \f[C]/\f[R] separator.
|
|
.PP
|
|
E.g.
|
|
\f[C]/directory/subdirectory/\f[R] is an rclone directory filter rule.
|
|
.PP
|
|
Rclone commands can use directory filter rules to determine whether they
|
|
recurse into subdirectories.
|
|
This potentially optimises access to a remote by avoiding listing
|
|
unnecessary directories.
|
|
Whether optimisation is desirable depends on the specific filter rules
|
|
and source remote content.
|
|
.PP
|
|
Directory recursion optimisation occurs if either:
|
|
.IP \[bu] 2
|
|
A source remote does not support the rclone \f[C]ListR\f[R] primitive.
|
|
local, sftp, Microsoft OneDrive and WebDav do not support
|
|
\f[C]ListR\f[R].
|
|
Google Drive and most bucket type storage do.
|
|
Full list (https://rclone.org/overview/#optional-features)
|
|
.IP \[bu] 2
|
|
On other remotes (those that support \f[C]ListR\f[R]), if the rclone
|
|
command is not naturally recursive, and provided it is not run with the
|
|
\f[C]--fast-list\f[R] flag.
|
|
\f[C]ls\f[R], \f[C]lsf -R\f[R] and \f[C]size\f[R] are naturally
|
|
recursive but \f[C]sync\f[R], \f[C]copy\f[R] and \f[C]move\f[R] are not.
|
|
.IP \[bu] 2
|
|
Whenever the \f[C]--disable ListR\f[R] flag is applied to an rclone
|
|
command.
|
|
.PP
|
|
Rclone commands imply directory filter rules from path/file filter
|
|
rules.
|
|
To view the directory filter rules rclone has implied for a command
|
|
specify the \f[C]--dump filters\f[R] flag.
|
|
.PP
|
|
E.g.
|
|
for an include rule
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/a/*.jpg
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Rclone implies the directory include rule
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/a/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Directory filter rules specified in an rclone command can limit the
|
|
scope of an rclone command but path/file filters still have to be
|
|
specified.
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone ls remote: --include /directory/\f[R] will not match any
|
|
files.
|
|
Because it is an \f[C]--include\f[R] option the \f[C]--exclude **\f[R]
|
|
rule is implied, and the \f[C]/directory/\f[R] pattern serves only to
|
|
optimise access to the remote by ignoring everything outside of that
|
|
directory.
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone ls remote: --filter-from filter-list.txt\f[R] with a file
|
|
\f[C]filter-list.txt\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
- /dir1/
|
|
- /dir2/
|
|
+ *.pdf
|
|
- **
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
All files in directories \f[C]dir1\f[R] or \f[C]dir2\f[R] or their
|
|
subdirectories are completely excluded from the listing.
|
|
Only files of suffix \f[C]pdf\f[R] in the root of \f[C]remote:\f[R] or
|
|
its subdirectories are listed.
|
|
The \f[C]- **\f[R] rule prevents listing of any path/files not
|
|
previously matched by the rules above.
|
|
.PP
|
|
Option \f[C]exclude-if-present\f[R] creates a directory exclude rule
|
|
based on the presence of a file in a directory and takes precedence over
|
|
other rclone directory filter rules.
|
|
.PP
|
|
When using pattern list syntax, if a pattern item contains either
|
|
\f[C]/\f[R] or \f[C]**\f[R], then rclone will not able to imply a
|
|
directory filter rule from this pattern list.
|
|
.PP
|
|
E.g.
|
|
for an include rule
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{dir1/**,dir2/**}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Rclone will match files below directories \f[C]dir1\f[R] or
|
|
\f[C]dir2\f[R] only, but will not be able to use this filter to exclude
|
|
a directory \f[C]dir3\f[R] from being traversed.
|
|
.PP
|
|
Directory recursion optimisation may affect performance, but normally
|
|
not the result.
|
|
One exception to this is sync operations with option
|
|
\f[C]--create-empty-src-dirs\f[R], where any traversed empty directories
|
|
will be created.
|
|
With the pattern list example \f[C]{dir1/**,dir2/**}\f[R] above, this
|
|
would create an empty directory \f[C]dir3\f[R] on destination (when it
|
|
exists on source).
|
|
Changing the filter to \f[C]{dir1,dir2}/**\f[R], or splitting it into
|
|
two include rules \f[C]--include dir1/** --include dir2/**\f[R], will
|
|
match the same files while also filtering directories, with the result
|
|
that an empty directory \f[C]dir3\f[R] will no longer be created.
|
|
.SS \f[C]--exclude\f[R] - Exclude files matching pattern
|
|
.PP
|
|
Excludes path/file names from an rclone command based on a single
|
|
exclude rule.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order filter flags are processed in.
|
|
.PP
|
|
\f[C]--exclude\f[R] should not be used with \f[C]--include\f[R],
|
|
\f[C]--include-from\f[R], \f[C]--filter\f[R] or \f[C]--filter-from\f[R]
|
|
flags.
|
|
.PP
|
|
\f[C]--exclude\f[R] has no effect when combined with
|
|
\f[C]--files-from\f[R] or \f[C]--files-from-raw\f[R] flags.
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone ls remote: --exclude *.bak\f[R] excludes all .bak files from
|
|
listing.
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone size remote: \[dq]--exclude /dir/**\[dq]\f[R] returns the
|
|
total size of all files on \f[C]remote:\f[R] excluding those in root
|
|
directory \f[C]dir\f[R] and sub directories.
|
|
.PP
|
|
E.g.
|
|
on Microsoft Windows
|
|
\f[C]rclone ls remote: --exclude \[dq]*\[rs][{JP,KR,HK}\[rs]]*\[dq]\f[R]
|
|
lists the files in \f[C]remote:\f[R] with \f[C][JP]\f[R] or
|
|
\f[C][KR]\f[R] or \f[C][HK]\f[R] in their name.
|
|
Quotes prevent the shell from interpreting the \f[C]\[rs]\f[R]
|
|
characters.\f[C]\[rs]\f[R] characters escape the \f[C][\f[R] and
|
|
\f[C]]\f[R] so an rclone filter treats them literally rather than as a
|
|
character-range.
|
|
The \f[C]{\f[R] and \f[C]}\f[R] define an rclone pattern list.
|
|
For other operating systems single quotes are required ie
|
|
\f[C]rclone ls remote: --exclude \[aq]*\[rs][{JP,KR,HK}\[rs]]*\[aq]\f[R]
|
|
.SS \f[C]--exclude-from\f[R] - Read exclude patterns from file
|
|
.PP
|
|
Excludes path/file names from an rclone command based on rules in a
|
|
named file.
|
|
The file contains a list of remarks and pattern rules.
|
|
.PP
|
|
For an example \f[C]exclude-file.txt\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
# a sample exclude rule file
|
|
*.bak
|
|
file2.jpg
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[C]rclone ls remote: --exclude-from exclude-file.txt\f[R] lists the
|
|
files on \f[C]remote:\f[R] except those named \f[C]file2.jpg\f[R] or
|
|
with a suffix \f[C].bak\f[R].
|
|
That is equivalent to
|
|
\f[C]rclone ls remote: --exclude file2.jpg --exclude \[dq]*.bak\[dq]\f[R].
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order filter flags are processed in.
|
|
.PP
|
|
The \f[C]--exclude-from\f[R] flag is useful where multiple exclude
|
|
filter rules are applied to an rclone command.
|
|
.PP
|
|
\f[C]--exclude-from\f[R] should not be used with \f[C]--include\f[R],
|
|
\f[C]--include-from\f[R], \f[C]--filter\f[R] or \f[C]--filter-from\f[R]
|
|
flags.
|
|
.PP
|
|
\f[C]--exclude-from\f[R] has no effect when combined with
|
|
\f[C]--files-from\f[R] or \f[C]--files-from-raw\f[R] flags.
|
|
.PP
|
|
\f[C]--exclude-from\f[R] followed by \f[C]-\f[R] reads filter rules from
|
|
standard input.
|
|
.SS \f[C]--include\f[R] - Include files matching pattern
|
|
.PP
|
|
Adds a single include rule based on path/file names to an rclone
|
|
command.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order filter flags are processed in.
|
|
.PP
|
|
\f[C]--include\f[R] has no effect when combined with
|
|
\f[C]--files-from\f[R] or \f[C]--files-from-raw\f[R] flags.
|
|
.PP
|
|
\f[C]--include\f[R] implies \f[C]--exclude **\f[R] at the end of an
|
|
rclone internal filter list.
|
|
Therefore if you mix \f[C]--include\f[R] and \f[C]--include-from\f[R]
|
|
flags with \f[C]--exclude\f[R], \f[C]--exclude-from\f[R],
|
|
\f[C]--filter\f[R] or \f[C]--filter-from\f[R], you must use include
|
|
rules for all the files you want in the include statement.
|
|
For more flexibility use the \f[C]--filter-from\f[R] flag.
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone ls remote: --include \[dq]*.{png,jpg}\[dq]\f[R] lists the
|
|
files on \f[C]remote:\f[R] with suffix \f[C].png\f[R] and
|
|
\f[C].jpg\f[R].
|
|
All other files are excluded.
|
|
.PP
|
|
E.g.
|
|
multiple rclone copy commands can be combined with \f[C]--include\f[R]
|
|
and a pattern-list.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /vol1/A remote:A
|
|
rclone copy /vol1/B remote:B
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
is equivalent to:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /vol1 remote: --include \[dq]{A,B}/**\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone ls remote:/wheat --include \[dq]??[\[ha][:punct:]]*\[dq]\f[R]
|
|
lists the files \f[C]remote:\f[R] directory \f[C]wheat\f[R] (and
|
|
subdirectories) whose third character is not punctuation.
|
|
This example uses an ASCII character
|
|
class (https://golang.org/pkg/regexp/syntax/).
|
|
.SS \f[C]--include-from\f[R] - Read include patterns from file
|
|
.PP
|
|
Adds path/file names to an rclone command based on rules in a named
|
|
file.
|
|
The file contains a list of remarks and pattern rules.
|
|
.PP
|
|
For an example \f[C]include-file.txt\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
# a sample include rule file
|
|
*.jpg
|
|
file2.avi
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[C]rclone ls remote: --include-from include-file.txt\f[R] lists the
|
|
files on \f[C]remote:\f[R] with name \f[C]file2.avi\f[R] or suffix
|
|
\f[C].jpg\f[R].
|
|
That is equivalent to
|
|
\f[C]rclone ls remote: --include file2.avi --include \[dq]*.jpg\[dq]\f[R].
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order filter flags are processed in.
|
|
.PP
|
|
The \f[C]--include-from\f[R] flag is useful where multiple include
|
|
filter rules are applied to an rclone command.
|
|
.PP
|
|
\f[C]--include-from\f[R] implies \f[C]--exclude **\f[R] at the end of an
|
|
rclone internal filter list.
|
|
Therefore if you mix \f[C]--include\f[R] and \f[C]--include-from\f[R]
|
|
flags with \f[C]--exclude\f[R], \f[C]--exclude-from\f[R],
|
|
\f[C]--filter\f[R] or \f[C]--filter-from\f[R], you must use include
|
|
rules for all the files you want in the include statement.
|
|
For more flexibility use the \f[C]--filter-from\f[R] flag.
|
|
.PP
|
|
\f[C]--exclude-from\f[R] has no effect when combined with
|
|
\f[C]--files-from\f[R] or \f[C]--files-from-raw\f[R] flags.
|
|
.PP
|
|
\f[C]--exclude-from\f[R] followed by \f[C]-\f[R] reads filter rules from
|
|
standard input.
|
|
.SS \f[C]--filter\f[R] - Add a file-filtering rule
|
|
.PP
|
|
Specifies path/file names to an rclone command, based on a single
|
|
include or exclude rule, in \f[C]+\f[R] or \f[C]-\f[R] format.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order filter flags are processed in.
|
|
.PP
|
|
\f[C]--filter +\f[R] differs from \f[C]--include\f[R].
|
|
In the case of \f[C]--include\f[R] rclone implies an
|
|
\f[C]--exclude *\f[R] rule which it adds to the bottom of the internal
|
|
rule list.
|
|
\f[C]--filter...+\f[R] does not imply that rule.
|
|
.PP
|
|
\f[C]--filter\f[R] has no effect when combined with
|
|
\f[C]--files-from\f[R] or \f[C]--files-from-raw\f[R] flags.
|
|
.PP
|
|
\f[C]--filter\f[R] should not be used with \f[C]--include\f[R],
|
|
\f[C]--include-from\f[R], \f[C]--exclude\f[R] or
|
|
\f[C]--exclude-from\f[R] flags.
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone ls remote: --filter \[dq]- *.bak\[dq]\f[R] excludes all
|
|
\f[C].bak\f[R] files from a list of \f[C]remote:\f[R].
|
|
.SS \f[C]--filter-from\f[R] - Read filtering patterns from a file
|
|
.PP
|
|
Adds path/file names to an rclone command based on rules in a named
|
|
file.
|
|
The file contains a list of remarks and pattern rules.
|
|
Include rules start with \f[C]+\f[R] and exclude rules with \f[C]-\f[R].
|
|
\f[C]!\f[R] clears existing rules.
|
|
Rules are processed in the order they are defined.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order filter flags are processed in.
|
|
.PP
|
|
Arrange the order of filter rules with the most restrictive first and
|
|
work down.
|
|
.PP
|
|
E.g.
|
|
for \f[C]filter-file.txt\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
# a sample filter rule file
|
|
- secret*.jpg
|
|
+ *.jpg
|
|
+ *.png
|
|
+ file2.avi
|
|
- /dir/Trash/**
|
|
+ /dir/**
|
|
# exclude everything else
|
|
- *
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[C]rclone ls remote: --filter-from filter-file.txt\f[R] lists the
|
|
path/files on \f[C]remote:\f[R] including all \f[C]jpg\f[R] and
|
|
\f[C]png\f[R] files, excluding any matching \f[C]secret*.jpg\f[R] and
|
|
including \f[C]file2.avi\f[R].
|
|
It also includes everything in the directory \f[C]dir\f[R] at the root
|
|
of \f[C]remote\f[R], except \f[C]remote:dir/Trash\f[R] which it
|
|
excludes.
|
|
Everything else is excluded.
|
|
.PP
|
|
E.g.
|
|
for an alternative \f[C]filter-file.txt\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
- secret*.jpg
|
|
+ *.jpg
|
|
+ *.png
|
|
+ file2.avi
|
|
- *
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Files \f[C]file1.jpg\f[R], \f[C]file3.png\f[R] and \f[C]file2.avi\f[R]
|
|
are listed whilst \f[C]secret17.jpg\f[R] and files without the suffix
|
|
\&.jpg\f[C]or\f[R].png\[ga] are excluded.
|
|
.PP
|
|
E.g.
|
|
for an alternative \f[C]filter-file.txt\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
+ *.jpg
|
|
+ *.gif
|
|
!
|
|
+ 42.doc
|
|
- *
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Only file 42.doc is listed.
|
|
Prior rules are cleared by the \f[C]!\f[R].
|
|
.SS \f[C]--files-from\f[R] - Read list of source-file names
|
|
.PP
|
|
Adds path/files to an rclone command from a list in a named file.
|
|
Rclone processes the path/file names in the order of the list, and no
|
|
others.
|
|
.PP
|
|
Other filter flags (\f[C]--include\f[R], \f[C]--include-from\f[R],
|
|
\f[C]--exclude\f[R], \f[C]--exclude-from\f[R], \f[C]--filter\f[R] and
|
|
\f[C]--filter-from\f[R]) are ignored when \f[C]--files-from\f[R] is
|
|
used.
|
|
.PP
|
|
\f[C]--files-from\f[R] expects a list of files as its input.
|
|
Leading or trailing whitespace is stripped from the input lines.
|
|
Lines starting with \f[C]#\f[R] or \f[C];\f[R] are ignored.
|
|
.PP
|
|
Rclone commands with a \f[C]--files-from\f[R] flag traverse the remote,
|
|
treating the names in \f[C]--files-from\f[R] as a set of filters.
|
|
.PP
|
|
If the \f[C]--no-traverse\f[R] and \f[C]--files-from\f[R] flags are used
|
|
together an rclone command does not traverse the remote.
|
|
Instead it addresses each path/file named in the file individually.
|
|
For each path/file name, that requires typically 1 API call.
|
|
This can be efficient for a short \f[C]--files-from\f[R] list and a
|
|
remote containing many files.
|
|
.PP
|
|
Rclone commands do not error if any names in the \f[C]--files-from\f[R]
|
|
file are missing from the source remote.
|
|
.PP
|
|
The \f[C]--files-from\f[R] flag can be repeated in a single rclone
|
|
command to read path/file names from more than one file.
|
|
The files are read from left to right along the command line.
|
|
.PP
|
|
Paths within the \f[C]--files-from\f[R] file are interpreted as starting
|
|
with the root specified in the rclone command.
|
|
Leading \f[C]/\f[R] separators are ignored.
|
|
See --files-from-raw if you need the input to be processed in a raw
|
|
manner.
|
|
.PP
|
|
E.g.
|
|
for a file \f[C]files-from.txt\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
# comment
|
|
file1.jpg
|
|
subdir/file2.jpg
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[C]rclone copy --files-from files-from.txt /home/me/pics remote:pics\f[R]
|
|
copies the following, if they exist, and only those files.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/me/pics/file1.jpg \[->] remote:pics/file1.jpg
|
|
/home/me/pics/subdir/file2.jpg \[->] remote:pics/subdir/file2.jpg
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
E.g.
|
|
to copy the following files referenced by their absolute paths:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/user1/42
|
|
/home/user1/dir/ford
|
|
/home/user2/prefect
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
First find a common subdirectory - in this case \f[C]/home\f[R] and put
|
|
the remaining files in \f[C]files-from.txt\f[R] with or without leading
|
|
\f[C]/\f[R], e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
user1/42
|
|
user1/dir/ford
|
|
user2/prefect
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Then copy these to a remote:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --files-from files-from.txt /home remote:backup
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The three files are transferred as follows:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/user1/42 \[->] remote:backup/user1/important
|
|
/home/user1/dir/ford \[->] remote:backup/user1/dir/file
|
|
/home/user2/prefect \[->] remote:backup/user2/stuff
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Alternatively if \f[C]/\f[R] is chosen as root \f[C]files-from.txt\f[R]
|
|
will be:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/user1/42
|
|
/home/user1/dir/ford
|
|
/home/user2/prefect
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The copy command will be:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --files-from files-from.txt / remote:backup
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Then there will be an extra \f[C]home\f[R] directory on the remote:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/user1/42 \[->] remote:backup/home/user1/42
|
|
/home/user1/dir/ford \[->] remote:backup/home/user1/dir/ford
|
|
/home/user2/prefect \[->] remote:backup/home/user2/prefect
|
|
\f[R]
|
|
.fi
|
|
.SS \f[C]--files-from-raw\f[R] - Read list of source-file names without any processing
|
|
.PP
|
|
This flag is the same as \f[C]--files-from\f[R] except that input is
|
|
read in a raw manner.
|
|
Lines with leading / trailing whitespace, and lines starting with
|
|
\f[C];\f[R] or \f[C]#\f[R] are read without any processing.
|
|
rclone lsf (https://rclone.org/commands/rclone_lsf/) has a compatible
|
|
format that can be used to export file lists from remotes for input to
|
|
\f[C]--files-from-raw\f[R].
|
|
.SS \f[C]--ignore-case\f[R] - make searches case insensitive
|
|
.PP
|
|
By default, rclone filter patterns are case sensitive.
|
|
The \f[C]--ignore-case\f[R] flag makes all of the filters patterns on
|
|
the command line case insensitive.
|
|
.PP
|
|
E.g.
|
|
\f[C]--include \[dq]zaphod.txt\[dq]\f[R] does not match a file
|
|
\f[C]Zaphod.txt\f[R].
|
|
With \f[C]--ignore-case\f[R] a match is made.
|
|
.SS Quoting shell metacharacters
|
|
.PP
|
|
Rclone commands with filter patterns containing shell metacharacters may
|
|
not as work as expected in your shell and may require quoting.
|
|
.PP
|
|
E.g.
|
|
linux, OSX (\f[C]*\f[R] metacharacter)
|
|
.IP \[bu] 2
|
|
\f[C]--include \[rs]*.jpg\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--include \[aq]*.jpg\[aq]\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--include=\[aq]*.jpg\[aq]\f[R]
|
|
.PP
|
|
Microsoft Windows expansion is done by the command, not shell, so
|
|
\f[C]--include *.jpg\f[R] does not require quoting.
|
|
.PP
|
|
If the rclone error
|
|
\f[C]Command .... needs .... arguments maximum: you provided .... non flag arguments:\f[R]
|
|
is encountered, the cause is commonly spaces within the name of a remote
|
|
or flag value.
|
|
The fix then is to quote values containing spaces.
|
|
.SS Other filters
|
|
.SS \f[C]--min-size\f[R] - Don\[aq]t transfer any file smaller than this
|
|
.PP
|
|
Controls the minimum size file within the scope of an rclone command.
|
|
Default units are \f[C]KiB\f[R] but abbreviations \f[C]K\f[R],
|
|
\f[C]M\f[R], \f[C]G\f[R], \f[C]T\f[R] or \f[C]P\f[R] are valid.
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone ls remote: --min-size 50k\f[R] lists files on
|
|
\f[C]remote:\f[R] of 50 KiB size or larger.
|
|
.SS \f[C]--max-size\f[R] - Don\[aq]t transfer any file larger than this
|
|
.PP
|
|
Controls the maximum size file within the scope of an rclone command.
|
|
Default units are \f[C]KiB\f[R] but abbreviations \f[C]K\f[R],
|
|
\f[C]M\f[R], \f[C]G\f[R], \f[C]T\f[R] or \f[C]P\f[R] are valid.
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone ls remote: --max-size 1G\f[R] lists files on
|
|
\f[C]remote:\f[R] of 1 GiB size or smaller.
|
|
.SS \f[C]--max-age\f[R] - Don\[aq]t transfer any file older than this
|
|
.PP
|
|
Controls the maximum age of files within the scope of an rclone command.
|
|
Default units are seconds or the following abbreviations are valid:
|
|
.IP \[bu] 2
|
|
\f[C]ms\f[R] - Milliseconds
|
|
.IP \[bu] 2
|
|
\f[C]s\f[R] - Seconds
|
|
.IP \[bu] 2
|
|
\f[C]m\f[R] - Minutes
|
|
.IP \[bu] 2
|
|
\f[C]h\f[R] - Hours
|
|
.IP \[bu] 2
|
|
\f[C]d\f[R] - Days
|
|
.IP \[bu] 2
|
|
\f[C]w\f[R] - Weeks
|
|
.IP \[bu] 2
|
|
\f[C]M\f[R] - Months
|
|
.IP \[bu] 2
|
|
\f[C]y\f[R] - Years
|
|
.PP
|
|
\f[C]--max-age\f[R] can also be specified as an absolute time in the
|
|
following formats:
|
|
.IP \[bu] 2
|
|
RFC3339 - e.g.
|
|
\f[C]2006-01-02T15:04:05Z\f[R] or \f[C]2006-01-02T15:04:05+07:00\f[R]
|
|
.IP \[bu] 2
|
|
ISO8601 Date and time, local timezone - \f[C]2006-01-02T15:04:05\f[R]
|
|
.IP \[bu] 2
|
|
ISO8601 Date and time, local timezone - \f[C]2006-01-02 15:04:05\f[R]
|
|
.IP \[bu] 2
|
|
ISO8601 Date - \f[C]2006-01-02\f[R] (YYYY-MM-DD)
|
|
.PP
|
|
\f[C]--max-age\f[R] applies only to files and not to directories.
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone ls remote: --max-age 2d\f[R] lists files on
|
|
\f[C]remote:\f[R] of 2 days old or less.
|
|
.SS \f[C]--min-age\f[R] - Don\[aq]t transfer any file younger than this
|
|
.PP
|
|
Controls the minimum age of files within the scope of an rclone command.
|
|
(see \f[C]--max-age\f[R] for valid formats)
|
|
.PP
|
|
\f[C]--min-age\f[R] applies only to files and not to directories.
|
|
.PP
|
|
E.g.
|
|
\f[C]rclone ls remote: --min-age 2d\f[R] lists files on
|
|
\f[C]remote:\f[R] of 2 days old or more.
|
|
.SS Other flags
|
|
.SS \f[C]--delete-excluded\f[R] - Delete files on dest excluded from sync
|
|
.PP
|
|
\f[B]Important\f[R] this flag is dangerous to your data - use with
|
|
\f[C]--dry-run\f[R] and \f[C]-v\f[R] first.
|
|
.PP
|
|
In conjunction with \f[C]rclone sync\f[R], \f[C]--delete-excluded\f[R]
|
|
deletes any files on the destination which are excluded from the
|
|
command.
|
|
.PP
|
|
E.g.
|
|
the scope of \f[C]rclone sync -i A: B:\f[R] can be restricted:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone --min-size 50k --delete-excluded sync A: B:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
All files on \f[C]B:\f[R] which are less than 50 KiB are deleted because
|
|
they are excluded from the rclone sync command.
|
|
.SS \f[C]--dump filters\f[R] - dump the filters to the output
|
|
.PP
|
|
Dumps the defined filters to standard output in regular expression
|
|
format.
|
|
.PP
|
|
Useful for debugging.
|
|
.SS Exclude directory based on a file
|
|
.PP
|
|
The \f[C]--exclude-if-present\f[R] flag controls whether a directory is
|
|
within the scope of an rclone command based on the presence of a named
|
|
file within it.
|
|
.PP
|
|
This flag has a priority over other filter flags.
|
|
.PP
|
|
E.g.
|
|
for the following directory structure:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
dir1/file1
|
|
dir1/dir2/file2
|
|
dir1/dir2/dir3/file3
|
|
dir1/dir2/dir3/.ignore
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The command \f[C]rclone ls --exclude-if-present .ignore dir1\f[R] does
|
|
not list \f[C]dir3\f[R], \f[C]file3\f[R] or \f[C].ignore\f[R].
|
|
.PP
|
|
\f[C]--exclude-if-present\f[R] can only be used once in an rclone
|
|
command.
|
|
.SS Common pitfalls
|
|
.PP
|
|
The most frequent filter support issues on the rclone
|
|
forum (https://forum.rclone.org/) are:
|
|
.IP \[bu] 2
|
|
Not using paths relative to the root of the remote
|
|
.IP \[bu] 2
|
|
Not using \f[C]/\f[R] to match from the root of a remote
|
|
.IP \[bu] 2
|
|
Not using \f[C]**\f[R] to match the contents of a directory
|
|
.SH GUI (Experimental)
|
|
.PP
|
|
Rclone can serve a web based GUI (graphical user interface).
|
|
This is somewhat experimental at the moment so things may be subject to
|
|
change.
|
|
.PP
|
|
Run this command in a terminal and rclone will download and then display
|
|
the GUI in a web browser.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rcd --rc-web-gui
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will produce logs like this and rclone needs to continue to run to
|
|
serve the GUI:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
2019/08/25 11:40:14 NOTICE: A new release for gui is present at https://github.com/rclone/rclone-webui-react/releases/download/v0.0.6/currentbuild.zip
|
|
2019/08/25 11:40:14 NOTICE: Downloading webgui binary. Please wait. [Size: 3813937, Path : /home/USER/.cache/rclone/webgui/v0.0.6.zip]
|
|
2019/08/25 11:40:16 NOTICE: Unzipping
|
|
2019/08/25 11:40:16 NOTICE: Serving remote control on http://127.0.0.1:5572/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This assumes you are running rclone locally on your machine.
|
|
It is possible to separate the rclone and the GUI - see below for
|
|
details.
|
|
.PP
|
|
If you wish to check for updates then you can add
|
|
\f[C]--rc-web-gui-update\f[R] to the command line.
|
|
.PP
|
|
If you find your GUI broken, you may force it to update by add
|
|
\f[C]--rc-web-gui-force-update\f[R].
|
|
.PP
|
|
By default, rclone will open your browser.
|
|
Add \f[C]--rc-web-gui-no-open-browser\f[R] to disable this feature.
|
|
.SS Using the GUI
|
|
.PP
|
|
Once the GUI opens, you will be looking at the dashboard which has an
|
|
overall overview.
|
|
.PP
|
|
On the left hand side you will see a series of view buttons you can
|
|
click on:
|
|
.IP \[bu] 2
|
|
Dashboard - main overview
|
|
.IP \[bu] 2
|
|
Configs - examine and create new configurations
|
|
.IP \[bu] 2
|
|
Explorer - view, download and upload files to the cloud storage systems
|
|
.IP \[bu] 2
|
|
Backend - view or alter the backend config
|
|
.IP \[bu] 2
|
|
Log out
|
|
.PP
|
|
(More docs and walkthrough video to come!)
|
|
.SS How it works
|
|
.PP
|
|
When you run the \f[C]rclone rcd --rc-web-gui\f[R] this is what happens
|
|
.IP \[bu] 2
|
|
Rclone starts but only runs the remote control API (\[dq]rc\[dq]).
|
|
.IP \[bu] 2
|
|
The API is bound to localhost with an auto-generated username and
|
|
password.
|
|
.IP \[bu] 2
|
|
If the API bundle is missing then rclone will download it.
|
|
.IP \[bu] 2
|
|
rclone will start serving the files from the API bundle over the same
|
|
port as the API
|
|
.IP \[bu] 2
|
|
rclone will open the browser with a \f[C]login_token\f[R] so it can log
|
|
straight in.
|
|
.SS Advanced use
|
|
.PP
|
|
The \f[C]rclone rcd\f[R] may use any of the flags documented on the rc
|
|
page (https://rclone.org/rc/#supported-parameters).
|
|
.PP
|
|
The flag \f[C]--rc-web-gui\f[R] is shorthand for
|
|
.IP \[bu] 2
|
|
Download the web GUI if necessary
|
|
.IP \[bu] 2
|
|
Check we are using some authentication
|
|
.IP \[bu] 2
|
|
\f[C]--rc-user gui\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--rc-pass <random password>\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--rc-serve\f[R]
|
|
.PP
|
|
These flags can be overridden as desired.
|
|
.PP
|
|
See also the rclone rcd
|
|
documentation (https://rclone.org/commands/rclone_rcd/).
|
|
.SS Example: Running a public GUI
|
|
.PP
|
|
For example the GUI could be served on a public port over SSL using an
|
|
htpasswd file using the following flags:
|
|
.IP \[bu] 2
|
|
\f[C]--rc-web-gui\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--rc-addr :443\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--rc-htpasswd /path/to/htpasswd\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--rc-cert /path/to/ssl.crt\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--rc-key /path/to/ssl.key\f[R]
|
|
.SS Example: Running a GUI behind a proxy
|
|
.PP
|
|
If you want to run the GUI behind a proxy at \f[C]/rclone\f[R] you could
|
|
use these flags:
|
|
.IP \[bu] 2
|
|
\f[C]--rc-web-gui\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--rc-baseurl rclone\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--rc-htpasswd /path/to/htpasswd\f[R]
|
|
.PP
|
|
Or instead of htpasswd if you just want a single user and password:
|
|
.IP \[bu] 2
|
|
\f[C]--rc-user me\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--rc-pass mypassword\f[R]
|
|
.SS Project
|
|
.PP
|
|
The GUI is being developed in the: rclone/rclone-webui-react
|
|
repository (https://github.com/rclone/rclone-webui-react).
|
|
.PP
|
|
Bug reports and contributions are very welcome :-)
|
|
.PP
|
|
If you have questions then please ask them on the rclone
|
|
forum (https://forum.rclone.org/).
|
|
.SH Remote controlling rclone with its API
|
|
.PP
|
|
If rclone is run with the \f[C]--rc\f[R] flag then it starts an HTTP
|
|
server which can be used to remote control rclone using its API.
|
|
.PP
|
|
You can either use the rclone rc command to access the API or use HTTP
|
|
directly.
|
|
.PP
|
|
If you just want to run a remote control then see the rcd
|
|
command (https://rclone.org/commands/rclone_rcd/).
|
|
.SS Supported parameters
|
|
.SS --rc
|
|
.PP
|
|
Flag to start the http server listen on remote requests
|
|
.SS --rc-addr=IP
|
|
.PP
|
|
IPaddress:Port or :Port to bind server to.
|
|
(default \[dq]localhost:5572\[dq])
|
|
.SS --rc-cert=KEY
|
|
.PP
|
|
SSL PEM key (concatenation of certificate and CA certificate)
|
|
.SS --rc-client-ca=PATH
|
|
.PP
|
|
Client certificate authority to verify clients with
|
|
.SS --rc-htpasswd=PATH
|
|
.PP
|
|
htpasswd file - if not provided no authentication is done
|
|
.SS --rc-key=PATH
|
|
.PP
|
|
SSL PEM Private key
|
|
.SS --rc-max-header-bytes=VALUE
|
|
.PP
|
|
Maximum size of request header (default 4096)
|
|
.SS --rc-user=VALUE
|
|
.PP
|
|
User name for authentication.
|
|
.SS --rc-pass=VALUE
|
|
.PP
|
|
Password for authentication.
|
|
.SS --rc-realm=VALUE
|
|
.PP
|
|
Realm for authentication (default \[dq]rclone\[dq])
|
|
.SS --rc-server-read-timeout=DURATION
|
|
.PP
|
|
Timeout for server reading data (default 1h0m0s)
|
|
.SS --rc-server-write-timeout=DURATION
|
|
.PP
|
|
Timeout for server writing data (default 1h0m0s)
|
|
.SS --rc-serve
|
|
.PP
|
|
Enable the serving of remote objects via the HTTP interface.
|
|
This means objects will be accessible at http://127.0.0.1:5572/ by
|
|
default, so you can browse to http://127.0.0.1:5572/ or
|
|
http://127.0.0.1:5572/* to see a listing of the remotes.
|
|
Objects may be requested from remotes using this syntax
|
|
http://127.0.0.1:5572/[remote:path]/path/to/object
|
|
.PP
|
|
Default Off.
|
|
.SS --rc-files /path/to/directory
|
|
.PP
|
|
Path to local files to serve on the HTTP server.
|
|
.PP
|
|
If this is set then rclone will serve the files in that directory.
|
|
It will also open the root in the web browser if specified.
|
|
This is for implementing browser based GUIs for rclone functions.
|
|
.PP
|
|
If \f[C]--rc-user\f[R] or \f[C]--rc-pass\f[R] is set then the URL that
|
|
is opened will have the authorization in the URL in the
|
|
\f[C]http://user:pass\[at]localhost/\f[R] style.
|
|
.PP
|
|
Default Off.
|
|
.SS --rc-enable-metrics
|
|
.PP
|
|
Enable OpenMetrics/Prometheus compatible endpoint at \f[C]/metrics\f[R].
|
|
.PP
|
|
Default Off.
|
|
.SS --rc-web-gui
|
|
.PP
|
|
Set this flag to serve the default web gui on the same port as rclone.
|
|
.PP
|
|
Default Off.
|
|
.SS --rc-allow-origin
|
|
.PP
|
|
Set the allowed Access-Control-Allow-Origin for rc requests.
|
|
.PP
|
|
Can be used with --rc-web-gui if the rclone is running on different IP
|
|
than the web-gui.
|
|
.PP
|
|
Default is IP address on which rc is running.
|
|
.SS --rc-web-fetch-url
|
|
.PP
|
|
Set the URL to fetch the rclone-web-gui files from.
|
|
.PP
|
|
Default
|
|
https://api.github.com/repos/rclone/rclone-webui-react/releases/latest.
|
|
.SS --rc-web-gui-update
|
|
.PP
|
|
Set this flag to check and update rclone-webui-react from the
|
|
rc-web-fetch-url.
|
|
.PP
|
|
Default Off.
|
|
.SS --rc-web-gui-force-update
|
|
.PP
|
|
Set this flag to force update rclone-webui-react from the
|
|
rc-web-fetch-url.
|
|
.PP
|
|
Default Off.
|
|
.SS --rc-web-gui-no-open-browser
|
|
.PP
|
|
Set this flag to disable opening browser automatically when using
|
|
web-gui.
|
|
.PP
|
|
Default Off.
|
|
.SS --rc-job-expire-duration=DURATION
|
|
.PP
|
|
Expire finished async jobs older than DURATION (default 60s).
|
|
.SS --rc-job-expire-interval=DURATION
|
|
.PP
|
|
Interval duration to check for expired async jobs (default 10s).
|
|
.SS --rc-no-auth
|
|
.PP
|
|
By default rclone will require authorisation to have been set up on the
|
|
rc interface in order to use any methods which access any rclone
|
|
remotes.
|
|
Eg \f[C]operations/list\f[R] is denied as it involved creating a remote
|
|
as is \f[C]sync/copy\f[R].
|
|
.PP
|
|
If this is set then no authorisation will be required on the server to
|
|
use these methods.
|
|
The alternative is to use \f[C]--rc-user\f[R] and \f[C]--rc-pass\f[R]
|
|
and use these credentials in the request.
|
|
.PP
|
|
Default Off.
|
|
.SS Accessing the remote control via the rclone rc command
|
|
.PP
|
|
Rclone itself implements the remote control protocol in its
|
|
\f[C]rclone rc\f[R] command.
|
|
.PP
|
|
You can use it like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone rc rc/noop param1=one param2=two
|
|
{
|
|
\[dq]param1\[dq]: \[dq]one\[dq],
|
|
\[dq]param2\[dq]: \[dq]two\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Run \f[C]rclone rc\f[R] on its own to see the help for the installed
|
|
remote control commands.
|
|
.SS JSON input
|
|
.PP
|
|
\f[C]rclone rc\f[R] also supports a \f[C]--json\f[R] flag which can be
|
|
used to send more complicated input parameters.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone rc --json \[aq]{ \[dq]p1\[dq]: [1,\[dq]2\[dq],null,4], \[dq]p2\[dq]: { \[dq]a\[dq]:1, \[dq]b\[dq]:2 } }\[aq] rc/noop
|
|
{
|
|
\[dq]p1\[dq]: [
|
|
1,
|
|
\[dq]2\[dq],
|
|
null,
|
|
4
|
|
],
|
|
\[dq]p2\[dq]: {
|
|
\[dq]a\[dq]: 1,
|
|
\[dq]b\[dq]: 2
|
|
}
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If the parameter being passed is an object then it can be passed as a
|
|
JSON string rather than using the \f[C]--json\f[R] flag which simplifies
|
|
the command line.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc operations/list fs=/tmp remote=test opt=\[aq]{\[dq]showHash\[dq]: true}\[aq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Rather than
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc operations/list --json \[aq]{\[dq]fs\[dq]: \[dq]/tmp\[dq], \[dq]remote\[dq]: \[dq]test\[dq], \[dq]opt\[dq]: {\[dq]showHash\[dq]: true}}\[aq]
|
|
\f[R]
|
|
.fi
|
|
.SS Special parameters
|
|
.PP
|
|
The rc interface supports some special parameters which apply to
|
|
\f[B]all\f[R] commands.
|
|
These start with \f[C]_\f[R] to show they are different.
|
|
.SS Running asynchronous jobs with _async = true
|
|
.PP
|
|
Each rc call is classified as a job and it is assigned its own id.
|
|
By default jobs are executed immediately as they are created or
|
|
synchronously.
|
|
.PP
|
|
If \f[C]_async\f[R] has a true value when supplied to an rc call then it
|
|
will return immediately with a job id and the task will be run in the
|
|
background.
|
|
The \f[C]job/status\f[R] call can be used to get information of the
|
|
background job.
|
|
The job can be queried for up to 1 minute after it has finished.
|
|
.PP
|
|
It is recommended that potentially long running jobs, e.g.
|
|
\f[C]sync/sync\f[R], \f[C]sync/copy\f[R], \f[C]sync/move\f[R],
|
|
\f[C]operations/purge\f[R] are run with the \f[C]_async\f[R] flag to
|
|
avoid any potential problems with the HTTP request and response timing
|
|
out.
|
|
.PP
|
|
Starting a job with the \f[C]_async\f[R] flag:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone rc --json \[aq]{ \[dq]p1\[dq]: [1,\[dq]2\[dq],null,4], \[dq]p2\[dq]: { \[dq]a\[dq]:1, \[dq]b\[dq]:2 }, \[dq]_async\[dq]: true }\[aq] rc/noop
|
|
{
|
|
\[dq]jobid\[dq]: 2
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Query the status to see if the job has finished.
|
|
For more information on the meaning of these return parameters see the
|
|
\f[C]job/status\f[R] call.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone rc --json \[aq]{ \[dq]jobid\[dq]:2 }\[aq] job/status
|
|
{
|
|
\[dq]duration\[dq]: 0.000124163,
|
|
\[dq]endTime\[dq]: \[dq]2018-10-27T11:38:07.911245881+01:00\[dq],
|
|
\[dq]error\[dq]: \[dq]\[dq],
|
|
\[dq]finished\[dq]: true,
|
|
\[dq]id\[dq]: 2,
|
|
\[dq]output\[dq]: {
|
|
\[dq]_async\[dq]: true,
|
|
\[dq]p1\[dq]: [
|
|
1,
|
|
\[dq]2\[dq],
|
|
null,
|
|
4
|
|
],
|
|
\[dq]p2\[dq]: {
|
|
\[dq]a\[dq]: 1,
|
|
\[dq]b\[dq]: 2
|
|
}
|
|
},
|
|
\[dq]startTime\[dq]: \[dq]2018-10-27T11:38:07.911121728+01:00\[dq],
|
|
\[dq]success\[dq]: true
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[C]job/list\f[R] can be used to show the running or recently completed
|
|
jobs
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone rc job/list
|
|
{
|
|
\[dq]jobids\[dq]: [
|
|
2
|
|
]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.SS Setting config flags with _config
|
|
.PP
|
|
If you wish to set config (the equivalent of the global flags) for the
|
|
duration of an rc call only then pass in the \f[C]_config\f[R]
|
|
parameter.
|
|
.PP
|
|
This should be in the same format as the \f[C]config\f[R] key returned
|
|
by options/get.
|
|
.PP
|
|
For example, if you wished to run a sync with the \f[C]--checksum\f[R]
|
|
parameter, you would pass this parameter in your JSON blob.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\[dq]_config\[dq]:{\[dq]CheckSum\[dq]: true}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If using \f[C]rclone rc\f[R] this could be passed as
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc operations/sync ... _config=\[aq]{\[dq]CheckSum\[dq]: true}\[aq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Any config parameters you don\[aq]t set will inherit the global defaults
|
|
which were set with command line flags or environment variables.
|
|
.PP
|
|
Note that it is possible to set some values as strings or integers - see
|
|
data types for more info.
|
|
Here is an example setting the equivalent of \f[C]--buffer-size\f[R] in
|
|
string or integer format.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\[dq]_config\[dq]:{\[dq]BufferSize\[dq]: \[dq]42M\[dq]}
|
|
\[dq]_config\[dq]:{\[dq]BufferSize\[dq]: 44040192}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you wish to check the \f[C]_config\f[R] assignment has worked
|
|
properly then calling \f[C]options/local\f[R] will show what the value
|
|
got set to.
|
|
.SS Setting filter flags with _filter
|
|
.PP
|
|
If you wish to set filters for the duration of an rc call only then pass
|
|
in the \f[C]_filter\f[R] parameter.
|
|
.PP
|
|
This should be in the same format as the \f[C]filter\f[R] key returned
|
|
by options/get.
|
|
.PP
|
|
For example, if you wished to run a sync with these flags
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--max-size 1M --max-age 42s --include \[dq]a\[dq] --include \[dq]b\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
you would pass this parameter in your JSON blob.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\[dq]_filter\[dq]:{\[dq]MaxSize\[dq]:\[dq]1M\[dq], \[dq]IncludeRule\[dq]:[\[dq]a\[dq],\[dq]b\[dq]], \[dq]MaxAge\[dq]:\[dq]42s\[dq]}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If using \f[C]rclone rc\f[R] this could be passed as
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc ... _filter=\[aq]{\[dq]MaxSize\[dq]:\[dq]1M\[dq], \[dq]IncludeRule\[dq]:[\[dq]a\[dq],\[dq]b\[dq]], \[dq]MaxAge\[dq]:\[dq]42s\[dq]}\[aq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Any filter parameters you don\[aq]t set will inherit the global defaults
|
|
which were set with command line flags or environment variables.
|
|
.PP
|
|
Note that it is possible to set some values as strings or integers - see
|
|
data types for more info.
|
|
Here is an example setting the equivalent of \f[C]--buffer-size\f[R] in
|
|
string or integer format.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\[dq]_filter\[dq]:{\[dq]MinSize\[dq]: \[dq]42M\[dq]}
|
|
\[dq]_filter\[dq]:{\[dq]MinSize\[dq]: 44040192}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you wish to check the \f[C]_filter\f[R] assignment has worked
|
|
properly then calling \f[C]options/local\f[R] will show what the value
|
|
got set to.
|
|
.SS Assigning operations to groups with _group = value
|
|
.PP
|
|
Each rc call has its own stats group for tracking its metrics.
|
|
By default grouping is done by the composite group name from prefix
|
|
\f[C]job/\f[R] and id of the job like so \f[C]job/1\f[R].
|
|
.PP
|
|
If \f[C]_group\f[R] has a value then stats for that request will be
|
|
grouped under that value.
|
|
This allows caller to group stats under their own name.
|
|
.PP
|
|
Stats for specific group can be accessed by passing \f[C]group\f[R] to
|
|
\f[C]core/stats\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone rc --json \[aq]{ \[dq]group\[dq]: \[dq]job/1\[dq] }\[aq] core/stats
|
|
{
|
|
\[dq]speed\[dq]: 12345
|
|
...
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.SS Data types
|
|
.PP
|
|
When the API returns types, these will mostly be straight forward
|
|
integer, string or boolean types.
|
|
.PP
|
|
However some of the types returned by the options/get call and taken by
|
|
the options/set calls as well as the \f[C]vfsOpt\f[R],
|
|
\f[C]mountOpt\f[R] and the \f[C]_config\f[R] parameters.
|
|
.IP \[bu] 2
|
|
\f[C]Duration\f[R] - these are returned as an integer duration in
|
|
nanoseconds.
|
|
They may be set as an integer, or they may be set with time string, eg
|
|
\[dq]5s\[dq].
|
|
See the options section (https://rclone.org/docs/#options) for more
|
|
info.
|
|
.IP \[bu] 2
|
|
\f[C]Size\f[R] - these are returned as an integer number of bytes.
|
|
They may be set as an integer or they may be set with a size suffix
|
|
string, eg \[dq]10M\[dq].
|
|
See the options section (https://rclone.org/docs/#options) for more
|
|
info.
|
|
.IP \[bu] 2
|
|
Enumerated type (such as \f[C]CutoffMode\f[R], \f[C]DumpFlags\f[R],
|
|
\f[C]LogLevel\f[R], \f[C]VfsCacheMode\f[R] - these will be returned as
|
|
an integer and may be set as an integer but more conveniently they can
|
|
be set as a string, eg \[dq]HARD\[dq] for \f[C]CutoffMode\f[R] or
|
|
\f[C]DEBUG\f[R] for \f[C]LogLevel\f[R].
|
|
.IP \[bu] 2
|
|
\f[C]BandwidthSpec\f[R] - this will be set and returned as a string, eg
|
|
\[dq]1M\[dq].
|
|
.SS Specifying remotes to work on
|
|
.PP
|
|
Remotes are specified with the \f[C]fs=\f[R], \f[C]srcFs=\f[R],
|
|
\f[C]dstFs=\f[R] parameters depending on the command being used.
|
|
.PP
|
|
The parameters can be a string as per the rest of rclone, eg
|
|
\f[C]s3:bucket/path\f[R] or \f[C]:sftp:/my/dir\f[R].
|
|
They can also be specified as JSON blobs.
|
|
.PP
|
|
If specifyng a JSON blob it should be a object mapping strings to
|
|
strings.
|
|
These values will be used to configure the remote.
|
|
There are 3 special values which may be set:
|
|
.IP \[bu] 2
|
|
\f[C]type\f[R] - set to \f[C]type\f[R] to specify a remote called
|
|
\f[C]:type:\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]_name\f[R] - set to \f[C]name\f[R] to specify a remote called
|
|
\f[C]name:\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]_root\f[R] - sets the root of the remote - may be empty
|
|
.PP
|
|
One of \f[C]_name\f[R] or \f[C]type\f[R] should normally be set.
|
|
If the \f[C]local\f[R] backend is desired then \f[C]type\f[R] should be
|
|
set to \f[C]local\f[R].
|
|
If \f[C]_root\f[R] isn\[aq]t specified then it defaults to the root of
|
|
the remote.
|
|
.PP
|
|
For example this JSON is equivalent to \f[C]remote:/tmp\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]_name\[dq]: \[dq]remote\[dq],
|
|
\[dq]_path\[dq]: \[dq]/tmp\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And this is equivalent to
|
|
\f[C]:sftp,host=\[aq]example.com\[aq]:/tmp\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]type\[dq]: \[dq]sftp\[dq],
|
|
\[dq]host\[dq]: \[dq]example.com\[dq],
|
|
\[dq]_path\[dq]: \[dq]/tmp\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And this is equivalent to \f[C]/tmp/dir\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
type = \[dq]local\[dq],
|
|
_ path = \[dq]/tmp/dir\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.SS Supported commands
|
|
.SS backend/command: Runs a backend command.
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
command - a string with the command name
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.IP \[bu] 2
|
|
arg - a list of arguments for the backend command
|
|
.IP \[bu] 2
|
|
opt - a map of string to string of options
|
|
.PP
|
|
Returns:
|
|
.IP \[bu] 2
|
|
result - result from the backend command
|
|
.PP
|
|
Example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc backend/command command=noop fs=. -o echo=yes -o blue -a path1 -a path2
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Returns
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]result\[dq]: {
|
|
\[dq]arg\[dq]: [
|
|
\[dq]path1\[dq],
|
|
\[dq]path2\[dq]
|
|
],
|
|
\[dq]name\[dq]: \[dq]noop\[dq],
|
|
\[dq]opt\[dq]: {
|
|
\[dq]blue\[dq]: \[dq]\[dq],
|
|
\[dq]echo\[dq]: \[dq]yes\[dq]
|
|
}
|
|
}
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that this is the direct equivalent of using this \[dq]backend\[dq]
|
|
command:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend noop . -o echo=yes -o blue path1 path2
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that arguments must be preceded by the \[dq]-a\[dq] flag
|
|
.PP
|
|
See the backend (https://rclone.org/commands/rclone_backend/) command
|
|
for more information.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS cache/expire: Purge a remote from cache
|
|
.PP
|
|
Purge a remote from the cache backend.
|
|
Supports either a directory or a file.
|
|
Params: - remote = path to remote (required) - withData = true/false to
|
|
delete cached data (chunks) as well (optional)
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc cache/expire remote=path/to/sub/folder/
|
|
rclone rc cache/expire remote=/ withData=true
|
|
\f[R]
|
|
.fi
|
|
.SS cache/fetch: Fetch file chunks
|
|
.PP
|
|
Ensure the specified file chunks are cached on disk.
|
|
.PP
|
|
The chunks= parameter specifies the file chunks to check.
|
|
It takes a comma separated list of array slice indices.
|
|
The slice indices are similar to Python slices: start[:end]
|
|
.PP
|
|
start is the 0 based chunk number from the beginning of the file to
|
|
fetch inclusive.
|
|
end is 0 based chunk number from the beginning of the file to fetch
|
|
exclusive.
|
|
Both values can be negative, in which case they count from the back of
|
|
the file.
|
|
The value \[dq]-5:\[dq] represents the last 5 chunks of a file.
|
|
.PP
|
|
Some valid examples are: \[dq]:5,-5:\[dq] -> the first and last five
|
|
chunks \[dq]0,-2\[dq] -> the first and the second last chunk
|
|
\[dq]0:10\[dq] -> the first ten chunks
|
|
.PP
|
|
Any parameter with a key that starts with \[dq]file\[dq] can be used to
|
|
specify files to fetch, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc cache/fetch chunks=0 file=hello file2=home/goodbye
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
File names will automatically be encrypted when the a crypt remote is
|
|
used on top of the cache.
|
|
.SS cache/stats: Get cache stats
|
|
.PP
|
|
Show statistics for the cache remote.
|
|
.SS config/create: create the config for a remote.
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
name - name of remote
|
|
.IP \[bu] 2
|
|
parameters - a map of { \[dq]key\[dq]: \[dq]value\[dq] } pairs
|
|
.IP \[bu] 2
|
|
type - type of the new remote
|
|
.IP \[bu] 2
|
|
opt - a dictionary of options to control the configuration
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
obscure - declare passwords are plain and need obscuring
|
|
.IP \[bu] 2
|
|
noObscure - declare passwords are already obscured and don\[aq]t need
|
|
obscuring
|
|
.IP \[bu] 2
|
|
nonInteractive - don\[aq]t interact with a user, return questions
|
|
.IP \[bu] 2
|
|
continue - continue the config process with an answer
|
|
.IP \[bu] 2
|
|
all - ask all the config questions not just the post config ones
|
|
.IP \[bu] 2
|
|
state - state to restart with - used with continue
|
|
.IP \[bu] 2
|
|
result - result to restart with - used with continue
|
|
.RE
|
|
.PP
|
|
See the config create
|
|
command (https://rclone.org/commands/rclone_config_create/) command for
|
|
more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS config/delete: Delete a remote in the config file.
|
|
.PP
|
|
Parameters:
|
|
.IP \[bu] 2
|
|
name - name of remote to delete
|
|
.PP
|
|
See the config delete
|
|
command (https://rclone.org/commands/rclone_config_delete/) command for
|
|
more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS config/dump: Dumps the config file.
|
|
.PP
|
|
Returns a JSON object: - key: value
|
|
.PP
|
|
Where keys are remote names and values are the config parameters.
|
|
.PP
|
|
See the config dump
|
|
command (https://rclone.org/commands/rclone_config_dump/) command for
|
|
more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS config/get: Get a remote in the config file.
|
|
.PP
|
|
Parameters:
|
|
.IP \[bu] 2
|
|
name - name of remote to get
|
|
.PP
|
|
See the config dump
|
|
command (https://rclone.org/commands/rclone_config_dump/) command for
|
|
more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS config/listremotes: Lists the remotes in the config file.
|
|
.PP
|
|
Returns - remotes - array of remote names
|
|
.PP
|
|
See the listremotes
|
|
command (https://rclone.org/commands/rclone_listremotes/) command for
|
|
more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS config/password: password the config for a remote.
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
name - name of remote
|
|
.IP \[bu] 2
|
|
parameters - a map of { \[dq]key\[dq]: \[dq]value\[dq] } pairs
|
|
.PP
|
|
See the config password
|
|
command (https://rclone.org/commands/rclone_config_password/) command
|
|
for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS config/providers: Shows how providers are configured in the config file.
|
|
.PP
|
|
Returns a JSON object: - providers - array of objects
|
|
.PP
|
|
See the config providers
|
|
command (https://rclone.org/commands/rclone_config_providers/) command
|
|
for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS config/update: update the config for a remote.
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
name - name of remote
|
|
.IP \[bu] 2
|
|
parameters - a map of { \[dq]key\[dq]: \[dq]value\[dq] } pairs
|
|
.IP \[bu] 2
|
|
opt - a dictionary of options to control the configuration
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
obscure - declare passwords are plain and need obscuring
|
|
.IP \[bu] 2
|
|
noObscure - declare passwords are already obscured and don\[aq]t need
|
|
obscuring
|
|
.IP \[bu] 2
|
|
nonInteractive - don\[aq]t interact with a user, return questions
|
|
.IP \[bu] 2
|
|
continue - continue the config process with an answer
|
|
.IP \[bu] 2
|
|
all - ask all the config questions not just the post config ones
|
|
.IP \[bu] 2
|
|
state - state to restart with - used with continue
|
|
.IP \[bu] 2
|
|
result - result to restart with - used with continue
|
|
.RE
|
|
.PP
|
|
See the config update
|
|
command (https://rclone.org/commands/rclone_config_update/) command for
|
|
more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS core/bwlimit: Set the bandwidth limit.
|
|
.PP
|
|
This sets the bandwidth limit to the string passed in.
|
|
This should be a single bandwidth limit entry or a pair of
|
|
upload:download bandwidth.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc core/bwlimit rate=off
|
|
{
|
|
\[dq]bytesPerSecond\[dq]: -1,
|
|
\[dq]bytesPerSecondTx\[dq]: -1,
|
|
\[dq]bytesPerSecondRx\[dq]: -1,
|
|
\[dq]rate\[dq]: \[dq]off\[dq]
|
|
}
|
|
rclone rc core/bwlimit rate=1M
|
|
{
|
|
\[dq]bytesPerSecond\[dq]: 1048576,
|
|
\[dq]bytesPerSecondTx\[dq]: 1048576,
|
|
\[dq]bytesPerSecondRx\[dq]: 1048576,
|
|
\[dq]rate\[dq]: \[dq]1M\[dq]
|
|
}
|
|
rclone rc core/bwlimit rate=1M:100k
|
|
{
|
|
\[dq]bytesPerSecond\[dq]: 1048576,
|
|
\[dq]bytesPerSecondTx\[dq]: 1048576,
|
|
\[dq]bytesPerSecondRx\[dq]: 131072,
|
|
\[dq]rate\[dq]: \[dq]1M\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If the rate parameter is not supplied then the bandwidth is queried
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc core/bwlimit
|
|
{
|
|
\[dq]bytesPerSecond\[dq]: 1048576,
|
|
\[dq]bytesPerSecondTx\[dq]: 1048576,
|
|
\[dq]bytesPerSecondRx\[dq]: 1048576,
|
|
\[dq]rate\[dq]: \[dq]1M\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The format of the parameter is exactly the same as passed to --bwlimit
|
|
except only one bandwidth may be specified.
|
|
.PP
|
|
In either case \[dq]rate\[dq] is returned as a human-readable string,
|
|
and \[dq]bytesPerSecond\[dq] is returned as a number.
|
|
.SS core/command: Run a rclone terminal command over rc.
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
command - a string with the command name.
|
|
.IP \[bu] 2
|
|
arg - a list of arguments for the backend command.
|
|
.IP \[bu] 2
|
|
opt - a map of string to string of options.
|
|
.IP \[bu] 2
|
|
returnType - one of (\[dq]COMBINED_OUTPUT\[dq], \[dq]STREAM\[dq],
|
|
\[dq]STREAM_ONLY_STDOUT\[dq], \[dq]STREAM_ONLY_STDERR\[dq]).
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Defaults to \[dq]COMBINED_OUTPUT\[dq] if not set.
|
|
.IP \[bu] 2
|
|
The STREAM returnTypes will write the output to the body of the HTTP
|
|
message.
|
|
.IP \[bu] 2
|
|
The COMBINED_OUTPUT will write the output to the \[dq]result\[dq]
|
|
parameter.
|
|
.RE
|
|
.PP
|
|
Returns:
|
|
.IP \[bu] 2
|
|
result - result from the backend command.
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Only set when using returnType \[dq]COMBINED_OUTPUT\[dq].
|
|
.RE
|
|
.IP \[bu] 2
|
|
error - set if rclone exits with an error code.
|
|
.IP \[bu] 2
|
|
returnType - one of (\[dq]COMBINED_OUTPUT\[dq], \[dq]STREAM\[dq],
|
|
\[dq]STREAM_ONLY_STDOUT\[dq], \[dq]STREAM_ONLY_STDERR\[dq]).
|
|
.PP
|
|
Example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc core/command command=ls -a mydrive:/ -o max-depth=1
|
|
rclone rc core/command -a ls -a mydrive:/ -o max-depth=1
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Returns:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]error\[dq]: false,
|
|
\[dq]result\[dq]: \[dq]<Raw command line output>\[dq]
|
|
}
|
|
|
|
OR
|
|
{
|
|
\[dq]error\[dq]: true,
|
|
\[dq]result\[dq]: \[dq]<Raw command line output>\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS core/gc: Runs a garbage collection.
|
|
.PP
|
|
This tells the go runtime to do a garbage collection run.
|
|
It isn\[aq]t necessary to call this normally, but it can be useful for
|
|
debugging memory problems.
|
|
.SS core/group-list: Returns list of stats.
|
|
.PP
|
|
This returns list of stats groups currently in memory.
|
|
.PP
|
|
Returns the following values:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]groups\[dq]: an array of group names:
|
|
[
|
|
\[dq]group1\[dq],
|
|
\[dq]group2\[dq],
|
|
...
|
|
]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.SS core/memstats: Returns the memory statistics
|
|
.PP
|
|
This returns the memory statistics of the running program.
|
|
What the values mean are explained in the go docs:
|
|
https://golang.org/pkg/runtime/#MemStats
|
|
.PP
|
|
The most interesting values for most people are:
|
|
.IP \[bu] 2
|
|
HeapAlloc - this is the amount of memory rclone is actually using
|
|
.IP \[bu] 2
|
|
HeapSys - this is the amount of memory rclone has obtained from the OS
|
|
.IP \[bu] 2
|
|
Sys - this is the total amount of memory requested from the OS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
It is virtual memory so may include unused memory
|
|
.RE
|
|
.SS core/obscure: Obscures a string passed in.
|
|
.PP
|
|
Pass a clear string and rclone will obscure it for the config file: -
|
|
clear - string
|
|
.PP
|
|
Returns: - obscured - string
|
|
.SS core/pid: Return PID of current process
|
|
.PP
|
|
This returns PID of current process.
|
|
Useful for stopping rclone process.
|
|
.SS core/quit: Terminates the app.
|
|
.PP
|
|
(Optional) Pass an exit code to be used for terminating the app: -
|
|
exitCode - int
|
|
.SS core/stats: Returns stats about current transfers.
|
|
.PP
|
|
This returns all available stats:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc core/stats
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If group is not provided then summed up stats for all groups will be
|
|
returned.
|
|
.PP
|
|
Parameters
|
|
.IP \[bu] 2
|
|
group - name of the stats group (string)
|
|
.PP
|
|
Returns the following values:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]bytes\[dq]: total transferred bytes since the start of the group,
|
|
\[dq]checks\[dq]: number of files checked,
|
|
\[dq]deletes\[dq] : number of files deleted,
|
|
\[dq]elapsedTime\[dq]: time in floating point seconds since rclone was started,
|
|
\[dq]errors\[dq]: number of errors,
|
|
\[dq]eta\[dq]: estimated time in seconds until the group completes,
|
|
\[dq]fatalError\[dq]: boolean whether there has been at least one fatal error,
|
|
\[dq]lastError\[dq]: last error string,
|
|
\[dq]renames\[dq] : number of files renamed,
|
|
\[dq]retryError\[dq]: boolean showing whether there has been at least one non-NoRetryError,
|
|
\[dq]speed\[dq]: average speed in bytes per second since start of the group,
|
|
\[dq]totalBytes\[dq]: total number of bytes in the group,
|
|
\[dq]totalChecks\[dq]: total number of checks in the group,
|
|
\[dq]totalTransfers\[dq]: total number of transfers in the group,
|
|
\[dq]transferTime\[dq] : total time spent on running jobs,
|
|
\[dq]transfers\[dq]: number of transferred files,
|
|
\[dq]transferring\[dq]: an array of currently active file transfers:
|
|
[
|
|
{
|
|
\[dq]bytes\[dq]: total transferred bytes for this file,
|
|
\[dq]eta\[dq]: estimated time in seconds until file transfer completion
|
|
\[dq]name\[dq]: name of the file,
|
|
\[dq]percentage\[dq]: progress of the file transfer in percent,
|
|
\[dq]speed\[dq]: average speed over the whole transfer in bytes per second,
|
|
\[dq]speedAvg\[dq]: current speed in bytes per second as an exponentially weighted moving average,
|
|
\[dq]size\[dq]: size of the file in bytes
|
|
}
|
|
],
|
|
\[dq]checking\[dq]: an array of names of currently active file checks
|
|
[]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Values for \[dq]transferring\[dq], \[dq]checking\[dq] and
|
|
\[dq]lastError\[dq] are only assigned if data is available.
|
|
The value for \[dq]eta\[dq] is null if an eta cannot be determined.
|
|
.SS core/stats-delete: Delete stats group.
|
|
.PP
|
|
This deletes entire stats group.
|
|
.PP
|
|
Parameters
|
|
.IP \[bu] 2
|
|
group - name of the stats group (string)
|
|
.SS core/stats-reset: Reset stats.
|
|
.PP
|
|
This clears counters, errors and finished transfers for all stats or
|
|
specific stats group if group is provided.
|
|
.PP
|
|
Parameters
|
|
.IP \[bu] 2
|
|
group - name of the stats group (string)
|
|
.SS core/transferred: Returns stats about completed transfers.
|
|
.PP
|
|
This returns stats about completed transfers:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc core/transferred
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If group is not provided then completed transfers for all groups will be
|
|
returned.
|
|
.PP
|
|
Note only the last 100 completed transfers are returned.
|
|
.PP
|
|
Parameters
|
|
.IP \[bu] 2
|
|
group - name of the stats group (string)
|
|
.PP
|
|
Returns the following values:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]transferred\[dq]: an array of completed transfers (including failed ones):
|
|
[
|
|
{
|
|
\[dq]name\[dq]: name of the file,
|
|
\[dq]size\[dq]: size of the file in bytes,
|
|
\[dq]bytes\[dq]: total transferred bytes for this file,
|
|
\[dq]checked\[dq]: if the transfer is only checked (skipped, deleted),
|
|
\[dq]timestamp\[dq]: integer representing millisecond unix epoch,
|
|
\[dq]error\[dq]: string description of the error (empty if successful),
|
|
\[dq]jobid\[dq]: id of the job that this transfer belongs to
|
|
}
|
|
]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.SS core/version: Shows the current version of rclone and the go runtime.
|
|
.PP
|
|
This shows the current version of go and the go runtime:
|
|
.IP \[bu] 2
|
|
version - rclone version, e.g.
|
|
\[dq]v1.53.0\[dq]
|
|
.IP \[bu] 2
|
|
decomposed - version number as [major, minor, patch]
|
|
.IP \[bu] 2
|
|
isGit - boolean - true if this was compiled from the git version
|
|
.IP \[bu] 2
|
|
isBeta - boolean - true if this is a beta version
|
|
.IP \[bu] 2
|
|
os - OS in use as according to Go
|
|
.IP \[bu] 2
|
|
arch - cpu architecture in use according to Go
|
|
.IP \[bu] 2
|
|
goVersion - version of Go runtime in use
|
|
.IP \[bu] 2
|
|
linking - type of rclone executable (static or dynamic)
|
|
.IP \[bu] 2
|
|
goTags - space separated build tags or \[dq]none\[dq]
|
|
.SS debug/set-block-profile-rate: Set runtime.SetBlockProfileRate for blocking profiling.
|
|
.PP
|
|
SetBlockProfileRate controls the fraction of goroutine blocking events
|
|
that are reported in the blocking profile.
|
|
The profiler aims to sample an average of one blocking event per rate
|
|
nanoseconds spent blocked.
|
|
.PP
|
|
To include every blocking event in the profile, pass rate = 1.
|
|
To turn off profiling entirely, pass rate <= 0.
|
|
.PP
|
|
After calling this you can use this to see the blocking profile:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
go tool pprof http://localhost:5572/debug/pprof/block
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Parameters:
|
|
.IP \[bu] 2
|
|
rate - int
|
|
.SS debug/set-mutex-profile-fraction: Set runtime.SetMutexProfileFraction for mutex profiling.
|
|
.PP
|
|
SetMutexProfileFraction controls the fraction of mutex contention events
|
|
that are reported in the mutex profile.
|
|
On average 1/rate events are reported.
|
|
The previous rate is returned.
|
|
.PP
|
|
To turn off profiling entirely, pass rate 0.
|
|
To just read the current rate, pass rate < 0.
|
|
(For n>1 the details of sampling may change.)
|
|
.PP
|
|
Once this is set you can look use this to profile the mutex contention:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
go tool pprof http://localhost:5572/debug/pprof/mutex
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Parameters:
|
|
.IP \[bu] 2
|
|
rate - int
|
|
.PP
|
|
Results:
|
|
.IP \[bu] 2
|
|
previousRate - int
|
|
.SS fscache/clear: Clear the Fs cache.
|
|
.PP
|
|
This clears the fs cache.
|
|
This is where remotes created from backends are cached for a short while
|
|
to make repeated rc calls more efficient.
|
|
.PP
|
|
If you change the parameters of a backend then you may want to call this
|
|
to clear an existing remote out of the cache before re-creating it.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS fscache/entries: Returns the number of entries in the fs cache.
|
|
.PP
|
|
This returns the number of entries in the fs cache.
|
|
.PP
|
|
Returns - entries - number of items in the cache
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS job/list: Lists the IDs of the running jobs
|
|
.PP
|
|
Parameters: None.
|
|
.PP
|
|
Results:
|
|
.IP \[bu] 2
|
|
jobids - array of integer job ids.
|
|
.SS job/status: Reads the status of the job ID
|
|
.PP
|
|
Parameters:
|
|
.IP \[bu] 2
|
|
jobid - id of the job (integer).
|
|
.PP
|
|
Results:
|
|
.IP \[bu] 2
|
|
finished - boolean
|
|
.IP \[bu] 2
|
|
duration - time in seconds that the job ran for
|
|
.IP \[bu] 2
|
|
endTime - time the job finished (e.g.
|
|
\[dq]2018-10-26T18:50:20.528746884+01:00\[dq])
|
|
.IP \[bu] 2
|
|
error - error from the job or empty string for no error
|
|
.IP \[bu] 2
|
|
finished - boolean whether the job has finished or not
|
|
.IP \[bu] 2
|
|
id - as passed in above
|
|
.IP \[bu] 2
|
|
startTime - time the job started (e.g.
|
|
\[dq]2018-10-26T18:50:20.528336039+01:00\[dq])
|
|
.IP \[bu] 2
|
|
success - boolean - true for success false otherwise
|
|
.IP \[bu] 2
|
|
output - output of the job as would have been returned if called
|
|
synchronously
|
|
.IP \[bu] 2
|
|
progress - output of the progress related to the underlying job
|
|
.SS job/stop: Stop the running job
|
|
.PP
|
|
Parameters:
|
|
.IP \[bu] 2
|
|
jobid - id of the job (integer).
|
|
.SS mount/listmounts: Show current mount points
|
|
.PP
|
|
This shows currently mounted points, which can be used for performing an
|
|
unmount.
|
|
.PP
|
|
This takes no parameters and returns
|
|
.IP \[bu] 2
|
|
mountPoints: list of current mount points
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc mount/listmounts
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS mount/mount: Create a new mount point
|
|
.PP
|
|
rclone allows Linux, FreeBSD, macOS and Windows to mount any of
|
|
Rclone\[aq]s cloud storage systems as a file system with FUSE.
|
|
.PP
|
|
If no mountType is provided, the priority is given as follows: 1.
|
|
mount 2.cmount 3.mount2
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote path to be mounted (required)
|
|
.IP \[bu] 2
|
|
mountPoint: valid path on the local machine (required)
|
|
.IP \[bu] 2
|
|
mountType: one of the values (mount, cmount, mount2) specifies the mount
|
|
implementation to use
|
|
.IP \[bu] 2
|
|
mountOpt: a JSON object with Mount options in.
|
|
.IP \[bu] 2
|
|
vfsOpt: a JSON object with VFS options in.
|
|
.PP
|
|
Example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc mount/mount fs=mydrive: mountPoint=/home/<user>/mountPoint
|
|
rclone rc mount/mount fs=mydrive: mountPoint=/home/<user>/mountPoint mountType=mount
|
|
rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt=\[aq]{\[dq]CacheMode\[dq]: 2}\[aq] mountOpt=\[aq]{\[dq]AllowOther\[dq]: true}\[aq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The vfsOpt are as described in options/get and can be seen in the the
|
|
\[dq]vfs\[dq] section when running and the mountOpt can be seen in the
|
|
\[dq]mount\[dq] section:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc options/get
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS mount/types: Show all possible mount types
|
|
.PP
|
|
This shows all possible mount types and returns them as a list.
|
|
.PP
|
|
This takes no parameters and returns
|
|
.IP \[bu] 2
|
|
mountTypes: list of mount types
|
|
.PP
|
|
The mount types are strings like \[dq]mount\[dq], \[dq]mount2\[dq],
|
|
\[dq]cmount\[dq] and can be passed to mount/mount as the mountType
|
|
parameter.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc mount/types
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS mount/unmount: Unmount selected active mount
|
|
.PP
|
|
rclone allows Linux, FreeBSD, macOS and Windows to mount any of
|
|
Rclone\[aq]s cloud storage systems as a file system with FUSE.
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
mountPoint: valid path on the local machine where the mount was created
|
|
(required)
|
|
.PP
|
|
Example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc mount/unmount mountPoint=/home/<user>/mountPoint
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS mount/unmountall: Show current mount points
|
|
.PP
|
|
This shows currently mounted points, which can be used for performing an
|
|
unmount.
|
|
.PP
|
|
This takes no parameters and returns error if unmount does not succeed.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc mount/unmountall
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/about: Return the space used on the remote
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.PP
|
|
The result is as returned from rclone about --json
|
|
.PP
|
|
See the about command (https://rclone.org/commands/rclone_size/) command
|
|
for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/cleanup: Remove trashed files in the remote or path
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.PP
|
|
See the cleanup command (https://rclone.org/commands/rclone_cleanup/)
|
|
command for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/copyfile: Copy a file from source remote to destination remote
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
srcFs - a remote name string e.g.
|
|
\[dq]drive:\[dq] for the source
|
|
.IP \[bu] 2
|
|
srcRemote - a path within that remote e.g.
|
|
\[dq]file.txt\[dq] for the source
|
|
.IP \[bu] 2
|
|
dstFs - a remote name string e.g.
|
|
\[dq]drive2:\[dq] for the destination
|
|
.IP \[bu] 2
|
|
dstRemote - a path within that remote e.g.
|
|
\[dq]file2.txt\[dq] for the destination
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/copyurl: Copy the URL to the object
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.IP \[bu] 2
|
|
remote - a path within that remote e.g.
|
|
\[dq]dir\[dq]
|
|
.IP \[bu] 2
|
|
url - string, URL to read from
|
|
.IP \[bu] 2
|
|
autoFilename - boolean, set to true to retrieve destination file name
|
|
from url See the copyurl
|
|
command (https://rclone.org/commands/rclone_copyurl/) command for more
|
|
information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/delete: Remove files in the path
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.PP
|
|
See the delete command (https://rclone.org/commands/rclone_delete/)
|
|
command for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/deletefile: Remove the single file pointed to
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.IP \[bu] 2
|
|
remote - a path within that remote e.g.
|
|
\[dq]dir\[dq]
|
|
.PP
|
|
See the deletefile
|
|
command (https://rclone.org/commands/rclone_deletefile/) command for
|
|
more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/fsinfo: Return information about the remote
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.PP
|
|
This returns info about the remote passed in;
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
// optional features and whether they are available or not
|
|
\[dq]Features\[dq]: {
|
|
\[dq]About\[dq]: true,
|
|
\[dq]BucketBased\[dq]: false,
|
|
\[dq]CanHaveEmptyDirectories\[dq]: true,
|
|
\[dq]CaseInsensitive\[dq]: false,
|
|
\[dq]ChangeNotify\[dq]: false,
|
|
\[dq]CleanUp\[dq]: false,
|
|
\[dq]Copy\[dq]: false,
|
|
\[dq]DirCacheFlush\[dq]: false,
|
|
\[dq]DirMove\[dq]: true,
|
|
\[dq]DuplicateFiles\[dq]: false,
|
|
\[dq]GetTier\[dq]: false,
|
|
\[dq]ListR\[dq]: false,
|
|
\[dq]MergeDirs\[dq]: false,
|
|
\[dq]Move\[dq]: true,
|
|
\[dq]OpenWriterAt\[dq]: true,
|
|
\[dq]PublicLink\[dq]: false,
|
|
\[dq]Purge\[dq]: true,
|
|
\[dq]PutStream\[dq]: true,
|
|
\[dq]PutUnchecked\[dq]: false,
|
|
\[dq]ReadMimeType\[dq]: false,
|
|
\[dq]ServerSideAcrossConfigs\[dq]: false,
|
|
\[dq]SetTier\[dq]: false,
|
|
\[dq]SetWrapper\[dq]: false,
|
|
\[dq]UnWrap\[dq]: false,
|
|
\[dq]WrapFs\[dq]: false,
|
|
\[dq]WriteMimeType\[dq]: false
|
|
},
|
|
// Names of hashes available
|
|
\[dq]Hashes\[dq]: [
|
|
\[dq]MD5\[dq],
|
|
\[dq]SHA-1\[dq],
|
|
\[dq]DropboxHash\[dq],
|
|
\[dq]QuickXorHash\[dq]
|
|
],
|
|
\[dq]Name\[dq]: \[dq]local\[dq], // Name as created
|
|
\[dq]Precision\[dq]: 1, // Precision of timestamps in ns
|
|
\[dq]Root\[dq]: \[dq]/\[dq], // Path as created
|
|
\[dq]String\[dq]: \[dq]Local file system at /\[dq] // how the remote will appear in logs
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This command does not have a command line equivalent so use this
|
|
instead:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc --loopback operations/fsinfo fs=remote:
|
|
\f[R]
|
|
.fi
|
|
.SS operations/list: List the given remote and path in JSON format
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.IP \[bu] 2
|
|
remote - a path within that remote e.g.
|
|
\[dq]dir\[dq]
|
|
.IP \[bu] 2
|
|
opt - a dictionary of options to control the listing (optional)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
recurse - If set recurse directories
|
|
.IP \[bu] 2
|
|
noModTime - If set return modification time
|
|
.IP \[bu] 2
|
|
showEncrypted - If set show decrypted names
|
|
.IP \[bu] 2
|
|
showOrigIDs - If set show the IDs for each item if known
|
|
.IP \[bu] 2
|
|
showHash - If set return a dictionary of hashes
|
|
.IP \[bu] 2
|
|
noMimeType - If set don\[aq]t show mime types
|
|
.IP \[bu] 2
|
|
dirsOnly - If set only show directories
|
|
.IP \[bu] 2
|
|
filesOnly - If set only show files
|
|
.IP \[bu] 2
|
|
hashTypes - array of strings of hash types to show if showHash set
|
|
.RE
|
|
.PP
|
|
Returns:
|
|
.IP \[bu] 2
|
|
list
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This is an array of objects as described in the lsjson command
|
|
.RE
|
|
.PP
|
|
See the lsjson command (https://rclone.org/commands/rclone_lsjson/) for
|
|
more information on the above and examples.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/mkdir: Make a destination directory or container
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.IP \[bu] 2
|
|
remote - a path within that remote e.g.
|
|
\[dq]dir\[dq]
|
|
.PP
|
|
See the mkdir command (https://rclone.org/commands/rclone_mkdir/)
|
|
command for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/movefile: Move a file from source remote to destination remote
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
srcFs - a remote name string e.g.
|
|
\[dq]drive:\[dq] for the source
|
|
.IP \[bu] 2
|
|
srcRemote - a path within that remote e.g.
|
|
\[dq]file.txt\[dq] for the source
|
|
.IP \[bu] 2
|
|
dstFs - a remote name string e.g.
|
|
\[dq]drive2:\[dq] for the destination
|
|
.IP \[bu] 2
|
|
dstRemote - a path within that remote e.g.
|
|
\[dq]file2.txt\[dq] for the destination
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/publiclink: Create or retrieve a public link to the given file or folder.
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.IP \[bu] 2
|
|
remote - a path within that remote e.g.
|
|
\[dq]dir\[dq]
|
|
.IP \[bu] 2
|
|
unlink - boolean - if set removes the link rather than adding it
|
|
(optional)
|
|
.IP \[bu] 2
|
|
expire - string - the expiry time of the link e.g.
|
|
\[dq]1d\[dq] (optional)
|
|
.PP
|
|
Returns:
|
|
.IP \[bu] 2
|
|
url - URL of the resource
|
|
.PP
|
|
See the link command (https://rclone.org/commands/rclone_link/) command
|
|
for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/purge: Remove a directory or container and all of its contents
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.IP \[bu] 2
|
|
remote - a path within that remote e.g.
|
|
\[dq]dir\[dq]
|
|
.PP
|
|
See the purge command (https://rclone.org/commands/rclone_purge/)
|
|
command for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/rmdir: Remove an empty directory or container
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.IP \[bu] 2
|
|
remote - a path within that remote e.g.
|
|
\[dq]dir\[dq]
|
|
.PP
|
|
See the rmdir command (https://rclone.org/commands/rclone_rmdir/)
|
|
command for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/rmdirs: Remove all the empty directories in the path
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.IP \[bu] 2
|
|
remote - a path within that remote e.g.
|
|
\[dq]dir\[dq]
|
|
.IP \[bu] 2
|
|
leaveRoot - boolean, set to true not to delete the root See the rmdirs
|
|
command (https://rclone.org/commands/rclone_rmdirs/) command for more
|
|
information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/size: Count the number of bytes and files in remote
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:path/to/dir\[dq]
|
|
.PP
|
|
Returns:
|
|
.IP \[bu] 2
|
|
count - number of files
|
|
.IP \[bu] 2
|
|
bytes - number of bytes in those files
|
|
.PP
|
|
See the size command (https://rclone.org/commands/rclone_size/) command
|
|
for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/stat: Give information about the supplied file or directory
|
|
.PP
|
|
This takes the following parameters
|
|
.IP \[bu] 2
|
|
fs - a remote name string eg \[dq]drive:\[dq]
|
|
.IP \[bu] 2
|
|
remote - a path within that remote eg \[dq]dir\[dq]
|
|
.IP \[bu] 2
|
|
opt - a dictionary of options to control the listing (optional)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
see operations/list for the options
|
|
.RE
|
|
.PP
|
|
The result is
|
|
.IP \[bu] 2
|
|
item - an object as described in the lsjson command.
|
|
Will be null if not found.
|
|
.PP
|
|
Note that if you are only interested in files then it is much more
|
|
efficient to set the filesOnly flag in the options.
|
|
.PP
|
|
See the lsjson command (https://rclone.org/commands/rclone_lsjson/) for
|
|
more information on the above and examples.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS operations/uploadfile: Upload file using multiform/form-data
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
fs - a remote name string e.g.
|
|
\[dq]drive:\[dq]
|
|
.IP \[bu] 2
|
|
remote - a path within that remote e.g.
|
|
\[dq]dir\[dq]
|
|
.IP \[bu] 2
|
|
each part in body represents a file to be uploaded See the uploadfile
|
|
command (https://rclone.org/commands/rclone_uploadfile/) command for
|
|
more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS options/blocks: List all the option blocks
|
|
.PP
|
|
Returns: - options - a list of the options block names
|
|
.SS options/get: Get all the global options
|
|
.PP
|
|
Returns an object where keys are option block names and values are an
|
|
object with the current option values in.
|
|
.PP
|
|
Note that these are the global options which are unaffected by use of
|
|
the _config and _filter parameters.
|
|
If you wish to read the parameters set in _config then use
|
|
options/config and for _filter use options/filter.
|
|
.PP
|
|
This shows the internal names of the option within rclone which should
|
|
map to the external options very easily with a few exceptions.
|
|
.SS options/local: Get the currently active config for this call
|
|
.PP
|
|
Returns an object with the keys \[dq]config\[dq] and \[dq]filter\[dq].
|
|
The \[dq]config\[dq] key contains the local config and the
|
|
\[dq]filter\[dq] key contains the local filters.
|
|
.PP
|
|
Note that these are the local options specific to this rc call.
|
|
If _config was not supplied then they will be the global options.
|
|
Likewise with \[dq]_filter\[dq].
|
|
.PP
|
|
This call is mostly useful for seeing if _config and _filter passing is
|
|
working.
|
|
.PP
|
|
This shows the internal names of the option within rclone which should
|
|
map to the external options very easily with a few exceptions.
|
|
.SS options/set: Set an option
|
|
.PP
|
|
Parameters:
|
|
.IP \[bu] 2
|
|
option block name containing an object with
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
key: value
|
|
.RE
|
|
.PP
|
|
Repeated as often as required.
|
|
.PP
|
|
Only supply the options you wish to change.
|
|
If an option is unknown it will be silently ignored.
|
|
Not all options will have an effect when changed like this.
|
|
.PP
|
|
For example:
|
|
.PP
|
|
This sets DEBUG level logs (-vv) (these can be set by number or string)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc options/set --json \[aq]{\[dq]main\[dq]: {\[dq]LogLevel\[dq]: \[dq]DEBUG\[dq]}}\[aq]
|
|
rclone rc options/set --json \[aq]{\[dq]main\[dq]: {\[dq]LogLevel\[dq]: 8}}\[aq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And this sets INFO level logs (-v)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc options/set --json \[aq]{\[dq]main\[dq]: {\[dq]LogLevel\[dq]: \[dq]INFO\[dq]}}\[aq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And this sets NOTICE level logs (normal without -v)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc options/set --json \[aq]{\[dq]main\[dq]: {\[dq]LogLevel\[dq]: \[dq]NOTICE\[dq]}}\[aq]
|
|
\f[R]
|
|
.fi
|
|
.SS pluginsctl/addPlugin: Add a plugin using url
|
|
.PP
|
|
Used for adding a plugin to the webgui.
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
url - http url of the github repo where the plugin is hosted
|
|
(http://github.com/rclone/rclone-webui-react).
|
|
.PP
|
|
Example:
|
|
.PP
|
|
rclone rc pluginsctl/addPlugin
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS pluginsctl/getPluginsForType: Get plugins with type criteria
|
|
.PP
|
|
This shows all possible plugins by a mime type.
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
type - supported mime type by a loaded plugin e.g.
|
|
(video/mp4, audio/mp3).
|
|
.IP \[bu] 2
|
|
pluginType - filter plugins based on their type e.g.
|
|
(DASHBOARD, FILE_HANDLER, TERMINAL).
|
|
.PP
|
|
Returns:
|
|
.IP \[bu] 2
|
|
loadedPlugins - list of current production plugins.
|
|
.IP \[bu] 2
|
|
testPlugins - list of temporarily loaded development plugins, usually
|
|
running on a different server.
|
|
.PP
|
|
Example:
|
|
.PP
|
|
rclone rc pluginsctl/getPluginsForType type=video/mp4
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS pluginsctl/listPlugins: Get the list of currently loaded plugins
|
|
.PP
|
|
This allows you to get the currently enabled plugins and their details.
|
|
.PP
|
|
This takes no parameters and returns:
|
|
.IP \[bu] 2
|
|
loadedPlugins - list of current production plugins.
|
|
.IP \[bu] 2
|
|
testPlugins - list of temporarily loaded development plugins, usually
|
|
running on a different server.
|
|
.PP
|
|
E.g.
|
|
.PP
|
|
rclone rc pluginsctl/listPlugins
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS pluginsctl/listTestPlugins: Show currently loaded test plugins
|
|
.PP
|
|
Allows listing of test plugins with the rclone.test set to true in
|
|
package.json of the plugin.
|
|
.PP
|
|
This takes no parameters and returns:
|
|
.IP \[bu] 2
|
|
loadedTestPlugins - list of currently available test plugins.
|
|
.PP
|
|
E.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc pluginsctl/listTestPlugins
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS pluginsctl/removePlugin: Remove a loaded plugin
|
|
.PP
|
|
This allows you to remove a plugin using it\[aq]s name.
|
|
.PP
|
|
This takes parameters:
|
|
.IP \[bu] 2
|
|
name - name of the plugin in the format
|
|
\f[C]author\f[R]/\f[C]plugin_name\f[R].
|
|
.PP
|
|
E.g.
|
|
.PP
|
|
rclone rc pluginsctl/removePlugin name=rclone/video-plugin
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS pluginsctl/removeTestPlugin: Remove a test plugin
|
|
.PP
|
|
This allows you to remove a plugin using it\[aq]s name.
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
name - name of the plugin in the format
|
|
\f[C]author\f[R]/\f[C]plugin_name\f[R].
|
|
.PP
|
|
Example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc pluginsctl/removeTestPlugin name=rclone/rclone-webui-react
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS rc/error: This returns an error
|
|
.PP
|
|
This returns an error with the input as part of its error string.
|
|
Useful for testing error handling.
|
|
.SS rc/list: List all the registered remote control commands
|
|
.PP
|
|
This lists all the registered remote control commands as a JSON map in
|
|
the commands response.
|
|
.SS rc/noop: Echo the input to the output parameters
|
|
.PP
|
|
This echoes the input parameters to the output parameters for testing
|
|
purposes.
|
|
It can be used to check that rclone is still alive and to check that
|
|
parameter passing is working properly.
|
|
.SS rc/noopauth: Echo the input to the output parameters requiring auth
|
|
.PP
|
|
This echoes the input parameters to the output parameters for testing
|
|
purposes.
|
|
It can be used to check that rclone is still alive and to check that
|
|
parameter passing is working properly.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS sync/bisync: Perform bidirectonal synchronization between two paths.
|
|
.PP
|
|
This takes the following parameters
|
|
.IP \[bu] 2
|
|
path1 - a remote directory string e.g.
|
|
\f[C]drive:path1\f[R]
|
|
.IP \[bu] 2
|
|
path2 - a remote directory string e.g.
|
|
\f[C]drive:path2\f[R]
|
|
.IP \[bu] 2
|
|
dryRun - dry-run mode
|
|
.IP \[bu] 2
|
|
resync - performs the resync run
|
|
.IP \[bu] 2
|
|
checkAccess - abort if RCLONE_TEST files are not found on both
|
|
filesystems
|
|
.IP \[bu] 2
|
|
checkFilename - file name for checkAccess (default: RCLONE_TEST)
|
|
.IP \[bu] 2
|
|
maxDelete - abort sync if percentage of deleted files is above this
|
|
threshold (default: 50)
|
|
.IP \[bu] 2
|
|
force - maxDelete safety check and run the sync
|
|
.IP \[bu] 2
|
|
checkSync - \f[C]true\f[R] by default, \f[C]false\f[R] disables
|
|
comparison of final listings, \f[C]only\f[R] will skip sync, only
|
|
compare listings from the last run
|
|
.IP \[bu] 2
|
|
removeEmptyDirs - remove empty directories at the final cleanup step
|
|
.IP \[bu] 2
|
|
filtersFile - read filtering patterns from a file
|
|
.IP \[bu] 2
|
|
workdir - server directory for history files (default:
|
|
/home/ncw/.cache/rclone/bisync)
|
|
.IP \[bu] 2
|
|
noCleanup - retain working files
|
|
.PP
|
|
See bisync command help (https://rclone.org/commands/rclone_bisync/) and
|
|
full bisync description (https://rclone.org/bisync/) for more
|
|
information.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS sync/copy: copy a directory from source remote to destination remote
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
srcFs - a remote name string e.g.
|
|
\[dq]drive:src\[dq] for the source
|
|
.IP \[bu] 2
|
|
dstFs - a remote name string e.g.
|
|
\[dq]drive:dst\[dq] for the destination
|
|
.IP \[bu] 2
|
|
createEmptySrcDirs - create empty src directories on destination if set
|
|
.PP
|
|
See the copy command (https://rclone.org/commands/rclone_copy/) command
|
|
for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS sync/move: move a directory from source remote to destination remote
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
srcFs - a remote name string e.g.
|
|
\[dq]drive:src\[dq] for the source
|
|
.IP \[bu] 2
|
|
dstFs - a remote name string e.g.
|
|
\[dq]drive:dst\[dq] for the destination
|
|
.IP \[bu] 2
|
|
createEmptySrcDirs - create empty src directories on destination if set
|
|
.IP \[bu] 2
|
|
deleteEmptySrcDirs - delete empty src directories if set
|
|
.PP
|
|
See the move command (https://rclone.org/commands/rclone_move/) command
|
|
for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS sync/sync: sync a directory from source remote to destination remote
|
|
.PP
|
|
This takes the following parameters:
|
|
.IP \[bu] 2
|
|
srcFs - a remote name string e.g.
|
|
\[dq]drive:src\[dq] for the source
|
|
.IP \[bu] 2
|
|
dstFs - a remote name string e.g.
|
|
\[dq]drive:dst\[dq] for the destination
|
|
.IP \[bu] 2
|
|
createEmptySrcDirs - create empty src directories on destination if set
|
|
.PP
|
|
See the sync command (https://rclone.org/commands/rclone_sync/) command
|
|
for more information on the above.
|
|
.PP
|
|
\f[B]Authentication is required for this call.\f[R]
|
|
.SS vfs/forget: Forget files or directories in the directory cache.
|
|
.PP
|
|
This forgets the paths in the directory cache causing them to be re-read
|
|
from the remote when needed.
|
|
.PP
|
|
If no paths are passed in then it will forget all the paths in the
|
|
directory cache.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Otherwise pass files or dirs in as file=path or dir=path.
|
|
Any parameter key starting with file will forget that file and any
|
|
starting with dir will forget that dir, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/forget file=hello file2=goodbye dir=home/junk
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This command takes an \[dq]fs\[dq] parameter.
|
|
If this parameter is not supplied and if there is only one VFS in use
|
|
then that VFS will be used.
|
|
If there is more than one VFS in use then the \[dq]fs\[dq] parameter
|
|
must be supplied.
|
|
.SS vfs/list: List active VFSes.
|
|
.PP
|
|
This lists the active VFSes.
|
|
.PP
|
|
It returns a list under the key \[dq]vfses\[dq] where the values are the
|
|
VFS names that could be passed to the other VFS commands in the
|
|
\[dq]fs\[dq] parameter.
|
|
.SS vfs/poll-interval: Get the status or update the value of the poll-interval option.
|
|
.PP
|
|
Without any parameter given this returns the current status of the
|
|
poll-interval setting.
|
|
.PP
|
|
When the interval=duration parameter is set, the poll-interval value is
|
|
updated and the polling function is notified.
|
|
Setting interval=0 disables poll-interval.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/poll-interval interval=5m
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The timeout=duration parameter can be used to specify a time to wait for
|
|
the current poll function to apply the new value.
|
|
If timeout is less or equal 0, which is the default, wait indefinitely.
|
|
.PP
|
|
The new poll-interval value will only be active when the timeout is not
|
|
reached.
|
|
.PP
|
|
If poll-interval is updated or disabled temporarily, some changes might
|
|
not get picked up by the polling function, depending on the used remote.
|
|
.PP
|
|
This command takes an \[dq]fs\[dq] parameter.
|
|
If this parameter is not supplied and if there is only one VFS in use
|
|
then that VFS will be used.
|
|
If there is more than one VFS in use then the \[dq]fs\[dq] parameter
|
|
must be supplied.
|
|
.SS vfs/refresh: Refresh the directory cache.
|
|
.PP
|
|
This reads the directories for the specified paths and freshens the
|
|
directory cache.
|
|
.PP
|
|
If no paths are passed in then it will refresh the root directory.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/refresh
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Otherwise pass directories in as dir=path.
|
|
Any parameter key starting with dir will refresh that directory, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rc vfs/refresh dir=home/junk dir2=data/misc
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If the parameter recursive=true is given the whole directory tree will
|
|
get refreshed.
|
|
This refresh will use --fast-list if enabled.
|
|
.PP
|
|
This command takes an \[dq]fs\[dq] parameter.
|
|
If this parameter is not supplied and if there is only one VFS in use
|
|
then that VFS will be used.
|
|
If there is more than one VFS in use then the \[dq]fs\[dq] parameter
|
|
must be supplied.
|
|
.SS vfs/stats: Stats for a VFS.
|
|
.PP
|
|
This returns stats for the selected VFS.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
// Status of the disk cache - only present if --vfs-cache-mode > off
|
|
\[dq]diskCache\[dq]: {
|
|
\[dq]bytesUsed\[dq]: 0,
|
|
\[dq]erroredFiles\[dq]: 0,
|
|
\[dq]files\[dq]: 0,
|
|
\[dq]hashType\[dq]: 1,
|
|
\[dq]outOfSpace\[dq]: false,
|
|
\[dq]path\[dq]: \[dq]/home/user/.cache/rclone/vfs/local/mnt/a\[dq],
|
|
\[dq]pathMeta\[dq]: \[dq]/home/user/.cache/rclone/vfsMeta/local/mnt/a\[dq],
|
|
\[dq]uploadsInProgress\[dq]: 0,
|
|
\[dq]uploadsQueued\[dq]: 0
|
|
},
|
|
\[dq]fs\[dq]: \[dq]/mnt/a\[dq],
|
|
\[dq]inUse\[dq]: 1,
|
|
// Status of the in memory metadata cache
|
|
\[dq]metadataCache\[dq]: {
|
|
\[dq]dirs\[dq]: 1,
|
|
\[dq]files\[dq]: 0
|
|
},
|
|
// Options as returned by options/get
|
|
\[dq]opt\[dq]: {
|
|
\[dq]CacheMaxAge\[dq]: 3600000000000,
|
|
// ...
|
|
\[dq]WriteWait\[dq]: 1000000000
|
|
}
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This command takes an \[dq]fs\[dq] parameter.
|
|
If this parameter is not supplied and if there is only one VFS in use
|
|
then that VFS will be used.
|
|
If there is more than one VFS in use then the \[dq]fs\[dq] parameter
|
|
must be supplied.
|
|
.SS Accessing the remote control via HTTP
|
|
.PP
|
|
Rclone implements a simple HTTP based protocol.
|
|
.PP
|
|
Each endpoint takes an JSON object and returns a JSON object or an
|
|
error.
|
|
The JSON objects are essentially a map of string names to values.
|
|
.PP
|
|
All calls must made using POST.
|
|
.PP
|
|
The input objects can be supplied using URL parameters, POST parameters
|
|
or by supplying \[dq]Content-Type: application/json\[dq] and a JSON blob
|
|
in the body.
|
|
There are examples of these below using \f[C]curl\f[R].
|
|
.PP
|
|
The response will be a JSON blob in the body of the response.
|
|
This is formatted to be reasonably human-readable.
|
|
.SS Error returns
|
|
.PP
|
|
If an error occurs then there will be an HTTP error status (e.g.
|
|
500) and the body of the response will contain a JSON encoded error
|
|
object, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]error\[dq]: \[dq]Expecting string value for key \[rs]\[dq]remote\[rs]\[dq] (was float64)\[dq],
|
|
\[dq]input\[dq]: {
|
|
\[dq]fs\[dq]: \[dq]/tmp\[dq],
|
|
\[dq]remote\[dq]: 3
|
|
},
|
|
\[dq]status\[dq]: 400
|
|
\[dq]path\[dq]: \[dq]operations/rmdir\[dq],
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The keys in the error response are - error - error string - input - the
|
|
input parameters to the call - status - the HTTP status code - path -
|
|
the path of the call
|
|
.SS CORS
|
|
.PP
|
|
The sever implements basic CORS support and allows all origins for that.
|
|
The response to a preflight OPTIONS request will echo the requested
|
|
\[dq]Access-Control-Request-Headers\[dq] back.
|
|
.SS Using POST with URL parameters only
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl -X POST \[aq]http://localhost:5572/rc/noop?potato=1&sausage=2\[aq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Response
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]potato\[dq]: \[dq]1\[dq],
|
|
\[dq]sausage\[dq]: \[dq]2\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Here is what an error response looks like:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl -X POST \[aq]http://localhost:5572/rc/error?potato=1&sausage=2\[aq]
|
|
\f[R]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]error\[dq]: \[dq]arbitrary error on input map[potato:1 sausage:2]\[dq],
|
|
\[dq]input\[dq]: {
|
|
\[dq]potato\[dq]: \[dq]1\[dq],
|
|
\[dq]sausage\[dq]: \[dq]2\[dq]
|
|
}
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that curl doesn\[aq]t return errors to the shell unless you use the
|
|
\f[C]-f\f[R] option
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ curl -f -X POST \[aq]http://localhost:5572/rc/error?potato=1&sausage=2\[aq]
|
|
curl: (22) The requested URL returned error: 400 Bad Request
|
|
$ echo $?
|
|
22
|
|
\f[R]
|
|
.fi
|
|
.SS Using POST with a form
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl --data \[dq]potato=1\[dq] --data \[dq]sausage=2\[dq] http://localhost:5572/rc/noop
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Response
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]potato\[dq]: \[dq]1\[dq],
|
|
\[dq]sausage\[dq]: \[dq]2\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that you can combine these with URL parameters too with the POST
|
|
parameters taking precedence.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl --data \[dq]potato=1\[dq] --data \[dq]sausage=2\[dq] \[dq]http://localhost:5572/rc/noop?rutabaga=3&sausage=4\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Response
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]potato\[dq]: \[dq]1\[dq],
|
|
\[dq]rutabaga\[dq]: \[dq]3\[dq],
|
|
\[dq]sausage\[dq]: \[dq]4\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.SS Using POST with a JSON blob
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl -H \[dq]Content-Type: application/json\[dq] -X POST -d \[aq]{\[dq]potato\[dq]:2,\[dq]sausage\[dq]:1}\[aq] http://localhost:5572/rc/noop
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
response
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]password\[dq]: \[dq]xyz\[dq],
|
|
\[dq]username\[dq]: \[dq]xyz\[dq]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This can be combined with URL parameters too if required.
|
|
The JSON blob takes precedence.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl -H \[dq]Content-Type: application/json\[dq] -X POST -d \[aq]{\[dq]potato\[dq]:2,\[dq]sausage\[dq]:1}\[aq] \[aq]http://localhost:5572/rc/noop?rutabaga=3&potato=4\[aq]
|
|
\f[R]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]potato\[dq]: 2,
|
|
\[dq]rutabaga\[dq]: \[dq]3\[dq],
|
|
\[dq]sausage\[dq]: 1
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.SS Debugging rclone with pprof
|
|
.PP
|
|
If you use the \f[C]--rc\f[R] flag this will also enable the use of the
|
|
go profiling tools on the same port.
|
|
.PP
|
|
To use these, first install go (https://golang.org/doc/install).
|
|
.SS Debugging memory use
|
|
.PP
|
|
To profile rclone\[aq]s memory use you can run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
go tool pprof -web http://localhost:5572/debug/pprof/heap
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This should open a page in your browser showing what is using what
|
|
memory.
|
|
.PP
|
|
You can also use the \f[C]-text\f[R] flag to produce a textual summary
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ go tool pprof -text http://localhost:5572/debug/pprof/heap
|
|
Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
|
|
flat flat% sum% cum cum%
|
|
1024.03kB 66.62% 66.62% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode
|
|
513kB 33.38% 100% 513kB 33.38% net/http.newBufioWriterSize
|
|
0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/all.init
|
|
0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve.init
|
|
0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve/restic.init
|
|
0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2.init
|
|
0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init
|
|
0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init.0
|
|
0 0% 100% 1024.03kB 66.62% main.init
|
|
0 0% 100% 513kB 33.38% net/http.(*conn).readRequest
|
|
0 0% 100% 513kB 33.38% net/http.(*conn).serve
|
|
0 0% 100% 1024.03kB 66.62% runtime.main
|
|
\f[R]
|
|
.fi
|
|
.SS Debugging go routine leaks
|
|
.PP
|
|
Memory leaks are most often caused by go routine leaks keeping memory
|
|
alive which should have been garbage collected.
|
|
.PP
|
|
See all active go routines using
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl http://localhost:5572/debug/pprof/goroutine?debug=1
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or go to http://localhost:5572/debug/pprof/goroutine?debug=1 in your
|
|
browser.
|
|
.SS Other profiles to look at
|
|
.PP
|
|
You can see a summary of profiles available at
|
|
http://localhost:5572/debug/pprof/
|
|
.PP
|
|
Here is how to use some of them:
|
|
.IP \[bu] 2
|
|
Memory: \f[C]go tool pprof http://localhost:5572/debug/pprof/heap\f[R]
|
|
.IP \[bu] 2
|
|
Go routines:
|
|
\f[C]curl http://localhost:5572/debug/pprof/goroutine?debug=1\f[R]
|
|
.IP \[bu] 2
|
|
30-second CPU profile:
|
|
\f[C]go tool pprof http://localhost:5572/debug/pprof/profile\f[R]
|
|
.IP \[bu] 2
|
|
5-second execution trace:
|
|
\f[C]wget http://localhost:5572/debug/pprof/trace?seconds=5\f[R]
|
|
.IP \[bu] 2
|
|
Goroutine blocking profile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enable first with:
|
|
\f[C]rclone rc debug/set-block-profile-rate rate=1\f[R] (docs)
|
|
.IP \[bu] 2
|
|
\f[C]go tool pprof http://localhost:5572/debug/pprof/block\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
Contended mutexes:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enable first with:
|
|
\f[C]rclone rc debug/set-mutex-profile-fraction rate=1\f[R] (docs)
|
|
.IP \[bu] 2
|
|
\f[C]go tool pprof http://localhost:5572/debug/pprof/mutex\f[R]
|
|
.RE
|
|
.PP
|
|
See the net/http/pprof docs (https://golang.org/pkg/net/http/pprof/) for
|
|
more info on how to use the profiling and for a general overview see the
|
|
Go team\[aq]s blog post on profiling go
|
|
programs (https://blog.golang.org/profiling-go-programs).
|
|
.PP
|
|
The profiling hook is zero overhead unless it is
|
|
used (https://stackoverflow.com/q/26545159/164234).
|
|
.SH Overview of cloud storage systems
|
|
.PP
|
|
Each cloud storage system is slightly different.
|
|
Rclone attempts to provide a unified interface to them, but some
|
|
underlying differences show through.
|
|
.SS Features
|
|
.PP
|
|
Here is an overview of the major features of each cloud storage system.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c c c c.
|
|
T{
|
|
Name
|
|
T}@T{
|
|
Hash
|
|
T}@T{
|
|
ModTime
|
|
T}@T{
|
|
Case Insensitive
|
|
T}@T{
|
|
Duplicate Files
|
|
T}@T{
|
|
MIME Type
|
|
T}
|
|
_
|
|
T{
|
|
1Fichier
|
|
T}@T{
|
|
Whirlpool
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
Akamai Netstorage
|
|
T}@T{
|
|
MD5, SHA256
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
Amazon Drive
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
Amazon S3 (or S3 compatible)
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Backblaze B2
|
|
T}@T{
|
|
SHA1
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Box
|
|
T}@T{
|
|
SHA1
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Citrix ShareFile
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Dropbox
|
|
T}@T{
|
|
DBHASH \[S1]
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Enterprise File Fabric
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
FTP
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Google Cloud Storage
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Google Drive
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Google Photos
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
HDFS
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
HTTP
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
Hubic
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Jottacloud
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
Koofr
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Mail.ru Cloud
|
|
T}@T{
|
|
Mailru \[u2076]
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Mega
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Memory
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Microsoft Azure Blob Storage
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Microsoft OneDrive
|
|
T}@T{
|
|
SHA1 \[u2075]
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
OpenDrive
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Partial \[u2078]
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
OpenStack Swift
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
pCloud
|
|
T}@T{
|
|
MD5, SHA1 \[u2077]
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
W
|
|
T}
|
|
T{
|
|
premiumize.me
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
put.io
|
|
T}@T{
|
|
CRC-32
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
QingStor
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Seafile
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
SFTP
|
|
T}@T{
|
|
MD5, SHA1 \[S2]
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Depends
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Sia
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
SugarSync
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Storj
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Uptobox
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
WebDAV
|
|
T}@T{
|
|
MD5, SHA1 \[S3]
|
|
T}@T{
|
|
Yes \[u2074]
|
|
T}@T{
|
|
Depends
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
Yandex Disk
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
Zoho WorkDrive
|
|
T}@T{
|
|
-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
T{
|
|
The local filesystem
|
|
T}@T{
|
|
All
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Depends
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
-
|
|
T}
|
|
.TE
|
|
.SS Notes
|
|
.PP
|
|
\[S1] Dropbox supports its own custom
|
|
hash (https://www.dropbox.com/developers/reference/content-hash).
|
|
This is an SHA256 sum of all the 4 MiB block SHA256s.
|
|
.PP
|
|
\[S2] SFTP supports checksums if the same login has shell access and
|
|
\f[C]md5sum\f[R] or \f[C]sha1sum\f[R] as well as \f[C]echo\f[R] are in
|
|
the remote\[aq]s PATH.
|
|
.PP
|
|
\[S3] WebDAV supports hashes when used with Owncloud and Nextcloud only.
|
|
.PP
|
|
\[u2074] WebDAV supports modtimes when used with Owncloud and Nextcloud
|
|
only.
|
|
.PP
|
|
\[u2075] Microsoft OneDrive Personal supports SHA1 hashes, whereas
|
|
OneDrive for business and SharePoint server support Microsoft\[aq]s own
|
|
QuickXorHash (https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash).
|
|
.PP
|
|
\[u2076] Mail.ru uses its own modified SHA1 hash
|
|
.PP
|
|
\[u2077] pCloud only supports SHA1 (not MD5) in its EU region
|
|
.PP
|
|
\[u2078] Opendrive does not support creation of duplicate files using
|
|
their web client interface or other stock clients, but the underlying
|
|
storage platform has been determined to allow duplicate files, and it is
|
|
possible to create them with \f[C]rclone\f[R].
|
|
It may be that this is a mistake or an unsupported feature.
|
|
.SS Hash
|
|
.PP
|
|
The cloud storage system supports various hash types of the objects.
|
|
The hashes are used when transferring data as an integrity check and can
|
|
be specifically used with the \f[C]--checksum\f[R] flag in syncs and in
|
|
the \f[C]check\f[R] command.
|
|
.PP
|
|
To use the verify checksums when transferring between cloud storage
|
|
systems they must support a common hash type.
|
|
.SS ModTime
|
|
.PP
|
|
The cloud storage system supports setting modification times on objects.
|
|
If it does then this enables a using the modification times as part of
|
|
the sync.
|
|
If not then only the size will be checked by default, though the MD5SUM
|
|
can be checked with the \f[C]--checksum\f[R] flag.
|
|
.PP
|
|
All cloud storage systems support some kind of date on the object and
|
|
these will be set when transferring from the cloud storage system.
|
|
.SS Case Insensitive
|
|
.PP
|
|
If a cloud storage systems is case sensitive then it is possible to have
|
|
two files which differ only in case, e.g.
|
|
\f[C]file.txt\f[R] and \f[C]FILE.txt\f[R].
|
|
If a cloud storage system is case insensitive then that isn\[aq]t
|
|
possible.
|
|
.PP
|
|
This can cause problems when syncing between a case insensitive system
|
|
and a case sensitive system.
|
|
The symptom of this is that no matter how many times you run the sync it
|
|
never completes fully.
|
|
.PP
|
|
The local filesystem and SFTP may or may not be case sensitive depending
|
|
on OS.
|
|
.IP \[bu] 2
|
|
Windows - usually case insensitive, though case is preserved
|
|
.IP \[bu] 2
|
|
OSX - usually case insensitive, though it is possible to format case
|
|
sensitive
|
|
.IP \[bu] 2
|
|
Linux - usually case sensitive, but there are case insensitive file
|
|
systems (e.g.
|
|
FAT formatted USB keys)
|
|
.PP
|
|
Most of the time this doesn\[aq]t cause any problems as people tend to
|
|
avoid files whose name differs only by case even on case sensitive
|
|
systems.
|
|
.SS Duplicate files
|
|
.PP
|
|
If a cloud storage system allows duplicate files then it can have two
|
|
objects with the same name.
|
|
.PP
|
|
This confuses rclone greatly when syncing - use the
|
|
\f[C]rclone dedupe\f[R] command to rename or remove duplicates.
|
|
.SS Restricted filenames
|
|
.PP
|
|
Some cloud storage systems might have restrictions on the characters
|
|
that are usable in file or directory names.
|
|
When \f[C]rclone\f[R] detects such a name during a file upload, it will
|
|
transparently replace the restricted characters with similar looking
|
|
Unicode characters.
|
|
To handle the different sets of restricted characters for different
|
|
backends, rclone uses something it calls encoding.
|
|
.PP
|
|
This process is designed to avoid ambiguous file names as much as
|
|
possible and allow to move files between many cloud storage systems
|
|
transparently.
|
|
.PP
|
|
The name shown by \f[C]rclone\f[R] to the user or during log output will
|
|
only contain a minimal set of replaced characters to ensure correct
|
|
formatting and not necessarily the actual name used on the cloud
|
|
storage.
|
|
.PP
|
|
This transformation is reversed when downloading a file or parsing
|
|
\f[C]rclone\f[R] arguments.
|
|
For example, when uploading a file named \f[C]my file?.txt\f[R] to
|
|
Onedrive, it will be displayed as \f[C]my file?.txt\f[R] on the console,
|
|
but stored as \f[C]my file\[uFF1F].txt\f[R] to Onedrive (the \f[C]?\f[R]
|
|
gets replaced by the similar looking \f[C]\[uFF1F]\f[R] character, the
|
|
so-called \[dq]fullwidth question mark\[dq]).
|
|
The reverse transformation allows to read a file
|
|
\f[C]unusual/name.txt\f[R] from Google Drive, by passing the name
|
|
\f[C]unusual\[uFF0F]name.txt\f[R] on the command line (the \f[C]/\f[R]
|
|
needs to be replaced by the similar looking \f[C]\[uFF0F]\f[R]
|
|
character).
|
|
.SS Caveats
|
|
.PP
|
|
The filename encoding system works well in most cases, at least where
|
|
file names are written in English or similar languages.
|
|
You might not even notice it: It just works.
|
|
In some cases it may lead to issues, though.
|
|
E.g.
|
|
when file names are written in Chinese, or Japanese, where it is always
|
|
the Unicode fullwidth variants of the punctuation marks that are used.
|
|
.PP
|
|
On Windows, the characters \f[C]:\f[R], \f[C]*\f[R] and \f[C]?\f[R] are
|
|
examples of restricted characters.
|
|
If these are used in filenames on a remote that supports it, Rclone will
|
|
transparently convert them to their fullwidth Unicode variants
|
|
\f[C]\[uFF0A]\f[R], \f[C]\[uFF1F]\f[R] and \f[C]\[uFF1A]\f[R] when
|
|
downloading to Windows, and back again when uploading.
|
|
This way files with names that are not allowed on Windows can still be
|
|
stored.
|
|
.PP
|
|
However, if you have files on your Windows system originally with these
|
|
same Unicode characters in their names, they will be included in the
|
|
same conversion process.
|
|
E.g.
|
|
if you create a file in your Windows filesystem with name
|
|
\f[C]Test\[uFF1A]1.jpg\f[R], where \f[C]\[uFF1A]\f[R] is the Unicode
|
|
fullwidth colon symbol, and use rclone to upload it to Google Drive,
|
|
which supports regular \f[C]:\f[R] (halfwidth question mark), rclone
|
|
will replace the fullwidth \f[C]:\f[R] with the halfwidth \f[C]:\f[R]
|
|
and store the file as \f[C]Test:1.jpg\f[R] in Google Drive.
|
|
Since both Windows and Google Drive allows the name
|
|
\f[C]Test\[uFF1A]1.jpg\f[R], it would probably be better if rclone just
|
|
kept the name as is in this case.
|
|
.PP
|
|
With the opposite situation; if you have a file named
|
|
\f[C]Test:1.jpg\f[R], in your Google Drive, e.g.
|
|
uploaded from a Linux system where \f[C]:\f[R] is valid in file names.
|
|
Then later use rclone to copy this file to your Windows computer you
|
|
will notice that on your local disk it gets renamed to
|
|
\f[C]Test\[uFF1A]1.jpg\f[R].
|
|
The original filename is not legal on Windows, due to the \f[C]:\f[R],
|
|
and rclone therefore renames it to make the copy possible.
|
|
That is all good.
|
|
However, this can also lead to an issue: If you already had a
|
|
\f[I]different\f[R] file named \f[C]Test\[uFF1A]1.jpg\f[R] on Windows,
|
|
and then use rclone to copy either way.
|
|
Rclone will then treat the file originally named \f[C]Test:1.jpg\f[R] on
|
|
Google Drive and the file originally named \f[C]Test\[uFF1A]1.jpg\f[R]
|
|
on Windows as the same file, and replace the contents from one with the
|
|
other.
|
|
.PP
|
|
Its virtually impossible to handle all cases like these correctly in all
|
|
situations, but by customizing the encoding option, changing the set of
|
|
characters that rclone should convert, you should be able to create a
|
|
configuration that works well for your specific situation.
|
|
See also the
|
|
example (https://rclone.org/overview/#encoding-example-windows) below.
|
|
.PP
|
|
(Windows was used as an example of a file system with many restricted
|
|
characters, and Google drive a storage system with few.)
|
|
.SS Default restricted characters
|
|
.PP
|
|
The table below shows the characters that are replaced by default.
|
|
.PP
|
|
When a replacement character is found in a filename, this character will
|
|
be escaped with the \f[C]\[u201B]\f[R] character to avoid ambiguous file
|
|
names.
|
|
(e.g.
|
|
a file named \f[C]\[u2400].txt\f[R] would shown as
|
|
\f[C]\[u201B]\[u2400].txt\f[R])
|
|
.PP
|
|
Each cloud storage backend can use a different set of characters, which
|
|
will be specified in the documentation for each backend.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
NUL
|
|
T}@T{
|
|
0x00
|
|
T}@T{
|
|
\[u2400]
|
|
T}
|
|
T{
|
|
SOH
|
|
T}@T{
|
|
0x01
|
|
T}@T{
|
|
\[u2401]
|
|
T}
|
|
T{
|
|
STX
|
|
T}@T{
|
|
0x02
|
|
T}@T{
|
|
\[u2402]
|
|
T}
|
|
T{
|
|
ETX
|
|
T}@T{
|
|
0x03
|
|
T}@T{
|
|
\[u2403]
|
|
T}
|
|
T{
|
|
EOT
|
|
T}@T{
|
|
0x04
|
|
T}@T{
|
|
\[u2404]
|
|
T}
|
|
T{
|
|
ENQ
|
|
T}@T{
|
|
0x05
|
|
T}@T{
|
|
\[u2405]
|
|
T}
|
|
T{
|
|
ACK
|
|
T}@T{
|
|
0x06
|
|
T}@T{
|
|
\[u2406]
|
|
T}
|
|
T{
|
|
BEL
|
|
T}@T{
|
|
0x07
|
|
T}@T{
|
|
\[u2407]
|
|
T}
|
|
T{
|
|
BS
|
|
T}@T{
|
|
0x08
|
|
T}@T{
|
|
\[u2408]
|
|
T}
|
|
T{
|
|
HT
|
|
T}@T{
|
|
0x09
|
|
T}@T{
|
|
\[u2409]
|
|
T}
|
|
T{
|
|
LF
|
|
T}@T{
|
|
0x0A
|
|
T}@T{
|
|
\[u240A]
|
|
T}
|
|
T{
|
|
VT
|
|
T}@T{
|
|
0x0B
|
|
T}@T{
|
|
\[u240B]
|
|
T}
|
|
T{
|
|
FF
|
|
T}@T{
|
|
0x0C
|
|
T}@T{
|
|
\[u240C]
|
|
T}
|
|
T{
|
|
CR
|
|
T}@T{
|
|
0x0D
|
|
T}@T{
|
|
\[u240D]
|
|
T}
|
|
T{
|
|
SO
|
|
T}@T{
|
|
0x0E
|
|
T}@T{
|
|
\[u240E]
|
|
T}
|
|
T{
|
|
SI
|
|
T}@T{
|
|
0x0F
|
|
T}@T{
|
|
\[u240F]
|
|
T}
|
|
T{
|
|
DLE
|
|
T}@T{
|
|
0x10
|
|
T}@T{
|
|
\[u2410]
|
|
T}
|
|
T{
|
|
DC1
|
|
T}@T{
|
|
0x11
|
|
T}@T{
|
|
\[u2411]
|
|
T}
|
|
T{
|
|
DC2
|
|
T}@T{
|
|
0x12
|
|
T}@T{
|
|
\[u2412]
|
|
T}
|
|
T{
|
|
DC3
|
|
T}@T{
|
|
0x13
|
|
T}@T{
|
|
\[u2413]
|
|
T}
|
|
T{
|
|
DC4
|
|
T}@T{
|
|
0x14
|
|
T}@T{
|
|
\[u2414]
|
|
T}
|
|
T{
|
|
NAK
|
|
T}@T{
|
|
0x15
|
|
T}@T{
|
|
\[u2415]
|
|
T}
|
|
T{
|
|
SYN
|
|
T}@T{
|
|
0x16
|
|
T}@T{
|
|
\[u2416]
|
|
T}
|
|
T{
|
|
ETB
|
|
T}@T{
|
|
0x17
|
|
T}@T{
|
|
\[u2417]
|
|
T}
|
|
T{
|
|
CAN
|
|
T}@T{
|
|
0x18
|
|
T}@T{
|
|
\[u2418]
|
|
T}
|
|
T{
|
|
EM
|
|
T}@T{
|
|
0x19
|
|
T}@T{
|
|
\[u2419]
|
|
T}
|
|
T{
|
|
SUB
|
|
T}@T{
|
|
0x1A
|
|
T}@T{
|
|
\[u241A]
|
|
T}
|
|
T{
|
|
ESC
|
|
T}@T{
|
|
0x1B
|
|
T}@T{
|
|
\[u241B]
|
|
T}
|
|
T{
|
|
FS
|
|
T}@T{
|
|
0x1C
|
|
T}@T{
|
|
\[u241C]
|
|
T}
|
|
T{
|
|
GS
|
|
T}@T{
|
|
0x1D
|
|
T}@T{
|
|
\[u241D]
|
|
T}
|
|
T{
|
|
RS
|
|
T}@T{
|
|
0x1E
|
|
T}@T{
|
|
\[u241E]
|
|
T}
|
|
T{
|
|
US
|
|
T}@T{
|
|
0x1F
|
|
T}@T{
|
|
\[u241F]
|
|
T}
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
T{
|
|
DEL
|
|
T}@T{
|
|
0x7F
|
|
T}@T{
|
|
\[u2421]
|
|
T}
|
|
.TE
|
|
.PP
|
|
The default encoding will also encode these file names as they are
|
|
problematic with many cloud storage systems.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c.
|
|
T{
|
|
File name
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\&.
|
|
T}@T{
|
|
\[uFF0E]
|
|
T}
|
|
T{
|
|
\&..
|
|
T}@T{
|
|
\[uFF0E]\[uFF0E]
|
|
T}
|
|
.TE
|
|
.SS Invalid UTF-8 bytes
|
|
.PP
|
|
Some backends only support a sequence of well formed UTF-8 bytes as file
|
|
or directory names.
|
|
.PP
|
|
In this case all invalid UTF-8 bytes will be replaced with a quoted
|
|
representation of the byte value to allow uploading a file to such a
|
|
backend.
|
|
For example, the invalid byte \f[C]0xFE\f[R] will be encoded as
|
|
\f[C]\[u201B]FE\f[R].
|
|
.PP
|
|
A common source of invalid UTF-8 bytes are local filesystems, that store
|
|
names in a different encoding than UTF-8 or UTF-16, like latin1.
|
|
See the local filenames (https://rclone.org/local/#filenames) section
|
|
for details.
|
|
.SS Encoding option
|
|
.PP
|
|
Most backends have an encoding option, specified as a flag
|
|
\f[C]--backend-encoding\f[R] where \f[C]backend\f[R] is the name of the
|
|
backend, or as a config parameter \f[C]encoding\f[R] (you\[aq]ll need to
|
|
select the Advanced config in \f[C]rclone config\f[R] to see it).
|
|
.PP
|
|
This will have default value which encodes and decodes characters in
|
|
such a way as to preserve the maximum number of characters (see above).
|
|
.PP
|
|
However this can be incorrect in some scenarios, for example if you have
|
|
a Windows file system with Unicode fullwidth characters
|
|
\f[C]\[uFF0A]\f[R], \f[C]\[uFF1F]\f[R] or \f[C]\[uFF1A]\f[R], that you
|
|
want to remain as those characters on the remote rather than being
|
|
translated to regular (halfwidth) \f[C]*\f[R], \f[C]?\f[R] and
|
|
\f[C]:\f[R].
|
|
.PP
|
|
The \f[C]--backend-encoding\f[R] flags allow you to change that.
|
|
You can disable the encoding completely with
|
|
\f[C]--backend-encoding None\f[R] or set \f[C]encoding = None\f[R] in
|
|
the config file.
|
|
.PP
|
|
Encoding takes a comma separated list of encodings.
|
|
You can see the list of all possible values by passing an invalid value
|
|
to this flag, e.g.
|
|
\f[C]--local-encoding \[dq]help\[dq]\f[R].
|
|
The command \f[C]rclone help flags encoding\f[R] will show you the
|
|
defaults for the backends.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l l.
|
|
T{
|
|
Encoding
|
|
T}@T{
|
|
Characters
|
|
T}
|
|
_
|
|
T{
|
|
Asterisk
|
|
T}@T{
|
|
\f[C]*\f[R]
|
|
T}
|
|
T{
|
|
BackQuote
|
|
T}@T{
|
|
\f[C]\[ga]\f[R]
|
|
T}
|
|
T{
|
|
BackSlash
|
|
T}@T{
|
|
\f[C]\[rs]\f[R]
|
|
T}
|
|
T{
|
|
Colon
|
|
T}@T{
|
|
\f[C]:\f[R]
|
|
T}
|
|
T{
|
|
CrLf
|
|
T}@T{
|
|
CR 0x0D, LF 0x0A
|
|
T}
|
|
T{
|
|
Ctl
|
|
T}@T{
|
|
All control characters 0x00-0x1F
|
|
T}
|
|
T{
|
|
Del
|
|
T}@T{
|
|
DEL 0x7F
|
|
T}
|
|
T{
|
|
Dollar
|
|
T}@T{
|
|
\f[C]$\f[R]
|
|
T}
|
|
T{
|
|
Dot
|
|
T}@T{
|
|
\f[C].\f[R] or \f[C]..\f[R] as entire string
|
|
T}
|
|
T{
|
|
DoubleQuote
|
|
T}@T{
|
|
\f[C]\[dq]\f[R]
|
|
T}
|
|
T{
|
|
Hash
|
|
T}@T{
|
|
\f[C]#\f[R]
|
|
T}
|
|
T{
|
|
InvalidUtf8
|
|
T}@T{
|
|
An invalid UTF-8 character (e.g.
|
|
latin1)
|
|
T}
|
|
T{
|
|
LeftCrLfHtVt
|
|
T}@T{
|
|
CR 0x0D, LF 0x0A,HT 0x09, VT 0x0B on the left of a string
|
|
T}
|
|
T{
|
|
LeftPeriod
|
|
T}@T{
|
|
\f[C].\f[R] on the left of a string
|
|
T}
|
|
T{
|
|
LeftSpace
|
|
T}@T{
|
|
SPACE on the left of a string
|
|
T}
|
|
T{
|
|
LeftTilde
|
|
T}@T{
|
|
\f[C]\[ti]\f[R] on the left of a string
|
|
T}
|
|
T{
|
|
LtGt
|
|
T}@T{
|
|
\f[C]<\f[R], \f[C]>\f[R]
|
|
T}
|
|
T{
|
|
None
|
|
T}@T{
|
|
No characters are encoded
|
|
T}
|
|
T{
|
|
Percent
|
|
T}@T{
|
|
\f[C]%\f[R]
|
|
T}
|
|
T{
|
|
Pipe
|
|
T}@T{
|
|
|
|
|
T}
|
|
T{
|
|
Question
|
|
T}@T{
|
|
\f[C]?\f[R]
|
|
T}
|
|
T{
|
|
RightCrLfHtVt
|
|
T}@T{
|
|
CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string
|
|
T}
|
|
T{
|
|
RightPeriod
|
|
T}@T{
|
|
\f[C].\f[R] on the right of a string
|
|
T}
|
|
T{
|
|
RightSpace
|
|
T}@T{
|
|
SPACE on the right of a string
|
|
T}
|
|
T{
|
|
SingleQuote
|
|
T}@T{
|
|
\f[C]\[aq]\f[R]
|
|
T}
|
|
T{
|
|
Slash
|
|
T}@T{
|
|
\f[C]/\f[R]
|
|
T}
|
|
T{
|
|
SquareBracket
|
|
T}@T{
|
|
\f[C][\f[R], \f[C]]\f[R]
|
|
T}
|
|
.TE
|
|
.SS Encoding example: FTP
|
|
.PP
|
|
To take a specific example, the FTP backend\[aq]s default encoding is
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--ftp-encoding \[dq]Slash,Del,Ctl,RightSpace,Dot\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
However, let\[aq]s say the FTP server is running on Windows and
|
|
can\[aq]t have any of the invalid Windows characters in file names.
|
|
You are backing up Linux servers to this FTP server which do have those
|
|
characters in file names.
|
|
So you would add the Windows set which are
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
to the existing ones, giving:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This can be specified using the \f[C]--ftp-encoding\f[R] flag or using
|
|
an \f[C]encoding\f[R] parameter in the config file.
|
|
.SS Encoding example: Windows
|
|
.PP
|
|
As a nother example, take a Windows system where there is a file with
|
|
name \f[C]Test\[uFF1A]1.jpg\f[R], where \f[C]\[uFF1A]\f[R] is the
|
|
Unicode fullwidth colon symbol.
|
|
When using rclone to copy this to a remote which supports \f[C]:\f[R],
|
|
the regular (halfwidth) colon (such as Google Drive), you will notice
|
|
that the file gets renamed to \f[C]Test:1.jpg\f[R].
|
|
.PP
|
|
To avoid this you can change the set of characters rclone should convert
|
|
for the local filesystem, using command-line argument
|
|
\f[C]--local-encoding\f[R].
|
|
Rclone\[aq]s default behavior on Windows corresponds to
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--local-encoding \[dq]Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you want to use fullwidth characters \f[C]\[uFF1A]\f[R],
|
|
\f[C]\[uFF0A]\f[R] and \f[C]\[uFF1F]\f[R] in your filenames without
|
|
rclone changing them when uploading to a remote, then set the same as
|
|
the default value but without \f[C]Colon,Question,Asterisk\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--local-encoding \[dq]Slash,LtGt,DoubleQuote,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Alternatively, you can disable the conversion of any characters with
|
|
\f[C]--local-encoding None\f[R].
|
|
.PP
|
|
Instead of using command-line argument \f[C]--local-encoding\f[R], you
|
|
may also set it as environment
|
|
variable (https://rclone.org/docs/#environment-variables)
|
|
\f[C]RCLONE_LOCAL_ENCODING\f[R], or
|
|
configure (https://rclone.org/docs/#configure) a remote of type
|
|
\f[C]local\f[R] in your config, and set the \f[C]encoding\f[R] option
|
|
there.
|
|
.PP
|
|
The risk by doing this is that if you have a filename with the regular
|
|
(halfwidth) \f[C]:\f[R], \f[C]*\f[R] and \f[C]?\f[R] in your cloud
|
|
storage, and you try to download it to your Windows filesystem, this
|
|
will fail.
|
|
These characters are not valid in filenames on Windows, and you have
|
|
told rclone not to work around this by converting them to valid
|
|
fullwidth variants.
|
|
.SS MIME Type
|
|
.PP
|
|
MIME types (also known as media types) classify types of documents using
|
|
a simple text classification, e.g.
|
|
\f[C]text/html\f[R] or \f[C]application/pdf\f[R].
|
|
.PP
|
|
Some cloud storage systems support reading (\f[C]R\f[R]) the MIME type
|
|
of objects and some support writing (\f[C]W\f[R]) the MIME type of
|
|
objects.
|
|
.PP
|
|
The MIME type can be important if you are serving files directly to HTTP
|
|
from the storage system.
|
|
.PP
|
|
If you are copying from a remote which supports reading (\f[C]R\f[R]) to
|
|
a remote which supports writing (\f[C]W\f[R]) then rclone will preserve
|
|
the MIME types.
|
|
Otherwise they will be guessed from the extension, or the remote itself
|
|
may assign the MIME type.
|
|
.SS Optional Features
|
|
.PP
|
|
All rclone remotes support a base command set.
|
|
Other features depend upon backend-specific capabilities.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c c c c c c c c c.
|
|
T{
|
|
Name
|
|
T}@T{
|
|
Purge
|
|
T}@T{
|
|
Copy
|
|
T}@T{
|
|
Move
|
|
T}@T{
|
|
DirMove
|
|
T}@T{
|
|
CleanUp
|
|
T}@T{
|
|
ListR
|
|
T}@T{
|
|
StreamUpload
|
|
T}@T{
|
|
LinkSharing
|
|
T}@T{
|
|
About
|
|
T}@T{
|
|
EmptyDir
|
|
T}
|
|
_
|
|
T{
|
|
1Fichier
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Amazon Drive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Amazon S3
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Backblaze B2
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Box
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes \[dd]\[dd]
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Citrix ShareFile
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Dropbox
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Enterprise File Fabric
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
FTP
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Google Cloud Storage
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Google Drive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Google Photos
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
HDFS
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
HTTP
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Hubic
|
|
T}@T{
|
|
Yes \[dg]
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Jottacloud
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Mail.ru Cloud
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Mega
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Memory
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Microsoft Azure Blob Storage
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Microsoft OneDrive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
OpenDrive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
OpenStack Swift
|
|
T}@T{
|
|
Yes \[dg]
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
pCloud
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
premiumize.me
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
put.io
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
QingStor
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Seafile
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
SFTP
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
SugarSync
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Storj
|
|
T}@T{
|
|
Yes \[dg]
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Uptobox
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
WebDAV
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes \[dd]
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Yandex Disk
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Zoho WorkDrive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
The local filesystem
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
.TE
|
|
.SS Purge
|
|
.PP
|
|
This deletes a directory quicker than just deleting all the files in the
|
|
directory.
|
|
.PP
|
|
\[dg] Note Swift, Hubic, and Storj implement this in order to delete
|
|
directory markers but they don\[aq]t actually have a quicker way of
|
|
deleting files other than deleting them individually.
|
|
.PP
|
|
\[dd] StreamUpload is not supported with Nextcloud
|
|
.SS Copy
|
|
.PP
|
|
Used when copying an object to and from the same remote.
|
|
This known as a server-side copy so you can copy a file without
|
|
downloading it and uploading it again.
|
|
It is used if you use \f[C]rclone copy\f[R] or \f[C]rclone move\f[R] if
|
|
the remote doesn\[aq]t support \f[C]Move\f[R] directly.
|
|
.PP
|
|
If the server doesn\[aq]t support \f[C]Copy\f[R] directly then for copy
|
|
operations the file is downloaded then re-uploaded.
|
|
.SS Move
|
|
.PP
|
|
Used when moving/renaming an object on the same remote.
|
|
This is known as a server-side move of a file.
|
|
This is used in \f[C]rclone move\f[R] if the server doesn\[aq]t support
|
|
\f[C]DirMove\f[R].
|
|
.PP
|
|
If the server isn\[aq]t capable of \f[C]Move\f[R] then rclone simulates
|
|
it with \f[C]Copy\f[R] then delete.
|
|
If the server doesn\[aq]t support \f[C]Copy\f[R] then rclone will
|
|
download the file and re-upload it.
|
|
.SS DirMove
|
|
.PP
|
|
This is used to implement \f[C]rclone move\f[R] to move a directory if
|
|
possible.
|
|
If it isn\[aq]t then it will use \f[C]Move\f[R] on each file (which
|
|
falls back to \f[C]Copy\f[R] then download and upload - see
|
|
\f[C]Move\f[R] section).
|
|
.SS CleanUp
|
|
.PP
|
|
This is used for emptying the trash for a remote by
|
|
\f[C]rclone cleanup\f[R].
|
|
.PP
|
|
If the server can\[aq]t do \f[C]CleanUp\f[R] then
|
|
\f[C]rclone cleanup\f[R] will return an error.
|
|
.PP
|
|
\[dd]\[dd] Note that while Box implements this it has to delete every
|
|
file individually so it will be slower than emptying the trash via the
|
|
WebUI
|
|
.SS ListR
|
|
.PP
|
|
The remote supports a recursive list to list all the contents beneath a
|
|
directory quickly.
|
|
This enables the \f[C]--fast-list\f[R] flag to work.
|
|
See the rclone docs (https://rclone.org/docs/#fast-list) for more
|
|
details.
|
|
.SS StreamUpload
|
|
.PP
|
|
Some remotes allow files to be uploaded without knowing the file size in
|
|
advance.
|
|
This allows certain operations to work without spooling the file to
|
|
local disk first, e.g.
|
|
\f[C]rclone rcat\f[R].
|
|
.SS LinkSharing
|
|
.PP
|
|
Sets the necessary permissions on a file or folder and prints a link
|
|
that allows others to access them, even if they don\[aq]t have an
|
|
account on the particular cloud provider.
|
|
.SS About
|
|
.PP
|
|
Rclone \f[C]about\f[R] prints quota information for a remote.
|
|
Typical output includes bytes used, free, quota and in trash.
|
|
.PP
|
|
If a remote lacks about capability \f[C]rclone about remote:\f[R]returns
|
|
an error.
|
|
.PP
|
|
Backends without about capability cannot determine free space for an
|
|
rclone mount, or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See rclone about command (https://rclone.org/commands/rclone_about/)
|
|
.SS EmptyDir
|
|
.PP
|
|
The remote supports empty directories.
|
|
See Limitations (https://rclone.org/bugs/#limitations) for details.
|
|
Most Object/Bucket-based remotes do not support this.
|
|
.SH Global Flags
|
|
.PP
|
|
This describes the global flags available to every rclone command split
|
|
into two groups, non backend and backend flags.
|
|
.SS Non Backend Flags
|
|
.PP
|
|
These flags are available for every command.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--ask-password Allow prompt for password for encrypted configuration (default true)
|
|
--auto-confirm If enabled, do not request console confirmation
|
|
--backup-dir string Make backups into hierarchy based in DIR
|
|
--bind string Local address to bind to for outgoing connections, IPv4, IPv6 or name
|
|
--buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi)
|
|
--bwlimit BwTimetable Bandwidth limit in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
|
|
--bwlimit-file BwTimetable Bandwidth limit per file in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
|
|
--ca-cert string CA certificate used to verify servers
|
|
--cache-dir string Directory rclone will use for caching (default \[dq]$HOME/.cache/rclone\[dq])
|
|
--check-first Do all the checks before starting transfers
|
|
--checkers int Number of checkers to run in parallel (default 8)
|
|
-c, --checksum Skip based on checksum (if available) & size, not mod-time & size
|
|
--client-cert string Client SSL certificate (PEM) for mutual TLS auth
|
|
--client-key string Client SSL private key (PEM) for mutual TLS auth
|
|
--compare-dest stringArray Include additional comma separated server-side paths during comparison
|
|
--config string Config file (default \[dq]$HOME/.config/rclone/rclone.conf\[dq])
|
|
--contimeout duration Connect timeout (default 1m0s)
|
|
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
|
|
--cpuprofile string Write cpu profile to file
|
|
--cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default \[dq]HARD\[dq])
|
|
--delete-after When synchronizing, delete files on destination after transferring (default)
|
|
--delete-before When synchronizing, delete files on destination before transferring
|
|
--delete-during When synchronizing, delete files during transfer
|
|
--delete-excluded Delete files on dest excluded from sync
|
|
--disable string Disable a comma separated list of features (use --disable help to see a list)
|
|
--disable-http2 Disable HTTP/2 in the global transport
|
|
-n, --dry-run Do a trial run with no permanent changes
|
|
--dscp string Set DSCP value to connections, value or name, e.g. CS1, LE, DF, AF21
|
|
--dump DumpFlags List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
|
|
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
|
--dump-headers Dump HTTP headers - may contain sensitive info
|
|
--error-on-no-transfer Sets exit code 9 if no files are transferred, useful in scripts
|
|
--exclude stringArray Exclude files matching pattern
|
|
--exclude-from stringArray Read exclude patterns from file (use - to read from stdin)
|
|
--exclude-if-present string Exclude directories if filename is present
|
|
--expect-continue-timeout duration Timeout when using expect / 100-continue in HTTP (default 1s)
|
|
--fast-list Use recursive list if available; uses more memory but fewer transactions
|
|
--files-from stringArray Read list of source-file names from file (use - to read from stdin)
|
|
--files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
|
|
-f, --filter stringArray Add a file-filtering rule
|
|
--filter-from stringArray Read filtering patterns from a file (use - to read from stdin)
|
|
--fs-cache-expire-duration duration Cache remotes for this long (0 to disable caching) (default 5m0s)
|
|
--fs-cache-expire-interval duration Interval to check for expired remotes (default 1m0s)
|
|
--header stringArray Set HTTP header for all transactions
|
|
--header-download stringArray Set HTTP header for download transactions
|
|
--header-upload stringArray Set HTTP header for upload transactions
|
|
--human-readable Print numbers in a human-readable format, sizes with suffix Ki|Mi|Gi|Ti|Pi
|
|
--ignore-case Ignore case in filters (case insensitive)
|
|
--ignore-case-sync Ignore case when synchronizing
|
|
--ignore-checksum Skip post copy check of checksums
|
|
--ignore-errors Delete even if there are I/O errors
|
|
--ignore-existing Skip all files that exist on destination
|
|
--ignore-size Ignore size when skipping use mod-time or checksum
|
|
-I, --ignore-times Don\[aq]t skip files that match size and time - transfer all files
|
|
--immutable Do not modify files, fail if existing files have been modified
|
|
--include stringArray Include files matching pattern
|
|
--include-from stringArray Read include patterns from file (use - to read from stdin)
|
|
-i, --interactive Enable interactive mode
|
|
--kv-lock-time duration Maximum time to keep key-value database locked by process (default 1s)
|
|
--log-file string Log everything to this file
|
|
--log-format string Comma separated list of log format options (default \[dq]date,time\[dq])
|
|
--log-level string Log level DEBUG|INFO|NOTICE|ERROR (default \[dq]NOTICE\[dq])
|
|
--log-systemd Activate systemd integration for the logger
|
|
--low-level-retries int Number of low level retries to do (default 10)
|
|
--max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
|
|
--max-backlog int Maximum number of objects in sync or check backlog (default 10000)
|
|
--max-delete int When synchronizing, limit the number of deletes (default -1)
|
|
--max-depth int If set limits the recursion depth to this (default -1)
|
|
--max-duration duration Maximum duration rclone will transfer data for
|
|
--max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
|
|
--max-stats-groups int Maximum number of stats groups to keep in memory, on max oldest is discarded (default 1000)
|
|
--max-transfer SizeSuffix Maximum size of data to transfer (default off)
|
|
--memprofile string Write memory profile to file
|
|
--min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
|
|
--min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
|
|
--modify-window duration Max time diff to be considered the same (default 1ns)
|
|
--multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size (default 250Mi)
|
|
--multi-thread-streams int Max number of streams to use for multi-thread downloads (default 4)
|
|
--no-check-certificate Do not verify the server SSL certificate (insecure)
|
|
--no-check-dest Don\[aq]t check the destination, copy regardless
|
|
--no-console Hide console window (supported on Windows only)
|
|
--no-gzip-encoding Don\[aq]t set Accept-Encoding: gzip
|
|
--no-traverse Don\[aq]t traverse destination file system on copy
|
|
--no-unicode-normalization Don\[aq]t normalize unicode characters in filenames
|
|
--no-update-modtime Don\[aq]t update destination mod-time if files identical
|
|
--order-by string Instructions on how to order the transfers, e.g. \[aq]size,descending\[aq]
|
|
--password-command SpaceSepList Command for supplying password for encrypted configuration
|
|
-P, --progress Show progress during transfer
|
|
--progress-terminal-title Show progress on the terminal title (requires -P/--progress)
|
|
-q, --quiet Print as little stuff as possible
|
|
--rc Enable the remote control server
|
|
--rc-addr string IPaddress:Port or :Port to bind server to (default \[dq]localhost:5572\[dq])
|
|
--rc-allow-origin string Set the allowed origin for CORS
|
|
--rc-baseurl string Prefix for URLs - leave blank for root
|
|
--rc-cert string SSL PEM key (concatenation of certificate and CA certificate)
|
|
--rc-client-ca string Client certificate authority to verify clients with
|
|
--rc-enable-metrics Enable prometheus metrics on /metrics
|
|
--rc-files string Path to local files to serve on the HTTP server
|
|
--rc-htpasswd string htpasswd file - if not provided no authentication is done
|
|
--rc-job-expire-duration duration Expire finished async jobs older than this value (default 1m0s)
|
|
--rc-job-expire-interval duration Interval to check for expired async jobs (default 10s)
|
|
--rc-key string SSL PEM Private key
|
|
--rc-max-header-bytes int Maximum size of request header (default 4096)
|
|
--rc-no-auth Don\[aq]t require auth for certain methods
|
|
--rc-pass string Password for authentication
|
|
--rc-realm string Realm for authentication (default \[dq]rclone\[dq])
|
|
--rc-serve Enable the serving of remote objects
|
|
--rc-server-read-timeout duration Timeout for server reading data (default 1h0m0s)
|
|
--rc-server-write-timeout duration Timeout for server writing data (default 1h0m0s)
|
|
--rc-template string User-specified template
|
|
--rc-user string User name for authentication
|
|
--rc-web-fetch-url string URL to fetch the releases for webgui (default \[dq]https://api.github.com/repos/rclone/rclone-webui-react/releases/latest\[dq])
|
|
--rc-web-gui Launch WebGUI on localhost
|
|
--rc-web-gui-force-update Force update to latest version of web gui
|
|
--rc-web-gui-no-open-browser Don\[aq]t open the browser automatically
|
|
--rc-web-gui-update Check and update to latest version of web gui
|
|
--refresh-times Refresh the modtime of remote files
|
|
--retries int Retry operations this many times if they fail (default 3)
|
|
--retries-sleep duration Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable)
|
|
--size-only Skip based on size only, not mod-time or checksum
|
|
--stats duration Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s)
|
|
--stats-file-name-length int Max file name length in stats (0 for no limit) (default 45)
|
|
--stats-log-level string Log level to show --stats output DEBUG|INFO|NOTICE|ERROR (default \[dq]INFO\[dq])
|
|
--stats-one-line Make the stats fit on one line
|
|
--stats-one-line-date Enable --stats-one-line and add current date/time prefix
|
|
--stats-one-line-date-format string Enable --stats-one-line-date and use custom formatted date: Enclose date string in double quotes (\[dq]), see https://golang.org/pkg/time/#Time.Format
|
|
--stats-unit string Show data rate in stats as either \[aq]bits\[aq] or \[aq]bytes\[aq] per second (default \[dq]bytes\[dq])
|
|
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
|
|
--suffix string Suffix to add to changed files
|
|
--suffix-keep-extension Preserve the extension when using --suffix
|
|
--syslog Use Syslog for logging
|
|
--syslog-facility string Facility for syslog, e.g. KERN,USER,... (default \[dq]DAEMON\[dq])
|
|
--temp-dir string Directory rclone will use for temporary files (default \[dq]/tmp\[dq])
|
|
--timeout duration IO idle timeout (default 5m0s)
|
|
--tpslimit float Limit HTTP transactions per second to this
|
|
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
|
|
--track-renames When synchronizing, track file renames and do a server-side move if possible
|
|
--track-renames-strategy string Strategies to use when synchronizing using track-renames hash|modtime|leaf (default \[dq]hash\[dq])
|
|
--transfers int Number of file transfers to run in parallel (default 4)
|
|
-u, --update Skip files that are newer on the destination
|
|
--use-cookies Enable session cookiejar
|
|
--use-json-log Use json log format
|
|
--use-mmap Use mmap allocator (see docs)
|
|
--use-server-modtime Use server modified time instead of object metadata
|
|
--user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.58.0\[dq])
|
|
-v, --verbose count Print lots more stuff (repeat for more)
|
|
\f[R]
|
|
.fi
|
|
.SS Backend Flags
|
|
.PP
|
|
These flags are available for every command.
|
|
They control the backends and may be set in the config file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--acd-auth-url string Auth server URL
|
|
--acd-client-id string OAuth Client Id
|
|
--acd-client-secret string OAuth Client Secret
|
|
--acd-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8,Dot)
|
|
--acd-templink-threshold SizeSuffix Files >= this size will be downloaded via their tempLink (default 9Gi)
|
|
--acd-token string OAuth Access Token as a JSON blob
|
|
--acd-token-url string Token server url
|
|
--acd-upload-wait-per-gb Duration Additional time per GiB to wait after a failed complete upload to see if it appears (default 3m0s)
|
|
--alias-remote string Remote or path to alias
|
|
--azureblob-access-tier string Access tier of blob: hot, cool or archive
|
|
--azureblob-account string Storage Account Name
|
|
--azureblob-archive-tier-delete Delete archive tier blobs before overwriting
|
|
--azureblob-chunk-size SizeSuffix Upload chunk size (default 4Mi)
|
|
--azureblob-disable-checksum Don\[aq]t store MD5 checksum with object metadata
|
|
--azureblob-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8)
|
|
--azureblob-endpoint string Endpoint for the service
|
|
--azureblob-key string Storage Account Key
|
|
--azureblob-list-chunk int Size of blob list (default 5000)
|
|
--azureblob-memory-pool-flush-time Duration How often internal memory buffer pools will be flushed (default 1m0s)
|
|
--azureblob-memory-pool-use-mmap Whether to use mmap buffers in internal memory pool
|
|
--azureblob-msi-client-id string Object ID of the user-assigned MSI to use, if any
|
|
--azureblob-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any
|
|
--azureblob-msi-object-id string Object ID of the user-assigned MSI to use, if any
|
|
--azureblob-no-head-object If set, do not do HEAD before GET when getting objects
|
|
--azureblob-public-access string Public access level of a container: blob or container
|
|
--azureblob-sas-url string SAS URL for container level access only
|
|
--azureblob-service-principal-file string Path to file containing credentials for use with a service principal
|
|
--azureblob-upload-concurrency int Concurrency for multipart uploads (default 16)
|
|
--azureblob-upload-cutoff string Cutoff for switching to chunked upload (<= 256 MiB) (deprecated)
|
|
--azureblob-use-emulator Uses local storage emulator if provided as \[aq]true\[aq]
|
|
--azureblob-use-msi Use a managed service identity to authenticate (only works in Azure)
|
|
--b2-account string Account ID or Application Key ID
|
|
--b2-chunk-size SizeSuffix Upload chunk size (default 96Mi)
|
|
--b2-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 4Gi)
|
|
--b2-disable-checksum Disable checksums for large (> upload cutoff) files
|
|
--b2-download-auth-duration Duration Time before the authorization token will expire in s or suffix ms|s|m|h|d (default 1w)
|
|
--b2-download-url string Custom endpoint for downloads
|
|
--b2-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
|
|
--b2-endpoint string Endpoint for the service
|
|
--b2-hard-delete Permanently delete files on remote removal, otherwise hide files
|
|
--b2-key string Application Key
|
|
--b2-memory-pool-flush-time Duration How often internal memory buffer pools will be flushed (default 1m0s)
|
|
--b2-memory-pool-use-mmap Whether to use mmap buffers in internal memory pool
|
|
--b2-test-mode string A flag string for X-Bz-Test-Mode header for debugging
|
|
--b2-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi)
|
|
--b2-versions Include old versions in directory listings
|
|
--box-access-token string Box App Primary Access Token
|
|
--box-auth-url string Auth server URL
|
|
--box-box-config-file string Box App config.json location
|
|
--box-box-sub-type string (default \[dq]user\[dq])
|
|
--box-client-id string OAuth Client Id
|
|
--box-client-secret string OAuth Client Secret
|
|
--box-commit-retries int Max number of times to try committing a multipart file (default 100)
|
|
--box-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot)
|
|
--box-list-chunk int Size of listing chunk 1-1000 (default 1000)
|
|
--box-owned-by string Only show items owned by the login (email address) passed in
|
|
--box-root-folder-id string Fill in for rclone to use a non root folder as its starting point
|
|
--box-token string OAuth Access Token as a JSON blob
|
|
--box-token-url string Token server url
|
|
--box-upload-cutoff SizeSuffix Cutoff for switching to multipart upload (>= 50 MiB) (default 50Mi)
|
|
--cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage (default 1m0s)
|
|
--cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming
|
|
--cache-chunk-path string Directory to cache chunk files (default \[dq]$HOME/.cache/rclone/cache-backend\[dq])
|
|
--cache-chunk-size SizeSuffix The size of a chunk (partial file data) (default 5Mi)
|
|
--cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk (default 10Gi)
|
|
--cache-db-path string Directory to store file structure metadata DB (default \[dq]$HOME/.cache/rclone/cache-backend\[dq])
|
|
--cache-db-purge Clear all the cached data for this remote on start
|
|
--cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s)
|
|
--cache-info-age Duration How long to cache file structure information (directory listings, file size, times, etc.) (default 6h0m0s)
|
|
--cache-plex-insecure string Skip all certificate verification when connecting to the Plex server
|
|
--cache-plex-password string The password of the Plex user (obscured)
|
|
--cache-plex-url string The URL of the Plex server
|
|
--cache-plex-username string The username of the Plex user
|
|
--cache-read-retries int How many times to retry a read from a cache storage (default 10)
|
|
--cache-remote string Remote to cache
|
|
--cache-rps int Limits the number of requests per second to the source FS (-1 to disable) (default -1)
|
|
--cache-tmp-upload-path string Directory to keep temporary files until they are uploaded
|
|
--cache-tmp-wait-time Duration How long should files be stored in local cache before being uploaded (default 15s)
|
|
--cache-workers int How many workers should run in parallel to download chunks (default 4)
|
|
--cache-writes Cache file data on writes through the FS
|
|
--chunker-chunk-size SizeSuffix Files larger than chunk size will be split in chunks (default 2Gi)
|
|
--chunker-fail-hard Choose how chunker should handle files with missing or invalid chunks
|
|
--chunker-hash-type string Choose how chunker handles hash sums (default \[dq]md5\[dq])
|
|
--chunker-remote string Remote to chunk/unchunk
|
|
--compress-level int GZIP compression level (-2 to 9) (default -1)
|
|
--compress-mode string Compression mode (default \[dq]gzip\[dq])
|
|
--compress-ram-cache-limit SizeSuffix Some remotes don\[aq]t allow the upload of files with unknown size (default 20Mi)
|
|
--compress-remote string Remote to compress
|
|
-L, --copy-links Follow symlinks and copy the pointed to item
|
|
--crypt-directory-name-encryption Option to either encrypt directory names or leave them intact (default true)
|
|
--crypt-filename-encoding string How to encode the encrypted filename to text string (default \[dq]base32\[dq])
|
|
--crypt-filename-encryption string How to encrypt the filenames (default \[dq]standard\[dq])
|
|
--crypt-no-data-encryption Option to either encrypt file data or leave it unencrypted
|
|
--crypt-password string Password or pass phrase for encryption (obscured)
|
|
--crypt-password2 string Password or pass phrase for salt (obscured)
|
|
--crypt-remote string Remote to encrypt/decrypt
|
|
--crypt-server-side-across-configs Allow server-side operations (e.g. copy) to work across different crypt configs
|
|
--crypt-show-mapping For all files listed show how the names encrypt
|
|
--drive-acknowledge-abuse Set to allow files which return cannotDownloadAbusiveFile to be downloaded
|
|
--drive-allow-import-name-change Allow the filetype to change when uploading Google docs
|
|
--drive-auth-owner-only Only consider files owned by the authenticated user
|
|
--drive-auth-url string Auth server URL
|
|
--drive-chunk-size SizeSuffix Upload chunk size (default 8Mi)
|
|
--drive-client-id string Google Application Client Id
|
|
--drive-client-secret string OAuth Client Secret
|
|
--drive-copy-shortcut-content Server side copy contents of shortcuts instead of the shortcut
|
|
--drive-disable-http2 Disable drive using http2 (default true)
|
|
--drive-encoding MultiEncoder The encoding for the backend (default InvalidUtf8)
|
|
--drive-export-formats string Comma separated list of preferred formats for downloading Google docs (default \[dq]docx,xlsx,pptx,svg\[dq])
|
|
--drive-formats string Deprecated: See export_formats
|
|
--drive-impersonate string Impersonate this user when using a service account
|
|
--drive-import-formats string Comma separated list of preferred formats for uploading Google docs
|
|
--drive-keep-revision-forever Keep new head revision of each file forever
|
|
--drive-list-chunk int Size of listing chunk 100-1000, 0 to disable (default 1000)
|
|
--drive-pacer-burst int Number of API calls to allow without sleeping (default 100)
|
|
--drive-pacer-min-sleep Duration Minimum time to sleep between API calls (default 100ms)
|
|
--drive-root-folder-id string ID of the root folder
|
|
--drive-scope string Scope that rclone should use when requesting access from drive
|
|
--drive-server-side-across-configs Allow server-side operations (e.g. copy) to work across different drive configs
|
|
--drive-service-account-credentials string Service Account Credentials JSON blob
|
|
--drive-service-account-file string Service Account Credentials JSON file path
|
|
--drive-shared-with-me Only show files that are shared with me
|
|
--drive-size-as-quota Show sizes as storage quota usage, not actual size
|
|
--drive-skip-checksum-gphotos Skip MD5 checksum on Google photos and videos only
|
|
--drive-skip-dangling-shortcuts If set skip dangling shortcut files
|
|
--drive-skip-gdocs Skip google documents in all listings
|
|
--drive-skip-shortcuts If set skip shortcut files
|
|
--drive-starred-only Only show files that are starred
|
|
--drive-stop-on-download-limit Make download limit errors be fatal
|
|
--drive-stop-on-upload-limit Make upload limit errors be fatal
|
|
--drive-team-drive string ID of the Shared Drive (Team Drive)
|
|
--drive-token string OAuth Access Token as a JSON blob
|
|
--drive-token-url string Token server url
|
|
--drive-trashed-only Only show files that are in the trash
|
|
--drive-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 8Mi)
|
|
--drive-use-created-date Use file created date instead of modified date
|
|
--drive-use-shared-date Use date file was shared instead of modified date
|
|
--drive-use-trash Send files to the trash instead of deleting permanently (default true)
|
|
--drive-v2-download-min-size SizeSuffix If Object\[aq]s are greater, use drive v2 API to download (default off)
|
|
--dropbox-auth-url string Auth server URL
|
|
--dropbox-batch-commit-timeout Duration Max time to wait for a batch to finish comitting (default 10m0s)
|
|
--dropbox-batch-mode string Upload file batching sync|async|off (default \[dq]sync\[dq])
|
|
--dropbox-batch-size int Max number of files in upload batch
|
|
--dropbox-batch-timeout Duration Max time to allow an idle upload batch before uploading (default 0s)
|
|
--dropbox-chunk-size SizeSuffix Upload chunk size (< 150Mi) (default 48Mi)
|
|
--dropbox-client-id string OAuth Client Id
|
|
--dropbox-client-secret string OAuth Client Secret
|
|
--dropbox-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot)
|
|
--dropbox-impersonate string Impersonate this user when using a business account
|
|
--dropbox-shared-files Instructs rclone to work on individual shared files
|
|
--dropbox-shared-folders Instructs rclone to work on shared folders
|
|
--dropbox-token string OAuth Access Token as a JSON blob
|
|
--dropbox-token-url string Token server url
|
|
--fichier-api-key string Your API Key, get it from https://1fichier.com/console/params.pl
|
|
--fichier-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot)
|
|
--fichier-file-password string If you want to download a shared file that is password protected, add this parameter (obscured)
|
|
--fichier-folder-password string If you want to list the files in a shared folder that is password protected, add this parameter (obscured)
|
|
--fichier-shared-folder string If you want to download a shared folder, add this parameter
|
|
--filefabric-encoding MultiEncoder The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot)
|
|
--filefabric-permanent-token string Permanent Authentication Token
|
|
--filefabric-root-folder-id string ID of the root folder
|
|
--filefabric-token string Session Token
|
|
--filefabric-token-expiry string Token expiry time
|
|
--filefabric-url string URL of the Enterprise File Fabric to connect to
|
|
--filefabric-version string Version read from the file fabric
|
|
--ftp-ask-password Allow asking for FTP password when needed
|
|
--ftp-close-timeout Duration Maximum time to wait for a response to close (default 1m0s)
|
|
--ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited
|
|
--ftp-disable-epsv Disable using EPSV even if server advertises support
|
|
--ftp-disable-mlsd Disable using MLSD even if server advertises support
|
|
--ftp-disable-tls13 Disable TLS 1.3 (workaround for FTP servers with buggy TLS)
|
|
--ftp-encoding MultiEncoder The encoding for the backend (default Slash,Del,Ctl,RightSpace,Dot)
|
|
--ftp-explicit-tls Use Explicit FTPS (FTP over TLS)
|
|
--ftp-host string FTP host to connect to
|
|
--ftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
|
|
--ftp-no-check-certificate Do not verify the TLS certificate of the server
|
|
--ftp-pass string FTP password (obscured)
|
|
--ftp-port int FTP port number (default 21)
|
|
--ftp-shut-timeout Duration Maximum time to wait for data connection closing status (default 1m0s)
|
|
--ftp-tls Use Implicit FTPS (FTP over TLS)
|
|
--ftp-tls-cache-size int Size of TLS session cache for all control and data connections (default 32)
|
|
--ftp-user string FTP username (default \[dq]$USER\[dq])
|
|
--ftp-writing-mdtm Use MDTM to set modification time (VsFtpd quirk)
|
|
--gcs-anonymous Access public buckets and objects without credentials
|
|
--gcs-auth-url string Auth server URL
|
|
--gcs-bucket-acl string Access Control List for new buckets
|
|
--gcs-bucket-policy-only Access checks should use bucket-level IAM policies
|
|
--gcs-client-id string OAuth Client Id
|
|
--gcs-client-secret string OAuth Client Secret
|
|
--gcs-encoding MultiEncoder The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
|
|
--gcs-location string Location for the newly created buckets
|
|
--gcs-object-acl string Access Control List for new objects
|
|
--gcs-project-number string Project number
|
|
--gcs-service-account-file string Service Account Credentials JSON file path
|
|
--gcs-storage-class string The storage class to use when storing objects in Google Cloud Storage
|
|
--gcs-token string OAuth Access Token as a JSON blob
|
|
--gcs-token-url string Token server url
|
|
--gphotos-auth-url string Auth server URL
|
|
--gphotos-client-id string OAuth Client Id
|
|
--gphotos-client-secret string OAuth Client Secret
|
|
--gphotos-encoding MultiEncoder The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
|
|
--gphotos-include-archived Also view and download archived media
|
|
--gphotos-read-only Set to make the Google Photos backend read only
|
|
--gphotos-read-size Set to read the size of media items
|
|
--gphotos-start-year int Year limits the photos to be downloaded to those which are uploaded after the given year (default 2000)
|
|
--gphotos-token string OAuth Access Token as a JSON blob
|
|
--gphotos-token-url string Token server url
|
|
--hasher-auto-size SizeSuffix Auto-update checksum for files smaller than this size (disabled by default)
|
|
--hasher-hashes CommaSepList Comma separated list of supported checksum types (default md5,sha1)
|
|
--hasher-max-age Duration Maximum time to keep checksums in cache (0 = no cache, off = cache forever) (default off)
|
|
--hasher-remote string Remote to cache checksums for (e.g. myRemote:path)
|
|
--hdfs-data-transfer-protection string Kerberos data transfer protection: authentication|integrity|privacy
|
|
--hdfs-encoding MultiEncoder The encoding for the backend (default Slash,Colon,Del,Ctl,InvalidUtf8,Dot)
|
|
--hdfs-namenode string Hadoop name node and port
|
|
--hdfs-service-principal-name string Kerberos service principal name for the namenode
|
|
--hdfs-username string Hadoop user name
|
|
--http-headers CommaSepList Set HTTP headers for all transactions
|
|
--http-no-head Don\[aq]t use HEAD requests
|
|
--http-no-slash Set this if the site doesn\[aq]t end directories with /
|
|
--http-url string URL of http host to connect to
|
|
--hubic-auth-url string Auth server URL
|
|
--hubic-chunk-size SizeSuffix Above this size files will be chunked into a _segments container (default 5Gi)
|
|
--hubic-client-id string OAuth Client Id
|
|
--hubic-client-secret string OAuth Client Secret
|
|
--hubic-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8)
|
|
--hubic-no-chunk Don\[aq]t chunk files during streaming upload
|
|
--hubic-token string OAuth Access Token as a JSON blob
|
|
--hubic-token-url string Token server url
|
|
--jottacloud-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot)
|
|
--jottacloud-hard-delete Delete files permanently rather than putting them into the trash
|
|
--jottacloud-md5-memory-limit SizeSuffix Files bigger than this will be cached on disk to calculate the MD5 if required (default 10Mi)
|
|
--jottacloud-no-versions Avoid server side versioning by deleting files and recreating files instead of overwriting them
|
|
--jottacloud-trashed-only Only show files that are in the trash
|
|
--jottacloud-upload-resume-limit SizeSuffix Files bigger than this can be resumed if the upload fail\[aq]s (default 10Mi)
|
|
--koofr-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
|
|
--koofr-endpoint string The Koofr API endpoint to use
|
|
--koofr-mountid string Mount ID of the mount to use
|
|
--koofr-password string Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password) (obscured)
|
|
--koofr-provider string Choose your storage provider
|
|
--koofr-setmtime Does the backend support setting modification time (default true)
|
|
--koofr-user string Your user name
|
|
-l, --links Translate symlinks to/from regular files with a \[aq].rclonelink\[aq] extension
|
|
--local-case-insensitive Force the filesystem to report itself as case insensitive
|
|
--local-case-sensitive Force the filesystem to report itself as case sensitive
|
|
--local-encoding MultiEncoder The encoding for the backend (default Slash,Dot)
|
|
--local-no-check-updated Don\[aq]t check to see if the files change during upload
|
|
--local-no-preallocate Disable preallocation of disk space for transferred files
|
|
--local-no-set-modtime Disable setting modtime
|
|
--local-no-sparse Disable sparse files for multi-thread downloads
|
|
--local-nounc string Disable UNC (long path names) conversion on Windows
|
|
--local-unicode-normalization Apply unicode NFC normalization to paths and filenames
|
|
--local-zero-size-links Assume the Stat size of links is zero (and read them instead) (deprecated)
|
|
--mailru-check-hash What should copy do if file checksum is mismatched or invalid (default true)
|
|
--mailru-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot)
|
|
--mailru-pass string Password (obscured)
|
|
--mailru-speedup-enable Skip full upload if there is another file with same data hash (default true)
|
|
--mailru-speedup-file-patterns string Comma separated list of file name patterns eligible for speedup (put by hash) (default \[dq]*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf\[dq])
|
|
--mailru-speedup-max-disk SizeSuffix This option allows you to disable speedup (put by hash) for large files (default 3Gi)
|
|
--mailru-speedup-max-memory SizeSuffix Files larger than the size given below will always be hashed on disk (default 32Mi)
|
|
--mailru-user string User name (usually email)
|
|
--mega-debug Output more debug from Mega
|
|
--mega-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8,Dot)
|
|
--mega-hard-delete Delete files permanently rather than putting them into the trash
|
|
--mega-pass string Password (obscured)
|
|
--mega-user string User name
|
|
--netstorage-account string Set the NetStorage account name
|
|
--netstorage-host string Domain+path of NetStorage host to connect to
|
|
--netstorage-protocol string Select between HTTP or HTTPS protocol (default \[dq]https\[dq])
|
|
--netstorage-secret string Set the NetStorage account secret/G2O key for authentication (obscured)
|
|
-x, --one-file-system Don\[aq]t cross filesystem boundaries (unix/macOS only)
|
|
--onedrive-auth-url string Auth server URL
|
|
--onedrive-chunk-size SizeSuffix Chunk size to upload files with - must be multiple of 320k (327,680 bytes) (default 10Mi)
|
|
--onedrive-client-id string OAuth Client Id
|
|
--onedrive-client-secret string OAuth Client Secret
|
|
--onedrive-disable-site-permission Disable the request for Sites.Read.All permission
|
|
--onedrive-drive-id string The ID of the drive to use
|
|
--onedrive-drive-type string The type of the drive (personal | business | documentLibrary)
|
|
--onedrive-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot)
|
|
--onedrive-expose-onenote-files Set to make OneNote files show up in directory listings
|
|
--onedrive-link-password string Set the password for links created by the link command
|
|
--onedrive-link-scope string Set the scope of the links created by the link command (default \[dq]anonymous\[dq])
|
|
--onedrive-link-type string Set the type of the links created by the link command (default \[dq]view\[dq])
|
|
--onedrive-list-chunk int Size of listing chunk (default 1000)
|
|
--onedrive-no-versions Remove all versions on modifying operations
|
|
--onedrive-region string Choose national cloud region for OneDrive (default \[dq]global\[dq])
|
|
--onedrive-root-folder-id string ID of the root folder
|
|
--onedrive-server-side-across-configs Allow server-side operations (e.g. copy) to work across different onedrive configs
|
|
--onedrive-token string OAuth Access Token as a JSON blob
|
|
--onedrive-token-url string Token server url
|
|
--opendrive-chunk-size SizeSuffix Files will be uploaded in chunks this size (default 10Mi)
|
|
--opendrive-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot)
|
|
--opendrive-password string Password (obscured)
|
|
--opendrive-username string Username
|
|
--pcloud-auth-url string Auth server URL
|
|
--pcloud-client-id string OAuth Client Id
|
|
--pcloud-client-secret string OAuth Client Secret
|
|
--pcloud-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
|
|
--pcloud-hostname string Hostname to connect to (default \[dq]api.pcloud.com\[dq])
|
|
--pcloud-root-folder-id string Fill in for rclone to use a non root folder as its starting point (default \[dq]d0\[dq])
|
|
--pcloud-token string OAuth Access Token as a JSON blob
|
|
--pcloud-token-url string Token server url
|
|
--premiumizeme-encoding MultiEncoder The encoding for the backend (default Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot)
|
|
--putio-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
|
|
--qingstor-access-key-id string QingStor Access Key ID
|
|
--qingstor-chunk-size SizeSuffix Chunk size to use for uploading (default 4Mi)
|
|
--qingstor-connection-retries int Number of connection retries (default 3)
|
|
--qingstor-encoding MultiEncoder The encoding for the backend (default Slash,Ctl,InvalidUtf8)
|
|
--qingstor-endpoint string Enter an endpoint URL to connection QingStor API
|
|
--qingstor-env-auth Get QingStor credentials from runtime
|
|
--qingstor-secret-access-key string QingStor Secret Access Key (password)
|
|
--qingstor-upload-concurrency int Concurrency for multipart uploads (default 1)
|
|
--qingstor-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi)
|
|
--qingstor-zone string Zone to connect to
|
|
--s3-access-key-id string AWS Access Key ID
|
|
--s3-acl string Canned ACL used when creating buckets and storing or copying objects
|
|
--s3-bucket-acl string Canned ACL used when creating buckets
|
|
--s3-chunk-size SizeSuffix Chunk size to use for uploading (default 5Mi)
|
|
--s3-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 4.656Gi)
|
|
--s3-disable-checksum Don\[aq]t store MD5 checksum with object metadata
|
|
--s3-disable-http2 Disable usage of http2 for S3 backends
|
|
--s3-download-url string Custom endpoint for downloads
|
|
--s3-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8,Dot)
|
|
--s3-endpoint string Endpoint for S3 API
|
|
--s3-env-auth Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars)
|
|
--s3-force-path-style If true use path style access if false use virtual hosted style (default true)
|
|
--s3-leave-parts-on-error If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery
|
|
--s3-list-chunk int Size of listing chunk (response list for each ListObject S3 request) (default 1000)
|
|
--s3-list-url-encode Tristate Whether to url encode listings: true/false/unset (default unset)
|
|
--s3-list-version int Version of ListObjects to use: 1,2 or 0 for auto
|
|
--s3-location-constraint string Location constraint - must be set to match the Region
|
|
--s3-max-upload-parts int Maximum number of parts in a multipart upload (default 10000)
|
|
--s3-memory-pool-flush-time Duration How often internal memory buffer pools will be flushed (default 1m0s)
|
|
--s3-memory-pool-use-mmap Whether to use mmap buffers in internal memory pool
|
|
--s3-no-check-bucket If set, don\[aq]t attempt to check the bucket exists or create it
|
|
--s3-no-head If set, don\[aq]t HEAD uploaded objects to check integrity
|
|
--s3-no-head-object If set, do not do HEAD before GET when getting objects
|
|
--s3-profile string Profile to use in the shared credentials file
|
|
--s3-provider string Choose your S3 provider
|
|
--s3-region string Region to connect to
|
|
--s3-requester-pays Enables requester pays option when interacting with S3 bucket
|
|
--s3-secret-access-key string AWS Secret Access Key (password)
|
|
--s3-server-side-encryption string The server-side encryption algorithm used when storing this object in S3
|
|
--s3-session-token string An AWS session token
|
|
--s3-shared-credentials-file string Path to the shared credentials file
|
|
--s3-sse-customer-algorithm string If using SSE-C, the server-side encryption algorithm used when storing this object in S3
|
|
--s3-sse-customer-key string If using SSE-C you must provide the secret encryption key used to encrypt/decrypt your data
|
|
--s3-sse-customer-key-md5 string If using SSE-C you may provide the secret encryption key MD5 checksum (optional)
|
|
--s3-sse-kms-key-id string If using KMS ID you must provide the ARN of Key
|
|
--s3-storage-class string The storage class to use when storing new objects in S3
|
|
--s3-upload-concurrency int Concurrency for multipart uploads (default 4)
|
|
--s3-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi)
|
|
--s3-use-accelerate-endpoint If true use the AWS S3 accelerated endpoint
|
|
--s3-use-multipart-etag Tristate Whether to use ETag in multipart uploads for verification (default unset)
|
|
--s3-v2-auth If true use v2 authentication
|
|
--seafile-2fa Two-factor authentication (\[aq]true\[aq] if the account has 2FA enabled)
|
|
--seafile-create-library Should rclone create a library if it doesn\[aq]t exist
|
|
--seafile-encoding MultiEncoder The encoding for the backend (default Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8)
|
|
--seafile-library string Name of the library
|
|
--seafile-library-key string Library password (for encrypted libraries only) (obscured)
|
|
--seafile-pass string Password (obscured)
|
|
--seafile-url string URL of seafile host to connect to
|
|
--seafile-user string User name (usually email address)
|
|
--sftp-ask-password Allow asking for SFTP password when needed
|
|
--sftp-disable-concurrent-reads If set don\[aq]t use concurrent reads
|
|
--sftp-disable-concurrent-writes If set don\[aq]t use concurrent writes
|
|
--sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available
|
|
--sftp-host string SSH host to connect to
|
|
--sftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
|
|
--sftp-key-file string Path to PEM-encoded private key file
|
|
--sftp-key-file-pass string The passphrase to decrypt the PEM-encoded private key file (obscured)
|
|
--sftp-key-pem string Raw PEM-encoded private key
|
|
--sftp-key-use-agent When set forces the usage of the ssh-agent
|
|
--sftp-known-hosts-file string Optional path to known_hosts file
|
|
--sftp-md5sum-command string The command used to read md5 hashes
|
|
--sftp-pass string SSH password, leave blank to use ssh-agent (obscured)
|
|
--sftp-path-override string Override path used by SSH connection
|
|
--sftp-port int SSH port number (default 22)
|
|
--sftp-pubkey-file string Optional path to public key file
|
|
--sftp-server-command string Specifies the path or command to run a sftp server on the remote host
|
|
--sftp-set-modtime Set the modified time on the remote if set (default true)
|
|
--sftp-sha1sum-command string The command used to read sha1 hashes
|
|
--sftp-skip-links Set to skip any symlinks and any other non regular files
|
|
--sftp-subsystem string Specifies the SSH2 subsystem on the remote host (default \[dq]sftp\[dq])
|
|
--sftp-use-fstat If set use fstat instead of stat
|
|
--sftp-use-insecure-cipher Enable the use of insecure ciphers and key exchange methods
|
|
--sftp-user string SSH username (default \[dq]$USER\[dq])
|
|
--sharefile-chunk-size SizeSuffix Upload chunk size (default 64Mi)
|
|
--sharefile-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot)
|
|
--sharefile-endpoint string Endpoint for API calls
|
|
--sharefile-root-folder-id string ID of the root folder
|
|
--sharefile-upload-cutoff SizeSuffix Cutoff for switching to multipart upload (default 128Mi)
|
|
--sia-api-password string Sia Daemon API Password (obscured)
|
|
--sia-api-url string Sia daemon API URL, like http://sia.daemon.host:9980 (default \[dq]http://127.0.0.1:9980\[dq])
|
|
--sia-encoding MultiEncoder The encoding for the backend (default Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot)
|
|
--sia-user-agent string Siad User Agent (default \[dq]Sia-Agent\[dq])
|
|
--skip-links Don\[aq]t warn about skipped symlinks
|
|
--storj-access-grant string Access grant
|
|
--storj-api-key string API key
|
|
--storj-passphrase string Encryption passphrase
|
|
--storj-provider string Choose an authentication method (default \[dq]existing\[dq])
|
|
--storj-satellite-address string Satellite address (default \[dq]us-central-1.storj.io\[dq])
|
|
--sugarsync-access-key-id string Sugarsync Access Key ID
|
|
--sugarsync-app-id string Sugarsync App ID
|
|
--sugarsync-authorization string Sugarsync authorization
|
|
--sugarsync-authorization-expiry string Sugarsync authorization expiry
|
|
--sugarsync-deleted-id string Sugarsync deleted folder id
|
|
--sugarsync-encoding MultiEncoder The encoding for the backend (default Slash,Ctl,InvalidUtf8,Dot)
|
|
--sugarsync-hard-delete Permanently delete files if true
|
|
--sugarsync-private-access-key string Sugarsync Private Access Key
|
|
--sugarsync-refresh-token string Sugarsync refresh token
|
|
--sugarsync-root-id string Sugarsync root id
|
|
--sugarsync-user string Sugarsync user
|
|
--swift-application-credential-id string Application Credential ID (OS_APPLICATION_CREDENTIAL_ID)
|
|
--swift-application-credential-name string Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME)
|
|
--swift-application-credential-secret string Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET)
|
|
--swift-auth string Authentication URL for server (OS_AUTH_URL)
|
|
--swift-auth-token string Auth Token from alternate authentication - optional (OS_AUTH_TOKEN)
|
|
--swift-auth-version int AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION)
|
|
--swift-chunk-size SizeSuffix Above this size files will be chunked into a _segments container (default 5Gi)
|
|
--swift-domain string User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME)
|
|
--swift-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8)
|
|
--swift-endpoint-type string Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) (default \[dq]public\[dq])
|
|
--swift-env-auth Get swift credentials from environment variables in standard OpenStack form
|
|
--swift-key string API key or password (OS_PASSWORD)
|
|
--swift-leave-parts-on-error If true avoid calling abort upload on a failure
|
|
--swift-no-chunk Don\[aq]t chunk files during streaming upload
|
|
--swift-region string Region name - optional (OS_REGION_NAME)
|
|
--swift-storage-policy string The storage policy to use when creating a new container
|
|
--swift-storage-url string Storage URL - optional (OS_STORAGE_URL)
|
|
--swift-tenant string Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME)
|
|
--swift-tenant-domain string Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME)
|
|
--swift-tenant-id string Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID)
|
|
--swift-user string User name to log in (OS_USERNAME)
|
|
--swift-user-id string User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID)
|
|
--union-action-policy string Policy to choose upstream on ACTION category (default \[dq]epall\[dq])
|
|
--union-cache-time int Cache time of usage and free space (in seconds) (default 120)
|
|
--union-create-policy string Policy to choose upstream on CREATE category (default \[dq]epmfs\[dq])
|
|
--union-search-policy string Policy to choose upstream on SEARCH category (default \[dq]ff\[dq])
|
|
--union-upstreams string List of space separated upstreams
|
|
--uptobox-access-token string Your access token
|
|
--uptobox-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,BackQuote,Del,Ctl,LeftSpace,InvalidUtf8,Dot)
|
|
--webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon)
|
|
--webdav-bearer-token-command string Command to run to get a bearer token
|
|
--webdav-encoding string The encoding for the backend
|
|
--webdav-headers CommaSepList Set HTTP headers for all transactions
|
|
--webdav-pass string Password (obscured)
|
|
--webdav-url string URL of http host to connect to
|
|
--webdav-user string User name
|
|
--webdav-vendor string Name of the Webdav site/service/software you are using
|
|
--yandex-auth-url string Auth server URL
|
|
--yandex-client-id string OAuth Client Id
|
|
--yandex-client-secret string OAuth Client Secret
|
|
--yandex-encoding MultiEncoder The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot)
|
|
--yandex-hard-delete Delete files permanently rather than putting them into the trash
|
|
--yandex-token string OAuth Access Token as a JSON blob
|
|
--yandex-token-url string Token server url
|
|
--zoho-auth-url string Auth server URL
|
|
--zoho-client-id string OAuth Client Id
|
|
--zoho-client-secret string OAuth Client Secret
|
|
--zoho-encoding MultiEncoder The encoding for the backend (default Del,Ctl,InvalidUtf8)
|
|
--zoho-region string Zoho region to connect to
|
|
--zoho-token string OAuth Access Token as a JSON blob
|
|
--zoho-token-url string Token server url
|
|
\f[R]
|
|
.fi
|
|
.SH Docker Volume Plugin
|
|
.SS Introduction
|
|
.PP
|
|
Docker 1.9 has added support for creating named
|
|
volumes (https://docs.docker.com/storage/volumes/) via command-line
|
|
interface (https://docs.docker.com/engine/reference/commandline/volume_create/)
|
|
and mounting them in containers as a way to share data between them.
|
|
Since Docker 1.10 you can create named volumes with Docker
|
|
Compose (https://docs.docker.com/compose/) by descriptions in
|
|
docker-compose.yml (https://docs.docker.com/compose/compose-file/compose-file-v2/#volume-configuration-reference)
|
|
files for use by container groups on a single host.
|
|
As of Docker 1.12 volumes are supported by Docker
|
|
Swarm (https://docs.docker.com/engine/swarm/key-concepts/) included with
|
|
Docker Engine and created from descriptions in swarm compose
|
|
v3 (https://docs.docker.com/compose/compose-file/compose-file-v3/#volume-configuration-reference)
|
|
files for use with \f[I]swarm stacks\f[R] across multiple cluster nodes.
|
|
.PP
|
|
Docker Volume
|
|
Plugins (https://docs.docker.com/engine/extend/plugins_volume/) augment
|
|
the default \f[C]local\f[R] volume driver included in Docker with
|
|
stateful volumes shared across containers and hosts.
|
|
Unlike local volumes, your data will \f[I]not\f[R] be deleted when such
|
|
volume is removed.
|
|
Plugins can run managed by the docker daemon, as a native system service
|
|
(under systemd, \f[I]sysv\f[R] or \f[I]upstart\f[R]) or as a standalone
|
|
executable.
|
|
Rclone can run as docker volume plugin in all these modes.
|
|
It interacts with the local docker daemon via plugin
|
|
API (https://docs.docker.com/engine/extend/plugin_api/) and handles
|
|
mounting of remote file systems into docker containers so it must run on
|
|
the same host as the docker daemon or on every Swarm node.
|
|
.SS Getting started
|
|
.PP
|
|
In the first example we will use the SFTP (https://rclone.org/sftp/)
|
|
rclone volume with Docker engine on a standalone Ubuntu machine.
|
|
.PP
|
|
Start from installing Docker (https://docs.docker.com/engine/install/)
|
|
on the host.
|
|
.PP
|
|
The \f[I]FUSE\f[R] driver is a prerequisite for rclone mounting and
|
|
should be installed on host:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo apt-get -y install fuse
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Create two directories required by rclone docker plugin:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo mkdir -p /var/lib/docker-plugins/rclone/config
|
|
sudo mkdir -p /var/lib/docker-plugins/rclone/cache
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Install the managed rclone docker plugin for your architecture (here
|
|
\f[C]amd64\f[R]):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker plugin install rclone/docker-volume-rclone:amd64 args=\[dq]-v\[dq] --alias rclone --grant-all-permissions
|
|
docker plugin list
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Create your SFTP volume (https://rclone.org/sftp/#standard-options):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker volume create firstvolume -d rclone -o type=sftp -o sftp-host=_hostname_ -o sftp-user=_username_ -o sftp-pass=_password_ -o allow-other=true
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that since all options are static, you don\[aq]t even have to run
|
|
\f[C]rclone config\f[R] or create the \f[C]rclone.conf\f[R] file (but
|
|
the \f[C]config\f[R] directory should still be present).
|
|
In the simplest case you can use \f[C]localhost\f[R] as
|
|
\f[I]hostname\f[R] and your SSH credentials as \f[I]username\f[R] and
|
|
\f[I]password\f[R].
|
|
You can also change the remote path to your home directory on the host,
|
|
for example \f[C]-o path=/home/username\f[R].
|
|
.PP
|
|
Time to create a test container and mount the volume into it:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker run --rm -it -v firstvolume:/mnt --workdir /mnt ubuntu:latest bash
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If all goes well, you will enter the new container and change right to
|
|
the mounted SFTP remote.
|
|
You can type \f[C]ls\f[R] to list the mounted directory or otherwise
|
|
play with it.
|
|
Type \f[C]exit\f[R] when you are done.
|
|
The container will stop but the volume will stay, ready to be reused.
|
|
When it\[aq]s not needed anymore, remove it:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker volume list
|
|
docker volume remove firstvolume
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Now let us try \f[B]something more elaborate\f[R]: Google
|
|
Drive (https://rclone.org/drive/) volume on multi-node Docker Swarm.
|
|
.PP
|
|
You should start from installing Docker and FUSE, creating plugin
|
|
directories and installing rclone plugin on \f[I]every\f[R] swarm node.
|
|
Then setup the Swarm (https://docs.docker.com/engine/swarm/swarm-mode/).
|
|
.PP
|
|
Google Drive volumes need an access token which can be setup via web
|
|
browser and will be periodically renewed by rclone.
|
|
The managed plugin cannot run a browser so we will use a technique
|
|
similar to the rclone setup on a headless
|
|
box (https://rclone.org/remote_setup/).
|
|
.PP
|
|
Run rclone config (https://rclone.org/commands/rclone_config_create/) on
|
|
\f[I]another\f[R] machine equipped with \f[I]web browser\f[R] and
|
|
graphical user interface.
|
|
Create the Google Drive
|
|
remote (https://rclone.org/drive/#standard-options).
|
|
When done, transfer the resulting \f[C]rclone.conf\f[R] to the Swarm
|
|
cluster and save as
|
|
\f[C]/var/lib/docker-plugins/rclone/config/rclone.conf\f[R] on
|
|
\f[I]every\f[R] node.
|
|
By default this location is accessible only to the root user so you will
|
|
need appropriate privileges.
|
|
The resulting config will look like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[gdrive]
|
|
type = drive
|
|
scope = drive
|
|
drive_id = 1234567...
|
|
root_folder_id = 0Abcd...
|
|
token = {\[dq]access_token\[dq]:...}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Now create the file named \f[C]example.yml\f[R] with a swarm stack
|
|
description like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
version: \[aq]3\[aq]
|
|
services:
|
|
heimdall:
|
|
image: linuxserver/heimdall:latest
|
|
ports: [8080:80]
|
|
volumes: [configdata:/config]
|
|
volumes:
|
|
configdata:
|
|
driver: rclone
|
|
driver_opts:
|
|
remote: \[aq]gdrive:heimdall\[aq]
|
|
allow_other: \[aq]true\[aq]
|
|
vfs_cache_mode: full
|
|
poll_interval: 0
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
and run the stack:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker stack deploy example -c ./example.yml
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
After a few seconds docker will spread the parsed stack description over
|
|
cluster, create the \f[C]example_heimdall\f[R] service on port
|
|
\f[I]8080\f[R], run service containers on one or more cluster nodes and
|
|
request the \f[C]example_configdata\f[R] volume from rclone plugins on
|
|
the node hosts.
|
|
You can use the following commands to confirm results:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker service ls
|
|
docker service ps example_heimdall
|
|
docker volume ls
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Point your browser to \f[C]http://cluster.host.address:8080\f[R] and
|
|
play with the service.
|
|
Stop it with \f[C]docker stack remove example\f[R] when you are done.
|
|
Note that the \f[C]example_configdata\f[R] volume(s) created on demand
|
|
at the cluster nodes will not be automatically removed together with the
|
|
stack but stay for future reuse.
|
|
You can remove them manually by invoking the
|
|
\f[C]docker volume remove example_configdata\f[R] command on every node.
|
|
.SS Creating Volumes via CLI
|
|
.PP
|
|
Volumes can be created with docker volume
|
|
create (https://docs.docker.com/engine/reference/commandline/volume_create/).
|
|
Here are a few examples:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker volume create vol1 -d rclone -o remote=storj: -o vfs-cache-mode=full
|
|
docker volume create vol2 -d rclone -o remote=:storj,access_grant=xxx:heimdall
|
|
docker volume create vol3 -d rclone -o type=storj -o path=heimdall -o storj-access-grant=xxx -o poll-interval=0
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note the \f[C]-d rclone\f[R] flag that tells docker to request volume
|
|
from the rclone driver.
|
|
This works even if you installed managed driver by its full name
|
|
\f[C]rclone/docker-volume-rclone\f[R] because you provided the
|
|
\f[C]--alias rclone\f[R] option.
|
|
.PP
|
|
Volumes can be inspected as follows:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker volume list
|
|
docker volume inspect vol1
|
|
\f[R]
|
|
.fi
|
|
.SS Volume Configuration
|
|
.PP
|
|
Rclone flags and volume options are set via the \f[C]-o\f[R] flag to the
|
|
\f[C]docker volume create\f[R] command.
|
|
They include backend-specific parameters as well as mount and
|
|
\f[I]VFS\f[R] options.
|
|
Also there are a few special \f[C]-o\f[R] options: \f[C]remote\f[R],
|
|
\f[C]fs\f[R], \f[C]type\f[R], \f[C]path\f[R], \f[C]mount-type\f[R] and
|
|
\f[C]persist\f[R].
|
|
.PP
|
|
\f[C]remote\f[R] determines an existing remote name from the config
|
|
file, with trailing colon and optionally with a remote path.
|
|
See the full syntax in the rclone
|
|
documentation (https://rclone.org/docs/#syntax-of-remote-paths).
|
|
This option can be aliased as \f[C]fs\f[R] to prevent confusion with the
|
|
\f[I]remote\f[R] parameter of such backends as \f[I]crypt\f[R] or
|
|
\f[I]alias\f[R].
|
|
.PP
|
|
The \f[C]remote=:backend:dir/subdir\f[R] syntax can be used to create
|
|
on-the-fly (config-less)
|
|
remotes (https://rclone.org/docs/#backend-path-to-dir), while the
|
|
\f[C]type\f[R] and \f[C]path\f[R] options provide a simpler alternative
|
|
for this.
|
|
Using two split options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-o type=backend -o path=dir/subdir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
is equivalent to the combined syntax
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-o remote=:backend:dir/subdir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
but is arguably easier to parameterize in scripts.
|
|
The \f[C]path\f[R] part is optional.
|
|
.PP
|
|
Mount and VFS
|
|
options (https://rclone.org/commands/rclone_serve_docker/#options) as
|
|
well as backend parameters (https://rclone.org/flags/#backend-flags) are
|
|
named like their twin command-line flags without the \f[C]--\f[R] CLI
|
|
prefix.
|
|
Optionally you can use underscores instead of dashes in option names.
|
|
For example, \f[C]--vfs-cache-mode full\f[R] becomes
|
|
\f[C]-o vfs-cache-mode=full\f[R] or \f[C]-o vfs_cache_mode=full\f[R].
|
|
Boolean CLI flags without value will gain the \f[C]true\f[R] value, e.g.
|
|
\f[C]--allow-other\f[R] becomes \f[C]-o allow-other=true\f[R] or
|
|
\f[C]-o allow_other=true\f[R].
|
|
.PP
|
|
Please note that you can provide parameters only for the backend
|
|
immediately referenced by the backend type of mounted \f[C]remote\f[R].
|
|
If this is a wrapping backend like \f[I]alias, chunker or crypt\f[R],
|
|
you cannot provide options for the referred to remote or backend.
|
|
This limitation is imposed by the rclone connection string parser.
|
|
The only workaround is to feed plugin with \f[C]rclone.conf\f[R] or
|
|
configure plugin arguments (see below).
|
|
.SS Special Volume Options
|
|
.PP
|
|
\f[C]mount-type\f[R] determines the mount method and in general can be
|
|
one of: \f[C]mount\f[R], \f[C]cmount\f[R], or \f[C]mount2\f[R].
|
|
This can be aliased as \f[C]mount_type\f[R].
|
|
It should be noted that the managed rclone docker plugin currently does
|
|
not support the \f[C]cmount\f[R] method and \f[C]mount2\f[R] is rarely
|
|
needed.
|
|
This option defaults to the first found method, which is usually
|
|
\f[C]mount\f[R] so you generally won\[aq]t need it.
|
|
.PP
|
|
\f[C]persist\f[R] is a reserved boolean (true/false) option.
|
|
In future it will allow to persist on-the-fly remotes in the plugin
|
|
\f[C]rclone.conf\f[R] file.
|
|
.SS Connection Strings
|
|
.PP
|
|
The \f[C]remote\f[R] value can be extended with connection
|
|
strings (https://rclone.org/docs/#connection-strings) as an alternative
|
|
way to supply backend parameters.
|
|
This is equivalent to the \f[C]-o\f[R] backend options with one
|
|
\f[I]syntactic difference\f[R].
|
|
Inside connection string the backend prefix must be dropped from
|
|
parameter names but in the \f[C]-o param=value\f[R] array it must be
|
|
present.
|
|
For instance, compare the following option array
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-o remote=:sftp:/home -o sftp-host=localhost
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
with equivalent connection string:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
-o remote=:sftp,host=localhost:/home
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This difference exists because flag options \f[C]-o key=val\f[R] include
|
|
not only backend parameters but also mount/VFS flags and possibly other
|
|
settings.
|
|
Also it allows to discriminate the \f[C]remote\f[R] option from the
|
|
\f[C]crypt-remote\f[R] (or similarly named backend parameters) and
|
|
arguably simplifies scripting due to clearer value substitution.
|
|
.SS Using with Swarm or Compose
|
|
.PP
|
|
Both \f[I]Docker Swarm\f[R] and \f[I]Docker Compose\f[R] use
|
|
YAML (http://yaml.org/spec/1.2/spec.html)-formatted text files to
|
|
describe groups (stacks) of containers, their properties, networks and
|
|
volumes.
|
|
\f[I]Compose\f[R] uses the compose
|
|
v2 (https://docs.docker.com/compose/compose-file/compose-file-v2/#volume-configuration-reference)
|
|
format, \f[I]Swarm\f[R] uses the compose
|
|
v3 (https://docs.docker.com/compose/compose-file/compose-file-v3/#volume-configuration-reference)
|
|
format.
|
|
They are mostly similar, differences are explained in the docker
|
|
documentation (https://docs.docker.com/compose/compose-file/compose-versioning/#upgrading).
|
|
.PP
|
|
Volumes are described by the children of the top-level
|
|
\f[C]volumes:\f[R] node.
|
|
Each of them should be named after its volume and have at least two
|
|
elements, the self-explanatory \f[C]driver: rclone\f[R] value and the
|
|
\f[C]driver_opts:\f[R] structure playing the same role as
|
|
\f[C]-o key=val\f[R] CLI flags:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
volumes:
|
|
volume_name_1:
|
|
driver: rclone
|
|
driver_opts:
|
|
remote: \[aq]gdrive:\[aq]
|
|
allow_other: \[aq]true\[aq]
|
|
vfs_cache_mode: full
|
|
token: \[aq]{\[dq]type\[dq]: \[dq]borrower\[dq], \[dq]expires\[dq]: \[dq]2021-12-31\[dq]}\[aq]
|
|
poll_interval: 0
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Notice a few important details: - YAML prefers \f[C]_\f[R] in option
|
|
names instead of \f[C]-\f[R].
|
|
- YAML treats single and double quotes interchangeably.
|
|
Simple strings and integers can be left unquoted.
|
|
- Boolean values must be quoted like \f[C]\[aq]true\[aq]\f[R] or
|
|
\f[C]\[dq]false\[dq]\f[R] because these two words are reserved by YAML.
|
|
- The filesystem string is keyed with \f[C]remote\f[R] (or with
|
|
\f[C]fs\f[R]).
|
|
Normally you can omit quotes here, but if the string ends with colon,
|
|
you \f[B]must\f[R] quote it like
|
|
\f[C]remote: \[dq]storage_box:\[dq]\f[R].
|
|
- YAML is picky about surrounding braces in values as this is in fact
|
|
another syntax for key/value
|
|
mappings (http://yaml.org/spec/1.2/spec.html#id2790832).
|
|
For example, JSON access tokens usually contain double quotes and
|
|
surrounding braces, so you must put them in single quotes.
|
|
.SS Installing as Managed Plugin
|
|
.PP
|
|
Docker daemon can install plugins from an image registry and run them
|
|
managed.
|
|
We maintain the
|
|
docker-volume-rclone (https://hub.docker.com/p/rclone/docker-volume-rclone/)
|
|
plugin image on Docker Hub (https://hub.docker.com).
|
|
.PP
|
|
Rclone volume plugin requires \f[B]Docker Engine >= 19.03.15\f[R]
|
|
.PP
|
|
The plugin requires presence of two directories on the host before it
|
|
can be installed.
|
|
Note that plugin will \f[B]not\f[R] create them automatically.
|
|
By default they must exist on host at the following locations (though
|
|
you can tweak the paths): -
|
|
\f[C]/var/lib/docker-plugins/rclone/config\f[R] is reserved for the
|
|
\f[C]rclone.conf\f[R] config file and \f[B]must\f[R] exist even if
|
|
it\[aq]s empty and the config file is not present.
|
|
- \f[C]/var/lib/docker-plugins/rclone/cache\f[R] holds the plugin state
|
|
file as well as optional VFS caches.
|
|
.PP
|
|
You can install managed
|
|
plugin (https://docs.docker.com/engine/reference/commandline/plugin_install/)
|
|
with default settings as follows:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker plugin install rclone/docker-volume-rclone:amd64 --grant-all-permissions --alias rclone
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The \f[C]:amd64\f[R] part of the image specification after colon is
|
|
called a \f[I]tag\f[R].
|
|
Usually you will want to install the latest plugin for your
|
|
architecture.
|
|
In this case the tag will just name it, like \f[C]amd64\f[R] above.
|
|
The following plugin architectures are currently available: -
|
|
\f[C]amd64\f[R] - \f[C]arm64\f[R] - \f[C]arm-v7\f[R]
|
|
.PP
|
|
Sometimes you might want a concrete plugin version, not the latest one.
|
|
Then you should use image tag in the form
|
|
\f[C]:ARCHITECTURE-VERSION\f[R].
|
|
For example, to install plugin version \f[C]v1.56.2\f[R] on architecture
|
|
\f[C]arm64\f[R] you will use tag \f[C]arm64-1.56.2\f[R] (note the
|
|
removed \f[C]v\f[R]) so the full image specification becomes
|
|
\f[C]rclone/docker-volume-rclone:arm64-1.56.2\f[R].
|
|
.PP
|
|
We also provide the \f[C]latest\f[R] plugin tag, but since docker does
|
|
not support multi-architecture plugins as of the time of this writing,
|
|
this tag is currently an \f[B]alias for \f[CB]amd64\f[B]\f[R].
|
|
By convention the \f[C]latest\f[R] tag is the default one and can be
|
|
omitted, thus both \f[C]rclone/docker-volume-rclone:latest\f[R] and just
|
|
\f[C]rclone/docker-volume-rclone\f[R] will refer to the latest plugin
|
|
release for the \f[C]amd64\f[R] platform.
|
|
.PP
|
|
Also the \f[C]amd64\f[R] part can be omitted from the versioned rclone
|
|
plugin tags.
|
|
For example, rclone image reference
|
|
\f[C]rclone/docker-volume-rclone:amd64-1.56.2\f[R] can be abbreviated as
|
|
\f[C]rclone/docker-volume-rclone:1.56.2\f[R] for convenience.
|
|
However, for non-intel architectures you still have to use the full tag
|
|
as \f[C]amd64\f[R] or \f[C]latest\f[R] will fail to start.
|
|
.PP
|
|
Managed plugin is in fact a special container running in a namespace
|
|
separate from normal docker containers.
|
|
Inside it runs the \f[C]rclone serve docker\f[R] command.
|
|
The config and cache directories are bind-mounted into the container at
|
|
start.
|
|
The docker daemon connects to a unix socket created by the command
|
|
inside the container.
|
|
The command creates on-demand remote mounts right inside, then docker
|
|
machinery propagates them through kernel mount namespaces and
|
|
bind-mounts into requesting user containers.
|
|
.PP
|
|
You can tweak a few plugin settings after installation when it\[aq]s
|
|
disabled (not in use), for instance:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker plugin disable rclone
|
|
docker plugin set rclone RCLONE_VERBOSE=2 config=/etc/rclone args=\[dq]--vfs-cache-mode=writes --allow-other\[dq]
|
|
docker plugin enable rclone
|
|
docker plugin inspect rclone
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that if docker refuses to disable the plugin, you should find and
|
|
remove all active volumes connected with it as well as containers and
|
|
swarm services that use them.
|
|
This is rather tedious so please carefully plan in advance.
|
|
.PP
|
|
You can tweak the following settings: \f[C]args\f[R], \f[C]config\f[R],
|
|
\f[C]cache\f[R], \f[C]HTTP_PROXY\f[R], \f[C]HTTPS_PROXY\f[R],
|
|
\f[C]NO_PROXY\f[R] and \f[C]RCLONE_VERBOSE\f[R].
|
|
It\[aq]s \f[I]your\f[R] task to keep plugin settings in sync across
|
|
swarm cluster nodes.
|
|
.PP
|
|
\f[C]args\f[R] sets command-line arguments for the
|
|
\f[C]rclone serve docker\f[R] command (\f[I]none\f[R] by default).
|
|
Arguments should be separated by space so you will normally want to put
|
|
them in quotes on the docker plugin
|
|
set (https://docs.docker.com/engine/reference/commandline/plugin_set/)
|
|
command line.
|
|
Both serve docker
|
|
flags (https://rclone.org/commands/rclone_serve_docker/#options) and
|
|
generic rclone flags (https://rclone.org/flags/) are supported,
|
|
including backend parameters that will be used as defaults for volume
|
|
creation.
|
|
Note that plugin will fail (due to this docker
|
|
bug (https://github.com/moby/moby/blob/v20.10.7/plugin/v2/plugin.go#L195))
|
|
if the \f[C]args\f[R] value is empty.
|
|
Use e.g.
|
|
\f[C]args=\[dq]-v\[dq]\f[R] as a workaround.
|
|
.PP
|
|
\f[C]config=/host/dir\f[R] sets alternative host location for the config
|
|
directory.
|
|
Plugin will look for \f[C]rclone.conf\f[R] here.
|
|
It\[aq]s not an error if the config file is not present but the
|
|
directory must exist.
|
|
Please note that plugin can periodically rewrite the config file, for
|
|
example when it renews storage access tokens.
|
|
Keep this in mind and try to avoid races between the plugin and other
|
|
instances of rclone on the host that might try to change the config
|
|
simultaneously resulting in corrupted \f[C]rclone.conf\f[R].
|
|
You can also put stuff like private key files for SFTP remotes in this
|
|
directory.
|
|
Just note that it\[aq]s bind-mounted inside the plugin container at the
|
|
predefined path \f[C]/data/config\f[R].
|
|
For example, if your key file is named \f[C]sftp-box1.key\f[R] on the
|
|
host, the corresponding volume config option should read
|
|
\f[C]-o sftp-key-file=/data/config/sftp-box1.key\f[R].
|
|
.PP
|
|
\f[C]cache=/host/dir\f[R] sets alternative host location for the
|
|
\f[I]cache\f[R] directory.
|
|
The plugin will keep VFS caches here.
|
|
Also it will create and maintain the \f[C]docker-plugin.state\f[R] file
|
|
in this directory.
|
|
When the plugin is restarted or reinstalled, it will look in this file
|
|
to recreate any volumes that existed previously.
|
|
However, they will not be re-mounted into consuming containers after
|
|
restart.
|
|
Usually this is not a problem as the docker daemon normally will restart
|
|
affected user containers after failures, daemon restarts or host
|
|
reboots.
|
|
.PP
|
|
\f[C]RCLONE_VERBOSE\f[R] sets plugin verbosity from \f[C]0\f[R] (errors
|
|
only, by default) to \f[C]2\f[R] (debugging).
|
|
Verbosity can be also tweaked via \f[C]args=\[dq]-v [-v] ...\[dq]\f[R].
|
|
Since arguments are more generic, you will rarely need this setting.
|
|
The plugin output by default feeds the docker daemon log on local host.
|
|
Log entries are reflected as \f[I]errors\f[R] in the docker log but
|
|
retain their actual level assigned by rclone in the encapsulated message
|
|
string.
|
|
.PP
|
|
\f[C]HTTP_PROXY\f[R], \f[C]HTTPS_PROXY\f[R], \f[C]NO_PROXY\f[R]
|
|
customize the plugin proxy settings.
|
|
.PP
|
|
You can set custom plugin options right when you install it, \f[I]in one
|
|
go\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker plugin remove rclone
|
|
docker plugin install rclone/docker-volume-rclone:amd64 \[rs]
|
|
--alias rclone --grant-all-permissions \[rs]
|
|
args=\[dq]-v --allow-other\[dq] config=/etc/rclone
|
|
docker plugin inspect rclone
|
|
\f[R]
|
|
.fi
|
|
.SS Healthchecks
|
|
.PP
|
|
The docker plugin volume protocol doesn\[aq]t provide a way for plugins
|
|
to inform the docker daemon that a volume is (un-)available.
|
|
As a workaround you can setup a healthcheck to verify that the mount is
|
|
responding, for example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
services:
|
|
my_service:
|
|
image: my_image
|
|
healthcheck:
|
|
test: ls /path/to/rclone/mount || exit 1
|
|
interval: 1m
|
|
timeout: 15s
|
|
retries: 3
|
|
start_period: 15s
|
|
\f[R]
|
|
.fi
|
|
.SS Running Plugin under Systemd
|
|
.PP
|
|
In most cases you should prefer managed mode.
|
|
Moreover, MacOS and Windows do not support native Docker plugins.
|
|
Please use managed mode on these systems.
|
|
Proceed further only if you are on Linux.
|
|
.PP
|
|
First, install rclone (https://rclone.org/install/).
|
|
You can just run it (type \f[C]rclone serve docker\f[R] and hit enter)
|
|
for the test.
|
|
.PP
|
|
Install \f[I]FUSE\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo apt-get -y install fuse
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Download two systemd configuration files:
|
|
docker-volume-rclone.service (https://raw.githubusercontent.com/rclone/rclone/master/contrib/docker-plugin/systemd/docker-volume-rclone.service)
|
|
and
|
|
docker-volume-rclone.socket (https://raw.githubusercontent.com/rclone/rclone/master/contrib/docker-plugin/systemd/docker-volume-rclone.socket).
|
|
.PP
|
|
Put them to the \f[C]/etc/systemd/system/\f[R] directory:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
cp docker-volume-plugin.service /etc/systemd/system/
|
|
cp docker-volume-plugin.socket /etc/systemd/system/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Please note that all commands in this section must be run as
|
|
\f[I]root\f[R] but we omit \f[C]sudo\f[R] prefix for brevity.
|
|
Now create directories required by the service:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
mkdir -p /var/lib/docker-volumes/rclone
|
|
mkdir -p /var/lib/docker-plugins/rclone/config
|
|
mkdir -p /var/lib/docker-plugins/rclone/cache
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Run the docker plugin service in the socket activated mode:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
systemctl daemon-reload
|
|
systemctl start docker-volume-rclone.service
|
|
systemctl enable docker-volume-rclone.socket
|
|
systemctl start docker-volume-rclone.socket
|
|
systemctl restart docker
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or run the service directly: - run \f[C]systemctl daemon-reload\f[R] to
|
|
let systemd pick up new config - run
|
|
\f[C]systemctl enable docker-volume-rclone.service\f[R] to make the new
|
|
service start automatically when you power on your machine.
|
|
- run \f[C]systemctl start docker-volume-rclone.service\f[R] to start
|
|
the service now.
|
|
- run \f[C]systemctl restart docker\f[R] to restart docker daemon and
|
|
let it detect the new plugin socket.
|
|
Note that this step is not needed in managed mode where docker knows
|
|
about plugin state changes.
|
|
.PP
|
|
The two methods are equivalent from the user perspective, but I
|
|
personally prefer socket activation.
|
|
.SS Troubleshooting
|
|
.PP
|
|
You can see managed plugin
|
|
settings (https://docs.docker.com/engine/extend/#debugging-plugins) with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker plugin list
|
|
docker plugin inspect rclone
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that docker (including latest 20.10.7) will not show actual values
|
|
of \f[C]args\f[R], just the defaults.
|
|
.PP
|
|
Use \f[C]journalctl --unit docker\f[R] to see managed plugin output as
|
|
part of the docker daemon log.
|
|
Note that docker reflects plugin lines as \f[I]errors\f[R] but their
|
|
actual level can be seen from encapsulated message string.
|
|
.PP
|
|
You will usually install the latest version of managed plugin for your
|
|
platform.
|
|
Use the following commands to print the actual installed version:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
PLUGID=$(docker plugin list --no-trunc | awk \[aq]/rclone/{print$1}\[aq])
|
|
sudo runc --root /run/docker/runtime-runc/plugins.moby exec $PLUGID rclone version
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can even use \f[C]runc\f[R] to run shell inside the plugin
|
|
container:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo runc --root /run/docker/runtime-runc/plugins.moby exec --tty $PLUGID bash
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Also you can use curl to check the plugin socket connectivity:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker plugin list --no-trunc
|
|
PLUGID=123abc...
|
|
sudo curl -H Content-Type:application/json -XPOST -d {} --unix-socket /run/docker/plugins/$PLUGID/rclone.sock http://localhost/Plugin.Activate
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
though this is rarely needed.
|
|
.SS Caveats
|
|
.PP
|
|
Finally I\[aq]d like to mention a \f[I]caveat with updating volume
|
|
settings\f[R].
|
|
Docker CLI does not have a dedicated command like
|
|
\f[C]docker volume update\f[R].
|
|
It may be tempting to invoke \f[C]docker volume create\f[R] with updated
|
|
options on existing volume, but there is a gotcha.
|
|
The command will do nothing, it won\[aq]t even return an error.
|
|
I hope that docker maintainers will fix this some day.
|
|
In the meantime be aware that you must remove your volume before
|
|
recreating it with new settings:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker volume remove my_vol
|
|
docker volume create my_vol -d rclone -o opt1=new_val1 ...
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
and verify that settings did update:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker volume list
|
|
docker volume inspect my_vol
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If docker refuses to remove the volume, you should find containers or
|
|
swarm services that use it and stop them first.
|
|
.SS Getting started
|
|
.IP \[bu] 2
|
|
Install rclone (https://rclone.org/install/) and setup your remotes.
|
|
.IP \[bu] 2
|
|
Bisync will create its working directory at
|
|
\f[C]\[ti]/.cache/rclone/bisync\f[R] on Linux or
|
|
\f[C]C:\[rs]Users\[rs]MyLogin\[rs]AppData\[rs]Local\[rs]rclone\[rs]bisync\f[R]
|
|
on Windows.
|
|
Make sure that this location is writable.
|
|
.IP \[bu] 2
|
|
Run bisync with the \f[C]--resync\f[R] flag, specifying the paths to the
|
|
local and remote sync directory roots.
|
|
.IP \[bu] 2
|
|
For successive sync runs, leave off the \f[C]--resync\f[R] flag.
|
|
.IP \[bu] 2
|
|
Consider using a filters file for excluding unnecessary files and
|
|
directories from the sync.
|
|
.IP \[bu] 2
|
|
Consider setting up the --check-access feature for safety.
|
|
.IP \[bu] 2
|
|
On Linux, consider setting up a crontab entry.
|
|
bisync can safely run in concurrent cron jobs thanks to lock files it
|
|
maintains.
|
|
.PP
|
|
Here is a typical run log (with timestamps removed for clarity):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone bisync /testdir/path1/ /testdir/path2/ --verbose
|
|
INFO : Synching Path1 \[dq]/testdir/path1/\[dq] with Path2 \[dq]/testdir/path2/\[dq]
|
|
INFO : Path1 checking for diffs
|
|
INFO : - Path1 File is new - file11.txt
|
|
INFO : - Path1 File is newer - file2.txt
|
|
INFO : - Path1 File is newer - file5.txt
|
|
INFO : - Path1 File is newer - file7.txt
|
|
INFO : - Path1 File was deleted - file4.txt
|
|
INFO : - Path1 File was deleted - file6.txt
|
|
INFO : - Path1 File was deleted - file8.txt
|
|
INFO : Path1: 7 changes: 1 new, 3 newer, 0 older, 3 deleted
|
|
INFO : Path2 checking for diffs
|
|
INFO : - Path2 File is new - file10.txt
|
|
INFO : - Path2 File is newer - file1.txt
|
|
INFO : - Path2 File is newer - file5.txt
|
|
INFO : - Path2 File is newer - file6.txt
|
|
INFO : - Path2 File was deleted - file3.txt
|
|
INFO : - Path2 File was deleted - file7.txt
|
|
INFO : - Path2 File was deleted - file8.txt
|
|
INFO : Path2: 7 changes: 1 new, 3 newer, 0 older, 3 deleted
|
|
INFO : Applying changes
|
|
INFO : - Path1 Queue copy to Path2 - /testdir/path2/file11.txt
|
|
INFO : - Path1 Queue copy to Path2 - /testdir/path2/file2.txt
|
|
INFO : - Path2 Queue delete - /testdir/path2/file4.txt
|
|
NOTICE: - WARNING New or changed in both paths - file5.txt
|
|
NOTICE: - Path1 Renaming Path1 copy - /testdir/path1/file5.txt..path1
|
|
NOTICE: - Path1 Queue copy to Path2 - /testdir/path2/file5.txt..path1
|
|
NOTICE: - Path2 Renaming Path2 copy - /testdir/path2/file5.txt..path2
|
|
NOTICE: - Path2 Queue copy to Path1 - /testdir/path1/file5.txt..path2
|
|
INFO : - Path2 Queue copy to Path1 - /testdir/path1/file6.txt
|
|
INFO : - Path1 Queue copy to Path2 - /testdir/path2/file7.txt
|
|
INFO : - Path2 Queue copy to Path1 - /testdir/path1/file1.txt
|
|
INFO : - Path2 Queue copy to Path1 - /testdir/path1/file10.txt
|
|
INFO : - Path1 Queue delete - /testdir/path1/file3.txt
|
|
INFO : - Path2 Do queued copies to - Path1
|
|
INFO : - Path1 Do queued copies to - Path2
|
|
INFO : - Do queued deletes on - Path1
|
|
INFO : - Do queued deletes on - Path2
|
|
INFO : Updating listings
|
|
INFO : Validating listings for Path1 \[dq]/testdir/path1/\[dq] vs Path2 \[dq]/testdir/path2/\[dq]
|
|
INFO : Bisync successful
|
|
\f[R]
|
|
.fi
|
|
.SS Command line syntax
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone bisync --help
|
|
Usage:
|
|
rclone bisync remote1:path1 remote2:path2 [flags]
|
|
|
|
Positional arguments:
|
|
Path1, Path2 Local path, or remote storage with \[aq]:\[aq] plus optional path.
|
|
Type \[aq]rclone listremotes\[aq] for list of configured remotes.
|
|
|
|
Optional Flags:
|
|
--check-access Ensure expected \[ga]RCLONE_TEST\[ga] files are found on
|
|
both Path1 and Path2 filesystems, else abort.
|
|
--check-filename FILENAME Filename for \[ga]--check-access\[ga] (default: \[ga]RCLONE_TEST\[ga])
|
|
--check-sync CHOICE Controls comparison of final listings:
|
|
\[ga]true | false | only\[ga] (default: true)
|
|
If set to \[ga]only\[ga], bisync will only compare listings
|
|
from the last run but skip actual sync.
|
|
--filters-file PATH Read filtering patterns from a file
|
|
--max-delete PERCENT Safety check on maximum percentage of deleted files allowed.
|
|
If exceeded, the bisync run will abort. (default: 50%)
|
|
--force Bypass \[ga]--max-delete\[ga] safety check and run the sync.
|
|
Consider using with \[ga]--verbose\[ga]
|
|
--remove-empty-dirs Remove empty directories at the final cleanup step.
|
|
-1, --resync Performs the resync run.
|
|
Warning: Path1 files may overwrite Path2 versions.
|
|
Consider using \[ga]--verbose\[ga] or \[ga]--dry-run\[ga] first.
|
|
--localtime Use local time in listings (default: UTC)
|
|
--no-cleanup Retain working files (useful for troubleshooting and testing).
|
|
--workdir PATH Use custom working directory (useful for testing).
|
|
(default: \[ga]\[ti]/.cache/rclone/bisync\[ga])
|
|
-n, --dry-run Go through the motions - No files are copied/deleted.
|
|
-v, --verbose Increases logging verbosity.
|
|
May be specified more than once for more details.
|
|
-h, --help help for bisync
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Arbitrary rclone flags may be specified on the bisync command
|
|
line (https://rclone.org/commands/rclone_bisync/), for example
|
|
\f[C]rclone bsync ./testdir/path1/ gdrive:testdir/path2/ --drive-skip-gdocs -v -v --timeout 10s\f[R]
|
|
Note that interactions of various rclone flags with bisync process flow
|
|
has not been fully tested yet.
|
|
.SS Paths
|
|
.PP
|
|
Path1 and Path2 arguments may be references to any mix of local
|
|
directory paths (absolute or relative), UNC paths
|
|
(\f[C]//server/share/path\f[R]), Windows drive paths (with a drive
|
|
letter and \f[C]:\f[R]) or configured
|
|
remotes (https://rclone.org/docs/#syntax-of-remote-paths) with optional
|
|
subdirectory paths.
|
|
Cloud references are distinguished by having a \f[C]:\f[R] in the
|
|
argument (see Windows support below).
|
|
.PP
|
|
Path1 and Path2 are treated equally, in that neither has priority for
|
|
file changes, and access efficiency does not change whether a remote is
|
|
on Path1 or Path2.
|
|
.PP
|
|
The listings in bisync working directory (default:
|
|
\f[C]\[ti]/.cache/rclone/bisync\f[R]) are named based on the Path1 and
|
|
Path2 arguments so that separate syncs to individual directories within
|
|
the tree may be set up, e.g.:
|
|
\f[C]path_to_local_tree..dropbox_subdir.lst\f[R].
|
|
.PP
|
|
Any empty directories after the sync on both the Path1 and Path2
|
|
filesystems are not deleted by default.
|
|
If the \f[C]--remove-empty-dirs\f[R] flag is specified, then both paths
|
|
will have any empty directories purged as the last step in the process.
|
|
.SS Command-line flags
|
|
.SS --resync
|
|
.PP
|
|
This will effectively make both Path1 and Path2 filesystems contain a
|
|
matching superset of all files.
|
|
Path2 files that do not exist in Path1 will be copied to Path1, and the
|
|
process will then sync the Path1 tree to Path2.
|
|
.PP
|
|
The base directories on the both Path1 and Path2 filesystems must exist
|
|
or bisync will fail.
|
|
This is required for safety - that bisync can verify that both paths are
|
|
valid.
|
|
.PP
|
|
When using \f[C]--resync\f[R] a newer version of a file on the Path2
|
|
filesystem will be overwritten by the Path1 filesystem version.
|
|
Carefully evaluate deltas using
|
|
--dry-run (https://rclone.org/flags/#non-backend-flags).
|
|
.PP
|
|
For a resync run, one of the paths may be empty (no files in the path
|
|
tree).
|
|
The resync run should result in files on both paths, else a normal
|
|
non-resync run will fail.
|
|
.PP
|
|
For a non-resync run, either path being empty (no files in the tree)
|
|
fails with
|
|
\f[C]Empty current PathN listing. Cannot sync to an empty directory: X.pathN.lst\f[R]
|
|
This is a safety check that an unexpected empty path does not result in
|
|
deleting \f[B]everything\f[R] in the other path.
|
|
.SS --check-access
|
|
.PP
|
|
Access check files are an additional safety measure against data loss.
|
|
bisync will ensure it can find matching \f[C]RCLONE_TEST\f[R] files in
|
|
the same places in the Path1 and Path2 filesystems.
|
|
Time stamps and file contents are not important, just the names and
|
|
locations.
|
|
Place one or more \f[C]RCLONE_TEST\f[R] files in the Path1 or Path2
|
|
filesystem and then do either a run without \f[C]--check-access\f[R] or
|
|
a \f[C]--resync\f[R] to set matching files on both filesystems.
|
|
If you have symbolic links in your sync tree it is recommended to place
|
|
\f[C]RCLONE_TEST\f[R] files in the linked-to directory tree to protect
|
|
against bisync assuming a bunch of deleted files if the linked-to tree
|
|
should not be accessible.
|
|
Also see the \f[C]--check-filename\f[R] flag.
|
|
.SS --max-delete
|
|
.PP
|
|
As a safety check, if greater than the \f[C]--max-delete\f[R] percent of
|
|
files were deleted on either the Path1 or Path2 filesystem, then bisync
|
|
will abort with a warning message, without making any changes.
|
|
The default \f[C]--max-delete\f[R] is \f[C]50%\f[R].
|
|
One way to trigger this limit is to rename a directory that contains
|
|
more than half of your files.
|
|
This will appear to bisync as a bunch of deleted files and a bunch of
|
|
new files.
|
|
This safety check is intended to block bisync from deleting all of the
|
|
files on both filesystems due to a temporary network access issue, or if
|
|
the user had inadvertently deleted the files on one side or the other.
|
|
To force the sync either set a different delete percentage limit, e.g.
|
|
\f[C]--max-delete 75\f[R] (allows up to 75% deletion), or use
|
|
\f[C]--force\f[R] to bypass the check.
|
|
.PP
|
|
Also see the all files changed check.
|
|
.SS --filters-file
|
|
.PP
|
|
By using rclone filter features you can exclude file types or directory
|
|
sub-trees from the sync.
|
|
See the bisync filters section and generic
|
|
--filter-from (https://rclone.org/filtering/#filter-from-read-filtering-patterns-from-a-file)
|
|
documentation.
|
|
An example filters file contains filters for non-allowed files for
|
|
synching with Dropbox.
|
|
.PP
|
|
If you make changes to your filters file then bisync requires a run with
|
|
\f[C]--resync\f[R].
|
|
This is a safety feature, which avoids existing files on the Path1
|
|
and/or Path2 side from seeming to disappear from view (since they are
|
|
excluded in the new listings), which would fool bisync into seeing them
|
|
as deleted (as compared to the prior run listings), and then bisync
|
|
would proceed to delete them for real.
|
|
.PP
|
|
To block this from happening bisync calculates an MD5 hash of the
|
|
filters file and stores the hash in a \f[C].md5\f[R] file in the same
|
|
place as your filters file.
|
|
On the next runs with \f[C]--filters-file\f[R] set, bisync re-calculates
|
|
the MD5 hash of the current filters file and compares it to the hash
|
|
stored in \f[C].md5\f[R] file.
|
|
If they don\[aq]t match the run aborts with a critical error and thus
|
|
forces you to do a \f[C]--resync\f[R], likely avoiding a disaster.
|
|
.SS --check-sync
|
|
.PP
|
|
Enabled by default, the check-sync function checks that all of the same
|
|
files exist in both the Path1 and Path2 history listings.
|
|
This \f[I]check-sync\f[R] integrity check is performed at the end of the
|
|
sync run by default.
|
|
Any untrapped failing copy/deletes between the two paths might result in
|
|
differences between the two listings and in the untracked file content
|
|
differences between the two paths.
|
|
A resync run would correct the error.
|
|
.PP
|
|
Note that the default-enabled integrity check locally executes a load of
|
|
both the final Path1 and Path2 listings, and thus adds to the run time
|
|
of a sync.
|
|
Using \f[C]--check-sync=false\f[R] will disable it and may significantly
|
|
reduce the sync run times for very large numbers of files.
|
|
.PP
|
|
The check may be run manually with \f[C]--check-sync=only\f[R].
|
|
It runs only the integrity check and terminates without actually
|
|
synching.
|
|
.SS Operation
|
|
.SS Runtime flow details
|
|
.PP
|
|
bisync retains the listings of the \f[C]Path1\f[R] and \f[C]Path2\f[R]
|
|
filesystems from the prior run.
|
|
On each successive run it will:
|
|
.IP \[bu] 2
|
|
list files on \f[C]path1\f[R] and \f[C]path2\f[R], and check for changes
|
|
on each side.
|
|
Changes include \f[C]New\f[R], \f[C]Newer\f[R], \f[C]Older\f[R], and
|
|
\f[C]Deleted\f[R] files.
|
|
.IP \[bu] 2
|
|
Propagate changes on \f[C]path1\f[R] to \f[C]path2\f[R], and vice-versa.
|
|
.SS Safety measures
|
|
.IP \[bu] 2
|
|
Lock file prevents multiple simultaneous runs when taking a while.
|
|
This can be particularly useful if bisync is run by cron scheduler.
|
|
.IP \[bu] 2
|
|
Handle change conflicts non-destructively by creating \f[C]..path1\f[R]
|
|
and \f[C]..path2\f[R] file versions.
|
|
.IP \[bu] 2
|
|
File system access health check using \f[C]RCLONE_TEST\f[R] files (see
|
|
the \f[C]--check-access\f[R] flag).
|
|
.IP \[bu] 2
|
|
Abort on excessive deletes - protects against a failed listing being
|
|
interpreted as all the files were deleted.
|
|
See the \f[C]--max-delete\f[R] and \f[C]--force\f[R] flags.
|
|
.IP \[bu] 2
|
|
If something evil happens, bisync goes into a safe state to block damage
|
|
by later runs.
|
|
(See Error Handling)
|
|
.SS Normal sync checks
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
lw(8.4n) lw(28.4n) lw(15.7n) lw(17.5n).
|
|
T{
|
|
Type
|
|
T}@T{
|
|
Description
|
|
T}@T{
|
|
Result
|
|
T}@T{
|
|
Implementation
|
|
T}
|
|
_
|
|
T{
|
|
Path2 new
|
|
T}@T{
|
|
File is new on Path2, does not exist on Path1
|
|
T}@T{
|
|
Path2 version survives
|
|
T}@T{
|
|
\f[C]rclone copy\f[R] Path2 to Path1
|
|
T}
|
|
T{
|
|
Path2 newer
|
|
T}@T{
|
|
File is newer on Path2, unchanged on Path1
|
|
T}@T{
|
|
Path2 version survives
|
|
T}@T{
|
|
\f[C]rclone copy\f[R] Path2 to Path1
|
|
T}
|
|
T{
|
|
Path2 deleted
|
|
T}@T{
|
|
File is deleted on Path2, unchanged on Path1
|
|
T}@T{
|
|
File is deleted
|
|
T}@T{
|
|
\f[C]rclone delete\f[R] Path1
|
|
T}
|
|
T{
|
|
Path1 new
|
|
T}@T{
|
|
File is new on Path1, does not exist on Path2
|
|
T}@T{
|
|
Path1 version survives
|
|
T}@T{
|
|
\f[C]rclone copy\f[R] Path1 to Path2
|
|
T}
|
|
T{
|
|
Path1 newer
|
|
T}@T{
|
|
File is newer on Path1, unchanged on Path2
|
|
T}@T{
|
|
Path1 version survives
|
|
T}@T{
|
|
\f[C]rclone copy\f[R] Path1 to Path2
|
|
T}
|
|
T{
|
|
Path1 older
|
|
T}@T{
|
|
File is older on Path1, unchanged on Path2
|
|
T}@T{
|
|
\f[I]Path1 version survives\f[R]
|
|
T}@T{
|
|
\f[C]rclone copy\f[R] Path1 to Path2
|
|
T}
|
|
T{
|
|
Path2 older
|
|
T}@T{
|
|
File is older on Path2, unchanged on Path1
|
|
T}@T{
|
|
\f[I]Path2 version survives\f[R]
|
|
T}@T{
|
|
\f[C]rclone copy\f[R] Path2 to Path1
|
|
T}
|
|
T{
|
|
Path1 deleted
|
|
T}@T{
|
|
File no longer exists on Path1
|
|
T}@T{
|
|
File is deleted
|
|
T}@T{
|
|
\f[C]rclone delete\f[R] Path2
|
|
T}
|
|
.TE
|
|
.SS Unusual sync checks
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
lw(17.2n) lw(21.0n) lw(19.4n) lw(12.4n).
|
|
T{
|
|
Type
|
|
T}@T{
|
|
Description
|
|
T}@T{
|
|
Result
|
|
T}@T{
|
|
Implementation
|
|
T}
|
|
_
|
|
T{
|
|
Path1 new AND Path2 new
|
|
T}@T{
|
|
File is new on Path1 AND new on Path2
|
|
T}@T{
|
|
Files renamed to _Path1 and _Path2
|
|
T}@T{
|
|
\f[C]rclone copy\f[R] _Path2 file to Path1, \f[C]rclone copy\f[R] _Path1
|
|
file to Path2
|
|
T}
|
|
T{
|
|
Path2 newer AND Path1 changed
|
|
T}@T{
|
|
File is newer on Path2 AND also changed (newer/older/size) on Path1
|
|
T}@T{
|
|
Files renamed to _Path1 and _Path2
|
|
T}@T{
|
|
\f[C]rclone copy\f[R] _Path2 file to Path1, \f[C]rclone copy\f[R] _Path1
|
|
file to Path2
|
|
T}
|
|
T{
|
|
Path2 newer AND Path1 deleted
|
|
T}@T{
|
|
File is newer on Path2 AND also deleted on Path1
|
|
T}@T{
|
|
Path2 version survives
|
|
T}@T{
|
|
\f[C]rclone copy\f[R] Path2 to Path1
|
|
T}
|
|
T{
|
|
Path2 deleted AND Path1 changed
|
|
T}@T{
|
|
File is deleted on Path2 AND changed (newer/older/size) on Path1
|
|
T}@T{
|
|
Path1 version survives
|
|
T}@T{
|
|
\f[C]rclone copy\f[R] Path1 to Path2
|
|
T}
|
|
T{
|
|
Path1 deleted AND Path2 changed
|
|
T}@T{
|
|
File is deleted on Path1 AND changed (newer/older/size) on Path2
|
|
T}@T{
|
|
Path2 version survives
|
|
T}@T{
|
|
\f[C]rclone copy\f[R] Path2 to Path1
|
|
T}
|
|
.TE
|
|
.SS All files changed check
|
|
.PP
|
|
if \f[I]all\f[R] prior existing files on either of the filesystems have
|
|
changed (e.g.
|
|
timestamps have changed due to changing the system\[aq]s timezone) then
|
|
bisync will abort without making any changes.
|
|
Any new files are not considered for this check.
|
|
You could use \f[C]--force\f[R] to force the sync (whichever side has
|
|
the changed timestamp files wins).
|
|
Alternately, a \f[C]--resync\f[R] may be used (Path1 versions will be
|
|
pushed to Path2).
|
|
Consider the situation carefully and perhaps use \f[C]--dry-run\f[R]
|
|
before you commit to the changes.
|
|
.SS Modification time
|
|
.PP
|
|
Bisync relies on file timestamps to identify changed files and will
|
|
\f[I]refuse\f[R] to operate if backend lacks the modification time
|
|
support.
|
|
.PP
|
|
If you or your application should change the content of a file without
|
|
changing the modification time then bisync will \f[I]not\f[R] notice the
|
|
change, and thus will not copy it to the other side.
|
|
.PP
|
|
Note that on some cloud storage systems it is not possible to have file
|
|
timestamps that match \f[I]precisely\f[R] between the local and other
|
|
filesystems.
|
|
.PP
|
|
Bisync\[aq]s approach to this problem is by tracking the changes on each
|
|
side \f[I]separately\f[R] over time with a local database of files in
|
|
that side then applying the resulting changes on the other side.
|
|
.SS Error handling
|
|
.PP
|
|
Certain bisync critical errors, such as file copy/move failing, will
|
|
result in a bisync lockout of following runs.
|
|
The lockout is asserted because the sync status and history of the Path1
|
|
and Path2 filesystems cannot be trusted, so it is safer to block any
|
|
further changes until someone checks things out.
|
|
The recovery is to do a \f[C]--resync\f[R] again.
|
|
.PP
|
|
It is recommended to use \f[C]--resync --dry-run --verbose\f[R]
|
|
initially and \f[I]carefully\f[R] review what changes will be made
|
|
before running the \f[C]--resync\f[R] without \f[C]--dry-run\f[R].
|
|
.PP
|
|
Most of these events come up due to a error status from an internal
|
|
call.
|
|
On such a critical error the \f[C]{...}.path1.lst\f[R] and
|
|
\f[C]{...}.path2.lst\f[R] listing files are renamed to extension
|
|
\f[C].lst-err\f[R], which blocks any future bisync runs (since the
|
|
normal \f[C].lst\f[R] files are not found).
|
|
Bisync keeps them under \f[C]bisync\f[R] subdirectory of the rclone
|
|
cache direcory, typically at \f[C]${HOME}/.cache/rclone/bisync/\f[R] on
|
|
Linux.
|
|
.PP
|
|
Some errors are considered temporary and re-running the bisync is not
|
|
blocked.
|
|
The \f[I]critical return\f[R] blocks further bisync runs.
|
|
.SS Lock file
|
|
.PP
|
|
When bisync is running, a lock file is created in the bisync working
|
|
directory, typically at
|
|
\f[C]\[ti]/.cache/rclone/bisync/PATH1..PATH2.lck\f[R] on Linux.
|
|
If bisync should crash or hang, the lock file will remain in place and
|
|
block any further runs of bisync \f[I]for the same paths\f[R].
|
|
Delete the lock file as part of debugging the situation.
|
|
The lock file effectively blocks follow-on (e.g., scheduled by
|
|
\f[I]cron\f[R]) runs when the prior invocation is taking a long time.
|
|
The lock file contains \f[I]PID\f[R] of the blocking process, which may
|
|
help in debug.
|
|
.PP
|
|
\f[B]Note\f[R] that while concurrent bisync runs are allowed, \f[I]be
|
|
very cautious\f[R] that there is no overlap in the trees being synched
|
|
between concurrent runs, lest there be replicated files, deleted files
|
|
and general mayhem.
|
|
.SS Return codes
|
|
.PP
|
|
\f[C]rclone bisync\f[R] returns the following codes to calling program:
|
|
- \f[C]0\f[R] on a successful run, - \f[C]1\f[R] for a non-critical
|
|
failing run (a rerun may be successful), - \f[C]2\f[R] for a critically
|
|
aborted run (requires a \f[C]--resync\f[R] to recover).
|
|
.SS Limitations
|
|
.SS Supported backends
|
|
.PP
|
|
Bisync is considered \f[I]BETA\f[R] and has been tested with the
|
|
following backends: - Local filesystem - Google Drive - Dropbox -
|
|
OneDrive - S3 - SFTP
|
|
.PP
|
|
It has not been fully tested with other services yet.
|
|
If it works, or sorta works, please let us know and we\[aq]ll update the
|
|
list.
|
|
Run the test suite to check for proper operation as described below.
|
|
.PP
|
|
First release of \f[C]rclone bisync\f[R] requires that underlying
|
|
backend supported the modification time feature and will refuse to run
|
|
otherwise.
|
|
This limitation will be lifted in a future \f[C]rclone bisync\f[R]
|
|
release.
|
|
.SS Concurrent modifications
|
|
.PP
|
|
When using \f[B]Local, FTP or SFTP\f[R] remotes rclone does not create
|
|
\f[I]temporary\f[R] files at the destination when copying, and thus if
|
|
the connection is lost the created file may be corrupt, which will
|
|
likely propagate back to the original path on the next sync, resulting
|
|
in data loss.
|
|
This will be solved in a future release, there is no workaround at the
|
|
moment.
|
|
.PP
|
|
Files that \f[B]change during\f[R] a bisync run may result in data loss.
|
|
This has been seen in a highly dynamic environment, where the filesystem
|
|
is getting hammered by running processes during the sync.
|
|
The solution is to sync at quiet times or filter out unnecessary
|
|
directories and files.
|
|
.SS Empty directories
|
|
.PP
|
|
New empty directories on one path are \f[I]not\f[R] propagated to the
|
|
other side.
|
|
This is because bisync (and rclone) natively works on files not
|
|
directories.
|
|
The following sequence is a workaround but will not propagate the delete
|
|
of an empty directory to the other side:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone bisync PATH1 PATH2
|
|
rclone copy PATH1 PATH2 --filter \[dq]+ */\[dq] --filter \[dq]- **\[dq] --create-empty-src-dirs
|
|
rclone copy PATH2 PATH2 --filter \[dq]+ */\[dq] --filter \[dq]- **\[dq] --create-empty-src-dirs
|
|
\f[R]
|
|
.fi
|
|
.SS Renamed directories
|
|
.PP
|
|
Renaming a folder on the Path1 side results is deleting all files on the
|
|
Path2 side and then copying all files again from Path1 to Path2.
|
|
Bisync sees this as all files in the old directory name as deleted and
|
|
all files in the new directory name as new.
|
|
Similarly, renaming a directory on both sides to the same name will
|
|
result in creating \f[C]..path1\f[R] and \f[C]..path2\f[R] files on both
|
|
sides.
|
|
Currently the most effective and efficient method of renaming a
|
|
directory is to rename it on both sides, then do a \f[C]--resync\f[R].
|
|
.SS Case sensitivity
|
|
.PP
|
|
Synching with \f[B]case-insensitive\f[R] filesystems, such as Windows or
|
|
\f[C]Box\f[R], can result in file name conflicts.
|
|
This will be fixed in a future release.
|
|
The near term workaround is to make sure that files on both sides
|
|
don\[aq]t have spelling case differences (\f[C]Smile.jpg\f[R] vs.
|
|
\f[C]smile.jpg\f[R]).
|
|
.SS Windows support
|
|
.PP
|
|
Bisync has been tested on Windows 8.1, Windows 10 Pro 64-bit and on
|
|
Windows Github runners.
|
|
.PP
|
|
Drive letters are allowed, including drive letters mapped to network
|
|
drives (\f[C]rclone bisync J:\[rs]localsync GDrive:\f[R]).
|
|
If a drive letter is omitted, the shell current drive is the default.
|
|
Drive letters are a single character follows by \f[C]:\f[R], so cloud
|
|
names must be more than one character long.
|
|
.PP
|
|
Absolute paths (with or without a drive letter), and relative paths
|
|
(with or without a drive letter) are supported.
|
|
.PP
|
|
Working directory is created at
|
|
\f[C]C:\[rs]Users\[rs]MyLogin\[rs]AppData\[rs]Local\[rs]rclone\[rs]bisync\f[R].
|
|
.PP
|
|
Note that bisync output may show a mix of forward \f[C]/\f[R] and back
|
|
\f[C]\[rs]\f[R] slashes.
|
|
.PP
|
|
Be careful of case independent directory and file naming on Windows vs.
|
|
case dependent Linux
|
|
.SS Filtering
|
|
.PP
|
|
See filtering documentation (https://rclone.org/filtering/) for how
|
|
filter rules are written and interpreted.
|
|
.PP
|
|
Bisync\[aq]s \f[C]--filters-file\f[R] flag slightly extends the
|
|
rclone\[aq]s
|
|
--filter-from (https://rclone.org/filtering/#filter-from-read-filtering-patterns-from-a-file)
|
|
filtering mechanism.
|
|
For a given bisync run you may provide \f[I]only one\f[R]
|
|
\f[C]--filters-file\f[R].
|
|
The \f[C]--include*\f[R], \f[C]--exclude*\f[R], and \f[C]--filter\f[R]
|
|
flags are also supported.
|
|
.SS How to filter directories
|
|
.PP
|
|
Filtering portions of the directory tree is a critical feature for
|
|
synching.
|
|
.PP
|
|
Examples of directory trees (always beneath the Path1/Path2 root level)
|
|
you may want to exclude from your sync: - Directory trees containing
|
|
only software build intermediate files.
|
|
- Directory trees containing application temporary files and data such
|
|
as the Windows \f[C]C:\[rs]Users\[rs]MyLogin\[rs]AppData\[rs]\f[R] tree.
|
|
- Directory trees containing files that are large, less important, or
|
|
are getting thrashed continuously by ongoing processes.
|
|
.PP
|
|
On the other hand, there may be only select directories that you
|
|
actually want to sync, and exclude all others.
|
|
See the Example include-style filters for Windows user directories
|
|
below.
|
|
.SS Filters file writing guidelines
|
|
.IP "1." 3
|
|
Begin with excluding directory trees:
|
|
.RS 4
|
|
.IP \[bu] 2
|
|
e.g.
|
|
\[ga]- /AppData/\[ga]
|
|
.IP \[bu] 2
|
|
\f[C]**\f[R] on the end is not necessary.
|
|
Once a given directory level is excluded then everything beneath it
|
|
won\[aq]t be looked at by rclone.
|
|
.IP \[bu] 2
|
|
Exclude such directories that are unneeded, are big, dynamically
|
|
thrashed, or where there may be access permission issues.
|
|
.IP \[bu] 2
|
|
Excluding such dirs first will make rclone operations (much) faster.
|
|
.IP \[bu] 2
|
|
Specific files may also be excluded, as with the Dropbox exclusions
|
|
example below.
|
|
.RE
|
|
.IP "2." 3
|
|
Decide if its easier (or cleaner) to:
|
|
.RS 4
|
|
.IP \[bu] 2
|
|
Include select directories and therefore \f[I]exclude everything
|
|
else\f[R] -- or --
|
|
.IP \[bu] 2
|
|
Exclude select directories and therefore \f[I]include everything
|
|
else\f[R]
|
|
.RE
|
|
.IP "3." 3
|
|
Include select directories:
|
|
.RS 4
|
|
.IP \[bu] 2
|
|
Add lines like: \[ga]+ /Documents/PersonalFiles/**\[ga] to select which
|
|
directories to include in the sync.
|
|
.IP \[bu] 2
|
|
\f[C]**\f[R] on the end specifies to include the full depth of the
|
|
specified tree.
|
|
.IP \[bu] 2
|
|
With Include-style filters, files at the Path1/Path2 root are not
|
|
included.
|
|
They may be included with \[ga]+ /*\[ga].
|
|
.IP \[bu] 2
|
|
Place RCLONE_TEST files within these included directory trees.
|
|
They will only be looked for in these directory trees.
|
|
.IP \[bu] 2
|
|
Finish by excluding everything else by adding \[ga]- **\[ga] at the end
|
|
of the filters file.
|
|
.IP \[bu] 2
|
|
Disregard step 4.
|
|
.RE
|
|
.IP "4." 3
|
|
Exclude select directories:
|
|
.RS 4
|
|
.IP \[bu] 2
|
|
Add more lines like in step 1.
|
|
For example: \f[C]-/Desktop/tempfiles/\f[R], or \[ga]-
|
|
/testdir/\f[C]. Again, a\f[R]**\[ga] on the end is not necessary.
|
|
.IP \[bu] 2
|
|
Do \f[I]not\f[R] add a \[ga]- **\[ga] in the file.
|
|
Without this line, everything will be included that has not be
|
|
explicitly excluded.
|
|
.IP \[bu] 2
|
|
Disregard step 3.
|
|
.RE
|
|
.PP
|
|
A few rules for the syntax of a filter file expanding on filtering
|
|
documentation (https://rclone.org/filtering/):
|
|
.IP \[bu] 2
|
|
Lines may start with spaces and tabs - rclone strips leading whitespace.
|
|
.IP \[bu] 2
|
|
If the first non-whitespace character is a \f[C]#\f[R] then the line is
|
|
a comment and will be ignored.
|
|
.IP \[bu] 2
|
|
Blank lines are ignored.
|
|
.IP \[bu] 2
|
|
The first non-whitespace character on a filter line must be a
|
|
\f[C]+\f[R] or \f[C]-\f[R].
|
|
.IP \[bu] 2
|
|
Exactly 1 space is allowed between the \f[C]+/-\f[R] and the path term.
|
|
.IP \[bu] 2
|
|
Only forward slashes (\f[C]/\f[R]) are used in path terms, even on
|
|
Windows.
|
|
.IP \[bu] 2
|
|
The rest of the line is taken as the path term.
|
|
Trailing whitespace is taken literally, and probably is an error.
|
|
.SS Example include-style filters for Windows user directories
|
|
.PP
|
|
This Windows \f[I]include-style\f[R] example is based on the sync root
|
|
(Path1) set to \f[C]C:\[rs]Users\[rs]MyLogin\f[R].
|
|
The strategy is to select specific directories to be synched with a
|
|
network drive (Path2).
|
|
.IP \[bu] 2
|
|
\[ga]- /AppData/\[ga] excludes an entire tree of Windows stored stuff
|
|
that need not be synched.
|
|
In my case, AppData has >11 GB of stuff I don\[aq]t care about, and
|
|
there are some subdirectories beneath AppData that are not accessible to
|
|
my user login, resulting in bisync critical aborts.
|
|
.IP \[bu] 2
|
|
Windows creates cache files starting with both upper and lowercase
|
|
\f[C]NTUSER\f[R] at \f[C]C:\[rs]Users\[rs]MyLogin\f[R].
|
|
These files may be dynamic, locked, and are generally \f[I]don\[aq]t
|
|
care\f[R].
|
|
.IP \[bu] 2
|
|
There are just a few directories with \f[I]my\f[R] data that I do want
|
|
synched, in the form of \[ga]+
|
|
/\f[C]. By selecting only the directory trees I want to avoid the dozen plus directories that various apps make at\f[R]C:\[ga].
|
|
.IP \[bu] 2
|
|
Include files in the root of the sync point,
|
|
\f[C]C:\[rs]Users\[rs]MyLogin\f[R], by adding the \[ga]+ /*\[ga] line.
|
|
.IP \[bu] 2
|
|
This is an Include-style filters file, therefore it ends with \[ga]-
|
|
**\[ga] which excludes everything not explicitly included.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
- /AppData/
|
|
- NTUSER*
|
|
- ntuser*
|
|
+ /Documents/Family/**
|
|
+ /Documents/Sketchup/**
|
|
+ /Documents/Microcapture_Photo/**
|
|
+ /Documents/Microcapture_Video/**
|
|
+ /Desktop/**
|
|
+ /Pictures/**
|
|
+ /*
|
|
- **
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note also that Windows implements several \[dq]library\[dq] links such
|
|
as \f[C]C:\[rs]Users\[rs]MyLogin\[rs]My Documents\[rs]My Music\f[R]
|
|
pointing to \f[C]C:\[rs]Users\[rs]MyLogin\[rs]Music\f[R].
|
|
rclone sees these as links, so you must add \f[C]--links\f[R] to the
|
|
bisync command line if you which to follow these links.
|
|
I find that I get permission errors in trying to follow the links, so I
|
|
don\[aq]t include the rclone \f[C]--links\f[R] flag, but then you get
|
|
lots of \f[C]Can\[aq]t follow symlink\&...\f[R] noise from rclone about
|
|
not following the links.
|
|
This noise can be quashed by adding \f[C]--quiet\f[R] to the bisync
|
|
command line.
|
|
.SS Example exclude-style filters files for use with Dropbox
|
|
.IP \[bu] 2
|
|
Dropbox disallows synching the listed temporary and configuration/data
|
|
files.
|
|
The \[ga]- \[ga] filters exclude these files where ever they may occur
|
|
in the sync tree.
|
|
Consider adding similar exclusions for file types you don\[aq]t need to
|
|
sync, such as core dump and software build files.
|
|
.IP \[bu] 2
|
|
bisync testing creates \f[C]/testdir/\f[R] at the top level of the sync
|
|
tree, and usually deletes the tree after the test.
|
|
If a normal sync should run while the \f[C]/testdir/\f[R] tree exists
|
|
the \f[C]--check-access\f[R] phase may fail due to unbalanced
|
|
RCLONE_TEST files.
|
|
The \[ga]- /testdir/\[ga] filter blocks this tree from being synched.
|
|
You don\[aq]t need this exclusion if you are not doing bisync
|
|
development testing.
|
|
.IP \[bu] 2
|
|
Everything else beneath the Path1/Path2 root will be synched.
|
|
.IP \[bu] 2
|
|
RCLONE_TEST files may be placed anywhere within the tree, including the
|
|
root.
|
|
.SS Example filters file for Dropbox
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
# Filter file for use with bisync
|
|
# See https://rclone.org/filtering/ for filtering rules
|
|
# NOTICE: If you make changes to this file you MUST do a --resync run.
|
|
# Run with --dry-run to see what changes will be made.
|
|
|
|
# Dropbox wont sync some files so filter them away here.
|
|
# See https://help.dropbox.com/installs-integrations/sync-uploads/files-not-syncing
|
|
- .dropbox.attr
|
|
- \[ti]*.tmp
|
|
- \[ti]$*
|
|
- .\[ti]*
|
|
- desktop.ini
|
|
- .dropbox
|
|
|
|
# Used for bisync testing, so excluded from normal runs
|
|
- /testdir/
|
|
|
|
# Other example filters
|
|
#- /TiBU/
|
|
#- /Photos/
|
|
\f[R]
|
|
.fi
|
|
.SS How --check-access handles filters
|
|
.PP
|
|
At the start of a bisync run, listings are gathered for Path1 and Path2
|
|
while using the user\[aq]s \f[C]--filters-file\f[R].
|
|
During the check access phase, bisync scans these listings for
|
|
\f[C]RCLONE_TEST\f[R] files.
|
|
Any \f[C]RCLONE_TEST\f[R] files hidden by the \f[C]--filters-file\f[R]
|
|
are \f[I]not\f[R] in the listings and thus not checked during the check
|
|
access phase.
|
|
.SS Troubleshooting
|
|
.SS Reading bisync logs
|
|
.PP
|
|
Here are two normal runs.
|
|
The first one has a newer file on the remote.
|
|
The second has no deltas between local and remote.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
2021/05/16 00:24:38 INFO : Synching Path1 \[dq]/path/to/local/tree/\[dq] with Path2 \[dq]dropbox:/\[dq]
|
|
2021/05/16 00:24:38 INFO : Path1 checking for diffs
|
|
2021/05/16 00:24:38 INFO : - Path1 File is new - file.txt
|
|
2021/05/16 00:24:38 INFO : Path1: 1 changes: 1 new, 0 newer, 0 older, 0 deleted
|
|
2021/05/16 00:24:38 INFO : Path2 checking for diffs
|
|
2021/05/16 00:24:38 INFO : Applying changes
|
|
2021/05/16 00:24:38 INFO : - Path1 Queue copy to Path2 - dropbox:/file.txt
|
|
2021/05/16 00:24:38 INFO : - Path1 Do queued copies to - Path2
|
|
2021/05/16 00:24:38 INFO : Updating listings
|
|
2021/05/16 00:24:38 INFO : Validating listings for Path1 \[dq]/path/to/local/tree/\[dq] vs Path2 \[dq]dropbox:/\[dq]
|
|
2021/05/16 00:24:38 INFO : Bisync successful
|
|
|
|
2021/05/16 00:36:52 INFO : Synching Path1 \[dq]/path/to/local/tree/\[dq] with Path2 \[dq]dropbox:/\[dq]
|
|
2021/05/16 00:36:52 INFO : Path1 checking for diffs
|
|
2021/05/16 00:36:52 INFO : Path2 checking for diffs
|
|
2021/05/16 00:36:52 INFO : No changes found
|
|
2021/05/16 00:36:52 INFO : Updating listings
|
|
2021/05/16 00:36:52 INFO : Validating listings for Path1 \[dq]/path/to/local/tree/\[dq] vs Path2 \[dq]dropbox:/\[dq]
|
|
2021/05/16 00:36:52 INFO : Bisync successful
|
|
\f[R]
|
|
.fi
|
|
.SS Dry run oddity
|
|
.PP
|
|
The \f[C]--dry-run\f[R] messages may indicate that it would try to
|
|
delete some files.
|
|
For example, if a file is new on Path2 and does not exist on Path1 then
|
|
it would normally be copied to Path1, but with \f[C]--dry-run\f[R]
|
|
enabled those copies don\[aq]t happen, which leads to the attempted
|
|
delete on the Path2, blocked again by --dry-run:
|
|
\f[C]... Not deleting as --dry-run\f[R].
|
|
.PP
|
|
This whole confusing situation is an artifact of the \f[C]--dry-run\f[R]
|
|
flag.
|
|
Scrutinize the proposed deletes carefully, and if the files would have
|
|
been copied to Path1 then the threatened deletes on Path2 may be
|
|
disregarded.
|
|
.SS Retries
|
|
.PP
|
|
Rclone has built in retries.
|
|
If you run with \f[C]--verbose\f[R] you\[aq]ll see error and retry
|
|
messages such as shown below.
|
|
This is usually not a bug.
|
|
If at the end of the run you see \f[C]Bisync successful\f[R] and not
|
|
\f[C]Bisync critical error\f[R] or \f[C]Bisync aborted\f[R] then the run
|
|
was successful, and you can ignore the error messages.
|
|
.PP
|
|
The following run shows an intermittent fail.
|
|
Lines \f[I]5\f[R] and _6- are low level messages.
|
|
Line \f[I]6\f[R] is a bubbled-up \f[I]warning\f[R] message, conveying
|
|
the error.
|
|
Rclone normally retries failing commands, so there may be numerous such
|
|
messages in the log.
|
|
.PP
|
|
Since there are no final error/warning messages on line \f[I]7\f[R],
|
|
rclone has recovered from failure after a retry, and the overall sync
|
|
was successful.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
1: 2021/05/14 00:44:12 INFO : Synching Path1 \[dq]/path/to/local/tree\[dq] with Path2 \[dq]dropbox:\[dq]
|
|
2: 2021/05/14 00:44:12 INFO : Path1 checking for diffs
|
|
3: 2021/05/14 00:44:12 INFO : Path2 checking for diffs
|
|
4: 2021/05/14 00:44:12 INFO : Path2: 113 changes: 22 new, 0 newer, 0 older, 91 deleted
|
|
5: 2021/05/14 00:44:12 ERROR : /path/to/local/tree/objects/af: error listing: unexpected end of JSON input
|
|
6: 2021/05/14 00:44:12 NOTICE: WARNING listing try 1 failed. - dropbox:
|
|
7: 2021/05/14 00:44:12 INFO : Bisync successful
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This log shows a \f[I]Critical failure\f[R] which requires a
|
|
\f[C]--resync\f[R] to recover from.
|
|
See the Runtime Error Handling section.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
2021/05/12 00:49:40 INFO : Google drive root \[aq]\[aq]: Waiting for checks to finish
|
|
2021/05/12 00:49:40 INFO : Google drive root \[aq]\[aq]: Waiting for transfers to finish
|
|
2021/05/12 00:49:40 INFO : Google drive root \[aq]\[aq]: not deleting files as there were IO errors
|
|
2021/05/12 00:49:40 ERROR : Attempt 3/3 failed with 3 errors and: not deleting files as there were IO errors
|
|
2021/05/12 00:49:40 ERROR : Failed to sync: not deleting files as there were IO errors
|
|
2021/05/12 00:49:40 NOTICE: WARNING rclone sync try 3 failed. - /path/to/local/tree/
|
|
2021/05/12 00:49:40 ERROR : Bisync aborted. Must run --resync to recover.
|
|
\f[R]
|
|
.fi
|
|
.SS Denied downloads of \[dq]infected\[dq] or \[dq]abusive\[dq] files
|
|
.PP
|
|
Google Drive has a filter for certain file types (\f[C].exe\f[R],
|
|
\f[C].apk\f[R], et cetera) that by default cannot be copied from Google
|
|
Drive to the local filesystem.
|
|
If you are having problems, run with \f[C]--verbose\f[R] to see
|
|
specifically which files are generating complaints.
|
|
If the error is
|
|
\f[C]This file has been identified as malware or spam and cannot be downloaded\f[R],
|
|
consider using the flag
|
|
--drive-acknowledge-abuse (https://rclone.org/drive/#drive-acknowledge-abuse).
|
|
.SS Google Doc files
|
|
.PP
|
|
Google docs exist as virtual files on Google Drive and cannot be
|
|
transferred to other filesystems natively.
|
|
While it is possible to export a Google doc to a normal file (with
|
|
\f[C].xlsx\f[R] extension, for example), it\[aq]s not possible to import
|
|
a normal file back into a Google document.
|
|
.PP
|
|
Bisync\[aq]s handling of Google Doc files is to flag them in the run log
|
|
output for user\[aq]s attention and ignore them for any file transfers,
|
|
deletes, or syncs.
|
|
They will show up with a length of \f[C]-1\f[R] in the listings.
|
|
This bisync run is otherwise successful:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
2021/05/11 08:23:15 INFO : Synching Path1 \[dq]/path/to/local/tree/base/\[dq] with Path2 \[dq]GDrive:\[dq]
|
|
2021/05/11 08:23:15 INFO : ...path2.lst-new: Ignoring incorrect line: \[dq]- -1 - - 2018-07-29T08:49:30.136000000+0000 GoogleDoc.docx\[dq]
|
|
2021/05/11 08:23:15 INFO : Bisync successful
|
|
\f[R]
|
|
.fi
|
|
.SS Usage examples
|
|
.SS Cron
|
|
.PP
|
|
Rclone does not yet have a built-in capability to monitor the local file
|
|
system for changes and must be blindly run periodically.
|
|
On Windows this can be done using a \f[I]Task Scheduler\f[R], on Linux
|
|
you can use \f[I]Cron\f[R] which is described below.
|
|
.PP
|
|
The 1st example runs a sync every 5 minutes between a local directory
|
|
and an OwnCloud server, with output logged to a runlog file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
# Minute (0-59)
|
|
# Hour (0-23)
|
|
# Day of Month (1-31)
|
|
# Month (1-12 or Jan-Dec)
|
|
# Day of Week (0-6 or Sun-Sat)
|
|
# Command
|
|
*/5 * * * * /path/to/rclone bisync /local/files MyCloud: --check-access --filters-file /path/to/bysync-filters.txt --log-file /path/to//bisync.log
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See crontab
|
|
syntax (https://www.man7.org/linux/man-pages/man1/crontab.1p.html#INPUT_FILES)).
|
|
for the details of crontab time interval expressions.
|
|
.PP
|
|
If you run \f[C]rclone bisync\f[R] as a cron job, redirect stdout/stderr
|
|
to a file.
|
|
The 2nd example runs a sync to Dropbox every hour and logs all stdout
|
|
(via the \f[C]>>\f[R]) and stderr (via \f[C]2>&1\f[R]) to a log file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
0 * * * * /path/to/rclone bisync /path/to/local/dropbox Dropbox: --check-access --filters-file /home/user/filters.txt >> /path/to/logs/dropbox-run.log 2>&1
|
|
\f[R]
|
|
.fi
|
|
.SS Sharing an encrypted folder tree between hosts
|
|
.PP
|
|
bisync can keep a local folder in sync with a cloud service, but what if
|
|
you have some highly sensitive files to be synched?
|
|
.PP
|
|
Usage of a cloud service is for exchanging both routine and sensitive
|
|
personal files between one\[aq]s home network, one\[aq]s personal
|
|
notebook when on the road, and with one\[aq]s work computer.
|
|
The routine data is not sensitive.
|
|
For the sensitive data, configure an rclone crypt
|
|
remote (https://rclone.org/crypt/) to point to a subdirectory within the
|
|
local disk tree that is bisync\[aq]d to Dropbox, and then set up an
|
|
bisync for this local crypt directory to a directory outside of the main
|
|
sync tree.
|
|
.SS Linux server setup
|
|
.IP \[bu] 2
|
|
\f[C]/path/to/DBoxroot\f[R] is the root of my local sync tree.
|
|
There are numerous subdirectories.
|
|
.IP \[bu] 2
|
|
\f[C]/path/to/DBoxroot/crypt\f[R] is the root subdirectory for files
|
|
that are encrypted.
|
|
This local directory target is setup as an rclone crypt remote named
|
|
\f[C]Dropcrypt:\f[R].
|
|
See rclone.conf snippet below.
|
|
.IP \[bu] 2
|
|
\f[C]/path/to/my/unencrypted/files\f[R] is the root of my sensitive
|
|
files - not encrypted, not within the tree synched to Dropbox.
|
|
.IP \[bu] 2
|
|
To sync my local unencrypted files with the encrypted Dropbox versions I
|
|
manually run \f[C]bisync /path/to/my/unencrypted/files DropCrypt:\f[R].
|
|
This step could be bundled into a script to run before and after the
|
|
full Dropbox tree sync in the last step, thus actively keeping the
|
|
sensitive files in sync.
|
|
.IP \[bu] 2
|
|
\f[C]bisync /path/to/DBoxroot Dropbox:\f[R] runs periodically via cron,
|
|
keeping my full local sync tree in sync with Dropbox.
|
|
.SS Windows notebook setup
|
|
.IP \[bu] 2
|
|
The Dropbox client runs keeping the local tree
|
|
\f[C]C:\[rs]Users\[rs]MyLogin\[rs]Dropbox\f[R] always in sync with
|
|
Dropbox.
|
|
I could have used \f[C]rclone bisync\f[R] instead.
|
|
.IP \[bu] 2
|
|
A separate directory tree at
|
|
\f[C]C:\[rs]Users\[rs]MyLogin\[rs]Documents\[rs]DropLocal\f[R] hosts the
|
|
tree of unencrypted files/folders.
|
|
.IP \[bu] 2
|
|
To sync my local unencrypted files with the encrypted Dropbox versions I
|
|
manually run the following command:
|
|
\f[C]rclone bisync C:\[rs]Users\[rs]MyLogin\[rs]Documents\[rs]DropLocal Dropcrypt:\f[R].
|
|
.IP \[bu] 2
|
|
The Dropbox client then syncs the changes with Dropbox.
|
|
.SS rclone.conf snippet
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[Dropbox]
|
|
type = dropbox
|
|
\&...
|
|
|
|
[Dropcrypt]
|
|
type = crypt
|
|
remote = /path/to/DBoxroot/crypt # on the Linux server
|
|
remote = C:\[rs]Users\[rs]MyLogin\[rs]Dropbox\[rs]crypt # on the Windows notebook
|
|
filename_encryption = standard
|
|
directory_name_encryption = true
|
|
password = ...
|
|
\&...
|
|
\f[R]
|
|
.fi
|
|
.SS Testing
|
|
.PP
|
|
You should read this section only if you are developing for rclone.
|
|
You need to have rclone source code locally to work with bisync tests.
|
|
.PP
|
|
Bisync has a dedicated test framework implemented in the
|
|
\f[C]bisync_test.go\f[R] file located in the rclone source tree.
|
|
The test suite is based on the \f[C]go test\f[R] command.
|
|
Series of tests are stored in subdirectories below the
|
|
\f[C]cmd/bisync/testdata\f[R] directory.
|
|
Individual tests can be invoked by their directory name, e.g.
|
|
\f[C]go test . -case basic -remote local -remote2 gdrive: -v\f[R]
|
|
.PP
|
|
Tests will make a temporary folder on remote and purge it afterwards.
|
|
If during test run there are intermittent errors and rclone retries,
|
|
these errors will be captured and flagged as invalid MISCOMPAREs.
|
|
Rerunning the test will let it pass.
|
|
Consider such failures as noise.
|
|
.SS Test command syntax
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
usage: go test ./cmd/bisync [options...]
|
|
|
|
Options:
|
|
-case NAME Name(s) of the test case(s) to run. Multiple names should
|
|
be separated by commas. You can remove the \[ga]test_\[ga] prefix
|
|
and replace \[ga]_\[ga] by \[ga]-\[ga] in test name for convenience.
|
|
If not \[ga]all\[ga], the name(s) should map to a directory under
|
|
\[ga]./cmd/bisync/testdata\[ga].
|
|
Use \[ga]all\[ga] to run all tests (default: all)
|
|
-remote PATH1 \[ga]local\[ga] or name of cloud service with \[ga]:\[ga] (default: local)
|
|
-remote2 PATH2 \[ga]local\[ga] or name of cloud service with \[ga]:\[ga] (default: local)
|
|
-no-compare Disable comparing test results with the golden directory
|
|
(default: compare)
|
|
-no-cleanup Disable cleanup of Path1 and Path2 testdirs.
|
|
Useful for troubleshooting. (default: cleanup)
|
|
-golden Store results in the golden directory (default: false)
|
|
This flag can be used with multiple tests.
|
|
-debug Print debug messages
|
|
-stop-at NUM Stop test after given step number. (default: run to the end)
|
|
Implies \[ga]-no-compare\[ga] and \[ga]-no-cleanup\[ga], if the test really
|
|
ends prematurely. Only meaningful for a single test case.
|
|
-refresh-times Force refreshing the target modtime, useful for Dropbox
|
|
(default: false)
|
|
-verbose Run tests verbosely
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note: unlike rclone flags which must be prefixed by double dash
|
|
(\f[C]--\f[R]), the test command flags can be equally prefixed by a
|
|
single \f[C]-\f[R] or double dash.
|
|
.SS Running tests
|
|
.IP \[bu] 2
|
|
\f[C]go test . -case basic -remote local -remote2 local\f[R] runs the
|
|
\f[C]test_basic\f[R] test case using only the local filesystem, synching
|
|
one local directory with another local directory.
|
|
Test script output is to the console, while commands within scenario.txt
|
|
have their output sent to the \f[C].../workdir/test.log\f[R] file, which
|
|
is finally compared to the golden copy.
|
|
.IP \[bu] 2
|
|
The first argument after \f[C]go test\f[R] should be a relative name of
|
|
the directory containing bisync source code.
|
|
If you run tests right from there, the argument will be \f[C].\f[R]
|
|
(current directory) as in most examples below.
|
|
If you run bisync tests from the rclone source directory, the command
|
|
should be \f[C]go test ./cmd/bisync ...\f[R].
|
|
.IP \[bu] 2
|
|
The test engine will mangle rclone output to ensure comparability with
|
|
golden listings and logs.
|
|
.IP \[bu] 2
|
|
Test scenarios are located in \f[C]./cmd/bisync/testdata\f[R].
|
|
The test \f[C]-case\f[R] argument should match the full name of a
|
|
subdirectory under that directory.
|
|
Every test subdirectory name on disk must start with \f[C]test_\f[R],
|
|
this prefix can be omitted on command line for brevity.
|
|
Also, underscores in the name can be replaced by dashes for convenience.
|
|
.IP \[bu] 2
|
|
\f[C]go test . -remote local -remote2 local -case all\f[R] runs all
|
|
tests.
|
|
.IP \[bu] 2
|
|
Path1 and Path2 may either be the keyword \f[C]local\f[R] or may be
|
|
names of configured cloud services.
|
|
\f[C]go test . -remote gdrive: -remote2 dropbox: -case basic\f[R] will
|
|
run the test between these two services, without transferring any files
|
|
to the local filesystem.
|
|
.IP \[bu] 2
|
|
Test run stdout and stderr console output may be directed to a file,
|
|
e.g.
|
|
\f[C]go test . -remote gdrive: -remote2 local -case all > runlog.txt 2>&1\f[R]
|
|
.SS Test execution flow
|
|
.IP "1." 3
|
|
The base setup in the \f[C]initial\f[R] directory of the testcase is
|
|
applied on the Path1 and Path2 filesystems (via rclone copy the initial
|
|
directory to Path1, then rclone sync Path1 to Path2).
|
|
.IP "2." 3
|
|
The commands in the scenario.txt file are applied, with output directed
|
|
to the \f[C]test.log\f[R] file in the test working directory.
|
|
Typically, the first actual command in the \f[C]scenario.txt\f[R] file
|
|
is to do a \f[C]--resync\f[R], which establishes the baseline
|
|
\f[C]{...}.path1.lst\f[R] and \f[C]{...}.path2.lst\f[R] files in the
|
|
test working directory (\f[C].../workdir/\f[R] relative to the temporary
|
|
test directory).
|
|
Various commands and listing snapshots are done within the test.
|
|
.IP "3." 3
|
|
Finally, the contents of the test working directory are compared to the
|
|
contents of the testcase\[aq]s golden directory.
|
|
.SS Notes about testing
|
|
.IP \[bu] 2
|
|
Test cases are in individual directories beneath
|
|
\f[C]./cmd/bisync/testdata\f[R].
|
|
A command line reference to a test is understood to reference a
|
|
directory beneath \f[C]testdata\f[R].
|
|
For example,
|
|
\f[C]go test ./cmd/bisync -case dry-run -remote gdrive: -remote2 local\f[R]
|
|
refers to the test case in \f[C]./cmd/bisync/testdata/test_dry_run\f[R].
|
|
.IP \[bu] 2
|
|
The test working directory is located at \f[C].../workdir\f[R] relative
|
|
to a temporary test directory, usually under \f[C]/tmp\f[R] on Linux.
|
|
.IP \[bu] 2
|
|
The local test sync tree is created at a temporary directory named like
|
|
\f[C]bisync.XXX\f[R] under system temporary directory.
|
|
.IP \[bu] 2
|
|
The remote test sync tree is located at a temporary directory under
|
|
\f[C]<remote:>/bisync.XXX/\f[R].
|
|
.IP \[bu] 2
|
|
\f[C]path1\f[R] and/or \f[C]path2\f[R] subdirectories are created in a
|
|
temporary directory under the respective local or cloud test remote.
|
|
.IP \[bu] 2
|
|
By default, the Path1 and Path2 test dirs and workdir will be deleted
|
|
after each test run.
|
|
The \f[C]-no-cleanup\f[R] flag disables purging these directories when
|
|
validating and debugging a given test.
|
|
These directories will be flushed before running another test,
|
|
independent of the \f[C]-no-cleanup\f[R] usage.
|
|
.IP \[bu] 2
|
|
You will likely want to add \[ga]-
|
|
/testdir/\f[C]to your normal bisync\f[R]--filters-file\f[C]so that normal syncs do not attempt to sync the test temporary directories, which may have\f[R]RCLONE_TEST\f[C]miscompares in some testcases which would otherwise trip the\f[R]--check-access\f[C]system. The\f[R]--check-access\f[C]mechanism is hard-coded to ignore\f[R]RCLONE_TEST\f[C]files beneath\f[R]bisync/testdata\[ga],
|
|
so the test cases may reside on the synched tree even if there are check
|
|
file mismatches in the test tree.
|
|
.IP \[bu] 2
|
|
Some Dropbox tests can fail, notably printing the following message:
|
|
\f[C]src and dst identical but can\[aq]t set mod time without deleting and re-uploading\f[R]
|
|
This is expected and happens due a way Dropbox handles modificaion
|
|
times.
|
|
You should use the \f[C]-refresh-times\f[R] test flag to make up for
|
|
this.
|
|
.IP \[bu] 2
|
|
If Dropbox tests hit request limit for you and print error message
|
|
\f[C]too_many_requests/...: Too many requests or write operations.\f[R]
|
|
then follow the Dropbox App ID
|
|
instructions (https://rclone.org/dropbox/#get-your-own-dropbox-app-id).
|
|
.SS Updating golden results
|
|
.PP
|
|
Sometimes even a slight change in the bisync source can cause little
|
|
changes spread around many log files.
|
|
Updating them manually would be a nighmare.
|
|
.PP
|
|
The \f[C]-golden\f[R] flag will store the \f[C]test.log\f[R] and
|
|
\f[C]*.lst\f[R] listings from each test case into respective golden
|
|
directories.
|
|
Golden results will automatically contain generic strings instead of
|
|
local or cloud paths which means that they should match when run with a
|
|
different cloud service.
|
|
.PP
|
|
Your normal workflow might be as follows: 1.
|
|
Git-clone the rclone sources locally 2.
|
|
Modify bisync source and check that it builds 3.
|
|
Run the whole test suite \f[C]go test ./cmd/bisync -remote local\f[R] 4.
|
|
If some tests show log difference, recheck them individually, e.g.:
|
|
\f[C]go test ./cmd/bisync -remote local -case basic\f[R] 5.
|
|
If you are convinced with the difference, goldenize all tests at once:
|
|
\f[C]go test ./cmd/bisync -remote local -golden\f[R] 6.
|
|
Use word diff: \f[C]git diff --word-diff ./cmd/bisync/testdata/\f[R].
|
|
Please note that normal line-level diff is generally useless here.
|
|
7.
|
|
Check the difference \f[I]carefully\f[R]! 8.
|
|
Commit the change (\f[C]git commit\f[R]) \f[I]only\f[R] if you are sure.
|
|
If unsure, save your code changes then wipe the log diffs from git:
|
|
\f[C]git reset [--hard]\f[R].
|
|
.SS Structure of test scenarios
|
|
.IP \[bu] 2
|
|
\f[C]<testname>/initial/\f[R] contains a tree of files that will be set
|
|
as the initial condition on both Path1 and Path2 testdirs.
|
|
.IP \[bu] 2
|
|
\f[C]<testname>/modfiles/\f[R] contains files that will be used to
|
|
modify the Path1 and/or Path2 filesystems.
|
|
.IP \[bu] 2
|
|
\f[C]<testname>/golden/\f[R] contains the expected content of the test
|
|
working directory (\f[C]workdir\f[R]) at the completion of the testcase.
|
|
.IP \[bu] 2
|
|
\f[C]<testname>/scenario.txt\f[R] contains the body of the test, in the
|
|
form of various commands to modify files, run bisync, and snapshot
|
|
listings.
|
|
Output from these commands is captured to \f[C].../workdir/test.log\f[R]
|
|
for comparison to the golden files.
|
|
.SS Supported test commands
|
|
.IP \[bu] 2
|
|
\f[C]test <some message>\f[R] Print the line to the console and to the
|
|
\f[C]test.log\f[R]:
|
|
\f[C]test sync is working correctly with options x, y, z\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]copy-listings <prefix>\f[R] Save a copy of all \f[C].lst\f[R]
|
|
listings in the test working directory with the specified prefix:
|
|
\f[C]save-listings exclude-pass-run\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]move-listings <prefix>\f[R] Similar to \f[C]copy-listings\f[R] but
|
|
removes the source
|
|
.IP \[bu] 2
|
|
\f[C]purge-children <dir>\f[R] This will delete all child files and
|
|
purge all child subdirs under given directory but keep the parent
|
|
intact.
|
|
This behavior is important for tests with Google Drive because removing
|
|
and re-creating the parent would change its ID.
|
|
.IP \[bu] 2
|
|
\f[C]delete-file <file>\f[R] Delete a single file.
|
|
.IP \[bu] 2
|
|
\f[C]delete-glob <dir> <pattern>\f[R] Delete a group of files located
|
|
one level deep in the given directory with names maching a given glob
|
|
pattern.
|
|
.IP \[bu] 2
|
|
\f[C]touch-glob YYYY-MM-DD <dir> <pattern>\f[R] Change modification time
|
|
on a group of files.
|
|
.IP \[bu] 2
|
|
\f[C]touch-copy YYYY-MM-DD <source-file> <dest-dir>\f[R] Change file
|
|
modification time then copy it to destination.
|
|
.IP \[bu] 2
|
|
\f[C]copy-file <source-file> <dest-dir>\f[R] Copy a single file to given
|
|
directory.
|
|
.IP \[bu] 2
|
|
\f[C]copy-as <source-file> <dest-file>\f[R] Similar to above but
|
|
destination must include both directory and the new file name at
|
|
destination.
|
|
.IP \[bu] 2
|
|
\f[C]copy-dir <src> <dst>\f[R] and \f[C]sync-dir <src> <dst>\f[R]
|
|
Copy/sync a directory.
|
|
Equivalent of \f[C]rclone copy\f[R] and \f[C]rclone sync\f[R].
|
|
.IP \[bu] 2
|
|
\f[C]list-dirs <dir>\f[R] Equivalent to
|
|
\f[C]rclone lsf -R --dirs-only <dir>\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]bisync [options]\f[R] Runs bisync against \f[C]-remote\f[R] and
|
|
\f[C]-remote2\f[R].
|
|
.SS Supported substitution terms
|
|
.IP \[bu] 2
|
|
\f[C]{testdir/}\f[R] - the root dir of the testcase
|
|
.IP \[bu] 2
|
|
\f[C]{datadir/}\f[R] - the \f[C]modfiles\f[R] dir under the testcase
|
|
root
|
|
.IP \[bu] 2
|
|
\f[C]{workdir/}\f[R] - the temporary test working directory
|
|
.IP \[bu] 2
|
|
\f[C]{path1/}\f[R] - the root of the Path1 test directory tree
|
|
.IP \[bu] 2
|
|
\f[C]{path2/}\f[R] - the root of the Path2 test directory tree
|
|
.IP \[bu] 2
|
|
\f[C]{session}\f[R] - base name of the test listings
|
|
.IP \[bu] 2
|
|
\f[C]{/}\f[R] - OS-specific path separator
|
|
.IP \[bu] 2
|
|
\f[C]{spc}\f[R], \f[C]{tab}\f[R], \f[C]{eol}\f[R] - whitespace
|
|
.IP \[bu] 2
|
|
\f[C]{chr:HH}\f[R] - raw byte with given hexadecimal code
|
|
.PP
|
|
Substitution results of the terms named like \f[C]{dir/}\f[R] will end
|
|
with \f[C]/\f[R] (or backslash on Windows), so it is not necessary to
|
|
include slash in the usage, for example
|
|
\f[C]delete-file {path1/}file1.txt\f[R].
|
|
.SS Benchmarks
|
|
.PP
|
|
\f[I]This section is work in progress.\f[R]
|
|
.PP
|
|
Here are a few data points for scale, execution times, and memory usage.
|
|
.PP
|
|
The first set of data was taken between a local disk to Dropbox.
|
|
The speedtest.net (https://speedtest.net) download speed was \[ti]170
|
|
Mbps, and upload speed was \[ti]10 Mbps.
|
|
500 files (\[ti]9.5 MB each) had been already synched.
|
|
50 files were added in a new directory, each \[ti]9.5 MB, \[ti]475 MB
|
|
total.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
lw(23.8n) lw(35.0n) lw(11.2n).
|
|
T{
|
|
Change
|
|
T}@T{
|
|
Operations and times
|
|
T}@T{
|
|
Overall run time
|
|
T}
|
|
_
|
|
T{
|
|
500 files synched (nothing to move)
|
|
T}@T{
|
|
1x listings for Path1 & Path2
|
|
T}@T{
|
|
1.5 sec
|
|
T}
|
|
T{
|
|
500 files synched with --check-access
|
|
T}@T{
|
|
1x listings for Path1 & Path2
|
|
T}@T{
|
|
1.5 sec
|
|
T}
|
|
T{
|
|
50 new files on remote
|
|
T}@T{
|
|
Queued 50 copies down: 27 sec
|
|
T}@T{
|
|
29 sec
|
|
T}
|
|
T{
|
|
Moved local dir
|
|
T}@T{
|
|
Queued 50 copies up: 410 sec, 50 deletes up: 9 sec
|
|
T}@T{
|
|
421 sec
|
|
T}
|
|
T{
|
|
Moved remote dir
|
|
T}@T{
|
|
Queued 50 copies down: 31 sec, 50 deletes down: <1 sec
|
|
T}@T{
|
|
33 sec
|
|
T}
|
|
T{
|
|
Delete local dir
|
|
T}@T{
|
|
Queued 50 deletes up: 9 sec
|
|
T}@T{
|
|
13 sec
|
|
T}
|
|
.TE
|
|
.PP
|
|
This next data is from a user\[aq]s application.
|
|
They had \[ti]400GB of data over 1.96 million files being sync\[aq]ed
|
|
between a Windows local disk and some remote cloud.
|
|
The file full path length was on average 35 characters (which factors
|
|
into load time and RAM required).
|
|
.IP \[bu] 2
|
|
Loading the prior listing into memory (1.96 million files, listing file
|
|
size 140 MB) took \[ti]30 sec and occupied about 1 GB of RAM.
|
|
.IP \[bu] 2
|
|
Getting a fresh listing of the local file system (producing the 140 MB
|
|
output file) took about XXX sec.
|
|
.IP \[bu] 2
|
|
Getting a fresh listing of the remote file system (producing the 140 MB
|
|
output file) took about XXX sec.
|
|
The network download speed was measured at XXX Mb/s.
|
|
.IP \[bu] 2
|
|
Once the prior and current Path1 and Path2 listings were loaded (a total
|
|
of four to be loaded, two at a time), determining the deltas was pretty
|
|
quick (a few seconds for this test case), and the transfer time for any
|
|
files to be copied was dominated by the network bandwidth.
|
|
.SS References
|
|
.PP
|
|
rclone\[aq]s bisync implementation was derived from the
|
|
rclonesync-V2 (https://github.com/cjnaz/rclonesync-V2) project,
|
|
including documentation and test mechanisms, with
|
|
[\[at]cjnaz](https://github.com/cjnaz)\[aq]s full support and
|
|
encouragement.
|
|
.PP
|
|
\f[C]rclone bisync\f[R] is similar in nature to a range of other
|
|
projects:
|
|
.IP \[bu] 2
|
|
unison (https://github.com/bcpierce00/unison)
|
|
.IP \[bu] 2
|
|
syncthing (https://github.com/syncthing/syncthing)
|
|
.IP \[bu] 2
|
|
cjnaz/rclonesync (https://github.com/cjnaz/rclonesync-V2)
|
|
.IP \[bu] 2
|
|
ConorWilliams/rsinc (https://github.com/ConorWilliams/rsinc)
|
|
.IP \[bu] 2
|
|
jwink3101/syncrclone (https://github.com/Jwink3101/syncrclone)
|
|
.IP \[bu] 2
|
|
DavideRossi/upback (https://github.com/DavideRossi/upback)
|
|
.PP
|
|
Bisync adopts the differential synchronization technique, which is based
|
|
on keeping history of changes performed by both synchronizing sides.
|
|
See the \f[I]Dual Shadow Method\f[R] section in the Neil Fraser\[aq]s
|
|
article (https://neil.fraser.name/writing/sync/).
|
|
.PP
|
|
Also note a number of academic publications by Benjamin
|
|
Pierce (http://www.cis.upenn.edu/%7Ebcpierce/papers/index.shtml#File%20Synchronization)
|
|
about \f[I]Unison\f[R] and synchronization in general.
|
|
.SH 1Fichier
|
|
.PP
|
|
This is a backend for the 1fichier (https://1fichier.com) cloud storage
|
|
service.
|
|
Note that a Premium subscription is required to use the API.
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for 1Fichier involves getting the API key from the
|
|
website which you need to do in your browser.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / 1Fichier
|
|
\[rs] \[dq]fichier\[dq]
|
|
[snip]
|
|
Storage> fichier
|
|
** See help for fichier backend at: https://rclone.org/fichier/ **
|
|
|
|
Your API Key, get it from https://1fichier.com/console/params.pl
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
api_key> example_key
|
|
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No
|
|
y/n>
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
type = fichier
|
|
api_key = example_key
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your 1Fichier account
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your 1Fichier account
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to a 1Fichier directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
1Fichier does not support modification times.
|
|
It supports the Whirlpool hash algorithm.
|
|
.SS Duplicated files
|
|
.PP
|
|
1Fichier can have two files with exactly the same name and path (unlike
|
|
a normal file system).
|
|
.PP
|
|
Duplicated files cause problems with the syncing and you will see
|
|
messages in the log about duplicates.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
T{
|
|
<
|
|
T}@T{
|
|
0x3C
|
|
T}@T{
|
|
\[uFF1C]
|
|
T}
|
|
T{
|
|
>
|
|
T}@T{
|
|
0x3E
|
|
T}@T{
|
|
\[uFF1E]
|
|
T}
|
|
T{
|
|
\[dq]
|
|
T}@T{
|
|
0x22
|
|
T}@T{
|
|
\[uFF02]
|
|
T}
|
|
T{
|
|
$
|
|
T}@T{
|
|
0x24
|
|
T}@T{
|
|
\[uFF04]
|
|
T}
|
|
T{
|
|
\[ga]
|
|
T}@T{
|
|
0x60
|
|
T}@T{
|
|
\[uFF40]
|
|
T}
|
|
T{
|
|
\[aq]
|
|
T}@T{
|
|
0x27
|
|
T}@T{
|
|
\[uFF07]
|
|
T}
|
|
.TE
|
|
.PP
|
|
File names can also not start or end with the following characters.
|
|
These only get replaced if they are the first or last character in the
|
|
name:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
SP
|
|
T}@T{
|
|
0x20
|
|
T}@T{
|
|
\[u2420]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to fichier (1Fichier).
|
|
.SS --fichier-api-key
|
|
.PP
|
|
Your API Key, get it from https://1fichier.com/console/params.pl.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: api_key
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FICHIER_API_KEY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to fichier (1Fichier).
|
|
.SS --fichier-shared-folder
|
|
.PP
|
|
If you want to download a shared folder, add this parameter.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: shared_folder
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FICHIER_SHARED_FOLDER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --fichier-file-password
|
|
.PP
|
|
If you want to download a shared file that is password protected, add
|
|
this parameter.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: file_password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FICHIER_FILE_PASSWORD
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --fichier-folder-password
|
|
.PP
|
|
If you want to list the files in a shared folder that is password
|
|
protected, add this parameter.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: folder_password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FICHIER_FOLDER_PASSWORD
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --fichier-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FICHIER_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default:
|
|
Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the 1Fichier backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SH Alias
|
|
.PP
|
|
The \f[C]alias\f[R] remote provides a new name for another remote.
|
|
.PP
|
|
Paths may be as deep as required or a local path, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R] or
|
|
\f[C]/directory/subdirectory\f[R].
|
|
.PP
|
|
During the initial setup with \f[C]rclone config\f[R] you will specify
|
|
the target remote.
|
|
The target remote can either be a local path or another remote.
|
|
.PP
|
|
Subfolders can be used in target remote.
|
|
Assume an alias remote named \f[C]backup\f[R] with the target
|
|
\f[C]mydrive:private/backup\f[R].
|
|
Invoking \f[C]rclone mkdir backup:desktop\f[R] is exactly the same as
|
|
invoking \f[C]rclone mkdir mydrive:private/backup/desktop\f[R].
|
|
.PP
|
|
There will be no special handling of paths containing \f[C]..\f[R]
|
|
segments.
|
|
Invoking \f[C]rclone mkdir backup:../desktop\f[R] is exactly the same as
|
|
invoking \f[C]rclone mkdir mydrive:private/backup/../desktop\f[R].
|
|
The empty path is not allowed as a remote.
|
|
To alias the current directory use \f[C].\f[R] instead.
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of how to make an alias called \f[C]remote\f[R] for
|
|
local folder.
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Alias for an existing remote
|
|
\[rs] \[dq]alias\[dq]
|
|
[snip]
|
|
Storage> alias
|
|
Remote or path to alias.
|
|
Can be \[dq]myremote:path/to/dir\[dq], \[dq]myremote:bucket\[dq], \[dq]myremote:\[dq] or \[dq]/local/path\[dq].
|
|
remote> /mnt/storage/backup
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
remote = /mnt/storage/backup
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
Current remotes:
|
|
|
|
Name Type
|
|
==== ====
|
|
remote alias
|
|
|
|
e) Edit existing remote
|
|
n) New remote
|
|
d) Delete remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/r/c/s/q> q
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level in \f[C]/mnt/storage/backup\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in \f[C]/mnt/storage/backup\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Copy another local directory to the alias directory called source
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:source
|
|
\f[R]
|
|
.fi
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to alias (Alias for an existing
|
|
remote).
|
|
.SS --alias-remote
|
|
.PP
|
|
Remote or path to alias.
|
|
.PP
|
|
Can be \[dq]myremote:path/to/dir\[dq], \[dq]myremote:bucket\[dq],
|
|
\[dq]myremote:\[dq] or \[dq]/local/path\[dq].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: remote
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ALIAS_REMOTE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SH Amazon Drive
|
|
.PP
|
|
Amazon Drive, formerly known as Amazon Cloud Drive, is a cloud storage
|
|
service run by Amazon for consumers.
|
|
.SS Status
|
|
.PP
|
|
\f[B]Important:\f[R] rclone supports Amazon Drive only if you have your
|
|
own set of API keys.
|
|
Unfortunately the Amazon Drive developer
|
|
program (https://developer.amazon.com/amazon-drive) is now closed to new
|
|
entries so if you don\[aq]t already have your own set of keys you will
|
|
not be able to use rclone with Amazon Drive.
|
|
.PP
|
|
For the history on why rclone no longer has a set of Amazon Drive API
|
|
keys see the
|
|
forum (https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/2314).
|
|
.PP
|
|
If you happen to know anyone who works at Amazon then please ask them to
|
|
re-instate rclone into the Amazon Drive developer program - thanks!
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for Amazon Drive involves getting a token from Amazon
|
|
which you need to do in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
The configuration process for Amazon Drive may involve using an oauth
|
|
proxy (https://github.com/ncw/oauthproxy).
|
|
This is used to keep the Amazon credentials out of the source code.
|
|
The proxy runs in Google\[aq]s very secure App Engine environment and
|
|
doesn\[aq]t store any credentials which pass through it.
|
|
.PP
|
|
Since rclone doesn\[aq]t currently have its own Amazon Drive credentials
|
|
so you will either need to have your own \f[C]client_id\f[R] and
|
|
\f[C]client_secret\f[R] with Amazon Drive, or use a third-party oauth
|
|
proxy in which case you will need to enter \f[C]client_id\f[R],
|
|
\f[C]client_secret\f[R], \f[C]auth_url\f[R] and \f[C]token_url\f[R].
|
|
.PP
|
|
Note also if you are not using Amazon\[aq]s \f[C]auth_url\f[R] and
|
|
\f[C]token_url\f[R], (ie you filled in something for those) then if
|
|
setting up on a remote machine you can only use the copying the config
|
|
method of
|
|
configuration (https://rclone.org/remote_setup/#configuring-by-copying-the-config-file)
|
|
- \f[C]rclone authorize\f[R] will not work.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/r/c/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Amazon Drive
|
|
\[rs] \[dq]amazon cloud drive\[dq]
|
|
[snip]
|
|
Storage> amazon cloud drive
|
|
Amazon Application Client Id - required.
|
|
client_id> your client ID goes here
|
|
Amazon Application Client Secret - required.
|
|
client_secret> your client secret goes here
|
|
Auth server URL - leave blank to use Amazon\[aq]s.
|
|
auth_url> Optional auth URL
|
|
Token server url - leave blank to use Amazon\[aq]s.
|
|
token_url> Optional token URL
|
|
Remote config
|
|
Make sure your Redirect URL is set to \[dq]http://127.0.0.1:53682/\[dq] in your custom config.
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
--------------------
|
|
[remote]
|
|
client_id = your client ID goes here
|
|
client_secret = your client secret goes here
|
|
auth_url = Optional auth URL
|
|
token_url = Optional token URL
|
|
token = {\[dq]access_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxx\[dq],\[dq]expiry\[dq]:\[dq]2015-09-06T16:07:39.658438471+01:00\[dq]}
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Amazon.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your Amazon Drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your Amazon Drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Amazon Drive directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time and MD5SUMs
|
|
.PP
|
|
Amazon Drive doesn\[aq]t allow modification times to be changed via the
|
|
API so these won\[aq]t be accurate or used for syncing.
|
|
.PP
|
|
It does store MD5SUMs so for a more accurate sync, you can use the
|
|
\f[C]--checksum\f[R] flag.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
NUL
|
|
T}@T{
|
|
0x00
|
|
T}@T{
|
|
\[u2400]
|
|
T}
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Deleting files
|
|
.PP
|
|
Any files you delete with rclone will end up in the trash.
|
|
Amazon don\[aq]t provide an API to permanently delete files, nor to
|
|
empty the trash, so you will have to do that with one of Amazon\[aq]s
|
|
apps or via the Amazon Drive website.
|
|
As of November 17, 2016, files are automatically deleted by Amazon from
|
|
the trash after 30 days.
|
|
.SS Using with non \f[C].com\f[R] Amazon accounts
|
|
.PP
|
|
Let\[aq]s say you usually use \f[C]amazon.co.uk\f[R].
|
|
When you authenticate with rclone it will take you to an
|
|
\f[C]amazon.com\f[R] page to log in.
|
|
Your \f[C]amazon.co.uk\f[R] email and password should work here just
|
|
fine.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to amazon cloud drive (Amazon
|
|
Drive).
|
|
.SS --acd-client-id
|
|
.PP
|
|
OAuth Client Id.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ACD_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --acd-client-secret
|
|
.PP
|
|
OAuth Client Secret.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ACD_CLIENT_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to amazon cloud drive (Amazon
|
|
Drive).
|
|
.SS --acd-token
|
|
.PP
|
|
OAuth Access Token as a JSON blob.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ACD_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --acd-auth-url
|
|
.PP
|
|
Auth server URL.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ACD_AUTH_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --acd-token-url
|
|
.PP
|
|
Token server url.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ACD_TOKEN_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --acd-checkpoint
|
|
.PP
|
|
Checkpoint for internal polling (debug).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: checkpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ACD_CHECKPOINT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --acd-upload-wait-per-gb
|
|
.PP
|
|
Additional time per GiB to wait after a failed complete upload to see if
|
|
it appears.
|
|
.PP
|
|
Sometimes Amazon Drive gives an error when a file has been fully
|
|
uploaded but the file appears anyway after a little while.
|
|
This happens sometimes for files over 1 GiB in size and nearly every
|
|
time for files bigger than 10 GiB.
|
|
This parameter controls the time rclone waits for the file to appear.
|
|
.PP
|
|
The default value for this parameter is 3 minutes per GiB, so by default
|
|
it will wait 3 minutes for every GiB uploaded to see if the file
|
|
appears.
|
|
.PP
|
|
You can disable this feature by setting it to 0.
|
|
This may cause conflict errors as rclone retries the failed upload but
|
|
the file will most likely appear correctly eventually.
|
|
.PP
|
|
These values were determined empirically by observing lots of uploads of
|
|
big files for a range of file sizes.
|
|
.PP
|
|
Upload with the \[dq]-v\[dq] flag to see more info about what rclone is
|
|
doing in this situation.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_wait_per_gb
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ACD_UPLOAD_WAIT_PER_GB
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 3m0s
|
|
.SS --acd-templink-threshold
|
|
.PP
|
|
Files >= this size will be downloaded via their tempLink.
|
|
.PP
|
|
Files this size or more will be downloaded via their \[dq]tempLink\[dq].
|
|
This is to work around a problem with Amazon Drive which blocks
|
|
downloads of files bigger than about 10 GiB.
|
|
The default for this is 9 GiB which shouldn\[aq]t need to be changed.
|
|
.PP
|
|
To download files above this threshold, rclone requests a
|
|
\[dq]tempLink\[dq] which downloads the file through a temporary URL
|
|
directly from the underlying S3 storage.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: templink_threshold
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ACD_TEMPLINK_THRESHOLD
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 9Gi
|
|
.SS --acd-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ACD_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
Note that Amazon Drive is case insensitive so you can\[aq]t have a file
|
|
called \[dq]Hello.doc\[dq] and one called \[dq]hello.doc\[dq].
|
|
.PP
|
|
Amazon Drive has rate limiting so you may notice errors in the sync (429
|
|
errors).
|
|
rclone will automatically retry the sync up to 3 times by default (see
|
|
\f[C]--retries\f[R] flag) which should hopefully work around this
|
|
problem.
|
|
.PP
|
|
Amazon Drive has an internal limit of file sizes that can be uploaded to
|
|
the service.
|
|
This limit is not officially published, but all files larger than this
|
|
will fail.
|
|
.PP
|
|
At the time of writing (Jan 2016) is in the area of 50 GiB per file.
|
|
This means that larger files are likely to fail.
|
|
.PP
|
|
Unfortunately there is no way for rclone to see that this failure is
|
|
because of file size, so it will retry the operation, as any other
|
|
failure.
|
|
To avoid this problem, use \f[C]--max-size 50000M\f[R] option to limit
|
|
the maximum size of uploaded files.
|
|
Note that \f[C]--max-size\f[R] does not split files into segments, it
|
|
only ignores files over this size.
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the Amazon Drive backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SH Amazon S3 Storage Providers
|
|
.PP
|
|
The S3 backend can be used with a number of different providers:
|
|
.IP \[bu] 2
|
|
AWS S3
|
|
.IP \[bu] 2
|
|
Alibaba Cloud (Aliyun) Object Storage System (OSS)
|
|
.IP \[bu] 2
|
|
Ceph
|
|
.IP \[bu] 2
|
|
DigitalOcean Spaces
|
|
.IP \[bu] 2
|
|
Dreamhost
|
|
.IP \[bu] 2
|
|
IBM COS S3
|
|
.IP \[bu] 2
|
|
Minio
|
|
.IP \[bu] 2
|
|
RackCorp Object Storage
|
|
.IP \[bu] 2
|
|
Scaleway
|
|
.IP \[bu] 2
|
|
Seagate Lyve Cloud
|
|
.IP \[bu] 2
|
|
SeaweedFS
|
|
.IP \[bu] 2
|
|
StackPath
|
|
.IP \[bu] 2
|
|
Storj
|
|
.IP \[bu] 2
|
|
Tencent Cloud Object Storage (COS)
|
|
.IP \[bu] 2
|
|
Wasabi
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[R] (or \f[C]remote:\f[R] for
|
|
the \f[C]lsd\f[R] command.) You may put subdirectories in too, e.g.
|
|
\f[C]remote:bucket/path/to/dir\f[R].
|
|
.PP
|
|
Once you have made a remote (see the provider specific section above)
|
|
you can use it like this:
|
|
.PP
|
|
See all buckets
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Make a new bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of making an s3 configuration for the AWS S3
|
|
provider.
|
|
Most applies to the other providers as well, any differences are
|
|
described below.
|
|
.PP
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Amazon S3 Compliant Storage Providers including AWS, Ceph, Dreamhost, IBM COS, Minio, and Tencent COS
|
|
\[rs] \[dq]s3\[dq]
|
|
[snip]
|
|
Storage> s3
|
|
Choose your S3 provider.
|
|
Choose a number from below, or type in your own value
|
|
1 / Amazon Web Services (AWS) S3
|
|
\[rs] \[dq]AWS\[dq]
|
|
2 / Ceph Object Storage
|
|
\[rs] \[dq]Ceph\[dq]
|
|
3 / Digital Ocean Spaces
|
|
\[rs] \[dq]DigitalOcean\[dq]
|
|
4 / Dreamhost DreamObjects
|
|
\[rs] \[dq]Dreamhost\[dq]
|
|
5 / IBM COS S3
|
|
\[rs] \[dq]IBMCOS\[dq]
|
|
6 / Minio Object Storage
|
|
\[rs] \[dq]Minio\[dq]
|
|
7 / Wasabi Object Storage
|
|
\[rs] \[dq]Wasabi\[dq]
|
|
8 / Any other S3 compatible provider
|
|
\[rs] \[dq]Other\[dq]
|
|
provider> 1
|
|
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
|
|
Choose a number from below, or type in your own value
|
|
1 / Enter AWS credentials in the next step
|
|
\[rs] \[dq]false\[dq]
|
|
2 / Get AWS credentials from the environment (env vars or IAM)
|
|
\[rs] \[dq]true\[dq]
|
|
env_auth> 1
|
|
AWS Access Key ID - leave blank for anonymous access or runtime credentials.
|
|
access_key_id> XXX
|
|
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
|
|
secret_access_key> YYY
|
|
Region to connect to.
|
|
Choose a number from below, or type in your own value
|
|
/ The default endpoint - a good choice if you are unsure.
|
|
1 | US Region, Northern Virginia, or Pacific Northwest.
|
|
| Leave location constraint empty.
|
|
\[rs] \[dq]us-east-1\[dq]
|
|
/ US East (Ohio) Region
|
|
2 | Needs location constraint us-east-2.
|
|
\[rs] \[dq]us-east-2\[dq]
|
|
/ US West (Oregon) Region
|
|
3 | Needs location constraint us-west-2.
|
|
\[rs] \[dq]us-west-2\[dq]
|
|
/ US West (Northern California) Region
|
|
4 | Needs location constraint us-west-1.
|
|
\[rs] \[dq]us-west-1\[dq]
|
|
/ Canada (Central) Region
|
|
5 | Needs location constraint ca-central-1.
|
|
\[rs] \[dq]ca-central-1\[dq]
|
|
/ EU (Ireland) Region
|
|
6 | Needs location constraint EU or eu-west-1.
|
|
\[rs] \[dq]eu-west-1\[dq]
|
|
/ EU (London) Region
|
|
7 | Needs location constraint eu-west-2.
|
|
\[rs] \[dq]eu-west-2\[dq]
|
|
/ EU (Frankfurt) Region
|
|
8 | Needs location constraint eu-central-1.
|
|
\[rs] \[dq]eu-central-1\[dq]
|
|
/ Asia Pacific (Singapore) Region
|
|
9 | Needs location constraint ap-southeast-1.
|
|
\[rs] \[dq]ap-southeast-1\[dq]
|
|
/ Asia Pacific (Sydney) Region
|
|
10 | Needs location constraint ap-southeast-2.
|
|
\[rs] \[dq]ap-southeast-2\[dq]
|
|
/ Asia Pacific (Tokyo) Region
|
|
11 | Needs location constraint ap-northeast-1.
|
|
\[rs] \[dq]ap-northeast-1\[dq]
|
|
/ Asia Pacific (Seoul)
|
|
12 | Needs location constraint ap-northeast-2.
|
|
\[rs] \[dq]ap-northeast-2\[dq]
|
|
/ Asia Pacific (Mumbai)
|
|
13 | Needs location constraint ap-south-1.
|
|
\[rs] \[dq]ap-south-1\[dq]
|
|
/ Asia Pacific (Hong Kong) Region
|
|
14 | Needs location constraint ap-east-1.
|
|
\[rs] \[dq]ap-east-1\[dq]
|
|
/ South America (Sao Paulo) Region
|
|
15 | Needs location constraint sa-east-1.
|
|
\[rs] \[dq]sa-east-1\[dq]
|
|
region> 1
|
|
Endpoint for S3 API.
|
|
Leave blank if using AWS to use the default endpoint for the region.
|
|
endpoint>
|
|
Location constraint - must be set to match the Region. Used when creating buckets only.
|
|
Choose a number from below, or type in your own value
|
|
1 / Empty for US Region, Northern Virginia, or Pacific Northwest.
|
|
\[rs] \[dq]\[dq]
|
|
2 / US East (Ohio) Region.
|
|
\[rs] \[dq]us-east-2\[dq]
|
|
3 / US West (Oregon) Region.
|
|
\[rs] \[dq]us-west-2\[dq]
|
|
4 / US West (Northern California) Region.
|
|
\[rs] \[dq]us-west-1\[dq]
|
|
5 / Canada (Central) Region.
|
|
\[rs] \[dq]ca-central-1\[dq]
|
|
6 / EU (Ireland) Region.
|
|
\[rs] \[dq]eu-west-1\[dq]
|
|
7 / EU (London) Region.
|
|
\[rs] \[dq]eu-west-2\[dq]
|
|
8 / EU Region.
|
|
\[rs] \[dq]EU\[dq]
|
|
9 / Asia Pacific (Singapore) Region.
|
|
\[rs] \[dq]ap-southeast-1\[dq]
|
|
10 / Asia Pacific (Sydney) Region.
|
|
\[rs] \[dq]ap-southeast-2\[dq]
|
|
11 / Asia Pacific (Tokyo) Region.
|
|
\[rs] \[dq]ap-northeast-1\[dq]
|
|
12 / Asia Pacific (Seoul)
|
|
\[rs] \[dq]ap-northeast-2\[dq]
|
|
13 / Asia Pacific (Mumbai)
|
|
\[rs] \[dq]ap-south-1\[dq]
|
|
14 / Asia Pacific (Hong Kong)
|
|
\[rs] \[dq]ap-east-1\[dq]
|
|
15 / South America (Sao Paulo) Region.
|
|
\[rs] \[dq]sa-east-1\[dq]
|
|
location_constraint> 1
|
|
Canned ACL used when creating buckets and/or storing objects in S3.
|
|
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
|
|
Choose a number from below, or type in your own value
|
|
1 / Owner gets FULL_CONTROL. No one else has access rights (default).
|
|
\[rs] \[dq]private\[dq]
|
|
2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
|
|
\[rs] \[dq]public-read\[dq]
|
|
/ Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
|
|
3 | Granting this on a bucket is generally not recommended.
|
|
\[rs] \[dq]public-read-write\[dq]
|
|
4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access.
|
|
\[rs] \[dq]authenticated-read\[dq]
|
|
/ Object owner gets FULL_CONTROL. Bucket owner gets READ access.
|
|
5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
|
|
\[rs] \[dq]bucket-owner-read\[dq]
|
|
/ Both the object owner and the bucket owner get FULL_CONTROL over the object.
|
|
6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
|
|
\[rs] \[dq]bucket-owner-full-control\[dq]
|
|
acl> 1
|
|
The server-side encryption algorithm used when storing this object in S3.
|
|
Choose a number from below, or type in your own value
|
|
1 / None
|
|
\[rs] \[dq]\[dq]
|
|
2 / AES256
|
|
\[rs] \[dq]AES256\[dq]
|
|
server_side_encryption> 1
|
|
The storage class to use when storing objects in S3.
|
|
Choose a number from below, or type in your own value
|
|
1 / Default
|
|
\[rs] \[dq]\[dq]
|
|
2 / Standard storage class
|
|
\[rs] \[dq]STANDARD\[dq]
|
|
3 / Reduced redundancy storage class
|
|
\[rs] \[dq]REDUCED_REDUNDANCY\[dq]
|
|
4 / Standard Infrequent Access storage class
|
|
\[rs] \[dq]STANDARD_IA\[dq]
|
|
5 / One Zone Infrequent Access storage class
|
|
\[rs] \[dq]ONEZONE_IA\[dq]
|
|
6 / Glacier storage class
|
|
\[rs] \[dq]GLACIER\[dq]
|
|
7 / Glacier Deep Archive storage class
|
|
\[rs] \[dq]DEEP_ARCHIVE\[dq]
|
|
8 / Intelligent-Tiering storage class
|
|
\[rs] \[dq]INTELLIGENT_TIERING\[dq]
|
|
9 / Glacier Instant Retrieval storage class
|
|
\[rs] \[dq]GLACIER_IR\[dq]
|
|
storage_class> 1
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
type = s3
|
|
provider = AWS
|
|
env_auth = false
|
|
access_key_id = XXX
|
|
secret_access_key = YYY
|
|
region = us-east-1
|
|
endpoint =
|
|
location_constraint =
|
|
acl = private
|
|
server_side_encryption =
|
|
storage_class =
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d>
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X-Amz-Meta-Mtime\f[R] as floating point since the epoch, accurate
|
|
to 1 ns.
|
|
.PP
|
|
If the modification time needs to be updated rclone will attempt to
|
|
perform a server side copy to update the modification if the object can
|
|
be copied in a single part.
|
|
In the case the object is larger than 5Gb or is in Glacier or Glacier
|
|
Deep Archive storage the object will be uploaded rather than copied.
|
|
.PP
|
|
Note that reading this from the object takes an additional
|
|
\f[C]HEAD\f[R] request as the metadata isn\[aq]t returned in object
|
|
listings.
|
|
.SS Reducing costs
|
|
.SS Avoiding HEAD requests to read the modification time
|
|
.PP
|
|
By default, rclone will use the modification time of objects stored in
|
|
S3 for syncing.
|
|
This is stored in object metadata which unfortunately takes an extra
|
|
HEAD request to read which can be expensive (in time and money).
|
|
.PP
|
|
The modification time is used by default for all operations that require
|
|
checking the time a file was last updated.
|
|
It allows rclone to treat the remote more like a true filesystem, but it
|
|
is inefficient on S3 because it requires an extra API call to retrieve
|
|
the metadata.
|
|
.PP
|
|
The extra API calls can be avoided when syncing (using
|
|
\f[C]rclone sync\f[R] or \f[C]rclone copy\f[R]) in a few different ways,
|
|
each with its own tradeoffs.
|
|
.IP \[bu] 2
|
|
\f[C]--size-only\f[R]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Only checks the size of files.
|
|
.IP \[bu] 2
|
|
Uses no extra transactions.
|
|
.IP \[bu] 2
|
|
If the file doesn\[aq]t change size then rclone won\[aq]t detect it has
|
|
changed.
|
|
.IP \[bu] 2
|
|
\f[C]rclone sync --size-only /path/to/source s3:bucket\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]--checksum\f[R]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Checks the size and MD5 checksum of files.
|
|
.IP \[bu] 2
|
|
Uses no extra transactions.
|
|
.IP \[bu] 2
|
|
The most accurate detection of changes possible.
|
|
.IP \[bu] 2
|
|
Will cause the source to read an MD5 checksum which, if it is a local
|
|
disk, will cause lots of disk activity.
|
|
.IP \[bu] 2
|
|
If the source and destination are both S3 this is the
|
|
\f[B]recommended\f[R] flag to use for maximum efficiency.
|
|
.IP \[bu] 2
|
|
\f[C]rclone sync --checksum /path/to/source s3:bucket\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]--update --use-server-modtime\f[R]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Uses no extra transactions.
|
|
.IP \[bu] 2
|
|
Modification time becomes the time the object was uploaded.
|
|
.IP \[bu] 2
|
|
For many operations this is sufficient to determine if it needs
|
|
uploading.
|
|
.IP \[bu] 2
|
|
Using \f[C]--update\f[R] along with \f[C]--use-server-modtime\f[R],
|
|
avoids the extra API call and uploads files whose local modification
|
|
time is newer than the time it was last uploaded.
|
|
.IP \[bu] 2
|
|
Files created with timestamps in the past will be missed by the sync.
|
|
.IP \[bu] 2
|
|
\f[C]rclone sync --update --use-server-modtime /path/to/source s3:bucket\f[R]
|
|
.RE
|
|
.PP
|
|
These flags can and should be used in combination with
|
|
\f[C]--fast-list\f[R] - see below.
|
|
.PP
|
|
If using \f[C]rclone mount\f[R] or any command using the VFS (eg
|
|
\f[C]rclone serve\f[R]) commands then you might want to consider using
|
|
the VFS flag \f[C]--no-modtime\f[R] which will stop rclone reading the
|
|
modification time for every object.
|
|
You could also use \f[C]--use-server-modtime\f[R] if you are happy with
|
|
the modification times of the objects being the time of upload.
|
|
.SS Avoiding GET requests to read directory listings
|
|
.PP
|
|
Rclone\[aq]s default directory traversal is to process each directory
|
|
individually.
|
|
This takes one API call per directory.
|
|
Using the \f[C]--fast-list\f[R] flag will read all info about the the
|
|
objects into memory first using a smaller number of API calls (one per
|
|
1000 objects).
|
|
See the rclone docs (https://rclone.org/docs/#fast-list) for more
|
|
details.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync --fast-list --checksum /path/to/source s3:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[C]--fast-list\f[R] trades off API transactions for memory use.
|
|
As a rough guide rclone uses 1k of memory per object stored, so using
|
|
\f[C]--fast-list\f[R] on a sync of a million objects will use roughly 1
|
|
GiB of RAM.
|
|
.PP
|
|
If you are only copying a small number of files into a big repository
|
|
then using \f[C]--no-traverse\f[R] is a good idea.
|
|
This finds objects directly instead of through directory listings.
|
|
You can do a \[dq]top-up\[dq] sync very cheaply by using
|
|
\f[C]--max-age\f[R] and \f[C]--no-traverse\f[R] to copy only recent
|
|
files, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --max-age 24h --no-traverse /path/to/source s3:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You\[aq]d then do a full \f[C]rclone sync\f[R] less often.
|
|
.PP
|
|
Note that \f[C]--fast-list\f[R] isn\[aq]t required in the top-up sync.
|
|
.SS Avoiding HEAD requests after PUT
|
|
.PP
|
|
By default, rclone will HEAD every object it uploads.
|
|
It does this to check the object got uploaded correctly.
|
|
.PP
|
|
You can disable this with the --s3-no-head option - see there for more
|
|
details.
|
|
.PP
|
|
Setting this flag increases the chance for undetected upload failures.
|
|
.SS Hashes
|
|
.PP
|
|
For small objects which weren\[aq]t uploaded as multipart uploads
|
|
(objects sized below \f[C]--s3-upload-cutoff\f[R] if uploaded with
|
|
rclone) rclone uses the \f[C]ETag:\f[R] header as an MD5 checksum.
|
|
.PP
|
|
However for objects which were uploaded as multipart uploads or with
|
|
server side encryption (SSE-AWS or SSE-C) the \f[C]ETag\f[R] header is
|
|
no longer the MD5 sum of the data, so rclone adds an additional piece of
|
|
metadata \f[C]X-Amz-Meta-Md5chksum\f[R] which is a base64 encoded MD5
|
|
hash (in the same format as is required for \f[C]Content-MD5\f[R]).
|
|
.PP
|
|
For large objects, calculating this hash can take some time so the
|
|
addition of this hash can be disabled with
|
|
\f[C]--s3-disable-checksum\f[R].
|
|
This will mean that these objects do not have an MD5 checksum.
|
|
.PP
|
|
Note that reading this from the object takes an additional
|
|
\f[C]HEAD\f[R] request as the metadata isn\[aq]t returned in object
|
|
listings.
|
|
.SS Cleanup
|
|
.PP
|
|
If you run \f[C]rclone cleanup s3:bucket\f[R] then it will remove all
|
|
pending multipart uploads older than 24 hours.
|
|
You can use the \f[C]-i\f[R] flag to see exactly what it will do.
|
|
If you want more control over the expiry date then run
|
|
\f[C]rclone backend cleanup s3:bucket -o max-age=1h\f[R] to expire all
|
|
uploads older than one hour.
|
|
You can use \f[C]rclone backend list-multipart-uploads s3:bucket\f[R] to
|
|
see the pending multipart uploads.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
S3 allows any valid UTF-8 string as a key.
|
|
.PP
|
|
Invalid UTF-8 bytes will be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in XML.
|
|
.PP
|
|
The following characters are replaced since these are problematic when
|
|
dealing with the REST API:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
NUL
|
|
T}@T{
|
|
0x00
|
|
T}@T{
|
|
\[u2400]
|
|
T}
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
.TE
|
|
.PP
|
|
The encoding will also encode these file names as they don\[aq]t seem to
|
|
work with the SDK properly:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c.
|
|
T{
|
|
File name
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\&.
|
|
T}@T{
|
|
\[uFF0E]
|
|
T}
|
|
T{
|
|
\&..
|
|
T}@T{
|
|
\[uFF0E]\[uFF0E]
|
|
T}
|
|
.TE
|
|
.SS Multipart uploads
|
|
.PP
|
|
rclone supports multipart uploads with S3 which means that it can upload
|
|
files bigger than 5 GiB.
|
|
.PP
|
|
Note that files uploaded \f[I]both\f[R] with multipart upload
|
|
\f[I]and\f[R] through crypt remotes do not have MD5 sums.
|
|
.PP
|
|
rclone switches from single part uploads to multipart uploads at the
|
|
point specified by \f[C]--s3-upload-cutoff\f[R].
|
|
This can be a maximum of 5 GiB and a minimum of 0 (ie always upload
|
|
multipart files).
|
|
.PP
|
|
The chunk sizes used in the multipart upload are specified by
|
|
\f[C]--s3-chunk-size\f[R] and the number of chunks uploaded concurrently
|
|
is specified by \f[C]--s3-upload-concurrency\f[R].
|
|
.PP
|
|
Multipart uploads will use \f[C]--transfers\f[R] *
|
|
\f[C]--s3-upload-concurrency\f[R] * \f[C]--s3-chunk-size\f[R] extra
|
|
memory.
|
|
Single part uploads to not use extra memory.
|
|
.PP
|
|
Single part transfers can be faster than multipart transfers or slower
|
|
depending on your latency from S3 - the more latency, the more likely
|
|
single part transfers will be faster.
|
|
.PP
|
|
Increasing \f[C]--s3-upload-concurrency\f[R] will increase throughput (8
|
|
would be a sensible value) and increasing \f[C]--s3-chunk-size\f[R] also
|
|
increases throughput (16M would be sensible).
|
|
Increasing either of these will use more memory.
|
|
The default values are high enough to gain most of the possible
|
|
performance without using too much memory.
|
|
.SS Buckets and Regions
|
|
.PP
|
|
With Amazon S3 you can list buckets (\f[C]rclone lsd\f[R]) using any
|
|
region, but you can only access the content of a bucket from the region
|
|
it was created in.
|
|
If you attempt to access a bucket from the wrong region, you will get an
|
|
error,
|
|
\f[C]incorrect region, the bucket is not in \[aq]XXX\[aq] region\f[R].
|
|
.SS Authentication
|
|
.PP
|
|
There are a number of ways to supply \f[C]rclone\f[R] with a set of AWS
|
|
credentials, with and without using the environment.
|
|
.PP
|
|
The different authentication methods are tried in this order:
|
|
.IP \[bu] 2
|
|
Directly in the rclone configuration file (\f[C]env_auth = false\f[R] in
|
|
the config file):
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]access_key_id\f[R] and \f[C]secret_access_key\f[R] are required.
|
|
.IP \[bu] 2
|
|
\f[C]session_token\f[R] can be optionally set when using AWS STS.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Runtime configuration (\f[C]env_auth = true\f[R] in the config file):
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Export the following environment variables before running
|
|
\f[C]rclone\f[R]:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Access Key ID: \f[C]AWS_ACCESS_KEY_ID\f[R] or \f[C]AWS_ACCESS_KEY\f[R]
|
|
.IP \[bu] 2
|
|
Secret Access Key: \f[C]AWS_SECRET_ACCESS_KEY\f[R] or
|
|
\f[C]AWS_SECRET_KEY\f[R]
|
|
.IP \[bu] 2
|
|
Session Token: \f[C]AWS_SESSION_TOKEN\f[R] (optional)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Or, use a named
|
|
profile (https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html):
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Profile files are standard files used by AWS CLI tools
|
|
.IP \[bu] 2
|
|
By default it will use the profile in your home directory (e.g.
|
|
\f[C]\[ti]/.aws/credentials\f[R] on unix based systems) file and the
|
|
\[dq]default\[dq] profile, to change set these environment variables:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]AWS_SHARED_CREDENTIALS_FILE\f[R] to control which file.
|
|
.IP \[bu] 2
|
|
\f[C]AWS_PROFILE\f[R] to control which profile to use.
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Or, run \f[C]rclone\f[R] in an ECS task with an IAM role (AWS only).
|
|
.IP \[bu] 2
|
|
Or, run \f[C]rclone\f[R] on an EC2 instance with an IAM role (AWS only).
|
|
.IP \[bu] 2
|
|
Or, run \f[C]rclone\f[R] in an EKS pod with an IAM role that is
|
|
associated with a service account (AWS only).
|
|
.RE
|
|
.PP
|
|
If none of these option actually end up providing \f[C]rclone\f[R] with
|
|
AWS credentials then S3 interaction will be non-authenticated (see
|
|
below).
|
|
.SS S3 Permissions
|
|
.PP
|
|
When using the \f[C]sync\f[R] subcommand of \f[C]rclone\f[R] the
|
|
following minimum permissions are required to be available on the bucket
|
|
being written to:
|
|
.IP \[bu] 2
|
|
\f[C]ListBucket\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]DeleteObject\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]GetObject\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]PutObject\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]PutObjectACL\f[R]
|
|
.PP
|
|
When using the \f[C]lsd\f[R] subcommand, the \f[C]ListAllMyBuckets\f[R]
|
|
permission is required.
|
|
.PP
|
|
Example policy:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]Version\[dq]: \[dq]2012-10-17\[dq],
|
|
\[dq]Statement\[dq]: [
|
|
{
|
|
\[dq]Effect\[dq]: \[dq]Allow\[dq],
|
|
\[dq]Principal\[dq]: {
|
|
\[dq]AWS\[dq]: \[dq]arn:aws:iam::USER_SID:user/USER_NAME\[dq]
|
|
},
|
|
\[dq]Action\[dq]: [
|
|
\[dq]s3:ListBucket\[dq],
|
|
\[dq]s3:DeleteObject\[dq],
|
|
\[dq]s3:GetObject\[dq],
|
|
\[dq]s3:PutObject\[dq],
|
|
\[dq]s3:PutObjectAcl\[dq]
|
|
],
|
|
\[dq]Resource\[dq]: [
|
|
\[dq]arn:aws:s3:::BUCKET_NAME/*\[dq],
|
|
\[dq]arn:aws:s3:::BUCKET_NAME\[dq]
|
|
]
|
|
},
|
|
{
|
|
\[dq]Effect\[dq]: \[dq]Allow\[dq],
|
|
\[dq]Action\[dq]: \[dq]s3:ListAllMyBuckets\[dq],
|
|
\[dq]Resource\[dq]: \[dq]arn:aws:s3:::*\[dq]
|
|
}
|
|
]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Notes on above:
|
|
.IP "1." 3
|
|
This is a policy that can be used when creating bucket.
|
|
It assumes that \f[C]USER_NAME\f[R] has been created.
|
|
.IP "2." 3
|
|
The Resource entry must include both resource ARNs, as one implies the
|
|
bucket and the other implies the bucket\[aq]s objects.
|
|
.PP
|
|
For reference, here\[aq]s an Ansible
|
|
script (https://gist.github.com/ebridges/ebfc9042dd7c756cd101cfa807b7ae2b)
|
|
that will generate one or more buckets that will work with
|
|
\f[C]rclone sync\f[R].
|
|
.SS Key Management System (KMS)
|
|
.PP
|
|
If you are using server-side encryption with KMS then you must make sure
|
|
rclone is configured with \f[C]server_side_encryption = aws:kms\f[R]
|
|
otherwise you will find you can\[aq]t transfer small objects - these
|
|
will create checksum errors.
|
|
.SS Glacier and Glacier Deep Archive
|
|
.PP
|
|
You can upload objects using the glacier storage class or transition
|
|
them to glacier using a lifecycle
|
|
policy (http://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html).
|
|
The bucket can still be synced or copied into normally, but if rclone
|
|
tries to access data from the glacier storage class you will see an
|
|
error like below.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
2017/09/11 19:07:43 Failed to sync: failed to open source object: Object in GLACIER, restore first: path/to/file
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
In this case you need to
|
|
restore (http://docs.aws.amazon.com/AmazonS3/latest/user-guide/restore-archived-objects.html)
|
|
the object(s) in question before using rclone.
|
|
.PP
|
|
Note that rclone only speaks the S3 API it does not speak the Glacier
|
|
Vault API, so rclone cannot directly access Glacier Vaults.
|
|
.SS Object-lock enabled S3 bucket
|
|
.PP
|
|
According to AWS\[aq]s documentation on S3 Object
|
|
Lock (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html#object-lock-permission):
|
|
.RS
|
|
.PP
|
|
If you configure a default retention period on a bucket, requests to
|
|
upload objects in such a bucket must include the Content-MD5 header.
|
|
.RE
|
|
.PP
|
|
As mentioned in the Hashes section, small files that are not uploaded as
|
|
multipart, use a different tag, causing the upload to fail.
|
|
A simple solution is to set the \f[C]--s3-upload-cutoff 0\f[R] and force
|
|
all the files to be uploaded as multipart.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to s3 (Amazon S3 Compliant
|
|
Storage Providers including AWS, Alibaba, Ceph, Digital Ocean,
|
|
Dreamhost, IBM COS, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent
|
|
COS).
|
|
.SS --s3-provider
|
|
.PP
|
|
Choose your S3 provider.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: provider
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_PROVIDER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]AWS\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Amazon Web Services (AWS) S3
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]Alibaba\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Alibaba Cloud Object Storage System (OSS) formerly Aliyun
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]Ceph\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ceph Object Storage
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]DigitalOcean\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Digital Ocean Spaces
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]Dreamhost\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Dreamhost DreamObjects
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]IBMCOS\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
IBM COS S3
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]LyveCloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Seagate Lyve Cloud
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]Minio\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Minio Object Storage
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]Netease\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Netease Object Storage (NOS)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]RackCorp\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
RackCorp Object Storage
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]Scaleway\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Scaleway Object Storage
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]SeaweedFS\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
SeaweedFS S3
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]StackPath\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
StackPath Object Storage
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]Storj\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Storj (S3 Compatible Gateway)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]TencentCOS\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Tencent Cloud Object Storage (COS)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]Wasabi\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Wasabi Object Storage
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]Other\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Any other S3 compatible provider
|
|
.RE
|
|
.RE
|
|
.SS --s3-env-auth
|
|
.PP
|
|
Get AWS credentials from runtime (environment variables or EC2/ECS meta
|
|
data if no env vars).
|
|
.PP
|
|
Only applies if access_key_id and secret_access_key is blank.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: env_auth
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ENV_AUTH
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]false\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enter AWS credentials in the next step.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]true\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Get AWS credentials from the environment (env vars or IAM).
|
|
.RE
|
|
.RE
|
|
.SS --s3-access-key-id
|
|
.PP
|
|
AWS Access Key ID.
|
|
.PP
|
|
Leave blank for anonymous access or runtime credentials.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: access_key_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ACCESS_KEY_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --s3-secret-access-key
|
|
.PP
|
|
AWS Secret Access Key (password).
|
|
.PP
|
|
Leave blank for anonymous access or runtime credentials.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: secret_access_key
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_SECRET_ACCESS_KEY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --s3-region
|
|
.PP
|
|
Region to connect to.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: region
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_REGION
|
|
.IP \[bu] 2
|
|
Provider: AWS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]us-east-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The default endpoint - a good choice if you are unsure.
|
|
.IP \[bu] 2
|
|
US Region, Northern Virginia, or Pacific Northwest.
|
|
.IP \[bu] 2
|
|
Leave location constraint empty.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-east-2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US East (Ohio) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint us-east-2.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-west-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US West (Northern California) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint us-west-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-west-2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US West (Oregon) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint us-west-2.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ca-central-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Canada (Central) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint ca-central-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-west-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU (Ireland) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint EU or eu-west-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-west-2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU (London) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint eu-west-2.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-west-3\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU (Paris) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint eu-west-3.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-north-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU (Stockholm) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint eu-north-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-south-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU (Milan) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint eu-south-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-central-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU (Frankfurt) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint eu-central-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-southeast-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Singapore) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint ap-southeast-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-southeast-2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Sydney) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint ap-southeast-2.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-northeast-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Tokyo) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint ap-northeast-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-northeast-2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Seoul).
|
|
.IP \[bu] 2
|
|
Needs location constraint ap-northeast-2.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-northeast-3\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Osaka-Local).
|
|
.IP \[bu] 2
|
|
Needs location constraint ap-northeast-3.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-south-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Mumbai).
|
|
.IP \[bu] 2
|
|
Needs location constraint ap-south-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-east-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Hong Kong) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint ap-east-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sa-east-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
South America (Sao Paulo) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint sa-east-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]me-south-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Middle East (Bahrain) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint me-south-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]af-south-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Africa (Cape Town) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint af-south-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cn-north-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
China (Beijing) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint cn-north-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cn-northwest-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
China (Ningxia) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint cn-northwest-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-gov-east-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
AWS GovCloud (US-East) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint us-gov-east-1.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-gov-west-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
AWS GovCloud (US) Region.
|
|
.IP \[bu] 2
|
|
Needs location constraint us-gov-west-1.
|
|
.RE
|
|
.RE
|
|
.SS --s3-region
|
|
.PP
|
|
region - the location where your bucket will be created and your data
|
|
stored.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: region
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_REGION
|
|
.IP \[bu] 2
|
|
Provider: RackCorp
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]global\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Global CDN (All locations) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Australia (All states)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-nsw\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
NSW (Australia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-qld\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
QLD (Australia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-vic\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
VIC (Australia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-wa\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Perth (Australia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ph\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Manila (Philippines) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]th\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Bangkok (Thailand) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]hk\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
HK (Hong Kong) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]mn\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ulaanbaatar (Mongolia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]kg\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Bishkek (Kyrgyzstan) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]id\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Jakarta (Indonesia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]jp\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Tokyo (Japan) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sg\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
SG (Singapore) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]de\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Frankfurt (Germany) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
USA (AnyCast) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-east-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New York (USA) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-west-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Freemont (USA) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]nz\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Auckland (New Zealand) Region
|
|
.RE
|
|
.RE
|
|
.SS --s3-region
|
|
.PP
|
|
Region to connect to.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: region
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_REGION
|
|
.IP \[bu] 2
|
|
Provider: Scaleway
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]nl-ams\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Amsterdam, The Netherlands
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]fr-par\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Paris, France
|
|
.RE
|
|
.RE
|
|
.SS --s3-region
|
|
.PP
|
|
Region to connect to.
|
|
.PP
|
|
Leave blank if you are using an S3 clone and you don\[aq]t have a
|
|
region.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: region
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_REGION
|
|
.IP \[bu] 2
|
|
Provider: !AWS,Alibaba,RackCorp,Scaleway,Storj,TencentCOS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use this if unsure.
|
|
.IP \[bu] 2
|
|
Will use v4 signatures and an empty region.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]other-v2-signature\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use this only if v4 signatures don\[aq]t work.
|
|
.IP \[bu] 2
|
|
E.g.
|
|
pre Jewel/v10 CEPH.
|
|
.RE
|
|
.RE
|
|
.SS --s3-endpoint
|
|
.PP
|
|
Endpoint for S3 API.
|
|
.PP
|
|
Leave blank if using AWS to use the default endpoint for the region.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ENDPOINT
|
|
.IP \[bu] 2
|
|
Provider: AWS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --s3-endpoint
|
|
.PP
|
|
Endpoint for IBM COS S3 API.
|
|
.PP
|
|
Specify if using an IBM COS On Premise.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ENDPOINT
|
|
.IP \[bu] 2
|
|
Provider: IBMCOS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]s3.us.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.dal.us.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region Dallas Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.wdc.us.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region Washington DC Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.sjc.us.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region San Jose Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.us.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.dal.us.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region Dallas Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.wdc.us.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region Washington DC Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.sjc.us.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region San Jose Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.us-east.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Region East Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.us-east.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Region East Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.us-south.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Region South Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.us-south.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Region South Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.eu.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.fra.eu.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Frankfurt Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.mil.eu.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Milan Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.ams.eu.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Amsterdam Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.eu.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.fra.eu.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Frankfurt Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.mil.eu.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Milan Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.ams.eu.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Amsterdam Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.eu-gb.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Great Britain Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.eu-gb.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Great Britain Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.eu-de.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Region DE Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.eu-de.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Region DE Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.ap.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Cross Regional Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.tok.ap.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Cross Regional Tokyo Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.hkg.ap.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Cross Regional HongKong Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.seo.ap.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Cross Regional Seoul Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.ap.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Cross Regional Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.tok.ap.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Cross Regional Tokyo Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.hkg.ap.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Cross Regional HongKong Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.seo.ap.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Cross Regional Seoul Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.jp-tok.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Region Japan Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.jp-tok.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Region Japan Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.au-syd.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Region Australia Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.au-syd.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Region Australia Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.ams03.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Amsterdam Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.ams03.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Amsterdam Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.che01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Chennai Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.che01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Chennai Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.mel01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Melbourne Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.mel01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Melbourne Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.osl01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Oslo Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.osl01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Oslo Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.tor01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Toronto Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.tor01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Toronto Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.seo01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Seoul Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.seo01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Seoul Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.mon01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Montreal Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.mon01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Montreal Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.mex01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Mexico Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.mex01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Mexico Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.sjc04.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
San Jose Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.sjc04.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
San Jose Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.mil01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Milan Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.mil01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Milan Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.hkg02.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Hong Kong Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.hkg02.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Hong Kong Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.par01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Paris Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.par01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Paris Single Site Private Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.sng01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Singapore Single Site Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.private.sng01.cloud-object-storage.appdomain.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Singapore Single Site Private Endpoint
|
|
.RE
|
|
.RE
|
|
.SS --s3-endpoint
|
|
.PP
|
|
Endpoint for OSS API.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ENDPOINT
|
|
.IP \[bu] 2
|
|
Provider: Alibaba
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]oss-accelerate.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Global Accelerate
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-accelerate-overseas.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Global Accelerate (outside mainland China)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-hangzhou.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
East China 1 (Hangzhou)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-shanghai.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
East China 2 (Shanghai)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-qingdao.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
North China 1 (Qingdao)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-beijing.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
North China 2 (Beijing)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-zhangjiakou.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
North China 3 (Zhangjiakou)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-huhehaote.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
North China 5 (Hohhot)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-wulanchabu.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
North China 6 (Ulanqab)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-shenzhen.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
South China 1 (Shenzhen)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-heyuan.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
South China 2 (Heyuan)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-guangzhou.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
South China 3 (Guangzhou)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-chengdu.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
West China 1 (Chengdu)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-cn-hongkong.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Hong Kong (Hong Kong)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-us-west-1.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US West 1 (Silicon Valley)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-us-east-1.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US East 1 (Virginia)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-ap-southeast-1.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Southeast Asia Southeast 1 (Singapore)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-ap-southeast-2.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific Southeast 2 (Sydney)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-ap-southeast-3.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Southeast Asia Southeast 3 (Kuala Lumpur)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-ap-southeast-5.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific Southeast 5 (Jakarta)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-ap-northeast-1.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific Northeast 1 (Japan)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-ap-south-1.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific South 1 (Mumbai)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-eu-central-1.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Central Europe 1 (Frankfurt)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-eu-west-1.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
West Europe (London)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]oss-me-east-1.aliyuncs.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Middle East 1 (Dubai)
|
|
.RE
|
|
.RE
|
|
.SS --s3-endpoint
|
|
.PP
|
|
Endpoint for Scaleway Object Storage.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ENDPOINT
|
|
.IP \[bu] 2
|
|
Provider: Scaleway
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]s3.nl-ams.scw.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Amsterdam Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.fr-par.scw.cloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Paris Endpoint
|
|
.RE
|
|
.RE
|
|
.SS --s3-endpoint
|
|
.PP
|
|
Endpoint for StackPath Object Storage.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ENDPOINT
|
|
.IP \[bu] 2
|
|
Provider: StackPath
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]s3.us-east-2.stackpathstorage.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US East Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.us-west-1.stackpathstorage.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US West Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.eu-central-1.stackpathstorage.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Endpoint
|
|
.RE
|
|
.RE
|
|
.SS --s3-endpoint
|
|
.PP
|
|
Endpoint of the Shared Gateway.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ENDPOINT
|
|
.IP \[bu] 2
|
|
Provider: Storj
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]gateway.eu1.storjshare.io\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU1 Shared Gateway
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]gateway.us1.storjshare.io\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US1 Shared Gateway
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]gateway.ap1.storjshare.io\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia-Pacific Shared Gateway
|
|
.RE
|
|
.RE
|
|
.SS --s3-endpoint
|
|
.PP
|
|
Endpoint for Tencent COS API.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ENDPOINT
|
|
.IP \[bu] 2
|
|
Provider: TencentCOS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-beijing.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Beijing Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-nanjing.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Nanjing Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-shanghai.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Shanghai Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-guangzhou.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Guangzhou Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-nanjing.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Nanjing Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-chengdu.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Chengdu Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-chongqing.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Chongqing Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-hongkong.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Hong Kong (China) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-singapore.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Singapore Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-mumbai.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Mumbai Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-seoul.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Seoul Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-bangkok.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Bangkok Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.ap-tokyo.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Tokyo Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.na-siliconvalley.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Silicon Valley Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.na-ashburn.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Virginia Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.na-toronto.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Toronto Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.eu-frankfurt.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Frankfurt Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.eu-moscow.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Moscow Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cos.accelerate.myqcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use Tencent COS Accelerate Endpoint
|
|
.RE
|
|
.RE
|
|
.SS --s3-endpoint
|
|
.PP
|
|
Endpoint for RackCorp Object Storage.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ENDPOINT
|
|
.IP \[bu] 2
|
|
Provider: RackCorp
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Global (AnyCast) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Australia (Anycast) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-nsw.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Sydney (Australia) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-qld.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Brisbane (Australia) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-vic.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Melbourne (Australia) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-wa.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Perth (Australia) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ph.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Manila (Philippines) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]th.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Bangkok (Thailand) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]hk.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
HK (Hong Kong) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]mn.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ulaanbaatar (Mongolia) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]kg.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Bishkek (Kyrgyzstan) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]id.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Jakarta (Indonesia) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]jp.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Tokyo (Japan) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sg.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
SG (Singapore) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]de.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Frankfurt (Germany) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
USA (AnyCast) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-east-1.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New York (USA) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-west-1.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Freemont (USA) Endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]nz.s3.rackcorp.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Auckland (New Zealand) Endpoint
|
|
.RE
|
|
.RE
|
|
.SS --s3-endpoint
|
|
.PP
|
|
Endpoint for S3 API.
|
|
.PP
|
|
Required when using an S3 clone.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ENDPOINT
|
|
.IP \[bu] 2
|
|
Provider:
|
|
!AWS,IBMCOS,TencentCOS,Alibaba,Scaleway,StackPath,Storj,RackCorp
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]objects-us-east-1.dream.io\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Dream Objects endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]nyc3.digitaloceanspaces.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Digital Ocean Spaces New York 3
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ams3.digitaloceanspaces.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Digital Ocean Spaces Amsterdam 3
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sgp1.digitaloceanspaces.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Digital Ocean Spaces Singapore 1
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]localhost:8333\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
SeaweedFS S3 localhost
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.us-east-1.lyvecloud.seagate.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Seagate Lyve Cloud US East 1 (Virginia)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.us-west-1.lyvecloud.seagate.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Seagate Lyve Cloud US West 1 (California)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.ap-southeast-1.lyvecloud.seagate.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Seagate Lyve Cloud AP Southeast 1 (Singapore)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.wasabisys.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Wasabi US East endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.us-west-1.wasabisys.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Wasabi US West endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.eu-central-1.wasabisys.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Wasabi EU Central endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.ap-northeast-1.wasabisys.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Wasabi AP Northeast 1 (Tokyo) endpoint
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]s3.ap-northeast-2.wasabisys.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Wasabi AP Northeast 2 (Osaka) endpoint
|
|
.RE
|
|
.RE
|
|
.SS --s3-location-constraint
|
|
.PP
|
|
Location constraint - must be set to match the Region.
|
|
.PP
|
|
Used when creating buckets only.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: location_constraint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_LOCATION_CONSTRAINT
|
|
.IP \[bu] 2
|
|
Provider: AWS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Empty for US Region, Northern Virginia, or Pacific Northwest
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-east-2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US East (Ohio) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-west-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US West (Northern California) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-west-2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US West (Oregon) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ca-central-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Canada (Central) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-west-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU (Ireland) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-west-2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU (London) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-west-3\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU (Paris) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-north-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU (Stockholm) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-south-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU (Milan) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]EU\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-southeast-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Singapore) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-southeast-2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Sydney) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-northeast-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Tokyo) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-northeast-2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Seoul) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-northeast-3\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Osaka-Local) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-south-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Mumbai) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-east-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia Pacific (Hong Kong) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sa-east-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
South America (Sao Paulo) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]me-south-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Middle East (Bahrain) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]af-south-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Africa (Cape Town) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cn-north-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
China (Beijing) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cn-northwest-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
China (Ningxia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-gov-east-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
AWS GovCloud (US-East) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-gov-west-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
AWS GovCloud (US) Region
|
|
.RE
|
|
.RE
|
|
.SS --s3-location-constraint
|
|
.PP
|
|
Location constraint - must match endpoint when using IBM Cloud Public.
|
|
.PP
|
|
For on-prem COS, do not make a selection from this list, hit enter.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: location_constraint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_LOCATION_CONSTRAINT
|
|
.IP \[bu] 2
|
|
Provider: IBMCOS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]us-standard\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region Standard
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-vault\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region Vault
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-cold\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region Cold
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-flex\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Cross Region Flex
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-east-standard\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US East Region Standard
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-east-vault\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US East Region Vault
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-east-cold\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US East Region Cold
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-east-flex\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US East Region Flex
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-south-standard\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US South Region Standard
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-south-vault\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US South Region Vault
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-south-cold\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US South Region Cold
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-south-flex\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US South Region Flex
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-standard\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Standard
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-vault\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Vault
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-cold\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Cold
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-flex\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU Cross Region Flex
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-gb-standard\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Great Britain Standard
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-gb-vault\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Great Britain Vault
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-gb-cold\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Great Britain Cold
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu-gb-flex\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Great Britain Flex
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-standard\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Standard
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-vault\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Vault
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-cold\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Cold
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ap-flex\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
APAC Flex
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]mel01-standard\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Melbourne Standard
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]mel01-vault\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Melbourne Vault
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]mel01-cold\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Melbourne Cold
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]mel01-flex\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Melbourne Flex
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]tor01-standard\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Toronto Standard
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]tor01-vault\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Toronto Vault
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]tor01-cold\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Toronto Cold
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]tor01-flex\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Toronto Flex
|
|
.RE
|
|
.RE
|
|
.SS --s3-location-constraint
|
|
.PP
|
|
Location constraint - the location where your bucket will be located and
|
|
your data stored.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: location_constraint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_LOCATION_CONSTRAINT
|
|
.IP \[bu] 2
|
|
Provider: RackCorp
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]global\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Global CDN Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Australia (All locations)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-nsw\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
NSW (Australia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-qld\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
QLD (Australia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-vic\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
VIC (Australia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]au-wa\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Perth (Australia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ph\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Manila (Philippines) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]th\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Bangkok (Thailand) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]hk\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
HK (Hong Kong) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]mn\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ulaanbaatar (Mongolia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]kg\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Bishkek (Kyrgyzstan) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]id\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Jakarta (Indonesia) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]jp\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Tokyo (Japan) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sg\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
SG (Singapore) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]de\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Frankfurt (Germany) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
USA (AnyCast) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-east-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New York (USA) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-west-1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Freemont (USA) Region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]nz\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Auckland (New Zealand) Region
|
|
.RE
|
|
.RE
|
|
.SS --s3-location-constraint
|
|
.PP
|
|
Location constraint - must be set to match the Region.
|
|
.PP
|
|
Leave blank if not sure.
|
|
Used when creating buckets only.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: location_constraint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_LOCATION_CONSTRAINT
|
|
.IP \[bu] 2
|
|
Provider:
|
|
!AWS,IBMCOS,Alibaba,RackCorp,Scaleway,StackPath,Storj,TencentCOS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --s3-acl
|
|
.PP
|
|
Canned ACL used when creating buckets and storing or copying objects.
|
|
.PP
|
|
This ACL is used for creating objects and if bucket_acl isn\[aq]t set,
|
|
for creating buckets too.
|
|
.PP
|
|
For more info visit
|
|
https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
|
|
.PP
|
|
Note that this ACL is applied when server-side copying objects as S3
|
|
doesn\[aq]t copy the ACL from the source but rather writes a fresh one.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: acl
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ACL
|
|
.IP \[bu] 2
|
|
Provider: !Storj
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]default\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets Full_CONTROL.
|
|
.IP \[bu] 2
|
|
No one else has access rights (default).
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]private\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
No one else has access rights (default).
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]public-read\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
The AllUsers group gets READ access.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]public-read-write\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
The AllUsers group gets READ and WRITE access.
|
|
.IP \[bu] 2
|
|
Granting this on a bucket is generally not recommended.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]authenticated-read\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
The AuthenticatedUsers group gets READ access.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]bucket-owner-read\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Object owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
Bucket owner gets READ access.
|
|
.IP \[bu] 2
|
|
If you specify this canned ACL when creating a bucket, Amazon S3 ignores
|
|
it.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]bucket-owner-full-control\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Both the object owner and the bucket owner get FULL_CONTROL over the
|
|
object.
|
|
.IP \[bu] 2
|
|
If you specify this canned ACL when creating a bucket, Amazon S3 ignores
|
|
it.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]private\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
No one else has access rights (default).
|
|
.IP \[bu] 2
|
|
This acl is available on IBM Cloud (Infra), IBM Cloud (Storage),
|
|
On-Premise COS.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]public-read\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
The AllUsers group gets READ access.
|
|
.IP \[bu] 2
|
|
This acl is available on IBM Cloud (Infra), IBM Cloud (Storage),
|
|
On-Premise IBM COS.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]public-read-write\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
The AllUsers group gets READ and WRITE access.
|
|
.IP \[bu] 2
|
|
This acl is available on IBM Cloud (Infra), On-Premise IBM COS.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]authenticated-read\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
The AuthenticatedUsers group gets READ access.
|
|
.IP \[bu] 2
|
|
Not supported on Buckets.
|
|
.IP \[bu] 2
|
|
This acl is available on IBM Cloud (Infra) and On-Premise IBM COS.
|
|
.RE
|
|
.RE
|
|
.SS --s3-server-side-encryption
|
|
.PP
|
|
The server-side encryption algorithm used when storing this object in
|
|
S3.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: server_side_encryption
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_SERVER_SIDE_ENCRYPTION
|
|
.IP \[bu] 2
|
|
Provider: AWS,Ceph,Minio
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
None
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]AES256\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
AES256
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]aws:kms\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
aws:kms
|
|
.RE
|
|
.RE
|
|
.SS --s3-sse-kms-key-id
|
|
.PP
|
|
If using KMS ID you must provide the ARN of Key.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: sse_kms_key_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_SSE_KMS_KEY_ID
|
|
.IP \[bu] 2
|
|
Provider: AWS,Ceph,Minio
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
None
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]arn:aws:kms:us-east-1:*\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
arn:aws:kms:*
|
|
.RE
|
|
.RE
|
|
.SS --s3-storage-class
|
|
.PP
|
|
The storage class to use when storing new objects in S3.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: storage_class
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_STORAGE_CLASS
|
|
.IP \[bu] 2
|
|
Provider: AWS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Default
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]STANDARD\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Standard storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]REDUCED_REDUNDANCY\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Reduced redundancy storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]STANDARD_IA\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Standard Infrequent Access storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ONEZONE_IA\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
One Zone Infrequent Access storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]GLACIER\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Glacier storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]DEEP_ARCHIVE\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Glacier Deep Archive storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]INTELLIGENT_TIERING\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Intelligent-Tiering storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]GLACIER_IR\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Glacier Instant Retrieval storage class
|
|
.RE
|
|
.RE
|
|
.SS --s3-storage-class
|
|
.PP
|
|
The storage class to use when storing new objects in OSS.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: storage_class
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_STORAGE_CLASS
|
|
.IP \[bu] 2
|
|
Provider: Alibaba
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Default
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]STANDARD\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Standard storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]GLACIER\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Archive storage mode
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]STANDARD_IA\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Infrequent access storage mode
|
|
.RE
|
|
.RE
|
|
.SS --s3-storage-class
|
|
.PP
|
|
The storage class to use when storing new objects in Tencent COS.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: storage_class
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_STORAGE_CLASS
|
|
.IP \[bu] 2
|
|
Provider: TencentCOS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Default
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]STANDARD\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Standard storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ARCHIVE\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Archive storage mode
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]STANDARD_IA\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Infrequent access storage mode
|
|
.RE
|
|
.RE
|
|
.SS --s3-storage-class
|
|
.PP
|
|
The storage class to use when storing new objects in S3.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: storage_class
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_STORAGE_CLASS
|
|
.IP \[bu] 2
|
|
Provider: Scaleway
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Default.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]STANDARD\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The Standard class for any upload.
|
|
.IP \[bu] 2
|
|
Suitable for on-demand content like streaming or CDN.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]GLACIER\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Archived storage.
|
|
.IP \[bu] 2
|
|
Prices are lower, but it needs to be restored first to be accessed.
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to s3 (Amazon S3 Compliant
|
|
Storage Providers including AWS, Alibaba, Ceph, Digital Ocean,
|
|
Dreamhost, IBM COS, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent
|
|
COS).
|
|
.SS --s3-bucket-acl
|
|
.PP
|
|
Canned ACL used when creating buckets.
|
|
.PP
|
|
For more info visit
|
|
https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
|
|
.PP
|
|
Note that this ACL is applied when only when creating buckets.
|
|
If it isn\[aq]t set then \[dq]acl\[dq] is used instead.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: bucket_acl
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_BUCKET_ACL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]private\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
No one else has access rights (default).
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]public-read\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
The AllUsers group gets READ access.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]public-read-write\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
The AllUsers group gets READ and WRITE access.
|
|
.IP \[bu] 2
|
|
Granting this on a bucket is generally not recommended.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]authenticated-read\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owner gets FULL_CONTROL.
|
|
.IP \[bu] 2
|
|
The AuthenticatedUsers group gets READ access.
|
|
.RE
|
|
.RE
|
|
.SS --s3-requester-pays
|
|
.PP
|
|
Enables requester pays option when interacting with S3 bucket.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: requester_pays
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_REQUESTER_PAYS
|
|
.IP \[bu] 2
|
|
Provider: AWS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --s3-sse-customer-algorithm
|
|
.PP
|
|
If using SSE-C, the server-side encryption algorithm used when storing
|
|
this object in S3.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: sse_customer_algorithm
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_SSE_CUSTOMER_ALGORITHM
|
|
.IP \[bu] 2
|
|
Provider: AWS,Ceph,Minio
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
None
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]AES256\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
AES256
|
|
.RE
|
|
.RE
|
|
.SS --s3-sse-customer-key
|
|
.PP
|
|
If using SSE-C you must provide the secret encryption key used to
|
|
encrypt/decrypt your data.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: sse_customer_key
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_SSE_CUSTOMER_KEY
|
|
.IP \[bu] 2
|
|
Provider: AWS,Ceph,Minio
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
None
|
|
.RE
|
|
.RE
|
|
.SS --s3-sse-customer-key-md5
|
|
.PP
|
|
If using SSE-C you may provide the secret encryption key MD5 checksum
|
|
(optional).
|
|
.PP
|
|
If you leave it blank, this is calculated automatically from the
|
|
sse_customer_key provided.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: sse_customer_key_md5
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_MD5
|
|
.IP \[bu] 2
|
|
Provider: AWS,Ceph,Minio
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
None
|
|
.RE
|
|
.RE
|
|
.SS --s3-upload-cutoff
|
|
.PP
|
|
Cutoff for switching to chunked upload.
|
|
.PP
|
|
Any files larger than this will be uploaded in chunks of chunk_size.
|
|
The minimum is 0 and the maximum is 5 GiB.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_cutoff
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_UPLOAD_CUTOFF
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 200Mi
|
|
.SS --s3-chunk-size
|
|
.PP
|
|
Chunk size to use for uploading.
|
|
.PP
|
|
When uploading files larger than upload_cutoff or files with unknown
|
|
size (e.g.
|
|
from \[dq]rclone rcat\[dq] or uploaded with \[dq]rclone mount\[dq] or
|
|
google photos or google docs) they will be uploaded as multipart uploads
|
|
using this chunk size.
|
|
.PP
|
|
Note that \[dq]--s3-upload-concurrency\[dq] chunks of this size are
|
|
buffered in memory per transfer.
|
|
.PP
|
|
If you are transferring large files over high-speed links and you have
|
|
enough memory, then increasing this will speed up the transfers.
|
|
.PP
|
|
Rclone will automatically increase the chunk size when uploading a large
|
|
file of known size to stay below the 10,000 chunks limit.
|
|
.PP
|
|
Files of unknown size are uploaded with the configured chunk_size.
|
|
Since the default chunk size is 5 MiB and there can be at most 10,000
|
|
chunks, this means that by default the maximum size of a file you can
|
|
stream upload is 48 GiB.
|
|
If you wish to stream upload larger files then you will need to increase
|
|
chunk_size.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 5Mi
|
|
.SS --s3-max-upload-parts
|
|
.PP
|
|
Maximum number of parts in a multipart upload.
|
|
.PP
|
|
This option defines the maximum number of multipart chunks to use when
|
|
doing a multipart upload.
|
|
.PP
|
|
This can be useful if a service does not support the AWS S3
|
|
specification of 10,000 chunks.
|
|
.PP
|
|
Rclone will automatically increase the chunk size when uploading a large
|
|
file of a known size to stay below this number of chunks limit.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: max_upload_parts
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_MAX_UPLOAD_PARTS
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 10000
|
|
.SS --s3-copy-cutoff
|
|
.PP
|
|
Cutoff for switching to multipart copy.
|
|
.PP
|
|
Any files larger than this that need to be server-side copied will be
|
|
copied in chunks of this size.
|
|
.PP
|
|
The minimum is 0 and the maximum is 5 GiB.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: copy_cutoff
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_COPY_CUTOFF
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 4.656Gi
|
|
.SS --s3-disable-checksum
|
|
.PP
|
|
Don\[aq]t store MD5 checksum with object metadata.
|
|
.PP
|
|
Normally rclone will calculate the MD5 checksum of the input before
|
|
uploading it so it can add it to metadata on the object.
|
|
This is great for data integrity checking but can cause long delays for
|
|
large files to start uploading.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_checksum
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_DISABLE_CHECKSUM
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --s3-shared-credentials-file
|
|
.PP
|
|
Path to the shared credentials file.
|
|
.PP
|
|
If env_auth = true then rclone can use a shared credentials file.
|
|
.PP
|
|
If this variable is empty rclone will look for the
|
|
\[dq]AWS_SHARED_CREDENTIALS_FILE\[dq] env variable.
|
|
If the env value is empty it will default to the current user\[aq]s home
|
|
directory.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Linux/OSX: \[dq]$HOME/.aws/credentials\[dq]
|
|
Windows: \[dq]%USERPROFILE%\[rs].aws\[rs]credentials\[dq]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: shared_credentials_file
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_SHARED_CREDENTIALS_FILE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --s3-profile
|
|
.PP
|
|
Profile to use in the shared credentials file.
|
|
.PP
|
|
If env_auth = true then rclone can use a shared credentials file.
|
|
This variable controls which profile is used in that file.
|
|
.PP
|
|
If empty it will default to the environment variable
|
|
\[dq]AWS_PROFILE\[dq] or \[dq]default\[dq] if that environment variable
|
|
is also not set.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: profile
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_PROFILE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --s3-session-token
|
|
.PP
|
|
An AWS session token.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: session_token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_SESSION_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --s3-upload-concurrency
|
|
.PP
|
|
Concurrency for multipart uploads.
|
|
.PP
|
|
This is the number of chunks of the same file that are uploaded
|
|
concurrently.
|
|
.PP
|
|
If you are uploading small numbers of large files over high-speed links
|
|
and these uploads do not fully utilize your bandwidth, then increasing
|
|
this may help to speed up the transfers.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_concurrency
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_UPLOAD_CONCURRENCY
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 4
|
|
.SS --s3-force-path-style
|
|
.PP
|
|
If true use path style access if false use virtual hosted style.
|
|
.PP
|
|
If this is true (the default) then rclone will use path style access, if
|
|
false then rclone will use virtual path style.
|
|
See the AWS S3
|
|
docs (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro)
|
|
for more info.
|
|
.PP
|
|
Some providers (e.g.
|
|
AWS, Aliyun OSS, Netease COS, or Tencent COS) require this set to false
|
|
- rclone will do this automatically based on the provider setting.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: force_path_style
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_FORCE_PATH_STYLE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: true
|
|
.SS --s3-v2-auth
|
|
.PP
|
|
If true use v2 authentication.
|
|
.PP
|
|
If this is false (the default) then rclone will use v4 authentication.
|
|
If it is set then rclone will use v2 authentication.
|
|
.PP
|
|
Use this only if v4 signatures don\[aq]t work, e.g.
|
|
pre Jewel/v10 CEPH.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: v2_auth
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_V2_AUTH
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --s3-use-accelerate-endpoint
|
|
.PP
|
|
If true use the AWS S3 accelerated endpoint.
|
|
.PP
|
|
See: AWS S3 Transfer
|
|
acceleration (https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration-examples.html)
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: use_accelerate_endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_USE_ACCELERATE_ENDPOINT
|
|
.IP \[bu] 2
|
|
Provider: AWS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --s3-leave-parts-on-error
|
|
.PP
|
|
If true avoid calling abort upload on a failure, leaving all
|
|
successfully uploaded parts on S3 for manual recovery.
|
|
.PP
|
|
It should be set to true for resuming uploads across different sessions.
|
|
.PP
|
|
WARNING: Storing parts of an incomplete multipart upload counts towards
|
|
space usage on S3 and will add additional costs if not cleaned up.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: leave_parts_on_error
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_LEAVE_PARTS_ON_ERROR
|
|
.IP \[bu] 2
|
|
Provider: AWS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --s3-list-chunk
|
|
.PP
|
|
Size of listing chunk (response list for each ListObject S3 request).
|
|
.PP
|
|
This option is also known as \[dq]MaxKeys\[dq], \[dq]max-items\[dq], or
|
|
\[dq]page-size\[dq] from the AWS S3 specification.
|
|
Most services truncate the response list to 1000 objects even if
|
|
requested more than that.
|
|
In AWS S3 this is a global maximum and cannot be changed, see AWS
|
|
S3 (https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html).
|
|
In Ceph, this can be increased with the \[dq]rgw list buckets max
|
|
chunk\[dq] option.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: list_chunk
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_LIST_CHUNK
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 1000
|
|
.SS --s3-list-version
|
|
.PP
|
|
Version of ListObjects to use: 1,2 or 0 for auto.
|
|
.PP
|
|
When S3 originally launched it only provided the ListObjects call to
|
|
enumerate objects in a bucket.
|
|
.PP
|
|
However in May 2016 the ListObjectsV2 call was introduced.
|
|
This is much higher performance and should be used if at all possible.
|
|
.PP
|
|
If set to the default, 0, rclone will guess according to the provider
|
|
set which list objects method to call.
|
|
If it guesses wrong, then it may be set manually here.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: list_version
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_LIST_VERSION
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 0
|
|
.SS --s3-list-url-encode
|
|
.PP
|
|
Whether to url encode listings: true/false/unset
|
|
.PP
|
|
Some providers support URL encoding listings and where this is available
|
|
this is more reliable when using control characters in file names.
|
|
If this is set to unset (the default) then rclone will choose according
|
|
to the provider setting what to apply, but you can override rclone\[aq]s
|
|
choice here.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: list_url_encode
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_LIST_URL_ENCODE
|
|
.IP \[bu] 2
|
|
Type: Tristate
|
|
.IP \[bu] 2
|
|
Default: unset
|
|
.SS --s3-no-check-bucket
|
|
.PP
|
|
If set, don\[aq]t attempt to check the bucket exists or create it.
|
|
.PP
|
|
This can be useful when trying to minimise the number of transactions
|
|
rclone does if you know the bucket exists already.
|
|
.PP
|
|
It can also be needed if the user you are using does not have bucket
|
|
creation permissions.
|
|
Before v1.52.0 this would have passed silently due to a bug.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_check_bucket
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_NO_CHECK_BUCKET
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --s3-no-head
|
|
.PP
|
|
If set, don\[aq]t HEAD uploaded objects to check integrity.
|
|
.PP
|
|
This can be useful when trying to minimise the number of transactions
|
|
rclone does.
|
|
.PP
|
|
Setting it means that if rclone receives a 200 OK message after
|
|
uploading an object with PUT then it will assume that it got uploaded
|
|
properly.
|
|
.PP
|
|
In particular it will assume:
|
|
.IP \[bu] 2
|
|
the metadata, including modtime, storage class and content type was as
|
|
uploaded
|
|
.IP \[bu] 2
|
|
the size was as uploaded
|
|
.PP
|
|
It reads the following items from the response for a single part PUT:
|
|
.IP \[bu] 2
|
|
the MD5SUM
|
|
.IP \[bu] 2
|
|
The uploaded date
|
|
.PP
|
|
For multipart uploads these items aren\[aq]t read.
|
|
.PP
|
|
If an source object of unknown length is uploaded then rclone
|
|
\f[B]will\f[R] do a HEAD request.
|
|
.PP
|
|
Setting this flag increases the chance for undetected upload failures,
|
|
in particular an incorrect size, so it isn\[aq]t recommended for normal
|
|
operation.
|
|
In practice the chance of an undetected upload failure is very small
|
|
even with this flag.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_head
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_NO_HEAD
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --s3-no-head-object
|
|
.PP
|
|
If set, do not do HEAD before GET when getting objects.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_head_object
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_NO_HEAD_OBJECT
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --s3-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,InvalidUtf8,Dot
|
|
.SS --s3-memory-pool-flush-time
|
|
.PP
|
|
How often internal memory buffer pools will be flushed.
|
|
.PP
|
|
Uploads which requires additional buffers (f.e multipart) will use
|
|
memory pool for allocations.
|
|
This option controls how often unused buffers will be removed from the
|
|
pool.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: memory_pool_flush_time
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_MEMORY_POOL_FLUSH_TIME
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 1m0s
|
|
.SS --s3-memory-pool-use-mmap
|
|
.PP
|
|
Whether to use mmap buffers in internal memory pool.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: memory_pool_use_mmap
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_MEMORY_POOL_USE_MMAP
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --s3-disable-http2
|
|
.PP
|
|
Disable usage of http2 for S3 backends.
|
|
.PP
|
|
There is currently an unsolved issue with the s3 (specifically minio)
|
|
backend and HTTP/2.
|
|
HTTP/2 is enabled by default for the s3 backend but can be disabled
|
|
here.
|
|
When the issue is solved this flag will be removed.
|
|
.PP
|
|
See: https://github.com/rclone/rclone/issues/4673,
|
|
https://github.com/rclone/rclone/issues/3631
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_http2
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_DISABLE_HTTP2
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --s3-download-url
|
|
.PP
|
|
Custom endpoint for downloads.
|
|
This is usually set to a CloudFront CDN URL as AWS S3 offers cheaper
|
|
egress for data downloaded through the CloudFront network.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: download_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_DOWNLOAD_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --s3-use-multipart-etag
|
|
.PP
|
|
Whether to use ETag in multipart uploads for verification
|
|
.PP
|
|
This should be true, false or left unset to use the default for the
|
|
provider.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: use_multipart_etag
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_S3_USE_MULTIPART_ETAG
|
|
.IP \[bu] 2
|
|
Type: Tristate
|
|
.IP \[bu] 2
|
|
Default: unset
|
|
.SS Backend commands
|
|
.PP
|
|
Here are the commands specific to the s3 backend.
|
|
.PP
|
|
Run them with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend COMMAND remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The help below will explain what arguments each command takes.
|
|
.PP
|
|
See the \[dq]rclone backend\[dq]
|
|
command (https://rclone.org/commands/rclone_backend/) for more info on
|
|
how to pass options and arguments.
|
|
.PP
|
|
These can be run on a running backend using the rc command
|
|
backend/command (https://rclone.org/rc/#backend-command).
|
|
.SS restore
|
|
.PP
|
|
Restore objects from GLACIER to normal storage
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend restore remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This command can be used to restore one or more objects from GLACIER to
|
|
normal storage.
|
|
.PP
|
|
Usage Examples:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend restore s3:bucket/path/to/object [-o priority=PRIORITY] [-o lifetime=DAYS]
|
|
rclone backend restore s3:bucket/path/to/directory [-o priority=PRIORITY] [-o lifetime=DAYS]
|
|
rclone backend restore s3:bucket [-o priority=PRIORITY] [-o lifetime=DAYS]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This flag also obeys the filters.
|
|
Test first with -i/--interactive or --dry-run flags
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone -i backend restore --include \[dq]*.txt\[dq] s3:bucket/path -o priority=Standard
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
All the objects shown will be marked for restore, then
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend restore --include \[dq]*.txt\[dq] s3:bucket/path -o priority=Standard
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
It returns a list of status dictionaries with Remote and Status keys.
|
|
The Status will be OK if it was successful or an error message if not.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[
|
|
{
|
|
\[dq]Status\[dq]: \[dq]OK\[dq],
|
|
\[dq]Path\[dq]: \[dq]test.txt\[dq]
|
|
},
|
|
{
|
|
\[dq]Status\[dq]: \[dq]OK\[dq],
|
|
\[dq]Path\[dq]: \[dq]test/file4.txt\[dq]
|
|
}
|
|
]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Options:
|
|
.IP \[bu] 2
|
|
\[dq]description\[dq]: The optional description for the job.
|
|
.IP \[bu] 2
|
|
\[dq]lifetime\[dq]: Lifetime of the active copy in days
|
|
.IP \[bu] 2
|
|
\[dq]priority\[dq]: Priority of restore: Standard|Expedited|Bulk
|
|
.SS list-multipart-uploads
|
|
.PP
|
|
List the unfinished multipart uploads
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend list-multipart-uploads remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This command lists the unfinished multipart uploads in JSON format.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend list-multipart s3:bucket/path/to/object
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
It returns a dictionary of buckets with values as lists of unfinished
|
|
multipart uploads.
|
|
.PP
|
|
You can call it with no bucket in which case it lists all bucket, with a
|
|
bucket or with a bucket and path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]rclone\[dq]: [
|
|
{
|
|
\[dq]Initiated\[dq]: \[dq]2020-06-26T14:20:36Z\[dq],
|
|
\[dq]Initiator\[dq]: {
|
|
\[dq]DisplayName\[dq]: \[dq]XXX\[dq],
|
|
\[dq]ID\[dq]: \[dq]arn:aws:iam::XXX:user/XXX\[dq]
|
|
},
|
|
\[dq]Key\[dq]: \[dq]KEY\[dq],
|
|
\[dq]Owner\[dq]: {
|
|
\[dq]DisplayName\[dq]: null,
|
|
\[dq]ID\[dq]: \[dq]XXX\[dq]
|
|
},
|
|
\[dq]StorageClass\[dq]: \[dq]STANDARD\[dq],
|
|
\[dq]UploadId\[dq]: \[dq]XXX\[dq]
|
|
}
|
|
],
|
|
\[dq]rclone-1000files\[dq]: [],
|
|
\[dq]rclone-dst\[dq]: []
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.SS cleanup
|
|
.PP
|
|
Remove unfinished multipart uploads.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend cleanup remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This command removes unfinished multipart uploads of age greater than
|
|
max-age which defaults to 24 hours.
|
|
.PP
|
|
Note that you can use -i/--dry-run with this command to see what it
|
|
would do.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend cleanup s3:bucket/path/to/object
|
|
rclone backend cleanup -o max-age=7w s3:bucket/path/to/object
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc.
|
|
.PP
|
|
Options:
|
|
.IP \[bu] 2
|
|
\[dq]max-age\[dq]: Max age of upload to delete
|
|
.SS Anonymous access to public buckets
|
|
.PP
|
|
If you want to use rclone to access a public bucket, configure with a
|
|
blank \f[C]access_key_id\f[R] and \f[C]secret_access_key\f[R].
|
|
Your config should end up looking like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[anons3]
|
|
type = s3
|
|
provider = AWS
|
|
env_auth = false
|
|
access_key_id =
|
|
secret_access_key =
|
|
region = us-east-1
|
|
endpoint =
|
|
location_constraint =
|
|
acl = private
|
|
server_side_encryption =
|
|
storage_class =
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Then use it as normal with the name of the public bucket, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd anons3:1000genomes
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You will be able to list and copy data but not upload it.
|
|
.SS Providers
|
|
.SS AWS S3
|
|
.PP
|
|
This is the provider used as main example and described in the
|
|
configuration section above.
|
|
.SS AWS Snowball Edge
|
|
.PP
|
|
AWS Snowball (https://aws.amazon.com/snowball/) is a hardware appliance
|
|
used for transferring bulk data back to AWS.
|
|
Its main software interface is S3 object storage.
|
|
.PP
|
|
To use rclone with AWS Snowball Edge devices, configure as standard for
|
|
an \[aq]S3 Compatible Service\[aq] be sure to set
|
|
\f[C]upload_cutoff = 0\f[R] otherwise you will run into authentication
|
|
header issues as the snowball device does not support query parameter
|
|
based authentication.
|
|
.PP
|
|
eg.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[snowball]
|
|
type = s3
|
|
provider = Other
|
|
access_key_id = YOUR_ACCESS_KEY
|
|
secret_access_key = YOUR_SECRET_KEY
|
|
endpoint = http://[IP of Snowball]:8080
|
|
upload_cutoff = 0
|
|
\f[R]
|
|
.fi
|
|
.SS Ceph
|
|
.PP
|
|
Ceph (https://ceph.com/) is an open-source, unified, distributed storage
|
|
system designed for excellent performance, reliability and scalability.
|
|
It has an S3 compatible object storage interface.
|
|
.PP
|
|
To use rclone with Ceph, configure as above but leave the region blank
|
|
and set the endpoint.
|
|
You should end up with something like this in your config:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[ceph]
|
|
type = s3
|
|
provider = Ceph
|
|
env_auth = false
|
|
access_key_id = XXX
|
|
secret_access_key = YYY
|
|
region =
|
|
endpoint = https://ceph.endpoint.example.com
|
|
location_constraint =
|
|
acl =
|
|
server_side_encryption =
|
|
storage_class =
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you are using an older version of CEPH, e.g.
|
|
10.2.x Jewel, then you may need to supply the parameter
|
|
\f[C]--s3-upload-cutoff 0\f[R] or put this in the config file as
|
|
\f[C]upload_cutoff 0\f[R] to work around a bug which causes uploading of
|
|
small files to fail.
|
|
.PP
|
|
Note also that Ceph sometimes puts \f[C]/\f[R] in the passwords it gives
|
|
users.
|
|
If you read the secret access key using the command line tools you will
|
|
get a JSON blob with the \f[C]/\f[R] escaped as \f[C]\[rs]/\f[R].
|
|
Make sure you only write \f[C]/\f[R] in the secret access key.
|
|
.PP
|
|
Eg the dump from Ceph looks something like this (irrelevant keys
|
|
removed).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]user_id\[dq]: \[dq]xxx\[dq],
|
|
\[dq]display_name\[dq]: \[dq]xxxx\[dq],
|
|
\[dq]keys\[dq]: [
|
|
{
|
|
\[dq]user\[dq]: \[dq]xxx\[dq],
|
|
\[dq]access_key\[dq]: \[dq]xxxxxx\[dq],
|
|
\[dq]secret_key\[dq]: \[dq]xxxxxx\[rs]/xxxx\[dq]
|
|
}
|
|
],
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Because this is a json dump, it is encoding the \f[C]/\f[R] as
|
|
\f[C]\[rs]/\f[R], so if you use the secret key as \f[C]xxxxxx/xxxx\f[R]
|
|
it will work fine.
|
|
.SS Dreamhost
|
|
.PP
|
|
Dreamhost DreamObjects (https://www.dreamhost.com/cloud/storage/) is an
|
|
object storage system based on CEPH.
|
|
.PP
|
|
To use rclone with Dreamhost, configure as above but leave the region
|
|
blank and set the endpoint.
|
|
You should end up with something like this in your config:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[dreamobjects]
|
|
type = s3
|
|
provider = DreamHost
|
|
env_auth = false
|
|
access_key_id = your_access_key
|
|
secret_access_key = your_secret_key
|
|
region =
|
|
endpoint = objects-us-west-1.dream.io
|
|
location_constraint =
|
|
acl = private
|
|
server_side_encryption =
|
|
storage_class =
|
|
\f[R]
|
|
.fi
|
|
.SS DigitalOcean Spaces
|
|
.PP
|
|
Spaces (https://www.digitalocean.com/products/object-storage/) is an
|
|
S3-interoperable (https://developers.digitalocean.com/documentation/spaces/)
|
|
object storage service from cloud provider DigitalOcean.
|
|
.PP
|
|
To connect to DigitalOcean Spaces you will need an access key and secret
|
|
key.
|
|
These can be retrieved on the \[dq]Applications &
|
|
API (https://cloud.digitalocean.com/settings/api/tokens)\[dq] page of
|
|
the DigitalOcean control panel.
|
|
They will be needed when prompted by \f[C]rclone config\f[R] for your
|
|
\f[C]access_key_id\f[R] and \f[C]secret_access_key\f[R].
|
|
.PP
|
|
When prompted for a \f[C]region\f[R] or \f[C]location_constraint\f[R],
|
|
press enter to use the default value.
|
|
The region must be included in the \f[C]endpoint\f[R] setting (e.g.
|
|
\f[C]nyc3.digitaloceanspaces.com\f[R]).
|
|
The default values can be used for other settings.
|
|
.PP
|
|
Going through the whole process of creating a new remote by running
|
|
\f[C]rclone config\f[R], each prompt should be answered as shown below:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Storage> s3
|
|
env_auth> 1
|
|
access_key_id> YOUR_ACCESS_KEY
|
|
secret_access_key> YOUR_SECRET_KEY
|
|
region>
|
|
endpoint> nyc3.digitaloceanspaces.com
|
|
location_constraint>
|
|
acl>
|
|
storage_class>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The resulting configuration file should look like:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[spaces]
|
|
type = s3
|
|
provider = DigitalOcean
|
|
env_auth = false
|
|
access_key_id = YOUR_ACCESS_KEY
|
|
secret_access_key = YOUR_SECRET_KEY
|
|
region =
|
|
endpoint = nyc3.digitaloceanspaces.com
|
|
location_constraint =
|
|
acl =
|
|
server_side_encryption =
|
|
storage_class =
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Once configured, you can create a new Space and begin copying files.
|
|
For example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir spaces:my-new-space
|
|
rclone copy /path/to/files spaces:my-new-space
|
|
\f[R]
|
|
.fi
|
|
.SS IBM COS (S3)
|
|
.PP
|
|
Information stored with IBM Cloud Object Storage is encrypted and
|
|
dispersed across multiple geographic locations, and accessed through an
|
|
implementation of the S3 API.
|
|
This service makes use of the distributed storage technologies provided
|
|
by IBM\[cq]s Cloud Object Storage System (formerly Cleversafe).
|
|
For more information visit: (http://www.ibm.com/cloud/object-storage)
|
|
.PP
|
|
To configure access to IBM COS S3, follow the steps below:
|
|
.IP "1." 3
|
|
Run rclone config and select n for a new remote.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
2018/02/14 14:13:11 NOTICE: Config file \[dq]C:\[rs]\[rs]Users\[rs]\[rs]a\[rs]\[rs].config\[rs]\[rs]rclone\[rs]\[rs]rclone.conf\[dq] not found - using defaults
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
\f[R]
|
|
.fi
|
|
.IP "2." 3
|
|
Enter the name for the configuration
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
name> <YOUR NAME>
|
|
\f[R]
|
|
.fi
|
|
.IP "3." 3
|
|
Select \[dq]s3\[dq] storage.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Choose a number from below, or type in your own value
|
|
1 / Alias for an existing remote
|
|
\[rs] \[dq]alias\[dq]
|
|
2 / Amazon Drive
|
|
\[rs] \[dq]amazon cloud drive\[dq]
|
|
3 / Amazon S3 Complaint Storage Providers (Dreamhost, Ceph, Minio, IBM COS)
|
|
\[rs] \[dq]s3\[dq]
|
|
4 / Backblaze B2
|
|
\[rs] \[dq]b2\[dq]
|
|
[snip]
|
|
23 / http Connection
|
|
\[rs] \[dq]http\[dq]
|
|
Storage> 3
|
|
\f[R]
|
|
.fi
|
|
.IP "4." 3
|
|
Select IBM COS as the S3 Storage Provider.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Choose the S3 provider.
|
|
Choose a number from below, or type in your own value
|
|
1 / Choose this option to configure Storage to AWS S3
|
|
\[rs] \[dq]AWS\[dq]
|
|
2 / Choose this option to configure Storage to Ceph Systems
|
|
\[rs] \[dq]Ceph\[dq]
|
|
3 / Choose this option to configure Storage to Dreamhost
|
|
\[rs] \[dq]Dreamhost\[dq]
|
|
4 / Choose this option to the configure Storage to IBM COS S3
|
|
\[rs] \[dq]IBMCOS\[dq]
|
|
5 / Choose this option to the configure Storage to Minio
|
|
\[rs] \[dq]Minio\[dq]
|
|
Provider>4
|
|
\f[R]
|
|
.fi
|
|
.IP "5." 3
|
|
Enter the Access Key and Secret.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
AWS Access Key ID - leave blank for anonymous access or runtime credentials.
|
|
access_key_id> <>
|
|
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
|
|
secret_access_key> <>
|
|
\f[R]
|
|
.fi
|
|
.IP "6." 3
|
|
Specify the endpoint for IBM COS.
|
|
For Public IBM COS, choose from the option below.
|
|
For On Premise IBM COS, enter an endpoint address.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Endpoint for IBM COS S3 API.
|
|
Specify if using an IBM COS On Premise.
|
|
Choose a number from below, or type in your own value
|
|
1 / US Cross Region Endpoint
|
|
\[rs] \[dq]s3-api.us-geo.objectstorage.softlayer.net\[dq]
|
|
2 / US Cross Region Dallas Endpoint
|
|
\[rs] \[dq]s3-api.dal.us-geo.objectstorage.softlayer.net\[dq]
|
|
3 / US Cross Region Washington DC Endpoint
|
|
\[rs] \[dq]s3-api.wdc-us-geo.objectstorage.softlayer.net\[dq]
|
|
4 / US Cross Region San Jose Endpoint
|
|
\[rs] \[dq]s3-api.sjc-us-geo.objectstorage.softlayer.net\[dq]
|
|
5 / US Cross Region Private Endpoint
|
|
\[rs] \[dq]s3-api.us-geo.objectstorage.service.networklayer.com\[dq]
|
|
6 / US Cross Region Dallas Private Endpoint
|
|
\[rs] \[dq]s3-api.dal-us-geo.objectstorage.service.networklayer.com\[dq]
|
|
7 / US Cross Region Washington DC Private Endpoint
|
|
\[rs] \[dq]s3-api.wdc-us-geo.objectstorage.service.networklayer.com\[dq]
|
|
8 / US Cross Region San Jose Private Endpoint
|
|
\[rs] \[dq]s3-api.sjc-us-geo.objectstorage.service.networklayer.com\[dq]
|
|
9 / US Region East Endpoint
|
|
\[rs] \[dq]s3.us-east.objectstorage.softlayer.net\[dq]
|
|
10 / US Region East Private Endpoint
|
|
\[rs] \[dq]s3.us-east.objectstorage.service.networklayer.com\[dq]
|
|
11 / US Region South Endpoint
|
|
[snip]
|
|
34 / Toronto Single Site Private Endpoint
|
|
\[rs] \[dq]s3.tor01.objectstorage.service.networklayer.com\[dq]
|
|
endpoint>1
|
|
\f[R]
|
|
.fi
|
|
.IP "7." 3
|
|
Specify a IBM COS Location Constraint.
|
|
The location constraint must match endpoint when using IBM Cloud Public.
|
|
For on-prem COS, do not make a selection from this list, hit enter
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
1 / US Cross Region Standard
|
|
\[rs] \[dq]us-standard\[dq]
|
|
2 / US Cross Region Vault
|
|
\[rs] \[dq]us-vault\[dq]
|
|
3 / US Cross Region Cold
|
|
\[rs] \[dq]us-cold\[dq]
|
|
4 / US Cross Region Flex
|
|
\[rs] \[dq]us-flex\[dq]
|
|
5 / US East Region Standard
|
|
\[rs] \[dq]us-east-standard\[dq]
|
|
6 / US East Region Vault
|
|
\[rs] \[dq]us-east-vault\[dq]
|
|
7 / US East Region Cold
|
|
\[rs] \[dq]us-east-cold\[dq]
|
|
8 / US East Region Flex
|
|
\[rs] \[dq]us-east-flex\[dq]
|
|
9 / US South Region Standard
|
|
\[rs] \[dq]us-south-standard\[dq]
|
|
10 / US South Region Vault
|
|
\[rs] \[dq]us-south-vault\[dq]
|
|
[snip]
|
|
32 / Toronto Flex
|
|
\[rs] \[dq]tor01-flex\[dq]
|
|
location_constraint>1
|
|
\f[R]
|
|
.fi
|
|
.IP "9." 3
|
|
Specify a canned ACL.
|
|
IBM Cloud (Storage) supports \[dq]public-read\[dq] and
|
|
\[dq]private\[dq].
|
|
IBM Cloud(Infra) supports all the canned ACLs.
|
|
On-Premise COS supports all the canned ACLs.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Canned ACL used when creating buckets and/or storing objects in S3.
|
|
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
|
|
Choose a number from below, or type in your own value
|
|
1 / Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS
|
|
\[rs] \[dq]private\[dq]
|
|
2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS
|
|
\[rs] \[dq]public-read\[dq]
|
|
3 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS
|
|
\[rs] \[dq]public-read-write\[dq]
|
|
4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS
|
|
\[rs] \[dq]authenticated-read\[dq]
|
|
acl> 1
|
|
\f[R]
|
|
.fi
|
|
.IP "12." 4
|
|
Review the displayed configuration and accept to save the
|
|
\[dq]remote\[dq] then quit.
|
|
The config file should look like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[xxx]
|
|
type = s3
|
|
Provider = IBMCOS
|
|
access_key_id = xxx
|
|
secret_access_key = yyy
|
|
endpoint = s3-api.us-geo.objectstorage.softlayer.net
|
|
location_constraint = us-standard
|
|
acl = private
|
|
\f[R]
|
|
.fi
|
|
.IP "13." 4
|
|
Execute rclone commands
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
1) Create a bucket.
|
|
rclone mkdir IBM-COS-XREGION:newbucket
|
|
2) List available buckets.
|
|
rclone lsd IBM-COS-XREGION:
|
|
-1 2017-11-08 21:16:22 -1 test
|
|
-1 2018-02-14 20:16:39 -1 newbucket
|
|
3) List contents of a bucket.
|
|
rclone ls IBM-COS-XREGION:newbucket
|
|
18685952 test.exe
|
|
4) Copy a file from local to remote.
|
|
rclone copy /Users/file.txt IBM-COS-XREGION:newbucket
|
|
5) Copy a file from remote to local.
|
|
rclone copy IBM-COS-XREGION:newbucket/file.txt .
|
|
6) Delete a file on remote.
|
|
rclone delete IBM-COS-XREGION:newbucket/file.txt
|
|
\f[R]
|
|
.fi
|
|
.SS Minio
|
|
.PP
|
|
Minio (https://minio.io/) is an object storage server built for cloud
|
|
application developers and devops.
|
|
.PP
|
|
It is very easy to install and provides an S3 compatible server which
|
|
can be used by rclone.
|
|
.PP
|
|
To use it, install Minio following the instructions
|
|
here (https://docs.minio.io/docs/minio-quickstart-guide).
|
|
.PP
|
|
When it configures itself Minio will print something like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Endpoint: http://192.168.1.106:9000 http://172.23.0.1:9000
|
|
AccessKey: USWUXHGYZQYFYFFIT3RE
|
|
SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
|
|
Region: us-east-1
|
|
SQS ARNs: arn:minio:sqs:us-east-1:1:redis arn:minio:sqs:us-east-1:2:redis
|
|
|
|
Browser Access:
|
|
http://192.168.1.106:9000 http://172.23.0.1:9000
|
|
|
|
Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
|
|
$ mc config host add myminio http://192.168.1.106:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
|
|
|
|
Object API (Amazon S3 compatible):
|
|
Go: https://docs.minio.io/docs/golang-client-quickstart-guide
|
|
Java: https://docs.minio.io/docs/java-client-quickstart-guide
|
|
Python: https://docs.minio.io/docs/python-client-quickstart-guide
|
|
JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
|
|
.NET: https://docs.minio.io/docs/dotnet-client-quickstart-guide
|
|
|
|
Drive Capacity: 26 GiB Free, 165 GiB Total
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
These details need to go into \f[C]rclone config\f[R] like this.
|
|
Note that it is important to put the region in as stated above.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
env_auth> 1
|
|
access_key_id> USWUXHGYZQYFYFFIT3RE
|
|
secret_access_key> MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
|
|
region> us-east-1
|
|
endpoint> http://192.168.1.106:9000
|
|
location_constraint>
|
|
server_side_encryption>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Which makes the config file look like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[minio]
|
|
type = s3
|
|
provider = Minio
|
|
env_auth = false
|
|
access_key_id = USWUXHGYZQYFYFFIT3RE
|
|
secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
|
|
region = us-east-1
|
|
endpoint = http://192.168.1.106:9000
|
|
location_constraint =
|
|
server_side_encryption =
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
So once set up, for example, to copy files into a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /path/to/files minio:bucket
|
|
\f[R]
|
|
.fi
|
|
.SS RackCorp
|
|
.PP
|
|
RackCorp Object Storage (https://www.rackcorp.com/storage/s3storage) is
|
|
an S3 compatible object storage platform from your friendly cloud
|
|
provider RackCorp.
|
|
The service is fast, reliable, well priced and located in many strategic
|
|
locations unserviced by others, to ensure you can maintain data
|
|
sovereignty.
|
|
.PP
|
|
Before you can use RackCorp Object Storage, you\[aq]ll need to \[dq]sign
|
|
up (https://www.rackcorp.com/signup)\[dq] for an account on our
|
|
\[dq]portal (https://portal.rackcorp.com)\[dq].
|
|
Next you can create an \f[C]access key\f[R], a \f[C]secret key\f[R] and
|
|
\f[C]buckets\f[R], in your location of choice with ease.
|
|
These details are required for the next steps of configuration, when
|
|
\f[C]rclone config\f[R] asks for your \f[C]access_key_id\f[R] and
|
|
\f[C]secret_access_key\f[R].
|
|
.PP
|
|
Your config should end up looking a bit like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[RCS3-demo-config]
|
|
type = s3
|
|
provider = RackCorp
|
|
env_auth = true
|
|
access_key_id = YOURACCESSKEY
|
|
secret_access_key = YOURSECRETACCESSKEY
|
|
region = au-nsw
|
|
endpoint = s3.rackcorp.com
|
|
location_constraint = au-nsw
|
|
\f[R]
|
|
.fi
|
|
.SS Scaleway
|
|
.PP
|
|
Scaleway (https://www.scaleway.com/object-storage/) The Object Storage
|
|
platform allows you to store anything from backups, logs and web assets
|
|
to documents and photos.
|
|
Files can be dropped from the Scaleway console or transferred through
|
|
our API and CLI or using any S3-compatible tool.
|
|
.PP
|
|
Scaleway provides an S3 interface which can be configured for use with
|
|
rclone like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[scaleway]
|
|
type = s3
|
|
provider = Scaleway
|
|
env_auth = false
|
|
endpoint = s3.nl-ams.scw.cloud
|
|
access_key_id = SCWXXXXXXXXXXXXXX
|
|
secret_access_key = 1111111-2222-3333-44444-55555555555555
|
|
region = nl-ams
|
|
location_constraint =
|
|
acl = private
|
|
server_side_encryption =
|
|
storage_class =
|
|
\f[R]
|
|
.fi
|
|
.SS Seagate Lyve Cloud
|
|
.PP
|
|
Seagate Lyve
|
|
Cloud (https://www.seagate.com/gb/en/services/cloud/storage/) is an S3
|
|
compatible object storage platform from Seagate (https://seagate.com/)
|
|
intended for enterprise use.
|
|
.PP
|
|
Here is a config run through for a remote called \f[C]remote\f[R] - you
|
|
may choose a different name of course.
|
|
Note that to create an access key and secret key you will need to create
|
|
a service account first.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone config
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Choose \f[C]s3\f[R] backend
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value.
|
|
[snip]
|
|
XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent COS
|
|
\[rs] (s3)
|
|
[snip]
|
|
Storage> s3
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Choose \f[C]LyveCloud\f[R] as S3 provider
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Choose your S3 provider.
|
|
Choose a number from below, or type in your own value.
|
|
Press Enter to leave empty.
|
|
[snip]
|
|
XX / Seagate Lyve Cloud
|
|
\[rs] (LyveCloud)
|
|
[snip]
|
|
provider> LyveCloud
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Take the default (just press enter) to enter access key and secret in
|
|
the config file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
|
|
Only applies if access_key_id and secret_access_key is blank.
|
|
Choose a number from below, or type in your own boolean value (true or false).
|
|
Press Enter for the default (false).
|
|
1 / Enter AWS credentials in the next step.
|
|
\[rs] (false)
|
|
2 / Get AWS credentials from the environment (env vars or IAM).
|
|
\[rs] (true)
|
|
env_auth>
|
|
\f[R]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
AWS Access Key ID.
|
|
Leave blank for anonymous access or runtime credentials.
|
|
Enter a value. Press Enter to leave empty.
|
|
access_key_id> XXX
|
|
\f[R]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
AWS Secret Access Key (password).
|
|
Leave blank for anonymous access or runtime credentials.
|
|
Enter a value. Press Enter to leave empty.
|
|
secret_access_key> YYY
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Leave region blank
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Region to connect to.
|
|
Leave blank if you are using an S3 clone and you don\[aq]t have a region.
|
|
Choose a number from below, or type in your own value.
|
|
Press Enter to leave empty.
|
|
/ Use this if unsure.
|
|
1 | Will use v4 signatures and an empty region.
|
|
\[rs] ()
|
|
/ Use this only if v4 signatures don\[aq]t work.
|
|
2 | E.g. pre Jewel/v10 CEPH.
|
|
\[rs] (other-v2-signature)
|
|
region>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Choose an endpoint from the list
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Endpoint for S3 API.
|
|
Required when using an S3 clone.
|
|
Choose a number from below, or type in your own value.
|
|
Press Enter to leave empty.
|
|
1 / Seagate Lyve Cloud US East 1 (Virginia)
|
|
\[rs] (s3.us-east-1.lyvecloud.seagate.com)
|
|
2 / Seagate Lyve Cloud US West 1 (California)
|
|
\[rs] (s3.us-west-1.lyvecloud.seagate.com)
|
|
3 / Seagate Lyve Cloud AP Southeast 1 (Singapore)
|
|
\[rs] (s3.ap-southeast-1.lyvecloud.seagate.com)
|
|
endpoint> 1
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Leave location constraint blank
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Location constraint - must be set to match the Region.
|
|
Leave blank if not sure. Used when creating buckets only.
|
|
Enter a value. Press Enter to leave empty.
|
|
location_constraint>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Choose default ACL (\f[C]private\f[R]).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Canned ACL used when creating buckets and storing or copying objects.
|
|
This ACL is used for creating objects and if bucket_acl isn\[aq]t set, for creating buckets too.
|
|
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
|
|
Note that this ACL is applied when server-side copying objects as S3
|
|
doesn\[aq]t copy the ACL from the source but rather writes a fresh one.
|
|
Choose a number from below, or type in your own value.
|
|
Press Enter to leave empty.
|
|
/ Owner gets FULL_CONTROL.
|
|
1 | No one else has access rights (default).
|
|
\[rs] (private)
|
|
[snip]
|
|
acl>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And the config file should end up looking like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[remote]
|
|
type = s3
|
|
provider = LyveCloud
|
|
access_key_id = XXX
|
|
secret_access_key = YYY
|
|
endpoint = s3.us-east-1.lyvecloud.seagate.com
|
|
\f[R]
|
|
.fi
|
|
.SS SeaweedFS
|
|
.PP
|
|
SeaweedFS (https://github.com/chrislusf/seaweedfs/) is a distributed
|
|
storage system for blobs, objects, files, and data lake, with O(1) disk
|
|
seek and a scalable file metadata store.
|
|
It has an S3 compatible object storage interface.
|
|
SeaweedFS can also act as a gateway to remote S3 compatible object
|
|
store (https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remote-Object-Storage)
|
|
to cache data and metadata with asynchronous write back, for fast local
|
|
speed and minimize access cost.
|
|
.PP
|
|
Assuming the SeaweedFS are configured with \f[C]weed shell\f[R] as such:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
> s3.bucket.create -name foo
|
|
> s3.configure -access_key=any -secret_key=any -buckets=foo -user=me -actions=Read,Write,List,Tagging,Admin -apply
|
|
{
|
|
\[dq]identities\[dq]: [
|
|
{
|
|
\[dq]name\[dq]: \[dq]me\[dq],
|
|
\[dq]credentials\[dq]: [
|
|
{
|
|
\[dq]accessKey\[dq]: \[dq]any\[dq],
|
|
\[dq]secretKey\[dq]: \[dq]any\[dq]
|
|
}
|
|
],
|
|
\[dq]actions\[dq]: [
|
|
\[dq]Read:foo\[dq],
|
|
\[dq]Write:foo\[dq],
|
|
\[dq]List:foo\[dq],
|
|
\[dq]Tagging:foo\[dq],
|
|
\[dq]Admin:foo\[dq]
|
|
]
|
|
}
|
|
]
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To use rclone with SeaweedFS, above configuration should end up with
|
|
something like this in your config:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[seaweedfs_s3]
|
|
type = s3
|
|
provider = SeaweedFS
|
|
access_key_id = any
|
|
secret_access_key = any
|
|
endpoint = localhost:8333
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
So once set up, for example to copy files into a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /path/to/files seaweedfs_s3:foo
|
|
\f[R]
|
|
.fi
|
|
.SS Wasabi
|
|
.PP
|
|
Wasabi (https://wasabi.com) is a cloud-based object storage service for
|
|
a broad range of applications and use cases.
|
|
Wasabi is designed for individuals and organizations that require a
|
|
high-performance, reliable, and secure data storage infrastructure at
|
|
minimal cost.
|
|
.PP
|
|
Wasabi provides an S3 interface which can be configured for use with
|
|
rclone like this.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
n/s> n
|
|
name> wasabi
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Amazon S3 (also Dreamhost, Ceph, Minio)
|
|
\[rs] \[dq]s3\[dq]
|
|
[snip]
|
|
Storage> s3
|
|
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
|
|
Choose a number from below, or type in your own value
|
|
1 / Enter AWS credentials in the next step
|
|
\[rs] \[dq]false\[dq]
|
|
2 / Get AWS credentials from the environment (env vars or IAM)
|
|
\[rs] \[dq]true\[dq]
|
|
env_auth> 1
|
|
AWS Access Key ID - leave blank for anonymous access or runtime credentials.
|
|
access_key_id> YOURACCESSKEY
|
|
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
|
|
secret_access_key> YOURSECRETACCESSKEY
|
|
Region to connect to.
|
|
Choose a number from below, or type in your own value
|
|
/ The default endpoint - a good choice if you are unsure.
|
|
1 | US Region, Northern Virginia, or Pacific Northwest.
|
|
| Leave location constraint empty.
|
|
\[rs] \[dq]us-east-1\[dq]
|
|
[snip]
|
|
region> us-east-1
|
|
Endpoint for S3 API.
|
|
Leave blank if using AWS to use the default endpoint for the region.
|
|
Specify if using an S3 clone such as Ceph.
|
|
endpoint> s3.wasabisys.com
|
|
Location constraint - must be set to match the Region. Used when creating buckets only.
|
|
Choose a number from below, or type in your own value
|
|
1 / Empty for US Region, Northern Virginia, or Pacific Northwest.
|
|
\[rs] \[dq]\[dq]
|
|
[snip]
|
|
location_constraint>
|
|
Canned ACL used when creating buckets and/or storing objects in S3.
|
|
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
|
|
Choose a number from below, or type in your own value
|
|
1 / Owner gets FULL_CONTROL. No one else has access rights (default).
|
|
\[rs] \[dq]private\[dq]
|
|
[snip]
|
|
acl>
|
|
The server-side encryption algorithm used when storing this object in S3.
|
|
Choose a number from below, or type in your own value
|
|
1 / None
|
|
\[rs] \[dq]\[dq]
|
|
2 / AES256
|
|
\[rs] \[dq]AES256\[dq]
|
|
server_side_encryption>
|
|
The storage class to use when storing objects in S3.
|
|
Choose a number from below, or type in your own value
|
|
1 / Default
|
|
\[rs] \[dq]\[dq]
|
|
2 / Standard storage class
|
|
\[rs] \[dq]STANDARD\[dq]
|
|
3 / Reduced redundancy storage class
|
|
\[rs] \[dq]REDUCED_REDUNDANCY\[dq]
|
|
4 / Standard Infrequent Access storage class
|
|
\[rs] \[dq]STANDARD_IA\[dq]
|
|
storage_class>
|
|
Remote config
|
|
--------------------
|
|
[wasabi]
|
|
env_auth = false
|
|
access_key_id = YOURACCESSKEY
|
|
secret_access_key = YOURSECRETACCESSKEY
|
|
region = us-east-1
|
|
endpoint = s3.wasabisys.com
|
|
location_constraint =
|
|
acl =
|
|
server_side_encryption =
|
|
storage_class =
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will leave the config file looking like this.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[wasabi]
|
|
type = s3
|
|
provider = Wasabi
|
|
env_auth = false
|
|
access_key_id = YOURACCESSKEY
|
|
secret_access_key = YOURSECRETACCESSKEY
|
|
region =
|
|
endpoint = s3.wasabisys.com
|
|
location_constraint =
|
|
acl =
|
|
server_side_encryption =
|
|
storage_class =
|
|
\f[R]
|
|
.fi
|
|
.SS Alibaba OSS
|
|
.PP
|
|
Here is an example of making an Alibaba Cloud (Aliyun)
|
|
OSS (https://www.alibabacloud.com/product/oss/) configuration.
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> oss
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, and Tencent COS
|
|
\[rs] \[dq]s3\[dq]
|
|
[snip]
|
|
Storage> s3
|
|
Choose your S3 provider.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Amazon Web Services (AWS) S3
|
|
\[rs] \[dq]AWS\[dq]
|
|
2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
|
|
\[rs] \[dq]Alibaba\[dq]
|
|
3 / Ceph Object Storage
|
|
\[rs] \[dq]Ceph\[dq]
|
|
[snip]
|
|
provider> Alibaba
|
|
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
|
|
Only applies if access_key_id and secret_access_key is blank.
|
|
Enter a boolean value (true or false). Press Enter for the default (\[dq]false\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Enter AWS credentials in the next step
|
|
\[rs] \[dq]false\[dq]
|
|
2 / Get AWS credentials from the environment (env vars or IAM)
|
|
\[rs] \[dq]true\[dq]
|
|
env_auth> 1
|
|
AWS Access Key ID.
|
|
Leave blank for anonymous access or runtime credentials.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
access_key_id> accesskeyid
|
|
AWS Secret Access Key (password)
|
|
Leave blank for anonymous access or runtime credentials.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
secret_access_key> secretaccesskey
|
|
Endpoint for OSS API.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / East China 1 (Hangzhou)
|
|
\[rs] \[dq]oss-cn-hangzhou.aliyuncs.com\[dq]
|
|
2 / East China 2 (Shanghai)
|
|
\[rs] \[dq]oss-cn-shanghai.aliyuncs.com\[dq]
|
|
3 / North China 1 (Qingdao)
|
|
\[rs] \[dq]oss-cn-qingdao.aliyuncs.com\[dq]
|
|
[snip]
|
|
endpoint> 1
|
|
Canned ACL used when creating buckets and storing or copying objects.
|
|
|
|
Note that this ACL is applied when server-side copying objects as S3
|
|
doesn\[aq]t copy the ACL from the source but rather writes a fresh one.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Owner gets FULL_CONTROL. No one else has access rights (default).
|
|
\[rs] \[dq]private\[dq]
|
|
2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
|
|
\[rs] \[dq]public-read\[dq]
|
|
/ Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
|
|
[snip]
|
|
acl> 1
|
|
The storage class to use when storing new objects in OSS.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Default
|
|
\[rs] \[dq]\[dq]
|
|
2 / Standard storage class
|
|
\[rs] \[dq]STANDARD\[dq]
|
|
3 / Archive storage mode.
|
|
\[rs] \[dq]GLACIER\[dq]
|
|
4 / Infrequent access storage mode.
|
|
\[rs] \[dq]STANDARD_IA\[dq]
|
|
storage_class> 1
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No
|
|
y/n> n
|
|
Remote config
|
|
--------------------
|
|
[oss]
|
|
type = s3
|
|
provider = Alibaba
|
|
env_auth = false
|
|
access_key_id = accesskeyid
|
|
secret_access_key = secretaccesskey
|
|
endpoint = oss-cn-hangzhou.aliyuncs.com
|
|
acl = private
|
|
storage_class = Standard
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.SS Tencent COS
|
|
.PP
|
|
Tencent Cloud Object Storage
|
|
(COS) (https://intl.cloud.tencent.com/product/cos) is a distributed
|
|
storage service offered by Tencent Cloud for unstructured data.
|
|
It is secure, stable, massive, convenient, low-delay and low-cost.
|
|
.PP
|
|
To configure access to Tencent COS, follow the steps below:
|
|
.IP "1." 3
|
|
Run \f[C]rclone config\f[R] and select \f[C]n\f[R] for a new remote.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
\f[R]
|
|
.fi
|
|
.IP "2." 3
|
|
Give the name of the configuration.
|
|
For example, name it \[aq]cos\[aq].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
name> cos
|
|
\f[R]
|
|
.fi
|
|
.IP "3." 3
|
|
Select \f[C]s3\f[R] storage.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Choose a number from below, or type in your own value
|
|
1 / 1Fichier
|
|
\[rs] \[dq]fichier\[dq]
|
|
2 / Alias for an existing remote
|
|
\[rs] \[dq]alias\[dq]
|
|
3 / Amazon Drive
|
|
\[rs] \[dq]amazon cloud drive\[dq]
|
|
4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, and Tencent COS
|
|
\[rs] \[dq]s3\[dq]
|
|
[snip]
|
|
Storage> s3
|
|
\f[R]
|
|
.fi
|
|
.IP "4." 3
|
|
Select \f[C]TencentCOS\f[R] provider.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Choose a number from below, or type in your own value
|
|
1 / Amazon Web Services (AWS) S3
|
|
\[rs] \[dq]AWS\[dq]
|
|
[snip]
|
|
11 / Tencent Cloud Object Storage (COS)
|
|
\[rs] \[dq]TencentCOS\[dq]
|
|
[snip]
|
|
provider> TencentCOS
|
|
\f[R]
|
|
.fi
|
|
.IP "5." 3
|
|
Enter your SecretId and SecretKey of Tencent Cloud.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
|
|
Only applies if access_key_id and secret_access_key is blank.
|
|
Enter a boolean value (true or false). Press Enter for the default (\[dq]false\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Enter AWS credentials in the next step
|
|
\[rs] \[dq]false\[dq]
|
|
2 / Get AWS credentials from the environment (env vars or IAM)
|
|
\[rs] \[dq]true\[dq]
|
|
env_auth> 1
|
|
AWS Access Key ID.
|
|
Leave blank for anonymous access or runtime credentials.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
access_key_id> AKIDxxxxxxxxxx
|
|
AWS Secret Access Key (password)
|
|
Leave blank for anonymous access or runtime credentials.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
secret_access_key> xxxxxxxxxxx
|
|
\f[R]
|
|
.fi
|
|
.IP "6." 3
|
|
Select endpoint for Tencent COS.
|
|
This is the standard endpoint for different region.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
1 / Beijing Region.
|
|
\[rs] \[dq]cos.ap-beijing.myqcloud.com\[dq]
|
|
2 / Nanjing Region.
|
|
\[rs] \[dq]cos.ap-nanjing.myqcloud.com\[dq]
|
|
3 / Shanghai Region.
|
|
\[rs] \[dq]cos.ap-shanghai.myqcloud.com\[dq]
|
|
4 / Guangzhou Region.
|
|
\[rs] \[dq]cos.ap-guangzhou.myqcloud.com\[dq]
|
|
[snip]
|
|
endpoint> 4
|
|
\f[R]
|
|
.fi
|
|
.IP "7." 3
|
|
Choose acl and storage class.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Note that this ACL is applied when server-side copying objects as S3
|
|
doesn\[aq]t copy the ACL from the source but rather writes a fresh one.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Owner gets Full_CONTROL. No one else has access rights (default).
|
|
\[rs] \[dq]default\[dq]
|
|
[snip]
|
|
acl> 1
|
|
The storage class to use when storing new objects in Tencent COS.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Default
|
|
\[rs] \[dq]\[dq]
|
|
[snip]
|
|
storage_class> 1
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
Remote config
|
|
--------------------
|
|
[cos]
|
|
type = s3
|
|
provider = TencentCOS
|
|
env_auth = false
|
|
access_key_id = xxx
|
|
secret_access_key = xxx
|
|
endpoint = cos.ap-guangzhou.myqcloud.com
|
|
acl = default
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
Current remotes:
|
|
|
|
Name Type
|
|
==== ====
|
|
cos s3
|
|
\f[R]
|
|
.fi
|
|
.SS Netease NOS
|
|
.PP
|
|
For Netease NOS configure as per the configurator
|
|
\f[C]rclone config\f[R] setting the provider \f[C]Netease\f[R].
|
|
This will automatically set \f[C]force_path_style = false\f[R] which is
|
|
necessary for it to run properly.
|
|
.SS Storj
|
|
.PP
|
|
Storj is a decentralized cloud storage which can be used through its
|
|
native protocol or an S3 compatible gateway.
|
|
.PP
|
|
The S3 compatible gateway is configured using \f[C]rclone config\f[R]
|
|
with a type of \f[C]s3\f[R] and with a provider name of \f[C]Storj\f[R].
|
|
Here is an example run of the configurator.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Type of storage to configure.
|
|
Storage> s3
|
|
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
|
|
Only applies if access_key_id and secret_access_key is blank.
|
|
Choose a number from below, or type in your own boolean value (true or false).
|
|
Press Enter for the default (false).
|
|
1 / Enter AWS credentials in the next step.
|
|
\[rs] (false)
|
|
2 / Get AWS credentials from the environment (env vars or IAM).
|
|
\[rs] (true)
|
|
env_auth> 1
|
|
Option access_key_id.
|
|
AWS Access Key ID.
|
|
Leave blank for anonymous access or runtime credentials.
|
|
Enter a value. Press Enter to leave empty.
|
|
access_key_id> XXXX (as shown when creating the access grant)
|
|
Option secret_access_key.
|
|
AWS Secret Access Key (password).
|
|
Leave blank for anonymous access or runtime credentials.
|
|
Enter a value. Press Enter to leave empty.
|
|
secret_access_key> XXXX (as shown when creating the access grant)
|
|
Option endpoint.
|
|
Endpoint of the Shared Gateway.
|
|
Choose a number from below, or type in your own value.
|
|
Press Enter to leave empty.
|
|
1 / EU1 Shared Gateway
|
|
\[rs] (gateway.eu1.storjshare.io)
|
|
2 / US1 Shared Gateway
|
|
\[rs] (gateway.us1.storjshare.io)
|
|
3 / Asia-Pacific Shared Gateway
|
|
\[rs] (gateway.ap1.storjshare.io)
|
|
endpoint> 1 (as shown when creating the access grant)
|
|
Edit advanced config?
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that s3 credentials are generated when you create an access
|
|
grant (https://docs.storj.io/dcs/api-reference/s3-compatible-gateway#usage).
|
|
.SS Backend quirks
|
|
.IP \[bu] 2
|
|
\f[C]--chunk-size\f[R] is forced to be 64 MiB or greater.
|
|
This will use more memory than the default of 5 MiB.
|
|
.IP \[bu] 2
|
|
Server side copy is disabled as it isn\[aq]t currently supported in the
|
|
gateway.
|
|
.IP \[bu] 2
|
|
GetTier and SetTier are not supported.
|
|
.SS Backend bugs
|
|
.PP
|
|
Due to issue #39 (https://github.com/storj/gateway-mt/issues/39)
|
|
uploading multipart files via the S3 gateway causes them to lose their
|
|
metadata.
|
|
For rclone\[aq]s purpose this means that the modification time is not
|
|
stored, nor is any MD5SUM (if one is available from the source).
|
|
.PP
|
|
This has the following consequences:
|
|
.IP \[bu] 2
|
|
Using \f[C]rclone rcat\f[R] will fail as the medatada doesn\[aq]t match
|
|
after upload
|
|
.IP \[bu] 2
|
|
Uploading files with \f[C]rclone mount\f[R] will fail for the same
|
|
reason
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This can worked around by using \f[C]--vfs-cache-mode writes\f[R] or
|
|
\f[C]--vfs-cache-mode full\f[R] or setting \f[C]--s3-upload-cutoff\f[R]
|
|
large
|
|
.RE
|
|
.IP \[bu] 2
|
|
Files uploaded via a multipart upload won\[aq]t have their modtimes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This will mean that \f[C]rclone sync\f[R] will likely keep trying to
|
|
upload files bigger than \f[C]--s3-upload-cutoff\f[R]
|
|
.IP \[bu] 2
|
|
This can be worked around with \f[C]--checksum\f[R] or
|
|
\f[C]--size-only\f[R] or setting \f[C]--s3-upload-cutoff\f[R] large
|
|
.IP \[bu] 2
|
|
The maximum value for \f[C]--s3-upload-cutoff\f[R] is 5GiB though
|
|
.RE
|
|
.PP
|
|
One general purpose workaround is to set
|
|
\f[C]--s3-upload-cutoff 5G\f[R].
|
|
This means that rclone will upload files smaller than 5GiB as single
|
|
parts.
|
|
Note that this can be set in the config file with
|
|
\f[C]upload_cutoff = 5G\f[R] or configured in the advanced settings.
|
|
If you regularly transfer files larger than 5G then using
|
|
\f[C]--checksum\f[R] or \f[C]--size-only\f[R] in \f[C]rclone sync\f[R]
|
|
is the recommended workaround.
|
|
.SS Comparison with the native protocol
|
|
.PP
|
|
Use the the native protocol to take advantage of client-side encryption
|
|
as well as to achieve the best possible download performance.
|
|
Uploads will be erasure-coded locally, thus a 1gb upload will result in
|
|
2.68gb of data being uploaded to storage nodes across the network.
|
|
.PP
|
|
Use this backend and the S3 compatible Hosted Gateway to increase upload
|
|
performance and reduce the load on your systems and network.
|
|
Uploads will be encrypted and erasure-coded server-side, thus a 1GB
|
|
upload will result in only in 1GB of data being uploaded to storage
|
|
nodes across the network.
|
|
.PP
|
|
For more detailed comparison please check the documentation of the storj
|
|
backend.
|
|
.SS Limitations
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the S3 backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SH Backblaze B2
|
|
.PP
|
|
B2 is Backblaze\[aq]s cloud storage
|
|
system (https://www.backblaze.com/b2/).
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[R] (or \f[C]remote:\f[R] for
|
|
the \f[C]lsd\f[R] command.) You may put subdirectories in too, e.g.
|
|
\f[C]remote:bucket/path/to/dir\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of making a b2 configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
To authenticate you will either need your Account ID (a short hex
|
|
number) and Master Application Key (a long hex number) OR an Application
|
|
Key, which is the recommended method.
|
|
See below for further details on generating and using an Application
|
|
Key.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
q) Quit config
|
|
n/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Backblaze B2
|
|
\[rs] \[dq]b2\[dq]
|
|
[snip]
|
|
Storage> b2
|
|
Account ID or Application Key ID
|
|
account> 123456789abc
|
|
Application Key
|
|
key> 0123456789abcdef0123456789abcdef0123456789
|
|
Endpoint for the service - leave blank normally.
|
|
endpoint>
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
account = 123456789abc
|
|
key = 0123456789abcdef0123456789abcdef0123456789
|
|
endpoint =
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[R] and can now be used like this
|
|
.PP
|
|
See all buckets
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Create a new bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.SS Application Keys
|
|
.PP
|
|
B2 supports multiple Application Keys for different access permission to
|
|
B2 Buckets (https://www.backblaze.com/b2/docs/application_keys.html).
|
|
.PP
|
|
You can use these with rclone too; you will need to use rclone version
|
|
1.43 or later.
|
|
.PP
|
|
Follow Backblaze\[aq]s docs to create an Application Key with the
|
|
required permission and add the \f[C]applicationKeyId\f[R] as the
|
|
\f[C]account\f[R] and the \f[C]Application Key\f[R] itself as the
|
|
\f[C]key\f[R].
|
|
.PP
|
|
Note that you must put the \f[I]applicationKeyId\f[R] as the
|
|
\f[C]account\f[R] \[en] you can\[aq]t use the master Account ID.
|
|
If you try then B2 will return 401 errors.
|
|
.SS --fast-list
|
|
.PP
|
|
This remote supports \f[C]--fast-list\f[R] which allows you to use fewer
|
|
transactions in exchange for more memory.
|
|
See the rclone docs (https://rclone.org/docs/#fast-list) for more
|
|
details.
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X-Bz-Info-src_last_modified_millis\f[R] as milliseconds since
|
|
1970-01-01 in the Backblaze standard.
|
|
Other tools should be able to use this as a modified time.
|
|
.PP
|
|
Modified times are used in syncing and are fully supported.
|
|
Note that if a modification time needs to be updated on an object then
|
|
it will create a new version of the object.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.PP
|
|
Note that in 2020-05 Backblaze started allowing \ characters in file
|
|
names.
|
|
Rclone hasn\[aq]t changed its encoding as this could cause syncs to
|
|
re-transfer files.
|
|
If you want rclone not to replace \ then see the \f[C]--b2-encoding\f[R]
|
|
flag below and remove the \f[C]BackSlash\f[R] from the string.
|
|
This can be set in the config.
|
|
.SS SHA1 checksums
|
|
.PP
|
|
The SHA1 checksums of the files are checked on upload and download and
|
|
will be used in the syncing process.
|
|
.PP
|
|
Large files (bigger than the limit in \f[C]--b2-upload-cutoff\f[R])
|
|
which are uploaded in chunks will store their SHA1 on the object as
|
|
\f[C]X-Bz-Info-large_file_sha1\f[R] as recommended by Backblaze.
|
|
.PP
|
|
For a large file to be uploaded with an SHA1 checksum, the source needs
|
|
to support SHA1 checksums.
|
|
The local disk supports SHA1 checksums so large file transfers from
|
|
local disk will have an SHA1.
|
|
See the overview (https://rclone.org/overview/#features) for exactly
|
|
which remotes support SHA1.
|
|
.PP
|
|
Sources which don\[aq]t support SHA1, in particular \f[C]crypt\f[R] will
|
|
upload large files without SHA1 checksums.
|
|
This may be fixed in the future (see
|
|
#1767 (https://github.com/rclone/rclone/issues/1767)).
|
|
.PP
|
|
Files sizes below \f[C]--b2-upload-cutoff\f[R] will always have an SHA1
|
|
regardless of the source.
|
|
.SS Transfers
|
|
.PP
|
|
Backblaze recommends that you do lots of transfers simultaneously for
|
|
maximum speed.
|
|
In tests from my SSD equipped laptop the optimum setting is about
|
|
\f[C]--transfers 32\f[R] though higher numbers may be used for a slight
|
|
speed improvement.
|
|
The optimum number for you may vary depending on your hardware, how big
|
|
the files are, how much you want to load your computer, etc.
|
|
The default of \f[C]--transfers 4\f[R] is definitely too low for
|
|
Backblaze B2 though.
|
|
.PP
|
|
Note that uploading big files (bigger than 200 MiB by default) will use
|
|
a 96 MiB RAM buffer by default.
|
|
There can be at most \f[C]--transfers\f[R] of these in use at any
|
|
moment, so this sets the upper limit on the memory used.
|
|
.SS Versions
|
|
.PP
|
|
When rclone uploads a new version of a file it creates a new version of
|
|
it (https://www.backblaze.com/b2/docs/file_versions.html).
|
|
Likewise when you delete a file, the old version will be marked hidden
|
|
and still be available.
|
|
Conversely, you may opt in to a \[dq]hard delete\[dq] of files with the
|
|
\f[C]--b2-hard-delete\f[R] flag which would permanently remove the file
|
|
instead of hiding it.
|
|
.PP
|
|
Old versions of files, where available, are visible using the
|
|
\f[C]--b2-versions\f[R] flag.
|
|
.PP
|
|
If you wish to remove all the old versions then you can use the
|
|
\f[C]rclone cleanup remote:bucket\f[R] command which will delete all the
|
|
old versions of files, leaving the current ones intact.
|
|
You can also supply a path and only old versions under that path will be
|
|
deleted, e.g.
|
|
\f[C]rclone cleanup remote:bucket/path/to/stuff\f[R].
|
|
.PP
|
|
Note that \f[C]cleanup\f[R] will remove partially uploaded files from
|
|
the bucket if they are more than a day old.
|
|
.PP
|
|
When you \f[C]purge\f[R] a bucket, the current and the old versions will
|
|
be deleted then the bucket will be deleted.
|
|
.PP
|
|
However \f[C]delete\f[R] will cause the current versions of the files to
|
|
become hidden old versions.
|
|
.PP
|
|
Here is a session showing the listing and retrieval of an old version
|
|
followed by a \f[C]cleanup\f[R] of the old versions.
|
|
.PP
|
|
Show current version and all the versions with \f[C]--b2-versions\f[R]
|
|
flag.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -q ls b2:cleanup-test
|
|
9 one.txt
|
|
|
|
$ rclone -q --b2-versions ls b2:cleanup-test
|
|
9 one.txt
|
|
8 one-v2016-07-04-141032-000.txt
|
|
16 one-v2016-07-04-141003-000.txt
|
|
15 one-v2016-07-02-155621-000.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Retrieve an old version
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
|
|
|
|
$ ls -l /tmp/one-v2016-07-04-141003-000.txt
|
|
-rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Clean up all the old versions and show that they\[aq]ve gone.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -q cleanup b2:cleanup-test
|
|
|
|
$ rclone -q ls b2:cleanup-test
|
|
9 one.txt
|
|
|
|
$ rclone -q --b2-versions ls b2:cleanup-test
|
|
9 one.txt
|
|
\f[R]
|
|
.fi
|
|
.SS Data usage
|
|
.PP
|
|
It is useful to know how many requests are sent to the server in
|
|
different scenarios.
|
|
.PP
|
|
All copy commands send the following 4 requests:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/b2api/v1/b2_authorize_account
|
|
/b2api/v1/b2_create_bucket
|
|
/b2api/v1/b2_list_buckets
|
|
/b2api/v1/b2_list_file_names
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The \f[C]b2_list_file_names\f[R] request will be sent once for every 1k
|
|
files in the remote path, providing the checksum and modification time
|
|
of the listed files.
|
|
As of version 1.33 issue
|
|
#818 (https://github.com/rclone/rclone/issues/818) causes extra requests
|
|
to be sent when using B2 with Crypt.
|
|
When a copy operation does not require any files to be uploaded, no more
|
|
requests will be sent.
|
|
.PP
|
|
Uploading files that do not require chunking, will send 2 requests per
|
|
file upload:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/b2api/v1/b2_get_upload_url
|
|
/b2api/v1/b2_upload_file/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Uploading files requiring chunking, will send 2 requests (one each to
|
|
start and finish the upload) and another 2 requests for each chunk:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/b2api/v1/b2_start_large_file
|
|
/b2api/v1/b2_get_upload_part_url
|
|
/b2api/v1/b2_upload_part/
|
|
/b2api/v1/b2_finish_large_file
|
|
\f[R]
|
|
.fi
|
|
.SS Versions
|
|
.PP
|
|
Versions can be viewed with the \f[C]--b2-versions\f[R] flag.
|
|
When it is set rclone will show and act on older versions of files.
|
|
For example
|
|
.PP
|
|
Listing without \f[C]--b2-versions\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -q ls b2:cleanup-test
|
|
9 one.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -q --b2-versions ls b2:cleanup-test
|
|
9 one.txt
|
|
8 one-v2016-07-04-141032-000.txt
|
|
16 one-v2016-07-04-141003-000.txt
|
|
15 one-v2016-07-02-155621-000.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Showing that the current version is unchanged but older versions can be
|
|
seen.
|
|
These have the UTC date that they were uploaded to the server to the
|
|
nearest millisecond appended to them.
|
|
.PP
|
|
Note that when using \f[C]--b2-versions\f[R] no file write operations
|
|
are permitted, so you can\[aq]t upload files or delete them.
|
|
.SS B2 and rclone link
|
|
.PP
|
|
Rclone supports generating file share links for private B2 buckets.
|
|
They can either be for a file for example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\&./rclone link B2:bucket/path/to/file.txt
|
|
https://f002.backblazeb2.com/file/bucket/path/to/file.txt?Authorization=xxxxxxxx
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
or if run on a directory you will get:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\&./rclone link B2:bucket/path
|
|
https://f002.backblazeb2.com/file/bucket/path?Authorization=xxxxxxxx
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
you can then use the authorization token (the part of the url from the
|
|
\f[C]?Authorization=\f[R] on) on any file path under that directory.
|
|
For example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
https://f002.backblazeb2.com/file/bucket/path/to/file1?Authorization=xxxxxxxx
|
|
https://f002.backblazeb2.com/file/bucket/path/file2?Authorization=xxxxxxxx
|
|
https://f002.backblazeb2.com/file/bucket/path/folder/file3?Authorization=xxxxxxxx
|
|
\f[R]
|
|
.fi
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to b2 (Backblaze B2).
|
|
.SS --b2-account
|
|
.PP
|
|
Account ID or Application Key ID.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: account
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_ACCOUNT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --b2-key
|
|
.PP
|
|
Application Key.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: key
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_KEY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --b2-hard-delete
|
|
.PP
|
|
Permanently delete files on remote removal, otherwise hide files.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: hard_delete
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_HARD_DELETE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to b2 (Backblaze B2).
|
|
.SS --b2-endpoint
|
|
.PP
|
|
Endpoint for the service.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_ENDPOINT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --b2-test-mode
|
|
.PP
|
|
A flag string for X-Bz-Test-Mode header for debugging.
|
|
.PP
|
|
This is for debugging purposes only.
|
|
Setting it to one of the strings below will cause b2 to return specific
|
|
errors:
|
|
.IP \[bu] 2
|
|
\[dq]fail_some_uploads\[dq]
|
|
.IP \[bu] 2
|
|
\[dq]expire_some_account_authorization_tokens\[dq]
|
|
.IP \[bu] 2
|
|
\[dq]force_cap_exceeded\[dq]
|
|
.PP
|
|
These will be set in the \[dq]X-Bz-Test-Mode\[dq] header which is
|
|
documented in the b2 integrations
|
|
checklist (https://www.backblaze.com/b2/docs/integration_checklist.html).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: test_mode
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_TEST_MODE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --b2-versions
|
|
.PP
|
|
Include old versions in directory listings.
|
|
.PP
|
|
Note that when using this no file write operations are permitted, so you
|
|
can\[aq]t upload files or delete them.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: versions
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_VERSIONS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --b2-upload-cutoff
|
|
.PP
|
|
Cutoff for switching to chunked upload.
|
|
.PP
|
|
Files above this size will be uploaded in chunks of
|
|
\[dq]--b2-chunk-size\[dq].
|
|
.PP
|
|
This value should be set no larger than 4.657 GiB (== 5 GB).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_cutoff
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_UPLOAD_CUTOFF
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 200Mi
|
|
.SS --b2-copy-cutoff
|
|
.PP
|
|
Cutoff for switching to multipart copy.
|
|
.PP
|
|
Any files larger than this that need to be server-side copied will be
|
|
copied in chunks of this size.
|
|
.PP
|
|
The minimum is 0 and the maximum is 4.6 GiB.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: copy_cutoff
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_COPY_CUTOFF
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 4Gi
|
|
.SS --b2-chunk-size
|
|
.PP
|
|
Upload chunk size.
|
|
.PP
|
|
When uploading large files, chunk the file into this size.
|
|
.PP
|
|
Must fit in memory.
|
|
These chunks are buffered in memory and there might a maximum of
|
|
\[dq]--transfers\[dq] chunks in progress at once.
|
|
.PP
|
|
5,000,000 Bytes is the minimum size.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 96Mi
|
|
.SS --b2-disable-checksum
|
|
.PP
|
|
Disable checksums for large (> upload cutoff) files.
|
|
.PP
|
|
Normally rclone will calculate the SHA1 checksum of the input before
|
|
uploading it so it can add it to metadata on the object.
|
|
This is great for data integrity checking but can cause long delays for
|
|
large files to start uploading.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_checksum
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_DISABLE_CHECKSUM
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --b2-download-url
|
|
.PP
|
|
Custom endpoint for downloads.
|
|
.PP
|
|
This is usually set to a Cloudflare CDN URL as Backblaze offers free
|
|
egress for data downloaded through the Cloudflare network.
|
|
Rclone works with private buckets by sending an \[dq]Authorization\[dq]
|
|
header.
|
|
If the custom endpoint rewrites the requests for authentication, e.g.,
|
|
in Cloudflare Workers, this header needs to be handled properly.
|
|
Leave blank if you want to use the endpoint provided by Backblaze.
|
|
.PP
|
|
The URL provided here SHOULD have the protocol and SHOULD NOT have a
|
|
trailing slash or specify the /file/bucket subpath as rclone will
|
|
request files with \[dq]{download_url}/file/{bucket_name}/{path}\[dq].
|
|
.PP
|
|
Example: > https://mysubdomain.mydomain.tld (No trailing \[dq]/\[dq],
|
|
\[dq]file\[dq] or \[dq]bucket\[dq])
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: download_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_DOWNLOAD_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --b2-download-auth-duration
|
|
.PP
|
|
Time before the authorization token will expire in s or suffix
|
|
ms|s|m|h|d.
|
|
.PP
|
|
The duration before the download authorization token will expire.
|
|
The minimum value is 1 second.
|
|
The maximum value is one week.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: download_auth_duration
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_DOWNLOAD_AUTH_DURATION
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 1w
|
|
.SS --b2-memory-pool-flush-time
|
|
.PP
|
|
How often internal memory buffer pools will be flushed.
|
|
Uploads which requires additional buffers (f.e multipart) will use
|
|
memory pool for allocations.
|
|
This option controls how often unused buffers will be removed from the
|
|
pool.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: memory_pool_flush_time
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_MEMORY_POOL_FLUSH_TIME
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 1m0s
|
|
.SS --b2-memory-pool-use-mmap
|
|
.PP
|
|
Whether to use mmap buffers in internal memory pool.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: memory_pool_use_mmap
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_MEMORY_POOL_USE_MMAP
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --b2-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_B2_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the B2 backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SH Box
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.PP
|
|
The initial setup for Box involves getting a token from Box which you
|
|
can do either in your browser, or with a config.json downloaded from Box
|
|
to use JWT authentication.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Box
|
|
\[rs] \[dq]box\[dq]
|
|
[snip]
|
|
Storage> box
|
|
Box App Client Id - leave blank normally.
|
|
client_id>
|
|
Box App Client Secret - leave blank normally.
|
|
client_secret>
|
|
Box App config.json location
|
|
Leave blank normally.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
box_config_file>
|
|
Box App Primary Access Token
|
|
Leave blank normally.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
access_token>
|
|
|
|
Enter a string value. Press Enter for the default (\[dq]user\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Rclone should act on behalf of a user
|
|
\[rs] \[dq]user\[dq]
|
|
2 / Rclone should act on behalf of a service account
|
|
\[rs] \[dq]enterprise\[dq]
|
|
box_sub_type>
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
--------------------
|
|
[remote]
|
|
client_id =
|
|
client_secret =
|
|
token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]XXX\[dq]}
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Box.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your Box
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your Box
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Box directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Using rclone with an Enterprise account with SSO
|
|
.PP
|
|
If you have an \[dq]Enterprise\[dq] account type with Box with single
|
|
sign on (SSO), you need to create a password to use Box with rclone.
|
|
This can be done at your Enterprise Box account by going to Settings,
|
|
\[dq]Account\[dq] Tab, and then set the password in the
|
|
\[dq]Authentication\[dq] field.
|
|
.PP
|
|
Once you have done this, you can setup your Enterprise Box account using
|
|
the same procedure detailed above in the, using the password you have
|
|
just set.
|
|
.SS Invalid refresh token
|
|
.PP
|
|
According to the box
|
|
docs (https://developer.box.com/v2.0/docs/oauth-20#section-6-using-the-access-and-refresh-tokens):
|
|
.RS
|
|
.PP
|
|
Each refresh_token is valid for one use in 60 days.
|
|
.RE
|
|
.PP
|
|
This means that if you
|
|
.IP \[bu] 2
|
|
Don\[aq]t use the box remote for 60 days
|
|
.IP \[bu] 2
|
|
Copy the config file with a box refresh token in and use it in two
|
|
places
|
|
.IP \[bu] 2
|
|
Get an error on a token refresh
|
|
.PP
|
|
then rclone will return an error which includes the text
|
|
\f[C]Invalid refresh token\f[R].
|
|
.PP
|
|
To fix this you will need to use oauth2 again to update the refresh
|
|
token.
|
|
You can use the methods in the remote setup
|
|
docs (https://rclone.org/remote_setup/), bearing in mind that if you use
|
|
the copy the config file method, you should not use that remote on the
|
|
computer you did the authentication on.
|
|
.PP
|
|
Here is how to do it.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone config
|
|
Current remotes:
|
|
|
|
Name Type
|
|
==== ====
|
|
remote box
|
|
|
|
e) Edit existing remote
|
|
n) New remote
|
|
d) Delete remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/r/c/s/q> e
|
|
Choose a number from below, or type in an existing value
|
|
1 > remote
|
|
remote> remote
|
|
--------------------
|
|
[remote]
|
|
type = box
|
|
token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2017-07-08T23:40:08.059167677+01:00\[dq]}
|
|
--------------------
|
|
Edit remote
|
|
Value \[dq]client_id\[dq] = \[dq]\[dq]
|
|
Edit? (y/n)>
|
|
y) Yes
|
|
n) No
|
|
y/n> n
|
|
Value \[dq]client_secret\[dq] = \[dq]\[dq]
|
|
Edit? (y/n)>
|
|
y) Yes
|
|
n) No
|
|
y/n> n
|
|
Remote config
|
|
Already have a token - refresh?
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
--------------------
|
|
[remote]
|
|
type = box
|
|
token = {\[dq]access_token\[dq]:\[dq]YYY\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]YYY\[dq],\[dq]expiry\[dq]:\[dq]2017-07-23T12:22:29.259137901+01:00\[dq]}
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
Box allows modification times to be set on objects accurate to 1 second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
.PP
|
|
Box supports SHA1 type hashes, so you can use the \f[C]--checksum\f[R]
|
|
flag.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
File names can also not end with the following characters.
|
|
These only get replaced if they are the last character in the name:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
SP
|
|
T}@T{
|
|
0x20
|
|
T}@T{
|
|
\[u2420]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Transfers
|
|
.PP
|
|
For files above 50 MiB rclone will use a chunked transfer.
|
|
Rclone will upload up to \f[C]--transfers\f[R] chunks at the same time
|
|
(shared among all the multipart uploads).
|
|
Chunks are buffered in memory and are normally 8 MiB so increasing
|
|
\f[C]--transfers\f[R] will increase memory use.
|
|
.SS Deleting files
|
|
.PP
|
|
Depending on the enterprise settings for your user, the item will either
|
|
be actually deleted from Box or moved to the trash.
|
|
.PP
|
|
Emptying the trash is supported via the rclone however cleanup command
|
|
however this deletes every trashed file and folder individually so it
|
|
may take a very long time.
|
|
Emptying the trash via the WebUI does not have this limitation so it is
|
|
advised to empty the trash via the WebUI.
|
|
.SS Root folder ID
|
|
.PP
|
|
You can set the \f[C]root_folder_id\f[R] for rclone.
|
|
This is the directory (identified by its \f[C]Folder ID\f[R]) that
|
|
rclone considers to be the root of your Box drive.
|
|
.PP
|
|
Normally you will leave this blank and rclone will determine the correct
|
|
root to use itself.
|
|
.PP
|
|
However you can set this to restrict rclone to a specific folder
|
|
hierarchy.
|
|
.PP
|
|
In order to do this you will have to find the \f[C]Folder ID\f[R] of the
|
|
directory you wish rclone to display.
|
|
This will be the last segment of the URL when you open the relevant
|
|
folder in the Box web interface.
|
|
.PP
|
|
So if the folder you want rclone to use has a URL which looks like
|
|
\f[C]https://app.box.com/folder/11xxxxxxxxx8\f[R] in the browser, then
|
|
you use \f[C]11xxxxxxxxx8\f[R] as the \f[C]root_folder_id\f[R] in the
|
|
config.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to box (Box).
|
|
.SS --box-client-id
|
|
.PP
|
|
OAuth Client Id.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --box-client-secret
|
|
.PP
|
|
OAuth Client Secret.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_CLIENT_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --box-box-config-file
|
|
.PP
|
|
Box App config.json location
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Leading \f[C]\[ti]\f[R] will be expanded in the file name as will
|
|
environment variables such as \f[C]${RCLONE_CONFIG_DIR}\f[R].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: box_config_file
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_BOX_CONFIG_FILE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --box-access-token
|
|
.PP
|
|
Box App Primary Access Token
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: access_token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_ACCESS_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --box-box-sub-type
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: box_sub_type
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_BOX_SUB_TYPE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]user\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]user\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rclone should act on behalf of a user.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]enterprise\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rclone should act on behalf of a service account.
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to box (Box).
|
|
.SS --box-token
|
|
.PP
|
|
OAuth Access Token as a JSON blob.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --box-auth-url
|
|
.PP
|
|
Auth server URL.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_AUTH_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --box-token-url
|
|
.PP
|
|
Token server url.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_TOKEN_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --box-root-folder-id
|
|
.PP
|
|
Fill in for rclone to use a non root folder as its starting point.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: root_folder_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_ROOT_FOLDER_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]0\[dq]
|
|
.SS --box-upload-cutoff
|
|
.PP
|
|
Cutoff for switching to multipart upload (>= 50 MiB).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_cutoff
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_UPLOAD_CUTOFF
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 50Mi
|
|
.SS --box-commit-retries
|
|
.PP
|
|
Max number of times to try committing a multipart file.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: commit_retries
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_COMMIT_RETRIES
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 100
|
|
.SS --box-list-chunk
|
|
.PP
|
|
Size of listing chunk 1-1000.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: list_chunk
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_LIST_CHUNK
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 1000
|
|
.SS --box-owned-by
|
|
.PP
|
|
Only show items owned by the login (email address) passed in.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: owned_by
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_OWNED_BY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --box-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_BOX_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
Note that Box is case insensitive so you can\[aq]t have a file called
|
|
\[dq]Hello.doc\[dq] and one called \[dq]hello.doc\[dq].
|
|
.PP
|
|
Box file names can\[aq]t have the \f[C]\[rs]\f[R] character in.
|
|
rclone maps this to and from an identical looking unicode equivalent
|
|
\f[C]\[uFF3C]\f[R] (U+FF3C Fullwidth Reverse Solidus).
|
|
.PP
|
|
Box only supports filenames up to 255 characters in length.
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the Box backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SH Cache (DEPRECATED)
|
|
.PP
|
|
The \f[C]cache\f[R] remote wraps another existing remote and stores file
|
|
structure and its data for long running tasks like
|
|
\f[C]rclone mount\f[R].
|
|
.SS Status
|
|
.PP
|
|
The cache backend code is working but it currently doesn\[aq]t have a
|
|
maintainer so there are outstanding
|
|
bugs (https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug+label%3A%22Remote%3A+Cache%22)
|
|
which aren\[aq]t getting fixed.
|
|
.PP
|
|
The cache backend is due to be phased out in favour of the VFS caching
|
|
layer eventually which is more tightly integrated into rclone.
|
|
.PP
|
|
Until this happens we recommend only using the cache backend if you find
|
|
you can\[aq]t work without it.
|
|
There are many docs online describing the use of the cache backend to
|
|
minimize API hits and by-and-large these are out of date and the cache
|
|
backend isn\[aq]t needed in those scenarios any more.
|
|
.SS Configuration
|
|
.PP
|
|
To get started you just need to have an existing remote which can be
|
|
configured with \f[C]cache\f[R].
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]test-cache\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/r/c/s/q> n
|
|
name> test-cache
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Cache a remote
|
|
\[rs] \[dq]cache\[dq]
|
|
[snip]
|
|
Storage> cache
|
|
Remote to cache.
|
|
Normally should contain a \[aq]:\[aq] and a path, e.g. \[dq]myremote:path/to/dir\[dq],
|
|
\[dq]myremote:bucket\[dq] or maybe \[dq]myremote:\[dq] (not recommended).
|
|
remote> local:/test
|
|
Optional: The URL of the Plex server
|
|
plex_url> http://127.0.0.1:32400
|
|
Optional: The username of the Plex user
|
|
plex_username> dummyusername
|
|
Optional: The password of the Plex user
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
n) No leave this optional password blank
|
|
y/g/n> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
The size of a chunk. Lower value good for slow connections but can affect seamless reading.
|
|
Default: 5M
|
|
Choose a number from below, or type in your own value
|
|
1 / 1 MiB
|
|
\[rs] \[dq]1M\[dq]
|
|
2 / 5 MiB
|
|
\[rs] \[dq]5M\[dq]
|
|
3 / 10 MiB
|
|
\[rs] \[dq]10M\[dq]
|
|
chunk_size> 2
|
|
How much time should object info (file size, file hashes, etc.) be stored in cache. Use a very high value if you don\[aq]t plan on changing the source FS from outside the cache.
|
|
Accepted units are: \[dq]s\[dq], \[dq]m\[dq], \[dq]h\[dq].
|
|
Default: 5m
|
|
Choose a number from below, or type in your own value
|
|
1 / 1 hour
|
|
\[rs] \[dq]1h\[dq]
|
|
2 / 24 hours
|
|
\[rs] \[dq]24h\[dq]
|
|
3 / 24 hours
|
|
\[rs] \[dq]48h\[dq]
|
|
info_age> 2
|
|
The maximum size of stored chunks. When the storage grows beyond this size, the oldest chunks will be deleted.
|
|
Default: 10G
|
|
Choose a number from below, or type in your own value
|
|
1 / 500 MiB
|
|
\[rs] \[dq]500M\[dq]
|
|
2 / 1 GiB
|
|
\[rs] \[dq]1G\[dq]
|
|
3 / 10 GiB
|
|
\[rs] \[dq]10G\[dq]
|
|
chunk_total_size> 3
|
|
Remote config
|
|
--------------------
|
|
[test-cache]
|
|
remote = local:/test
|
|
plex_url = http://127.0.0.1:32400
|
|
plex_username = dummyusername
|
|
plex_password = *** ENCRYPTED ***
|
|
chunk_size = 5M
|
|
info_age = 48h
|
|
chunk_total_size = 10G
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can then use it like this,
|
|
.PP
|
|
List directories in top level of your drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd test-cache:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls test-cache:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To start a cached mount
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mount --allow-other test-cache: /var/tmp/test-cache
|
|
\f[R]
|
|
.fi
|
|
.SS Write Features
|
|
.SS Offline uploading
|
|
.PP
|
|
In an effort to make writing through cache more reliable, the backend
|
|
now supports this feature which can be activated by specifying a
|
|
\f[C]cache-tmp-upload-path\f[R].
|
|
.PP
|
|
A files goes through these states when using this feature:
|
|
.IP "1." 3
|
|
An upload is started (usually by copying a file on the cache remote)
|
|
.IP "2." 3
|
|
When the copy to the temporary location is complete the file is part of
|
|
the cached remote and looks and behaves like any other file (reading
|
|
included)
|
|
.IP "3." 3
|
|
After \f[C]cache-tmp-wait-time\f[R] passes and the file is next in line,
|
|
\f[C]rclone move\f[R] is used to move the file to the cloud provider
|
|
.IP "4." 3
|
|
Reading the file still works during the upload but most modifications on
|
|
it will be prohibited
|
|
.IP "5." 3
|
|
Once the move is complete the file is unlocked for modifications as it
|
|
becomes as any other regular file
|
|
.IP "6." 3
|
|
If the file is being read through \f[C]cache\f[R] when it\[aq]s actually
|
|
deleted from the temporary path then \f[C]cache\f[R] will simply swap
|
|
the source to the cloud provider without interrupting the reading (small
|
|
blip can happen though)
|
|
.PP
|
|
Files are uploaded in sequence and only one file is uploaded at a time.
|
|
Uploads will be stored in a queue and be processed based on the order
|
|
they were added.
|
|
The queue and the temporary storage is persistent across restarts but
|
|
can be cleared on startup with the \f[C]--cache-db-purge\f[R] flag.
|
|
.SS Write Support
|
|
.PP
|
|
Writes are supported through \f[C]cache\f[R].
|
|
One caveat is that a mounted cache remote does not add any retry or
|
|
fallback mechanism to the upload operation.
|
|
This will depend on the implementation of the wrapped remote.
|
|
Consider using \f[C]Offline uploading\f[R] for reliable writes.
|
|
.PP
|
|
One special case is covered with \f[C]cache-writes\f[R] which will cache
|
|
the file data at the same time as the upload when it is enabled making
|
|
it available from the cache store immediately once the upload is
|
|
finished.
|
|
.SS Read Features
|
|
.SS Multiple connections
|
|
.PP
|
|
To counter the high latency between a local PC where rclone is running
|
|
and cloud providers, the cache remote can split multiple requests to the
|
|
cloud provider for smaller file chunks and combines them together
|
|
locally where they can be available almost immediately before the reader
|
|
usually needs them.
|
|
.PP
|
|
This is similar to buffering when media files are played online.
|
|
Rclone will stay around the current marker but always try its best to
|
|
stay ahead and prepare the data before.
|
|
.SS Plex Integration
|
|
.PP
|
|
There is a direct integration with Plex which allows cache to detect
|
|
during reading if the file is in playback or not.
|
|
This helps cache to adapt how it queries the cloud provider depending on
|
|
what is needed for.
|
|
.PP
|
|
Scans will have a minimum amount of workers (1) while in a confirmed
|
|
playback cache will deploy the configured number of workers.
|
|
.PP
|
|
This integration opens the doorway to additional performance
|
|
improvements which will be explored in the near future.
|
|
.PP
|
|
\f[B]Note:\f[R] If Plex options are not configured, \f[C]cache\f[R] will
|
|
function with its configured options without adapting any of its
|
|
settings.
|
|
.PP
|
|
How to enable?
|
|
Run \f[C]rclone config\f[R] and add all the Plex options (endpoint,
|
|
username and password) in your remote and it will be automatically
|
|
enabled.
|
|
.PP
|
|
Affected settings: - \f[C]cache-workers\f[R]: \f[I]Configured value\f[R]
|
|
during confirmed playback or \f[I]1\f[R] all the other times
|
|
.SS Certificate Validation
|
|
.PP
|
|
When the Plex server is configured to only accept secure connections, it
|
|
is possible to use \f[C].plex.direct\f[R] URLs to ensure certificate
|
|
validation succeeds.
|
|
These URLs are used by Plex internally to connect to the Plex server
|
|
securely.
|
|
.PP
|
|
The format for these URLs is the following:
|
|
.PP
|
|
\f[C]https://ip-with-dots-replaced.server-hash.plex.direct:32400/\f[R]
|
|
.PP
|
|
The \f[C]ip-with-dots-replaced\f[R] part can be any IPv4 address, where
|
|
the dots have been replaced with dashes, e.g.
|
|
\f[C]127.0.0.1\f[R] becomes \f[C]127-0-0-1\f[R].
|
|
.PP
|
|
To get the \f[C]server-hash\f[R] part, the easiest way is to visit
|
|
.PP
|
|
https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token
|
|
.PP
|
|
This page will list all the available Plex servers for your account with
|
|
at least one \f[C].plex.direct\f[R] link for each.
|
|
Copy one URL and replace the IP address with the desired address.
|
|
This can be used as the \f[C]plex_url\f[R] value.
|
|
.SS Known issues
|
|
.SS Mount and --dir-cache-time
|
|
.PP
|
|
--dir-cache-time controls the first layer of directory caching which
|
|
works at the mount layer.
|
|
Being an independent caching mechanism from the \f[C]cache\f[R] backend,
|
|
it will manage its own entries based on the configured time.
|
|
.PP
|
|
To avoid getting in a scenario where dir cache has obsolete data and
|
|
cache would have the correct one, try to set \f[C]--dir-cache-time\f[R]
|
|
to a lower time than \f[C]--cache-info-age\f[R].
|
|
Default values are already configured in this way.
|
|
.SS Windows support - Experimental
|
|
.PP
|
|
There are a couple of issues with Windows \f[C]mount\f[R] functionality
|
|
that still require some investigations.
|
|
It should be considered as experimental thus far as fixes come in for
|
|
this OS.
|
|
.PP
|
|
Most of the issues seem to be related to the difference between
|
|
filesystems on Linux flavors and Windows as cache is heavily dependent
|
|
on them.
|
|
.PP
|
|
Any reports or feedback on how cache behaves on this OS is greatly
|
|
appreciated.
|
|
.IP \[bu] 2
|
|
https://github.com/rclone/rclone/issues/1935
|
|
.IP \[bu] 2
|
|
https://github.com/rclone/rclone/issues/1907
|
|
.IP \[bu] 2
|
|
https://github.com/rclone/rclone/issues/1834
|
|
.SS Risk of throttling
|
|
.PP
|
|
Future iterations of the cache backend will make use of the pooling
|
|
functionality of the cloud provider to synchronize and at the same time
|
|
make writing through it more tolerant to failures.
|
|
.PP
|
|
There are a couple of enhancements in track to add these but in the
|
|
meantime there is a valid concern that the expiring cache listings can
|
|
lead to cloud provider throttles or bans due to repeated queries on it
|
|
for very large mounts.
|
|
.PP
|
|
Some recommendations: - don\[aq]t use a very small interval for entry
|
|
information (\f[C]--cache-info-age\f[R]) - while writes aren\[aq]t yet
|
|
optimised, you can still write through \f[C]cache\f[R] which gives you
|
|
the advantage of adding the file in the cache at the same time if
|
|
configured to do so.
|
|
.PP
|
|
Future enhancements:
|
|
.IP \[bu] 2
|
|
https://github.com/rclone/rclone/issues/1937
|
|
.IP \[bu] 2
|
|
https://github.com/rclone/rclone/issues/1936
|
|
.SS cache and crypt
|
|
.PP
|
|
One common scenario is to keep your data encrypted in the cloud provider
|
|
using the \f[C]crypt\f[R] remote.
|
|
\f[C]crypt\f[R] uses a similar technique to wrap around an existing
|
|
remote and handles this translation in a seamless way.
|
|
.PP
|
|
There is an issue with wrapping the remotes in this order: \f[B]cloud
|
|
remote\f[R] -> \f[B]crypt\f[R] -> \f[B]cache\f[R]
|
|
.PP
|
|
During testing, I experienced a lot of bans with the remotes in this
|
|
order.
|
|
I suspect it might be related to how crypt opens files on the cloud
|
|
provider which makes it think we\[aq]re downloading the full file
|
|
instead of small chunks.
|
|
Organizing the remotes in this order yields better results: \f[B]cloud
|
|
remote\f[R] -> \f[B]cache\f[R] -> \f[B]crypt\f[R]
|
|
.SS absolute remote paths
|
|
.PP
|
|
\f[C]cache\f[R] can not differentiate between relative and absolute
|
|
paths for the wrapped remote.
|
|
Any path given in the \f[C]remote\f[R] config setting and on the command
|
|
line will be passed to the wrapped remote as is, but for storing the
|
|
chunks on disk the path will be made relative by removing any leading
|
|
\f[C]/\f[R] character.
|
|
.PP
|
|
This behavior is irrelevant for most backend types, but there are
|
|
backends where a leading \f[C]/\f[R] changes the effective directory,
|
|
e.g.
|
|
in the \f[C]sftp\f[R] backend paths starting with a \f[C]/\f[R] are
|
|
relative to the root of the SSH server and paths without are relative to
|
|
the user home directory.
|
|
As a result \f[C]sftp:bin\f[R] and \f[C]sftp:/bin\f[R] will share the
|
|
same cache folder, even if they represent a different directory on the
|
|
SSH server.
|
|
.SS Cache and Remote Control (--rc)
|
|
.PP
|
|
Cache supports the new \f[C]--rc\f[R] mode in rclone and can be remote
|
|
controlled through the following end points: By default, the listener is
|
|
disabled if you do not add the flag.
|
|
.SS rc cache/expire
|
|
.PP
|
|
Purge a remote from the cache backend.
|
|
Supports either a directory or a file.
|
|
It supports both encrypted and unencrypted file names if cache is
|
|
wrapped by crypt.
|
|
.PP
|
|
Params: - \f[B]remote\f[R] = path to remote \f[B](required)\f[R] -
|
|
\f[B]withData\f[R] = true/false to delete cached data (chunks) as well
|
|
\f[I](optional, false by default)\f[R]
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to cache (Cache a remote).
|
|
.SS --cache-remote
|
|
.PP
|
|
Remote to cache.
|
|
.PP
|
|
Normally should contain a \[aq]:\[aq] and a path, e.g.
|
|
\[dq]myremote:path/to/dir\[dq], \[dq]myremote:bucket\[dq] or maybe
|
|
\[dq]myremote:\[dq] (not recommended).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: remote
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_REMOTE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --cache-plex-url
|
|
.PP
|
|
The URL of the Plex server.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: plex_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_PLEX_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --cache-plex-username
|
|
.PP
|
|
The username of the Plex user.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: plex_username
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_PLEX_USERNAME
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --cache-plex-password
|
|
.PP
|
|
The password of the Plex user.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: plex_password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_PLEX_PASSWORD
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --cache-chunk-size
|
|
.PP
|
|
The size of a chunk (partial file data).
|
|
.PP
|
|
Use lower numbers for slower connections.
|
|
If the chunk size is changed, any downloaded chunks will be invalid and
|
|
cache-chunk-path will need to be cleared or unexpected EOF errors will
|
|
occur.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 5Mi
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]1M\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
1 MiB
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]5M\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
5 MiB
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]10M\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
10 MiB
|
|
.RE
|
|
.RE
|
|
.SS --cache-info-age
|
|
.PP
|
|
How long to cache file structure information (directory listings, file
|
|
size, times, etc.).
|
|
If all write operations are done through the cache then you can safely
|
|
make this value very large as the cache store will also be updated in
|
|
real time.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: info_age
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_INFO_AGE
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 6h0m0s
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]1h\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
1 hour
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]24h\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
24 hours
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]48h\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
48 hours
|
|
.RE
|
|
.RE
|
|
.SS --cache-chunk-total-size
|
|
.PP
|
|
The total size that the chunks can take up on the local disk.
|
|
.PP
|
|
If the cache exceeds this value then it will start to delete the oldest
|
|
chunks until it goes under this value.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_total_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_CHUNK_TOTAL_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 10Gi
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]500M\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
500 MiB
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]1G\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
1 GiB
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]10G\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
10 GiB
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to cache (Cache a remote).
|
|
.SS --cache-plex-token
|
|
.PP
|
|
The plex token for authentication - auto set normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: plex_token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_PLEX_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --cache-plex-insecure
|
|
.PP
|
|
Skip all certificate verification when connecting to the Plex server.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: plex_insecure
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_PLEX_INSECURE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --cache-db-path
|
|
.PP
|
|
Directory to store file structure metadata DB.
|
|
.PP
|
|
The remote name is used as the DB file name.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: db_path
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_DB_PATH
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]$HOME/.cache/rclone/cache-backend\[dq]
|
|
.SS --cache-chunk-path
|
|
.PP
|
|
Directory to cache chunk files.
|
|
.PP
|
|
Path to where partial file data (chunks) are stored locally.
|
|
The remote name is appended to the final path.
|
|
.PP
|
|
This config follows the \[dq]--cache-db-path\[dq].
|
|
If you specify a custom location for \[dq]--cache-db-path\[dq] and
|
|
don\[aq]t specify one for \[dq]--cache-chunk-path\[dq] then
|
|
\[dq]--cache-chunk-path\[dq] will use the same path as
|
|
\[dq]--cache-db-path\[dq].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_path
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_CHUNK_PATH
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]$HOME/.cache/rclone/cache-backend\[dq]
|
|
.SS --cache-db-purge
|
|
.PP
|
|
Clear all the cached data for this remote on start.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: db_purge
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_DB_PURGE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --cache-chunk-clean-interval
|
|
.PP
|
|
How often should the cache perform cleanups of the chunk storage.
|
|
.PP
|
|
The default value should be ok for most people.
|
|
If you find that the cache goes over \[dq]cache-chunk-total-size\[dq]
|
|
too often then try to lower this value to force it to perform cleanups
|
|
more often.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_clean_interval
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_CHUNK_CLEAN_INTERVAL
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 1m0s
|
|
.SS --cache-read-retries
|
|
.PP
|
|
How many times to retry a read from a cache storage.
|
|
.PP
|
|
Since reading from a cache stream is independent from downloading file
|
|
data, readers can get to a point where there\[aq]s no more data in the
|
|
cache.
|
|
Most of the times this can indicate a connectivity issue if cache
|
|
isn\[aq]t able to provide file data anymore.
|
|
.PP
|
|
For really slow connections, increase this to a point where the stream
|
|
is able to provide data but your experience will be very stuttering.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: read_retries
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_READ_RETRIES
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 10
|
|
.SS --cache-workers
|
|
.PP
|
|
How many workers should run in parallel to download chunks.
|
|
.PP
|
|
Higher values will mean more parallel processing (better CPU needed) and
|
|
more concurrent requests on the cloud provider.
|
|
This impacts several aspects like the cloud provider API limits, more
|
|
stress on the hardware that rclone runs on but it also means that
|
|
streams will be more fluid and data will be available much more faster
|
|
to readers.
|
|
.PP
|
|
\f[B]Note\f[R]: If the optional Plex integration is enabled then this
|
|
setting will adapt to the type of reading performed and the value
|
|
specified here will be used as a maximum number of workers to use.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: workers
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_WORKERS
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 4
|
|
.SS --cache-chunk-no-memory
|
|
.PP
|
|
Disable the in-memory cache for storing chunks during streaming.
|
|
.PP
|
|
By default, cache will keep file data during streaming in RAM as well to
|
|
provide it to readers as fast as possible.
|
|
.PP
|
|
This transient data is evicted as soon as it is read and the number of
|
|
chunks stored doesn\[aq]t exceed the number of workers.
|
|
However, depending on other settings like \[dq]cache-chunk-size\[dq] and
|
|
\[dq]cache-workers\[dq] this footprint can increase if there are
|
|
parallel streams too (multiple files being read at the same time).
|
|
.PP
|
|
If the hardware permits it, use this feature to provide an overall
|
|
better performance during streaming but it can also be disabled if RAM
|
|
is not available on the local machine.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_no_memory
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_CHUNK_NO_MEMORY
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --cache-rps
|
|
.PP
|
|
Limits the number of requests per second to the source FS (-1 to
|
|
disable).
|
|
.PP
|
|
This setting places a hard limit on the number of requests per second
|
|
that cache will be doing to the cloud provider remote and try to respect
|
|
that value by setting waits between reads.
|
|
.PP
|
|
If you find that you\[aq]re getting banned or limited on the cloud
|
|
provider through cache and know that a smaller number of requests per
|
|
second will allow you to work with it then you can use this setting for
|
|
that.
|
|
.PP
|
|
A good balance of all the other settings should make this setting
|
|
useless but it is available to set for more special cases.
|
|
.PP
|
|
\f[B]NOTE\f[R]: This will limit the number of requests during streams
|
|
but other API calls to the cloud provider like directory listings will
|
|
still pass.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: rps
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_RPS
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: -1
|
|
.SS --cache-writes
|
|
.PP
|
|
Cache file data on writes through the FS.
|
|
.PP
|
|
If you need to read files immediately after you upload them through
|
|
cache you can enable this flag to have their data stored in the cache
|
|
store at the same time during upload.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: writes
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_WRITES
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --cache-tmp-upload-path
|
|
.PP
|
|
Directory to keep temporary files until they are uploaded.
|
|
.PP
|
|
This is the path where cache will use as a temporary storage for new
|
|
files that need to be uploaded to the cloud provider.
|
|
.PP
|
|
Specifying a value will enable this feature.
|
|
Without it, it is completely disabled and files will be uploaded
|
|
directly to the cloud provider
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: tmp_upload_path
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_TMP_UPLOAD_PATH
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --cache-tmp-wait-time
|
|
.PP
|
|
How long should files be stored in local cache before being uploaded.
|
|
.PP
|
|
This is the duration that a file must wait in the temporary location
|
|
\f[I]cache-tmp-upload-path\f[R] before it is selected for upload.
|
|
.PP
|
|
Note that only one file is uploaded at a time and it can take longer to
|
|
start the upload if a queue formed for this purpose.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: tmp_wait_time
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_TMP_WAIT_TIME
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 15s
|
|
.SS --cache-db-wait-time
|
|
.PP
|
|
How long to wait for the DB to be available - 0 is unlimited.
|
|
.PP
|
|
Only one process can have the DB open at any one time, so rclone waits
|
|
for this duration for the DB to become available before it gives an
|
|
error.
|
|
.PP
|
|
If you set it to 0 then it will wait forever.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: db_wait_time
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CACHE_DB_WAIT_TIME
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 1s
|
|
.SS Backend commands
|
|
.PP
|
|
Here are the commands specific to the cache backend.
|
|
.PP
|
|
Run them with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend COMMAND remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The help below will explain what arguments each command takes.
|
|
.PP
|
|
See the \[dq]rclone backend\[dq]
|
|
command (https://rclone.org/commands/rclone_backend/) for more info on
|
|
how to pass options and arguments.
|
|
.PP
|
|
These can be run on a running backend using the rc command
|
|
backend/command (https://rclone.org/rc/#backend-command).
|
|
.SS stats
|
|
.PP
|
|
Print stats on the cache backend in JSON format.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend stats remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.SH Chunker (BETA)
|
|
.PP
|
|
The \f[C]chunker\f[R] overlay transparently splits large files into
|
|
smaller chunks during upload to wrapped remote and transparently
|
|
assembles them back when the file is downloaded.
|
|
This allows to effectively overcome size limits imposed by storage
|
|
providers.
|
|
.SS Configuration
|
|
.PP
|
|
To use it, first set up the underlying remote following the
|
|
configuration instructions for that remote.
|
|
You can also use a local pathname instead of a remote.
|
|
.PP
|
|
First check your chosen remote is working - we\[aq]ll call it
|
|
\f[C]remote:path\f[R] here.
|
|
Note that anything inside \f[C]remote:path\f[R] will be chunked and
|
|
anything outside won\[aq]t.
|
|
This means that if you are using a bucket-based remote (e.g.
|
|
S3, B2, swift) then you should probably put the bucket in the remote
|
|
\f[C]s3:bucket\f[R].
|
|
.PP
|
|
Now configure \f[C]chunker\f[R] using \f[C]rclone config\f[R].
|
|
We will call this one \f[C]overlay\f[R] to separate it from the
|
|
\f[C]remote\f[R] itself.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> overlay
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Transparently chunk/split large files
|
|
\[rs] \[dq]chunker\[dq]
|
|
[snip]
|
|
Storage> chunker
|
|
Remote to chunk/unchunk.
|
|
Normally should contain a \[aq]:\[aq] and a path, e.g. \[dq]myremote:path/to/dir\[dq],
|
|
\[dq]myremote:bucket\[dq] or maybe \[dq]myremote:\[dq] (not recommended).
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
remote> remote:path
|
|
Files larger than chunk size will be split in chunks.
|
|
Enter a size with suffix K,M,G,T. Press Enter for the default (\[dq]2G\[dq]).
|
|
chunk_size> 100M
|
|
Choose how chunker handles hash sums. All modes but \[dq]none\[dq] require metadata.
|
|
Enter a string value. Press Enter for the default (\[dq]md5\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise
|
|
\[rs] \[dq]none\[dq]
|
|
2 / MD5 for composite files
|
|
\[rs] \[dq]md5\[dq]
|
|
3 / SHA1 for composite files
|
|
\[rs] \[dq]sha1\[dq]
|
|
4 / MD5 for all files
|
|
\[rs] \[dq]md5all\[dq]
|
|
5 / SHA1 for all files
|
|
\[rs] \[dq]sha1all\[dq]
|
|
6 / Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported
|
|
\[rs] \[dq]md5quick\[dq]
|
|
7 / Similar to \[dq]md5quick\[dq] but prefers SHA1 over MD5
|
|
\[rs] \[dq]sha1quick\[dq]
|
|
hash_type> md5
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No
|
|
y/n> n
|
|
Remote config
|
|
--------------------
|
|
[overlay]
|
|
type = chunker
|
|
remote = remote:bucket
|
|
chunk_size = 100M
|
|
hash_type = md5
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.SS Specifying the remote
|
|
.PP
|
|
In normal use, make sure the remote has a \f[C]:\f[R] in.
|
|
If you specify the remote without a \f[C]:\f[R] then rclone will use a
|
|
local directory of that name.
|
|
So if you use a remote of \f[C]/path/to/secret/files\f[R] then rclone
|
|
will chunk stuff in that directory.
|
|
If you use a remote of \f[C]name\f[R] then rclone will put files in a
|
|
directory called \f[C]name\f[R] in the current directory.
|
|
.SS Chunking
|
|
.PP
|
|
When rclone starts a file upload, chunker checks the file size.
|
|
If it doesn\[aq]t exceed the configured chunk size, chunker will just
|
|
pass the file to the wrapped remote.
|
|
If a file is large, chunker will transparently cut data in pieces with
|
|
temporary names and stream them one by one, on the fly.
|
|
Each data chunk will contain the specified number of bytes, except for
|
|
the last one which may have less data.
|
|
If file size is unknown in advance (this is called a streaming upload),
|
|
chunker will internally create a temporary copy, record its size and
|
|
repeat the above process.
|
|
.PP
|
|
When upload completes, temporary chunk files are finally renamed.
|
|
This scheme guarantees that operations can be run in parallel and look
|
|
from outside as atomic.
|
|
A similar method with hidden temporary chunks is used for other
|
|
operations (copy/move/rename, etc.).
|
|
If an operation fails, hidden chunks are normally destroyed, and the
|
|
target composite file stays intact.
|
|
.PP
|
|
When a composite file download is requested, chunker transparently
|
|
assembles it by concatenating data chunks in order.
|
|
As the split is trivial one could even manually concatenate data chunks
|
|
together to obtain the original content.
|
|
.PP
|
|
When the \f[C]list\f[R] rclone command scans a directory on wrapped
|
|
remote, the potential chunk files are accounted for, grouped and
|
|
assembled into composite directory entries.
|
|
Any temporary chunks are hidden.
|
|
.PP
|
|
List and other commands can sometimes come across composite files with
|
|
missing or invalid chunks, e.g.
|
|
shadowed by like-named directory or another file.
|
|
This usually means that wrapped file system has been directly tampered
|
|
with or damaged.
|
|
If chunker detects a missing chunk it will by default print warning,
|
|
skip the whole incomplete group of chunks but proceed with current
|
|
command.
|
|
You can set the \f[C]--chunker-fail-hard\f[R] flag to have commands
|
|
abort with error message in such cases.
|
|
.SS Chunk names
|
|
.PP
|
|
The default chunk name format is \f[C]*.rclone_chunk.###\f[R], hence by
|
|
default chunk names are \f[C]BIG_FILE_NAME.rclone_chunk.001\f[R],
|
|
\f[C]BIG_FILE_NAME.rclone_chunk.002\f[R] etc.
|
|
You can configure another name format using the \f[C]name_format\f[R]
|
|
configuration file option.
|
|
The format uses asterisk \f[C]*\f[R] as a placeholder for the base file
|
|
name and one or more consecutive hash characters \f[C]#\f[R] as a
|
|
placeholder for sequential chunk number.
|
|
There must be one and only one asterisk.
|
|
The number of consecutive hash characters defines the minimum length of
|
|
a string representing a chunk number.
|
|
If decimal chunk number has less digits than the number of hashes, it is
|
|
left-padded by zeros.
|
|
If the decimal string is longer, it is left intact.
|
|
By default numbering starts from 1 but there is another option that
|
|
allows user to start from 0, e.g.
|
|
for compatibility with legacy software.
|
|
.PP
|
|
For example, if name format is \f[C]big_*-##.part\f[R] and original file
|
|
name is \f[C]data.txt\f[R] and numbering starts from 0, then the first
|
|
chunk will be named \f[C]big_data.txt-00.part\f[R], the 99th chunk will
|
|
be \f[C]big_data.txt-98.part\f[R] and the 302nd chunk will become
|
|
\f[C]big_data.txt-301.part\f[R].
|
|
.PP
|
|
Note that \f[C]list\f[R] assembles composite directory entries only when
|
|
chunk names match the configured format and treats non-conforming file
|
|
names as normal non-chunked files.
|
|
.PP
|
|
When using \f[C]norename\f[R] transactions, chunk names will
|
|
additionally have a unique file version suffix.
|
|
For example, \f[C]BIG_FILE_NAME.rclone_chunk.001_bp562k\f[R].
|
|
.SS Metadata
|
|
.PP
|
|
Besides data chunks chunker will by default create metadata object for a
|
|
composite file.
|
|
The object is named after the original file.
|
|
Chunker allows user to disable metadata completely (the \f[C]none\f[R]
|
|
format).
|
|
Note that metadata is normally not created for files smaller than the
|
|
configured chunk size.
|
|
This may change in future rclone releases.
|
|
.SS Simple JSON metadata format
|
|
.PP
|
|
This is the default format.
|
|
It supports hash sums and chunk validation for composite files.
|
|
Meta objects carry the following fields:
|
|
.IP \[bu] 2
|
|
\f[C]ver\f[R] - version of format, currently \f[C]1\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]size\f[R] - total size of composite file
|
|
.IP \[bu] 2
|
|
\f[C]nchunks\f[R] - number of data chunks in file
|
|
.IP \[bu] 2
|
|
\f[C]md5\f[R] - MD5 hashsum of composite file (if present)
|
|
.IP \[bu] 2
|
|
\f[C]sha1\f[R] - SHA1 hashsum (if present)
|
|
.IP \[bu] 2
|
|
\f[C]txn\f[R] - identifies current version of the file
|
|
.PP
|
|
There is no field for composite file name as it\[aq]s simply equal to
|
|
the name of meta object on the wrapped remote.
|
|
Please refer to respective sections for details on hashsums and modified
|
|
time handling.
|
|
.SS No metadata
|
|
.PP
|
|
You can disable meta objects by setting the meta format option to
|
|
\f[C]none\f[R].
|
|
In this mode chunker will scan directory for all files that follow
|
|
configured chunk name format, group them by detecting chunks with the
|
|
same base name and show group names as virtual composite files.
|
|
This method is more prone to missing chunk errors (especially missing
|
|
last chunk) than format with metadata enabled.
|
|
.SS Hashsums
|
|
.PP
|
|
Chunker supports hashsums only when a compatible metadata is present.
|
|
Hence, if you choose metadata format of \f[C]none\f[R], chunker will
|
|
report hashsum as \f[C]UNSUPPORTED\f[R].
|
|
.PP
|
|
Please note that by default metadata is stored only for composite files.
|
|
If a file is smaller than configured chunk size, chunker will
|
|
transparently redirect hash requests to wrapped remote, so support
|
|
depends on that.
|
|
You will see the empty string as a hashsum of requested type for small
|
|
files if the wrapped remote doesn\[aq]t support it.
|
|
.PP
|
|
Many storage backends support MD5 and SHA1 hash types, so does chunker.
|
|
With chunker you can choose one or another but not both.
|
|
MD5 is set by default as the most supported type.
|
|
Since chunker keeps hashes for composite files and falls back to the
|
|
wrapped remote hash for non-chunked ones, we advise you to choose the
|
|
same hash type as supported by wrapped remote so that your file listings
|
|
look coherent.
|
|
.PP
|
|
If your storage backend does not support MD5 or SHA1 but you need
|
|
consistent file hashing, configure chunker with \f[C]md5all\f[R] or
|
|
\f[C]sha1all\f[R].
|
|
These two modes guarantee given hash for all files.
|
|
If wrapped remote doesn\[aq]t support it, chunker will then add metadata
|
|
to all files, even small.
|
|
However, this can double the amount of small files in storage and incur
|
|
additional service charges.
|
|
You can even use chunker to force md5/sha1 support in any other remote
|
|
at expense of sidecar meta objects by setting e.g.
|
|
\f[C]chunk_type=sha1all\f[R] to force hashsums and
|
|
\f[C]chunk_size=1P\f[R] to effectively disable chunking.
|
|
.PP
|
|
Normally, when a file is copied to chunker controlled remote, chunker
|
|
will ask the file source for compatible file hash and revert to
|
|
on-the-fly calculation if none is found.
|
|
This involves some CPU overhead but provides a guarantee that given
|
|
hashsum is available.
|
|
Also, chunker will reject a server-side copy or move operation if source
|
|
and destination hashsum types are different resulting in the extra
|
|
network bandwidth, too.
|
|
In some rare cases this may be undesired, so chunker provides two
|
|
optional choices: \f[C]sha1quick\f[R] and \f[C]md5quick\f[R].
|
|
If the source does not support primary hash type and the quick mode is
|
|
enabled, chunker will try to fall back to the secondary type.
|
|
This will save CPU and bandwidth but can result in empty hashsums at
|
|
destination.
|
|
Beware of consequences: the \f[C]sync\f[R] command will revert
|
|
(sometimes silently) to time/size comparison if compatible hashsums
|
|
between source and target are not found.
|
|
.SS Modified time
|
|
.PP
|
|
Chunker stores modification times using the wrapped remote so support
|
|
depends on that.
|
|
For a small non-chunked file the chunker overlay simply manipulates
|
|
modification time of the wrapped remote file.
|
|
For a composite file with metadata chunker will get and set modification
|
|
time of the metadata object on the wrapped remote.
|
|
If file is chunked but metadata format is \f[C]none\f[R] then chunker
|
|
will use modification time of the first data chunk.
|
|
.SS Migrations
|
|
.PP
|
|
The idiomatic way to migrate to a different chunk size, hash type,
|
|
transaction style or chunk naming scheme is to:
|
|
.IP \[bu] 2
|
|
Collect all your chunked files under a directory and have your chunker
|
|
remote point to it.
|
|
.IP \[bu] 2
|
|
Create another directory (most probably on the same cloud storage) and
|
|
configure a new remote with desired metadata format, hash type, chunk
|
|
naming etc.
|
|
.IP \[bu] 2
|
|
Now run \f[C]rclone sync -i oldchunks: newchunks:\f[R] and all your data
|
|
will be transparently converted in transfer.
|
|
This may take some time, yet chunker will try server-side copy if
|
|
possible.
|
|
.IP \[bu] 2
|
|
After checking data integrity you may remove configuration section of
|
|
the old remote.
|
|
.PP
|
|
If rclone gets killed during a long operation on a big composite file,
|
|
hidden temporary chunks may stay in the directory.
|
|
They will not be shown by the \f[C]list\f[R] command but will eat up
|
|
your account quota.
|
|
Please note that the \f[C]deletefile\f[R] command deletes only active
|
|
chunks of a file.
|
|
As a workaround, you can use remote of the wrapped file system to see
|
|
them.
|
|
An easy way to get rid of hidden garbage is to copy littered directory
|
|
somewhere using the chunker remote and purge the original directory.
|
|
The \f[C]copy\f[R] command will copy only active chunks while the
|
|
\f[C]purge\f[R] will remove everything including garbage.
|
|
.SS Caveats and Limitations
|
|
.PP
|
|
Chunker requires wrapped remote to support server-side \f[C]move\f[R]
|
|
(or \f[C]copy\f[R] + \f[C]delete\f[R]) operations, otherwise it will
|
|
explicitly refuse to start.
|
|
This is because it internally renames temporary chunk files to their
|
|
final names when an operation completes successfully.
|
|
.PP
|
|
Chunker encodes chunk number in file name, so with default
|
|
\f[C]name_format\f[R] setting it adds 17 characters.
|
|
Also chunker adds 7 characters of temporary suffix during operations.
|
|
Many file systems limit base file name without path by 255 characters.
|
|
Using rclone\[aq]s crypt remote as a base file system limits file name
|
|
by 143 characters.
|
|
Thus, maximum name length is 231 for most files and 119 for
|
|
chunker-over-crypt.
|
|
A user in need can change name format to e.g.
|
|
\f[C]*.rcc##\f[R] and save 10 characters (provided at most 99 chunks per
|
|
file).
|
|
.PP
|
|
Note that a move implemented using the copy-and-delete method may incur
|
|
double charging with some cloud storage providers.
|
|
.PP
|
|
Chunker will not automatically rename existing chunks when you run
|
|
\f[C]rclone config\f[R] on a live remote and change the chunk name
|
|
format.
|
|
Beware that in result of this some files which have been treated as
|
|
chunks before the change can pop up in directory listings as normal
|
|
files and vice versa.
|
|
The same warning holds for the chunk size.
|
|
If you desperately need to change critical chunking settings, you should
|
|
run data migration as described above.
|
|
.PP
|
|
If wrapped remote is case insensitive, the chunker overlay will inherit
|
|
that property (so you can\[aq]t have a file called \[dq]Hello.doc\[dq]
|
|
and \[dq]hello.doc\[dq] in the same directory).
|
|
.PP
|
|
Chunker included in rclone releases up to \f[C]v1.54\f[R] can sometimes
|
|
fail to detect metadata produced by recent versions of rclone.
|
|
We recommend users to keep rclone up-to-date to avoid data corruption.
|
|
.PP
|
|
Changing \f[C]transactions\f[R] is dangerous and requires explicit
|
|
migration.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to chunker (Transparently
|
|
chunk/split large files).
|
|
.SS --chunker-remote
|
|
.PP
|
|
Remote to chunk/unchunk.
|
|
.PP
|
|
Normally should contain a \[aq]:\[aq] and a path, e.g.
|
|
\[dq]myremote:path/to/dir\[dq], \[dq]myremote:bucket\[dq] or maybe
|
|
\[dq]myremote:\[dq] (not recommended).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: remote
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CHUNKER_REMOTE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --chunker-chunk-size
|
|
.PP
|
|
Files larger than chunk size will be split in chunks.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CHUNKER_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 2Gi
|
|
.SS --chunker-hash-type
|
|
.PP
|
|
Choose how chunker handles hash sums.
|
|
.PP
|
|
All modes but \[dq]none\[dq] require metadata.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: hash_type
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CHUNKER_HASH_TYPE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]md5\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]none\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Pass any hash supported by wrapped remote for non-chunked files.
|
|
.IP \[bu] 2
|
|
Return nothing otherwise.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]md5\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
MD5 for composite files.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sha1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
SHA1 for composite files.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]md5all\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
MD5 for all files.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sha1all\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
SHA1 for all files.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]md5quick\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Copying a file to chunker will request MD5 from the source.
|
|
.IP \[bu] 2
|
|
Falling back to SHA1 if unsupported.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sha1quick\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Similar to \[dq]md5quick\[dq] but prefers SHA1 over MD5.
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to chunker (Transparently
|
|
chunk/split large files).
|
|
.SS --chunker-name-format
|
|
.PP
|
|
String format of chunk file names.
|
|
.PP
|
|
The two placeholders are: base file name (*) and chunk number (#...).
|
|
There must be one and only one asterisk and one or more consecutive hash
|
|
characters.
|
|
If chunk number has less digits than the number of hashes, it is
|
|
left-padded by zeros.
|
|
If there are more digits in the number, they are left as is.
|
|
Possible chunk files are ignored if their name does not match given
|
|
format.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: name_format
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CHUNKER_NAME_FORMAT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]*.rclone_chunk.###\[dq]
|
|
.SS --chunker-start-from
|
|
.PP
|
|
Minimum valid chunk number.
|
|
Usually 0 or 1.
|
|
.PP
|
|
By default chunk numbers start from 1.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: start_from
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CHUNKER_START_FROM
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 1
|
|
.SS --chunker-meta-format
|
|
.PP
|
|
Format of the metadata object or \[dq]none\[dq].
|
|
.PP
|
|
By default \[dq]simplejson\[dq].
|
|
Metadata is a small JSON file named after the composite file.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: meta_format
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CHUNKER_META_FORMAT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]simplejson\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]none\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Do not use metadata files at all.
|
|
.IP \[bu] 2
|
|
Requires hash type \[dq]none\[dq].
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]simplejson\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Simple JSON supports hash sums and chunk validation.
|
|
.IP \[bu] 2
|
|
It has the following fields: ver, size, nchunks, md5, sha1.
|
|
.RE
|
|
.RE
|
|
.SS --chunker-fail-hard
|
|
.PP
|
|
Choose how chunker should handle files with missing or invalid chunks.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: fail_hard
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CHUNKER_FAIL_HARD
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]true\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Report errors and abort current command.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]false\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Warn user, skip incomplete file and proceed.
|
|
.RE
|
|
.RE
|
|
.SS --chunker-transactions
|
|
.PP
|
|
Choose how chunker should handle temporary files during transactions.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: transactions
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CHUNKER_TRANSACTIONS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]rename\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]rename\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rename temporary files after a successful transaction.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]norename\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Leave temporary file names and write transaction ID to metadata file.
|
|
.IP \[bu] 2
|
|
Metadata is required for no rename transactions (meta format cannot be
|
|
\[dq]none\[dq]).
|
|
.IP \[bu] 2
|
|
If you are using norename transactions you should be careful not to
|
|
downgrade Rclone
|
|
.IP \[bu] 2
|
|
as older versions of Rclone don\[aq]t support this transaction style and
|
|
will misinterpret
|
|
.IP \[bu] 2
|
|
files manipulated by norename transactions.
|
|
.IP \[bu] 2
|
|
This method is EXPERIMENTAL, don\[aq]t use on production systems.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]auto\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rename or norename will be used depending on capabilities of the
|
|
backend.
|
|
.IP \[bu] 2
|
|
If meta format is set to \[dq]none\[dq], rename transactions will always
|
|
be used.
|
|
.IP \[bu] 2
|
|
This method is EXPERIMENTAL, don\[aq]t use on production systems.
|
|
.RE
|
|
.RE
|
|
.SH Citrix ShareFile
|
|
.PP
|
|
Citrix ShareFile (https://sharefile.com) is a secure file sharing and
|
|
transfer service aimed as business.
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for Citrix ShareFile involves getting a token from
|
|
Citrix ShareFile which you can in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
XX / Citrix Sharefile
|
|
\[rs] \[dq]sharefile\[dq]
|
|
Storage> sharefile
|
|
** See help for sharefile backend at: https://rclone.org/sharefile/ **
|
|
|
|
ID of the root folder
|
|
|
|
Leave blank to access \[dq]Personal Folders\[dq]. You can use one of the
|
|
standard values here or any folder ID (long hex number ID).
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Access the Personal Folders. (Default)
|
|
\[rs] \[dq]\[dq]
|
|
2 / Access the Favorites folder.
|
|
\[rs] \[dq]favorites\[dq]
|
|
3 / Access all the shared folders.
|
|
\[rs] \[dq]allshared\[dq]
|
|
4 / Access all the individual connectors.
|
|
\[rs] \[dq]connectors\[dq]
|
|
5 / Access the home, favorites, and shared folders as well as the connectors.
|
|
\[rs] \[dq]top\[dq]
|
|
root_folder_id>
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No
|
|
y/n> n
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth?state=XXX
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
--------------------
|
|
[remote]
|
|
type = sharefile
|
|
endpoint = https://XXX.sharefile.com
|
|
token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2019-09-30T19:41:45.878561877+01:00\[dq]}
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Citrix ShareFile.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your ShareFile
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your ShareFile
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an ShareFile directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Modified time and hashes
|
|
.PP
|
|
ShareFile allows modification times to be set on objects accurate to 1
|
|
second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
.PP
|
|
ShareFile supports MD5 type hashes, so you can use the
|
|
\f[C]--checksum\f[R] flag.
|
|
.SS Transfers
|
|
.PP
|
|
For files above 128 MiB rclone will use a chunked transfer.
|
|
Rclone will upload up to \f[C]--transfers\f[R] chunks at the same time
|
|
(shared among all the multipart uploads).
|
|
Chunks are buffered in memory and are normally 64 MiB so increasing
|
|
\f[C]--transfers\f[R] will increase memory use.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
T{
|
|
*
|
|
T}@T{
|
|
0x2A
|
|
T}@T{
|
|
\[uFF0A]
|
|
T}
|
|
T{
|
|
<
|
|
T}@T{
|
|
0x3C
|
|
T}@T{
|
|
\[uFF1C]
|
|
T}
|
|
T{
|
|
>
|
|
T}@T{
|
|
0x3E
|
|
T}@T{
|
|
\[uFF1E]
|
|
T}
|
|
T{
|
|
?
|
|
T}@T{
|
|
0x3F
|
|
T}@T{
|
|
\[uFF1F]
|
|
T}
|
|
T{
|
|
:
|
|
T}@T{
|
|
0x3A
|
|
T}@T{
|
|
\[uFF1A]
|
|
T}
|
|
T{
|
|
|
|
|
T}@T{
|
|
0x7C
|
|
T}@T{
|
|
\[uFF5C]
|
|
T}
|
|
T{
|
|
\[dq]
|
|
T}@T{
|
|
0x22
|
|
T}@T{
|
|
\[uFF02]
|
|
T}
|
|
.TE
|
|
.PP
|
|
File names can also not start or end with the following characters.
|
|
These only get replaced if they are the first or last character in the
|
|
name:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
SP
|
|
T}@T{
|
|
0x20
|
|
T}@T{
|
|
\[u2420]
|
|
T}
|
|
T{
|
|
\&.
|
|
T}@T{
|
|
0x2E
|
|
T}@T{
|
|
\[uFF0E]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to sharefile (Citrix Sharefile).
|
|
.SS --sharefile-root-folder-id
|
|
.PP
|
|
ID of the root folder.
|
|
.PP
|
|
Leave blank to access \[dq]Personal Folders\[dq].
|
|
You can use one of the standard values here or any folder ID (long hex
|
|
number ID).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: root_folder_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SHAREFILE_ROOT_FOLDER_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Access the Personal Folders (default).
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]favorites\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Access the Favorites folder.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]allshared\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Access all the shared folders.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]connectors\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Access all the individual connectors.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]top\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Access the home, favorites, and shared folders as well as the
|
|
connectors.
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to sharefile (Citrix Sharefile).
|
|
.SS --sharefile-upload-cutoff
|
|
.PP
|
|
Cutoff for switching to multipart upload.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_cutoff
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SHAREFILE_UPLOAD_CUTOFF
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 128Mi
|
|
.SS --sharefile-chunk-size
|
|
.PP
|
|
Upload chunk size.
|
|
.PP
|
|
Must a power of 2 >= 256k.
|
|
.PP
|
|
Making this larger will improve performance, but note that each chunk is
|
|
buffered in memory one per transfer.
|
|
.PP
|
|
Reducing this will reduce memory usage but decrease performance.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SHAREFILE_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 64Mi
|
|
.SS --sharefile-endpoint
|
|
.PP
|
|
Endpoint for API calls.
|
|
.PP
|
|
This is usually auto discovered as part of the oauth process, but can be
|
|
set manually to something like: https://XXX.sharefile.com
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SHAREFILE_ENDPOINT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sharefile-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SHAREFILE_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default:
|
|
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
Note that ShareFile is case insensitive so you can\[aq]t have a file
|
|
called \[dq]Hello.doc\[dq] and one called \[dq]hello.doc\[dq].
|
|
.PP
|
|
ShareFile only supports filenames up to 256 characters in length.
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the Citrix ShareFile backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SH Crypt
|
|
.PP
|
|
Rclone \f[C]crypt\f[R] remotes encrypt and decrypt other remotes.
|
|
.PP
|
|
A remote of type \f[C]crypt\f[R] does not access a storage
|
|
system (https://rclone.org/overview/) directly, but instead wraps
|
|
another remote, which in turn accesses the storage system.
|
|
This is similar to how alias (https://rclone.org/alias/),
|
|
union (https://rclone.org/union/), chunker (https://rclone.org/chunker/)
|
|
and a few others work.
|
|
It makes the usage very flexible, as you can add a layer, in this case
|
|
an encryption layer, on top of any other backend, even in multiple
|
|
layers.
|
|
Rclone\[aq]s functionality can be used as with any other remote, for
|
|
example you can mount (https://rclone.org/commands/rclone_mount/) a
|
|
crypt remote.
|
|
.PP
|
|
Accessing a storage system through a crypt remote realizes client-side
|
|
encryption, which makes it safe to keep your data in a location you do
|
|
not trust will not get compromised.
|
|
When working against the \f[C]crypt\f[R] remote, rclone will
|
|
automatically encrypt (before uploading) and decrypt (after downloading)
|
|
on your local system as needed on the fly, leaving the data encrypted at
|
|
rest in the wrapped remote.
|
|
If you access the storage system using an application other than rclone,
|
|
or access the wrapped remote directly using rclone, there will not be
|
|
any encryption/decryption: Downloading existing content will just give
|
|
you the encrypted (scrambled) format, and anything you upload will
|
|
\f[I]not\f[R] become encrypted.
|
|
.PP
|
|
The encryption is a secret-key encryption (also called symmetric key
|
|
encryption) algorithm, where a password (or pass phrase) is used to
|
|
generate real encryption key.
|
|
The password can be supplied by user, or you may chose to let rclone
|
|
generate one.
|
|
It will be stored in the configuration file, in a lightly obscured form.
|
|
If you are in an environment where you are not able to keep your
|
|
configuration secured, you should add configuration
|
|
encryption (https://rclone.org/docs/#configuration-encryption) as
|
|
protection.
|
|
As long as you have this configuration file, you will be able to decrypt
|
|
your data.
|
|
Without the configuration file, as long as you remember the password (or
|
|
keep it in a safe place), you can re-create the configuration and gain
|
|
access to the existing data.
|
|
You may also configure a corresponding remote in a different
|
|
installation to access the same data.
|
|
See below for guidance to changing password.
|
|
.PP
|
|
Encryption uses cryptographic
|
|
salt (https://en.wikipedia.org/wiki/Salt_(cryptography)), to permute the
|
|
encryption key so that the same string may be encrypted in different
|
|
ways.
|
|
When configuring the crypt remote it is optional to enter a salt, or to
|
|
let rclone generate a unique salt.
|
|
If omitted, rclone uses a built-in unique string.
|
|
Normally in cryptography, the salt is stored together with the encrypted
|
|
content, and do not have to be memorized by the user.
|
|
This is not the case in rclone, because rclone does not store any
|
|
additional information on the remotes.
|
|
Use of custom salt is effectively a second password that must be
|
|
memorized.
|
|
.PP
|
|
File content encryption is performed using NaCl
|
|
SecretBox (https://godoc.org/golang.org/x/crypto/nacl/secretbox), based
|
|
on XSalsa20 cipher and Poly1305 for integrity.
|
|
Names (file- and directory names) are also encrypted by default, but
|
|
this has some implications and is therefore possible to turned off.
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]secret\f[R].
|
|
.PP
|
|
To use \f[C]crypt\f[R], first set up the underlying remote.
|
|
Follow the \f[C]rclone config\f[R] instructions for the specific
|
|
backend.
|
|
.PP
|
|
Before configuring the crypt remote, check the underlying remote is
|
|
working.
|
|
In this example the underlying remote is called \f[C]remote\f[R].
|
|
We will configure a path \f[C]path\f[R] within this remote to contain
|
|
the encrypted content.
|
|
Anything inside \f[C]remote:path\f[R] will be encrypted and anything
|
|
outside will not.
|
|
.PP
|
|
Configure \f[C]crypt\f[R] using \f[C]rclone config\f[R].
|
|
In this example the \f[C]crypt\f[R] remote is called \f[C]secret\f[R],
|
|
to differentiate it from the underlying \f[C]remote\f[R].
|
|
.PP
|
|
When you are done you can use the crypt remote named \f[C]secret\f[R]
|
|
just as you would with any other remote, e.g.
|
|
\f[C]rclone copy D:\[rs]docs secret:\[rs]docs\f[R], and rclone will
|
|
encrypt and decrypt as needed on the fly.
|
|
If you access the wrapped remote \f[C]remote:path\f[R] directly you will
|
|
bypass the encryption, and anything you read will be in encrypted form,
|
|
and anything you write will be unencrypted.
|
|
To avoid issues it is best to configure a dedicated path for encrypted
|
|
content, and access it exclusively through a crypt remote.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> secret
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Encrypt/Decrypt a remote
|
|
\[rs] \[dq]crypt\[dq]
|
|
[snip]
|
|
Storage> crypt
|
|
** See help for crypt backend at: https://rclone.org/crypt/ **
|
|
|
|
Remote to encrypt/decrypt.
|
|
Normally should contain a \[aq]:\[aq] and a path, eg \[dq]myremote:path/to/dir\[dq],
|
|
\[dq]myremote:bucket\[dq] or maybe \[dq]myremote:\[dq] (not recommended).
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
remote> remote:path
|
|
How to encrypt the filenames.
|
|
Enter a string value. Press Enter for the default (\[dq]standard\[dq]).
|
|
Choose a number from below, or type in your own value.
|
|
/ Encrypt the filenames.
|
|
1 | See the docs for the details.
|
|
\[rs] \[dq]standard\[dq]
|
|
2 / Very simple filename obfuscation.
|
|
\[rs] \[dq]obfuscate\[dq]
|
|
/ Don\[aq]t encrypt the file names.
|
|
3 | Adds a \[dq].bin\[dq] extension only.
|
|
\[rs] \[dq]off\[dq]
|
|
filename_encryption>
|
|
Option to either encrypt directory names or leave them intact.
|
|
|
|
NB If filename_encryption is \[dq]off\[dq] then this option will do nothing.
|
|
Enter a boolean value (true or false). Press Enter for the default (\[dq]true\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Encrypt directory names.
|
|
\[rs] \[dq]true\[dq]
|
|
2 / Don\[aq]t encrypt directory names, leave them intact.
|
|
\[rs] \[dq]false\[dq]
|
|
directory_name_encryption>
|
|
Password or pass phrase for encryption.
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
y/g> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
Password or pass phrase for salt. Optional but recommended.
|
|
Should be different to the previous password.
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
n) No leave this optional password blank (default)
|
|
y/g/n> g
|
|
Password strength in bits.
|
|
64 is just about memorable
|
|
128 is secure
|
|
1024 is the maximum
|
|
Bits> 128
|
|
Your password is: JAsJvRcgR-_veXNfy_sGmQ
|
|
Use this password? Please note that an obscured version of this
|
|
password (and not the password itself) will be stored under your
|
|
configuration file, so keep this generated password in a safe place.
|
|
y) Yes (default)
|
|
n) No
|
|
y/n>
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No (default)
|
|
y/n>
|
|
Remote config
|
|
--------------------
|
|
[secret]
|
|
type = crypt
|
|
remote = remote:path
|
|
password = *** ENCRYPTED ***
|
|
password2 = *** ENCRYPTED ***
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]Important\f[R] The crypt password stored in \f[C]rclone.conf\f[R]
|
|
is lightly obscured.
|
|
That only protects it from cursory inspection.
|
|
It is not secure unless configuration
|
|
encryption (https://rclone.org/docs/#configuration-encryption) of
|
|
\f[C]rclone.conf\f[R] is specified.
|
|
.PP
|
|
A long passphrase is recommended, or \f[C]rclone config\f[R] can
|
|
generate a random one.
|
|
.PP
|
|
The obscured password is created using AES-CTR with a static key.
|
|
The salt is stored verbatim at the beginning of the obscured password.
|
|
This static key is shared between all versions of rclone.
|
|
.PP
|
|
If you reconfigure rclone with the same passwords/passphrases elsewhere
|
|
it will be compatible, but the obscured version will be different due to
|
|
the different salt.
|
|
.PP
|
|
Rclone does not encrypt
|
|
.IP \[bu] 2
|
|
file length - this can be calculated within 16 bytes
|
|
.IP \[bu] 2
|
|
modification time - used for syncing
|
|
.SS Specifying the remote
|
|
.PP
|
|
When configuring the remote to encrypt/decrypt, you may specify any
|
|
string that rclone accepts as a source/destination of other commands.
|
|
.PP
|
|
The primary use case is to specify the path into an already configured
|
|
remote (e.g.
|
|
\f[C]remote:path/to/dir\f[R] or \f[C]remote:bucket\f[R]), such that data
|
|
in a remote untrusted location can be stored encrypted.
|
|
.PP
|
|
You may also specify a local filesystem path, such as
|
|
\f[C]/path/to/dir\f[R] on Linux, \f[C]C:\[rs]path\[rs]to\[rs]dir\f[R] on
|
|
Windows.
|
|
By creating a crypt remote pointing to such a local filesystem path, you
|
|
can use rclone as a utility for pure local file encryption, for example
|
|
to keep encrypted files on a removable USB drive.
|
|
.PP
|
|
\f[B]Note\f[R]: A string which do not contain a \f[C]:\f[R] will by
|
|
rclone be treated as a relative path in the local filesystem.
|
|
For example, if you enter the name \f[C]remote\f[R] without the trailing
|
|
\f[C]:\f[R], it will be treated as a subdirectory of the current
|
|
directory with name \[dq]remote\[dq].
|
|
.PP
|
|
If a path \f[C]remote:path/to/dir\f[R] is specified, rclone stores
|
|
encrypted files in \f[C]path/to/dir\f[R] on the remote.
|
|
With file name encryption, files saved to
|
|
\f[C]secret:subdir/subfile\f[R] are stored in the unencrypted path
|
|
\f[C]path/to/dir\f[R] but the \f[C]subdir/subpath\f[R] element is
|
|
encrypted.
|
|
.PP
|
|
The path you specify does not have to exist, rclone will create it when
|
|
needed.
|
|
.PP
|
|
If you intend to use the wrapped remote both directly for keeping
|
|
unencrypted content, as well as through a crypt remote for encrypted
|
|
content, it is recommended to point the crypt remote to a separate
|
|
directory within the wrapped remote.
|
|
If you use a bucket-based storage system (e.g.
|
|
Swift, S3, Google Compute Storage, B2, Hubic) it is generally advisable
|
|
to wrap the crypt remote around a specific bucket (\f[C]s3:bucket\f[R]).
|
|
If wrapping around the entire root of the storage (\f[C]s3:\f[R]), and
|
|
use the optional file name encryption, rclone will encrypt the bucket
|
|
name.
|
|
.SS Changing password
|
|
.PP
|
|
Should the password, or the configuration file containing a lightly
|
|
obscured form of the password, be compromised, you need to re-encrypt
|
|
your data with a new password.
|
|
Since rclone uses secret-key encryption, where the encryption key is
|
|
generated directly from the password kept on the client, it is not
|
|
possible to change the password/key of already encrypted content.
|
|
Just changing the password configured for an existing crypt remote means
|
|
you will no longer able to decrypt any of the previously encrypted
|
|
content.
|
|
The only possibility is to re-upload everything via a crypt remote
|
|
configured with your new password.
|
|
.PP
|
|
Depending on the size of your data, your bandwith, storage quota etc,
|
|
there are different approaches you can take: - If you have everything in
|
|
a different location, for example on your local system, you could remove
|
|
all of the prior encrypted files, change the password for your
|
|
configured crypt remote (or delete and re-create the crypt
|
|
configuration), and then re-upload everything from the alternative
|
|
location.
|
|
- If you have enough space on the storage system you can create a new
|
|
crypt remote pointing to a separate directory on the same backend, and
|
|
then use rclone to copy everything from the original crypt remote to the
|
|
new, effectively decrypting everything on the fly using the old password
|
|
and re-encrypting using the new password.
|
|
When done, delete the original crypt remote directory and finally the
|
|
rclone crypt configuration with the old password.
|
|
All data will be streamed from the storage system and back, so you will
|
|
get half the bandwith and be charged twice if you have upload and
|
|
download quota on the storage system.
|
|
.PP
|
|
\f[B]Note\f[R]: A security problem related to the random password
|
|
generator was fixed in rclone version 1.53.3 (released 2020-11-19).
|
|
Passwords generated by rclone config in version 1.49.0 (released
|
|
2019-08-26) to 1.53.2 (released 2020-10-26) are not considered secure
|
|
and should be changed.
|
|
If you made up your own password, or used rclone version older than
|
|
1.49.0 or newer than 1.53.2 to generate it, you are \f[I]not\f[R]
|
|
affected by this issue.
|
|
See issue #4783 (https://github.com/rclone/rclone/issues/4783) for more
|
|
details, and a tool you can use to check if you are affected.
|
|
.SS Example
|
|
.PP
|
|
Create the following file structure using \[dq]standard\[dq] file name
|
|
encryption.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
plaintext/
|
|
\[u251C]\[u2500]\[u2500] file0.txt
|
|
\[u251C]\[u2500]\[u2500] file1.txt
|
|
\[u2514]\[u2500]\[u2500] subdir
|
|
\[u251C]\[u2500]\[u2500] file2.txt
|
|
\[u251C]\[u2500]\[u2500] file3.txt
|
|
\[u2514]\[u2500]\[u2500] subsubdir
|
|
\[u2514]\[u2500]\[u2500] file4.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Copy these to the remote, and list them
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -q copy plaintext secret:
|
|
$ rclone -q ls secret:
|
|
7 file1.txt
|
|
6 file0.txt
|
|
8 subdir/file2.txt
|
|
10 subdir/subsubdir/file4.txt
|
|
9 subdir/file3.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The crypt remote looks like
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -q ls remote:path
|
|
55 hagjclgavj2mbiqm6u6cnjjqcg
|
|
54 v05749mltvv1tf4onltun46gls
|
|
57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo
|
|
58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc
|
|
56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The directory structure is preserved
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -q ls secret:subdir
|
|
8 file2.txt
|
|
9 file3.txt
|
|
10 subsubdir/file4.txt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Without file name encryption \f[C].bin\f[R] extensions are added to
|
|
underlying names.
|
|
This prevents the cloud provider attempting to interpret file content.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -q ls remote:path
|
|
54 file0.txt.bin
|
|
57 subdir/file3.txt.bin
|
|
56 subdir/file2.txt.bin
|
|
58 subdir/subsubdir/file4.txt.bin
|
|
55 file1.txt.bin
|
|
\f[R]
|
|
.fi
|
|
.SS File name encryption modes
|
|
.PP
|
|
Off
|
|
.IP \[bu] 2
|
|
doesn\[aq]t hide file names or directory structure
|
|
.IP \[bu] 2
|
|
allows for longer file names (\[ti]246 characters)
|
|
.IP \[bu] 2
|
|
can use sub paths and copy single files
|
|
.PP
|
|
Standard
|
|
.IP \[bu] 2
|
|
file names encrypted
|
|
.IP \[bu] 2
|
|
file names can\[aq]t be as long (\[ti]143 characters)
|
|
.IP \[bu] 2
|
|
can use sub paths and copy single files
|
|
.IP \[bu] 2
|
|
directory structure visible
|
|
.IP \[bu] 2
|
|
identical files names will have identical uploaded names
|
|
.IP \[bu] 2
|
|
can use shortcuts to shorten the directory recursion
|
|
.PP
|
|
Obfuscation
|
|
.PP
|
|
This is a simple \[dq]rotate\[dq] of the filename, with each file having
|
|
a rot distance based on the filename.
|
|
Rclone stores the distance at the beginning of the filename.
|
|
A file called \[dq]hello\[dq] may become \[dq]53.jgnnq\[dq].
|
|
.PP
|
|
Obfuscation is not a strong encryption of filenames, but hinders
|
|
automated scanning tools picking up on filename patterns.
|
|
It is an intermediate between \[dq]off\[dq] and \[dq]standard\[dq] which
|
|
allows for longer path segment names.
|
|
.PP
|
|
There is a possibility with some unicode based filenames that the
|
|
obfuscation is weak and may map lower case characters to upper case
|
|
equivalents.
|
|
.PP
|
|
Obfuscation cannot be relied upon for strong protection.
|
|
.IP \[bu] 2
|
|
file names very lightly obfuscated
|
|
.IP \[bu] 2
|
|
file names can be longer than standard encryption
|
|
.IP \[bu] 2
|
|
can use sub paths and copy single files
|
|
.IP \[bu] 2
|
|
directory structure visible
|
|
.IP \[bu] 2
|
|
identical files names will have identical uploaded names
|
|
.PP
|
|
Cloud storage systems have limits on file name length and total path
|
|
length which rclone is more likely to breach using \[dq]Standard\[dq]
|
|
file name encryption.
|
|
Where file names are less than 156 characters in length issues should
|
|
not be encountered, irrespective of cloud storage provider.
|
|
.PP
|
|
An experimental advanced option \f[C]filename_encoding\f[R] is now
|
|
provided to address this problem to a certain degree.
|
|
For cloud storage systems with case sensitive file names (e.g.
|
|
Google Drive), \f[C]base64\f[R] can be used to reduce file name length.
|
|
For cloud storage systems using UTF-16 to store file names internally
|
|
(e.g.
|
|
OneDrive), \f[C]base32768\f[R] can be used to drastically reduce file
|
|
name length.
|
|
.PP
|
|
An alternative, future rclone file name encryption mode may tolerate
|
|
backend provider path length limits.
|
|
.SS Directory name encryption
|
|
.PP
|
|
Crypt offers the option of encrypting dir names or leaving them intact.
|
|
There are two options:
|
|
.PP
|
|
True
|
|
.PP
|
|
Encrypts the whole file path including directory names Example:
|
|
\f[C]1/12/123.txt\f[R] is encrypted to
|
|
\f[C]p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0\f[R]
|
|
.PP
|
|
False
|
|
.PP
|
|
Only encrypts file names, skips directory names Example:
|
|
\f[C]1/12/123.txt\f[R] is encrypted to
|
|
\f[C]1/12/qgm4avr35m5loi1th53ato71v0\f[R]
|
|
.SS Modified time and hashes
|
|
.PP
|
|
Crypt stores modification times using the underlying remote so support
|
|
depends on that.
|
|
.PP
|
|
Hashes are not stored for crypt.
|
|
However the data integrity is protected by an extremely strong crypto
|
|
authenticator.
|
|
.PP
|
|
Use the \f[C]rclone cryptcheck\f[R] command to check the integrity of a
|
|
crypted remote instead of \f[C]rclone check\f[R] which can\[aq]t check
|
|
the checksums properly.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to crypt (Encrypt/Decrypt a
|
|
remote).
|
|
.SS --crypt-remote
|
|
.PP
|
|
Remote to encrypt/decrypt.
|
|
.PP
|
|
Normally should contain a \[aq]:\[aq] and a path, e.g.
|
|
\[dq]myremote:path/to/dir\[dq], \[dq]myremote:bucket\[dq] or maybe
|
|
\[dq]myremote:\[dq] (not recommended).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: remote
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CRYPT_REMOTE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --crypt-filename-encryption
|
|
.PP
|
|
How to encrypt the filenames.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: filename_encryption
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CRYPT_FILENAME_ENCRYPTION
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]standard\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]standard\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Encrypt the filenames.
|
|
.IP \[bu] 2
|
|
See the docs for the details.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]obfuscate\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Very simple filename obfuscation.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]off\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t encrypt the file names.
|
|
.IP \[bu] 2
|
|
Adds a \[dq].bin\[dq] extension only.
|
|
.RE
|
|
.RE
|
|
.SS --crypt-directory-name-encryption
|
|
.PP
|
|
Option to either encrypt directory names or leave them intact.
|
|
.PP
|
|
NB If filename_encryption is \[dq]off\[dq] then this option will do
|
|
nothing.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: directory_name_encryption
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CRYPT_DIRECTORY_NAME_ENCRYPTION
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: true
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]true\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Encrypt directory names.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]false\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t encrypt directory names, leave them intact.
|
|
.RE
|
|
.RE
|
|
.SS --crypt-password
|
|
.PP
|
|
Password or pass phrase for encryption.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CRYPT_PASSWORD
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --crypt-password2
|
|
.PP
|
|
Password or pass phrase for salt.
|
|
.PP
|
|
Optional but recommended.
|
|
Should be different to the previous password.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: password2
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CRYPT_PASSWORD2
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to crypt (Encrypt/Decrypt a
|
|
remote).
|
|
.SS --crypt-server-side-across-configs
|
|
.PP
|
|
Allow server-side operations (e.g.
|
|
copy) to work across different crypt configs.
|
|
.PP
|
|
Normally this option is not what you want, but if you have two crypts
|
|
pointing to the same backend you can use it.
|
|
.PP
|
|
This can be used, for example, to change file name encryption type
|
|
without re-uploading all the data.
|
|
Just make two crypt backends pointing to two different directories with
|
|
the single changed parameter and use rclone move to move the files
|
|
between the crypt remotes.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: server_side_across_configs
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --crypt-show-mapping
|
|
.PP
|
|
For all files listed show how the names encrypt.
|
|
.PP
|
|
If this flag is set then for each file that the remote is asked to list,
|
|
it will log (at level INFO) a line stating the decrypted file name and
|
|
the encrypted file name.
|
|
.PP
|
|
This is so you can work out which encrypted names are which decrypted
|
|
names just in case you need to do something with the encrypted file
|
|
names, or for debugging purposes.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: show_mapping
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CRYPT_SHOW_MAPPING
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --crypt-no-data-encryption
|
|
.PP
|
|
Option to either encrypt file data or leave it unencrypted.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_data_encryption
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]true\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t encrypt file data, leave it unencrypted.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]false\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Encrypt file data.
|
|
.RE
|
|
.RE
|
|
.SS --crypt-filename-encoding
|
|
.PP
|
|
How to encode the encrypted filename to text string.
|
|
.PP
|
|
This option could help with shortening the encrypted filename.
|
|
The suitable option would depend on the way your remote count the
|
|
filename length and if it\[aq]s case sensitve.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: filename_encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_CRYPT_FILENAME_ENCODING
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]base32\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]base32\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Encode using base32.
|
|
Suitable for all remote.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]base64\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Encode using base64.
|
|
Suitable for case sensitive remote.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]base32768\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Encode using base32768.
|
|
Suitable if your remote counts UTF-16 or
|
|
.IP \[bu] 2
|
|
Unicode codepoint instead of UTF-8 byte length.
|
|
(Eg.
|
|
Onedrive)
|
|
.RE
|
|
.RE
|
|
.SS Backend commands
|
|
.PP
|
|
Here are the commands specific to the crypt backend.
|
|
.PP
|
|
Run them with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend COMMAND remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The help below will explain what arguments each command takes.
|
|
.PP
|
|
See the \[dq]rclone backend\[dq]
|
|
command (https://rclone.org/commands/rclone_backend/) for more info on
|
|
how to pass options and arguments.
|
|
.PP
|
|
These can be run on a running backend using the rc command
|
|
backend/command (https://rclone.org/rc/#backend-command).
|
|
.SS encode
|
|
.PP
|
|
Encode the given filename(s)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend encode remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This encodes the filenames given as arguments returning a list of
|
|
strings of the encoded results.
|
|
.PP
|
|
Usage Example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend encode crypt: file1 [file2...]
|
|
rclone rc backend/command command=encode fs=crypt: file1 [file2...]
|
|
\f[R]
|
|
.fi
|
|
.SS decode
|
|
.PP
|
|
Decode the given filename(s)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend decode remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This decodes the filenames given as arguments returning a list of
|
|
strings of the decoded results.
|
|
It will return an error if any of the inputs are invalid.
|
|
.PP
|
|
Usage Example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend decode crypt: encryptedfile1 [encryptedfile2...]
|
|
rclone rc backend/command command=decode fs=crypt: encryptedfile1 [encryptedfile2...]
|
|
\f[R]
|
|
.fi
|
|
.SS Backing up a crypted remote
|
|
.PP
|
|
If you wish to backup a crypted remote, it is recommended that you use
|
|
\f[C]rclone sync\f[R] on the encrypted files, and make sure the
|
|
passwords are the same in the new encrypted remote.
|
|
.PP
|
|
This will have the following advantages
|
|
.IP \[bu] 2
|
|
\f[C]rclone sync\f[R] will check the checksums while copying
|
|
.IP \[bu] 2
|
|
you can use \f[C]rclone check\f[R] between the encrypted remotes
|
|
.IP \[bu] 2
|
|
you don\[aq]t decrypt and encrypt unnecessarily
|
|
.PP
|
|
For example, let\[aq]s say you have your original remote at
|
|
\f[C]remote:\f[R] with the encrypted version at \f[C]eremote:\f[R] with
|
|
path \f[C]remote:crypt\f[R].
|
|
You would then set up the new remote \f[C]remote2:\f[R] and then the
|
|
encrypted version \f[C]eremote2:\f[R] with path \f[C]remote2:crypt\f[R]
|
|
using the same passwords as \f[C]eremote:\f[R].
|
|
.PP
|
|
To sync the two remotes you would do
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i remote:crypt remote2:crypt
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And to check the integrity you would do
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone check remote:crypt remote2:crypt
|
|
\f[R]
|
|
.fi
|
|
.SS File formats
|
|
.SS File encryption
|
|
.PP
|
|
Files are encrypted 1:1 source file to destination object.
|
|
The file has a header and is divided into chunks.
|
|
.SS Header
|
|
.IP \[bu] 2
|
|
8 bytes magic string \f[C]RCLONE\[rs]x00\[rs]x00\f[R]
|
|
.IP \[bu] 2
|
|
24 bytes Nonce (IV)
|
|
.PP
|
|
The initial nonce is generated from the operating systems crypto strong
|
|
random number generator.
|
|
The nonce is incremented for each chunk read making sure each nonce is
|
|
unique for each block written.
|
|
The chance of a nonce being re-used is minuscule.
|
|
If you wrote an exabyte of data (10\[S1]\[u2078] bytes) you would have a
|
|
probability of approximately 2\[tmu]10\[u207B]\[S3]\[S2] of re-using a
|
|
nonce.
|
|
.SS Chunk
|
|
.PP
|
|
Each chunk will contain 64 KiB of data, except for the last one which
|
|
may have less data.
|
|
The data chunk is in standard NaCl SecretBox format.
|
|
SecretBox uses XSalsa20 and Poly1305 to encrypt and authenticate
|
|
messages.
|
|
.PP
|
|
Each chunk contains:
|
|
.IP \[bu] 2
|
|
16 Bytes of Poly1305 authenticator
|
|
.IP \[bu] 2
|
|
1 - 65536 bytes XSalsa20 encrypted data
|
|
.PP
|
|
64k chunk size was chosen as the best performing chunk size (the
|
|
authenticator takes too much time below this and the performance drops
|
|
off due to cache effects above this).
|
|
Note that these chunks are buffered in memory so they can\[aq]t be too
|
|
big.
|
|
.PP
|
|
This uses a 32 byte (256 bit key) key derived from the user password.
|
|
.SS Examples
|
|
.PP
|
|
1 byte file will encrypt to
|
|
.IP \[bu] 2
|
|
32 bytes header
|
|
.IP \[bu] 2
|
|
17 bytes data chunk
|
|
.PP
|
|
49 bytes total
|
|
.PP
|
|
1 MiB (1048576 bytes) file will encrypt to
|
|
.IP \[bu] 2
|
|
32 bytes header
|
|
.IP \[bu] 2
|
|
16 chunks of 65568 bytes
|
|
.PP
|
|
1049120 bytes total (a 0.05% overhead).
|
|
This is the overhead for big files.
|
|
.SS Name encryption
|
|
.PP
|
|
File names are encrypted segment by segment - the path is broken up into
|
|
\f[C]/\f[R] separated strings and these are encrypted individually.
|
|
.PP
|
|
File segments are padded using PKCS#7 to a multiple of 16 bytes before
|
|
encryption.
|
|
.PP
|
|
They are then encrypted with EME using AES with 256 bit key.
|
|
EME (ECB-Mix-ECB) is a wide-block encryption mode presented in the 2003
|
|
paper \[dq]A Parallelizable Enciphering Mode\[dq] by Halevi and Rogaway.
|
|
.PP
|
|
This makes for deterministic encryption which is what we want - the same
|
|
filename must encrypt to the same thing otherwise we can\[aq]t find it
|
|
on the cloud storage system.
|
|
.PP
|
|
This means that
|
|
.IP \[bu] 2
|
|
filenames with the same name will encrypt the same
|
|
.IP \[bu] 2
|
|
filenames which start the same won\[aq]t have a common prefix
|
|
.PP
|
|
This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of
|
|
which are derived from the user password.
|
|
.PP
|
|
After encryption they are written out using a modified version of
|
|
standard \f[C]base32\f[R] encoding as described in RFC4648.
|
|
The standard encoding is modified in two ways:
|
|
.IP \[bu] 2
|
|
it becomes lower case (no-one likes upper case filenames!)
|
|
.IP \[bu] 2
|
|
we strip the padding character \f[C]=\f[R]
|
|
.PP
|
|
\f[C]base32\f[R] is used rather than the more efficient \f[C]base64\f[R]
|
|
so rclone can be used on case insensitive remotes (e.g.
|
|
Windows, Amazon Drive).
|
|
.SS Key derivation
|
|
.PP
|
|
Rclone uses \f[C]scrypt\f[R] with parameters \f[C]N=16384, r=8, p=1\f[R]
|
|
with an optional user supplied salt (password2) to derive the 32+32+16 =
|
|
80 bytes of key material required.
|
|
If the user doesn\[aq]t supply a salt then rclone uses an internal one.
|
|
.PP
|
|
\f[C]scrypt\f[R] makes it impractical to mount a dictionary attack on
|
|
rclone encrypted data.
|
|
For full protection against this you should always use a salt.
|
|
.SS SEE ALSO
|
|
.IP \[bu] 2
|
|
rclone cryptdecode (https://rclone.org/commands/rclone_cryptdecode/) -
|
|
Show forward/reverse mapping of encrypted filenames
|
|
.SH Compress (Experimental)
|
|
.SS Warning
|
|
.PP
|
|
This remote is currently \f[B]experimental\f[R].
|
|
Things may break and data may be lost.
|
|
Anything you do with this remote is at your own risk.
|
|
Please understand the risks associated with using experimental code and
|
|
don\[aq]t use this remote in critical applications.
|
|
.PP
|
|
The \f[C]Compress\f[R] remote adds compression to another remote.
|
|
It is best used with remotes containing many large compressible files.
|
|
.SS Configuration
|
|
.PP
|
|
To use this remote, all you need to do is specify another remote and a
|
|
compression mode to use:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Current remotes:
|
|
|
|
Name Type
|
|
==== ====
|
|
remote_to_press sometype
|
|
|
|
e) Edit existing remote
|
|
$ rclone config
|
|
n) New remote
|
|
d) Delete remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/r/c/s/q> n
|
|
name> compress
|
|
\&...
|
|
8 / Compress a remote
|
|
\[rs] \[dq]compress\[dq]
|
|
\&...
|
|
Storage> compress
|
|
** See help for compress backend at: https://rclone.org/compress/ **
|
|
|
|
Remote to compress.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
remote> remote_to_press:subdir
|
|
Compression mode.
|
|
Enter a string value. Press Enter for the default (\[dq]gzip\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Gzip compression balanced for speed and compression strength.
|
|
\[rs] \[dq]gzip\[dq]
|
|
compression_mode> gzip
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
Remote config
|
|
--------------------
|
|
[compress]
|
|
type = compress
|
|
remote = remote_to_press:subdir
|
|
compression_mode = gzip
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.SS Compression Modes
|
|
.PP
|
|
Currently only gzip compression is supported.
|
|
It provides a decent balance between speed and size and is well
|
|
supported by other applications.
|
|
Compression strength can further be configured via an advanced setting
|
|
where 0 is no compression and 9 is strongest compression.
|
|
.SS File types
|
|
.PP
|
|
If you open a remote wrapped by compress, you will see that there are
|
|
many files with an extension corresponding to the compression algorithm
|
|
you chose.
|
|
These files are standard files that can be opened by various archive
|
|
programs, but they have some hidden metadata that allows them to be used
|
|
by rclone.
|
|
While you may download and decompress these files at will, do
|
|
\f[B]not\f[R] manually delete or rename files.
|
|
Files without correct metadata files will not be recognized by rclone.
|
|
.SS File names
|
|
.PP
|
|
The compressed files will be named \f[C]*.###########.gz\f[R] where
|
|
\f[C]*\f[R] is the base file and the \f[C]#\f[R] part is base64 encoded
|
|
size of the uncompressed file.
|
|
The file names should not be changed by anything other than the rclone
|
|
compression backend.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to compress (Compress a remote).
|
|
.SS --compress-remote
|
|
.PP
|
|
Remote to compress.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: remote
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_COMPRESS_REMOTE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --compress-mode
|
|
.PP
|
|
Compression mode.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: mode
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_COMPRESS_MODE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]gzip\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]gzip\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Standard gzip compression with fastest parameters.
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to compress (Compress a remote).
|
|
.SS --compress-level
|
|
.PP
|
|
GZIP compression level (-2 to 9).
|
|
.PP
|
|
Generally -1 (default, equivalent to 5) is recommended.
|
|
Levels 1 to 9 increase compression at the cost of speed.
|
|
Going past 6 generally offers very little return.
|
|
.PP
|
|
Level -2 uses Huffmann encoding only.
|
|
Only use if you know what you are doing.
|
|
Level 0 turns off compression.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: level
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_COMPRESS_LEVEL
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: -1
|
|
.SS --compress-ram-cache-limit
|
|
.PP
|
|
Some remotes don\[aq]t allow the upload of files with unknown size.
|
|
In this case the compressed file will need to be cached to determine
|
|
it\[aq]s size.
|
|
.PP
|
|
Files smaller than this limit will be cached in RAM, files larger than
|
|
this limit will be cached on disk.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: ram_cache_limit
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_COMPRESS_RAM_CACHE_LIMIT
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 20Mi
|
|
.SH Dropbox
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Dropbox paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for dropbox involves getting a token from Dropbox
|
|
which you need to do in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n) New remote
|
|
d) Delete remote
|
|
q) Quit config
|
|
e/n/d/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Dropbox
|
|
\[rs] \[dq]dropbox\[dq]
|
|
[snip]
|
|
Storage> dropbox
|
|
Dropbox App Key - leave blank normally.
|
|
app_key>
|
|
Dropbox App Secret - leave blank normally.
|
|
app_secret>
|
|
Remote config
|
|
Please visit:
|
|
https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code
|
|
Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX
|
|
--------------------
|
|
[remote]
|
|
app_key =
|
|
app_secret =
|
|
token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can then use it like this,
|
|
.PP
|
|
List directories in top level of your dropbox
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your dropbox
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to a dropbox directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Dropbox for business
|
|
.PP
|
|
Rclone supports Dropbox for business and Team Folders.
|
|
.PP
|
|
When using Dropbox for business \f[C]remote:\f[R] and
|
|
\f[C]remote:path/to/file\f[R] will refer to your personal folder.
|
|
.PP
|
|
If you wish to see Team Folders you must use a leading \f[C]/\f[R] in
|
|
the path, so \f[C]rclone lsd remote:/\f[R] will refer to the root and
|
|
show you all Team Folders and your User Folder.
|
|
.PP
|
|
You can then use team folders like this \f[C]remote:/TeamFolder\f[R] and
|
|
\f[C]remote:/TeamFolder/path/to/file\f[R].
|
|
.PP
|
|
A leading \f[C]/\f[R] for a Dropbox personal account will do nothing,
|
|
but it will take an extra HTTP transaction so it should be avoided.
|
|
.SS Modified time and Hashes
|
|
.PP
|
|
Dropbox supports modified times, but the only way to set a modification
|
|
time is to re-upload the file.
|
|
.PP
|
|
This means that if you uploaded your data with an older version of
|
|
rclone which didn\[aq]t support the v2 API and modified times, rclone
|
|
will decide to upload all your old data to fix the modification times.
|
|
If you don\[aq]t want this to happen use \f[C]--size-only\f[R] or
|
|
\f[C]--checksum\f[R] flag to stop it.
|
|
.PP
|
|
Dropbox supports its own hash
|
|
type (https://www.dropbox.com/developers/reference/content-hash) which
|
|
is checked for all transfers.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
NUL
|
|
T}@T{
|
|
0x00
|
|
T}@T{
|
|
\[u2400]
|
|
T}
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
T{
|
|
DEL
|
|
T}@T{
|
|
0x7F
|
|
T}@T{
|
|
\[u2421]
|
|
T}
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
File names can also not end with the following characters.
|
|
These only get replaced if they are the last character in the name:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
SP
|
|
T}@T{
|
|
0x20
|
|
T}@T{
|
|
\[u2420]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Batch mode uploads
|
|
.PP
|
|
Using batch mode uploads is very important for performance when using
|
|
the Dropbox API.
|
|
See the dropbox performance
|
|
guide (https://developers.dropbox.com/dbx-performance-guide) for more
|
|
info.
|
|
.PP
|
|
There are 3 modes rclone can use for uploads.
|
|
.SS --dropbox-batch-mode off
|
|
.PP
|
|
In this mode rclone will not use upload batching.
|
|
This was the default before rclone v1.55.
|
|
It has the disadvantage that it is very likely to encounter
|
|
\f[C]too_many_requests\f[R] errors like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 15 seconds.
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
When rclone receives these it has to wait for 15s or sometimes 300s
|
|
before continuing which really slows down transfers.
|
|
.PP
|
|
This will happen especially if \f[C]--transfers\f[R] is large, so this
|
|
mode isn\[aq]t recommended except for compatibility or investigating
|
|
problems.
|
|
.SS --dropbox-batch-mode sync
|
|
.PP
|
|
In this mode rclone will batch up uploads to the size specified by
|
|
\f[C]--dropbox-batch-size\f[R] and commit them together.
|
|
.PP
|
|
Using this mode means you can use a much higher \f[C]--transfers\f[R]
|
|
parameter (32 or 64 works fine) without receiving
|
|
\f[C]too_many_requests\f[R] errors.
|
|
.PP
|
|
This mode ensures full data integrity.
|
|
.PP
|
|
Note that there may be a pause when quitting rclone while rclone
|
|
finishes up the last batch using this mode.
|
|
.SS --dropbox-batch-mode async
|
|
.PP
|
|
In this mode rclone will batch up uploads to the size specified by
|
|
\f[C]--dropbox-batch-size\f[R] and commit them together.
|
|
.PP
|
|
However it will not wait for the status of the batch to be returned to
|
|
the caller.
|
|
This means rclone can use a much bigger batch size (much bigger than
|
|
\f[C]--transfers\f[R]), at the cost of not being able to check the
|
|
status of the upload.
|
|
.PP
|
|
This provides the maximum possible upload speed especially with lots of
|
|
small files, however rclone can\[aq]t check the file got uploaded
|
|
properly using this mode.
|
|
.PP
|
|
If you are using this mode then using \[dq]rclone check\[dq] after the
|
|
transfer completes is recommended.
|
|
Or you could do an initial transfer with
|
|
\f[C]--dropbox-batch-mode async\f[R] then do a final transfer with
|
|
\f[C]--dropbox-batch-mode sync\f[R] (the default).
|
|
.PP
|
|
Note that there may be a pause when quitting rclone while rclone
|
|
finishes up the last batch using this mode.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to dropbox (Dropbox).
|
|
.SS --dropbox-client-id
|
|
.PP
|
|
OAuth Client Id.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --dropbox-client-secret
|
|
.PP
|
|
OAuth Client Secret.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_CLIENT_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to dropbox (Dropbox).
|
|
.SS --dropbox-token
|
|
.PP
|
|
OAuth Access Token as a JSON blob.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --dropbox-auth-url
|
|
.PP
|
|
Auth server URL.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_AUTH_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --dropbox-token-url
|
|
.PP
|
|
Token server url.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_TOKEN_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --dropbox-chunk-size
|
|
.PP
|
|
Upload chunk size (< 150Mi).
|
|
.PP
|
|
Any files larger than this will be uploaded in chunks of this size.
|
|
.PP
|
|
Note that chunks are buffered in memory (one at a time) so rclone can
|
|
deal with retries.
|
|
Setting this larger will increase the speed slightly (at most 10% for
|
|
128 MiB in tests) at the cost of using more memory.
|
|
It can be set smaller if you are tight on memory.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 48Mi
|
|
.SS --dropbox-impersonate
|
|
.PP
|
|
Impersonate this user when using a business account.
|
|
.PP
|
|
Note that if you want to use impersonate, you should make sure this flag
|
|
is set when running \[dq]rclone config\[dq] as this will cause rclone to
|
|
request the \[dq]members.read\[dq] scope which it won\[aq]t normally.
|
|
This is needed to lookup a members email address into the internal ID
|
|
that dropbox uses in the API.
|
|
.PP
|
|
Using the \[dq]members.read\[dq] scope will require a Dropbox Team Admin
|
|
to approve during the OAuth flow.
|
|
.PP
|
|
You will have to use your own App (setting your own client_id and
|
|
client_secret) to use this option as currently rclone\[aq]s default set
|
|
of permissions doesn\[aq]t include \[dq]members.read\[dq].
|
|
This can be added once v1.55 or later is in use everywhere.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: impersonate
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_IMPERSONATE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --dropbox-shared-files
|
|
.PP
|
|
Instructs rclone to work on individual shared files.
|
|
.PP
|
|
In this mode rclone\[aq]s features are extremely limited - only list
|
|
(ls, lsl, etc.) operations and read operations (e.g.
|
|
downloading) are supported in this mode.
|
|
All other operations will be disabled.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: shared_files
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_SHARED_FILES
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --dropbox-shared-folders
|
|
.PP
|
|
Instructs rclone to work on shared folders.
|
|
.PP
|
|
When this flag is used with no path only the List operation is supported
|
|
and all available shared folders will be listed.
|
|
If you specify a path the first part will be interpreted as the name of
|
|
shared folder.
|
|
Rclone will then try to mount this shared to the root namespace.
|
|
On success shared folder rclone proceeds normally.
|
|
The shared folder is now pretty much a normal folder and all normal
|
|
operations are supported.
|
|
.PP
|
|
Note that we don\[aq]t unmount the shared folder afterwards so the
|
|
--dropbox-shared-folders can be omitted after the first use of a
|
|
particular shared folder.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: shared_folders
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_SHARED_FOLDERS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --dropbox-batch-mode
|
|
.PP
|
|
Upload file batching sync|async|off.
|
|
.PP
|
|
This sets the batch mode used by rclone.
|
|
.PP
|
|
For full info see the main docs (https://rclone.org/dropbox/#batch-mode)
|
|
.PP
|
|
This has 3 possible values
|
|
.IP \[bu] 2
|
|
off - no batching
|
|
.IP \[bu] 2
|
|
sync - batch uploads and check completion (default)
|
|
.IP \[bu] 2
|
|
async - batch upload and don\[aq]t check completion
|
|
.PP
|
|
Rclone will close any outstanding batches when it exits which may make a
|
|
delay on quit.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: batch_mode
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_BATCH_MODE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]sync\[dq]
|
|
.SS --dropbox-batch-size
|
|
.PP
|
|
Max number of files in upload batch.
|
|
.PP
|
|
This sets the batch size of files to upload.
|
|
It has to be less than 1000.
|
|
.PP
|
|
By default this is 0 which means rclone which calculate the batch size
|
|
depending on the setting of batch_mode.
|
|
.IP \[bu] 2
|
|
batch_mode: async - default batch_size is 100
|
|
.IP \[bu] 2
|
|
batch_mode: sync - default batch_size is the same as --transfers
|
|
.IP \[bu] 2
|
|
batch_mode: off - not in use
|
|
.PP
|
|
Rclone will close any outstanding batches when it exits which may make a
|
|
delay on quit.
|
|
.PP
|
|
Setting this is a great idea if you are uploading lots of small files as
|
|
it will make them a lot quicker.
|
|
You can use --transfers 32 to maximise throughput.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: batch_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_BATCH_SIZE
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 0
|
|
.SS --dropbox-batch-timeout
|
|
.PP
|
|
Max time to allow an idle upload batch before uploading.
|
|
.PP
|
|
If an upload batch is idle for more than this long then it will be
|
|
uploaded.
|
|
.PP
|
|
The default for this is 0 which means rclone will choose a sensible
|
|
default based on the batch_mode in use.
|
|
.IP \[bu] 2
|
|
batch_mode: async - default batch_timeout is 500ms
|
|
.IP \[bu] 2
|
|
batch_mode: sync - default batch_timeout is 10s
|
|
.IP \[bu] 2
|
|
batch_mode: off - not in use
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: batch_timeout
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_BATCH_TIMEOUT
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 0s
|
|
.SS --dropbox-batch-commit-timeout
|
|
.PP
|
|
Max time to wait for a batch to finish comitting
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: batch_commit_timeout
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_BATCH_COMMIT_TIMEOUT
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 10m0s
|
|
.SS --dropbox-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DROPBOX_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
Note that Dropbox is case insensitive so you can\[aq]t have a file
|
|
called \[dq]Hello.doc\[dq] and one called \[dq]hello.doc\[dq].
|
|
.PP
|
|
There are some file names such as \f[C]thumbs.db\f[R] which Dropbox
|
|
can\[aq]t store.
|
|
There is a full list of them in the \[dq]Ignored Files\[dq] section of
|
|
this document (https://www.dropbox.com/en/help/145).
|
|
Rclone will issue an error message
|
|
\f[C]File name disallowed - not uploading\f[R] if it attempts to upload
|
|
one of those file names, but the sync won\[aq]t fail.
|
|
.PP
|
|
Some errors may occur if you try to sync copyright-protected files
|
|
because Dropbox has its own copyright
|
|
detector (https://techcrunch.com/2014/03/30/how-dropbox-knows-when-youre-sharing-copyrighted-stuff-without-actually-looking-at-your-stuff/)
|
|
that prevents this sort of file being downloaded.
|
|
This will return the error
|
|
\f[C]ERROR : /path/to/your/file: Failed to copy: failed to open source object: path/restricted_content/.\f[R]
|
|
.PP
|
|
If you have more than 10,000 files in a directory then
|
|
\f[C]rclone purge dropbox:dir\f[R] will return the error
|
|
\f[C]Failed to purge: There are too many files involved in this operation\f[R].
|
|
As a work-around do an \f[C]rclone delete dropbox:dir\f[R] followed by
|
|
an \f[C]rclone rmdir dropbox:dir\f[R].
|
|
.PP
|
|
When using \f[C]rclone link\f[R] you\[aq]ll need to set
|
|
\f[C]--expire\f[R] if using a non-personal account otherwise the
|
|
visibility may not be correct.
|
|
(Note that \f[C]--expire\f[R] isn\[aq]t supported on personal accounts).
|
|
See the forum
|
|
discussion (https://forum.rclone.org/t/rclone-link-dropbox-permissions/23211)
|
|
and the dropbox SDK
|
|
issue (https://github.com/dropbox/dropbox-sdk-go-unofficial/issues/75).
|
|
.SS Get your own Dropbox App ID
|
|
.PP
|
|
When you use rclone with Dropbox in its default configuration you are
|
|
using rclone\[aq]s App ID.
|
|
This is shared between all the rclone users.
|
|
.PP
|
|
Here is how to create your own Dropbox App ID for rclone:
|
|
.IP "1." 3
|
|
Log into the Dropbox App
|
|
console (https://www.dropbox.com/developers/apps/create) with your
|
|
Dropbox Account (It need not to be the same account as the Dropbox you
|
|
want to access)
|
|
.IP "2." 3
|
|
Choose an API => Usually this should be \f[C]Dropbox API\f[R]
|
|
.IP "3." 3
|
|
Choose the type of access you want to use => \f[C]Full Dropbox\f[R] or
|
|
\f[C]App Folder\f[R]
|
|
.IP "4." 3
|
|
Name your App.
|
|
The app name is global, so you can\[aq]t use \f[C]rclone\f[R] for
|
|
example
|
|
.IP "5." 3
|
|
Click the button \f[C]Create App\f[R]
|
|
.IP "6." 3
|
|
Switch to the \f[C]Permissions\f[R] tab.
|
|
Enable at least the following permissions: \f[C]account_info.read\f[R],
|
|
\f[C]files.metadata.write\f[R], \f[C]files.content.write\f[R],
|
|
\f[C]files.content.read\f[R], \f[C]sharing.write\f[R].
|
|
The \f[C]files.metadata.read\f[R] and \f[C]sharing.read\f[R] checkboxes
|
|
will be marked too.
|
|
Click \f[C]Submit\f[R]
|
|
.IP "7." 3
|
|
Switch to the \f[C]Settings\f[R] tab.
|
|
Fill \f[C]OAuth2 - Redirect URIs\f[R] as
|
|
\f[C]http://localhost:53682/\f[R]
|
|
.IP "8." 3
|
|
Find the \f[C]App key\f[R] and \f[C]App secret\f[R] values on the
|
|
\f[C]Settings\f[R] tab.
|
|
Use these values in rclone config to add a new remote or edit an
|
|
existing remote.
|
|
The \f[C]App key\f[R] setting corresponds to \f[C]client_id\f[R] in
|
|
rclone config, the \f[C]App secret\f[R] corresponds to
|
|
\f[C]client_secret\f[R]
|
|
.SH Enterprise File Fabric
|
|
.PP
|
|
This backend supports Storage Made Easy\[aq]s Enterprise File
|
|
Fabric\[tm] (https://storagemadeeasy.com/about/) which provides a
|
|
software solution to integrate and unify File and Object Storage
|
|
accessible through a global file system.
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for the Enterprise File Fabric backend involves
|
|
getting a token from the the Enterprise File Fabric which you need to do
|
|
in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Enterprise File Fabric
|
|
\[rs] \[dq]filefabric\[dq]
|
|
[snip]
|
|
Storage> filefabric
|
|
** See help for filefabric backend at: https://rclone.org/filefabric/ **
|
|
|
|
URL of the Enterprise File Fabric to connect to
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Storage Made Easy US
|
|
\[rs] \[dq]https://storagemadeeasy.com\[dq]
|
|
2 / Storage Made Easy EU
|
|
\[rs] \[dq]https://eu.storagemadeeasy.com\[dq]
|
|
3 / Connect to your Enterprise File Fabric
|
|
\[rs] \[dq]https://yourfabric.smestorage.com\[dq]
|
|
url> https://yourfabric.smestorage.com/
|
|
ID of the root folder
|
|
Leave blank normally.
|
|
|
|
Fill in to make rclone start with directory of a given ID.
|
|
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
root_folder_id>
|
|
Permanent Authentication Token
|
|
|
|
A Permanent Authentication Token can be created in the Enterprise File
|
|
Fabric, on the users Dashboard under Security, there is an entry
|
|
you\[aq]ll see called \[dq]My Authentication Tokens\[dq]. Click the Manage button
|
|
to create one.
|
|
|
|
These tokens are normally valid for several years.
|
|
|
|
For more info see: https://docs.storagemadeeasy.com/organisationcloud/api-tokens
|
|
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
permanent_token> xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
type = filefabric
|
|
url = https://yourfabric.smestorage.com/
|
|
permanent_token = xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your Enterprise File Fabric
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your Enterprise File Fabric
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Enterprise File Fabric directory called
|
|
backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
The Enterprise File Fabric allows modification times to be set on files
|
|
accurate to 1 second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
.PP
|
|
The Enterprise File Fabric does not support any data hashes at this
|
|
time.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
The default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) will be
|
|
replaced.
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Empty files
|
|
.PP
|
|
Empty files aren\[aq]t supported by the Enterprise File Fabric.
|
|
Rclone will therefore upload an empty file as a single space with a mime
|
|
type of \f[C]application/vnd.rclone.empty.file\f[R] and files with that
|
|
mime type are treated as empty.
|
|
.SS Root folder ID
|
|
.PP
|
|
You can set the \f[C]root_folder_id\f[R] for rclone.
|
|
This is the directory (identified by its \f[C]Folder ID\f[R]) that
|
|
rclone considers to be the root of your Enterprise File Fabric.
|
|
.PP
|
|
Normally you will leave this blank and rclone will determine the correct
|
|
root to use itself.
|
|
.PP
|
|
However you can set this to restrict rclone to a specific folder
|
|
hierarchy.
|
|
.PP
|
|
In order to do this you will have to find the \f[C]Folder ID\f[R] of the
|
|
directory you wish rclone to display.
|
|
These aren\[aq]t displayed in the web interface, but you can use
|
|
\f[C]rclone lsf\f[R] to find them, for example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsf --dirs-only -Fip --csv filefabric:
|
|
120673758,Burnt PDFs/
|
|
120673759,My Quick Uploads/
|
|
120673755,My Syncs/
|
|
120673756,My backups/
|
|
120673757,My contacts/
|
|
120673761,S3 Storage/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The ID for \[dq]S3 Storage\[dq] would be \f[C]120673761\f[R].
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to filefabric (Enterprise File
|
|
Fabric).
|
|
.SS --filefabric-url
|
|
.PP
|
|
URL of the Enterprise File Fabric to connect to.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FILEFABRIC_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]https://storagemadeeasy.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Storage Made Easy US
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]https://eu.storagemadeeasy.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Storage Made Easy EU
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]https://yourfabric.smestorage.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Connect to your Enterprise File Fabric
|
|
.RE
|
|
.RE
|
|
.SS --filefabric-root-folder-id
|
|
.PP
|
|
ID of the root folder.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Fill in to make rclone start with directory of a given ID.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: root_folder_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FILEFABRIC_ROOT_FOLDER_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --filefabric-permanent-token
|
|
.PP
|
|
Permanent Authentication Token.
|
|
.PP
|
|
A Permanent Authentication Token can be created in the Enterprise File
|
|
Fabric, on the users Dashboard under Security, there is an entry
|
|
you\[aq]ll see called \[dq]My Authentication Tokens\[dq].
|
|
Click the Manage button to create one.
|
|
.PP
|
|
These tokens are normally valid for several years.
|
|
.PP
|
|
For more info see:
|
|
https://docs.storagemadeeasy.com/organisationcloud/api-tokens
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: permanent_token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FILEFABRIC_PERMANENT_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to filefabric (Enterprise File
|
|
Fabric).
|
|
.SS --filefabric-token
|
|
.PP
|
|
Session Token.
|
|
.PP
|
|
This is a session token which rclone caches in the config file.
|
|
It is usually valid for 1 hour.
|
|
.PP
|
|
Don\[aq]t set this value - rclone will set it automatically.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FILEFABRIC_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --filefabric-token-expiry
|
|
.PP
|
|
Token expiry time.
|
|
.PP
|
|
Don\[aq]t set this value - rclone will set it automatically.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_expiry
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FILEFABRIC_TOKEN_EXPIRY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --filefabric-version
|
|
.PP
|
|
Version read from the file fabric.
|
|
.PP
|
|
Don\[aq]t set this value - rclone will set it automatically.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: version
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FILEFABRIC_VERSION
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --filefabric-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FILEFABRIC_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,Del,Ctl,InvalidUtf8,Dot
|
|
.SH FTP
|
|
.PP
|
|
FTP is the File Transfer Protocol.
|
|
Rclone FTP support is provided using the
|
|
github.com/jlaffaye/ftp (https://godoc.org/github.com/jlaffaye/ftp)
|
|
package.
|
|
.PP
|
|
Limitations of Rclone\[aq]s FTP backend
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R].
|
|
If the path does not begin with a \f[C]/\f[R] it is relative to the home
|
|
directory of the user.
|
|
An empty path \f[C]remote:\f[R] refers to the user\[aq]s home directory.
|
|
.SS Configuration
|
|
.PP
|
|
To create an FTP configuration named \f[C]remote\f[R], run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Rclone config guides you through an interactive setup process.
|
|
A minimal rclone FTP remote definition only requires host, username and
|
|
password.
|
|
For an anonymous FTP server, use \f[C]anonymous\f[R] as username and
|
|
your email address as password.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/r/c/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / FTP Connection
|
|
\[rs] \[dq]ftp\[dq]
|
|
[snip]
|
|
Storage> ftp
|
|
** See help for ftp backend at: https://rclone.org/ftp/ **
|
|
|
|
FTP host to connect to
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Connect to ftp.example.com
|
|
\[rs] \[dq]ftp.example.com\[dq]
|
|
host> ftp.example.com
|
|
FTP username
|
|
Enter a string value. Press Enter for the default (\[dq]$USER\[dq]).
|
|
user>
|
|
FTP port number
|
|
Enter a signed integer. Press Enter for the default (21).
|
|
port>
|
|
FTP password
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
y/g> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
Use FTP over TLS (Implicit)
|
|
Enter a boolean value (true or false). Press Enter for the default (\[dq]false\[dq]).
|
|
tls>
|
|
Use FTP over TLS (Explicit)
|
|
Enter a boolean value (true or false). Press Enter for the default (\[dq]false\[dq]).
|
|
explicit_tls>
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
type = ftp
|
|
host = ftp.example.com
|
|
pass = *** ENCRYPTED ***
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To see all directories in the home directory of \f[C]remote\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Make a new directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:path/to/directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:path/to/directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote directory, deleting
|
|
any excess files in the directory.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory remote:directory
|
|
\f[R]
|
|
.fi
|
|
.SS Example without a config file
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=\[ga]rclone obscure dummy\[ga]
|
|
\f[R]
|
|
.fi
|
|
.SS Implicit TLS
|
|
.PP
|
|
Rlone FTP supports implicit FTP over TLS servers (FTPS).
|
|
This has to be enabled in the FTP backend config for the remote, or with
|
|
\f[C]--ftp-tls\f[R].
|
|
The default FTPS port is \f[C]990\f[R], not \f[C]21\f[R] and can be set
|
|
with \f[C]--ftp-port\f[R].
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
File names cannot end with the following characters.
|
|
Repacement is limited to the last character in a file name:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
SP
|
|
T}@T{
|
|
0x20
|
|
T}@T{
|
|
\[u2420]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Not all FTP servers can have all characters in file names, for example:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c.
|
|
T{
|
|
FTP Server
|
|
T}@T{
|
|
Forbidden characters
|
|
T}
|
|
_
|
|
T{
|
|
proftpd
|
|
T}@T{
|
|
\f[C]*\f[R]
|
|
T}
|
|
T{
|
|
pureftpd
|
|
T}@T{
|
|
\f[C]\[rs] [ ]\f[R]
|
|
T}
|
|
.TE
|
|
.PP
|
|
This backend\[aq]s interactive configuration wizard provides a selection
|
|
of sensible encoding settings for major FTP servers: ProFTPd, PureFTPd,
|
|
VsFTPd.
|
|
Just hit a selection number when prompted.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to ftp (FTP Connection).
|
|
.SS --ftp-host
|
|
.PP
|
|
FTP host to connect to.
|
|
.PP
|
|
E.g.
|
|
\[dq]ftp.example.com\[dq].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: host
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_HOST
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --ftp-user
|
|
.PP
|
|
FTP username.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_USER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]$USER\[dq]
|
|
.SS --ftp-port
|
|
.PP
|
|
FTP port number.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: port
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_PORT
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 21
|
|
.SS --ftp-pass
|
|
.PP
|
|
FTP password.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: pass
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_PASS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --ftp-tls
|
|
.PP
|
|
Use Implicit FTPS (FTP over TLS).
|
|
.PP
|
|
When using implicit FTP over TLS the client connects using TLS right
|
|
from the start which breaks compatibility with non-TLS-aware servers.
|
|
This is usually served over port 990 rather than port 21.
|
|
Cannot be used in combination with explicit FTP.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: tls
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_TLS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --ftp-explicit-tls
|
|
.PP
|
|
Use Explicit FTPS (FTP over TLS).
|
|
.PP
|
|
When using explicit FTP over TLS the client explicitly requests security
|
|
from the server in order to upgrade a plain text connection to an
|
|
encrypted one.
|
|
Cannot be used in combination with implicit FTP.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: explicit_tls
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_EXPLICIT_TLS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to ftp (FTP Connection).
|
|
.SS --ftp-concurrency
|
|
.PP
|
|
Maximum number of FTP simultaneous connections, 0 for unlimited.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: concurrency
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_CONCURRENCY
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 0
|
|
.SS --ftp-no-check-certificate
|
|
.PP
|
|
Do not verify the TLS certificate of the server.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_check_certificate
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_NO_CHECK_CERTIFICATE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --ftp-disable-epsv
|
|
.PP
|
|
Disable using EPSV even if server advertises support.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_epsv
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_DISABLE_EPSV
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --ftp-disable-mlsd
|
|
.PP
|
|
Disable using MLSD even if server advertises support.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_mlsd
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_DISABLE_MLSD
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --ftp-writing-mdtm
|
|
.PP
|
|
Use MDTM to set modification time (VsFtpd quirk)
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: writing_mdtm
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_WRITING_MDTM
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --ftp-idle-timeout
|
|
.PP
|
|
Max time before closing idle connections.
|
|
.PP
|
|
If no connections have been returned to the connection pool in the time
|
|
given, rclone will empty the connection pool.
|
|
.PP
|
|
Set to 0 to keep connections indefinitely.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: idle_timeout
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_IDLE_TIMEOUT
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 1m0s
|
|
.SS --ftp-close-timeout
|
|
.PP
|
|
Maximum time to wait for a response to close.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: close_timeout
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_CLOSE_TIMEOUT
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 1m0s
|
|
.SS --ftp-tls-cache-size
|
|
.PP
|
|
Size of TLS session cache for all control and data connections.
|
|
.PP
|
|
TLS cache allows to resume TLS sessions and reuse PSK between
|
|
connections.
|
|
Increase if default size is not enough resulting in TLS resumption
|
|
errors.
|
|
Enabled by default.
|
|
Use 0 to disable.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: tls_cache_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_TLS_CACHE_SIZE
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 32
|
|
.SS --ftp-disable-tls13
|
|
.PP
|
|
Disable TLS 1.3 (workaround for FTP servers with buggy TLS)
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_tls13
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_DISABLE_TLS13
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --ftp-shut-timeout
|
|
.PP
|
|
Maximum time to wait for data connection closing status.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: shut_timeout
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_SHUT_TIMEOUT
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 1m0s
|
|
.SS --ftp-ask-password
|
|
.PP
|
|
Allow asking for FTP password when needed.
|
|
.PP
|
|
If this is set and no password is supplied then rclone will ask for a
|
|
password
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: ask_password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_ASK_PASSWORD
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --ftp-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_FTP_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,Del,Ctl,RightSpace,Dot
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]Asterisk,Ctl,Dot,Slash\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
ProFTPd can\[aq]t handle \[aq]*\[aq] in file names
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]BackSlash,Ctl,Del,Dot,RightSpace,Slash,SquareBracket\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
PureFTPd can\[aq]t handle \[aq][]\[aq] or \[aq]*\[aq] in file names
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]Ctl,LeftPeriod,Slash\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
VsFTPd can\[aq]t handle file names starting with dot
|
|
.RE
|
|
.RE
|
|
.SS Limitations
|
|
.PP
|
|
FTP servers acting as rclone remotes must support \f[C]passive\f[R]
|
|
mode.
|
|
The mode cannot be configured as \f[C]passive\f[R] is the only supported
|
|
one.
|
|
Rclone\[aq]s FTP implementation is not compatible with \f[C]active\f[R]
|
|
mode as the library it uses doesn\[aq]t support
|
|
it (https://github.com/jlaffaye/ftp/issues/29).
|
|
This will likely never be supported due to security concerns.
|
|
.PP
|
|
Rclone\[aq]s FTP backend does not support any checksums but can compare
|
|
file sizes.
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the FTP backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.PP
|
|
The implementation of : \f[C]--dump headers\f[R],
|
|
\f[C]--dump bodies\f[R], \f[C]--dump auth\f[R] for debugging isn\[aq]t
|
|
the same as for rclone HTTP based backends - it has less fine grained
|
|
control.
|
|
.PP
|
|
\f[C]--timeout\f[R] isn\[aq]t supported (but \f[C]--contimeout\f[R] is).
|
|
.PP
|
|
\f[C]--bind\f[R] isn\[aq]t supported.
|
|
.PP
|
|
Rclone\[aq]s FTP backend could support server-side move but does not at
|
|
present.
|
|
.PP
|
|
The \f[C]ftp_proxy\f[R] environment variable is not currently supported.
|
|
.SS Modified time
|
|
.PP
|
|
File modification time (timestamps) is supported to 1 second resolution
|
|
for major FTP servers: ProFTPd, PureFTPd, VsFTPd, and FileZilla FTP
|
|
server.
|
|
The \f[C]VsFTPd\f[R] server has non-standard implementation of time
|
|
related protocol commands and needs a special configuration setting:
|
|
\f[C]writing_mdtm = true\f[R].
|
|
.PP
|
|
Support for precise file time with other FTP servers varies depending on
|
|
what protocol extensions they advertise.
|
|
If all the \f[C]MLSD\f[R], \f[C]MDTM\f[R] and \f[C]MFTM\f[R] extensions
|
|
are present, rclone will use them together to provide precise time.
|
|
Otherwise the times you see on the FTP server through rclone are those
|
|
of the last file upload.
|
|
.PP
|
|
You can use the following command to check whether rclone can use
|
|
precise time with your FTP server:
|
|
\f[C]rclone backend features your_ftp_remote:\f[R] (the trailing colon
|
|
is important).
|
|
Look for the number in the line tagged by \f[C]Precision\f[R]
|
|
designating the remote time precision expressed as nanoseconds.
|
|
A value of \f[C]1000000000\f[R] means that file time precision of 1
|
|
second is available.
|
|
A value of \f[C]3153600000000000000\f[R] (or another large number) means
|
|
\[dq]unsupported\[dq].
|
|
.SH Google Cloud Storage
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[R] (or \f[C]remote:\f[R] for
|
|
the \f[C]lsd\f[R] command.) You may put subdirectories in too, e.g.
|
|
\f[C]remote:bucket/path/to/dir\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for google cloud storage involves getting a token from
|
|
Google Cloud Storage which you need to do in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n) New remote
|
|
d) Delete remote
|
|
q) Quit config
|
|
e/n/d/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Google Cloud Storage (this is not Google Drive)
|
|
\[rs] \[dq]google cloud storage\[dq]
|
|
[snip]
|
|
Storage> google cloud storage
|
|
Google Application Client Id - leave blank normally.
|
|
client_id>
|
|
Google Application Client Secret - leave blank normally.
|
|
client_secret>
|
|
Project number optional - needed only for list/create/delete buckets - see your developer console.
|
|
project_number> 12345678
|
|
Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login.
|
|
service_account_file>
|
|
Access Control List for new objects.
|
|
Choose a number from below, or type in your own value
|
|
1 / Object owner gets OWNER access, and all Authenticated Users get READER access.
|
|
\[rs] \[dq]authenticatedRead\[dq]
|
|
2 / Object owner gets OWNER access, and project team owners get OWNER access.
|
|
\[rs] \[dq]bucketOwnerFullControl\[dq]
|
|
3 / Object owner gets OWNER access, and project team owners get READER access.
|
|
\[rs] \[dq]bucketOwnerRead\[dq]
|
|
4 / Object owner gets OWNER access [default if left blank].
|
|
\[rs] \[dq]private\[dq]
|
|
5 / Object owner gets OWNER access, and project team members get access according to their roles.
|
|
\[rs] \[dq]projectPrivate\[dq]
|
|
6 / Object owner gets OWNER access, and all Users get READER access.
|
|
\[rs] \[dq]publicRead\[dq]
|
|
object_acl> 4
|
|
Access Control List for new buckets.
|
|
Choose a number from below, or type in your own value
|
|
1 / Project team owners get OWNER access, and all Authenticated Users get READER access.
|
|
\[rs] \[dq]authenticatedRead\[dq]
|
|
2 / Project team owners get OWNER access [default if left blank].
|
|
\[rs] \[dq]private\[dq]
|
|
3 / Project team members get access according to their roles.
|
|
\[rs] \[dq]projectPrivate\[dq]
|
|
4 / Project team owners get OWNER access, and all Users get READER access.
|
|
\[rs] \[dq]publicRead\[dq]
|
|
5 / Project team owners get OWNER access, and all Users get WRITER access.
|
|
\[rs] \[dq]publicReadWrite\[dq]
|
|
bucket_acl> 2
|
|
Location for the newly created buckets.
|
|
Choose a number from below, or type in your own value
|
|
1 / Empty for default location (US).
|
|
\[rs] \[dq]\[dq]
|
|
2 / Multi-regional location for Asia.
|
|
\[rs] \[dq]asia\[dq]
|
|
3 / Multi-regional location for Europe.
|
|
\[rs] \[dq]eu\[dq]
|
|
4 / Multi-regional location for United States.
|
|
\[rs] \[dq]us\[dq]
|
|
5 / Taiwan.
|
|
\[rs] \[dq]asia-east1\[dq]
|
|
6 / Tokyo.
|
|
\[rs] \[dq]asia-northeast1\[dq]
|
|
7 / Singapore.
|
|
\[rs] \[dq]asia-southeast1\[dq]
|
|
8 / Sydney.
|
|
\[rs] \[dq]australia-southeast1\[dq]
|
|
9 / Belgium.
|
|
\[rs] \[dq]europe-west1\[dq]
|
|
10 / London.
|
|
\[rs] \[dq]europe-west2\[dq]
|
|
11 / Iowa.
|
|
\[rs] \[dq]us-central1\[dq]
|
|
12 / South Carolina.
|
|
\[rs] \[dq]us-east1\[dq]
|
|
13 / Northern Virginia.
|
|
\[rs] \[dq]us-east4\[dq]
|
|
14 / Oregon.
|
|
\[rs] \[dq]us-west1\[dq]
|
|
location> 12
|
|
The storage class to use when storing objects in Google Cloud Storage.
|
|
Choose a number from below, or type in your own value
|
|
1 / Default
|
|
\[rs] \[dq]\[dq]
|
|
2 / Multi-regional storage class
|
|
\[rs] \[dq]MULTI_REGIONAL\[dq]
|
|
3 / Regional storage class
|
|
\[rs] \[dq]REGIONAL\[dq]
|
|
4 / Nearline storage class
|
|
\[rs] \[dq]NEARLINE\[dq]
|
|
5 / Coldline storage class
|
|
\[rs] \[dq]COLDLINE\[dq]
|
|
6 / Durable reduced availability storage class
|
|
\[rs] \[dq]DURABLE_REDUCED_AVAILABILITY\[dq]
|
|
storage_class> 5
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine or Y didn\[aq]t work
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
--------------------
|
|
[remote]
|
|
type = google cloud storage
|
|
client_id =
|
|
client_secret =
|
|
token = {\[dq]AccessToken\[dq]:\[dq]xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]RefreshToken\[dq]:\[dq]x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx\[dq],\[dq]Expiry\[dq]:\[dq]2014-07-17T20:49:14.929208288+01:00\[dq],\[dq]Extra\[dq]:null}
|
|
project_number = 12345678
|
|
object_acl = private
|
|
bucket_acl = private
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Google if you use auto config mode.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall, or use
|
|
manual mode.
|
|
.PP
|
|
This remote is called \f[C]remote\f[R] and can now be used like this
|
|
.PP
|
|
See all the buckets in your project
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Make a new bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.SS Service Account support
|
|
.PP
|
|
You can set up rclone with Google Cloud Storage in an unattended mode,
|
|
i.e.
|
|
not tied to a specific end-user Google account.
|
|
This is useful when you want to synchronise files onto machines that
|
|
don\[aq]t have actively logged-in users, for example build machines.
|
|
.PP
|
|
To get credentials for Google Cloud Platform IAM Service
|
|
Accounts (https://cloud.google.com/iam/docs/service-accounts), please
|
|
head to the Service
|
|
Account (https://console.cloud.google.com/permissions/serviceaccounts)
|
|
section of the Google Developer Console.
|
|
Service Accounts behave just like normal \f[C]User\f[R] permissions in
|
|
Google Cloud Storage
|
|
ACLs (https://cloud.google.com/storage/docs/access-control), so you can
|
|
limit their access (e.g.
|
|
make them read only).
|
|
After creating an account, a JSON file containing the Service
|
|
Account\[aq]s credentials will be downloaded onto your machines.
|
|
These credentials are what rclone will use for authentication.
|
|
.PP
|
|
To use a Service Account instead of OAuth2 token flow, enter the path to
|
|
your Service Account credentials at the \f[C]service_account_file\f[R]
|
|
prompt and rclone won\[aq]t use the browser based authentication flow.
|
|
If you\[aq]d rather stuff the contents of the credentials file into the
|
|
rclone config file, you can set \f[C]service_account_credentials\f[R]
|
|
with the actual contents of the file instead, or set the equivalent
|
|
environment variable.
|
|
.SS Anonymous Access
|
|
.PP
|
|
For downloads of objects that permit public access you can configure
|
|
rclone to use anonymous access by setting \f[C]anonymous\f[R] to
|
|
\f[C]true\f[R].
|
|
With unauthorized access you can\[aq]t write or create files but only
|
|
read or list those buckets and objects that have public read access.
|
|
.SS Application Default Credentials
|
|
.PP
|
|
If no other source of credentials is provided, rclone will fall back to
|
|
Application Default
|
|
Credentials (https://cloud.google.com/video-intelligence/docs/common/auth#authenticating_with_application_default_credentials)
|
|
this is useful both when you already have configured authentication for
|
|
your developer account, or in production when running on a google
|
|
compute host.
|
|
Note that if running in docker, you may need to run additional commands
|
|
on your google compute machine - see this
|
|
page (https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud_as_a_docker_credential_helper).
|
|
.PP
|
|
Note that in the case application default credentials are used, there is
|
|
no need to explicitly configure a project number.
|
|
.SS --fast-list
|
|
.PP
|
|
This remote supports \f[C]--fast-list\f[R] which allows you to use fewer
|
|
transactions in exchange for more memory.
|
|
See the rclone docs (https://rclone.org/docs/#fast-list) for more
|
|
details.
|
|
.SS Custom upload headers
|
|
.PP
|
|
You can set custom upload headers with the \f[C]--header-upload\f[R]
|
|
flag.
|
|
Google Cloud Storage supports the headers as described in the working
|
|
with metadata
|
|
documentation (https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata)
|
|
.IP \[bu] 2
|
|
Cache-Control
|
|
.IP \[bu] 2
|
|
Content-Disposition
|
|
.IP \[bu] 2
|
|
Content-Encoding
|
|
.IP \[bu] 2
|
|
Content-Language
|
|
.IP \[bu] 2
|
|
Content-Type
|
|
.IP \[bu] 2
|
|
X-Goog-Storage-Class
|
|
.IP \[bu] 2
|
|
X-Goog-Meta-
|
|
.PP
|
|
Eg \f[C]--header-upload \[dq]Content-Type text/potato\[dq]\f[R]
|
|
.PP
|
|
Note that the last of these is for setting custom metadata in the form
|
|
\f[C]--header-upload \[dq]x-goog-meta-key: value\[dq]\f[R]
|
|
.SS Modification time
|
|
.PP
|
|
Google Cloud Storage stores md5sum natively.
|
|
Google\[aq]s gsutil (https://cloud.google.com/storage/docs/gsutil) tool
|
|
stores modification time with one-second precision as
|
|
\f[C]goog-reserved-file-mtime\f[R] in file metadata.
|
|
.PP
|
|
To ensure compatibility with gsutil, rclone stores modification time in
|
|
2 separate metadata entries.
|
|
\f[C]mtime\f[R] uses RFC3339 format with one-nanosecond precision.
|
|
\f[C]goog-reserved-file-mtime\f[R] uses Unix timestamp format with
|
|
one-second precision.
|
|
To get modification time from object metadata, rclone reads the metadata
|
|
in the following order: \f[C]mtime\f[R],
|
|
\f[C]goog-reserved-file-mtime\f[R], object updated time.
|
|
.PP
|
|
Note that rclone\[aq]s default modify window is 1ns.
|
|
Files uploaded by gsutil only contain timestamps with one-second
|
|
precision.
|
|
If you use rclone to sync files previously uploaded by gsutil, rclone
|
|
will attempt to update modification time for all these files.
|
|
To avoid these possibly unnecessary updates, use
|
|
\f[C]--modify-window 1s\f[R].
|
|
.SS Restricted filename characters
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
NUL
|
|
T}@T{
|
|
0x00
|
|
T}@T{
|
|
\[u2400]
|
|
T}
|
|
T{
|
|
LF
|
|
T}@T{
|
|
0x0A
|
|
T}@T{
|
|
\[u240A]
|
|
T}
|
|
T{
|
|
CR
|
|
T}@T{
|
|
0x0D
|
|
T}@T{
|
|
\[u240D]
|
|
T}
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to google cloud storage (Google
|
|
Cloud Storage (this is not Google Drive)).
|
|
.SS --gcs-client-id
|
|
.PP
|
|
OAuth Client Id.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gcs-client-secret
|
|
.PP
|
|
OAuth Client Secret.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_CLIENT_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gcs-project-number
|
|
.PP
|
|
Project number.
|
|
.PP
|
|
Optional - needed only for list/create/delete buckets - see your
|
|
developer console.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: project_number
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_PROJECT_NUMBER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gcs-service-account-file
|
|
.PP
|
|
Service Account Credentials JSON file path.
|
|
.PP
|
|
Leave blank normally.
|
|
Needed only if you want use SA instead of interactive login.
|
|
.PP
|
|
Leading \f[C]\[ti]\f[R] will be expanded in the file name as will
|
|
environment variables such as \f[C]${RCLONE_CONFIG_DIR}\f[R].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: service_account_file
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_SERVICE_ACCOUNT_FILE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gcs-service-account-credentials
|
|
.PP
|
|
Service Account Credentials JSON blob.
|
|
.PP
|
|
Leave blank normally.
|
|
Needed only if you want use SA instead of interactive login.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: service_account_credentials
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_SERVICE_ACCOUNT_CREDENTIALS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gcs-anonymous
|
|
.PP
|
|
Access public buckets and objects without credentials.
|
|
.PP
|
|
Set to \[aq]true\[aq] if you just want to download files and don\[aq]t
|
|
configure credentials.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: anonymous
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_ANONYMOUS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --gcs-object-acl
|
|
.PP
|
|
Access Control List for new objects.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: object_acl
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_OBJECT_ACL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]authenticatedRead\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Object owner gets OWNER access.
|
|
.IP \[bu] 2
|
|
All Authenticated Users get READER access.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]bucketOwnerFullControl\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Object owner gets OWNER access.
|
|
.IP \[bu] 2
|
|
Project team owners get OWNER access.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]bucketOwnerRead\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Object owner gets OWNER access.
|
|
.IP \[bu] 2
|
|
Project team owners get READER access.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]private\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Object owner gets OWNER access.
|
|
.IP \[bu] 2
|
|
Default if left blank.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]projectPrivate\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Object owner gets OWNER access.
|
|
.IP \[bu] 2
|
|
Project team members get access according to their roles.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]publicRead\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Object owner gets OWNER access.
|
|
.IP \[bu] 2
|
|
All Users get READER access.
|
|
.RE
|
|
.RE
|
|
.SS --gcs-bucket-acl
|
|
.PP
|
|
Access Control List for new buckets.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: bucket_acl
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_BUCKET_ACL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]authenticatedRead\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Project team owners get OWNER access.
|
|
.IP \[bu] 2
|
|
All Authenticated Users get READER access.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]private\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Project team owners get OWNER access.
|
|
.IP \[bu] 2
|
|
Default if left blank.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]projectPrivate\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Project team members get access according to their roles.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]publicRead\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Project team owners get OWNER access.
|
|
.IP \[bu] 2
|
|
All Users get READER access.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]publicReadWrite\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Project team owners get OWNER access.
|
|
.IP \[bu] 2
|
|
All Users get WRITER access.
|
|
.RE
|
|
.RE
|
|
.SS --gcs-bucket-policy-only
|
|
.PP
|
|
Access checks should use bucket-level IAM policies.
|
|
.PP
|
|
If you want to upload objects to a bucket with Bucket Policy Only set
|
|
then you will need to set this.
|
|
.PP
|
|
When it is set, rclone:
|
|
.IP \[bu] 2
|
|
ignores ACLs set on buckets
|
|
.IP \[bu] 2
|
|
ignores ACLs set on objects
|
|
.IP \[bu] 2
|
|
creates buckets with Bucket Policy Only set
|
|
.PP
|
|
Docs: https://cloud.google.com/storage/docs/bucket-policy-only
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: bucket_policy_only
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_BUCKET_POLICY_ONLY
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --gcs-location
|
|
.PP
|
|
Location for the newly created buckets.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: location
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_LOCATION
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Empty for default location (US)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Multi-regional location for Asia
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Multi-regional location for Europe
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Multi-regional location for United States
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia-east1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Taiwan
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia-east2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Hong Kong
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia-northeast1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Tokyo
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia-northeast2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Osaka
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia-northeast3\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Seoul
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia-south1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Mumbai
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia-south2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Delhi
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia-southeast1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Singapore
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia-southeast2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Jakarta
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]australia-southeast1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Sydney
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]australia-southeast2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Melbourne
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]europe-north1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Finland
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]europe-west1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Belgium
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]europe-west2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
London
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]europe-west3\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Frankfurt
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]europe-west4\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Netherlands
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]europe-west6\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Z\[:u]rich
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]europe-central2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Warsaw
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-central1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Iowa
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-east1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
South Carolina
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-east4\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Northern Virginia
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-west1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Oregon
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-west2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
California
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-west3\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Salt Lake City
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us-west4\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Las Vegas
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]northamerica-northeast1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Montr\['e]al
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]northamerica-northeast2\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Toronto
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]southamerica-east1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
S\[~a]o Paulo
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]southamerica-west1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Santiago
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia1\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Dual region: asia-northeast1 and asia-northeast2.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eur4\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Dual region: europe-north1 and europe-west4.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]nam4\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Dual region: us-central1 and us-east1.
|
|
.RE
|
|
.RE
|
|
.SS --gcs-storage-class
|
|
.PP
|
|
The storage class to use when storing objects in Google Cloud Storage.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: storage_class
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_STORAGE_CLASS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Default
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]MULTI_REGIONAL\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Multi-regional storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]REGIONAL\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Regional storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]NEARLINE\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Nearline storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]COLDLINE\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Coldline storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]ARCHIVE\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Archive storage class
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]DURABLE_REDUCED_AVAILABILITY\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Durable reduced availability storage class
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to google cloud storage (Google
|
|
Cloud Storage (this is not Google Drive)).
|
|
.SS --gcs-token
|
|
.PP
|
|
OAuth Access Token as a JSON blob.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gcs-auth-url
|
|
.PP
|
|
Auth server URL.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_AUTH_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gcs-token-url
|
|
.PP
|
|
Token server url.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_TOKEN_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gcs-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GCS_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,CrLf,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the Google Cloud Storage
|
|
backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SH Google Drive
|
|
.PP
|
|
Paths are specified as \f[C]drive:path\f[R]
|
|
.PP
|
|
Drive paths may be as deep as required, e.g.
|
|
\f[C]drive:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for drive involves getting a token from Google drive
|
|
which you need to do in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/r/c/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Google Drive
|
|
\[rs] \[dq]drive\[dq]
|
|
[snip]
|
|
Storage> drive
|
|
Google Application Client Id - leave blank normally.
|
|
client_id>
|
|
Google Application Client Secret - leave blank normally.
|
|
client_secret>
|
|
Scope that rclone should use when requesting access from drive.
|
|
Choose a number from below, or type in your own value
|
|
1 / Full access all files, excluding Application Data Folder.
|
|
\[rs] \[dq]drive\[dq]
|
|
2 / Read-only access to file metadata and file contents.
|
|
\[rs] \[dq]drive.readonly\[dq]
|
|
/ Access to files created by rclone only.
|
|
3 | These are visible in the drive website.
|
|
| File authorization is revoked when the user deauthorizes the app.
|
|
\[rs] \[dq]drive.file\[dq]
|
|
/ Allows read and write access to the Application Data folder.
|
|
4 | This is not visible in the drive website.
|
|
\[rs] \[dq]drive.appfolder\[dq]
|
|
/ Allows read-only access to file metadata but
|
|
5 | does not allow any access to read or download file content.
|
|
\[rs] \[dq]drive.metadata.readonly\[dq]
|
|
scope> 1
|
|
ID of the root folder - leave blank normally. Fill in to access \[dq]Computers\[dq] folders. (see docs).
|
|
root_folder_id>
|
|
Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login.
|
|
service_account_file>
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine or Y didn\[aq]t work
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
Configure this as a Shared Drive (Team Drive)?
|
|
y) Yes
|
|
n) No
|
|
y/n> n
|
|
--------------------
|
|
[remote]
|
|
client_id =
|
|
client_secret =
|
|
scope = drive
|
|
root_folder_id =
|
|
service_account_file =
|
|
token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2014-03-16T13:57:58.955387075Z\[dq]}
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Google if you use auto config mode.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and it may require you to
|
|
unblock it temporarily if you are running a host firewall, or use manual
|
|
mode.
|
|
.PP
|
|
You can then use it like this,
|
|
.PP
|
|
List directories in top level of your drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to a drive directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Scopes
|
|
.PP
|
|
Rclone allows you to select which scope you would like for rclone to
|
|
use.
|
|
This changes what type of token is granted to rclone.
|
|
The scopes are defined
|
|
here (https://developers.google.com/drive/v3/web/about-auth).
|
|
.PP
|
|
The scope are
|
|
.SS drive
|
|
.PP
|
|
This is the default scope and allows full access to all files, except
|
|
for the Application Data Folder (see below).
|
|
.PP
|
|
Choose this one if you aren\[aq]t sure.
|
|
.SS drive.readonly
|
|
.PP
|
|
This allows read only access to all files.
|
|
Files may be listed and downloaded but not uploaded, renamed or deleted.
|
|
.SS drive.file
|
|
.PP
|
|
With this scope rclone can read/view/modify only those files and folders
|
|
it creates.
|
|
.PP
|
|
So if you uploaded files to drive via the web interface (or any other
|
|
means) they will not be visible to rclone.
|
|
.PP
|
|
This can be useful if you are using rclone to backup data and you want
|
|
to be sure confidential data on your drive is not visible to rclone.
|
|
.PP
|
|
Files created with this scope are visible in the web interface.
|
|
.SS drive.appfolder
|
|
.PP
|
|
This gives rclone its own private area to store files.
|
|
Rclone will not be able to see any other files on your drive and you
|
|
won\[aq]t be able to see rclone\[aq]s files from the web interface
|
|
either.
|
|
.SS drive.metadata.readonly
|
|
.PP
|
|
This allows read only access to file names only.
|
|
It does not allow rclone to download or upload data, or rename or delete
|
|
files or directories.
|
|
.SS Root folder ID
|
|
.PP
|
|
You can set the \f[C]root_folder_id\f[R] for rclone.
|
|
This is the directory (identified by its \f[C]Folder ID\f[R]) that
|
|
rclone considers to be the root of your drive.
|
|
.PP
|
|
Normally you will leave this blank and rclone will determine the correct
|
|
root to use itself.
|
|
.PP
|
|
However you can set this to restrict rclone to a specific folder
|
|
hierarchy or to access data within the \[dq]Computers\[dq] tab on the
|
|
drive web interface (where files from Google\[aq]s Backup and Sync
|
|
desktop program go).
|
|
.PP
|
|
In order to do this you will have to find the \f[C]Folder ID\f[R] of the
|
|
directory you wish rclone to display.
|
|
This will be the last segment of the URL when you open the relevant
|
|
folder in the drive web interface.
|
|
.PP
|
|
So if the folder you want rclone to use has a URL which looks like
|
|
\f[C]https://drive.google.com/drive/folders/1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh\f[R]
|
|
in the browser, then you use \f[C]1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh\f[R]
|
|
as the \f[C]root_folder_id\f[R] in the config.
|
|
.PP
|
|
\f[B]NB\f[R] folders under the \[dq]Computers\[dq] tab seem to be read
|
|
only (drive gives a 500 error) when using rclone.
|
|
.PP
|
|
There doesn\[aq]t appear to be an API to discover the folder IDs of the
|
|
\[dq]Computers\[dq] tab - please contact us if you know otherwise!
|
|
.PP
|
|
Note also that rclone can\[aq]t access any data under the
|
|
\[dq]Backups\[dq] tab on the google drive web interface yet.
|
|
.SS Service Account support
|
|
.PP
|
|
You can set up rclone with Google Drive in an unattended mode, i.e.
|
|
not tied to a specific end-user Google account.
|
|
This is useful when you want to synchronise files onto machines that
|
|
don\[aq]t have actively logged-in users, for example build machines.
|
|
.PP
|
|
To use a Service Account instead of OAuth2 token flow, enter the path to
|
|
your Service Account credentials at the \f[C]service_account_file\f[R]
|
|
prompt during \f[C]rclone config\f[R] and rclone won\[aq]t use the
|
|
browser based authentication flow.
|
|
If you\[aq]d rather stuff the contents of the credentials file into the
|
|
rclone config file, you can set \f[C]service_account_credentials\f[R]
|
|
with the actual contents of the file instead, or set the equivalent
|
|
environment variable.
|
|
.SS Use case - Google Apps/G-suite account and individual Drive
|
|
.PP
|
|
Let\[aq]s say that you are the administrator of a Google Apps (old) or
|
|
G-suite account.
|
|
The goal is to store data on an individual\[aq]s Drive account, who IS a
|
|
member of the domain.
|
|
We\[aq]ll call the domain \f[B]example.com\f[R], and the user
|
|
\f[B]foo\[at]example.com\f[R].
|
|
.PP
|
|
There\[aq]s a few steps we need to go through to accomplish this:
|
|
.SS 1. Create a service account for example.com
|
|
.IP \[bu] 2
|
|
To create a service account and obtain its credentials, go to the Google
|
|
Developer Console (https://console.developers.google.com).
|
|
.IP \[bu] 2
|
|
You must have a project - create one if you don\[aq]t.
|
|
.IP \[bu] 2
|
|
Then go to \[dq]IAM & admin\[dq] -> \[dq]Service Accounts\[dq].
|
|
.IP \[bu] 2
|
|
Use the \[dq]Create Credentials\[dq] button.
|
|
Fill in \[dq]Service account name\[dq] with something that identifies
|
|
your client.
|
|
\[dq]Role\[dq] can be empty.
|
|
.IP \[bu] 2
|
|
Tick \[dq]Furnish a new private key\[dq] - select \[dq]Key type
|
|
JSON\[dq].
|
|
.IP \[bu] 2
|
|
Tick \[dq]Enable G Suite Domain-wide Delegation\[dq].
|
|
This option makes \[dq]impersonation\[dq] possible, as documented here:
|
|
Delegating domain-wide authority to the service
|
|
account (https://developers.google.com/identity/protocols/OAuth2ServiceAccount#delegatingauthority)
|
|
.IP \[bu] 2
|
|
These credentials are what rclone will use for authentication.
|
|
If you ever need to remove access, press the \[dq]Delete service account
|
|
key\[dq] button.
|
|
.SS 2. Allowing API access to example.com Google Drive
|
|
.IP \[bu] 2
|
|
Go to example.com\[aq]s admin console
|
|
.IP \[bu] 2
|
|
Go into \[dq]Security\[dq] (or use the search bar)
|
|
.IP \[bu] 2
|
|
Select \[dq]Show more\[dq] and then \[dq]Advanced settings\[dq]
|
|
.IP \[bu] 2
|
|
Select \[dq]Manage API client access\[dq] in the
|
|
\[dq]Authentication\[dq] section
|
|
.IP \[bu] 2
|
|
In the \[dq]Client Name\[dq] field enter the service account\[aq]s
|
|
\[dq]Client ID\[dq] - this can be found in the Developer Console under
|
|
\[dq]IAM & Admin\[dq] -> \[dq]Service Accounts\[dq], then \[dq]View
|
|
Client ID\[dq] for the newly created service account.
|
|
It is a \[ti]21 character numerical string.
|
|
.IP \[bu] 2
|
|
In the next field, \[dq]One or More API Scopes\[dq], enter
|
|
\f[C]https://www.googleapis.com/auth/drive\f[R] to grant access to
|
|
Google Drive specifically.
|
|
.SS 3. Configure rclone, assuming a new install
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
|
|
n/s/q> n # New
|
|
name>gdrive # Gdrive is an example name
|
|
Storage> # Select the number shown for Google Drive
|
|
client_id> # Can be left blank
|
|
client_secret> # Can be left blank
|
|
scope> # Select your scope, 1 for example
|
|
root_folder_id> # Can be left blank
|
|
service_account_file> /home/foo/myJSONfile.json # This is where the JSON file goes!
|
|
y/n> # Auto config, n
|
|
\f[R]
|
|
.fi
|
|
.SS 4. Verify that it\[aq]s working
|
|
.IP \[bu] 2
|
|
\f[C]rclone -v --drive-impersonate foo\[at]example.com lsf gdrive:backup\f[R]
|
|
.IP \[bu] 2
|
|
The arguments do:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]-v\f[R] - verbose logging
|
|
.IP \[bu] 2
|
|
\f[C]--drive-impersonate foo\[at]example.com\f[R] - this is what does
|
|
the magic, pretending to be user foo.
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[R] - list files in a parsing friendly way
|
|
.IP \[bu] 2
|
|
\f[C]gdrive:backup\f[R] - use the remote called gdrive, work in the
|
|
folder named backup.
|
|
.RE
|
|
.PP
|
|
Note: in case you configured a specific root folder on gdrive and rclone
|
|
is unable to access the contents of that folder when using
|
|
\f[C]--drive-impersonate\f[R], do this instead: - in the gdrive web
|
|
interface, share your root folder with the user/email of the new Service
|
|
Account you created/selected at step #1 - use rclone without specifying
|
|
the \f[C]--drive-impersonate\f[R] option, like this:
|
|
\f[C]rclone -v lsf gdrive:backup\f[R]
|
|
.SS Shared drives (team drives)
|
|
.PP
|
|
If you want to configure the remote to point to a Google Shared Drive
|
|
(previously known as Team Drives) then answer \f[C]y\f[R] to the
|
|
question \f[C]Configure this as a Shared Drive (Team Drive)?\f[R].
|
|
.PP
|
|
This will fetch the list of Shared Drives from google and allow you to
|
|
configure which one you want to use.
|
|
You can also type in a Shared Drive ID if you prefer.
|
|
.PP
|
|
For example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Configure this as a Shared Drive (Team Drive)?
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
Fetching Shared Drive list...
|
|
Choose a number from below, or type in your own value
|
|
1 / Rclone Test
|
|
\[rs] \[dq]xxxxxxxxxxxxxxxxxxxx\[dq]
|
|
2 / Rclone Test 2
|
|
\[rs] \[dq]yyyyyyyyyyyyyyyyyyyy\[dq]
|
|
3 / Rclone Test 3
|
|
\[rs] \[dq]zzzzzzzzzzzzzzzzzzzz\[dq]
|
|
Enter a Shared Drive ID> 1
|
|
--------------------
|
|
[remote]
|
|
client_id =
|
|
client_secret =
|
|
token = {\[dq]AccessToken\[dq]:\[dq]xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]RefreshToken\[dq]:\[dq]1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]Expiry\[dq]:\[dq]2014-03-16T13:57:58.955387075Z\[dq],\[dq]Extra\[dq]:null}
|
|
team_drive = xxxxxxxxxxxxxxxxxxxx
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.SS --fast-list
|
|
.PP
|
|
This remote supports \f[C]--fast-list\f[R] which allows you to use fewer
|
|
transactions in exchange for more memory.
|
|
See the rclone docs (https://rclone.org/docs/#fast-list) for more
|
|
details.
|
|
.PP
|
|
It does this by combining multiple \f[C]list\f[R] calls into a single
|
|
API request.
|
|
.PP
|
|
This works by combining many \f[C]\[aq]%s\[aq] in parents\f[R] filters
|
|
into one expression.
|
|
To list the contents of directories a, b and c, the following requests
|
|
will be send by the regular \f[C]List\f[R] function:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
trashed=false and \[aq]a\[aq] in parents
|
|
trashed=false and \[aq]b\[aq] in parents
|
|
trashed=false and \[aq]c\[aq] in parents
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
These can now be combined into a single request:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
trashed=false and (\[aq]a\[aq] in parents or \[aq]b\[aq] in parents or \[aq]c\[aq] in parents)
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The implementation of \f[C]ListR\f[R] will put up to 50
|
|
\f[C]parents\f[R] filters into one request.
|
|
It will use the \f[C]--checkers\f[R] value to specify the number of
|
|
requests to run in parallel.
|
|
.PP
|
|
In tests, these batch requests were up to 20x faster than the regular
|
|
method.
|
|
Running the following command against different sized folders gives:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsjson -vv -R --checkers=6 gdrive:folder
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
small folder (220 directories, 700 files):
|
|
.IP \[bu] 2
|
|
without \f[C]--fast-list\f[R]: 38s
|
|
.IP \[bu] 2
|
|
with \f[C]--fast-list\f[R]: 10s
|
|
.PP
|
|
large folder (10600 directories, 39000 files):
|
|
.IP \[bu] 2
|
|
without \f[C]--fast-list\f[R]: 22:05 min
|
|
.IP \[bu] 2
|
|
with \f[C]--fast-list\f[R]: 58s
|
|
.SS Modified time
|
|
.PP
|
|
Google drive stores modification times accurate to 1 ms.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
Only Invalid UTF-8 bytes will be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.PP
|
|
In contrast to other backends, \f[C]/\f[R] can also be used in names and
|
|
\f[C].\f[R] or \f[C]..\f[R] are valid names.
|
|
.SS Revisions
|
|
.PP
|
|
Google drive stores revisions of files.
|
|
When you upload a change to an existing file to google drive using
|
|
rclone it will create a new revision of that file.
|
|
.PP
|
|
Revisions follow the standard google policy which at time of writing was
|
|
.IP \[bu] 2
|
|
They are deleted after 30 days or 100 revisions (whatever comes first).
|
|
.IP \[bu] 2
|
|
They do not count towards a user storage quota.
|
|
.SS Deleting files
|
|
.PP
|
|
By default rclone will send all files to the trash when deleting files.
|
|
If deleting them permanently is required then use the
|
|
\f[C]--drive-use-trash=false\f[R] flag, or set the equivalent
|
|
environment variable.
|
|
.SS Shortcuts
|
|
.PP
|
|
In March 2020 Google introduced a new feature in Google Drive called
|
|
drive shortcuts (https://support.google.com/drive/answer/9700156)
|
|
(API (https://developers.google.com/drive/api/v3/shortcuts)).
|
|
These will (by September 2020) replace the ability for files or folders
|
|
to be in multiple folders at
|
|
once (https://cloud.google.com/blog/products/g-suite/simplifying-google-drives-folder-structure-and-sharing-models).
|
|
.PP
|
|
Shortcuts are files that link to other files on Google Drive somewhat
|
|
like a symlink in unix, except they point to the underlying file data
|
|
(e.g.
|
|
the inode in unix terms) so they don\[aq]t break if the source is
|
|
renamed or moved about.
|
|
.PP
|
|
Be default rclone treats these as follows.
|
|
.PP
|
|
For shortcuts pointing to files:
|
|
.IP \[bu] 2
|
|
When listing a file shortcut appears as the destination file.
|
|
.IP \[bu] 2
|
|
When downloading the contents of the destination file is downloaded.
|
|
.IP \[bu] 2
|
|
When updating shortcut file with a non shortcut file, the shortcut is
|
|
removed then a new file is uploaded in place of the shortcut.
|
|
.IP \[bu] 2
|
|
When server-side moving (renaming) the shortcut is renamed, not the
|
|
destination file.
|
|
.IP \[bu] 2
|
|
When server-side copying the shortcut is copied, not the contents of the
|
|
shortcut.
|
|
(unless \f[C]--drive-copy-shortcut-content\f[R] is in use in which case
|
|
the contents of the shortcut gets copied).
|
|
.IP \[bu] 2
|
|
When deleting the shortcut is deleted not the linked file.
|
|
.IP \[bu] 2
|
|
When setting the modification time, the modification time of the linked
|
|
file will be set.
|
|
.PP
|
|
For shortcuts pointing to folders:
|
|
.IP \[bu] 2
|
|
When listing the shortcut appears as a folder and that folder will
|
|
contain the contents of the linked folder appear (including any sub
|
|
folders)
|
|
.IP \[bu] 2
|
|
When downloading the contents of the linked folder and sub contents are
|
|
downloaded
|
|
.IP \[bu] 2
|
|
When uploading to a shortcut folder the file will be placed in the
|
|
linked folder
|
|
.IP \[bu] 2
|
|
When server-side moving (renaming) the shortcut is renamed, not the
|
|
destination folder
|
|
.IP \[bu] 2
|
|
When server-side copying the contents of the linked folder is copied,
|
|
not the shortcut.
|
|
.IP \[bu] 2
|
|
When deleting with \f[C]rclone rmdir\f[R] or \f[C]rclone purge\f[R] the
|
|
shortcut is deleted not the linked folder.
|
|
.IP \[bu] 2
|
|
\f[B]NB\f[R] When deleting with \f[C]rclone remove\f[R] or
|
|
\f[C]rclone mount\f[R] the contents of the linked folder will be
|
|
deleted.
|
|
.PP
|
|
The rclone backend (https://rclone.org/commands/rclone_backend/) command
|
|
can be used to create shortcuts.
|
|
.PP
|
|
Shortcuts can be completely ignored with the
|
|
\f[C]--drive-skip-shortcuts\f[R] flag or the corresponding
|
|
\f[C]skip_shortcuts\f[R] configuration setting.
|
|
.SS Emptying trash
|
|
.PP
|
|
If you wish to empty your trash you can use the
|
|
\f[C]rclone cleanup remote:\f[R] command which will permanently delete
|
|
all your trashed files.
|
|
This command does not take any path arguments.
|
|
.PP
|
|
Note that Google Drive takes some time (minutes to days) to empty the
|
|
trash even though the command returns within a few seconds.
|
|
No output is echoed, so there will be no confirmation even using -v or
|
|
-vv.
|
|
.SS Quota information
|
|
.PP
|
|
To view your current quota you can use the
|
|
\f[C]rclone about remote:\f[R] command which will display your usage
|
|
limit (quota), the usage in Google Drive, the size of all files in the
|
|
Trash and the space used by other Google services such as Gmail.
|
|
This command does not take any path arguments.
|
|
.SS Import/Export of google documents
|
|
.PP
|
|
Google documents can be exported from and uploaded to Google Drive.
|
|
.PP
|
|
When rclone downloads a Google doc it chooses a format to download
|
|
depending upon the \f[C]--drive-export-formats\f[R] setting.
|
|
By default the export formats are \f[C]docx,xlsx,pptx,svg\f[R] which are
|
|
a sensible default for an editable document.
|
|
.PP
|
|
When choosing a format, rclone runs down the list provided in order and
|
|
chooses the first file format the doc can be exported as from the list.
|
|
If the file can\[aq]t be exported to a format on the formats list, then
|
|
rclone will choose a format from the default list.
|
|
.PP
|
|
If you prefer an archive copy then you might use
|
|
\f[C]--drive-export-formats pdf\f[R], or if you prefer
|
|
openoffice/libreoffice formats you might use
|
|
\f[C]--drive-export-formats ods,odt,odp\f[R].
|
|
.PP
|
|
Note that rclone adds the extension to the google doc, so if it is
|
|
called \f[C]My Spreadsheet\f[R] on google docs, it will be exported as
|
|
\f[C]My Spreadsheet.xlsx\f[R] or \f[C]My Spreadsheet.pdf\f[R] etc.
|
|
.PP
|
|
When importing files into Google Drive, rclone will convert all files
|
|
with an extension in \f[C]--drive-import-formats\f[R] to their
|
|
associated document type.
|
|
rclone will not convert any files by default, since the conversion is
|
|
lossy process.
|
|
.PP
|
|
The conversion must result in a file with the same extension when the
|
|
\f[C]--drive-export-formats\f[R] rules are applied to the uploaded
|
|
document.
|
|
.PP
|
|
Here are some examples for allowed and prohibited conversions.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l l l l l.
|
|
T{
|
|
export-formats
|
|
T}@T{
|
|
import-formats
|
|
T}@T{
|
|
Upload Ext
|
|
T}@T{
|
|
Document Ext
|
|
T}@T{
|
|
Allowed
|
|
T}
|
|
_
|
|
T{
|
|
odt
|
|
T}@T{
|
|
odt
|
|
T}@T{
|
|
odt
|
|
T}@T{
|
|
odt
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
odt
|
|
T}@T{
|
|
docx,odt
|
|
T}@T{
|
|
odt
|
|
T}@T{
|
|
odt
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
T}@T{
|
|
docx
|
|
T}@T{
|
|
docx
|
|
T}@T{
|
|
docx
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
T}@T{
|
|
odt
|
|
T}@T{
|
|
odt
|
|
T}@T{
|
|
docx
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
odt,docx
|
|
T}@T{
|
|
docx,odt
|
|
T}@T{
|
|
docx
|
|
T}@T{
|
|
odt
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
docx,odt
|
|
T}@T{
|
|
docx,odt
|
|
T}@T{
|
|
docx
|
|
T}@T{
|
|
docx
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
docx,odt
|
|
T}@T{
|
|
docx,odt
|
|
T}@T{
|
|
odt
|
|
T}@T{
|
|
docx
|
|
T}@T{
|
|
No
|
|
T}
|
|
.TE
|
|
.PP
|
|
This limitation can be disabled by specifying
|
|
\f[C]--drive-allow-import-name-change\f[R].
|
|
When using this flag, rclone can convert multiple files types resulting
|
|
in the same document type at once, e.g.
|
|
with \f[C]--drive-import-formats docx,odt,txt\f[R], all files having
|
|
these extension would result in a document represented as a docx file.
|
|
This brings the additional risk of overwriting a document, if multiple
|
|
files have the same stem.
|
|
Many rclone operations will not handle this name change in any way.
|
|
They assume an equal name when copying files and might copy the file
|
|
again or delete them when the name changes.
|
|
.PP
|
|
Here are the possible export extensions with their corresponding mime
|
|
types.
|
|
Most of these can also be used for importing, but there more that are
|
|
not listed here.
|
|
Some of these additional ones might only be available when the operating
|
|
system provides the correct MIME type entries.
|
|
.PP
|
|
This list can be changed by Google Drive at any time and might not
|
|
represent the currently available conversions.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
lw(19.7n) lw(24.1n) lw(26.2n).
|
|
T{
|
|
Extension
|
|
T}@T{
|
|
Mime Type
|
|
T}@T{
|
|
Description
|
|
T}
|
|
_
|
|
T{
|
|
csv
|
|
T}@T{
|
|
text/csv
|
|
T}@T{
|
|
Standard CSV format for Spreadsheets
|
|
T}
|
|
T{
|
|
docx
|
|
T}@T{
|
|
application/vnd.openxmlformats-officedocument.wordprocessingml.document
|
|
T}@T{
|
|
Microsoft Office Document
|
|
T}
|
|
T{
|
|
epub
|
|
T}@T{
|
|
application/epub+zip
|
|
T}@T{
|
|
E-book format
|
|
T}
|
|
T{
|
|
html
|
|
T}@T{
|
|
text/html
|
|
T}@T{
|
|
An HTML Document
|
|
T}
|
|
T{
|
|
jpg
|
|
T}@T{
|
|
image/jpeg
|
|
T}@T{
|
|
A JPEG Image File
|
|
T}
|
|
T{
|
|
json
|
|
T}@T{
|
|
application/vnd.google-apps.script+json
|
|
T}@T{
|
|
JSON Text Format
|
|
T}
|
|
T{
|
|
odp
|
|
T}@T{
|
|
application/vnd.oasis.opendocument.presentation
|
|
T}@T{
|
|
Openoffice Presentation
|
|
T}
|
|
T{
|
|
ods
|
|
T}@T{
|
|
application/vnd.oasis.opendocument.spreadsheet
|
|
T}@T{
|
|
Openoffice Spreadsheet
|
|
T}
|
|
T{
|
|
ods
|
|
T}@T{
|
|
application/x-vnd.oasis.opendocument.spreadsheet
|
|
T}@T{
|
|
Openoffice Spreadsheet
|
|
T}
|
|
T{
|
|
odt
|
|
T}@T{
|
|
application/vnd.oasis.opendocument.text
|
|
T}@T{
|
|
Openoffice Document
|
|
T}
|
|
T{
|
|
pdf
|
|
T}@T{
|
|
application/pdf
|
|
T}@T{
|
|
Adobe PDF Format
|
|
T}
|
|
T{
|
|
png
|
|
T}@T{
|
|
image/png
|
|
T}@T{
|
|
PNG Image Format
|
|
T}
|
|
T{
|
|
pptx
|
|
T}@T{
|
|
application/vnd.openxmlformats-officedocument.presentationml.presentation
|
|
T}@T{
|
|
Microsoft Office Powerpoint
|
|
T}
|
|
T{
|
|
rtf
|
|
T}@T{
|
|
application/rtf
|
|
T}@T{
|
|
Rich Text Format
|
|
T}
|
|
T{
|
|
svg
|
|
T}@T{
|
|
image/svg+xml
|
|
T}@T{
|
|
Scalable Vector Graphics Format
|
|
T}
|
|
T{
|
|
tsv
|
|
T}@T{
|
|
text/tab-separated-values
|
|
T}@T{
|
|
Standard TSV format for spreadsheets
|
|
T}
|
|
T{
|
|
txt
|
|
T}@T{
|
|
text/plain
|
|
T}@T{
|
|
Plain Text
|
|
T}
|
|
T{
|
|
xlsx
|
|
T}@T{
|
|
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
|
|
T}@T{
|
|
Microsoft Office Spreadsheet
|
|
T}
|
|
T{
|
|
zip
|
|
T}@T{
|
|
application/zip
|
|
T}@T{
|
|
A ZIP file of HTML, Images CSS
|
|
T}
|
|
.TE
|
|
.PP
|
|
Google documents can also be exported as link files.
|
|
These files will open a browser window for the Google Docs website of
|
|
that document when opened.
|
|
The link file extension has to be specified as a
|
|
\f[C]--drive-export-formats\f[R] parameter.
|
|
They will match all available Google Documents.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l l l.
|
|
T{
|
|
Extension
|
|
T}@T{
|
|
Description
|
|
T}@T{
|
|
OS Support
|
|
T}
|
|
_
|
|
T{
|
|
desktop
|
|
T}@T{
|
|
freedesktop.org specified desktop entry
|
|
T}@T{
|
|
Linux
|
|
T}
|
|
T{
|
|
link.html
|
|
T}@T{
|
|
An HTML Document with a redirect
|
|
T}@T{
|
|
All
|
|
T}
|
|
T{
|
|
url
|
|
T}@T{
|
|
INI style link file
|
|
T}@T{
|
|
macOS, Windows
|
|
T}
|
|
T{
|
|
webloc
|
|
T}@T{
|
|
macOS specific XML format
|
|
T}@T{
|
|
macOS
|
|
T}
|
|
.TE
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to drive (Google Drive).
|
|
.SS --drive-client-id
|
|
.PP
|
|
Google Application Client Id Setting your own is recommended.
|
|
See https://rclone.org/drive/#making-your-own-client-id for how to
|
|
create your own.
|
|
If you leave this blank, it will use an internal key which is low
|
|
performance.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-client-secret
|
|
.PP
|
|
OAuth Client Secret.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_CLIENT_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-scope
|
|
.PP
|
|
Scope that rclone should use when requesting access from drive.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: scope
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_SCOPE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]drive\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Full access all files, excluding Application Data Folder.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]drive.readonly\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Read-only access to file metadata and file contents.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]drive.file\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Access to files created by rclone only.
|
|
.IP \[bu] 2
|
|
These are visible in the drive website.
|
|
.IP \[bu] 2
|
|
File authorization is revoked when the user deauthorizes the app.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]drive.appfolder\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allows read and write access to the Application Data folder.
|
|
.IP \[bu] 2
|
|
This is not visible in the drive website.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]drive.metadata.readonly\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allows read-only access to file metadata but
|
|
.IP \[bu] 2
|
|
does not allow any access to read or download file content.
|
|
.RE
|
|
.RE
|
|
.SS --drive-root-folder-id
|
|
.PP
|
|
ID of the root folder.
|
|
Leave blank normally.
|
|
.PP
|
|
Fill in to access \[dq]Computers\[dq] folders (see docs), or for rclone
|
|
to use a non root folder as its starting point.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: root_folder_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_ROOT_FOLDER_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-service-account-file
|
|
.PP
|
|
Service Account Credentials JSON file path.
|
|
.PP
|
|
Leave blank normally.
|
|
Needed only if you want use SA instead of interactive login.
|
|
.PP
|
|
Leading \f[C]\[ti]\f[R] will be expanded in the file name as will
|
|
environment variables such as \f[C]${RCLONE_CONFIG_DIR}\f[R].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: service_account_file
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_FILE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-alternate-export
|
|
.PP
|
|
Deprecated: No longer needed.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: alternate_export
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_ALTERNATE_EXPORT
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to drive (Google Drive).
|
|
.SS --drive-token
|
|
.PP
|
|
OAuth Access Token as a JSON blob.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-auth-url
|
|
.PP
|
|
Auth server URL.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_AUTH_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-token-url
|
|
.PP
|
|
Token server url.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_TOKEN_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-service-account-credentials
|
|
.PP
|
|
Service Account Credentials JSON blob.
|
|
.PP
|
|
Leave blank normally.
|
|
Needed only if you want use SA instead of interactive login.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: service_account_credentials
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_CREDENTIALS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-team-drive
|
|
.PP
|
|
ID of the Shared Drive (Team Drive).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: team_drive
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_TEAM_DRIVE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-auth-owner-only
|
|
.PP
|
|
Only consider files owned by the authenticated user.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_owner_only
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_AUTH_OWNER_ONLY
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-use-trash
|
|
.PP
|
|
Send files to the trash instead of deleting permanently.
|
|
.PP
|
|
Defaults to true, namely sending files to the trash.
|
|
Use \f[C]--drive-use-trash=false\f[R] to delete files permanently
|
|
instead.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: use_trash
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_USE_TRASH
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: true
|
|
.SS --drive-copy-shortcut-content
|
|
.PP
|
|
Server side copy contents of shortcuts instead of the shortcut.
|
|
.PP
|
|
When doing server side copies, normally rclone will copy shortcuts as
|
|
shortcuts.
|
|
.PP
|
|
If this flag is used then rclone will copy the contents of shortcuts
|
|
rather than shortcuts themselves when doing server side copies.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: copy_shortcut_content
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_COPY_SHORTCUT_CONTENT
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-skip-gdocs
|
|
.PP
|
|
Skip google documents in all listings.
|
|
.PP
|
|
If given, gdocs practically become invisible to rclone.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: skip_gdocs
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_SKIP_GDOCS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-skip-checksum-gphotos
|
|
.PP
|
|
Skip MD5 checksum on Google photos and videos only.
|
|
.PP
|
|
Use this if you get checksum errors when transferring Google photos or
|
|
videos.
|
|
.PP
|
|
Setting this flag will cause Google photos and videos to return a blank
|
|
MD5 checksum.
|
|
.PP
|
|
Google photos are identified by being in the \[dq]photos\[dq] space.
|
|
.PP
|
|
Corrupted checksums are caused by Google modifying the image/video but
|
|
not updating the checksum.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: skip_checksum_gphotos
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_SKIP_CHECKSUM_GPHOTOS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-shared-with-me
|
|
.PP
|
|
Only show files that are shared with me.
|
|
.PP
|
|
Instructs rclone to operate on your \[dq]Shared with me\[dq] folder
|
|
(where Google Drive lets you access the files and folders others have
|
|
shared with you).
|
|
.PP
|
|
This works both with the \[dq]list\[dq] (lsd, lsl, etc.) and the
|
|
\[dq]copy\[dq] commands (copy, sync, etc.), and with all other commands
|
|
too.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: shared_with_me
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_SHARED_WITH_ME
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-trashed-only
|
|
.PP
|
|
Only show files that are in the trash.
|
|
.PP
|
|
This will show trashed files in their original directory structure.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: trashed_only
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_TRASHED_ONLY
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-starred-only
|
|
.PP
|
|
Only show files that are starred.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: starred_only
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_STARRED_ONLY
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-formats
|
|
.PP
|
|
Deprecated: See export_formats.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: formats
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_FORMATS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-export-formats
|
|
.PP
|
|
Comma separated list of preferred formats for downloading Google docs.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: export_formats
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_EXPORT_FORMATS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]docx,xlsx,pptx,svg\[dq]
|
|
.SS --drive-import-formats
|
|
.PP
|
|
Comma separated list of preferred formats for uploading Google docs.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: import_formats
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_IMPORT_FORMATS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-allow-import-name-change
|
|
.PP
|
|
Allow the filetype to change when uploading Google docs.
|
|
.PP
|
|
E.g.
|
|
file.doc to file.docx.
|
|
This will confuse sync and reupload every time.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: allow_import_name_change
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_ALLOW_IMPORT_NAME_CHANGE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-use-created-date
|
|
.PP
|
|
Use file created date instead of modified date.
|
|
.PP
|
|
Useful when downloading data and you want the creation date used in
|
|
place of the last modified date.
|
|
.PP
|
|
\f[B]WARNING\f[R]: This flag may have some unexpected consequences.
|
|
.PP
|
|
When uploading to your drive all files will be overwritten unless they
|
|
haven\[aq]t been modified since their creation.
|
|
And the inverse will occur while downloading.
|
|
This side effect can be avoided by using the \[dq]--checksum\[dq] flag.
|
|
.PP
|
|
This feature was implemented to retain photos capture date as recorded
|
|
by google photos.
|
|
You will first need to check the \[dq]Create a Google Photos folder\[dq]
|
|
option in your google drive settings.
|
|
You can then copy or move the photos locally and use the date the image
|
|
was taken (created) set as the modification date.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: use_created_date
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_USE_CREATED_DATE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-use-shared-date
|
|
.PP
|
|
Use date file was shared instead of modified date.
|
|
.PP
|
|
Note that, as with \[dq]--drive-use-created-date\[dq], this flag may
|
|
have unexpected consequences when uploading/downloading files.
|
|
.PP
|
|
If both this flag and \[dq]--drive-use-created-date\[dq] are set, the
|
|
created date is used.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: use_shared_date
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_USE_SHARED_DATE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-list-chunk
|
|
.PP
|
|
Size of listing chunk 100-1000, 0 to disable.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: list_chunk
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_LIST_CHUNK
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 1000
|
|
.SS --drive-impersonate
|
|
.PP
|
|
Impersonate this user when using a service account.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: impersonate
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_IMPERSONATE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --drive-upload-cutoff
|
|
.PP
|
|
Cutoff for switching to chunked upload.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_cutoff
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_UPLOAD_CUTOFF
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 8Mi
|
|
.SS --drive-chunk-size
|
|
.PP
|
|
Upload chunk size.
|
|
.PP
|
|
Must a power of 2 >= 256k.
|
|
.PP
|
|
Making this larger will improve performance, but note that each chunk is
|
|
buffered in memory one per transfer.
|
|
.PP
|
|
Reducing this will reduce memory usage but decrease performance.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 8Mi
|
|
.SS --drive-acknowledge-abuse
|
|
.PP
|
|
Set to allow files which return cannotDownloadAbusiveFile to be
|
|
downloaded.
|
|
.PP
|
|
If downloading a file returns the error \[dq]This file has been
|
|
identified as malware or spam and cannot be downloaded\[dq] with the
|
|
error code \[dq]cannotDownloadAbusiveFile\[dq] then supply this flag to
|
|
rclone to indicate you acknowledge the risks of downloading the file and
|
|
rclone will download it anyway.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: acknowledge_abuse
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_ACKNOWLEDGE_ABUSE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-keep-revision-forever
|
|
.PP
|
|
Keep new head revision of each file forever.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: keep_revision_forever
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_KEEP_REVISION_FOREVER
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-size-as-quota
|
|
.PP
|
|
Show sizes as storage quota usage, not actual size.
|
|
.PP
|
|
Show the size of a file as the storage quota used.
|
|
This is the current version plus any older versions that have been set
|
|
to keep forever.
|
|
.PP
|
|
\f[B]WARNING\f[R]: This flag may have some unexpected consequences.
|
|
.PP
|
|
It is not recommended to set this flag in your config - the recommended
|
|
usage is using the flag form --drive-size-as-quota when doing rclone
|
|
ls/lsl/lsf/lsjson/etc only.
|
|
.PP
|
|
If you do use this flag for syncing (not recommended) then you will need
|
|
to use --ignore size also.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: size_as_quota
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_SIZE_AS_QUOTA
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-v2-download-min-size
|
|
.PP
|
|
If Object\[aq]s are greater, use drive v2 API to download.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: v2_download_min_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_V2_DOWNLOAD_MIN_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: off
|
|
.SS --drive-pacer-min-sleep
|
|
.PP
|
|
Minimum time to sleep between API calls.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: pacer_min_sleep
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_PACER_MIN_SLEEP
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 100ms
|
|
.SS --drive-pacer-burst
|
|
.PP
|
|
Number of API calls to allow without sleeping.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: pacer_burst
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_PACER_BURST
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 100
|
|
.SS --drive-server-side-across-configs
|
|
.PP
|
|
Allow server-side operations (e.g.
|
|
copy) to work across different drive configs.
|
|
.PP
|
|
This can be useful if you wish to do a server-side copy between two
|
|
different Google drives.
|
|
Note that this isn\[aq]t enabled by default because it isn\[aq]t easy to
|
|
tell if it will work between any two configurations.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: server_side_across_configs
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_SERVER_SIDE_ACROSS_CONFIGS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-disable-http2
|
|
.PP
|
|
Disable drive using http2.
|
|
.PP
|
|
There is currently an unsolved issue with the google drive backend and
|
|
HTTP/2.
|
|
HTTP/2 is therefore disabled by default for the drive backend but can be
|
|
re-enabled here.
|
|
When the issue is solved this flag will be removed.
|
|
.PP
|
|
See: https://github.com/rclone/rclone/issues/3631
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_http2
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_DISABLE_HTTP2
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: true
|
|
.SS --drive-stop-on-upload-limit
|
|
.PP
|
|
Make upload limit errors be fatal.
|
|
.PP
|
|
At the time of writing it is only possible to upload 750 GiB of data to
|
|
Google Drive a day (this is an undocumented limit).
|
|
When this limit is reached Google Drive produces a slightly different
|
|
error message.
|
|
When this flag is set it causes these errors to be fatal.
|
|
These will stop the in-progress sync.
|
|
.PP
|
|
Note that this detection is relying on error message strings which
|
|
Google don\[aq]t document so it may break in the future.
|
|
.PP
|
|
See: https://github.com/rclone/rclone/issues/3857
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: stop_on_upload_limit
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_STOP_ON_UPLOAD_LIMIT
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-stop-on-download-limit
|
|
.PP
|
|
Make download limit errors be fatal.
|
|
.PP
|
|
At the time of writing it is only possible to download 10 TiB of data
|
|
from Google Drive a day (this is an undocumented limit).
|
|
When this limit is reached Google Drive produces a slightly different
|
|
error message.
|
|
When this flag is set it causes these errors to be fatal.
|
|
These will stop the in-progress sync.
|
|
.PP
|
|
Note that this detection is relying on error message strings which
|
|
Google don\[aq]t document so it may break in the future.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: stop_on_download_limit
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_STOP_ON_DOWNLOAD_LIMIT
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-skip-shortcuts
|
|
.PP
|
|
If set skip shortcut files.
|
|
.PP
|
|
Normally rclone dereferences shortcut files making them appear as if
|
|
they are the original file (see the shortcuts section).
|
|
If this flag is set then rclone will ignore shortcut files completely.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: skip_shortcuts
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_SKIP_SHORTCUTS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-skip-dangling-shortcuts
|
|
.PP
|
|
If set skip dangling shortcut files.
|
|
.PP
|
|
If this is set then rclone will not show any dangling shortcuts in
|
|
listings.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: skip_dangling_shortcuts
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_SKIP_DANGLING_SHORTCUTS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --drive-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_DRIVE_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: InvalidUtf8
|
|
.SS Backend commands
|
|
.PP
|
|
Here are the commands specific to the drive backend.
|
|
.PP
|
|
Run them with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend COMMAND remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The help below will explain what arguments each command takes.
|
|
.PP
|
|
See the \[dq]rclone backend\[dq]
|
|
command (https://rclone.org/commands/rclone_backend/) for more info on
|
|
how to pass options and arguments.
|
|
.PP
|
|
These can be run on a running backend using the rc command
|
|
backend/command (https://rclone.org/rc/#backend-command).
|
|
.SS get
|
|
.PP
|
|
Get command for fetching the drive config parameters
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend get remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This is a get command which will be used to fetch the various drive
|
|
config parameters
|
|
.PP
|
|
Usage Examples:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend get drive: [-o service_account_file] [-o chunk_size]
|
|
rclone rc backend/command command=get fs=drive: [-o service_account_file] [-o chunk_size]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Options:
|
|
.IP \[bu] 2
|
|
\[dq]chunk_size\[dq]: show the current upload chunk size
|
|
.IP \[bu] 2
|
|
\[dq]service_account_file\[dq]: show the current service account file
|
|
.SS set
|
|
.PP
|
|
Set command for updating the drive config parameters
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend set remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This is a set command which will be used to update the various drive
|
|
config parameters
|
|
.PP
|
|
Usage Examples:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend set drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
|
|
rclone rc backend/command command=set fs=drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Options:
|
|
.IP \[bu] 2
|
|
\[dq]chunk_size\[dq]: update the current upload chunk size
|
|
.IP \[bu] 2
|
|
\[dq]service_account_file\[dq]: update the current service account file
|
|
.SS shortcut
|
|
.PP
|
|
Create shortcuts from files or directories
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend shortcut remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This command creates shortcuts from files or directories.
|
|
.PP
|
|
Usage:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend shortcut drive: source_item destination_shortcut
|
|
rclone backend shortcut drive: source_item -o target=drive2: destination_shortcut
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
In the first example this creates a shortcut from the
|
|
\[dq]source_item\[dq] which can be a file or a directory to the
|
|
\[dq]destination_shortcut\[dq].
|
|
The \[dq]source_item\[dq] and the \[dq]destination_shortcut\[dq] should
|
|
be relative paths from \[dq]drive:\[dq]
|
|
.PP
|
|
In the second example this creates a shortcut from the
|
|
\[dq]source_item\[dq] relative to \[dq]drive:\[dq] to the
|
|
\[dq]destination_shortcut\[dq] relative to \[dq]drive2:\[dq].
|
|
This may fail with a permission error if the user authenticated with
|
|
\[dq]drive2:\[dq] can\[aq]t read files from \[dq]drive:\[dq].
|
|
.PP
|
|
Options:
|
|
.IP \[bu] 2
|
|
\[dq]target\[dq]: optional target remote for the shortcut destination
|
|
.SS drives
|
|
.PP
|
|
List the Shared Drives available to this account
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend drives remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This command lists the Shared Drives (Team Drives) available to this
|
|
account.
|
|
.PP
|
|
Usage:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend [-o config] drives drive:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will return a JSON list of objects like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[
|
|
{
|
|
\[dq]id\[dq]: \[dq]0ABCDEF-01234567890\[dq],
|
|
\[dq]kind\[dq]: \[dq]drive#teamDrive\[dq],
|
|
\[dq]name\[dq]: \[dq]My Drive\[dq]
|
|
},
|
|
{
|
|
\[dq]id\[dq]: \[dq]0ABCDEFabcdefghijkl\[dq],
|
|
\[dq]kind\[dq]: \[dq]drive#teamDrive\[dq],
|
|
\[dq]name\[dq]: \[dq]Test Drive\[dq]
|
|
}
|
|
]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
With the -o config parameter it will output the list in a format
|
|
suitable for adding to a config file to make aliases for all the drives
|
|
found.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[My Drive]
|
|
type = alias
|
|
remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:
|
|
|
|
[Test Drive]
|
|
type = alias
|
|
remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Adding this to the rclone config file will cause those team drives to be
|
|
accessible with the aliases shown.
|
|
This may require manual editing of the names.
|
|
.SS untrash
|
|
.PP
|
|
Untrash files and directories
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend untrash remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This command untrashes all the files and directories in the directory
|
|
passed in recursively.
|
|
.PP
|
|
Usage:
|
|
.PP
|
|
This takes an optional directory to trash which make this easier to use
|
|
via the API.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend untrash drive:directory
|
|
rclone backend -i untrash drive:directory subdir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Use the -i flag to see what would be restored before restoring it.
|
|
.PP
|
|
Result:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\[dq]Untrashed\[dq]: 17,
|
|
\[dq]Errors\[dq]: 0
|
|
}
|
|
\f[R]
|
|
.fi
|
|
.SS copyid
|
|
.PP
|
|
Copy files by ID
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend copyid remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This command copies files by ID
|
|
.PP
|
|
Usage:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend copyid drive: ID path
|
|
rclone backend copyid drive: ID1 path1 ID2 path2
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
It copies the drive file with ID given to the path (an rclone path which
|
|
will be passed internally to rclone copyto).
|
|
The ID and path pairs can be repeated.
|
|
.PP
|
|
The path should end with a / to indicate copy the file as named to this
|
|
directory.
|
|
If it doesn\[aq]t end with a / then the last path component will be used
|
|
as the file name.
|
|
.PP
|
|
If the destination is a drive backend then server-side copying will be
|
|
attempted if possible.
|
|
.PP
|
|
Use the -i flag to see what would be copied before copying.
|
|
.SS Limitations
|
|
.PP
|
|
Drive has quite a lot of rate limiting.
|
|
This causes rclone to be limited to transferring about 2 files per
|
|
second only.
|
|
Individual files may be transferred much faster at 100s of MiB/s but
|
|
lots of small files can take a long time.
|
|
.PP
|
|
Server side copies are also subject to a separate rate limit.
|
|
If you see User rate limit exceeded errors, wait at least 24 hours and
|
|
retry.
|
|
You can disable server-side copies with \f[C]--disable copy\f[R] to
|
|
download and upload the files if you prefer.
|
|
.SS Limitations of Google Docs
|
|
.PP
|
|
Google docs will appear as size -1 in \f[C]rclone ls\f[R] and as size 0
|
|
in anything which uses the VFS layer, e.g.
|
|
\f[C]rclone mount\f[R], \f[C]rclone serve\f[R].
|
|
.PP
|
|
This is because rclone can\[aq]t find out the size of the Google docs
|
|
without downloading them.
|
|
.PP
|
|
Google docs will transfer correctly with \f[C]rclone sync\f[R],
|
|
\f[C]rclone copy\f[R] etc as rclone knows to ignore the size when doing
|
|
the transfer.
|
|
.PP
|
|
However an unfortunate consequence of this is that you may not be able
|
|
to download Google docs using \f[C]rclone mount\f[R].
|
|
If it doesn\[aq]t work you will get a 0 sized file.
|
|
If you try again the doc may gain its correct size and be downloadable.
|
|
Whether it will work on not depends on the application accessing the
|
|
mount and the OS you are running - experiment to find out if it does
|
|
work for you!
|
|
.SS Duplicated files
|
|
.PP
|
|
Sometimes, for no reason I\[aq]ve been able to track down, drive will
|
|
duplicate a file that rclone uploads.
|
|
Drive unlike all the other remotes can have duplicated files.
|
|
.PP
|
|
Duplicated files cause problems with the syncing and you will see
|
|
messages in the log about duplicates.
|
|
.PP
|
|
Use \f[C]rclone dedupe\f[R] to fix duplicated files.
|
|
.PP
|
|
Note that this isn\[aq]t just a problem with rclone, even Google Photos
|
|
on Android duplicates files on drive sometimes.
|
|
.SS Rclone appears to be re-copying files it shouldn\[aq]t
|
|
.PP
|
|
The most likely cause of this is the duplicated file issue above - run
|
|
\f[C]rclone dedupe\f[R] and check your logs for duplicate object or
|
|
directory messages.
|
|
.PP
|
|
This can also be caused by a delay/caching on google drive\[aq]s end
|
|
when comparing directory listings.
|
|
Specifically with team drives used in combination with --fast-list.
|
|
Files that were uploaded recently may not appear on the directory list
|
|
sent to rclone when using --fast-list.
|
|
.PP
|
|
Waiting a moderate period of time between attempts (estimated to be
|
|
approximately 1 hour) and/or not using --fast-list both seem to be
|
|
effective in preventing the problem.
|
|
.SS Making your own client_id
|
|
.PP
|
|
When you use rclone with Google drive in its default configuration you
|
|
are using rclone\[aq]s client_id.
|
|
This is shared between all the rclone users.
|
|
There is a global rate limit on the number of queries per second that
|
|
each client_id can do set by Google.
|
|
rclone already has a high quota and I will continue to make sure it is
|
|
high enough by contacting Google.
|
|
.PP
|
|
It is strongly recommended to use your own client ID as the default
|
|
rclone ID is heavily used.
|
|
If you have multiple services running, it is recommended to use an API
|
|
key for each service.
|
|
The default Google quota is 10 transactions per second so it is
|
|
recommended to stay under that number as if you use more than that, it
|
|
will cause rclone to rate limit and make things slower.
|
|
.PP
|
|
Here is how to create your own Google Drive client ID for rclone:
|
|
.IP "1." 3
|
|
Log into the Google API Console (https://console.developers.google.com/)
|
|
with your Google account.
|
|
It doesn\[aq]t matter what Google account you use.
|
|
(It need not be the same account as the Google Drive you want to access)
|
|
.IP "2." 3
|
|
Select a project or create a new project.
|
|
.IP "3." 3
|
|
Under \[dq]ENABLE APIS AND SERVICES\[dq] search for \[dq]Drive\[dq], and
|
|
enable the \[dq]Google Drive API\[dq].
|
|
.IP "4." 3
|
|
Click \[dq]Credentials\[dq] in the left-side panel (not \[dq]Create
|
|
credentials\[dq], which opens the wizard), then \[dq]Create
|
|
credentials\[dq]
|
|
.IP "5." 3
|
|
If you already configured an \[dq]Oauth Consent Screen\[dq], then skip
|
|
to the next step; if not, click on \[dq]CONFIGURE CONSENT SCREEN\[dq]
|
|
button (near the top right corner of the right panel), then select
|
|
\[dq]External\[dq] and click on \[dq]CREATE\[dq]; on the next screen,
|
|
enter an \[dq]Application name\[dq] (\[dq]rclone\[dq] is OK); enter
|
|
\[dq]User Support Email\[dq] (your own email is OK); enter
|
|
\[dq]Developer Contact Email\[dq] (your own email is OK); then click on
|
|
\[dq]Save\[dq] (all other data is optional).
|
|
Click again on \[dq]Credentials\[dq] on the left panel to go back to the
|
|
\[dq]Credentials\[dq] screen.
|
|
.PP
|
|
(PS: if you are a GSuite user, you could also select \[dq]Internal\[dq]
|
|
instead of \[dq]External\[dq] above, but this has not been
|
|
tested/documented so far).
|
|
.IP " 6." 4
|
|
Click on the \[dq]+ CREATE CREDENTIALS\[dq] button at the top of the
|
|
screen, then select \[dq]OAuth client ID\[dq].
|
|
.IP " 7." 4
|
|
Choose an application type of \[dq]Desktop app\[dq] and click
|
|
\[dq]Create\[dq].
|
|
(the default name is fine)
|
|
.IP " 8." 4
|
|
It will show you a client ID and client secret.
|
|
Make a note of these.
|
|
.IP " 9." 4
|
|
Go to \[dq]Oauth consent screen\[dq] and press \[dq]Publish App\[dq]
|
|
.IP "10." 4
|
|
Provide the noted client ID and client secret to rclone.
|
|
.IP "11." 4
|
|
Click \[dq]OAuth consent screen\[dq], then click \[dq]PUBLISH APP\[dq]
|
|
button and confirm, or add your account under \[dq]Test users\[dq].
|
|
.PP
|
|
Be aware that, due to the \[dq]enhanced security\[dq] recently
|
|
introduced by Google, you are theoretically expected to \[dq]submit your
|
|
app for verification\[dq] and then wait a few weeks(!) for their
|
|
response; in practice, you can go right ahead and use the client ID and
|
|
client secret with rclone, the only issue will be a very scary
|
|
confirmation screen shown when you connect via your browser for rclone
|
|
to be able to get its token-id (but as this only happens during the
|
|
remote configuration, it\[aq]s not such a big deal).
|
|
.PP
|
|
(Thanks to \[at]balazer on github for these instructions.)
|
|
.PP
|
|
Sometimes, creation of an OAuth consent in Google API Console fails due
|
|
to an error message \[lq]The request failed because changes to one of
|
|
the field of the resource is not supported\[rq].
|
|
As a convenient workaround, the necessary Google Drive API key can be
|
|
created on the Python
|
|
Quickstart (https://developers.google.com/drive/api/v3/quickstart/python)
|
|
page.
|
|
Just push the Enable the Drive API button to receive the Client ID and
|
|
Secret.
|
|
Note that it will automatically create a new project in the API Console.
|
|
.SH Google Photos
|
|
.PP
|
|
The rclone backend for Google
|
|
Photos (https://www.google.com/photos/about/) is a specialized backend
|
|
for transferring photos and videos to and from Google Photos.
|
|
.PP
|
|
\f[B]NB\f[R] The Google Photos API which rclone uses has quite a few
|
|
limitations, so please read the limitations section carefully to make
|
|
sure it is suitable for your use.
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for google cloud storage involves getting a token from
|
|
Google Photos which you need to do in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Google Photos
|
|
\[rs] \[dq]google photos\[dq]
|
|
[snip]
|
|
Storage> google photos
|
|
** See help for google photos backend at: https://rclone.org/googlephotos/ **
|
|
|
|
Google Application Client Id
|
|
Leave blank normally.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
client_id>
|
|
Google Application Client Secret
|
|
Leave blank normally.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
client_secret>
|
|
Set to make the Google Photos backend read only.
|
|
|
|
If you choose read only then rclone will only request read only access
|
|
to your photos, otherwise rclone will request full access.
|
|
Enter a boolean value (true or false). Press Enter for the default (\[dq]false\[dq]).
|
|
read_only>
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No
|
|
y/n> n
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
|
|
*** IMPORTANT: All media items uploaded to Google Photos with rclone
|
|
*** are stored in full resolution at original quality. These uploads
|
|
*** will count towards storage in your Google Account.
|
|
|
|
--------------------
|
|
[remote]
|
|
type = google photos
|
|
token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2019-06-28T17:38:04.644930156+01:00\[dq]}
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Google if you use auto config mode.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and this may require you to
|
|
unblock it temporarily if you are running a host firewall, or use manual
|
|
mode.
|
|
.PP
|
|
This remote is called \f[C]remote\f[R] and can now be used like this
|
|
.PP
|
|
See all the albums in your photos
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:album
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Make a new album
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:album/newAlbum
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of an album
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:album/newAlbum
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/images\f[R] to the Google Photos, removing any
|
|
excess files in the album.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/image remote:album/newAlbum
|
|
\f[R]
|
|
.fi
|
|
.SS Layout
|
|
.PP
|
|
As Google Photos is not a general purpose cloud storage system the
|
|
backend is laid out to help you navigate it.
|
|
.PP
|
|
The directories under \f[C]media\f[R] show different ways of
|
|
categorizing the media.
|
|
Each file will appear multiple times.
|
|
So if you want to make a backup of your google photos you might choose
|
|
to backup \f[C]remote:media/by-month\f[R].
|
|
(\f[B]NB\f[R] \f[C]remote:media/by-day\f[R] is rather slow at the moment
|
|
so avoid for syncing.)
|
|
.PP
|
|
Note that all your photos and videos will appear somewhere under
|
|
\f[C]media\f[R], but they may not appear under \f[C]album\f[R] unless
|
|
you\[aq]ve put them into albums.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/
|
|
- upload
|
|
- file1.jpg
|
|
- file2.jpg
|
|
- ...
|
|
- media
|
|
- all
|
|
- file1.jpg
|
|
- file2.jpg
|
|
- ...
|
|
- by-year
|
|
- 2000
|
|
- file1.jpg
|
|
- ...
|
|
- 2001
|
|
- file2.jpg
|
|
- ...
|
|
- ...
|
|
- by-month
|
|
- 2000
|
|
- 2000-01
|
|
- file1.jpg
|
|
- ...
|
|
- 2000-02
|
|
- file2.jpg
|
|
- ...
|
|
- ...
|
|
- by-day
|
|
- 2000
|
|
- 2000-01-01
|
|
- file1.jpg
|
|
- ...
|
|
- 2000-01-02
|
|
- file2.jpg
|
|
- ...
|
|
- ...
|
|
- album
|
|
- album name
|
|
- album name/sub
|
|
- shared-album
|
|
- album name
|
|
- album name/sub
|
|
- feature
|
|
- favorites
|
|
- file1.jpg
|
|
- file2.jpg
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
There are two writable parts of the tree, the \f[C]upload\f[R] directory
|
|
and sub directories of the \f[C]album\f[R] directory.
|
|
.PP
|
|
The \f[C]upload\f[R] directory is for uploading files you don\[aq]t want
|
|
to put into albums.
|
|
This will be empty to start with and will contain the files you\[aq]ve
|
|
uploaded for one rclone session only, becoming empty again when you
|
|
restart rclone.
|
|
The use case for this would be if you have a load of files you just want
|
|
to once off dump into Google Photos.
|
|
For repeated syncing, uploading to \f[C]album\f[R] will work better.
|
|
.PP
|
|
Directories within the \f[C]album\f[R] directory are also writeable and
|
|
you may create new directories (albums) under \f[C]album\f[R].
|
|
If you copy files with a directory hierarchy in there then rclone will
|
|
create albums with the \f[C]/\f[R] character in them.
|
|
For example if you do
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /path/to/images remote:album/images
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
and the images directory contains
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
images
|
|
- file1.jpg
|
|
dir
|
|
file2.jpg
|
|
dir2
|
|
dir3
|
|
file3.jpg
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Then rclone will create the following albums with the following files in
|
|
.IP \[bu] 2
|
|
images
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
file1.jpg
|
|
.RE
|
|
.IP \[bu] 2
|
|
images/dir
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
file2.jpg
|
|
.RE
|
|
.IP \[bu] 2
|
|
images/dir2/dir3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
file3.jpg
|
|
.RE
|
|
.PP
|
|
This means that you can use the \f[C]album\f[R] path pretty much like a
|
|
normal filesystem and it is a good target for repeated syncing.
|
|
.PP
|
|
The \f[C]shared-album\f[R] directory shows albums shared with you or by
|
|
you.
|
|
This is similar to the Sharing tab in the Google Photos web interface.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to google photos (Google Photos).
|
|
.SS --gphotos-client-id
|
|
.PP
|
|
OAuth Client Id.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GPHOTOS_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gphotos-client-secret
|
|
.PP
|
|
OAuth Client Secret.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GPHOTOS_CLIENT_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gphotos-read-only
|
|
.PP
|
|
Set to make the Google Photos backend read only.
|
|
.PP
|
|
If you choose read only then rclone will only request read only access
|
|
to your photos, otherwise rclone will request full access.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: read_only
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GPHOTOS_READ_ONLY
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to google photos (Google Photos).
|
|
.SS --gphotos-token
|
|
.PP
|
|
OAuth Access Token as a JSON blob.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GPHOTOS_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gphotos-auth-url
|
|
.PP
|
|
Auth server URL.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GPHOTOS_AUTH_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gphotos-token-url
|
|
.PP
|
|
Token server url.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GPHOTOS_TOKEN_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --gphotos-read-size
|
|
.PP
|
|
Set to read the size of media items.
|
|
.PP
|
|
Normally rclone does not read the size of media items since this takes
|
|
another transaction.
|
|
This isn\[aq]t necessary for syncing.
|
|
However rclone mount needs to know the size of files in advance of
|
|
reading them, so setting this flag when using rclone mount is
|
|
recommended if you want to read the media.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: read_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GPHOTOS_READ_SIZE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --gphotos-start-year
|
|
.PP
|
|
Year limits the photos to be downloaded to those which are uploaded
|
|
after the given year.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: start_year
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GPHOTOS_START_YEAR
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 2000
|
|
.SS --gphotos-include-archived
|
|
.PP
|
|
Also view and download archived media.
|
|
.PP
|
|
By default, rclone does not request archived media.
|
|
Thus, when syncing, archived media is not visible in directory listings
|
|
or transferred.
|
|
.PP
|
|
Note that media in albums is always visible and synced, no matter their
|
|
archive status.
|
|
.PP
|
|
With this flag, archived media are always visible in directory listings
|
|
and transferred.
|
|
.PP
|
|
Without this flag, archived media will not be visible in directory
|
|
listings and won\[aq]t be transferred.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: include_archived
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GPHOTOS_INCLUDE_ARCHIVED
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --gphotos-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_GPHOTOS_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,CrLf,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
Only images and videos can be uploaded.
|
|
If you attempt to upload non videos or images or formats that Google
|
|
Photos doesn\[aq]t understand, rclone will upload the file, then Google
|
|
Photos will give an error when it is put turned into a media item.
|
|
.PP
|
|
Note that all media items uploaded to Google Photos through the API are
|
|
stored in full resolution at \[dq]original quality\[dq] and
|
|
\f[B]will\f[R] count towards your storage quota in your Google Account.
|
|
The API does \f[B]not\f[R] offer a way to upload in \[dq]high
|
|
quality\[dq] mode..
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the Google Photos backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SS Downloading Images
|
|
.PP
|
|
When Images are downloaded this strips EXIF location (according to the
|
|
docs and my tests).
|
|
This is a limitation of the Google Photos API and is covered by bug
|
|
#112096115 (https://issuetracker.google.com/issues/112096115).
|
|
.PP
|
|
\f[B]The current google API does not allow photos to be downloaded at
|
|
original resolution. This is very important if you are, for example,
|
|
relying on \[dq]Google Photos\[dq] as a backup of your photos. You will
|
|
not be able to use rclone to redownload original images. You could use
|
|
\[aq]google takeout\[aq] to recover the original photos as a last
|
|
resort\f[R]
|
|
.SS Downloading Videos
|
|
.PP
|
|
When videos are downloaded they are downloaded in a really compressed
|
|
version of the video compared to downloading it via the Google Photos
|
|
web interface.
|
|
This is covered by bug
|
|
#113672044 (https://issuetracker.google.com/issues/113672044).
|
|
.SS Duplicates
|
|
.PP
|
|
If a file name is duplicated in a directory then rclone will add the
|
|
file ID into its name.
|
|
So two files called \f[C]file.jpg\f[R] would then appear as
|
|
\f[C]file {123456}.jpg\f[R] and \f[C]file {ABCDEF}.jpg\f[R] (the actual
|
|
IDs are a lot longer alas!).
|
|
.PP
|
|
If you upload the same image (with the same binary data) twice then
|
|
Google Photos will deduplicate it.
|
|
However it will retain the filename from the first upload which may
|
|
confuse rclone.
|
|
For example if you uploaded an image to \f[C]upload\f[R] then uploaded
|
|
the same image to \f[C]album/my_album\f[R] the filename of the image in
|
|
\f[C]album/my_album\f[R] will be what it was uploaded with initially,
|
|
not what you uploaded it with to \f[C]album\f[R].
|
|
In practise this shouldn\[aq]t cause too many problems.
|
|
.SS Modified time
|
|
.PP
|
|
The date shown of media in Google Photos is the creation date as
|
|
determined by the EXIF information, or the upload date if that is not
|
|
known.
|
|
.PP
|
|
This is not changeable by rclone and is not the modification date of the
|
|
media on local disk.
|
|
This means that rclone cannot use the dates from Google Photos for
|
|
syncing purposes.
|
|
.SS Size
|
|
.PP
|
|
The Google Photos API does not return the size of media.
|
|
This means that when syncing to Google Photos, rclone can only do a file
|
|
existence check.
|
|
.PP
|
|
It is possible to read the size of the media, but this needs an extra
|
|
HTTP HEAD request per media item so is \f[B]very slow\f[R] and uses up a
|
|
lot of transactions.
|
|
This can be enabled with the \f[C]--gphotos-read-size\f[R] option or the
|
|
\f[C]read_size = true\f[R] config parameter.
|
|
.PP
|
|
If you want to use the backend with \f[C]rclone mount\f[R] you may need
|
|
to enable this flag (depending on your OS and application using the
|
|
photos) otherwise you may not be able to read media off the mount.
|
|
You\[aq]ll need to experiment to see if it works for you without the
|
|
flag.
|
|
.SS Albums
|
|
.PP
|
|
Rclone can only upload files to albums it created.
|
|
This is a limitation of the Google Photos
|
|
API (https://developers.google.com/photos/library/guides/manage-albums).
|
|
.PP
|
|
Rclone can remove files it uploaded from albums it created only.
|
|
.SS Deleting files
|
|
.PP
|
|
Rclone can remove files from albums it created, but note that the Google
|
|
Photos API does not allow media to be deleted permanently so this media
|
|
will still remain.
|
|
See bug #109759781 (https://issuetracker.google.com/issues/109759781).
|
|
.PP
|
|
Rclone cannot delete files anywhere except under \f[C]album\f[R].
|
|
.SS Deleting albums
|
|
.PP
|
|
The Google Photos API does not support deleting albums - see bug
|
|
#135714733 (https://issuetracker.google.com/issues/135714733).
|
|
.SH Hasher (EXPERIMENTAL)
|
|
.PP
|
|
Hasher is a special overlay backend to create remotes which handle
|
|
checksums for other remotes.
|
|
It\[aq]s main functions include: - Emulate hash types unimplemented by
|
|
backends - Cache checksums to help with slow hashing of large local or
|
|
(S)FTP files - Warm up checksum cache from external SUM files
|
|
.SS Getting started
|
|
.PP
|
|
To use Hasher, first set up the underlying remote following the
|
|
configuration instructions for that remote.
|
|
You can also use a local pathname instead of a remote.
|
|
Check that your base remote is working.
|
|
.PP
|
|
Let\[aq]s call the base remote \f[C]myRemote:path\f[R] here.
|
|
Note that anything inside \f[C]myRemote:path\f[R] will be handled by
|
|
hasher and anything outside won\[aq]t.
|
|
This means that if you are using a bucket based remote (S3, B2, Swift)
|
|
then you should put the bucket in the remote \f[C]s3:bucket\f[R].
|
|
.PP
|
|
Now proceed to interactive or manual configuration.
|
|
.SS Interactive configuration
|
|
.PP
|
|
Run \f[C]rclone config\f[R]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> Hasher1
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Handle checksums for other remotes
|
|
\[rs] \[dq]hasher\[dq]
|
|
[snip]
|
|
Storage> hasher
|
|
Remote to cache checksums for, like myremote:mypath.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
remote> myRemote:path
|
|
Comma separated list of supported checksum types.
|
|
Enter a string value. Press Enter for the default (\[dq]md5,sha1\[dq]).
|
|
hashsums> md5
|
|
Maximum time to keep checksums in cache. 0 = no cache, off = cache forever.
|
|
max_age> off
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No
|
|
y/n> n
|
|
Remote config
|
|
--------------------
|
|
[Hasher1]
|
|
type = hasher
|
|
remote = myRemote:path
|
|
hashsums = md5
|
|
max_age = off
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.SS Manual configuration
|
|
.PP
|
|
Run \f[C]rclone config path\f[R] to see the path of current active
|
|
config file, usually \f[C]YOURHOME/.config/rclone/rclone.conf\f[R].
|
|
Open it in your favorite text editor, find section for the base remote
|
|
and create new section for hasher like in the following examples:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[Hasher1]
|
|
type = hasher
|
|
remote = myRemote:path
|
|
hashes = md5
|
|
max_age = off
|
|
|
|
[Hasher2]
|
|
type = hasher
|
|
remote = /local/path
|
|
hashes = dropbox,sha1
|
|
max_age = 24h
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Hasher takes basically the following parameters: - \f[C]remote\f[R] is
|
|
required, - \f[C]hashes\f[R] is a comma separated list of supported
|
|
checksums (by default \f[C]md5,sha1\f[R]), - \f[C]max_age\f[R] - maximum
|
|
time to keep a checksum value in the cache, \f[C]0\f[R] will disable
|
|
caching completely, \f[C]off\f[R] will cache \[dq]forever\[dq] (that is
|
|
until the files get changed).
|
|
.PP
|
|
Make sure the \f[C]remote\f[R] has \f[C]:\f[R] (colon) in.
|
|
If you specify the remote without a colon then rclone will use a local
|
|
directory of that name.
|
|
So if you use a remote of \f[C]/local/path\f[R] then rclone will handle
|
|
hashes for that directory.
|
|
If you use \f[C]remote = name\f[R] literally then rclone will put files
|
|
\f[B]in a directory called \f[CB]name\f[B] located under current
|
|
directory\f[R].
|
|
.SS Usage
|
|
.SS Basic operations
|
|
.PP
|
|
Now you can use it as \f[C]Hasher2:subdir/file\f[R] instead of base
|
|
remote.
|
|
Hasher will transparently update cache with new checksums when a file is
|
|
fully read or overwritten, like:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy External:path/file Hasher:dest/path
|
|
|
|
rclone cat Hasher:path/to/file > /dev/null
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The way to refresh \f[B]all\f[R] cached checksums (even unsupported by
|
|
the base backend) for a subtree is to \f[B]re-download\f[R] all files in
|
|
the subtree.
|
|
For example, use \f[C]hashsum --download\f[R] using \f[B]any\f[R]
|
|
supported hashsum on the command line (we just care to re-read):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone hashsum MD5 --download Hasher:path/to/subtree > /dev/null
|
|
|
|
rclone backend dump Hasher:path/to/subtree
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can print or drop hashsum cache using custom backend commands:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend dump Hasher:dir/subdir
|
|
|
|
rclone backend drop Hasher:
|
|
\f[R]
|
|
.fi
|
|
.SS Pre-Seed from a SUM File
|
|
.PP
|
|
Hasher supports two backend commands: generic SUM file \f[C]import\f[R]
|
|
and faster but less consistent \f[C]stickyimport\f[R].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend import Hasher:dir/subdir SHA1 /path/to/SHA1SUM [--checkers 4]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Instead of SHA1 it can be any hash supported by the remote.
|
|
The last argument can point to either a local or an
|
|
\f[C]other-remote:path\f[R] text file in SUM format.
|
|
The command will parse the SUM file, then walk down the path given by
|
|
the first argument, snapshot current fingerprints and fill in the cache
|
|
entries correspondingly.
|
|
- Paths in the SUM file are treated as relative to
|
|
\f[C]hasher:dir/subdir\f[R].
|
|
- The command will \f[B]not\f[R] check that supplied values are correct.
|
|
You \f[B]must know\f[R] what you are doing.
|
|
- This is a one-time action.
|
|
The SUM file will not get \[dq]attached\[dq] to the remote.
|
|
Cache entries can still be overwritten later, should the object\[aq]s
|
|
fingerprint change.
|
|
- The tree walk can take long depending on the tree size.
|
|
You can increase \f[C]--checkers\f[R] to make it faster.
|
|
Or use \f[C]stickyimport\f[R] if you don\[aq]t care about fingerprints
|
|
and consistency.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend stickyimport hasher:path/to/data sha1 remote:/path/to/sum.sha1
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[C]stickyimport\f[R] is similar to \f[C]import\f[R] but works much
|
|
faster because it does not need to stat existing files and skips initial
|
|
tree walk.
|
|
Instead of binding cache entries to file fingerprints it creates
|
|
\f[I]sticky\f[R] entries bound to the file name alone ignoring size,
|
|
modification time etc.
|
|
Such hash entries can be replaced only by \f[C]purge\f[R],
|
|
\f[C]delete\f[R], \f[C]backend drop\f[R] or by full re-read/re-write of
|
|
the files.
|
|
.SS Configuration reference
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to hasher (Better checksums for
|
|
other remotes).
|
|
.SS --hasher-remote
|
|
.PP
|
|
Remote to cache checksums for (e.g.
|
|
myRemote:path).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: remote
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HASHER_REMOTE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --hasher-hashes
|
|
.PP
|
|
Comma separated list of supported checksum types.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: hashes
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HASHER_HASHES
|
|
.IP \[bu] 2
|
|
Type: CommaSepList
|
|
.IP \[bu] 2
|
|
Default: md5,sha1
|
|
.SS --hasher-max-age
|
|
.PP
|
|
Maximum time to keep checksums in cache (0 = no cache, off = cache
|
|
forever).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: max_age
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HASHER_MAX_AGE
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: off
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to hasher (Better checksums for
|
|
other remotes).
|
|
.SS --hasher-auto-size
|
|
.PP
|
|
Auto-update checksum for files smaller than this size (disabled by
|
|
default).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auto_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HASHER_AUTO_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 0
|
|
.SS Backend commands
|
|
.PP
|
|
Here are the commands specific to the hasher backend.
|
|
.PP
|
|
Run them with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend COMMAND remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The help below will explain what arguments each command takes.
|
|
.PP
|
|
See the \[dq]rclone backend\[dq]
|
|
command (https://rclone.org/commands/rclone_backend/) for more info on
|
|
how to pass options and arguments.
|
|
.PP
|
|
These can be run on a running backend using the rc command
|
|
backend/command (https://rclone.org/rc/#backend-command).
|
|
.SS drop
|
|
.PP
|
|
Drop cache
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend drop remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Completely drop checksum cache.
|
|
Usage Example: rclone backend drop hasher:
|
|
.SS dump
|
|
.PP
|
|
Dump the database
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend dump remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Dump cache records covered by the current remote
|
|
.SS fulldump
|
|
.PP
|
|
Full dump of the database
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend fulldump remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Dump all cache records in the database
|
|
.SS import
|
|
.PP
|
|
Import a SUM file
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend import remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Amend hash cache from a SUM file and bind checksums to files by
|
|
size/time.
|
|
Usage Example: rclone backend import hasher:subdir md5 /path/to/sum.md5
|
|
.SS stickyimport
|
|
.PP
|
|
Perform fast import of a SUM file
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend stickyimport remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Fill hash cache from a SUM file without verifying file fingerprints.
|
|
Usage Example: rclone backend stickyimport hasher:subdir md5
|
|
remote:path/to/sum.md5
|
|
.SS Implementation details (advanced)
|
|
.PP
|
|
This section explains how various rclone operations work on a hasher
|
|
remote.
|
|
.PP
|
|
\f[B]Disclaimer. This section describes current implementation which can
|
|
change in future rclone versions!.\f[R]
|
|
.SS Hashsum command
|
|
.PP
|
|
The \f[C]rclone hashsum\f[R] (or \f[C]md5sum\f[R] or \f[C]sha1sum\f[R])
|
|
command will:
|
|
.IP "1." 3
|
|
if requested hash is supported by lower level, just pass it.
|
|
.IP "2." 3
|
|
if object size is below \f[C]auto_size\f[R] then download object and
|
|
calculate \f[I]requested\f[R] hashes on the fly.
|
|
.IP "3." 3
|
|
if unsupported and the size is big enough, build object
|
|
\f[C]fingerprint\f[R] (including size, modtime if supported, first-found
|
|
\f[I]other\f[R] hash if any).
|
|
.IP "4." 3
|
|
if the strict match is found in cache for the requested remote, return
|
|
the stored hash.
|
|
.IP "5." 3
|
|
if remote found but fingerprint mismatched, then purge the entry and
|
|
proceed to step 6.
|
|
.IP "6." 3
|
|
if remote not found or had no requested hash type or after step 5:
|
|
download object, calculate all \f[I]supported\f[R] hashes on the fly and
|
|
store in cache; return requested hash.
|
|
.SS Other operations
|
|
.IP \[bu] 2
|
|
whenever a file is uploaded or downloaded \f[B]in full\f[R], capture the
|
|
stream to calculate all supported hashes on the fly and update database
|
|
.IP \[bu] 2
|
|
server-side \f[C]move\f[R] will update keys of existing cache entries
|
|
.IP \[bu] 2
|
|
\f[C]deletefile\f[R] will remove a single cache entry
|
|
.IP \[bu] 2
|
|
\f[C]purge\f[R] will remove all cache entries under the purged path
|
|
.PP
|
|
Note that setting \f[C]max_age = 0\f[R] will disable checksum caching
|
|
completely.
|
|
.PP
|
|
If you set \f[C]max_age = off\f[R], checksums in cache will never age,
|
|
unless you fully rewrite or delete the file.
|
|
.SS Cache storage
|
|
.PP
|
|
Cached checksums are stored as \f[C]bolt\f[R] database files under
|
|
rclone cache directory, usually \f[C]\[ti]/.cache/rclone/kv/\f[R].
|
|
Databases are maintained one per \f[I]base\f[R] backend, named like
|
|
\f[C]BaseRemote\[ti]hasher.bolt\f[R].
|
|
Checksums for multiple \f[C]alias\f[R]-es into a single base backend
|
|
will be stored in the single database.
|
|
All local paths are treated as aliases into the \f[C]local\f[R] backend
|
|
(unless crypted or chunked) and stored in
|
|
\f[C]\[ti]/.cache/rclone/kv/local\[ti]hasher.bolt\f[R].
|
|
Databases can be shared between multiple rclone processes.
|
|
.SH HDFS
|
|
.PP
|
|
HDFS (https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html)
|
|
is a distributed file-system, part of the Apache
|
|
Hadoop (https://hadoop.apache.org/) framework.
|
|
.PP
|
|
Paths are specified as \f[C]remote:\f[R] or
|
|
\f[C]remote:path/to/dir\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[skip]
|
|
XX / Hadoop distributed file system
|
|
\[rs] \[dq]hdfs\[dq]
|
|
[skip]
|
|
Storage> hdfs
|
|
** See help for hdfs backend at: https://rclone.org/hdfs/ **
|
|
|
|
hadoop name node and port
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Connect to host namenode at port 8020
|
|
\[rs] \[dq]namenode:8020\[dq]
|
|
namenode> namenode.hadoop:8020
|
|
hadoop user name
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Connect to hdfs as root
|
|
\[rs] \[dq]root\[dq]
|
|
username> root
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
type = hdfs
|
|
namenode = namenode.hadoop:8020
|
|
username = root
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
Current remotes:
|
|
|
|
Name Type
|
|
==== ====
|
|
hadoop hdfs
|
|
|
|
e) Edit existing remote
|
|
n) New remote
|
|
d) Delete remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/r/c/s/q> q
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[R] and can now be used like this
|
|
.PP
|
|
See all the top level directories
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync the remote \f[C]directory\f[R] to \f[C]/home/local/directory\f[R],
|
|
deleting any excess files.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i remote:directory /home/local/directory
|
|
\f[R]
|
|
.fi
|
|
.SS Setting up your own HDFS instance for testing
|
|
.PP
|
|
You may start with a manual
|
|
setup (https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html)
|
|
or use the docker image from the tests:
|
|
.PP
|
|
If you want to build the docker image
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
git clone https://github.com/rclone/rclone.git
|
|
cd rclone/fstest/testserver/images/test-hdfs
|
|
docker build --rm -t rclone/test-hdfs .
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or you can just use the latest one pushed
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
docker run --rm --name \[dq]rclone-hdfs\[dq] -p 127.0.0.1:9866:9866 -p 127.0.0.1:8020:8020 --hostname \[dq]rclone-hdfs\[dq] rclone/test-hdfs
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]NB\f[R] it need few seconds to startup.
|
|
.PP
|
|
For this docker image the remote needs to be configured like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[remote]
|
|
type = hdfs
|
|
namenode = 127.0.0.1:8020
|
|
username = root
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can stop this image with \f[C]docker kill rclone-hdfs\f[R]
|
|
(\f[B]NB\f[R] it does not use volumes, so all data uploaded will be
|
|
lost.)
|
|
.SS Modified time
|
|
.PP
|
|
Time accurate to 1 second is stored.
|
|
.SS Checksum
|
|
.PP
|
|
No checksums are implemented.
|
|
.SS Usage information
|
|
.PP
|
|
You can use the \f[C]rclone about remote:\f[R] command which will
|
|
display filesystem size and current usage.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
:
|
|
T}@T{
|
|
0x3A
|
|
T}@T{
|
|
\[uFF1A]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8).
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to hdfs (Hadoop distributed file
|
|
system).
|
|
.SS --hdfs-namenode
|
|
.PP
|
|
Hadoop name node and port.
|
|
.PP
|
|
E.g.
|
|
\[dq]namenode:8020\[dq] to connect to host namenode at port 8020.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: namenode
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HDFS_NAMENODE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --hdfs-username
|
|
.PP
|
|
Hadoop user name.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: username
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HDFS_USERNAME
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]root\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Connect to hdfs as root.
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to hdfs (Hadoop distributed file
|
|
system).
|
|
.SS --hdfs-service-principal-name
|
|
.PP
|
|
Kerberos service principal name for the namenode.
|
|
.PP
|
|
Enables KERBEROS authentication.
|
|
Specifies the Service Principal Name (SERVICE/FQDN) for the namenode.
|
|
E.g.
|
|
\[dq]hdfs/namenode.hadoop.docker\[dq] for namenode running as service
|
|
\[aq]hdfs\[aq] with FQDN \[aq]namenode.hadoop.docker\[aq].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: service_principal_name
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HDFS_SERVICE_PRINCIPAL_NAME
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --hdfs-data-transfer-protection
|
|
.PP
|
|
Kerberos data transfer protection: authentication|integrity|privacy.
|
|
.PP
|
|
Specifies whether or not authentication, data signature integrity
|
|
checks, and wire encryption is required when communicating the the
|
|
datanodes.
|
|
Possible values are \[aq]authentication\[aq], \[aq]integrity\[aq] and
|
|
\[aq]privacy\[aq].
|
|
Used only with KERBEROS enabled.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: data_transfer_protection
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HDFS_DATA_TRANSFER_PROTECTION
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]privacy\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ensure authentication, integrity and encryption enabled.
|
|
.RE
|
|
.RE
|
|
.SS --hdfs-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HDFS_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,Colon,Del,Ctl,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.IP \[bu] 2
|
|
No server-side \f[C]Move\f[R] or \f[C]DirMove\f[R].
|
|
.IP \[bu] 2
|
|
Checksums not implemented.
|
|
.SH HTTP
|
|
.PP
|
|
The HTTP remote is a read only remote for reading files of a webserver.
|
|
The webserver should provide file listings which rclone will read and
|
|
turn into a remote.
|
|
This has been tested with common webservers such as Apache/Nginx/Caddy
|
|
and will likely work with file listings from most web servers.
|
|
(If it doesn\[aq]t then please file an issue, or send a pull request!)
|
|
.PP
|
|
Paths are specified as \f[C]remote:\f[R] or \f[C]remote:path\f[R].
|
|
.PP
|
|
The \f[C]remote:\f[R] represents the configured url, and any path
|
|
following it will be resolved relative to this url, according to the URL
|
|
standard.
|
|
This means with remote url \f[C]https://beta.rclone.org/branch\f[R] and
|
|
path \f[C]fix\f[R], the resolved URL will be
|
|
\f[C]https://beta.rclone.org/branch/fix\f[R], while with path
|
|
\f[C]/fix\f[R] the resolved URL will be
|
|
\f[C]https://beta.rclone.org/fix\f[R] as the absolute path is resolved
|
|
from the root of the domain.
|
|
.PP
|
|
If the path following the \f[C]remote:\f[R] ends with \f[C]/\f[R] it
|
|
will be assumed to point to a directory.
|
|
If the path does not end with \f[C]/\f[R], then a HEAD request is sent
|
|
and the response used to decide if it it is treated as a file or a
|
|
directory (run with \f[C]-vv\f[R] to see details).
|
|
When --http-no-head is specified, a path without ending \f[C]/\f[R] is
|
|
always assumed to be a file.
|
|
If rclone incorrectly assumes the path is a file, the solution is to
|
|
specify the path with ending \f[C]/\f[R].
|
|
When you know the path is a directory, ending it with \f[C]/\f[R] is
|
|
always better as it avoids the initial HEAD request.
|
|
.PP
|
|
To just download a single file it is easier to use
|
|
copyurl (https://rclone.org/commands/rclone_copyurl/).
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / http Connection
|
|
\[rs] \[dq]http\[dq]
|
|
[snip]
|
|
Storage> http
|
|
URL of http host to connect to
|
|
Choose a number from below, or type in your own value
|
|
1 / Connect to example.com
|
|
\[rs] \[dq]https://example.com\[dq]
|
|
url> https://beta.rclone.org
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
url = https://beta.rclone.org
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
Current remotes:
|
|
|
|
Name Type
|
|
==== ====
|
|
remote http
|
|
|
|
e) Edit existing remote
|
|
n) New remote
|
|
d) Delete remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/r/c/s/q> q
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[R] and can now be used like this
|
|
.PP
|
|
See all the top level directories
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync the remote \f[C]directory\f[R] to \f[C]/home/local/directory\f[R],
|
|
deleting any excess files.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i remote:directory /home/local/directory
|
|
\f[R]
|
|
.fi
|
|
.SS Read only
|
|
.PP
|
|
This remote is read only - you can\[aq]t upload files to an HTTP server.
|
|
.SS Modified time
|
|
.PP
|
|
Most HTTP servers store time accurate to 1 second.
|
|
.SS Checksum
|
|
.PP
|
|
No checksums are stored.
|
|
.SS Usage without a config file
|
|
.PP
|
|
Since the http remote only has one config parameter it is easy to use
|
|
without a config file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd --http-url https://beta.rclone.org :http:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
or:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd :http,url=\[aq]https://beta.rclone.org\[aq]:
|
|
\f[R]
|
|
.fi
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to http (http Connection).
|
|
.SS --http-url
|
|
.PP
|
|
URL of http host to connect to.
|
|
.PP
|
|
E.g.
|
|
\[dq]https://example.com\[dq], or
|
|
\[dq]https://user:pass\[at]example.com\[dq] to use a username and
|
|
password.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HTTP_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to http (http Connection).
|
|
.SS --http-headers
|
|
.PP
|
|
Set HTTP headers for all transactions.
|
|
.PP
|
|
Use this to set additional HTTP headers for all transactions.
|
|
.PP
|
|
The input format is comma separated list of key,value pairs.
|
|
Standard CSV encoding (https://godoc.org/encoding/csv) may be used.
|
|
.PP
|
|
For example, to set a Cookie use \[aq]Cookie,name=value\[aq], or
|
|
\[aq]\[dq]Cookie\[dq],\[dq]name=value\[dq]\[aq].
|
|
.PP
|
|
You can set multiple headers, e.g.
|
|
\[aq]\[dq]Cookie\[dq],\[dq]name=value\[dq],\[dq]Authorization\[dq],\[dq]xxx\[dq]\[aq].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: headers
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HTTP_HEADERS
|
|
.IP \[bu] 2
|
|
Type: CommaSepList
|
|
.IP \[bu] 2
|
|
Default:
|
|
.SS --http-no-slash
|
|
.PP
|
|
Set this if the site doesn\[aq]t end directories with /.
|
|
.PP
|
|
Use this if your target website does not use / on the end of
|
|
directories.
|
|
.PP
|
|
A / on the end of a path is how rclone normally tells the difference
|
|
between files and directories.
|
|
If this flag is set, then rclone will treat all files with Content-Type:
|
|
text/html as directories and read URLs from them rather than downloading
|
|
them.
|
|
.PP
|
|
Note that this may cause rclone to confuse genuine HTML files with
|
|
directories.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_slash
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HTTP_NO_SLASH
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --http-no-head
|
|
.PP
|
|
Don\[aq]t use HEAD requests.
|
|
.PP
|
|
HEAD requests are mainly used to find file sizes in dir listing.
|
|
If your site is being very slow to load then you can try this option.
|
|
Normally rclone does a HEAD request for each potential file in a
|
|
directory listing to:
|
|
.IP \[bu] 2
|
|
find its size
|
|
.IP \[bu] 2
|
|
check it really exists
|
|
.IP \[bu] 2
|
|
check to see if it is a directory
|
|
.PP
|
|
If you set this option, rclone will not do the HEAD request.
|
|
This will mean that directory listings are much quicker, but rclone
|
|
won\[aq]t have the times or sizes of any files, and some files that
|
|
don\[aq]t exist may be in the listing.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_head
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HTTP_NO_HEAD
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS Limitations
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the HTTP backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SH Hubic
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths are specified as \f[C]remote:container\f[R] (or \f[C]remote:\f[R]
|
|
for the \f[C]lsd\f[R] command.) You may put subdirectories in too, e.g.
|
|
\f[C]remote:container/path/to/dir\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for Hubic involves getting a token from Hubic which
|
|
you need to do in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n) New remote
|
|
s) Set configuration password
|
|
n/s> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Hubic
|
|
\[rs] \[dq]hubic\[dq]
|
|
[snip]
|
|
Storage> hubic
|
|
Hubic Client Id - leave blank normally.
|
|
client_id>
|
|
Hubic Client Secret - leave blank normally.
|
|
client_secret>
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
--------------------
|
|
[remote]
|
|
client_id =
|
|
client_secret =
|
|
token = {\[dq]access_token\[dq]:\[dq]XXXXXX\[dq]}
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Hubic.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List containers in the top level of your Hubic
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your Hubic
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Hubic directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you want the directory to be visible in the official \f[I]Hubic
|
|
browser\f[R], you need to copy your files to the \f[C]default\f[R]
|
|
directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:default/backup
|
|
\f[R]
|
|
.fi
|
|
.SS --fast-list
|
|
.PP
|
|
This remote supports \f[C]--fast-list\f[R] which allows you to use fewer
|
|
transactions in exchange for more memory.
|
|
See the rclone docs (https://rclone.org/docs/#fast-list) for more
|
|
details.
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X-Object-Meta-Mtime\f[R] as floating point since the epoch accurate
|
|
to 1 ns.
|
|
.PP
|
|
This is a de facto standard (used in the official python-swiftclient
|
|
amongst others) for storing the modification time for an object.
|
|
.PP
|
|
Note that Hubic wraps the Swift backend, so most of the properties of
|
|
are the same.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to hubic (Hubic).
|
|
.SS --hubic-client-id
|
|
.PP
|
|
OAuth Client Id.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HUBIC_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --hubic-client-secret
|
|
.PP
|
|
OAuth Client Secret.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HUBIC_CLIENT_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to hubic (Hubic).
|
|
.SS --hubic-token
|
|
.PP
|
|
OAuth Access Token as a JSON blob.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HUBIC_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --hubic-auth-url
|
|
.PP
|
|
Auth server URL.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HUBIC_AUTH_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --hubic-token-url
|
|
.PP
|
|
Token server url.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HUBIC_TOKEN_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --hubic-chunk-size
|
|
.PP
|
|
Above this size files will be chunked into a _segments container.
|
|
.PP
|
|
Above this size files will be chunked into a _segments container.
|
|
The default for this is 5 GiB which is its maximum value.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HUBIC_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 5Gi
|
|
.SS --hubic-no-chunk
|
|
.PP
|
|
Don\[aq]t chunk files during streaming upload.
|
|
.PP
|
|
When doing streaming uploads (e.g.
|
|
using rcat or mount) setting this flag will cause the swift backend to
|
|
not upload chunked files.
|
|
.PP
|
|
This will limit the maximum upload size to 5 GiB.
|
|
However non chunked files are easier to deal with and have an MD5SUM.
|
|
.PP
|
|
Rclone will still chunk files bigger than chunk_size when doing normal
|
|
copy operations.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_chunk
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HUBIC_NO_CHUNK
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --hubic-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_HUBIC_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,InvalidUtf8
|
|
.SS Limitations
|
|
.PP
|
|
This uses the normal OpenStack Swift mechanism to refresh the Swift API
|
|
credentials and ignores the expires field returned by the Hubic API.
|
|
.PP
|
|
The Swift API doesn\[aq]t return a correct MD5SUM for segmented files
|
|
(Dynamic or Static Large Objects) so rclone won\[aq]t check or use the
|
|
MD5SUM for these.
|
|
.SH Jottacloud
|
|
.PP
|
|
Jottacloud is a cloud storage service provider from a Norwegian company,
|
|
using its own datacenters in Norway.
|
|
In addition to the official service at
|
|
jottacloud.com (https://www.jottacloud.com/), it also provides
|
|
white-label solutions to different companies, such as: * Telia * Telia
|
|
Cloud (cloud.telia.se) * Telia Sky (sky.telia.no) * Tele2 * Tele2 Cloud
|
|
(mittcloud.tele2.se) * Elkj\[/o]p (with subsidiaries): * Elkj\[/o]p
|
|
Cloud (cloud.elkjop.no) * Elgiganten Sweden (cloud.elgiganten.se) *
|
|
Elgiganten Denmark (cloud.elgiganten.dk) * Giganti Cloud
|
|
(cloud.gigantti.fi) * ELKO Clouud (cloud.elko.is)
|
|
.PP
|
|
Most of the white-label versions are supported by this backend, although
|
|
may require different authentication setup - described below.
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Authentication types
|
|
.PP
|
|
Some of the whitelabel versions uses a different authentication method
|
|
than the official service, and you have to choose the correct one when
|
|
setting up the remote.
|
|
.SS Standard authentication
|
|
.PP
|
|
To configure Jottacloud you will need to generate a personal security
|
|
token in the Jottacloud web interface.
|
|
You will the option to do in your account security
|
|
settings (https://www.jottacloud.com/web/secure) (for whitelabel version
|
|
you need to find this page in its web interface).
|
|
Note that the web interface may refer to this token as a JottaCli token.
|
|
.SS Legacy authentication
|
|
.PP
|
|
If you are using one of the whitelabel versions (e.g.
|
|
from Elkj\[/o]p) you may not have the option to generate a CLI token.
|
|
In this case you\[aq]ll have to use the legacy authentication.
|
|
To do this select yes when the setup asks for legacy authentication and
|
|
enter your username and password.
|
|
The rest of the setup is identical to the default setup.
|
|
.SS Telia Cloud authentication
|
|
.PP
|
|
Similar to other whitelabel versions Telia Cloud doesn\[aq]t offer the
|
|
option of creating a CLI token, and additionally uses a separate
|
|
authentication flow where the username is generated internally.
|
|
To setup rclone to use Telia Cloud, choose Telia Cloud authentication in
|
|
the setup.
|
|
The rest of the setup is identical to the default setup.
|
|
.SS Tele2 Cloud authentication
|
|
.PP
|
|
As Tele2-Com Hem merger was completed this authentication can be used
|
|
for former Com Hem Cloud and Tele2 Cloud customers as no support for
|
|
creating a CLI token exists, and additionally uses a separate
|
|
authentication flow where the username is generated internally.
|
|
To setup rclone to use Tele2 Cloud, choose Tele2 Cloud authentication in
|
|
the setup.
|
|
The rest of the setup is identical to the default setup.
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R] with
|
|
the default setup.
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Jottacloud
|
|
\[rs] \[dq]jottacloud\[dq]
|
|
[snip]
|
|
Storage> jottacloud
|
|
** See help for jottacloud backend at: https://rclone.org/jottacloud/ **
|
|
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No
|
|
y/n> n
|
|
Remote config
|
|
Use legacy authentication?.
|
|
This is only required for certain whitelabel versions of Jottacloud and not recommended for normal users.
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
|
|
Generate a personal login token here: https://www.jottacloud.com/web/secure
|
|
Login Token> <your token here>
|
|
|
|
Do you want to use a non standard device/mountpoint e.g. for accessing files uploaded using the official Jottacloud client?
|
|
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
Please select the device to use. Normally this will be Jotta
|
|
Choose a number from below, or type in an existing value
|
|
1 > DESKTOP-3H31129
|
|
2 > Jotta
|
|
Devices> 2
|
|
Please select the mountpoint to user. Normally this will be Archive
|
|
Choose a number from below, or type in an existing value
|
|
1 > Archive
|
|
2 > Links
|
|
3 > Sync
|
|
|
|
Mountpoints> 1
|
|
--------------------
|
|
[jotta]
|
|
type = jottacloud
|
|
token = {........}
|
|
device = Jotta
|
|
mountpoint = Archive
|
|
configVersion = 1
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your Jottacloud
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your Jottacloud
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Jottacloud directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Devices and Mountpoints
|
|
.PP
|
|
The official Jottacloud client registers a device for each computer you
|
|
install it on, and then creates a mountpoint for each folder you select
|
|
for Backup.
|
|
The web interface uses a special device called Jotta for the Archive and
|
|
Sync mountpoints.
|
|
.PP
|
|
With rclone you\[aq]ll want to use the Jotta/Archive device/mountpoint
|
|
in most cases, however if you want to access files uploaded by any of
|
|
the official clients rclone provides the option to select other devices
|
|
and mountpoints during config.
|
|
Note that uploading files is currently not supported to other devices
|
|
than Jotta.
|
|
.PP
|
|
The built-in Jotta device may also contain several other mountpoints,
|
|
such as: Latest, Links, Shared and Trash.
|
|
These are special mountpoints with a different internal representation
|
|
than the \[dq]regular\[dq] mountpoints.
|
|
Rclone will only to a very limited degree support them.
|
|
Generally you should avoid these, unless you know what you are doing.
|
|
.SS --fast-list
|
|
.PP
|
|
This remote supports \f[C]--fast-list\f[R] which allows you to use fewer
|
|
transactions in exchange for more memory.
|
|
See the rclone docs (https://rclone.org/docs/#fast-list) for more
|
|
details.
|
|
.PP
|
|
Note that the implementation in Jottacloud always uses only a single API
|
|
request to get the entire list, so for large folders this could lead to
|
|
long wait time before the first results are shown.
|
|
.PP
|
|
Note also that with rclone version 1.58 and newer information about MIME
|
|
types (https://rclone.org/overview/#mime-type) are not available when
|
|
using \f[C]--fast-list\f[R].
|
|
.SS Modified time and hashes
|
|
.PP
|
|
Jottacloud allows modification times to be set on objects accurate to 1
|
|
second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
.PP
|
|
Jottacloud supports MD5 type hashes, so you can use the
|
|
\f[C]--checksum\f[R] flag.
|
|
.PP
|
|
Note that Jottacloud requires the MD5 hash before upload so if the
|
|
source does not have an MD5 checksum then the file will be cached
|
|
temporarily on disk (in location given by
|
|
--temp-dir (https://rclone.org/docs/#temp-dir-dir)) before it is
|
|
uploaded.
|
|
Small files will be cached in memory - see the
|
|
--jottacloud-md5-memory-limit flag.
|
|
When uploading from local disk the source checksum is always available,
|
|
so this does not apply.
|
|
Starting with rclone version 1.52 the same is true for crypted remotes
|
|
(in older versions the crypt backend would not calculate hashes for
|
|
uploads from local disk, so the Jottacloud backend had to do it as
|
|
described above).
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[dq]
|
|
T}@T{
|
|
0x22
|
|
T}@T{
|
|
\[uFF02]
|
|
T}
|
|
T{
|
|
*
|
|
T}@T{
|
|
0x2A
|
|
T}@T{
|
|
\[uFF0A]
|
|
T}
|
|
T{
|
|
:
|
|
T}@T{
|
|
0x3A
|
|
T}@T{
|
|
\[uFF1A]
|
|
T}
|
|
T{
|
|
<
|
|
T}@T{
|
|
0x3C
|
|
T}@T{
|
|
\[uFF1C]
|
|
T}
|
|
T{
|
|
>
|
|
T}@T{
|
|
0x3E
|
|
T}@T{
|
|
\[uFF1E]
|
|
T}
|
|
T{
|
|
?
|
|
T}@T{
|
|
0x3F
|
|
T}@T{
|
|
\[uFF1F]
|
|
T}
|
|
T{
|
|
|
|
|
T}@T{
|
|
0x7C
|
|
T}@T{
|
|
\[uFF5C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in XML strings.
|
|
.SS Deleting files
|
|
.PP
|
|
By default, rclone will send all files to the trash when deleting files.
|
|
They will be permanently deleted automatically after 30 days.
|
|
You may bypass the trash and permanently delete files immediately by
|
|
using the --jottacloud-hard-delete flag, or set the equivalent
|
|
environment variable.
|
|
Emptying the trash is supported by the
|
|
cleanup (https://rclone.org/commands/rclone_cleanup/) command.
|
|
.SS Versions
|
|
.PP
|
|
Jottacloud supports file versioning.
|
|
When rclone uploads a new version of a file it creates a new version of
|
|
it.
|
|
Currently rclone only supports retrieving the current version but older
|
|
versions can be accessed via the Jottacloud Website.
|
|
.PP
|
|
Versioning can be disabled by \f[C]--jottacloud-no-versions\f[R] option.
|
|
This is achieved by deleting the remote file prior to uploading a new
|
|
version.
|
|
If the upload the fails no version of the file will be available in the
|
|
remote.
|
|
.SS Quota information
|
|
.PP
|
|
To view your current quota you can use the
|
|
\f[C]rclone about remote:\f[R] command which will display your usage
|
|
limit (unless it is unlimited) and the current usage.
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to jottacloud (Jottacloud).
|
|
.SS --jottacloud-md5-memory-limit
|
|
.PP
|
|
Files bigger than this will be cached on disk to calculate the MD5 if
|
|
required.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: md5_memory_limit
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_JOTTACLOUD_MD5_MEMORY_LIMIT
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 10Mi
|
|
.SS --jottacloud-trashed-only
|
|
.PP
|
|
Only show files that are in the trash.
|
|
.PP
|
|
This will show trashed files in their original directory structure.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: trashed_only
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_JOTTACLOUD_TRASHED_ONLY
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --jottacloud-hard-delete
|
|
.PP
|
|
Delete files permanently rather than putting them into the trash.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: hard_delete
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_JOTTACLOUD_HARD_DELETE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --jottacloud-upload-resume-limit
|
|
.PP
|
|
Files bigger than this can be resumed if the upload fail\[aq]s.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_resume_limit
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_JOTTACLOUD_UPLOAD_RESUME_LIMIT
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 10Mi
|
|
.SS --jottacloud-no-versions
|
|
.PP
|
|
Avoid server side versioning by deleting files and recreating files
|
|
instead of overwriting them.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_versions
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_JOTTACLOUD_NO_VERSIONS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --jottacloud-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_JOTTACLOUD_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default:
|
|
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
Note that Jottacloud is case insensitive so you can\[aq]t have a file
|
|
called \[dq]Hello.doc\[dq] and one called \[dq]hello.doc\[dq].
|
|
.PP
|
|
There are quite a few characters that can\[aq]t be in Jottacloud file
|
|
names.
|
|
Rclone will map these names to and from an identical looking unicode
|
|
equivalent.
|
|
For example if a file has a ?
|
|
in it will be mapped to \[uFF1F] instead.
|
|
.PP
|
|
Jottacloud only supports filenames up to 255 characters in length.
|
|
.SS Troubleshooting
|
|
.PP
|
|
Jottacloud exhibits some inconsistent behaviours regarding deleted files
|
|
and folders which may cause Copy, Move and DirMove operations to
|
|
previously deleted paths to fail.
|
|
Emptying the trash should help in such cases.
|
|
.SH Koofr
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for Koofr involves creating an application password
|
|
for rclone.
|
|
You can do that by opening the Koofr web
|
|
application (https://app.koofr.net/app/admin/preferences/password),
|
|
giving the password a nice name like \f[C]rclone\f[R] and clicking on
|
|
generate.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]koofr\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> koofr
|
|
Option Storage.
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value.
|
|
[snip]
|
|
22 / Koofr, Digi Storage and other Koofr-compatible storage providers
|
|
\[rs] (koofr)
|
|
[snip]
|
|
Storage> koofr
|
|
Option provider.
|
|
Choose your storage provider.
|
|
Choose a number from below, or type in your own value.
|
|
Press Enter to leave empty.
|
|
1 / Koofr, https://app.koofr.net/
|
|
\[rs] (koofr)
|
|
2 / Digi Storage, https://storage.rcs-rds.ro/
|
|
\[rs] (digistorage)
|
|
3 / Any other Koofr API compatible storage service
|
|
\[rs] (other)
|
|
provider> 1
|
|
Option user.
|
|
Your user name.
|
|
Enter a value.
|
|
user> USERNAME
|
|
Option password.
|
|
Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password).
|
|
Choose an alternative below.
|
|
y) Yes, type in my own password
|
|
g) Generate random password
|
|
y/g> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
Edit advanced config?
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
Remote config
|
|
--------------------
|
|
[koofr]
|
|
type = koofr
|
|
provider = koofr
|
|
user = USERNAME
|
|
password = *** ENCRYPTED ***
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can choose to edit advanced config in order to enter your own
|
|
service URL if you use an on-premise or white label Koofr instance, or
|
|
choose an alternative mount instead of your primary storage.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your Koofr
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd koofr:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your Koofr
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls koofr:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Koofr directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source koofr:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in XML strings.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to koofr (Koofr, Digi Storage and
|
|
other Koofr-compatible storage providers).
|
|
.SS --koofr-provider
|
|
.PP
|
|
Choose your storage provider.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: provider
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_KOOFR_PROVIDER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]koofr\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Koofr, https://app.koofr.net/
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]digistorage\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Digi Storage, https://storage.rcs-rds.ro/
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]other\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Any other Koofr API compatible storage service
|
|
.RE
|
|
.RE
|
|
.SS --koofr-endpoint
|
|
.PP
|
|
The Koofr API endpoint to use.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_KOOFR_ENDPOINT
|
|
.IP \[bu] 2
|
|
Provider: other
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --koofr-user
|
|
.PP
|
|
Your user name.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_KOOFR_USER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --koofr-password
|
|
.PP
|
|
Your password for rclone (generate one at
|
|
https://app.koofr.net/app/admin/preferences/password).
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_KOOFR_PASSWORD
|
|
.IP \[bu] 2
|
|
Provider: koofr
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --koofr-password
|
|
.PP
|
|
Your password for rclone (generate one at
|
|
https://storage.rcs-rds.ro/app/admin/preferences/password).
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_KOOFR_PASSWORD
|
|
.IP \[bu] 2
|
|
Provider: digistorage
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --koofr-password
|
|
.PP
|
|
Your password for rclone (generate one at your service\[aq]s settings
|
|
page).
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_KOOFR_PASSWORD
|
|
.IP \[bu] 2
|
|
Provider: other
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to koofr (Koofr, Digi Storage and
|
|
other Koofr-compatible storage providers).
|
|
.SS --koofr-mountid
|
|
.PP
|
|
Mount ID of the mount to use.
|
|
.PP
|
|
If omitted, the primary mount is used.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: mountid
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_KOOFR_MOUNTID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --koofr-setmtime
|
|
.PP
|
|
Does the backend support setting modification time.
|
|
.PP
|
|
Set this to false if you use a mount ID that points to a Dropbox or
|
|
Amazon Drive backend.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: setmtime
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_KOOFR_SETMTIME
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: true
|
|
.SS --koofr-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_KOOFR_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
Note that Koofr is case insensitive so you can\[aq]t have a file called
|
|
\[dq]Hello.doc\[dq] and one called \[dq]hello.doc\[dq].
|
|
.SS Providers
|
|
.SS Koofr
|
|
.PP
|
|
This is the original Koofr (https://koofr.eu) storage provider used as
|
|
main example and described in the configuration section above.
|
|
.SS Digi Storage
|
|
.PP
|
|
Digi Storage (https://www.digi.ro/servicii/online/digi-storage) is a
|
|
cloud storage service run by Digi.ro (https://www.digi.ro/) that
|
|
provides a Koofr API.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]ds\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> ds
|
|
Option Storage.
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value.
|
|
[snip]
|
|
22 / Koofr, Digi Storage and other Koofr-compatible storage providers
|
|
\[rs] (koofr)
|
|
[snip]
|
|
Storage> koofr
|
|
Option provider.
|
|
Choose your storage provider.
|
|
Choose a number from below, or type in your own value.
|
|
Press Enter to leave empty.
|
|
1 / Koofr, https://app.koofr.net/
|
|
\[rs] (koofr)
|
|
2 / Digi Storage, https://storage.rcs-rds.ro/
|
|
\[rs] (digistorage)
|
|
3 / Any other Koofr API compatible storage service
|
|
\[rs] (other)
|
|
provider> 2
|
|
Option user.
|
|
Your user name.
|
|
Enter a value.
|
|
user> USERNAME
|
|
Option password.
|
|
Your password for rclone (generate one at https://storage.rcs-rds.ro/app/admin/preferences/password).
|
|
Choose an alternative below.
|
|
y) Yes, type in my own password
|
|
g) Generate random password
|
|
y/g> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
Edit advanced config?
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
--------------------
|
|
[ds]
|
|
type = koofr
|
|
provider = digistorage
|
|
user = USERNAME
|
|
password = *** ENCRYPTED ***
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.SS Other
|
|
.PP
|
|
You may also want to use another, public or private storage provider
|
|
that runs a Koofr API compatible service, by simply providing the base
|
|
URL to connect to.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]other\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> other
|
|
Option Storage.
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value.
|
|
[snip]
|
|
22 / Koofr, Digi Storage and other Koofr-compatible storage providers
|
|
\[rs] (koofr)
|
|
[snip]
|
|
Storage> koofr
|
|
Option provider.
|
|
Choose your storage provider.
|
|
Choose a number from below, or type in your own value.
|
|
Press Enter to leave empty.
|
|
1 / Koofr, https://app.koofr.net/
|
|
\[rs] (koofr)
|
|
2 / Digi Storage, https://storage.rcs-rds.ro/
|
|
\[rs] (digistorage)
|
|
3 / Any other Koofr API compatible storage service
|
|
\[rs] (other)
|
|
provider> 3
|
|
Option endpoint.
|
|
The Koofr API endpoint to use.
|
|
Enter a value.
|
|
endpoint> https://koofr.other.org
|
|
Option user.
|
|
Your user name.
|
|
Enter a value.
|
|
user> USERNAME
|
|
Option password.
|
|
Your password for rclone (generate one at your service\[aq]s settings page).
|
|
Choose an alternative below.
|
|
y) Yes, type in my own password
|
|
g) Generate random password
|
|
y/g> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
Edit advanced config?
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
--------------------
|
|
[other]
|
|
type = koofr
|
|
provider = other
|
|
endpoint = https://koofr.other.org
|
|
user = USERNAME
|
|
password = *** ENCRYPTED ***
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.SH Mail.ru Cloud
|
|
.PP
|
|
Mail.ru Cloud (https://cloud.mail.ru/) is a cloud storage provided by a
|
|
Russian internet company Mail.Ru Group (https://mail.ru).
|
|
The official desktop client is Disk-O: (https://disk-o.cloud/en),
|
|
available on Windows and Mac OS.
|
|
.PP
|
|
Currently it is recommended to disable 2FA on Mail.ru accounts intended
|
|
for rclone until it gets eventually implemented.
|
|
.SS Features highlights
|
|
.IP \[bu] 2
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R]
|
|
.IP \[bu] 2
|
|
Files have a \f[C]last modified time\f[R] property, directories
|
|
don\[aq]t
|
|
.IP \[bu] 2
|
|
Deleted files are by default moved to the trash
|
|
.IP \[bu] 2
|
|
Files and directories can be shared via public links
|
|
.IP \[bu] 2
|
|
Partial uploads or streaming are not supported, file size must be known
|
|
before upload
|
|
.IP \[bu] 2
|
|
Maximum file size is limited to 2G for a free account, unlimited for
|
|
paid accounts
|
|
.IP \[bu] 2
|
|
Storage keeps hash for all files and performs transparent deduplication,
|
|
the hash algorithm is a modified SHA1
|
|
.IP \[bu] 2
|
|
If a particular file is already present in storage, one can quickly
|
|
submit file hash instead of long file upload (this optimization is
|
|
supported by rclone)
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of making a mailru configuration.
|
|
First create a Mail.ru Cloud account and choose a tariff, then run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Mail.ru Cloud
|
|
\[rs] \[dq]mailru\[dq]
|
|
[snip]
|
|
Storage> mailru
|
|
User name (usually email)
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
user> username\[at]mail.ru
|
|
Password
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
y/g> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
Skip full upload if there is another file with same data hash.
|
|
This feature is called \[dq]speedup\[dq] or \[dq]put by hash\[dq]. It is especially efficient
|
|
in case of generally available files like popular books, video or audio clips
|
|
[snip]
|
|
Enter a boolean value (true or false). Press Enter for the default (\[dq]true\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Enable
|
|
\[rs] \[dq]true\[dq]
|
|
2 / Disable
|
|
\[rs] \[dq]false\[dq]
|
|
speedup_enable> 1
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No
|
|
y/n> n
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
type = mailru
|
|
user = username\[at]mail.ru
|
|
pass = *** ENCRYPTED ***
|
|
speedup_enable = true
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Configuration of this backend does not require a local web browser.
|
|
You can use the configured backend as shown below:
|
|
.PP
|
|
See top level directories
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Make a new directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote path, deleting any
|
|
excess files in the path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory remote:directory
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time
|
|
.PP
|
|
Files support a modification time attribute with up to 1 second
|
|
precision.
|
|
Directories do not have a modification time, which is shown as \[dq]Jan
|
|
1 1970\[dq].
|
|
.SS Hash checksums
|
|
.PP
|
|
Hash sums use a custom Mail.ru algorithm based on SHA1.
|
|
If file size is less than or equal to the SHA1 block size (20 bytes),
|
|
its hash is simply its data right-padded with zero bytes.
|
|
Hash sum of a larger file is computed as a SHA1 sum of the file data
|
|
bytes concatenated with a decimal representation of the data length.
|
|
.SS Emptying Trash
|
|
.PP
|
|
Removing a file or directory actually moves it to the trash, which is
|
|
not visible to rclone but can be seen in a web browser.
|
|
The trashed file still occupies part of total quota.
|
|
If you wish to empty your trash and free some quota, you can use the
|
|
\f[C]rclone cleanup remote:\f[R] command, which will permanently delete
|
|
all your trashed files.
|
|
This command does not take any path arguments.
|
|
.SS Quota information
|
|
.PP
|
|
To view your current quota you can use the
|
|
\f[C]rclone about remote:\f[R] command which will display your usage
|
|
limit (quota) and the current usage.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[dq]
|
|
T}@T{
|
|
0x22
|
|
T}@T{
|
|
\[uFF02]
|
|
T}
|
|
T{
|
|
*
|
|
T}@T{
|
|
0x2A
|
|
T}@T{
|
|
\[uFF0A]
|
|
T}
|
|
T{
|
|
:
|
|
T}@T{
|
|
0x3A
|
|
T}@T{
|
|
\[uFF1A]
|
|
T}
|
|
T{
|
|
<
|
|
T}@T{
|
|
0x3C
|
|
T}@T{
|
|
\[uFF1C]
|
|
T}
|
|
T{
|
|
>
|
|
T}@T{
|
|
0x3E
|
|
T}@T{
|
|
\[uFF1E]
|
|
T}
|
|
T{
|
|
?
|
|
T}@T{
|
|
0x3F
|
|
T}@T{
|
|
\[uFF1F]
|
|
T}
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
T{
|
|
|
|
|
T}@T{
|
|
0x7C
|
|
T}@T{
|
|
\[uFF5C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to mailru (Mail.ru Cloud).
|
|
.SS --mailru-user
|
|
.PP
|
|
User name (usually email).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MAILRU_USER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --mailru-pass
|
|
.PP
|
|
Password.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: pass
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MAILRU_PASS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --mailru-speedup-enable
|
|
.PP
|
|
Skip full upload if there is another file with same data hash.
|
|
.PP
|
|
This feature is called \[dq]speedup\[dq] or \[dq]put by hash\[dq].
|
|
It is especially efficient in case of generally available files like
|
|
popular books, video or audio clips, because files are searched by hash
|
|
in all accounts of all mailru users.
|
|
It is meaningless and ineffective if source file is unique or encrypted.
|
|
Please note that rclone may need local memory and disk space to
|
|
calculate content hash in advance and decide whether full upload is
|
|
required.
|
|
Also, if rclone does not know file size in advance (e.g.
|
|
in case of streaming or partial uploads), it will not even try this
|
|
optimization.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: speedup_enable
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MAILRU_SPEEDUP_ENABLE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: true
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]true\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enable
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]false\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Disable
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to mailru (Mail.ru Cloud).
|
|
.SS --mailru-speedup-file-patterns
|
|
.PP
|
|
Comma separated list of file name patterns eligible for speedup (put by
|
|
hash).
|
|
.PP
|
|
Patterns are case insensitive and can contain \[aq]*\[aq] or \[aq]?\[aq]
|
|
meta characters.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: speedup_file_patterns
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MAILRU_SPEEDUP_FILE_PATTERNS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default:
|
|
\[dq]\f[I].mkv,\f[R].avi,\f[I].mp4,\f[R].mp3,\f[I].zip,\f[R].gz,\f[I].rar,\f[R].pdf\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Empty list completely disables speedup (put by hash).
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]*\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
All files will be attempted for speedup.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]\f[I].mkv,\f[R].avi,\f[I].mp4,\f[R].mp3\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Only common audio/video files will be tried for put by hash.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]\f[I].zip,\f[R].gz,\f[I].rar,\f[R].pdf\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Only common archives or PDF books will be tried for speedup.
|
|
.RE
|
|
.RE
|
|
.SS --mailru-speedup-max-disk
|
|
.PP
|
|
This option allows you to disable speedup (put by hash) for large files.
|
|
.PP
|
|
Reason is that preliminary hashing can exhaust your RAM or disk space.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: speedup_max_disk
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MAILRU_SPEEDUP_MAX_DISK
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 3Gi
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]0\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Completely disable speedup (put by hash).
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]1G\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Files larger than 1Gb will be uploaded directly.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]3G\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Choose this option if you have less than 3Gb free on local disk.
|
|
.RE
|
|
.RE
|
|
.SS --mailru-speedup-max-memory
|
|
.PP
|
|
Files larger than the size given below will always be hashed on disk.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: speedup_max_memory
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MAILRU_SPEEDUP_MAX_MEMORY
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 32Mi
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]0\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Preliminary hashing will always be done in a temporary disk location.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]32M\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Do not dedicate more than 32Mb RAM for preliminary hashing.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]256M\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
You have at most 256Mb RAM free for hash calculations.
|
|
.RE
|
|
.RE
|
|
.SS --mailru-check-hash
|
|
.PP
|
|
What should copy do if file checksum is mismatched or invalid.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: check_hash
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MAILRU_CHECK_HASH
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: true
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]true\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fail with error.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]false\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ignore and continue.
|
|
.RE
|
|
.RE
|
|
.SS --mailru-user-agent
|
|
.PP
|
|
HTTP user agent used internally by client.
|
|
.PP
|
|
Defaults to \[dq]rclone/VERSION\[dq] or \[dq]--user-agent\[dq] provided
|
|
on command line.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user_agent
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MAILRU_USER_AGENT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --mailru-quirks
|
|
.PP
|
|
Comma separated list of internal maintenance flags.
|
|
.PP
|
|
This option must not be used by an ordinary user.
|
|
It is intended only to facilitate remote troubleshooting of backend
|
|
issues.
|
|
Strict meaning of flags is not documented and not guaranteed to persist
|
|
between releases.
|
|
Quirks will be removed when the backend grows stable.
|
|
Supported quirks: atomicmkdir binlist unknowndirs
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: quirks
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MAILRU_QUIRKS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --mailru-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MAILRU_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default:
|
|
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
File size limits depend on your account.
|
|
A single file size is limited by 2G for a free account and unlimited for
|
|
paid tariffs.
|
|
Please refer to the Mail.ru site for the total uploaded size limits.
|
|
.PP
|
|
Note that Mailru is case insensitive so you can\[aq]t have a file called
|
|
\[dq]Hello.doc\[dq] and one called \[dq]hello.doc\[dq].
|
|
.SH Mega
|
|
.PP
|
|
Mega (https://mega.nz/) is a cloud storage and file hosting service
|
|
known for its security feature where all files are encrypted locally
|
|
before they are uploaded.
|
|
This prevents anyone (including employees of Mega) from accessing the
|
|
files without knowledge of the key used for encryption.
|
|
.PP
|
|
This is an rclone backend for Mega which supports the file transfer
|
|
features of Mega using the same client side encryption.
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Mega
|
|
\[rs] \[dq]mega\[dq]
|
|
[snip]
|
|
Storage> mega
|
|
User name
|
|
user> you\[at]example.com
|
|
Password.
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
n) No leave this optional password blank
|
|
y/g/n> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
type = mega
|
|
user = you\[at]example.com
|
|
pass = *** ENCRYPTED ***
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]NOTE:\f[R] The encryption keys need to have been already generated
|
|
after a regular login via the browser, otherwise attempting to use the
|
|
credentials in \f[C]rclone\f[R] will fail.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your Mega
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your Mega
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Mega directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
Mega does not support modification times or hashes yet.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
NUL
|
|
T}@T{
|
|
0x00
|
|
T}@T{
|
|
\[u2400]
|
|
T}
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Duplicated files
|
|
.PP
|
|
Mega can have two files with exactly the same name and path (unlike a
|
|
normal file system).
|
|
.PP
|
|
Duplicated files cause problems with the syncing and you will see
|
|
messages in the log about duplicates.
|
|
.PP
|
|
Use \f[C]rclone dedupe\f[R] to fix duplicated files.
|
|
.SS Failure to log-in
|
|
.PP
|
|
Mega remotes seem to get blocked (reject logins) under \[dq]heavy
|
|
use\[dq].
|
|
We haven\[aq]t worked out the exact blocking rules but it seems to be
|
|
related to fast paced, successive rclone commands.
|
|
.PP
|
|
For example, executing this command 90 times in a row
|
|
\f[C]rclone link remote:file\f[R] will cause the remote to become
|
|
\[dq]blocked\[dq].
|
|
This is not an abnormal situation, for example if you wish to get the
|
|
public links of a directory with hundred of files...
|
|
After more or less a week, the remote will remote accept rclone logins
|
|
normally again.
|
|
.PP
|
|
You can mitigate this issue by mounting the remote it with
|
|
\f[C]rclone mount\f[R].
|
|
This will log-in when mounting and a log-out when unmounting only.
|
|
You can also run \f[C]rclone rcd\f[R] and then use \f[C]rclone rc\f[R]
|
|
to run the commands over the API to avoid logging in each time.
|
|
.PP
|
|
Rclone does not currently close mega sessions (you can see them in the
|
|
web interface), however closing the sessions does not solve the issue.
|
|
.PP
|
|
If you space rclone commands by 3 seconds it will avoid blocking the
|
|
remote.
|
|
We haven\[aq]t identified the exact blocking rules, so perhaps one could
|
|
execute the command 80 times without waiting and avoid blocking by
|
|
waiting 3 seconds, then continuing...
|
|
.PP
|
|
Note that this has been observed by trial and error and might not be set
|
|
in stone.
|
|
.PP
|
|
Other tools seem not to produce this blocking effect, as they use a
|
|
different working approach (state-based, using sessionIDs instead of
|
|
log-in) which isn\[aq]t compatible with the current stateless rclone
|
|
approach.
|
|
.PP
|
|
Note that once blocked, the use of other tools (such as megacmd) is not
|
|
a sure workaround: following megacmd login times have been observed in
|
|
succession for blocked remote: 7 minutes, 20 min, 30min, 30 min, 30min.
|
|
Web access looks unaffected though.
|
|
.PP
|
|
Investigation is continuing in relation to workarounds based on
|
|
timeouts, pacers, retrials and tpslimits - if you discover something
|
|
relevant, please post on the forum.
|
|
.PP
|
|
So, if rclone was working nicely and suddenly you are unable to log-in
|
|
and you are sure the user and the password are correct, likely you have
|
|
got the remote blocked for a while.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to mega (Mega).
|
|
.SS --mega-user
|
|
.PP
|
|
User name.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MEGA_USER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --mega-pass
|
|
.PP
|
|
Password.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: pass
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MEGA_PASS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to mega (Mega).
|
|
.SS --mega-debug
|
|
.PP
|
|
Output more debug from Mega.
|
|
.PP
|
|
If this flag is set (along with -vv) it will print further debugging
|
|
information from the mega backend.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: debug
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MEGA_DEBUG
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --mega-hard-delete
|
|
.PP
|
|
Delete files permanently rather than putting them into the trash.
|
|
.PP
|
|
Normally the mega backend will put all deletions into the trash rather
|
|
than permanently deleting them.
|
|
If you specify this then rclone will permanently delete objects instead.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: hard_delete
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MEGA_HARD_DELETE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --mega-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_MEGA_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
This backend uses the go-mega go
|
|
library (https://github.com/t3rm1n4l/go-mega) which is an opensource go
|
|
library implementing the Mega API.
|
|
There doesn\[aq]t appear to be any documentation for the mega protocol
|
|
beyond the mega C++ SDK (https://github.com/meganz/sdk) source code so
|
|
there are likely quite a few errors still remaining in this library.
|
|
.PP
|
|
Mega allows duplicate files which may confuse rclone.
|
|
.SH Memory
|
|
.PP
|
|
The memory backend is an in RAM backend.
|
|
It does not persist its data - use the local backend for that.
|
|
.PP
|
|
The memory backend behaves like a bucket-based remote (e.g.
|
|
like s3).
|
|
Because it has no parameters you can just use it with the
|
|
\f[C]:memory:\f[R] remote name.
|
|
.SS Configuration
|
|
.PP
|
|
You can configure it as a remote like this with \f[C]rclone config\f[R]
|
|
too if you want to:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Memory
|
|
\[rs] \[dq]memory\[dq]
|
|
[snip]
|
|
Storage> memory
|
|
** See help for memory backend at: https://rclone.org/memory/ **
|
|
|
|
Remote config
|
|
|
|
--------------------
|
|
[remote]
|
|
type = memory
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Because the memory backend isn\[aq]t persistent it is most useful for
|
|
testing or with an rclone server or rclone mount, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mount :memory: /mnt/tmp
|
|
rclone serve webdav :memory:
|
|
rclone serve sftp :memory:
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
The memory backend supports MD5 hashes and modification times accurate
|
|
to 1 nS.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
The memory backend replaces the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters).
|
|
.SS Akamai NetStorage
|
|
.PP
|
|
Paths are specified as \f[C]remote:\f[R] You may put subdirectories in
|
|
too, e.g.
|
|
\f[C]remote:/path/to/dir\f[R].
|
|
If you have a CP code you can use that as the folder after the domain
|
|
such as <domain>/<cpcode>/<internal directories within cpcode>.
|
|
.PP
|
|
For example, this is commonly configured with or without a CP code: *
|
|
\f[B]With a CP code\f[R].
|
|
\f[C][your-domain-prefix]-nsu.akamaihd.net/123456/subdirectory/\f[R] *
|
|
\f[B]Without a CP code\f[R].
|
|
\f[C][your-domain-prefix]-nsu.akamaihd.net\f[R]
|
|
.PP
|
|
See all buckets rclone lsd remote: The initial setup for Netstorage
|
|
involves getting an account and secret.
|
|
Use \f[C]rclone config\f[R] to walk you through the setup process.
|
|
.PP
|
|
Here\[aq]s an example of how to make a remote called \f[C]ns1\f[R].
|
|
.IP "1." 3
|
|
To begin the interactive configuration process, enter this command:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.IP "2." 3
|
|
Type \f[C]n\f[R] to create a new remote.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n) New remote
|
|
d) Delete remote
|
|
q) Quit config
|
|
e/n/d/q> n
|
|
\f[R]
|
|
.fi
|
|
.IP "3." 3
|
|
For this example, enter \f[C]ns1\f[R] when you reach the name> prompt.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
name> ns1
|
|
\f[R]
|
|
.fi
|
|
.IP "4." 3
|
|
Enter \f[C]netstorage\f[R] as the type of storage to configure.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
XX / NetStorage
|
|
\[rs] \[dq]netstorage\[dq]
|
|
Storage> netstorage
|
|
\f[R]
|
|
.fi
|
|
.IP "5." 3
|
|
Select between the HTTP or HTTPS protocol.
|
|
Most users should choose HTTPS, which is the default.
|
|
HTTP is provided primarily for debugging purposes.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / HTTP protocol
|
|
\[rs] \[dq]http\[dq]
|
|
2 / HTTPS protocol
|
|
\[rs] \[dq]https\[dq]
|
|
protocol> 1
|
|
\f[R]
|
|
.fi
|
|
.IP "6." 3
|
|
Specify your NetStorage host, CP code, and any necessary content paths
|
|
using this format: \f[C]<domain>/<cpcode>/<content>/\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
host> baseball-nsu.akamaihd.net/123456/content/
|
|
\f[R]
|
|
.fi
|
|
.IP "7." 3
|
|
Set the netstorage account name
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
account> username
|
|
\f[R]
|
|
.fi
|
|
.IP "8." 3
|
|
Set the Netstorage account secret/G2O key which will be used for
|
|
authentication purposes.
|
|
Select the \f[C]y\f[R] option to set your own password then enter your
|
|
secret.
|
|
Note: The secret is stored in the \f[C]rclone.conf\f[R] file with
|
|
hex-encoded encryption.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
y/g> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
\f[R]
|
|
.fi
|
|
.IP "9." 3
|
|
View the summary and confirm your remote configuration.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[ns1]
|
|
type = netstorage
|
|
protocol = http
|
|
host = baseball-nsu.akamaihd.net/123456/content/
|
|
account = username
|
|
secret = *** ENCRYPTED ***
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]ns1\f[R] and can now be used.
|
|
.SS Example operations
|
|
.PP
|
|
Get started with rclone and NetStorage with these examples.
|
|
For additional rclone commands, visit https://rclone.org/commands/.
|
|
.SS See contents of a directory in your project
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd ns1:/974012/testing/
|
|
\f[R]
|
|
.fi
|
|
.SS Sync the contents local with remote
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync . ns1:/974012/testing/
|
|
\f[R]
|
|
.fi
|
|
.SS Upload local content to remote
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy notes.txt ns1:/974012/testing/
|
|
\f[R]
|
|
.fi
|
|
.SS Delete content on remote
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone delete ns1:/974012/testing/notes.txt
|
|
\f[R]
|
|
.fi
|
|
.SS Move or copy content between CP codes.
|
|
.PP
|
|
Your credentials must have access to two CP codes on the same remote.
|
|
You can\[aq]t perform operations between different remotes.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone move ns1:/974012/testing/notes.txt ns1:/974450/testing2/
|
|
\f[R]
|
|
.fi
|
|
.SS Symlink Support
|
|
.PP
|
|
The Netstorage backend changes the rclone \f[C]--links, -l\f[R]
|
|
behavior.
|
|
When uploading, instead of creating the .rclonelink file, use the
|
|
\[dq]symlink\[dq] API in order to create the corresponding symlink on
|
|
the remote.
|
|
The .rclonelink file will not be created, the upload will be intercepted
|
|
and only the symlink file that matches the source file name with no
|
|
suffix will be created on the remote.
|
|
.PP
|
|
This will effectively allow commands like copy/copyto, move/moveto and
|
|
sync to upload from local to remote and download from remote to local
|
|
directories with symlinks.
|
|
Due to internal rclone limitations, it is not possible to upload an
|
|
individual symlink file to any remote backend.
|
|
You can always use the \[dq]backend symlink\[dq] command to create a
|
|
symlink on the NetStorage server, refer to \[dq]symlink\[dq] section
|
|
below.
|
|
.PP
|
|
Individual symlink files on the remote can be used with the commands
|
|
like \[dq]cat\[dq] to print the destination name, or \[dq]delete\[dq] to
|
|
delete symlink, or copy, copy/to and move/moveto to download from the
|
|
remote to local.
|
|
Note: individual symlink files on the remote should be specified
|
|
including the suffix .rclonelink.
|
|
.PP
|
|
\f[B]Note\f[R]: No file with the suffix .rclonelink should ever exist on
|
|
the server since it is not possible to actually upload/create a file
|
|
with .rclonelink suffix with rclone, it can only exist if it is manually
|
|
created through a non-rclone method on the remote.
|
|
.SS Implicit vs. Explicit Directories
|
|
.PP
|
|
With NetStorage, directories can exist in one of two forms:
|
|
.IP "1." 3
|
|
\f[B]Explicit Directory\f[R].
|
|
This is an actual, physical directory that you have created in a storage
|
|
group.
|
|
.IP "2." 3
|
|
\f[B]Implicit Directory\f[R].
|
|
This refers to a directory within a path that has not been physically
|
|
created.
|
|
For example, during upload of a file, non-existent subdirectories can be
|
|
specified in the target path.
|
|
NetStorage creates these as \[dq]implicit.\[dq] While the directories
|
|
aren\[aq]t physically created, they exist implicitly and the noted path
|
|
is connected with the uploaded file.
|
|
.PP
|
|
Rclone will intercept all file uploads and mkdir commands for the
|
|
NetStorage remote and will explicitly issue the mkdir command for each
|
|
directory in the uploading path.
|
|
This will help with the interoperability with the other Akamai services
|
|
such as SFTP and the Content Management Shell (CMShell).
|
|
Rclone will not guarantee correctness of operations with implicit
|
|
directories which might have been created as a result of using an upload
|
|
API directly.
|
|
.SS ListR Feature
|
|
.PP
|
|
NetStorage remote supports the ListR feature by using the \[dq]list\[dq]
|
|
NetStorage API action to return a lexicographical list of all objects
|
|
within the specified CP code, recursing into subdirectories as
|
|
they\[aq]re encountered.
|
|
.IP \[bu] 2
|
|
\f[B]Rclone will use the ListR method for some commands by default\f[R].
|
|
Commands such as \f[C]lsf -R\f[R] will use ListR by default.
|
|
To disable this, include the \f[C]--disable listR\f[R] option to use the
|
|
non-recursive method of listing objects.
|
|
.IP \[bu] 2
|
|
\f[B]Rclone will not use the ListR method for some commands\f[R].
|
|
Commands such as \f[C]sync\f[R] don\[aq]t use ListR by default.
|
|
To force using the ListR method, include the \f[C]--fast-list\f[R]
|
|
option.
|
|
.PP
|
|
There are pros and cons of using the ListR method, refer to rclone
|
|
documentation (https://rclone.org/docs/#fast-list).
|
|
In general, the sync command over an existing deep tree on the remote
|
|
will run faster with the \[dq]--fast-list\[dq] flag but with extra
|
|
memory usage as a side effect.
|
|
It might also result in higher CPU utilization but the whole task can be
|
|
completed faster.
|
|
.PP
|
|
\f[B]Note\f[R]: There is a known limitation that \[dq]lsf -R\[dq] will
|
|
display number of files in the directory and directory size as -1 when
|
|
ListR method is used.
|
|
The workaround is to pass \[dq]--disable listR\[dq] flag if these
|
|
numbers are important in the output.
|
|
.SS Purge Feature
|
|
.PP
|
|
NetStorage remote supports the purge feature by using the
|
|
\[dq]quick-delete\[dq] NetStorage API action.
|
|
The quick-delete action is disabled by default for security reasons and
|
|
can be enabled for the account through the Akamai portal.
|
|
Rclone will first try to use quick-delete action for the purge command
|
|
and if this functionality is disabled then will fall back to a standard
|
|
delete method.
|
|
.PP
|
|
\f[B]Note\f[R]: Read the NetStorage Usage
|
|
API (https://learn.akamai.com/en-us/webhelp/netstorage/netstorage-http-api-developer-guide/GUID-15836617-9F50-405A-833C-EA2556756A30.html)
|
|
for considerations when using \[dq]quick-delete\[dq].
|
|
In general, using quick-delete method will not delete the tree
|
|
immediately and objects targeted for quick-delete may still be
|
|
accessible.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to netstorage (Akamai
|
|
NetStorage).
|
|
.SS --netstorage-host
|
|
.PP
|
|
Domain+path of NetStorage host to connect to.
|
|
.PP
|
|
Format should be \f[C]<domain>/<internal folders>\f[R]
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: host
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_NETSTORAGE_HOST
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --netstorage-account
|
|
.PP
|
|
Set the NetStorage account name
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: account
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_NETSTORAGE_ACCOUNT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --netstorage-secret
|
|
.PP
|
|
Set the NetStorage account secret/G2O key for authentication.
|
|
.PP
|
|
Please choose the \[aq]y\[aq] option to set your own password then enter
|
|
your secret.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_NETSTORAGE_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to netstorage (Akamai
|
|
NetStorage).
|
|
.SS --netstorage-protocol
|
|
.PP
|
|
Select between HTTP or HTTPS protocol.
|
|
.PP
|
|
Most users should choose HTTPS, which is the default.
|
|
HTTP is provided primarily for debugging purposes.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: protocol
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_NETSTORAGE_PROTOCOL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]https\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]http\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
HTTP protocol
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]https\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
HTTPS protocol
|
|
.RE
|
|
.RE
|
|
.SS Backend commands
|
|
.PP
|
|
Here are the commands specific to the netstorage backend.
|
|
.PP
|
|
Run them with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend COMMAND remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The help below will explain what arguments each command takes.
|
|
.PP
|
|
See the \[dq]rclone backend\[dq]
|
|
command (https://rclone.org/commands/rclone_backend/) for more info on
|
|
how to pass options and arguments.
|
|
.PP
|
|
These can be run on a running backend using the rc command
|
|
backend/command (https://rclone.org/rc/#backend-command).
|
|
.SS du
|
|
.PP
|
|
Return disk usage information for a specified directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend du remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The usage information returned, includes the targeted directory as well
|
|
as all files stored in any sub-directories that may exist.
|
|
.SS symlink
|
|
.PP
|
|
You can create a symbolic link in ObjectStore with the symlink action.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend symlink remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The desired path location (including applicable sub-directories) ending
|
|
in the object that will be the target of the symlink (for example,
|
|
/links/mylink).
|
|
Include the file extension for the object, if applicable.
|
|
\f[C]rclone backend symlink <src> <path>\f[R]
|
|
.SH Microsoft Azure Blob Storage
|
|
.PP
|
|
Paths are specified as \f[C]remote:container\f[R] (or \f[C]remote:\f[R]
|
|
for the \f[C]lsd\f[R] command.) You may put subdirectories in too, e.g.
|
|
\f[C]remote:container/path/to/dir\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of making a Microsoft Azure Blob Storage
|
|
configuration.
|
|
For a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Microsoft Azure Blob Storage
|
|
\[rs] \[dq]azureblob\[dq]
|
|
[snip]
|
|
Storage> azureblob
|
|
Storage Account Name
|
|
account> account_name
|
|
Storage Account Key
|
|
key> base64encodedkey==
|
|
Endpoint for the service - leave blank normally.
|
|
endpoint>
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
account = account_name
|
|
key = base64encodedkey==
|
|
endpoint =
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See all containers
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Make a new container
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:container
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a container
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:container
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote container, deleting
|
|
any excess files in the container.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory remote:container
|
|
\f[R]
|
|
.fi
|
|
.SS --fast-list
|
|
.PP
|
|
This remote supports \f[C]--fast-list\f[R] which allows you to use fewer
|
|
transactions in exchange for more memory.
|
|
See the rclone docs (https://rclone.org/docs/#fast-list) for more
|
|
details.
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object with the
|
|
\f[C]mtime\f[R] key.
|
|
It is stored using RFC3339 Format time with nanosecond precision.
|
|
The metadata is supplied during directory listings so there is no
|
|
overhead to using it.
|
|
.SS Performance
|
|
.PP
|
|
When uploading large files, increasing the value of
|
|
\f[C]--azureblob-upload-concurrency\f[R] will increase performance at
|
|
the cost of using more memory.
|
|
The default of 16 is set quite conservatively to use less memory.
|
|
It maybe be necessary raise it to 64 or higher to fully utilize a 1
|
|
GBit/s link with a single file transfer.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
File names can also not end with the following characters.
|
|
These only get replaced if they are the last character in the name:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\&.
|
|
T}@T{
|
|
0x2E
|
|
T}@T{
|
|
\[uFF0E]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Hashes
|
|
.PP
|
|
MD5 hashes are stored with blobs.
|
|
However blobs that were uploaded in chunks only have an MD5 if the
|
|
source remote was capable of MD5 hashes, e.g.
|
|
the local disk.
|
|
.SS Authenticating with Azure Blob Storage
|
|
.PP
|
|
Rclone has 3 ways of authenticating with Azure Blob Storage:
|
|
.SS Account and Key
|
|
.PP
|
|
This is the most straight forward and least flexible way.
|
|
Just fill in the \f[C]account\f[R] and \f[C]key\f[R] lines and leave the
|
|
rest blank.
|
|
.SS SAS URL
|
|
.PP
|
|
This can be an account level SAS URL or container level SAS URL.
|
|
.PP
|
|
To use it leave \f[C]account\f[R], \f[C]key\f[R] blank and fill in
|
|
\f[C]sas_url\f[R].
|
|
.PP
|
|
An account level SAS URL or container level SAS URL can be obtained from
|
|
the Azure portal or the Azure Storage Explorer.
|
|
To get a container level SAS URL right click on a container in the Azure
|
|
Blob explorer in the Azure portal.
|
|
.PP
|
|
If you use a container level SAS URL, rclone operations are permitted
|
|
only on a particular container, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls azureblob:container
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You can also list the single container from the root.
|
|
This will only show the container specified by the SAS URL.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone lsd azureblob:
|
|
container/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that you can\[aq]t see or access any other containers - this will
|
|
fail
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls azureblob:othercontainer
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Container level SAS URLs are useful for temporarily allowing third
|
|
parties access to a single container or putting credentials into an
|
|
untrusted environment such as a CI build server.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to azureblob (Microsoft Azure
|
|
Blob Storage).
|
|
.SS --azureblob-account
|
|
.PP
|
|
Storage Account Name.
|
|
.PP
|
|
Leave blank to use SAS URL or Emulator.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: account
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_ACCOUNT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --azureblob-service-principal-file
|
|
.PP
|
|
Path to file containing credentials for use with a service principal.
|
|
.PP
|
|
Leave blank normally.
|
|
Needed only if you want to use a service principal instead of
|
|
interactive login.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ az ad sp create-for-rbac --name \[dq]<name>\[dq] \[rs]
|
|
--role \[dq]Storage Blob Data Owner\[dq] \[rs]
|
|
--scopes \[dq]/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>\[dq] \[rs]
|
|
> azure-principal.json
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See \[dq]Create an Azure service
|
|
principal\[dq] (https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli)
|
|
and \[dq]Assign an Azure role for access to blob
|
|
data\[dq] (https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli)
|
|
pages for more details.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: service_principal_file
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_SERVICE_PRINCIPAL_FILE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --azureblob-key
|
|
.PP
|
|
Storage Account Key.
|
|
.PP
|
|
Leave blank to use SAS URL or Emulator.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: key
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_KEY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --azureblob-sas-url
|
|
.PP
|
|
SAS URL for container level access only.
|
|
.PP
|
|
Leave blank if using account/key or Emulator.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: sas_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_SAS_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --azureblob-use-msi
|
|
.PP
|
|
Use a managed service identity to authenticate (only works in Azure).
|
|
.PP
|
|
When true, use a managed service
|
|
identity (https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/)
|
|
to authenticate to Azure Storage instead of a SAS token or account key.
|
|
.PP
|
|
If the VM(SS) on which this program is running has a system-assigned
|
|
identity, it will be used by default.
|
|
If the resource has no system-assigned but exactly one user-assigned
|
|
identity, the user-assigned identity will be used by default.
|
|
If the resource has multiple user-assigned identities, the identity to
|
|
use must be explicitly specified using exactly one of the msi_object_id,
|
|
msi_client_id, or msi_mi_res_id parameters.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: use_msi
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_USE_MSI
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --azureblob-use-emulator
|
|
.PP
|
|
Uses local storage emulator if provided as \[aq]true\[aq].
|
|
.PP
|
|
Leave blank if using real azure storage endpoint.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: use_emulator
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_USE_EMULATOR
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to azureblob (Microsoft Azure
|
|
Blob Storage).
|
|
.SS --azureblob-msi-object-id
|
|
.PP
|
|
Object ID of the user-assigned MSI to use, if any.
|
|
.PP
|
|
Leave blank if msi_client_id or msi_mi_res_id specified.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: msi_object_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_MSI_OBJECT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --azureblob-msi-client-id
|
|
.PP
|
|
Object ID of the user-assigned MSI to use, if any.
|
|
.PP
|
|
Leave blank if msi_object_id or msi_mi_res_id specified.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: msi_client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_MSI_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --azureblob-msi-mi-res-id
|
|
.PP
|
|
Azure resource ID of the user-assigned MSI to use, if any.
|
|
.PP
|
|
Leave blank if msi_client_id or msi_object_id specified.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: msi_mi_res_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_MSI_MI_RES_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --azureblob-endpoint
|
|
.PP
|
|
Endpoint for the service.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_ENDPOINT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --azureblob-upload-cutoff
|
|
.PP
|
|
Cutoff for switching to chunked upload (<= 256 MiB) (deprecated).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_cutoff
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_UPLOAD_CUTOFF
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --azureblob-chunk-size
|
|
.PP
|
|
Upload chunk size.
|
|
.PP
|
|
Note that this is stored in memory and there may be up to
|
|
\[dq]--transfers\[dq] * \[dq]--azureblob-upload-concurrency\[dq] chunks
|
|
stored at once in memory.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 4Mi
|
|
.SS --azureblob-upload-concurrency
|
|
.PP
|
|
Concurrency for multipart uploads.
|
|
.PP
|
|
This is the number of chunks of the same file that are uploaded
|
|
concurrently.
|
|
.PP
|
|
If you are uploading small numbers of large files over high-speed links
|
|
and these uploads do not fully utilize your bandwidth, then increasing
|
|
this may help to speed up the transfers.
|
|
.PP
|
|
In tests, upload speed increases almost linearly with upload
|
|
concurrency.
|
|
For example to fill a gigabit pipe it may be necessary to raise this to
|
|
64.
|
|
Note that this will use more memory.
|
|
.PP
|
|
Note that chunks are stored in memory and there may be up to
|
|
\[dq]--transfers\[dq] * \[dq]--azureblob-upload-concurrency\[dq] chunks
|
|
stored at once in memory.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_concurrency
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_UPLOAD_CONCURRENCY
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 16
|
|
.SS --azureblob-list-chunk
|
|
.PP
|
|
Size of blob list.
|
|
.PP
|
|
This sets the number of blobs requested in each listing chunk.
|
|
Default is the maximum, 5000.
|
|
\[dq]List blobs\[dq] requests are permitted 2 minutes per megabyte to
|
|
complete.
|
|
If an operation is taking longer than 2 minutes per megabyte on average,
|
|
it will time out (
|
|
source (https://docs.microsoft.com/en-us/rest/api/storageservices/setting-timeouts-for-blob-service-operations#exceptions-to-default-timeout-interval)
|
|
).
|
|
This can be used to limit the number of blobs items to return, to avoid
|
|
the time out.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: list_chunk
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_LIST_CHUNK
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 5000
|
|
.SS --azureblob-access-tier
|
|
.PP
|
|
Access tier of blob: hot, cool or archive.
|
|
.PP
|
|
Archived blobs can be restored by setting access tier to hot or cool.
|
|
Leave blank if you intend to use default access tier, which is set at
|
|
account level
|
|
.PP
|
|
If there is no \[dq]access tier\[dq] specified, rclone doesn\[aq]t apply
|
|
any tier.
|
|
rclone performs \[dq]Set Tier\[dq] operation on blobs while uploading,
|
|
if objects are not modified, specifying \[dq]access tier\[dq] to new one
|
|
will have no effect.
|
|
If blobs are in \[dq]archive tier\[dq] at remote, trying to perform data
|
|
transfer operations from remote will not be allowed.
|
|
User should first restore by tiering blob to \[dq]Hot\[dq] or
|
|
\[dq]Cool\[dq].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: access_tier
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_ACCESS_TIER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --azureblob-archive-tier-delete
|
|
.PP
|
|
Delete archive tier blobs before overwriting.
|
|
.PP
|
|
Archive tier blobs cannot be updated.
|
|
So without this flag, if you attempt to update an archive tier blob,
|
|
then rclone will produce the error:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
can\[aq]t update archive tier blob without --azureblob-archive-tier-delete
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
With this flag set then before rclone attempts to overwrite an archive
|
|
tier blob, it will delete the existing blob before uploading its
|
|
replacement.
|
|
This has the potential for data loss if the upload fails (unlike
|
|
updating a normal blob) and also may cost more since deleting archive
|
|
tier blobs early may be chargable.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: archive_tier_delete
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_ARCHIVE_TIER_DELETE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --azureblob-disable-checksum
|
|
.PP
|
|
Don\[aq]t store MD5 checksum with object metadata.
|
|
.PP
|
|
Normally rclone will calculate the MD5 checksum of the input before
|
|
uploading it so it can add it to metadata on the object.
|
|
This is great for data integrity checking but can cause long delays for
|
|
large files to start uploading.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_checksum
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_DISABLE_CHECKSUM
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --azureblob-memory-pool-flush-time
|
|
.PP
|
|
How often internal memory buffer pools will be flushed.
|
|
.PP
|
|
Uploads which requires additional buffers (f.e multipart) will use
|
|
memory pool for allocations.
|
|
This option controls how often unused buffers will be removed from the
|
|
pool.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: memory_pool_flush_time
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_FLUSH_TIME
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 1m0s
|
|
.SS --azureblob-memory-pool-use-mmap
|
|
.PP
|
|
Whether to use mmap buffers in internal memory pool.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: memory_pool_use_mmap
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_USE_MMAP
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --azureblob-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8
|
|
.SS --azureblob-public-access
|
|
.PP
|
|
Public access level of a container: blob or container.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: public_access
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_PUBLIC_ACCESS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The container and its blobs can be accessed only with an authorized
|
|
request.
|
|
.IP \[bu] 2
|
|
It\[aq]s a default value.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]blob\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Blob data within this container can be read via anonymous request.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]container\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow full public read access for container and blob data.
|
|
.RE
|
|
.RE
|
|
.SS --azureblob-no-head-object
|
|
.PP
|
|
If set, do not do HEAD before GET when getting objects.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_head_object
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_AZUREBLOB_NO_HEAD_OBJECT
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS Limitations
|
|
.PP
|
|
MD5 sums are only uploaded with chunked files if the source has an MD5
|
|
sum.
|
|
This will always be the case for a local to azure copy.
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the Microsoft Azure Blob
|
|
storage backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SS Azure Storage Emulator Support
|
|
.PP
|
|
You can test rclone with storage emulator locally, to do this make sure
|
|
azure storage emulator installed locally and set up a new remote with
|
|
\f[C]rclone config\f[R] follow instructions described in introduction,
|
|
set \f[C]use_emulator\f[R] config as \f[C]true\f[R], you do not need to
|
|
provide default account name or key if using emulator.
|
|
.SH Microsoft OneDrive
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for OneDrive involves getting a token from Microsoft
|
|
which you need to do in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
e) Edit existing remote
|
|
n) New remote
|
|
d) Delete remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/r/c/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Microsoft OneDrive
|
|
\[rs] \[dq]onedrive\[dq]
|
|
[snip]
|
|
Storage> onedrive
|
|
Microsoft App Client Id
|
|
Leave blank normally.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
client_id>
|
|
Microsoft App Client Secret
|
|
Leave blank normally.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
client_secret>
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No
|
|
y/n> n
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
Choose a number from below, or type in an existing value
|
|
1 / OneDrive Personal or Business
|
|
\[rs] \[dq]onedrive\[dq]
|
|
2 / Sharepoint site
|
|
\[rs] \[dq]sharepoint\[dq]
|
|
3 / Type in driveID
|
|
\[rs] \[dq]driveid\[dq]
|
|
4 / Type in SiteID
|
|
\[rs] \[dq]siteid\[dq]
|
|
5 / Search a Sharepoint site
|
|
\[rs] \[dq]search\[dq]
|
|
Your choice> 1
|
|
Found 1 drives, please select the one you want to use:
|
|
0: OneDrive (business) id=b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk
|
|
Chose drive to use:> 0
|
|
Found drive \[aq]root\[aq] of type \[aq]business\[aq], URL: https://org-my.sharepoint.com/personal/you/Documents
|
|
Is that okay?
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
--------------------
|
|
[remote]
|
|
type = onedrive
|
|
token = {\[dq]access_token\[dq]:\[dq]youraccesstoken\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]yourrefreshtoken\[dq],\[dq]expiry\[dq]:\[dq]2018-08-26T22:39:52.486512262+08:00\[dq]}
|
|
drive_id = b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk
|
|
drive_type = business
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Microsoft.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your OneDrive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your OneDrive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an OneDrive directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Getting your own Client ID and Key
|
|
.PP
|
|
You can use your own Client ID if the default (\f[C]client_id\f[R] left
|
|
blank) one doesn\[aq]t work for you or you see lots of throttling.
|
|
The default Client ID and Key is shared by all rclone users when
|
|
performing requests.
|
|
.PP
|
|
If you are having problems with them (E.g., seeing a lot of throttling),
|
|
you can get your own Client ID and Key by following the steps below:
|
|
.IP "1." 3
|
|
Open
|
|
https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade
|
|
and then click \f[C]New registration\f[R].
|
|
.IP "2." 3
|
|
Enter a name for your app, choose account type
|
|
\f[C]Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox)\f[R],
|
|
select \f[C]Web\f[R] in \f[C]Redirect URI\f[R], then type (do not copy
|
|
and paste) \f[C]http://localhost:53682/\f[R] and click Register.
|
|
Copy and keep the \f[C]Application (client) ID\f[R] under the app name
|
|
for later use.
|
|
.IP "3." 3
|
|
Under \f[C]manage\f[R] select \f[C]Certificates & secrets\f[R], click
|
|
\f[C]New client secret\f[R].
|
|
Enter a description (can be anything) and set \f[C]Expires\f[R] to 24
|
|
months.
|
|
Copy and keep that secret \f[I]Value\f[R] for later use (you
|
|
\f[I]won\[aq]t\f[R] be able to see this value afterwards).
|
|
.IP "4." 3
|
|
Under \f[C]manage\f[R] select \f[C]API permissions\f[R], click
|
|
\f[C]Add a permission\f[R] and select \f[C]Microsoft Graph\f[R] then
|
|
select \f[C]delegated permissions\f[R].
|
|
.IP "5." 3
|
|
Search and select the following permissions: \f[C]Files.Read\f[R],
|
|
\f[C]Files.ReadWrite\f[R], \f[C]Files.Read.All\f[R],
|
|
\f[C]Files.ReadWrite.All\f[R], \f[C]offline_access\f[R],
|
|
\f[C]User.Read\f[R], and optionally \f[C]Sites.Read.All\f[R] (see
|
|
below).
|
|
Once selected click \f[C]Add permissions\f[R] at the bottom.
|
|
.PP
|
|
Now the application is complete.
|
|
Run \f[C]rclone config\f[R] to create or edit a OneDrive remote.
|
|
Supply the app ID and password as Client ID and Secret, respectively.
|
|
rclone will walk you through the remaining steps.
|
|
.PP
|
|
The \f[C]Sites.Read.All\f[R] permission is required if you need to
|
|
search SharePoint sites when configuring the
|
|
remote (https://github.com/rclone/rclone/pull/5883).
|
|
However, if that permission is not assigned, you need to set
|
|
\f[C]disable_site_permission\f[R] option to true in the advanced
|
|
options.
|
|
.SS Modification time and hashes
|
|
.PP
|
|
OneDrive allows modification times to be set on objects accurate to 1
|
|
second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
.PP
|
|
OneDrive personal supports SHA1 type hashes.
|
|
OneDrive for business and Sharepoint Server support
|
|
QuickXorHash (https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash).
|
|
.PP
|
|
For all types of OneDrive you can use the \f[C]--checksum\f[R] flag.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[dq]
|
|
T}@T{
|
|
0x22
|
|
T}@T{
|
|
\[uFF02]
|
|
T}
|
|
T{
|
|
*
|
|
T}@T{
|
|
0x2A
|
|
T}@T{
|
|
\[uFF0A]
|
|
T}
|
|
T{
|
|
:
|
|
T}@T{
|
|
0x3A
|
|
T}@T{
|
|
\[uFF1A]
|
|
T}
|
|
T{
|
|
<
|
|
T}@T{
|
|
0x3C
|
|
T}@T{
|
|
\[uFF1C]
|
|
T}
|
|
T{
|
|
>
|
|
T}@T{
|
|
0x3E
|
|
T}@T{
|
|
\[uFF1E]
|
|
T}
|
|
T{
|
|
?
|
|
T}@T{
|
|
0x3F
|
|
T}@T{
|
|
\[uFF1F]
|
|
T}
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
T{
|
|
|
|
|
T}@T{
|
|
0x7C
|
|
T}@T{
|
|
\[uFF5C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
File names can also not end with the following characters.
|
|
These only get replaced if they are the last character in the name:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
SP
|
|
T}@T{
|
|
0x20
|
|
T}@T{
|
|
\[u2420]
|
|
T}
|
|
T{
|
|
\&.
|
|
T}@T{
|
|
0x2E
|
|
T}@T{
|
|
\[uFF0E]
|
|
T}
|
|
.TE
|
|
.PP
|
|
File names can also not begin with the following characters.
|
|
These only get replaced if they are the first character in the name:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
SP
|
|
T}@T{
|
|
0x20
|
|
T}@T{
|
|
\[u2420]
|
|
T}
|
|
T{
|
|
\[ti]
|
|
T}@T{
|
|
0x7E
|
|
T}@T{
|
|
\[uFF5E]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Deleting files
|
|
.PP
|
|
Any files you delete with rclone will end up in the trash.
|
|
Microsoft doesn\[aq]t provide an API to permanently delete files, nor to
|
|
empty the trash, so you will have to do that with one of Microsoft\[aq]s
|
|
apps or via the OneDrive website.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to onedrive (Microsoft OneDrive).
|
|
.SS --onedrive-client-id
|
|
.PP
|
|
OAuth Client Id.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --onedrive-client-secret
|
|
.PP
|
|
OAuth Client Secret.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_CLIENT_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --onedrive-region
|
|
.PP
|
|
Choose national cloud region for OneDrive.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: region
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_REGION
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]global\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]global\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Microsoft Cloud Global
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]us\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Microsoft Cloud for US Government
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]de\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Microsoft Cloud Germany
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]cn\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Azure and Office 365 operated by 21Vianet in China
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to onedrive (Microsoft OneDrive).
|
|
.SS --onedrive-token
|
|
.PP
|
|
OAuth Access Token as a JSON blob.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --onedrive-auth-url
|
|
.PP
|
|
Auth server URL.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_AUTH_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --onedrive-token-url
|
|
.PP
|
|
Token server url.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_TOKEN_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --onedrive-chunk-size
|
|
.PP
|
|
Chunk size to upload files with - must be multiple of 320k (327,680
|
|
bytes).
|
|
.PP
|
|
Above this size files will be chunked - must be multiple of 320k
|
|
(327,680 bytes) and should not exceed 250M (262,144,000 bytes) else you
|
|
may encounter
|
|
\[dq]Microsoft.SharePoint.Client.InvalidClientQueryException: The
|
|
request message is too big.\[dq] Note that the chunks will be buffered
|
|
into memory.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 10Mi
|
|
.SS --onedrive-drive-id
|
|
.PP
|
|
The ID of the drive to use.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: drive_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_DRIVE_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --onedrive-drive-type
|
|
.PP
|
|
The type of the drive (personal | business | documentLibrary).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: drive_type
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_DRIVE_TYPE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --onedrive-root-folder-id
|
|
.PP
|
|
ID of the root folder.
|
|
.PP
|
|
This isn\[aq]t normally needed, but in special circumstances you might
|
|
know the folder ID that you wish to access but not be able to get there
|
|
through a path traversal.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: root_folder_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_ROOT_FOLDER_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --onedrive-disable-site-permission
|
|
.PP
|
|
Disable the request for Sites.Read.All permission.
|
|
.PP
|
|
If set to true, you will no longer be able to search for a SharePoint
|
|
site when configuring drive ID, because rclone will not request
|
|
Sites.Read.All permission.
|
|
Set it to true if your organization didn\[aq]t assign Sites.Read.All
|
|
permission to the application, and your organization disallows users to
|
|
consent app permission request on their own.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_site_permission
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_DISABLE_SITE_PERMISSION
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --onedrive-expose-onenote-files
|
|
.PP
|
|
Set to make OneNote files show up in directory listings.
|
|
.PP
|
|
By default, rclone will hide OneNote files in directory listings because
|
|
operations like \[dq]Open\[dq] and \[dq]Update\[dq] won\[aq]t work on
|
|
them.
|
|
But this behaviour may also prevent you from deleting them.
|
|
If you want to delete OneNote files or otherwise want them to show up in
|
|
directory listing, set this option.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: expose_onenote_files
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_EXPOSE_ONENOTE_FILES
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --onedrive-server-side-across-configs
|
|
.PP
|
|
Allow server-side operations (e.g.
|
|
copy) to work across different onedrive configs.
|
|
.PP
|
|
This will only work if you are copying between two OneDrive
|
|
\f[I]Personal\f[R] drives AND the files to copy are already shared
|
|
between them.
|
|
In other cases, rclone will fall back to normal copy (which will be
|
|
slightly slower).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: server_side_across_configs
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_SERVER_SIDE_ACROSS_CONFIGS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --onedrive-list-chunk
|
|
.PP
|
|
Size of listing chunk.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: list_chunk
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_LIST_CHUNK
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 1000
|
|
.SS --onedrive-no-versions
|
|
.PP
|
|
Remove all versions on modifying operations.
|
|
.PP
|
|
Onedrive for business creates versions when rclone uploads new files
|
|
overwriting an existing one and when it sets the modification time.
|
|
.PP
|
|
These versions take up space out of the quota.
|
|
.PP
|
|
This flag checks for versions after file upload and setting modification
|
|
time and removes all but the last version.
|
|
.PP
|
|
\f[B]NB\f[R] Onedrive personal can\[aq]t currently delete versions so
|
|
don\[aq]t use this flag there.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_versions
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_NO_VERSIONS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --onedrive-link-scope
|
|
.PP
|
|
Set the scope of the links created by the link command.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: link_scope
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_LINK_SCOPE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]anonymous\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]anonymous\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Anyone with the link has access, without needing to sign in.
|
|
.IP \[bu] 2
|
|
This may include people outside of your organization.
|
|
.IP \[bu] 2
|
|
Anonymous link support may be disabled by an administrator.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]organization\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Anyone signed into your organization (tenant) can use the link to get
|
|
access.
|
|
.IP \[bu] 2
|
|
Only available in OneDrive for Business and SharePoint.
|
|
.RE
|
|
.RE
|
|
.SS --onedrive-link-type
|
|
.PP
|
|
Set the type of the links created by the link command.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: link_type
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_LINK_TYPE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]view\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]view\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Creates a read-only link to the item.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]edit\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Creates a read-write link to the item.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]embed\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Creates an embeddable link to the item.
|
|
.RE
|
|
.RE
|
|
.SS --onedrive-link-password
|
|
.PP
|
|
Set the password for links created by the link command.
|
|
.PP
|
|
At the time of writing this only works with OneDrive personal paid
|
|
accounts.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: link_password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_LINK_PASSWORD
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --onedrive-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ONEDRIVE_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default:
|
|
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
If you don\[aq]t use rclone for 90 days the refresh token will expire.
|
|
This will result in authorization problems.
|
|
This is easy to fix by running the
|
|
\f[C]rclone config reconnect remote:\f[R] command to get a new token and
|
|
refresh token.
|
|
.SS Naming
|
|
.PP
|
|
Note that OneDrive is case insensitive so you can\[aq]t have a file
|
|
called \[dq]Hello.doc\[dq] and one called \[dq]hello.doc\[dq].
|
|
.PP
|
|
There are quite a few characters that can\[aq]t be in OneDrive file
|
|
names.
|
|
These can\[aq]t occur on Windows platforms, but on non-Windows platforms
|
|
they are common.
|
|
Rclone will map these names to and from an identical looking unicode
|
|
equivalent.
|
|
For example if a file has a \f[C]?\f[R] in it will be mapped to
|
|
\f[C]\[uFF1F]\f[R] instead.
|
|
.SS File sizes
|
|
.PP
|
|
The largest allowed file size is 250 GiB for both OneDrive Personal and
|
|
OneDrive for Business (Updated 13 Jan
|
|
2021) (https://support.microsoft.com/en-us/office/invalid-file-names-and-file-types-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa?ui=en-us&rs=en-us&ad=us#individualfilesize).
|
|
.SS Path length
|
|
.PP
|
|
The entire path, including the file name, must contain fewer than 400
|
|
characters for OneDrive, OneDrive for Business and SharePoint Online.
|
|
If you are encrypting file and folder names with rclone, you may want to
|
|
pay attention to this limitation because the encrypted names are
|
|
typically longer than the original ones.
|
|
.SS Number of files
|
|
.PP
|
|
OneDrive seems to be OK with at least 50,000 files in a folder, but at
|
|
100,000 rclone will get errors listing the directory like
|
|
\f[C]couldn\[cq]t list files: UnknownError:\f[R].
|
|
See #2707 (https://github.com/rclone/rclone/issues/2707) for more info.
|
|
.PP
|
|
An official document about the limitations for different types of
|
|
OneDrive can be found
|
|
here (https://support.office.com/en-us/article/invalid-file-names-and-file-types-in-onedrive-onedrive-for-business-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa).
|
|
.SS Versions
|
|
.PP
|
|
Every change in a file OneDrive causes the service to create a new
|
|
version of the the file.
|
|
This counts against a users quota.
|
|
For example changing the modification time of a file creates a second
|
|
version, so the file apparently uses twice the space.
|
|
.PP
|
|
For example the \f[C]copy\f[R] command is affected by this as rclone
|
|
copies the file and then afterwards sets the modification time to match
|
|
the source file which uses another version.
|
|
.PP
|
|
You can use the \f[C]rclone cleanup\f[R] command (see below) to remove
|
|
all old versions.
|
|
.PP
|
|
Or you can set the \f[C]no_versions\f[R] parameter to \f[C]true\f[R] and
|
|
rclone will remove versions after operations which create new versions.
|
|
This takes extra transactions so only enable it if you need it.
|
|
.PP
|
|
\f[B]Note\f[R] At the time of writing Onedrive Personal creates versions
|
|
(but not for setting the modification time) but the API for removing
|
|
them returns \[dq]API not found\[dq] so cleanup and
|
|
\f[C]no_versions\f[R] should not be used on Onedrive Personal.
|
|
.SS Disabling versioning
|
|
.PP
|
|
Starting October 2018, users will no longer be able to disable
|
|
versioning by default.
|
|
This is because Microsoft has brought an
|
|
update (https://techcommunity.microsoft.com/t5/Microsoft-OneDrive-Blog/New-Updates-to-OneDrive-and-SharePoint-Team-Site-Versioning/ba-p/204390)
|
|
to the mechanism.
|
|
To change this new default setting, a PowerShell command is required to
|
|
be run by a SharePoint admin.
|
|
If you are an admin, you can run these commands in PowerShell to change
|
|
that setting:
|
|
.IP "1." 3
|
|
\f[C]Install-Module -Name Microsoft.Online.SharePoint.PowerShell\f[R]
|
|
(in case you haven\[aq]t installed this already)
|
|
.IP "2." 3
|
|
\f[C]Import-Module Microsoft.Online.SharePoint.PowerShell -DisableNameChecking\f[R]
|
|
.IP "3." 3
|
|
\f[C]Connect-SPOService -Url https://YOURSITE-admin.sharepoint.com -Credential YOU\[at]YOURSITE.COM\f[R]
|
|
(replacing \f[C]YOURSITE\f[R], \f[C]YOU\f[R], \f[C]YOURSITE.COM\f[R]
|
|
with the actual values; this will prompt for your credentials)
|
|
.IP "4." 3
|
|
\f[C]Set-SPOTenant -EnableMinimumVersionRequirement $False\f[R]
|
|
.IP "5." 3
|
|
\f[C]Disconnect-SPOService\f[R] (to disconnect from the server)
|
|
.PP
|
|
\f[I]Below are the steps for normal users to disable versioning. If you
|
|
don\[aq]t see the \[dq]No Versioning\[dq] option, make sure the above
|
|
requirements are met.\f[R]
|
|
.PP
|
|
User Weropol (https://github.com/Weropol) has found a method to disable
|
|
versioning on OneDrive
|
|
.IP "1." 3
|
|
Open the settings menu by clicking on the gear symbol at the top of the
|
|
OneDrive Business page.
|
|
.IP "2." 3
|
|
Click Site settings.
|
|
.IP "3." 3
|
|
Once on the Site settings page, navigate to Site Administration > Site
|
|
libraries and lists.
|
|
.IP "4." 3
|
|
Click Customize \[dq]Documents\[dq].
|
|
.IP "5." 3
|
|
Click General Settings > Versioning Settings.
|
|
.IP "6." 3
|
|
Under Document Version History select the option No versioning.
|
|
Note: This will disable the creation of new file versions, but will not
|
|
remove any previous versions.
|
|
Your documents are safe.
|
|
.IP "7." 3
|
|
Apply the changes by clicking OK.
|
|
.IP "8." 3
|
|
Use rclone to upload or modify files.
|
|
(I also use the --no-update-modtime flag)
|
|
.IP "9." 3
|
|
Restore the versioning settings after using rclone.
|
|
(Optional)
|
|
.SS Cleanup
|
|
.PP
|
|
OneDrive supports \f[C]rclone cleanup\f[R] which causes rclone to look
|
|
through every file under the path supplied and delete all version but
|
|
the current version.
|
|
Because this involves traversing all the files, then querying each file
|
|
for versions it can be quite slow.
|
|
Rclone does \f[C]--checkers\f[R] tests in parallel.
|
|
The command also supports \f[C]-i\f[R] which is a great way to see what
|
|
it would do.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone cleanup -i remote:path/subdir # interactively remove all old version for path/subdir
|
|
rclone cleanup remote:path/subdir # unconditionally remove all old version for path/subdir
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]NB\f[R] Onedrive personal can\[aq]t currently delete versions
|
|
.SS Troubleshooting
|
|
.SS Excessive throttling or blocked on SharePoint
|
|
.PP
|
|
If you experience excessive throttling or is being blocked on SharePoint
|
|
then it may help to set the user agent explicitly with a flag like this:
|
|
\f[C]--user-agent \[dq]ISV|rclone.org|rclone/v1.55.1\[dq]\f[R]
|
|
.PP
|
|
The specific details can be found in the Microsoft document: Avoid
|
|
getting throttled or blocked in SharePoint
|
|
Online (https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#how-to-decorate-your-http-traffic-to-avoid-throttling)
|
|
.SS Unexpected file size/hash differences on Sharepoint
|
|
.PP
|
|
It is a
|
|
known (https://github.com/OneDrive/onedrive-api-docs/issues/935#issuecomment-441741631)
|
|
issue that Sharepoint (not OneDrive or OneDrive for Business) silently
|
|
modifies uploaded files, mainly Office files (.docx, .xlsx, etc.),
|
|
causing file size and hash checks to fail.
|
|
There are also other situations that will cause OneDrive to report
|
|
inconsistent file sizes.
|
|
To use rclone with such affected files on Sharepoint, you may disable
|
|
these checks with the following command line arguments:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--ignore-checksum --ignore-size
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Alternatively, if you have write access to the OneDrive files, it may be
|
|
possible to fix this problem for certain files, by attempting the steps
|
|
below.
|
|
Open the web interface for OneDrive (https://onedrive.live.com) and find
|
|
the affected files (which will be in the error messages/log for rclone).
|
|
Simply click on each of these files, causing OneDrive to open them on
|
|
the web.
|
|
This will cause each file to be converted in place to a format that is
|
|
functionally equivalent but which will no longer trigger the size
|
|
discrepancy.
|
|
Once all problematic files are converted you will no longer need the
|
|
ignore options above.
|
|
.SS Replacing/deleting existing files on Sharepoint gets \[dq]item not found\[dq]
|
|
.PP
|
|
It is a
|
|
known (https://github.com/OneDrive/onedrive-api-docs/issues/1068) issue
|
|
that Sharepoint (not OneDrive or OneDrive for Business) may return
|
|
\[dq]item not found\[dq] errors when users try to replace or delete
|
|
uploaded files; this seems to mainly affect Office files (.docx, .xlsx,
|
|
etc.).
|
|
As a workaround, you may use the \f[C]--backup-dir <BACKUP_DIR>\f[R]
|
|
command line argument so rclone moves the files to be replaced/deleted
|
|
into a given backup directory (instead of directly replacing/deleting
|
|
them).
|
|
For example, to instruct rclone to move the files into the directory
|
|
\f[C]rclone-backup-dir\f[R] on backend \f[C]mysharepoint\f[R], you may
|
|
use:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--backup-dir mysharepoint:rclone-backup-dir
|
|
\f[R]
|
|
.fi
|
|
.SS access_denied (AADSTS65005)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Error: access_denied
|
|
Code: AADSTS65005
|
|
Description: Using application \[aq]rclone\[aq] is currently not supported for your organization [YOUR_ORGANIZATION] because it is in an unmanaged state. An administrator needs to claim ownership of the company by DNS validation of [YOUR_ORGANIZATION] before the application rclone can be provisioned.
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This means that rclone can\[aq]t use the OneDrive for Business API with
|
|
your account.
|
|
You can\[aq]t do much about it, maybe write an email to your admins.
|
|
.PP
|
|
However, there are other ways to interact with your OneDrive account.
|
|
Have a look at the webdav backend: https://rclone.org/webdav/#sharepoint
|
|
.SS invalid_grant (AADSTS50076)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Error: invalid_grant
|
|
Code: AADSTS50076
|
|
Description: Due to a configuration change made by your administrator, or because you moved to a new location, you must use multi-factor authentication to access \[aq]...\[aq].
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you see the error above after enabling multi-factor authentication
|
|
for your account, you can fix it by refreshing your OAuth refresh token.
|
|
To do that, run \f[C]rclone config\f[R], and choose to edit your
|
|
OneDrive backend.
|
|
Then, you don\[aq]t need to actually make any changes until you reach
|
|
this question: \f[C]Already have a token - refresh?\f[R].
|
|
For this question, answer \f[C]y\f[R] and go through the process to
|
|
refresh your token, just like the first time the backend is configured.
|
|
After this, rclone should work again for this backend.
|
|
.SS Invalid request when making public links
|
|
.PP
|
|
On Sharepoint and OneDrive for Business, \f[C]rclone link\f[R] may
|
|
return an \[dq]Invalid request\[dq] error.
|
|
A possible cause is that the organisation admin didn\[aq]t allow public
|
|
links to be made for the organisation/sharepoint library.
|
|
To fix the permissions as an admin, take a look at the docs:
|
|
1 (https://docs.microsoft.com/en-us/sharepoint/turn-external-sharing-on-or-off),
|
|
2 (https://support.microsoft.com/en-us/office/set-up-and-manage-access-requests-94b26e0b-2822-49d4-929a-8455698654b3).
|
|
.SH OpenDrive
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n) New remote
|
|
d) Delete remote
|
|
q) Quit config
|
|
e/n/d/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / OpenDrive
|
|
\[rs] \[dq]opendrive\[dq]
|
|
[snip]
|
|
Storage> opendrive
|
|
Username
|
|
username>
|
|
Password
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
y/g> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
--------------------
|
|
[remote]
|
|
username =
|
|
password = *** ENCRYPTED ***
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List directories in top level of your OpenDrive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your OpenDrive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an OpenDrive directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time and MD5SUMs
|
|
.PP
|
|
OpenDrive allows modification times to be set on objects accurate to 1
|
|
second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
NUL
|
|
T}@T{
|
|
0x00
|
|
T}@T{
|
|
\[u2400]
|
|
T}
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
T{
|
|
\[dq]
|
|
T}@T{
|
|
0x22
|
|
T}@T{
|
|
\[uFF02]
|
|
T}
|
|
T{
|
|
*
|
|
T}@T{
|
|
0x2A
|
|
T}@T{
|
|
\[uFF0A]
|
|
T}
|
|
T{
|
|
:
|
|
T}@T{
|
|
0x3A
|
|
T}@T{
|
|
\[uFF1A]
|
|
T}
|
|
T{
|
|
<
|
|
T}@T{
|
|
0x3C
|
|
T}@T{
|
|
\[uFF1C]
|
|
T}
|
|
T{
|
|
>
|
|
T}@T{
|
|
0x3E
|
|
T}@T{
|
|
\[uFF1E]
|
|
T}
|
|
T{
|
|
?
|
|
T}@T{
|
|
0x3F
|
|
T}@T{
|
|
\[uFF1F]
|
|
T}
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
T{
|
|
|
|
|
T}@T{
|
|
0x7C
|
|
T}@T{
|
|
\[uFF5C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
File names can also not begin or end with the following characters.
|
|
These only get replaced if they are the first or last character in the
|
|
name:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
SP
|
|
T}@T{
|
|
0x20
|
|
T}@T{
|
|
\[u2420]
|
|
T}
|
|
T{
|
|
HT
|
|
T}@T{
|
|
0x09
|
|
T}@T{
|
|
\[u2409]
|
|
T}
|
|
T{
|
|
LF
|
|
T}@T{
|
|
0x0A
|
|
T}@T{
|
|
\[u240A]
|
|
T}
|
|
T{
|
|
VT
|
|
T}@T{
|
|
0x0B
|
|
T}@T{
|
|
\[u240B]
|
|
T}
|
|
T{
|
|
CR
|
|
T}@T{
|
|
0x0D
|
|
T}@T{
|
|
\[u240D]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to opendrive (OpenDrive).
|
|
.SS --opendrive-username
|
|
.PP
|
|
Username.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: username
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_OPENDRIVE_USERNAME
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --opendrive-password
|
|
.PP
|
|
Password.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_OPENDRIVE_PASSWORD
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to opendrive (OpenDrive).
|
|
.SS --opendrive-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_OPENDRIVE_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default:
|
|
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot
|
|
.SS --opendrive-chunk-size
|
|
.PP
|
|
Files will be uploaded in chunks this size.
|
|
.PP
|
|
Note that these chunks are buffered in memory so increasing them will
|
|
increase memory use.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_OPENDRIVE_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 10Mi
|
|
.SS Limitations
|
|
.PP
|
|
Note that OpenDrive is case insensitive so you can\[aq]t have a file
|
|
called \[dq]Hello.doc\[dq] and one called \[dq]hello.doc\[dq].
|
|
.PP
|
|
There are quite a few characters that can\[aq]t be in OpenDrive file
|
|
names.
|
|
These can\[aq]t occur on Windows platforms, but on non-Windows platforms
|
|
they are common.
|
|
Rclone will map these names to and from an identical looking unicode
|
|
equivalent.
|
|
For example if a file has a \f[C]?\f[R] in it will be mapped to
|
|
\f[C]\[uFF1F]\f[R] instead.
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the OpenDrive backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SH QingStor
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[R] (or \f[C]remote:\f[R] for
|
|
the \f[C]lsd\f[R] command.) You may put subdirectories in too, e.g.
|
|
\f[C]remote:bucket/path/to/dir\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of making an QingStor configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/r/c/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / QingStor Object Storage
|
|
\[rs] \[dq]qingstor\[dq]
|
|
[snip]
|
|
Storage> qingstor
|
|
Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank.
|
|
Choose a number from below, or type in your own value
|
|
1 / Enter QingStor credentials in the next step
|
|
\[rs] \[dq]false\[dq]
|
|
2 / Get QingStor credentials from the environment (env vars or IAM)
|
|
\[rs] \[dq]true\[dq]
|
|
env_auth> 1
|
|
QingStor Access Key ID - leave blank for anonymous access or runtime credentials.
|
|
access_key_id> access_key
|
|
QingStor Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
|
|
secret_access_key> secret_key
|
|
Enter an endpoint URL to connection QingStor API.
|
|
Leave blank will use the default value \[dq]https://qingstor.com:443\[dq]
|
|
endpoint>
|
|
Zone connect to. Default is \[dq]pek3a\[dq].
|
|
Choose a number from below, or type in your own value
|
|
/ The Beijing (China) Three Zone
|
|
1 | Needs location constraint pek3a.
|
|
\[rs] \[dq]pek3a\[dq]
|
|
/ The Shanghai (China) First Zone
|
|
2 | Needs location constraint sh1a.
|
|
\[rs] \[dq]sh1a\[dq]
|
|
zone> 1
|
|
Number of connection retry.
|
|
Leave blank will use the default value \[dq]3\[dq].
|
|
connection_retries>
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
env_auth = false
|
|
access_key_id = access_key
|
|
secret_access_key = secret_key
|
|
endpoint =
|
|
zone = pek3a
|
|
connection_retries =
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[R] and can now be used like this
|
|
.PP
|
|
See all buckets
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Make a new bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.SS --fast-list
|
|
.PP
|
|
This remote supports \f[C]--fast-list\f[R] which allows you to use fewer
|
|
transactions in exchange for more memory.
|
|
See the rclone docs (https://rclone.org/docs/#fast-list) for more
|
|
details.
|
|
.SS Multipart uploads
|
|
.PP
|
|
rclone supports multipart uploads with QingStor which means that it can
|
|
upload files bigger than 5 GiB.
|
|
Note that files uploaded with multipart upload don\[aq]t have an MD5SUM.
|
|
.PP
|
|
Note that incomplete multipart uploads older than 24 hours can be
|
|
removed with \f[C]rclone cleanup remote:bucket\f[R] just for one bucket
|
|
\f[C]rclone cleanup remote:\f[R] for all buckets.
|
|
QingStor does not ever remove incomplete multipart uploads so it may be
|
|
necessary to run this from time to time.
|
|
.SS Buckets and Zone
|
|
.PP
|
|
With QingStor you can list buckets (\f[C]rclone lsd\f[R]) using any
|
|
zone, but you can only access the content of a bucket from the zone it
|
|
was created in.
|
|
If you attempt to access a bucket from the wrong zone, you will get an
|
|
error,
|
|
\f[C]incorrect zone, the bucket is not in \[aq]XXX\[aq] zone\f[R].
|
|
.SS Authentication
|
|
.PP
|
|
There are two ways to supply \f[C]rclone\f[R] with a set of QingStor
|
|
credentials.
|
|
In order of precedence:
|
|
.IP \[bu] 2
|
|
Directly in the rclone configuration file (as configured by
|
|
\f[C]rclone config\f[R])
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
set \f[C]access_key_id\f[R] and \f[C]secret_access_key\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
Runtime configuration:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
set \f[C]env_auth\f[R] to \f[C]true\f[R] in the config file
|
|
.IP \[bu] 2
|
|
Exporting the following environment variables before running
|
|
\f[C]rclone\f[R]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Access Key ID: \f[C]QS_ACCESS_KEY_ID\f[R] or \f[C]QS_ACCESS_KEY\f[R]
|
|
.IP \[bu] 2
|
|
Secret Access Key: \f[C]QS_SECRET_ACCESS_KEY\f[R] or
|
|
\f[C]QS_SECRET_KEY\f[R]
|
|
.RE
|
|
.RE
|
|
.SS Restricted filename characters
|
|
.PP
|
|
The control characters 0x00-0x1F and / are replaced as in the default
|
|
restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters).
|
|
Note that 0x7F is not replaced.
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to qingstor (QingCloud Object
|
|
Storage).
|
|
.SS --qingstor-env-auth
|
|
.PP
|
|
Get QingStor credentials from runtime.
|
|
.PP
|
|
Only applies if access_key_id and secret_access_key is blank.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: env_auth
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_QINGSTOR_ENV_AUTH
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]false\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enter QingStor credentials in the next step.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]true\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Get QingStor credentials from the environment (env vars or IAM).
|
|
.RE
|
|
.RE
|
|
.SS --qingstor-access-key-id
|
|
.PP
|
|
QingStor Access Key ID.
|
|
.PP
|
|
Leave blank for anonymous access or runtime credentials.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: access_key_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_QINGSTOR_ACCESS_KEY_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --qingstor-secret-access-key
|
|
.PP
|
|
QingStor Secret Access Key (password).
|
|
.PP
|
|
Leave blank for anonymous access or runtime credentials.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: secret_access_key
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_QINGSTOR_SECRET_ACCESS_KEY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --qingstor-endpoint
|
|
.PP
|
|
Enter an endpoint URL to connection QingStor API.
|
|
.PP
|
|
Leave blank will use the default value
|
|
\[dq]https://qingstor.com:443\[dq].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_QINGSTOR_ENDPOINT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --qingstor-zone
|
|
.PP
|
|
Zone to connect to.
|
|
.PP
|
|
Default is \[dq]pek3a\[dq].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: zone
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_QINGSTOR_ZONE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]pek3a\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The Beijing (China) Three Zone.
|
|
.IP \[bu] 2
|
|
Needs location constraint pek3a.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sh1a\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The Shanghai (China) First Zone.
|
|
.IP \[bu] 2
|
|
Needs location constraint sh1a.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]gd2a\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The Guangdong (China) Second Zone.
|
|
.IP \[bu] 2
|
|
Needs location constraint gd2a.
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to qingstor (QingCloud Object
|
|
Storage).
|
|
.SS --qingstor-connection-retries
|
|
.PP
|
|
Number of connection retries.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: connection_retries
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_QINGSTOR_CONNECTION_RETRIES
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 3
|
|
.SS --qingstor-upload-cutoff
|
|
.PP
|
|
Cutoff for switching to chunked upload.
|
|
.PP
|
|
Any files larger than this will be uploaded in chunks of chunk_size.
|
|
The minimum is 0 and the maximum is 5 GiB.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_cutoff
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_QINGSTOR_UPLOAD_CUTOFF
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 200Mi
|
|
.SS --qingstor-chunk-size
|
|
.PP
|
|
Chunk size to use for uploading.
|
|
.PP
|
|
When uploading files larger than upload_cutoff they will be uploaded as
|
|
multipart uploads using this chunk size.
|
|
.PP
|
|
Note that \[dq]--qingstor-upload-concurrency\[dq] chunks of this size
|
|
are buffered in memory per transfer.
|
|
.PP
|
|
If you are transferring large files over high-speed links and you have
|
|
enough memory, then increasing this will speed up the transfers.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_QINGSTOR_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 4Mi
|
|
.SS --qingstor-upload-concurrency
|
|
.PP
|
|
Concurrency for multipart uploads.
|
|
.PP
|
|
This is the number of chunks of the same file that are uploaded
|
|
concurrently.
|
|
.PP
|
|
NB if you set this to > 1 then the checksums of multipart uploads become
|
|
corrupted (the uploads themselves are not corrupted though).
|
|
.PP
|
|
If you are uploading small numbers of large files over high-speed links
|
|
and these uploads do not fully utilize your bandwidth, then increasing
|
|
this may help to speed up the transfers.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upload_concurrency
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_QINGSTOR_UPLOAD_CONCURRENCY
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 1
|
|
.SS --qingstor-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_QINGSTOR_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,Ctl,InvalidUtf8
|
|
.SS Limitations
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the qingstor backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SH Sia
|
|
.PP
|
|
Sia (sia.tech (https://sia.tech/)) is a decentralized cloud storage
|
|
platform based on the blockchain (https://wikipedia.org/wiki/Blockchain)
|
|
technology.
|
|
With rclone you can use it like any other remote filesystem or mount Sia
|
|
folders locally.
|
|
The technology behind it involves a number of new concepts such as
|
|
Siacoins and Wallet, Blockchain and Consensus, Renting and Hosting, and
|
|
so on.
|
|
If you are new to it, you\[aq]d better first familiarize yourself using
|
|
their excellent support documentation (https://support.sia.tech/).
|
|
.SS Introduction
|
|
.PP
|
|
Before you can use rclone with Sia, you will need to have a running copy
|
|
of \f[C]Sia-UI\f[R] or \f[C]siad\f[R] (the Sia daemon) locally on your
|
|
computer or on local network (e.g.
|
|
a NAS).
|
|
Please follow the Get started (https://sia.tech/get-started) guide and
|
|
install one.
|
|
.PP
|
|
rclone interacts with Sia network by talking to the Sia daemon via HTTP
|
|
API (https://sia.tech/docs/) which is usually available on port
|
|
\f[I]9980\f[R].
|
|
By default you will run the daemon locally on the same computer so
|
|
it\[aq]s safe to leave the API password blank (the API URL will be
|
|
\f[C]http://127.0.0.1:9980\f[R] making external access impossible).
|
|
.PP
|
|
However, if you want to access Sia daemon running on another node, for
|
|
example due to memory constraints or because you want to share single
|
|
daemon between several rclone and Sia-UI instances, you\[aq]ll need to
|
|
make a few more provisions: - Ensure you have \f[I]Sia daemon\f[R]
|
|
installed directly or in a docker
|
|
container (https://github.com/SiaFoundation/siad/pkgs/container/siad)
|
|
because Sia-UI does not support this mode natively.
|
|
- Run it on externally accessible port, for example provide
|
|
\f[C]--api-addr :9980\f[R] and \f[C]--disable-api-security\f[R]
|
|
arguments on the daemon command line.
|
|
- Enforce API password for the \f[C]siad\f[R] daemon via environment
|
|
variable \f[C]SIA_API_PASSWORD\f[R] or text file named
|
|
\f[C]apipassword\f[R] in the daemon directory.
|
|
- Set rclone backend option \f[C]api_password\f[R] taking it from above
|
|
locations.
|
|
.PP
|
|
Notes: 1.
|
|
If your wallet is locked, rclone cannot unlock it automatically.
|
|
You should either unlock it in advance by using Sia-UI or via command
|
|
line \f[C]siac wallet unlock\f[R].
|
|
Alternatively you can make \f[C]siad\f[R] unlock your wallet
|
|
automatically upon startup by running it with environment variable
|
|
\f[C]SIA_WALLET_PASSWORD\f[R].
|
|
2.
|
|
If \f[C]siad\f[R] cannot find the \f[C]SIA_API_PASSWORD\f[R] variable or
|
|
the \f[C]apipassword\f[R] file in the \f[C]SIA_DIR\f[R] directory, it
|
|
will generate a random password and store in the text file named
|
|
\f[C]apipassword\f[R] under \f[C]YOUR_HOME/.sia/\f[R] directory on Unix
|
|
or
|
|
\f[C]C:\[rs]Users\[rs]YOUR_HOME\[rs]AppData\[rs]Local\[rs]Sia\[rs]apipassword\f[R]
|
|
on Windows.
|
|
Remember this when you configure password in rclone.
|
|
3.
|
|
The only way to use \f[C]siad\f[R] without API password is to run it
|
|
\f[B]on localhost\f[R] with command line argument
|
|
\f[C]--authorize-api=false\f[R], but this is insecure and \f[B]strongly
|
|
discouraged\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of how to make a \f[C]sia\f[R] remote called
|
|
\f[C]mySia\f[R].
|
|
First, run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> mySia
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
\&...
|
|
29 / Sia Decentralized Cloud
|
|
\[rs] \[dq]sia\[dq]
|
|
\&...
|
|
Storage> sia
|
|
Sia daemon API URL, like http://sia.daemon.host:9980.
|
|
Note that siad must run with --disable-api-security to open API port for other hosts (not recommended).
|
|
Keep default if Sia daemon runs on localhost.
|
|
Enter a string value. Press Enter for the default (\[dq]http://127.0.0.1:9980\[dq]).
|
|
api_url> http://127.0.0.1:9980
|
|
Sia Daemon API Password.
|
|
Can be found in the apipassword file located in HOME/.sia/ or in the daemon directory.
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
n) No leave this optional password blank (default)
|
|
y/g/n> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
Edit advanced config?
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
--------------------
|
|
[mySia]
|
|
type = sia
|
|
api_url = http://127.0.0.1:9980
|
|
api_password = *** ENCRYPTED ***
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Once configured, you can then use \f[C]rclone\f[R] like this:
|
|
.IP \[bu] 2
|
|
List directories in top level of your Sia storage
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd mySia:
|
|
\f[R]
|
|
.fi
|
|
.IP \[bu] 2
|
|
List all the files in your Sia storage
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls mySia:
|
|
\f[R]
|
|
.fi
|
|
.IP \[bu] 2
|
|
Upload a local directory to the Sia directory called \f[I]backup\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source mySia:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to sia (Sia Decentralized Cloud).
|
|
.SS --sia-api-url
|
|
.PP
|
|
Sia daemon API URL, like http://sia.daemon.host:9980.
|
|
.PP
|
|
Note that siad must run with --disable-api-security to open API port for
|
|
other hosts (not recommended).
|
|
Keep default if Sia daemon runs on localhost.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: api_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SIA_API_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]http://127.0.0.1:9980\[dq]
|
|
.SS --sia-api-password
|
|
.PP
|
|
Sia Daemon API Password.
|
|
.PP
|
|
Can be found in the apipassword file located in HOME/.sia/ or in the
|
|
daemon directory.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: api_password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SIA_API_PASSWORD
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to sia (Sia Decentralized Cloud).
|
|
.SS --sia-user-agent
|
|
.PP
|
|
Siad User Agent
|
|
.PP
|
|
Sia daemon requires the \[aq]Sia-Agent\[aq] user agent by default for
|
|
security
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user_agent
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SIA_USER_AGENT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]Sia-Agent\[dq]
|
|
.SS --sia-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SIA_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.IP \[bu] 2
|
|
Modification times not supported
|
|
.IP \[bu] 2
|
|
Checksums not supported
|
|
.IP \[bu] 2
|
|
\f[C]rclone about\f[R] not supported
|
|
.IP \[bu] 2
|
|
rclone can work only with \f[I]Siad\f[R] or \f[I]Sia-UI\f[R] at the
|
|
moment, the \f[B]SkyNet daemon is not supported yet.\f[R]
|
|
.IP \[bu] 2
|
|
Sia does not allow control characters or symbols like question and pound
|
|
signs in file names.
|
|
rclone will transparently encode (https://rclone.org/overview/#encoding)
|
|
them for you, but you\[aq]d better be aware
|
|
.SH Swift
|
|
.PP
|
|
Swift refers to OpenStack Object
|
|
Storage (https://docs.openstack.org/swift/latest/).
|
|
Commercial implementations of that being:
|
|
.IP \[bu] 2
|
|
Rackspace Cloud Files (https://www.rackspace.com/cloud/files/)
|
|
.IP \[bu] 2
|
|
Memset Memstore (https://www.memset.com/cloud/storage/)
|
|
.IP \[bu] 2
|
|
OVH Object
|
|
Storage (https://www.ovh.co.uk/public-cloud/storage/object-storage/)
|
|
.IP \[bu] 2
|
|
Oracle Cloud Storage (https://cloud.oracle.com/object-storage/buckets)
|
|
.IP \[bu] 2
|
|
IBM Bluemix Cloud ObjectStorage
|
|
Swift (https://console.bluemix.net/docs/infrastructure/objectstorage-swift/index.html)
|
|
.PP
|
|
Paths are specified as \f[C]remote:container\f[R] (or \f[C]remote:\f[R]
|
|
for the \f[C]lsd\f[R] command.) You may put subdirectories in too, e.g.
|
|
\f[C]remote:container/path/to/dir\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of making a swift configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
|
|
\[rs] \[dq]swift\[dq]
|
|
[snip]
|
|
Storage> swift
|
|
Get swift credentials from environment variables in standard OpenStack form.
|
|
Choose a number from below, or type in your own value
|
|
1 / Enter swift credentials in the next step
|
|
\[rs] \[dq]false\[dq]
|
|
2 / Get swift credentials from environment vars. Leave other fields blank if using this.
|
|
\[rs] \[dq]true\[dq]
|
|
env_auth> true
|
|
User name to log in (OS_USERNAME).
|
|
user>
|
|
API key or password (OS_PASSWORD).
|
|
key>
|
|
Authentication URL for server (OS_AUTH_URL).
|
|
Choose a number from below, or type in your own value
|
|
1 / Rackspace US
|
|
\[rs] \[dq]https://auth.api.rackspacecloud.com/v1.0\[dq]
|
|
2 / Rackspace UK
|
|
\[rs] \[dq]https://lon.auth.api.rackspacecloud.com/v1.0\[dq]
|
|
3 / Rackspace v2
|
|
\[rs] \[dq]https://identity.api.rackspacecloud.com/v2.0\[dq]
|
|
4 / Memset Memstore UK
|
|
\[rs] \[dq]https://auth.storage.memset.com/v1.0\[dq]
|
|
5 / Memset Memstore UK v2
|
|
\[rs] \[dq]https://auth.storage.memset.com/v2.0\[dq]
|
|
6 / OVH
|
|
\[rs] \[dq]https://auth.cloud.ovh.net/v3\[dq]
|
|
auth>
|
|
User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID).
|
|
user_id>
|
|
User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME)
|
|
domain>
|
|
Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME)
|
|
tenant>
|
|
Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID)
|
|
tenant_id>
|
|
Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME)
|
|
tenant_domain>
|
|
Region name - optional (OS_REGION_NAME)
|
|
region>
|
|
Storage URL - optional (OS_STORAGE_URL)
|
|
storage_url>
|
|
Auth Token from alternate authentication - optional (OS_AUTH_TOKEN)
|
|
auth_token>
|
|
AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION)
|
|
auth_version>
|
|
Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE)
|
|
Choose a number from below, or type in your own value
|
|
1 / Public (default, choose this if not sure)
|
|
\[rs] \[dq]public\[dq]
|
|
2 / Internal (use internal service net)
|
|
\[rs] \[dq]internal\[dq]
|
|
3 / Admin
|
|
\[rs] \[dq]admin\[dq]
|
|
endpoint_type>
|
|
Remote config
|
|
--------------------
|
|
[test]
|
|
env_auth = true
|
|
user =
|
|
key =
|
|
auth =
|
|
user_id =
|
|
domain =
|
|
tenant =
|
|
tenant_id =
|
|
tenant_domain =
|
|
region =
|
|
storage_url =
|
|
auth_token =
|
|
auth_version =
|
|
endpoint_type =
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[R] and can now be used like this
|
|
.PP
|
|
See all containers
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Make a new container
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:container
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a container
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:container
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote container, deleting
|
|
any excess files in the container.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory remote:container
|
|
\f[R]
|
|
.fi
|
|
.SS Configuration from an OpenStack credentials file
|
|
.PP
|
|
An OpenStack credentials file typically looks something something like
|
|
this (without the comments)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
export OS_AUTH_URL=https://a.provider.net/v2.0
|
|
export OS_TENANT_ID=ffffffffffffffffffffffffffffffff
|
|
export OS_TENANT_NAME=\[dq]1234567890123456\[dq]
|
|
export OS_USERNAME=\[dq]123abc567xy\[dq]
|
|
echo \[dq]Please enter your OpenStack Password: \[dq]
|
|
read -sr OS_PASSWORD_INPUT
|
|
export OS_PASSWORD=$OS_PASSWORD_INPUT
|
|
export OS_REGION_NAME=\[dq]SBG1\[dq]
|
|
if [ -z \[dq]$OS_REGION_NAME\[dq] ]; then unset OS_REGION_NAME; fi
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The config file needs to look something like this where
|
|
\f[C]$OS_USERNAME\f[R] represents the value of the \f[C]OS_USERNAME\f[R]
|
|
variable - \f[C]123abc567xy\f[R] in the example above.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[remote]
|
|
type = swift
|
|
user = $OS_USERNAME
|
|
key = $OS_PASSWORD
|
|
auth = $OS_AUTH_URL
|
|
tenant = $OS_TENANT_NAME
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that you may (or may not) need to set \f[C]region\f[R] too - try
|
|
without first.
|
|
.SS Configuration from the environment
|
|
.PP
|
|
If you prefer you can configure rclone to use swift using a standard set
|
|
of OpenStack environment variables.
|
|
.PP
|
|
When you run through the config, make sure you choose \f[C]true\f[R] for
|
|
\f[C]env_auth\f[R] and leave everything else blank.
|
|
.PP
|
|
rclone will then set any empty config parameters from the environment
|
|
using standard OpenStack environment variables.
|
|
There is a list of the
|
|
variables (https://godoc.org/github.com/ncw/swift#Connection.ApplyEnvironment)
|
|
in the docs for the swift library.
|
|
.SS Using an alternate authentication method
|
|
.PP
|
|
If your OpenStack installation uses a non-standard authentication method
|
|
that might not be yet supported by rclone or the underlying swift
|
|
library, you can authenticate externally (e.g.
|
|
calling manually the \f[C]openstack\f[R] commands to get a token).
|
|
Then, you just need to pass the two configuration variables
|
|
\f[C]auth_token\f[R] and \f[C]storage_url\f[R].
|
|
If they are both provided, the other variables are ignored.
|
|
rclone will not try to authenticate but instead assume it is already
|
|
authenticated and use these two variables to access the OpenStack
|
|
installation.
|
|
.SS Using rclone without a config file
|
|
.PP
|
|
You can use rclone with swift without a config file, if desired, like
|
|
this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
source openstack-credentials-file
|
|
export RCLONE_CONFIG_MYREMOTE_TYPE=swift
|
|
export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true
|
|
rclone lsd myremote:
|
|
\f[R]
|
|
.fi
|
|
.SS --fast-list
|
|
.PP
|
|
This remote supports \f[C]--fast-list\f[R] which allows you to use fewer
|
|
transactions in exchange for more memory.
|
|
See the rclone docs (https://rclone.org/docs/#fast-list) for more
|
|
details.
|
|
.SS --update and --use-server-modtime
|
|
.PP
|
|
As noted below, the modified time is stored on metadata on the object.
|
|
It is used by default for all operations that require checking the time
|
|
a file was last updated.
|
|
It allows rclone to treat the remote more like a true filesystem, but it
|
|
is inefficient because it requires an extra API call to retrieve the
|
|
metadata.
|
|
.PP
|
|
For many operations, the time the object was last uploaded to the remote
|
|
is sufficient to determine if it is \[dq]dirty\[dq].
|
|
By using \f[C]--update\f[R] along with \f[C]--use-server-modtime\f[R],
|
|
you can avoid the extra API call and simply upload files whose local
|
|
modtime is newer than the time it was last uploaded.
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X-Object-Meta-Mtime\f[R] as floating point since the epoch accurate
|
|
to 1 ns.
|
|
.PP
|
|
This is a de facto standard (used in the official python-swiftclient
|
|
amongst others) for storing the modification time for an object.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
NUL
|
|
T}@T{
|
|
0x00
|
|
T}@T{
|
|
\[u2400]
|
|
T}
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to swift (OpenStack Swift
|
|
(Rackspace Cloud Files, Memset Memstore, OVH)).
|
|
.SS --swift-env-auth
|
|
.PP
|
|
Get swift credentials from environment variables in standard OpenStack
|
|
form.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: env_auth
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_ENV_AUTH
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]false\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enter swift credentials in the next step.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]true\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Get swift credentials from environment vars.
|
|
.IP \[bu] 2
|
|
Leave other fields blank if using this.
|
|
.RE
|
|
.RE
|
|
.SS --swift-user
|
|
.PP
|
|
User name to log in (OS_USERNAME).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_USER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-key
|
|
.PP
|
|
API key or password (OS_PASSWORD).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: key
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_KEY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-auth
|
|
.PP
|
|
Authentication URL for server (OS_AUTH_URL).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_AUTH
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]https://auth.api.rackspacecloud.com/v1.0\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rackspace US
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]https://lon.auth.api.rackspacecloud.com/v1.0\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rackspace UK
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]https://identity.api.rackspacecloud.com/v2.0\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rackspace v2
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]https://auth.storage.memset.com/v1.0\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Memset Memstore UK
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]https://auth.storage.memset.com/v2.0\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Memset Memstore UK v2
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]https://auth.cloud.ovh.net/v3\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
OVH
|
|
.RE
|
|
.RE
|
|
.SS --swift-user-id
|
|
.PP
|
|
User ID to log in - optional - most swift systems use user and leave
|
|
this blank (v3 auth) (OS_USER_ID).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_USER_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-domain
|
|
.PP
|
|
User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME)
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: domain
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_DOMAIN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-tenant
|
|
.PP
|
|
Tenant name - optional for v1 auth, this or tenant_id required otherwise
|
|
(OS_TENANT_NAME or OS_PROJECT_NAME).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: tenant
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_TENANT
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-tenant-id
|
|
.PP
|
|
Tenant ID - optional for v1 auth, this or tenant required otherwise
|
|
(OS_TENANT_ID).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: tenant_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_TENANT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-tenant-domain
|
|
.PP
|
|
Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: tenant_domain
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_TENANT_DOMAIN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-region
|
|
.PP
|
|
Region name - optional (OS_REGION_NAME).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: region
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_REGION
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-storage-url
|
|
.PP
|
|
Storage URL - optional (OS_STORAGE_URL).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: storage_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_STORAGE_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-auth-token
|
|
.PP
|
|
Auth Token from alternate authentication - optional (OS_AUTH_TOKEN).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_AUTH_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-application-credential-id
|
|
.PP
|
|
Application Credential ID (OS_APPLICATION_CREDENTIAL_ID).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: application_credential_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-application-credential-name
|
|
.PP
|
|
Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: application_credential_name
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_NAME
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-application-credential-secret
|
|
.PP
|
|
Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: application_credential_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --swift-auth-version
|
|
.PP
|
|
AuthVersion - optional - set to (1,2,3) if your auth URL has no version
|
|
(ST_AUTH_VERSION).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_version
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_AUTH_VERSION
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 0
|
|
.SS --swift-endpoint-type
|
|
.PP
|
|
Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: endpoint_type
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_ENDPOINT_TYPE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]public\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]public\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Public (default, choose this if not sure)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]internal\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Internal (use internal service net)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]admin\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Admin
|
|
.RE
|
|
.RE
|
|
.SS --swift-storage-policy
|
|
.PP
|
|
The storage policy to use when creating a new container.
|
|
.PP
|
|
This applies the specified storage policy when creating a new container.
|
|
The policy cannot be changed afterwards.
|
|
The allowed configuration values and their meaning depend on your Swift
|
|
storage provider.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: storage_policy
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_STORAGE_POLICY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Default
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]pcs\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
OVH Public Cloud Storage
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]pca\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
OVH Public Cloud Archive
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to swift (OpenStack Swift
|
|
(Rackspace Cloud Files, Memset Memstore, OVH)).
|
|
.SS --swift-leave-parts-on-error
|
|
.PP
|
|
If true avoid calling abort upload on a failure.
|
|
.PP
|
|
It should be set to true for resuming uploads across different sessions.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: leave_parts_on_error
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_LEAVE_PARTS_ON_ERROR
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --swift-chunk-size
|
|
.PP
|
|
Above this size files will be chunked into a _segments container.
|
|
.PP
|
|
Above this size files will be chunked into a _segments container.
|
|
The default for this is 5 GiB which is its maximum value.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: chunk_size
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_CHUNK_SIZE
|
|
.IP \[bu] 2
|
|
Type: SizeSuffix
|
|
.IP \[bu] 2
|
|
Default: 5Gi
|
|
.SS --swift-no-chunk
|
|
.PP
|
|
Don\[aq]t chunk files during streaming upload.
|
|
.PP
|
|
When doing streaming uploads (e.g.
|
|
using rcat or mount) setting this flag will cause the swift backend to
|
|
not upload chunked files.
|
|
.PP
|
|
This will limit the maximum upload size to 5 GiB.
|
|
However non chunked files are easier to deal with and have an MD5SUM.
|
|
.PP
|
|
Rclone will still chunk files bigger than chunk_size when doing normal
|
|
copy operations.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_chunk
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_NO_CHUNK
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --swift-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SWIFT_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,InvalidUtf8
|
|
.SS Limitations
|
|
.PP
|
|
The Swift API doesn\[aq]t return a correct MD5SUM for segmented files
|
|
(Dynamic or Static Large Objects) so rclone won\[aq]t check or use the
|
|
MD5SUM for these.
|
|
.SS Troubleshooting
|
|
.SS Rclone gives Failed to create file system for \[dq]remote:\[dq]: Bad Request
|
|
.PP
|
|
Due to an oddity of the underlying swift library, it gives a \[dq]Bad
|
|
Request\[dq] error rather than a more sensible error when the
|
|
authentication fails for Swift.
|
|
.PP
|
|
So this most likely means your username / password is wrong.
|
|
You can investigate further with the \f[C]--dump-bodies\f[R] flag.
|
|
.PP
|
|
This may also be caused by specifying the region when you shouldn\[aq]t
|
|
have (e.g.
|
|
OVH).
|
|
.SS Rclone gives Failed to create file system: Response didn\[aq]t have storage url and auth token
|
|
.PP
|
|
This is most likely caused by forgetting to specify your tenant when
|
|
setting up a swift remote.
|
|
.SS OVH Cloud Archive
|
|
.PP
|
|
To use rclone with OVH cloud archive, first use \f[C]rclone config\f[R]
|
|
to set up a \f[C]swift\f[R] backend with OVH, choosing \f[C]pca\f[R] as
|
|
the \f[C]storage_policy\f[R].
|
|
.SS Uploading Objects
|
|
.PP
|
|
Uploading objects to OVH cloud archive is no different to object
|
|
storage, you just simply run the command you like (move, copy or sync)
|
|
to upload the objects.
|
|
Once uploaded the objects will show in a \[dq]Frozen\[dq] state within
|
|
the OVH control panel.
|
|
.SS Retrieving Objects
|
|
.PP
|
|
To retrieve objects use \f[C]rclone copy\f[R] as normal.
|
|
If the objects are in a frozen state then rclone will ask for them all
|
|
to be unfrozen and it will wait at the end of the output with a message
|
|
like the following:
|
|
.PP
|
|
\f[C]2019/03/23 13:06:33 NOTICE: Received retry after error - sleeping until 2019-03-23T13:16:33.481657164+01:00 (9m59.99985121s)\f[R]
|
|
.PP
|
|
Rclone will wait for the time specified then retry the copy.
|
|
.SH pCloud
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for pCloud involves getting a token from pCloud which
|
|
you need to do in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Pcloud
|
|
\[rs] \[dq]pcloud\[dq]
|
|
[snip]
|
|
Storage> pcloud
|
|
Pcloud App Client Id - leave blank normally.
|
|
client_id>
|
|
Pcloud App Client Secret - leave blank normally.
|
|
client_secret>
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
--------------------
|
|
[remote]
|
|
client_id =
|
|
client_secret =
|
|
token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]bearer\[dq],\[dq]expiry\[dq]:\[dq]0001-01-01T00:00:00Z\[dq]}
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from pCloud.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your pCloud
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your pCloud
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to a pCloud directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
pCloud allows modification times to be set on objects accurate to 1
|
|
second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
In order to set a Modification time pCloud requires the object be
|
|
re-uploaded.
|
|
.PP
|
|
pCloud supports MD5 and SHA1 hashes in the US region, and SHA1 and
|
|
SHA256 hashes in the EU region, so you can use the \f[C]--checksum\f[R]
|
|
flag.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Deleting files
|
|
.PP
|
|
Deleted files will be moved to the trash.
|
|
Your subscription level will determine how long items stay in the trash.
|
|
\f[C]rclone cleanup\f[R] can be used to empty the trash.
|
|
.SS Root folder ID
|
|
.PP
|
|
You can set the \f[C]root_folder_id\f[R] for rclone.
|
|
This is the directory (identified by its \f[C]Folder ID\f[R]) that
|
|
rclone considers to be the root of your pCloud drive.
|
|
.PP
|
|
Normally you will leave this blank and rclone will determine the correct
|
|
root to use itself.
|
|
.PP
|
|
However you can set this to restrict rclone to a specific folder
|
|
hierarchy.
|
|
.PP
|
|
In order to do this you will have to find the \f[C]Folder ID\f[R] of the
|
|
directory you wish rclone to display.
|
|
This will be the \f[C]folder\f[R] field of the URL when you open the
|
|
relevant folder in the pCloud web interface.
|
|
.PP
|
|
So if the folder you want rclone to use has a URL which looks like
|
|
\f[C]https://my.pcloud.com/#page=filemanager&folder=5xxxxxxxx8&tpl=foldergrid\f[R]
|
|
in the browser, then you use \f[C]5xxxxxxxx8\f[R] as the
|
|
\f[C]root_folder_id\f[R] in the config.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to pcloud (Pcloud).
|
|
.SS --pcloud-client-id
|
|
.PP
|
|
OAuth Client Id.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_PCLOUD_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --pcloud-client-secret
|
|
.PP
|
|
OAuth Client Secret.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_PCLOUD_CLIENT_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to pcloud (Pcloud).
|
|
.SS --pcloud-token
|
|
.PP
|
|
OAuth Access Token as a JSON blob.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_PCLOUD_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --pcloud-auth-url
|
|
.PP
|
|
Auth server URL.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_PCLOUD_AUTH_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --pcloud-token-url
|
|
.PP
|
|
Token server url.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_PCLOUD_TOKEN_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --pcloud-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_PCLOUD_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
|
|
.SS --pcloud-root-folder-id
|
|
.PP
|
|
Fill in for rclone to use a non root folder as its starting point.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: root_folder_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_PCLOUD_ROOT_FOLDER_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]d0\[dq]
|
|
.SS --pcloud-hostname
|
|
.PP
|
|
Hostname to connect to.
|
|
.PP
|
|
This is normally set when rclone initially does the oauth connection,
|
|
however you will need to set it by hand if you are using remote config
|
|
with rclone authorize.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: hostname
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_PCLOUD_HOSTNAME
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]api.pcloud.com\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]api.pcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Original/US region
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eapi.pcloud.com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
EU region
|
|
.RE
|
|
.RE
|
|
.SH premiumize.me
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for premiumize.me (https://premiumize.me/) involves
|
|
getting a token from premiumize.me which you need to do in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / premiumize.me
|
|
\[rs] \[dq]premiumizeme\[dq]
|
|
[snip]
|
|
Storage> premiumizeme
|
|
** See help for premiumizeme backend at: https://rclone.org/premiumizeme/ **
|
|
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
--------------------
|
|
[remote]
|
|
type = premiumizeme
|
|
token = {\[dq]access_token\[dq]:\[dq]XXX\[dq],\[dq]token_type\[dq]:\[dq]Bearer\[dq],\[dq]refresh_token\[dq]:\[dq]XXX\[dq],\[dq]expiry\[dq]:\[dq]2029-08-07T18:44:15.548915378+01:00\[dq]}
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from premiumize.me.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your premiumize.me
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your premiumize.me
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an premiumize.me directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
premiumize.me does not support modification times or hashes, therefore
|
|
syncing will default to \f[C]--size-only\f[R] checking.
|
|
Note that using \f[C]--update\f[R] will work.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
T{
|
|
\[dq]
|
|
T}@T{
|
|
0x22
|
|
T}@T{
|
|
\[uFF02]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to premiumizeme (premiumize.me).
|
|
.SS --premiumizeme-api-key
|
|
.PP
|
|
API Key.
|
|
.PP
|
|
This is not normally used - use oauth instead.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: api_key
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_PREMIUMIZEME_API_KEY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to premiumizeme (premiumize.me).
|
|
.SS --premiumizeme-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_PREMIUMIZEME_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
Note that premiumize.me is case insensitive so you can\[aq]t have a file
|
|
called \[dq]Hello.doc\[dq] and one called \[dq]hello.doc\[dq].
|
|
.PP
|
|
premiumize.me file names can\[aq]t have the \f[C]\[rs]\f[R] or
|
|
\f[C]\[dq]\f[R] characters in.
|
|
rclone maps these to and from an identical looking unicode equivalents
|
|
\f[C]\[uFF3C]\f[R] and \f[C]\[uFF02]\f[R]
|
|
.PP
|
|
premiumize.me only supports filenames up to 255 characters in length.
|
|
.SH put.io
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
put.io paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for put.io involves getting a token from put.io which
|
|
you need to do in your browser.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> putio
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Put.io
|
|
\[rs] \[dq]putio\[dq]
|
|
[snip]
|
|
Storage> putio
|
|
** See help for putio backend at: https://rclone.org/putio/ **
|
|
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
--------------------
|
|
[putio]
|
|
type = putio
|
|
token = {\[dq]access_token\[dq]:\[dq]XXXXXXXX\[dq],\[dq]expiry\[dq]:\[dq]0001-01-01T00:00:00Z\[dq]}
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
Current remotes:
|
|
|
|
Name Type
|
|
==== ====
|
|
putio putio
|
|
|
|
e) Edit existing remote
|
|
n) New remote
|
|
d) Delete remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/r/c/s/q> q
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Google if you use auto config mode.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall, or use
|
|
manual mode.
|
|
.PP
|
|
You can then use it like this,
|
|
.PP
|
|
List directories in top level of your put.io
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your put.io
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to a put.io directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to putio (Put.io).
|
|
.SS --putio-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_PUTIO_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
|
|
.SH Seafile
|
|
.PP
|
|
This is a backend for the Seafile (https://www.seafile.com/) storage
|
|
service: - It works with both the free community edition or the
|
|
professional edition.
|
|
- Seafile versions 6.x and 7.x are all supported.
|
|
- Encrypted libraries are also supported.
|
|
- It supports 2FA enabled users
|
|
.SS Configuration
|
|
.PP
|
|
There are two distinct modes you can setup your remote: - you point your
|
|
remote to the \f[B]root of the server\f[R], meaning you don\[aq]t
|
|
specify a library during the configuration: Paths are specified as
|
|
\f[C]remote:library\f[R].
|
|
You may put subdirectories in too, e.g.
|
|
\f[C]remote:library/path/to/dir\f[R].
|
|
- you point your remote to a specific library during the configuration:
|
|
Paths are specified as \f[C]remote:path/to/dir\f[R].
|
|
\f[B]This is the recommended mode when using encrypted libraries\f[R].
|
|
(\f[I]This mode is possibly slightly faster than the root mode\f[R])
|
|
.SS Configuration in root mode
|
|
.PP
|
|
Here is an example of making a seafile configuration for a user with
|
|
\f[B]no\f[R] two-factor authentication.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
To authenticate you will need the URL of your server, your email (or
|
|
username) and your password.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> seafile
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Seafile
|
|
\[rs] \[dq]seafile\[dq]
|
|
[snip]
|
|
Storage> seafile
|
|
** See help for seafile backend at: https://rclone.org/seafile/ **
|
|
|
|
URL of seafile host to connect to
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Connect to cloud.seafile.com
|
|
\[rs] \[dq]https://cloud.seafile.com/\[dq]
|
|
url> http://my.seafile.server/
|
|
User name (usually email address)
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
user> me\[at]example.com
|
|
Password
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
n) No leave this optional password blank (default)
|
|
y/g> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
Two-factor authentication (\[aq]true\[aq] if the account has 2FA enabled)
|
|
Enter a boolean value (true or false). Press Enter for the default (\[dq]false\[dq]).
|
|
2fa> false
|
|
Name of the library. Leave blank to access all non-encrypted libraries.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
library>
|
|
Library password (for encrypted libraries only). Leave blank if you pass it through the command line.
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
n) No leave this optional password blank (default)
|
|
y/g/n> n
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
Remote config
|
|
Two-factor authentication is not enabled on this account.
|
|
--------------------
|
|
[seafile]
|
|
type = seafile
|
|
url = http://my.seafile.server/
|
|
user = me\[at]example.com
|
|
pass = *** ENCRYPTED ***
|
|
2fa = false
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]seafile\f[R].
|
|
It\[aq]s pointing to the root of your seafile server and can now be used
|
|
like this:
|
|
.PP
|
|
See all libraries
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd seafile:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Create a new library
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir seafile:library
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a library
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls seafile:library
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote library, deleting any
|
|
excess files in the library.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory seafile:library
|
|
\f[R]
|
|
.fi
|
|
.SS Configuration in library mode
|
|
.PP
|
|
Here\[aq]s an example of a configuration in library mode with a user
|
|
that has the two-factor authentication enabled.
|
|
Your 2FA code will be asked at the end of the configuration, and will
|
|
attempt to authenticate you:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> seafile
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Seafile
|
|
\[rs] \[dq]seafile\[dq]
|
|
[snip]
|
|
Storage> seafile
|
|
** See help for seafile backend at: https://rclone.org/seafile/ **
|
|
|
|
URL of seafile host to connect to
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Connect to cloud.seafile.com
|
|
\[rs] \[dq]https://cloud.seafile.com/\[dq]
|
|
url> http://my.seafile.server/
|
|
User name (usually email address)
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
user> me\[at]example.com
|
|
Password
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
n) No leave this optional password blank (default)
|
|
y/g> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
Two-factor authentication (\[aq]true\[aq] if the account has 2FA enabled)
|
|
Enter a boolean value (true or false). Press Enter for the default (\[dq]false\[dq]).
|
|
2fa> true
|
|
Name of the library. Leave blank to access all non-encrypted libraries.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
library> My Library
|
|
Library password (for encrypted libraries only). Leave blank if you pass it through the command line.
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
n) No leave this optional password blank (default)
|
|
y/g/n> n
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
Remote config
|
|
Two-factor authentication: please enter your 2FA code
|
|
2fa code> 123456
|
|
Authenticating...
|
|
Success!
|
|
--------------------
|
|
[seafile]
|
|
type = seafile
|
|
url = http://my.seafile.server/
|
|
user = me\[at]example.com
|
|
pass =
|
|
2fa = true
|
|
library = My Library
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
You\[aq]ll notice your password is blank in the configuration.
|
|
It\[aq]s because we only need the password to authenticate you once.
|
|
.PP
|
|
You specified \f[C]My Library\f[R] during the configuration.
|
|
The root of the remote is pointing at the root of the library
|
|
\f[C]My Library\f[R]:
|
|
.PP
|
|
See all files in the library:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd seafile:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Create a new directory inside the library
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir seafile:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls seafile:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote library, deleting any
|
|
excess files in the library.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory seafile:
|
|
\f[R]
|
|
.fi
|
|
.SS --fast-list
|
|
.PP
|
|
Seafile version 7+ supports \f[C]--fast-list\f[R] which allows you to
|
|
use fewer transactions in exchange for more memory.
|
|
See the rclone docs (https://rclone.org/docs/#fast-list) for more
|
|
details.
|
|
Please note this is not supported on seafile server version 6.x
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
T{
|
|
\[dq]
|
|
T}@T{
|
|
0x22
|
|
T}@T{
|
|
\[uFF02]
|
|
T}
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Seafile and rclone link
|
|
.PP
|
|
Rclone supports generating share links for non-encrypted libraries only.
|
|
They can either be for a file or a directory:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone link seafile:seafile-tutorial.doc
|
|
http://my.seafile.server/f/fdcd8a2f93f84b8b90f4/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
or if run on a directory you will get:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone link seafile:dir
|
|
http://my.seafile.server/d/9ea2455f6f55478bbb0d/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Please note a share link is unique for each file or directory.
|
|
If you run a link command on a file/dir that has already been shared,
|
|
you will get the exact same link.
|
|
.SS Compatibility
|
|
.PP
|
|
It has been actively tested using the seafile docker
|
|
image (https://github.com/haiwen/seafile-docker) of these versions: -
|
|
6.3.4 community edition - 7.0.5 community edition - 7.1.3 community
|
|
edition
|
|
.PP
|
|
Versions below 6.0 are not supported.
|
|
Versions between 6.0 and 6.3 haven\[aq]t been tested and might not work
|
|
properly.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to seafile (seafile).
|
|
.SS --seafile-url
|
|
.PP
|
|
URL of seafile host to connect to.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SEAFILE_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]https://cloud.seafile.com/\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Connect to cloud.seafile.com.
|
|
.RE
|
|
.RE
|
|
.SS --seafile-user
|
|
.PP
|
|
User name (usually email address).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SEAFILE_USER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --seafile-pass
|
|
.PP
|
|
Password.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: pass
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SEAFILE_PASS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --seafile-2fa
|
|
.PP
|
|
Two-factor authentication (\[aq]true\[aq] if the account has 2FA
|
|
enabled).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: 2fa
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SEAFILE_2FA
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --seafile-library
|
|
.PP
|
|
Name of the library.
|
|
.PP
|
|
Leave blank to access all non-encrypted libraries.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: library
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SEAFILE_LIBRARY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --seafile-library-key
|
|
.PP
|
|
Library password (for encrypted libraries only).
|
|
.PP
|
|
Leave blank if you pass it through the command line.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: library_key
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SEAFILE_LIBRARY_KEY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --seafile-auth-token
|
|
.PP
|
|
Authentication token.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SEAFILE_AUTH_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to seafile (seafile).
|
|
.SS --seafile-create-library
|
|
.PP
|
|
Should rclone create a library if it doesn\[aq]t exist.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: create_library
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SEAFILE_CREATE_LIBRARY
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --seafile-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SEAFILE_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8
|
|
.SH SFTP
|
|
.PP
|
|
SFTP is the Secure (or SSH) File Transfer
|
|
Protocol (https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).
|
|
.PP
|
|
The SFTP backend can be used with a number of different providers:
|
|
.IP \[bu] 2
|
|
C14
|
|
.IP \[bu] 2
|
|
rsync.net
|
|
.PP
|
|
SFTP runs over SSH v2 and is installed as standard with most modern SSH
|
|
installations.
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R].
|
|
If the path does not begin with a \f[C]/\f[R] it is relative to the home
|
|
directory of the user.
|
|
An empty path \f[C]remote:\f[R] refers to the user\[aq]s home directory.
|
|
For example, \f[C]rclone lsd remote:\f[R] would list the home directory
|
|
of the user cofigured in the rclone remote config
|
|
(\f[C]i.e /home/sftpuser\f[R]).
|
|
However, \f[C]rclone lsd remote:/\f[R] would list the root directory for
|
|
remote machine (i.e.
|
|
\f[C]/\f[R])
|
|
.PP
|
|
Note that some SFTP servers will need the leading / - Synology is a good
|
|
example of this.
|
|
rsync.net, on the other hand, requires users to OMIT the leading /.
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of making an SFTP configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / SSH/SFTP Connection
|
|
\[rs] \[dq]sftp\[dq]
|
|
[snip]
|
|
Storage> sftp
|
|
SSH host to connect to
|
|
Choose a number from below, or type in your own value
|
|
1 / Connect to example.com
|
|
\[rs] \[dq]example.com\[dq]
|
|
host> example.com
|
|
SSH username
|
|
Enter a string value. Press Enter for the default (\[dq]$USER\[dq]).
|
|
user> sftpuser
|
|
SSH port number
|
|
Enter a signed integer. Press Enter for the default (22).
|
|
port>
|
|
SSH password, leave blank to use ssh-agent.
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
n) No leave this optional password blank
|
|
y/g/n> n
|
|
Path to unencrypted PEM-encoded private key file, leave blank to use ssh-agent.
|
|
key_file>
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
host = example.com
|
|
user = sftpuser
|
|
port =
|
|
pass =
|
|
key_file =
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[R] and can now be used like this:
|
|
.PP
|
|
See all directories in the home directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See all directories in the root directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Make a new directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:path/to/directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:path/to/directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote directory, deleting
|
|
any excess files in the directory.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory remote:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Mount the remote path \f[C]/srv/www-data/\f[R] to the local path
|
|
\f[C]/mnt/www-data\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mount remote:/srv/www-data/ /mnt/www-data
|
|
\f[R]
|
|
.fi
|
|
.SS SSH Authentication
|
|
.PP
|
|
The SFTP remote supports three authentication methods:
|
|
.IP \[bu] 2
|
|
Password
|
|
.IP \[bu] 2
|
|
Key file, including certificate signed keys
|
|
.IP \[bu] 2
|
|
ssh-agent
|
|
.PP
|
|
Key files should be PEM-encoded private key files.
|
|
For instance \f[C]/home/$USER/.ssh/id_rsa\f[R].
|
|
Only unencrypted OpenSSH or PEM encrypted files are supported.
|
|
.PP
|
|
The key file can be specified in either an external file (key_file) or
|
|
contained within the rclone config file (key_pem).
|
|
If using key_pem in the config file, the entry should be on a single
|
|
line with new line (\[aq]\[aq] or \[aq]\[aq]) separating lines.
|
|
i.e.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
key_pem = -----BEGIN RSA PRIVATE KEY-----\[rs]nMaMbaIXtE\[rs]n0gAMbMbaSsd\[rs]nMbaass\[rs]n-----END RSA PRIVATE KEY-----
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will generate it correctly for key_pem for use in the config:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
awk \[aq]{printf \[dq]%s\[rs]\[rs]n\[dq], $0}\[aq] < \[ti]/.ssh/id_rsa
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you don\[aq]t specify \f[C]pass\f[R], \f[C]key_file\f[R], or
|
|
\f[C]key_pem\f[R] or \f[C]ask_password\f[R] then rclone will attempt to
|
|
contact an ssh-agent.
|
|
You can also specify \f[C]key_use_agent\f[R] to force the usage of an
|
|
ssh-agent.
|
|
In this case \f[C]key_file\f[R] or \f[C]key_pem\f[R] can also be
|
|
specified to force the usage of a specific key in the ssh-agent.
|
|
.PP
|
|
Using an ssh-agent is the only way to load encrypted OpenSSH keys at the
|
|
moment.
|
|
.PP
|
|
If you set the \f[C]ask_password\f[R] option, rclone will prompt for a
|
|
password when needed and no password has been configured.
|
|
.SS Certificate-signed keys
|
|
.PP
|
|
With traditional key-based authentication, you configure your private
|
|
key only, and the public key built into it will be used during the
|
|
authentication process.
|
|
.PP
|
|
If you have a certificate you may use it to sign your public key,
|
|
creating a separate SSH user certificate that should be used instead of
|
|
the plain public key extracted from the private key.
|
|
Then you must provide the path to the user certificate public key file
|
|
in \f[C]pubkey_file\f[R].
|
|
.PP
|
|
Note: This is not the traditional public key paired with your private
|
|
key, typically saved as \f[C]/home/$USER/.ssh/id_rsa.pub\f[R].
|
|
Setting this path in \f[C]pubkey_file\f[R] will not work.
|
|
.PP
|
|
Example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[remote]
|
|
type = sftp
|
|
host = example.com
|
|
user = sftpuser
|
|
key_file = \[ti]/id_rsa
|
|
pubkey_file = \[ti]/id_rsa-cert.pub
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you concatenate a cert with a private key then you can specify the
|
|
merged file in both places.
|
|
.PP
|
|
Note: the cert must come first in the file.
|
|
e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
cat id_rsa-cert.pub id_rsa > merged_key
|
|
\f[R]
|
|
.fi
|
|
.SS Host key validation
|
|
.PP
|
|
By default rclone will not check the server\[aq]s host key for
|
|
validation.
|
|
This can allow an attacker to replace a server with their own and if you
|
|
use password authentication then this can lead to that password being
|
|
exposed.
|
|
.PP
|
|
Host key matching, using standard \f[C]known_hosts\f[R] files can be
|
|
turned on by enabling the \f[C]known_hosts_file\f[R] option.
|
|
This can point to the file maintained by \f[C]OpenSSH\f[R] or can point
|
|
to a unique file.
|
|
.PP
|
|
e.g.
|
|
using the OpenSSH \f[C]known_hosts\f[R] file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[remote]
|
|
type = sftp
|
|
host = example.com
|
|
user = sftpuser
|
|
pass =
|
|
known_hosts_file = \[ti]/.ssh/known_hosts
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Alternatively you can create your own known hosts file like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
ssh-keyscan -t dsa,rsa,ecdsa,ed25519 example.com >> known_hosts
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
There are some limitations:
|
|
.IP \[bu] 2
|
|
\f[C]rclone\f[R] will not \f[I]manage\f[R] this file for you.
|
|
If the key is missing or wrong then the connection will be refused.
|
|
.IP \[bu] 2
|
|
If the server is set up for a certificate host key then the entry in the
|
|
\f[C]known_hosts\f[R] file \f[I]must\f[R] be the
|
|
\f[C]\[at]cert-authority\f[R] entry for the CA
|
|
.PP
|
|
If the host key provided by the server does not match the one in the
|
|
file (or is missing) then the connection will be aborted and an error
|
|
returned such as
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
NewFs: couldn\[aq]t connect SSH: ssh: handshake failed: knownhosts: key mismatch
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
or
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
NewFs: couldn\[aq]t connect SSH: ssh: handshake failed: knownhosts: key is unknown
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you see an error such as
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
NewFs: couldn\[aq]t connect SSH: ssh: handshake failed: ssh: no authorities for hostname: example.com:22
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
then it is likely the server has presented a CA signed host certificate
|
|
and you will need to add the appropriate \f[C]\[at]cert-authority\f[R]
|
|
entry.
|
|
.PP
|
|
The \f[C]known_hosts_file\f[R] setting can be set during
|
|
\f[C]rclone config\f[R] as an advanced option.
|
|
.SS ssh-agent on macOS
|
|
.PP
|
|
Note that there seem to be various problems with using an ssh-agent on
|
|
macOS due to recent changes in the OS.
|
|
The most effective work-around seems to be to start an ssh-agent in each
|
|
session, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
eval \[ga]ssh-agent -s\[ga] && ssh-add -A
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And then at the end of the session
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
eval \[ga]ssh-agent -k\[ga]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
These commands can be used in scripts of course.
|
|
.SS Modified time
|
|
.PP
|
|
Modified times are stored on the server to 1 second precision.
|
|
.PP
|
|
Modified times are used in syncing and are fully supported.
|
|
.PP
|
|
Some SFTP servers disable setting/modifying the file modification time
|
|
after upload (for example, certain configurations of ProFTPd with
|
|
mod_sftp).
|
|
If you are using one of these servers, you can set the option
|
|
\f[C]set_modtime = false\f[R] in your RClone backend configuration to
|
|
disable this behaviour.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to sftp (SSH/SFTP Connection).
|
|
.SS --sftp-host
|
|
.PP
|
|
SSH host to connect to.
|
|
.PP
|
|
E.g.
|
|
\[dq]example.com\[dq].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: host
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_HOST
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --sftp-user
|
|
.PP
|
|
SSH username.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_USER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]$USER\[dq]
|
|
.SS --sftp-port
|
|
.PP
|
|
SSH port number.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: port
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_PORT
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 22
|
|
.SS --sftp-pass
|
|
.PP
|
|
SSH password, leave blank to use ssh-agent.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: pass
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_PASS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sftp-key-pem
|
|
.PP
|
|
Raw PEM-encoded private key.
|
|
.PP
|
|
If specified, will override key_file parameter.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: key_pem
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_KEY_PEM
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sftp-key-file
|
|
.PP
|
|
Path to PEM-encoded private key file.
|
|
.PP
|
|
Leave blank or set key-use-agent to use ssh-agent.
|
|
.PP
|
|
Leading \f[C]\[ti]\f[R] will be expanded in the file name as will
|
|
environment variables such as \f[C]${RCLONE_CONFIG_DIR}\f[R].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: key_file
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_KEY_FILE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sftp-key-file-pass
|
|
.PP
|
|
The passphrase to decrypt the PEM-encoded private key file.
|
|
.PP
|
|
Only PEM encrypted key files (old OpenSSH format) are supported.
|
|
Encrypted keys in the new OpenSSH format can\[aq]t be used.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: key_file_pass
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_KEY_FILE_PASS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sftp-pubkey-file
|
|
.PP
|
|
Optional path to public key file.
|
|
.PP
|
|
Set this if you have a signed certificate you want to use for
|
|
authentication.
|
|
.PP
|
|
Leading \f[C]\[ti]\f[R] will be expanded in the file name as will
|
|
environment variables such as \f[C]${RCLONE_CONFIG_DIR}\f[R].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: pubkey_file
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_PUBKEY_FILE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sftp-key-use-agent
|
|
.PP
|
|
When set forces the usage of the ssh-agent.
|
|
.PP
|
|
When key-file is also set, the \[dq].pub\[dq] file of the specified
|
|
key-file is read and only the associated key is requested from the
|
|
ssh-agent.
|
|
This allows to avoid
|
|
\f[C]Too many authentication failures for *username*\f[R] errors when
|
|
the ssh-agent contains many keys.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: key_use_agent
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_KEY_USE_AGENT
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --sftp-use-insecure-cipher
|
|
.PP
|
|
Enable the use of insecure ciphers and key exchange methods.
|
|
.PP
|
|
This enables the use of the following insecure ciphers and key exchange
|
|
methods:
|
|
.IP \[bu] 2
|
|
aes128-cbc
|
|
.IP \[bu] 2
|
|
aes192-cbc
|
|
.IP \[bu] 2
|
|
aes256-cbc
|
|
.IP \[bu] 2
|
|
3des-cbc
|
|
.IP \[bu] 2
|
|
diffie-hellman-group-exchange-sha256
|
|
.IP \[bu] 2
|
|
diffie-hellman-group-exchange-sha1
|
|
.PP
|
|
Those algorithms are insecure and may allow plaintext data to be
|
|
recovered by an attacker.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: use_insecure_cipher
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_USE_INSECURE_CIPHER
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]false\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use default Cipher list.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]true\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enables the use of the aes128-cbc cipher and
|
|
diffie-hellman-group-exchange-sha256, diffie-hellman-group-exchange-sha1
|
|
key exchange.
|
|
.RE
|
|
.RE
|
|
.SS --sftp-disable-hashcheck
|
|
.PP
|
|
Disable the execution of SSH commands to determine if remote file
|
|
hashing is available.
|
|
.PP
|
|
Leave blank or set to false to enable hashing (recommended), set to true
|
|
to disable hashing.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_hashcheck
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_DISABLE_HASHCHECK
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to sftp (SSH/SFTP Connection).
|
|
.SS --sftp-known-hosts-file
|
|
.PP
|
|
Optional path to known_hosts file.
|
|
.PP
|
|
Set this value to enable server host key validation.
|
|
.PP
|
|
Leading \f[C]\[ti]\f[R] will be expanded in the file name as will
|
|
environment variables such as \f[C]${RCLONE_CONFIG_DIR}\f[R].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: known_hosts_file
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_KNOWN_HOSTS_FILE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]\[ti]/.ssh/known_hosts\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use OpenSSH\[aq]s known_hosts file.
|
|
.RE
|
|
.RE
|
|
.SS --sftp-ask-password
|
|
.PP
|
|
Allow asking for SFTP password when needed.
|
|
.PP
|
|
If this is set and no password is supplied then rclone will: - ask for a
|
|
password - not contact the ssh agent
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: ask_password
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_ASK_PASSWORD
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --sftp-path-override
|
|
.PP
|
|
Override path used by SSH connection.
|
|
.PP
|
|
This allows checksum calculation when SFTP and SSH paths are different.
|
|
This issue affects among others Synology NAS boxes.
|
|
.PP
|
|
Shared folders can be found in directories representing volumes
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync /home/local/directory remote:/directory --sftp-path-override /volume2/directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Home directory can be found in a shared folder called \[dq]home\[dq]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync /home/local/directory remote:/home/directory --sftp-path-override /volume1/homes/USER/directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: path_override
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_PATH_OVERRIDE
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sftp-set-modtime
|
|
.PP
|
|
Set the modified time on the remote if set.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: set_modtime
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_SET_MODTIME
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: true
|
|
.SS --sftp-md5sum-command
|
|
.PP
|
|
The command used to read md5 hashes.
|
|
.PP
|
|
Leave blank for autodetect.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: md5sum_command
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_MD5SUM_COMMAND
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sftp-sha1sum-command
|
|
.PP
|
|
The command used to read sha1 hashes.
|
|
.PP
|
|
Leave blank for autodetect.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: sha1sum_command
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_SHA1SUM_COMMAND
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sftp-skip-links
|
|
.PP
|
|
Set to skip any symlinks and any other non regular files.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: skip_links
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_SKIP_LINKS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --sftp-subsystem
|
|
.PP
|
|
Specifies the SSH2 subsystem on the remote host.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: subsystem
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_SUBSYSTEM
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]sftp\[dq]
|
|
.SS --sftp-server-command
|
|
.PP
|
|
Specifies the path or command to run a sftp server on the remote host.
|
|
.PP
|
|
The subsystem option is ignored when server_command is defined.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: server_command
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_SERVER_COMMAND
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sftp-use-fstat
|
|
.PP
|
|
If set use fstat instead of stat.
|
|
.PP
|
|
Some servers limit the amount of open files and calling Stat after
|
|
opening the file will throw an error from the server.
|
|
Setting this flag will call Fstat instead of Stat which is called on an
|
|
already open file handle.
|
|
.PP
|
|
It has been found that this helps with IBM Sterling SFTP servers which
|
|
have \[dq]extractability\[dq] level set to 1 which means only 1 file can
|
|
be opened at any given time.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: use_fstat
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_USE_FSTAT
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --sftp-disable-concurrent-reads
|
|
.PP
|
|
If set don\[aq]t use concurrent reads.
|
|
.PP
|
|
Normally concurrent reads are safe to use and not using them will
|
|
degrade performance, so this option is disabled by default.
|
|
.PP
|
|
Some servers limit the amount number of times a file can be downloaded.
|
|
Using concurrent reads can trigger this limit, so if you have a server
|
|
which returns
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Failed to copy: file does not exist
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Then you may need to enable this flag.
|
|
.PP
|
|
If concurrent reads are disabled, the use_fstat option is ignored.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_concurrent_reads
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_DISABLE_CONCURRENT_READS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --sftp-disable-concurrent-writes
|
|
.PP
|
|
If set don\[aq]t use concurrent writes.
|
|
.PP
|
|
Normally rclone uses concurrent writes to upload files.
|
|
This improves the performance greatly, especially for distant servers.
|
|
.PP
|
|
This option disables concurrent writes should that be necessary.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: disable_concurrent_writes
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_DISABLE_CONCURRENT_WRITES
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --sftp-idle-timeout
|
|
.PP
|
|
Max time before closing idle connections.
|
|
.PP
|
|
If no connections have been returned to the connection pool in the time
|
|
given, rclone will empty the connection pool.
|
|
.PP
|
|
Set to 0 to keep connections indefinitely.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: idle_timeout
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SFTP_IDLE_TIMEOUT
|
|
.IP \[bu] 2
|
|
Type: Duration
|
|
.IP \[bu] 2
|
|
Default: 1m0s
|
|
.SS Limitations
|
|
.PP
|
|
SFTP supports checksums if the same login has shell access and
|
|
\f[C]md5sum\f[R] or \f[C]sha1sum\f[R] as well as \f[C]echo\f[R] are in
|
|
the remote\[aq]s PATH.
|
|
This remote checksumming (file hashing) is recommended and enabled by
|
|
default.
|
|
Disabling the checksumming may be required if you are connecting to SFTP
|
|
servers which are not under your control, and to which the execution of
|
|
remote commands is prohibited.
|
|
Set the configuration option \f[C]disable_hashcheck\f[R] to
|
|
\f[C]true\f[R] to disable checksumming.
|
|
.PP
|
|
SFTP also supports \f[C]about\f[R] if the same login has shell access
|
|
and \f[C]df\f[R] are in the remote\[aq]s PATH.
|
|
\f[C]about\f[R] will return the total space, free space, and used space
|
|
on the remote for the disk of the specified path on the remote or, if
|
|
not set, the disk of the root on the remote.
|
|
\f[C]about\f[R] will fail if it does not have shell access or if
|
|
\f[C]df\f[R] is not in the remote\[aq]s PATH.
|
|
.PP
|
|
Note that some SFTP servers (e.g.
|
|
Synology) the paths are different for SSH and SFTP so the hashes
|
|
can\[aq]t be calculated properly.
|
|
For them using \f[C]disable_hashcheck\f[R] is a good idea.
|
|
.PP
|
|
The only ssh agent supported under Windows is Putty\[aq]s pageant.
|
|
.PP
|
|
The Go SSH library disables the use of the aes128-cbc cipher by default,
|
|
due to security concerns.
|
|
This can be re-enabled on a per-connection basis by setting the
|
|
\f[C]use_insecure_cipher\f[R] setting in the configuration file to
|
|
\f[C]true\f[R].
|
|
Further details on the insecurity of this cipher can be found in this
|
|
paper (http://www.isg.rhul.ac.uk/~kp/SandPfinal.pdf).
|
|
.PP
|
|
SFTP isn\[aq]t supported under plan9 until this
|
|
issue (https://github.com/pkg/sftp/issues/156) is fixed.
|
|
.PP
|
|
Note that since SFTP isn\[aq]t HTTP based the following flags don\[aq]t
|
|
work with it: \f[C]--dump-headers\f[R], \f[C]--dump-bodies\f[R],
|
|
\f[C]--dump-auth\f[R]
|
|
.PP
|
|
Note that \f[C]--timeout\f[R] and \f[C]--contimeout\f[R] are both
|
|
supported.
|
|
.SS C14
|
|
.PP
|
|
C14 is supported through the SFTP backend.
|
|
.PP
|
|
See C14\[aq]s
|
|
documentation (https://www.online.net/en/storage/c14-cold-storage)
|
|
.SS rsync.net
|
|
.PP
|
|
rsync.net is supported through the SFTP backend.
|
|
.PP
|
|
See rsync.net\[aq]s documentation of rclone
|
|
examples (https://www.rsync.net/products/rclone.html).
|
|
.SH Storj
|
|
.PP
|
|
Storj (https://storj.io) is an encrypted, secure, and cost-effective
|
|
object storage service that enables you to store, back up, and archive
|
|
large amounts of data in a decentralized manner.
|
|
.SS Backend options
|
|
.PP
|
|
Storj can be used both with this native backend and with the s3 backend
|
|
using the Storj S3 compatible gateway (https://rclone.org/s3/#storj)
|
|
(shared or private).
|
|
.PP
|
|
Use this backend to take advantage of client-side encryption as well as
|
|
to achieve the best possible download performance.
|
|
Uploads will be erasure-coded locally, thus a 1gb upload will result in
|
|
2.68gb of data being uploaded to storage nodes across the network.
|
|
.PP
|
|
Use the s3 backend and one of the S3 compatible Hosted Gateways to
|
|
increase upload performance and reduce the load on your systems and
|
|
network.
|
|
Uploads will be encrypted and erasure-coded server-side, thus a 1GB
|
|
upload will result in only in 1GB of data being uploaded to storage
|
|
nodes across the network.
|
|
.PP
|
|
Side by side comparison with more details:
|
|
.IP \[bu] 2
|
|
Characteristics:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[I]Storj backend\f[R]: Uses native RPC protocol, connects directly to
|
|
the storage nodes which hosts the data.
|
|
Requires more CPU resource of encoding/decoding and has network
|
|
amplification (especially during the upload), uses lots of TCP
|
|
connections
|
|
.IP \[bu] 2
|
|
\f[I]S3 backend\f[R]: Uses S3 compatible HTTP Rest API via the shared
|
|
gateways.
|
|
There is no network amplification, but performance depends on the shared
|
|
gateways and the secret encryption key is shared with the gateway.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Typical usage:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[I]Storj backend\f[R]: Server environments and desktops with enough
|
|
resources, internet speed and connectivity - and applications where
|
|
storjs client-side encryption is required.
|
|
.IP \[bu] 2
|
|
\f[I]S3 backend\f[R]: Desktops and similar with limited resources,
|
|
internet speed or connectivity.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Security:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[I]Storj backend\f[R]: \f[B]strong\f[R].
|
|
Private encryption key doesn\[aq]t need to leave the local computer.
|
|
.IP \[bu] 2
|
|
\f[I]S3 backend\f[R]: \f[B]weaker\f[R].
|
|
Private encryption key is shared
|
|
with (https://docs.storj.io/dcs/api-reference/s3-compatible-gateway#security-and-encryption)
|
|
the authentication service of the hosted gateway, where it\[aq]s stored
|
|
encrypted.
|
|
It can be stronger when combining with the rclone crypt backend.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bandwidth usage (upload):
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[I]Storj backend\f[R]: \f[B]higher\f[R].
|
|
As data is erasure coded on the client side both the original data and
|
|
the parities should be uploaded.
|
|
About \[ti]2.7 times more data is required to be uploaded.
|
|
Client may start to upload with even higher number of nodes (\[ti]3.7
|
|
times more) and abandon/stop the slow uploads.
|
|
.IP \[bu] 2
|
|
\f[I]S3 backend\f[R]: \f[B]normal\f[R].
|
|
Only the raw data is uploaded, erasure coding happens on the gateway.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bandwidth usage (download)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[I]Storj backend\f[R]: \f[B]almost normal\f[R].
|
|
Only the minimal number of data is required, but to avoid very slow data
|
|
providers a few more sources are used and the slowest are ignored (max
|
|
1.2x overhead).
|
|
.IP \[bu] 2
|
|
\f[I]S3 backend\f[R]: \f[B]normal\f[R].
|
|
Only the raw data is downloaded, erasure coding happens on the shared
|
|
gateway.
|
|
.RE
|
|
.IP \[bu] 2
|
|
CPU usage:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[I]Storj backend\f[R]: \f[B]higher\f[R], but more predictable.
|
|
Erasure code and encryption/decryption happens locally which requires
|
|
significant CPU usage.
|
|
.IP \[bu] 2
|
|
\f[I]S3 backend\f[R]: \f[B]less\f[R].
|
|
Erasure code and encryption/decryption happens on shared s3 gateways
|
|
(and as is, it depends on the current load on the gateways)
|
|
.RE
|
|
.IP \[bu] 2
|
|
TCP connection usage:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[I]Storj backend\f[R]: \f[B]high\f[R].
|
|
A direct connection is required to each of the Storj nodes resulting in
|
|
110 connections on upload and 35 on download per 64 MB segment.
|
|
Not all the connections are actively used (slow ones are pruned), but
|
|
they are all opened.
|
|
Adjusting the max open file
|
|
limit (https://rclone.org/storj/#known-issues) may be required.
|
|
.IP \[bu] 2
|
|
\f[I]S3 backend\f[R]: \f[B]normal\f[R].
|
|
Only one connection per download/upload thread is required to the shared
|
|
gateway.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Overall performance:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[I]Storj backend\f[R]: with enough resources (CPU and bandwidth)
|
|
\f[I]storj\f[R] backend can provide even 2x better performance.
|
|
Data is directly downloaded to / uploaded from to the client instead of
|
|
the gateway.
|
|
.IP \[bu] 2
|
|
\f[I]S3 backend\f[R]: Can be faster on edge devices where CPU and
|
|
network bandwidth is limited as the shared S3 compatible gateways take
|
|
care about the encrypting/decryption and erasure coding and no
|
|
download/upload amplification.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Decentralization:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[I]Storj backend\f[R]: \f[B]high\f[R].
|
|
Data is downloaded directly from the distributed cloud of storage
|
|
providers.
|
|
.IP \[bu] 2
|
|
\f[I]S3 backend\f[R]: \f[B]low\f[R].
|
|
Requires a running S3 gateway (either self-hosted or Storj-hosted).
|
|
.RE
|
|
.IP \[bu] 2
|
|
Limitations:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[I]Storj backend\f[R]: \f[C]rclone checksum\f[R] is not possible
|
|
without download, as checksum metadata is not calculated during upload
|
|
.IP \[bu] 2
|
|
\f[I]S3 backend\f[R]: secret encryption key is shared with the gateway
|
|
.RE
|
|
.SS Configuration
|
|
.PP
|
|
To make a new Storj configuration you need one of the following: *
|
|
Access Grant that someone else shared with you.
|
|
* API
|
|
Key (https://documentation.storj.io/getting-started/uploading-your-first-object/create-an-api-key)
|
|
of a Storj project you are a member of.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.SS Setup with access grant
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Storj Decentralized Cloud Storage
|
|
\[rs] \[dq]storj\[dq]
|
|
[snip]
|
|
Storage> storj
|
|
** See help for storj backend at: https://rclone.org/storj/ **
|
|
|
|
Choose an authentication method.
|
|
Enter a string value. Press Enter for the default (\[dq]existing\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Use an existing access grant.
|
|
\[rs] \[dq]existing\[dq]
|
|
2 / Create a new access grant from satellite address, API key, and passphrase.
|
|
\[rs] \[dq]new\[dq]
|
|
provider> existing
|
|
Access Grant.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
access_grant> your-access-grant-received-by-someone-else
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
type = storj
|
|
access_grant = your-access-grant-received-by-someone-else
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.SS Setup with API key and passphrase
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Storj Decentralized Cloud Storage
|
|
\[rs] \[dq]storj\[dq]
|
|
[snip]
|
|
Storage> storj
|
|
** See help for storj backend at: https://rclone.org/storj/ **
|
|
|
|
Choose an authentication method.
|
|
Enter a string value. Press Enter for the default (\[dq]existing\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / Use an existing access grant.
|
|
\[rs] \[dq]existing\[dq]
|
|
2 / Create a new access grant from satellite address, API key, and passphrase.
|
|
\[rs] \[dq]new\[dq]
|
|
provider> new
|
|
Satellite Address. Custom satellite address should match the format: \[ga]<nodeid>\[at]<address>:<port>\[ga].
|
|
Enter a string value. Press Enter for the default (\[dq]us-central-1.storj.io\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
1 / US Central 1
|
|
\[rs] \[dq]us-central-1.storj.io\[dq]
|
|
2 / Europe West 1
|
|
\[rs] \[dq]europe-west-1.storj.io\[dq]
|
|
3 / Asia East 1
|
|
\[rs] \[dq]asia-east-1.storj.io\[dq]
|
|
satellite_address> 1
|
|
API Key.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
api_key> your-api-key-for-your-storj-project
|
|
Encryption Passphrase. To access existing objects enter passphrase used for uploading.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
passphrase> your-human-readable-encryption-passphrase
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
type = storj
|
|
satellite_address = 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S\[at]us-central-1.tardigrade.io:7777
|
|
api_key = your-api-key-for-your-storj-project
|
|
passphrase = your-human-readable-encryption-passphrase
|
|
access_grant = the-access-grant-generated-from-the-api-key-and-passphrase
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to storj (Storj Decentralized
|
|
Cloud Storage).
|
|
.SS --storj-provider
|
|
.PP
|
|
Choose an authentication method.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: provider
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_STORJ_PROVIDER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]existing\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]existing\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use an existing access grant.
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]new\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Create a new access grant from satellite address, API key, and
|
|
passphrase.
|
|
.RE
|
|
.RE
|
|
.SS --storj-access-grant
|
|
.PP
|
|
Access grant.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: access_grant
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_STORJ_ACCESS_GRANT
|
|
.IP \[bu] 2
|
|
Provider: existing
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --storj-satellite-address
|
|
.PP
|
|
Satellite address.
|
|
.PP
|
|
Custom satellite address should match the format:
|
|
\f[C]<nodeid>\[at]<address>:<port>\f[R].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: satellite_address
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_STORJ_SATELLITE_ADDRESS
|
|
.IP \[bu] 2
|
|
Provider: new
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]us-central-1.storj.io\[dq]
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]us-central-1.storj.io\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
US Central 1
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]europe-west-1.storj.io\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Europe West 1
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]asia-east-1.storj.io\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Asia East 1
|
|
.RE
|
|
.RE
|
|
.SS --storj-api-key
|
|
.PP
|
|
API key.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: api_key
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_STORJ_API_KEY
|
|
.IP \[bu] 2
|
|
Provider: new
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --storj-passphrase
|
|
.PP
|
|
Encryption passphrase.
|
|
.PP
|
|
To access existing objects enter passphrase used for uploading.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: passphrase
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_STORJ_PASSPHRASE
|
|
.IP \[bu] 2
|
|
Provider: new
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Usage
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[R] (or \f[C]remote:\f[R] for
|
|
the \f[C]lsf\f[R] command.) You may put subdirectories in too, e.g.
|
|
\f[C]remote:bucket/path/to/dir\f[R].
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this.
|
|
.SS Create a new bucket
|
|
.PP
|
|
Use the \f[C]mkdir\f[R] command to create new bucket, e.g.
|
|
\f[C]bucket\f[R].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.SS List all buckets
|
|
.PP
|
|
Use the \f[C]lsf\f[R] command to list all buckets.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsf remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note the colon (\f[C]:\f[R]) character at the end of the command line.
|
|
.SS Delete a bucket
|
|
.PP
|
|
Use the \f[C]rmdir\f[R] command to delete an empty bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone rmdir remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Use the \f[C]purge\f[R] command to delete a non-empty bucket with all
|
|
its content.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone purge remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.SS Upload objects
|
|
.PP
|
|
Use the \f[C]copy\f[R] command to upload an object.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --progress /home/local/directory/file.ext remote:bucket/path/to/dir/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The \f[C]--progress\f[R] flag is for displaying progress information.
|
|
Remove it if you don\[aq]t need this information.
|
|
.PP
|
|
Use a folder in the local path to upload all its objects.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --progress /home/local/directory/ remote:bucket/path/to/dir/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Only modified files will be copied.
|
|
.SS List objects
|
|
.PP
|
|
Use the \f[C]ls\f[R] command to list recursively all objects in a
|
|
bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:bucket
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Add the folder to the remote path to list recursively all objects in
|
|
this folder.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:bucket/path/to/dir/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Use the \f[C]lsf\f[R] command to list non-recursively all objects in a
|
|
bucket or a folder.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsf remote:bucket/path/to/dir/
|
|
\f[R]
|
|
.fi
|
|
.SS Download objects
|
|
.PP
|
|
Use the \f[C]copy\f[R] command to download an object.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --progress remote:bucket/path/to/dir/file.ext /home/local/directory/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The \f[C]--progress\f[R] flag is for displaying progress information.
|
|
Remove it if you don\[aq]t need this information.
|
|
.PP
|
|
Use a folder in the remote path to download all its objects.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy --progress remote:bucket/path/to/dir/ /home/local/directory/
|
|
\f[R]
|
|
.fi
|
|
.SS Delete objects
|
|
.PP
|
|
Use the \f[C]deletefile\f[R] command to delete a single object.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone deletefile remote:bucket/path/to/dir/file.ext
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Use the \f[C]delete\f[R] command to delete all object in a folder.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone delete remote:bucket/path/to/dir/
|
|
\f[R]
|
|
.fi
|
|
.SS Print the total size of objects
|
|
.PP
|
|
Use the \f[C]size\f[R] command to print the total size of objects in a
|
|
bucket or a folder.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone size remote:bucket/path/to/dir/
|
|
\f[R]
|
|
.fi
|
|
.SS Sync two Locations
|
|
.PP
|
|
Use the \f[C]sync\f[R] command to sync the source to the destination,
|
|
changing the destination only, deleting any excess files.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i --progress /home/local/directory/ remote:bucket/path/to/dir/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The \f[C]--progress\f[R] flag is for displaying progress information.
|
|
Remove it if you don\[aq]t need this information.
|
|
.PP
|
|
Since this can cause data loss, test first with the \f[C]--dry-run\f[R]
|
|
flag to see exactly what would be copied and deleted.
|
|
.PP
|
|
The sync can be done also from Storj to the local file system.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i --progress remote:bucket/path/to/dir/ /home/local/directory/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or between two Storj buckets.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i --progress remote-us:bucket/path/to/dir/ remote-europe:bucket/path/to/dir/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Or even between another cloud storage and Storj.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i --progress s3:bucket/path/to/dir/ storj:bucket/path/to/dir/
|
|
\f[R]
|
|
.fi
|
|
.SS Limitations
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the rclone Storj backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SS Known issues
|
|
.PP
|
|
If you get errors like \f[C]too many open files\f[R] this usually
|
|
happens when the default \f[C]ulimit\f[R] for system max open files is
|
|
exceeded.
|
|
Native Storj protocol opens a large number of TCP connections (each of
|
|
which is counted as an open file).
|
|
For a single upload stream you can expect 110 TCP connections to be
|
|
opened.
|
|
For a single download stream you can expect 35.
|
|
This batch of connections will be opened for every 64 MiB segment and
|
|
you should also expect TCP connections to be reused.
|
|
If you do many transfers you eventually open a connection to most
|
|
storage nodes (thousands of nodes).
|
|
.PP
|
|
To fix these, please raise your system limits.
|
|
You can do this issuing a \f[C]ulimit -n 65536\f[R] just before you run
|
|
rclone.
|
|
To change the limits more permanently you can add this to your shell
|
|
startup script, e.g.
|
|
\f[C]$HOME/.bashrc\f[R], or change the system-wide configuration,
|
|
usually \f[C]/etc/sysctl.conf\f[R] and/or
|
|
\f[C]/etc/security/limits.conf\f[R], but please refer to your operating
|
|
system manual.
|
|
.SH SugarSync
|
|
.PP
|
|
SugarSync (https://sugarsync.com) is a cloud service that enables active
|
|
synchronization of files across computers and other devices for file
|
|
backup, access, syncing, and sharing.
|
|
.SS Configuration
|
|
.PP
|
|
The initial setup for SugarSync involves getting a token from SugarSync
|
|
which you can do with rclone.
|
|
\f[C]rclone config\f[R] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Sugarsync
|
|
\[rs] \[dq]sugarsync\[dq]
|
|
[snip]
|
|
Storage> sugarsync
|
|
** See help for sugarsync backend at: https://rclone.org/sugarsync/ **
|
|
|
|
Sugarsync App ID.
|
|
Leave blank to use rclone\[aq]s.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
app_id>
|
|
Sugarsync Access Key ID.
|
|
Leave blank to use rclone\[aq]s.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
access_key_id>
|
|
Sugarsync Private Access Key
|
|
Leave blank to use rclone\[aq]s.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
private_access_key>
|
|
Permanently delete files if true
|
|
otherwise put them in the deleted files.
|
|
Enter a boolean value (true or false). Press Enter for the default (\[dq]false\[dq]).
|
|
hard_delete>
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
Remote config
|
|
Username (email address)> nick\[at]craig-wood.com
|
|
Your Sugarsync password is only required during setup and will not be stored.
|
|
password:
|
|
--------------------
|
|
[remote]
|
|
type = sugarsync
|
|
refresh_token = https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that the config asks for your email and password but doesn\[aq]t
|
|
store them, it only uses them to get the initial token.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories (sync folders) in top level of your SugarSync
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your SugarSync folder \[dq]Test\[dq]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:Test
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an SugarSync folder called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.PP
|
|
\f[B]NB\f[R] you can\[aq]t create files in the top level folder you have
|
|
to create a folder, which rclone will create as a \[dq]Sync Folder\[dq]
|
|
with SugarSync.
|
|
.SS Modified time and hashes
|
|
.PP
|
|
SugarSync does not support modification times or hashes, therefore
|
|
syncing will default to \f[C]--size-only\f[R] checking.
|
|
Note that using \f[C]--update\f[R] will work as rclone can read the time
|
|
files were uploaded.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
SugarSync replaces the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) except for DEL.
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in XML strings.
|
|
.SS Deleting files
|
|
.PP
|
|
Deleted files will be moved to the \[dq]Deleted items\[dq] folder by
|
|
default.
|
|
.PP
|
|
However you can supply the flag \f[C]--sugarsync-hard-delete\f[R] or set
|
|
the config parameter \f[C]hard_delete = true\f[R] if you would like
|
|
files to be deleted straight away.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to sugarsync (Sugarsync).
|
|
.SS --sugarsync-app-id
|
|
.PP
|
|
Sugarsync App ID.
|
|
.PP
|
|
Leave blank to use rclone\[aq]s.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: app_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SUGARSYNC_APP_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sugarsync-access-key-id
|
|
.PP
|
|
Sugarsync Access Key ID.
|
|
.PP
|
|
Leave blank to use rclone\[aq]s.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: access_key_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SUGARSYNC_ACCESS_KEY_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sugarsync-private-access-key
|
|
.PP
|
|
Sugarsync Private Access Key.
|
|
.PP
|
|
Leave blank to use rclone\[aq]s.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: private_access_key
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SUGARSYNC_PRIVATE_ACCESS_KEY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sugarsync-hard-delete
|
|
.PP
|
|
Permanently delete files if true otherwise put them in the deleted
|
|
files.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: hard_delete
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SUGARSYNC_HARD_DELETE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to sugarsync (Sugarsync).
|
|
.SS --sugarsync-refresh-token
|
|
.PP
|
|
Sugarsync refresh token.
|
|
.PP
|
|
Leave blank normally, will be auto configured by rclone.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: refresh_token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SUGARSYNC_REFRESH_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sugarsync-authorization
|
|
.PP
|
|
Sugarsync authorization.
|
|
.PP
|
|
Leave blank normally, will be auto configured by rclone.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: authorization
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SUGARSYNC_AUTHORIZATION
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sugarsync-authorization-expiry
|
|
.PP
|
|
Sugarsync authorization expiry.
|
|
.PP
|
|
Leave blank normally, will be auto configured by rclone.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: authorization_expiry
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SUGARSYNC_AUTHORIZATION_EXPIRY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sugarsync-user
|
|
.PP
|
|
Sugarsync user.
|
|
.PP
|
|
Leave blank normally, will be auto configured by rclone.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SUGARSYNC_USER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sugarsync-root-id
|
|
.PP
|
|
Sugarsync root id.
|
|
.PP
|
|
Leave blank normally, will be auto configured by rclone.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: root_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SUGARSYNC_ROOT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sugarsync-deleted-id
|
|
.PP
|
|
Sugarsync deleted folder id.
|
|
.PP
|
|
Leave blank normally, will be auto configured by rclone.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: deleted_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SUGARSYNC_DELETED_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --sugarsync-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_SUGARSYNC_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,Ctl,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by the SugarSync backend.
|
|
Backends without this capability cannot determine free space for an
|
|
rclone mount or use policy \f[C]mfs\f[R] (most free space) as a member
|
|
of an rclone union remote.
|
|
.PP
|
|
See List of backends that do not support rclone
|
|
about (https://rclone.org/overview/#optional-features) See rclone
|
|
about (https://rclone.org/commands/rclone_about/)
|
|
.SH Tardigrade
|
|
.PP
|
|
The Tardigrade backend has been renamed to be the Storj
|
|
backend (https://rclone.org/storj/).
|
|
Old configuration files will continue to work.
|
|
.SH Uptobox
|
|
.PP
|
|
This is a Backend for Uptobox file storage service.
|
|
Uptobox is closer to a one-click hoster than a traditional cloud storage
|
|
provider and therefore not suitable for long term storage.
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
To configure an Uptobox backend you\[aq]ll need your personal api token.
|
|
You\[aq]ll find it in your account
|
|
settings (https://uptobox.com/my_account)
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R] with
|
|
the default setup.
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Current remotes:
|
|
|
|
Name Type
|
|
==== ====
|
|
TestUptobox uptobox
|
|
|
|
e) Edit existing remote
|
|
n) New remote
|
|
d) Delete remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/r/c/s/q> n
|
|
name> uptobox
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[...]
|
|
37 / Uptobox
|
|
\[rs] \[dq]uptobox\[dq]
|
|
[...]
|
|
Storage> uptobox
|
|
** See help for uptobox backend at: https://rclone.org/uptobox/ **
|
|
|
|
Your API Key, get it from https://uptobox.com/my_account
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
api_key> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
Remote config
|
|
--------------------
|
|
[uptobox]
|
|
type = uptobox
|
|
api_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your Uptobox
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your Uptobox
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Uptobox directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
Uptobox supports neither modified times nor checksums.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
In addition to the default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) the following
|
|
characters are also replaced:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
\[dq]
|
|
T}@T{
|
|
0x22
|
|
T}@T{
|
|
\[uFF02]
|
|
T}
|
|
T{
|
|
\[ga]
|
|
T}@T{
|
|
0x41
|
|
T}@T{
|
|
\[uFF40]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in XML strings.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to uptobox (Uptobox).
|
|
.SS --uptobox-access-token
|
|
.PP
|
|
Your access token.
|
|
.PP
|
|
Get it from https://uptobox.com/my_account.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: access_token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_UPTOBOX_ACCESS_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to uptobox (Uptobox).
|
|
.SS --uptobox-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_UPTOBOX_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default:
|
|
Slash,LtGt,DoubleQuote,BackQuote,Del,Ctl,LeftSpace,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
Uptobox will delete inactive files that have not been accessed in 60
|
|
days.
|
|
.PP
|
|
\f[C]rclone about\f[R] is not supported by this backend an overview of
|
|
used space can however been seen in the uptobox web interface.
|
|
.SH Union
|
|
.PP
|
|
The \f[C]union\f[R] remote provides a unification similar to UnionFS
|
|
using other remotes.
|
|
.PP
|
|
Paths may be as deep as required or a local path, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R] or
|
|
\f[C]/directory/subdirectory\f[R].
|
|
.PP
|
|
During the initial setup with \f[C]rclone config\f[R] you will specify
|
|
the upstream remotes as a space separated list.
|
|
The upstream remotes can either be a local paths or other remotes.
|
|
.PP
|
|
Attribute \f[C]:ro\f[R] and \f[C]:nc\f[R] can be attach to the end of
|
|
path to tag the remote as \f[B]read only\f[R] or \f[B]no create\f[R],
|
|
e.g.
|
|
\f[C]remote:directory/subdirectory:ro\f[R] or
|
|
\f[C]remote:directory/subdirectory:nc\f[R].
|
|
.PP
|
|
Subfolders can be used in upstream remotes.
|
|
Assume a union remote named \f[C]backup\f[R] with the remotes
|
|
\f[C]mydrive:private/backup\f[R].
|
|
Invoking \f[C]rclone mkdir backup:desktop\f[R] is exactly the same as
|
|
invoking \f[C]rclone mkdir mydrive:private/backup/desktop\f[R].
|
|
.PP
|
|
There will be no special handling of paths containing \f[C]..\f[R]
|
|
segments.
|
|
Invoking \f[C]rclone mkdir backup:../desktop\f[R] is exactly the same as
|
|
invoking \f[C]rclone mkdir mydrive:private/backup/../desktop\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of how to make a union called \f[C]remote\f[R] for
|
|
local folders.
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Union merges the contents of several remotes
|
|
\[rs] \[dq]union\[dq]
|
|
[snip]
|
|
Storage> union
|
|
List of space separated upstreams.
|
|
Can be \[aq]upstreama:test/dir upstreamb:\[aq], \[aq]\[rs]\[dq]upstreama:test/space:ro dir\[rs]\[dq] upstreamb:\[aq], etc.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
upstreams> remote1:dir1 remote2:dir2 remote3:dir3
|
|
Policy to choose upstream on ACTION class.
|
|
Enter a string value. Press Enter for the default (\[dq]epall\[dq]).
|
|
action_policy>
|
|
Policy to choose upstream on CREATE class.
|
|
Enter a string value. Press Enter for the default (\[dq]epmfs\[dq]).
|
|
create_policy>
|
|
Policy to choose upstream on SEARCH class.
|
|
Enter a string value. Press Enter for the default (\[dq]ff\[dq]).
|
|
search_policy>
|
|
Cache time of usage and free space (in seconds). This option is only useful when a path preserving policy is used.
|
|
Enter a signed integer. Press Enter for the default (\[dq]120\[dq]).
|
|
cache_time>
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
type = union
|
|
upstreams = remote1:dir1 remote2:dir2 remote3:dir3
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
Current remotes:
|
|
|
|
Name Type
|
|
==== ====
|
|
remote union
|
|
|
|
e) Edit existing remote
|
|
n) New remote
|
|
d) Delete remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/r/c/s/q> q
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level in \f[C]remote1:dir1\f[R],
|
|
\f[C]remote2:dir2\f[R] and \f[C]remote3:dir3\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in \f[C]remote1:dir1\f[R], \f[C]remote2:dir2\f[R] and
|
|
\f[C]remote3:dir3\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Copy another local directory to the union directory called source, which
|
|
will be placed into \f[C]remote3:dir3\f[R]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy C:\[rs]source remote:source
|
|
\f[R]
|
|
.fi
|
|
.SS Behavior / Policies
|
|
.PP
|
|
The behavior of union backend is inspired by
|
|
trapexit/mergerfs (https://github.com/trapexit/mergerfs).
|
|
All functions are grouped into 3 categories: \f[B]action\f[R],
|
|
\f[B]create\f[R] and \f[B]search\f[R].
|
|
These functions and categories can be assigned a policy which dictates
|
|
what file or directory is chosen when performing that behavior.
|
|
Any policy can be assigned to a function or category though some may not
|
|
be very useful in practice.
|
|
For instance: \f[B]rand\f[R] (random) may be useful for file creation
|
|
(create) but could lead to very odd behavior if used for
|
|
\f[C]delete\f[R] if there were more than one copy of the file.
|
|
.SS Function / Category classifications
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
lw(5.8n) lw(15.0n) lw(49.2n).
|
|
T{
|
|
Category
|
|
T}@T{
|
|
Description
|
|
T}@T{
|
|
Functions
|
|
T}
|
|
_
|
|
T{
|
|
action
|
|
T}@T{
|
|
Writing Existing file
|
|
T}@T{
|
|
move, rmdir, rmdirs, delete, purge and copy, sync (as destination when
|
|
file exist)
|
|
T}
|
|
T{
|
|
create
|
|
T}@T{
|
|
Create non-existing file
|
|
T}@T{
|
|
copy, sync (as destination when file not exist)
|
|
T}
|
|
T{
|
|
search
|
|
T}@T{
|
|
Reading and listing file
|
|
T}@T{
|
|
ls, lsd, lsl, cat, md5sum, sha1sum and copy, sync (as source)
|
|
T}
|
|
T{
|
|
N/A
|
|
T}@T{
|
|
T}@T{
|
|
size, about
|
|
T}
|
|
.TE
|
|
.SS Path Preservation
|
|
.PP
|
|
Policies, as described below, are of two basic types.
|
|
\f[C]path preserving\f[R] and \f[C]non-path preserving\f[R].
|
|
.PP
|
|
All policies which start with \f[C]ep\f[R] (\f[B]epff\f[R],
|
|
\f[B]eplfs\f[R], \f[B]eplus\f[R], \f[B]epmfs\f[R], \f[B]eprand\f[R]) are
|
|
\f[C]path preserving\f[R].
|
|
\f[C]ep\f[R] stands for \f[C]existing path\f[R].
|
|
.PP
|
|
A path preserving policy will only consider upstreams where the relative
|
|
path being accessed already exists.
|
|
.PP
|
|
When using non-path preserving policies paths will be created in target
|
|
upstreams as necessary.
|
|
.SS Quota Relevant Policies
|
|
.PP
|
|
Some policies rely on quota information.
|
|
These policies should be used only if your upstreams support the
|
|
respective quota fields.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l l.
|
|
T{
|
|
Policy
|
|
T}@T{
|
|
Required Field
|
|
T}
|
|
_
|
|
T{
|
|
lfs, eplfs
|
|
T}@T{
|
|
Free
|
|
T}
|
|
T{
|
|
mfs, epmfs
|
|
T}@T{
|
|
Free
|
|
T}
|
|
T{
|
|
lus, eplus
|
|
T}@T{
|
|
Used
|
|
T}
|
|
T{
|
|
lno, eplno
|
|
T}@T{
|
|
Objects
|
|
T}
|
|
.TE
|
|
.PP
|
|
To check if your upstream supports the field, run
|
|
\f[C]rclone about remote: [flags]\f[R] and see if the required field
|
|
exists.
|
|
.SS Filters
|
|
.PP
|
|
Policies basically search upstream remotes and create a list of files /
|
|
paths for functions to work on.
|
|
The policy is responsible for filtering and sorting.
|
|
The policy type defines the sorting but filtering is mostly uniform as
|
|
described below.
|
|
.IP \[bu] 2
|
|
No \f[B]search\f[R] policies filter.
|
|
.IP \[bu] 2
|
|
All \f[B]action\f[R] policies will filter out remotes which are tagged
|
|
as \f[B]read-only\f[R].
|
|
.IP \[bu] 2
|
|
All \f[B]create\f[R] policies will filter out remotes which are tagged
|
|
\f[B]read-only\f[R] or \f[B]no-create\f[R].
|
|
.PP
|
|
If all remotes are filtered an error will be returned.
|
|
.SS Policy descriptions
|
|
.PP
|
|
The policies definition are inspired by
|
|
trapexit/mergerfs (https://github.com/trapexit/mergerfs) but not exactly
|
|
the same.
|
|
Some policy definition could be different due to the much larger latency
|
|
of remote file systems.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
lw(16.2n) lw(53.8n).
|
|
T{
|
|
Policy
|
|
T}@T{
|
|
Description
|
|
T}
|
|
_
|
|
T{
|
|
all
|
|
T}@T{
|
|
Search category: same as \f[B]epall\f[R].
|
|
Action category: same as \f[B]epall\f[R].
|
|
Create category: act on all upstreams.
|
|
T}
|
|
T{
|
|
epall (existing path, all)
|
|
T}@T{
|
|
Search category: Given this order configured, act on the first one found
|
|
where the relative path exists.
|
|
Action category: apply to all found.
|
|
Create category: act on all upstreams where the relative path exists.
|
|
T}
|
|
T{
|
|
epff (existing path, first found)
|
|
T}@T{
|
|
Act on the first one found, by the time upstreams reply, where the
|
|
relative path exists.
|
|
T}
|
|
T{
|
|
eplfs (existing path, least free space)
|
|
T}@T{
|
|
Of all the upstreams on which the relative path exists choose the one
|
|
with the least free space.
|
|
T}
|
|
T{
|
|
eplus (existing path, least used space)
|
|
T}@T{
|
|
Of all the upstreams on which the relative path exists choose the one
|
|
with the least used space.
|
|
T}
|
|
T{
|
|
eplno (existing path, least number of objects)
|
|
T}@T{
|
|
Of all the upstreams on which the relative path exists choose the one
|
|
with the least number of objects.
|
|
T}
|
|
T{
|
|
epmfs (existing path, most free space)
|
|
T}@T{
|
|
Of all the upstreams on which the relative path exists choose the one
|
|
with the most free space.
|
|
T}
|
|
T{
|
|
eprand (existing path, random)
|
|
T}@T{
|
|
Calls \f[B]epall\f[R] and then randomizes.
|
|
Returns only one upstream.
|
|
T}
|
|
T{
|
|
ff (first found)
|
|
T}@T{
|
|
Search category: same as \f[B]epff\f[R].
|
|
Action category: same as \f[B]epff\f[R].
|
|
Create category: Act on the first one found by the time upstreams reply.
|
|
T}
|
|
T{
|
|
lfs (least free space)
|
|
T}@T{
|
|
Search category: same as \f[B]eplfs\f[R].
|
|
Action category: same as \f[B]eplfs\f[R].
|
|
Create category: Pick the upstream with the least available free space.
|
|
T}
|
|
T{
|
|
lus (least used space)
|
|
T}@T{
|
|
Search category: same as \f[B]eplus\f[R].
|
|
Action category: same as \f[B]eplus\f[R].
|
|
Create category: Pick the upstream with the least used space.
|
|
T}
|
|
T{
|
|
lno (least number of objects)
|
|
T}@T{
|
|
Search category: same as \f[B]eplno\f[R].
|
|
Action category: same as \f[B]eplno\f[R].
|
|
Create category: Pick the upstream with the least number of objects.
|
|
T}
|
|
T{
|
|
mfs (most free space)
|
|
T}@T{
|
|
Search category: same as \f[B]epmfs\f[R].
|
|
Action category: same as \f[B]epmfs\f[R].
|
|
Create category: Pick the upstream with the most available free space.
|
|
T}
|
|
T{
|
|
newest
|
|
T}@T{
|
|
Pick the file / directory with the largest mtime.
|
|
T}
|
|
T{
|
|
rand (random)
|
|
T}@T{
|
|
Calls \f[B]all\f[R] and then randomizes.
|
|
Returns only one upstream.
|
|
T}
|
|
.TE
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to union (Union merges the
|
|
contents of several upstream fs).
|
|
.SS --union-upstreams
|
|
.PP
|
|
List of space separated upstreams.
|
|
.PP
|
|
Can be \[aq]upstreama:test/dir upstreamb:\[aq],
|
|
\[aq]\[dq]upstreama:test/space:ro dir\[dq] upstreamb:\[aq], etc.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: upstreams
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_UNION_UPSTREAMS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --union-action-policy
|
|
.PP
|
|
Policy to choose upstream on ACTION category.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: action_policy
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_UNION_ACTION_POLICY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]epall\[dq]
|
|
.SS --union-create-policy
|
|
.PP
|
|
Policy to choose upstream on CREATE category.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: create_policy
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_UNION_CREATE_POLICY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]epmfs\[dq]
|
|
.SS --union-search-policy
|
|
.PP
|
|
Policy to choose upstream on SEARCH category.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: search_policy
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_UNION_SEARCH_POLICY
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Default: \[dq]ff\[dq]
|
|
.SS --union-cache-time
|
|
.PP
|
|
Cache time of usage and free space (in seconds).
|
|
.PP
|
|
This option is only useful when a path preserving policy is used.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: cache_time
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_UNION_CACHE_TIME
|
|
.IP \[bu] 2
|
|
Type: int
|
|
.IP \[bu] 2
|
|
Default: 120
|
|
.SH WebDAV
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[R]
|
|
.PP
|
|
Paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
To configure the WebDAV remote you will need to have a URL for it, and a
|
|
username and password.
|
|
If you know what kind of system you are connecting to then rclone can
|
|
enable extra features.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[R].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
n/s/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Webdav
|
|
\[rs] \[dq]webdav\[dq]
|
|
[snip]
|
|
Storage> webdav
|
|
URL of http host to connect to
|
|
Choose a number from below, or type in your own value
|
|
1 / Connect to example.com
|
|
\[rs] \[dq]https://example.com\[dq]
|
|
url> https://example.com/remote.php/webdav/
|
|
Name of the Webdav site/service/software you are using
|
|
Choose a number from below, or type in your own value
|
|
1 / Nextcloud
|
|
\[rs] \[dq]nextcloud\[dq]
|
|
2 / Owncloud
|
|
\[rs] \[dq]owncloud\[dq]
|
|
3 / Sharepoint Online, authenticated by Microsoft account.
|
|
\[rs] \[dq]sharepoint\[dq]
|
|
4 / Sharepoint with NTLM authentication. Usually self-hosted or on-premises.
|
|
\[rs] \[dq]sharepoint-ntlm\[dq]
|
|
5 / Other site/service or software
|
|
\[rs] \[dq]other\[dq]
|
|
vendor> 1
|
|
User name
|
|
user> user
|
|
Password.
|
|
y) Yes type in my own password
|
|
g) Generate random password
|
|
n) No leave this optional password blank
|
|
y/g/n> y
|
|
Enter the password:
|
|
password:
|
|
Confirm the password:
|
|
password:
|
|
Bearer token instead of user/pass (e.g. a Macaroon)
|
|
bearer_token>
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
type = webdav
|
|
url = https://example.com/remote.php/webdav/
|
|
vendor = nextcloud
|
|
user = user
|
|
pass = *** ENCRYPTED ***
|
|
bearer_token =
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
List directories in top level of your WebDAV
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List all the files in your WebDAV
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an WebDAV directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone copy /home/source remote:backup
|
|
\f[R]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
Plain WebDAV does not support modified times.
|
|
However when used with Owncloud or Nextcloud rclone will support
|
|
modified times.
|
|
.PP
|
|
Likewise plain WebDAV does not support hashes, however when used with
|
|
Owncloud or Nextcloud rclone will support SHA1 and MD5 hashes.
|
|
Depending on the exact version of Owncloud or Nextcloud hashes may
|
|
appear on all objects, or only on objects which had a hash uploaded with
|
|
them.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to webdav (Webdav).
|
|
.SS --webdav-url
|
|
.PP
|
|
URL of http host to connect to.
|
|
.PP
|
|
E.g.
|
|
https://example.com.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_WEBDAV_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: true
|
|
.SS --webdav-vendor
|
|
.PP
|
|
Name of the Webdav site/service/software you are using.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: vendor
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_WEBDAV_VENDOR
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]nextcloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Nextcloud
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]owncloud\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Owncloud
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sharepoint\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Sharepoint Online, authenticated by Microsoft account
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]sharepoint-ntlm\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Sharepoint with NTLM authentication, usually self-hosted or on-premises
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]other\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Other site/service or software
|
|
.RE
|
|
.RE
|
|
.SS --webdav-user
|
|
.PP
|
|
User name.
|
|
.PP
|
|
In case NTLM authentication is used, the username should be in the
|
|
format \[aq]Domain\[aq].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: user
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_WEBDAV_USER
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --webdav-pass
|
|
.PP
|
|
Password.
|
|
.PP
|
|
\f[B]NB\f[R] Input to this must be obscured - see rclone
|
|
obscure (https://rclone.org/commands/rclone_obscure/).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: pass
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_WEBDAV_PASS
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --webdav-bearer-token
|
|
.PP
|
|
Bearer token instead of user/pass (e.g.
|
|
a Macaroon).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: bearer_token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_WEBDAV_BEARER_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to webdav (Webdav).
|
|
.SS --webdav-bearer-token-command
|
|
.PP
|
|
Command to run to get a bearer token.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: bearer_token_command
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_WEBDAV_BEARER_TOKEN_COMMAND
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --webdav-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Default encoding is
|
|
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Hash,Percent,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8
|
|
for sharepoint-ntlm or identity otherwise.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_WEBDAV_ENCODING
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --webdav-headers
|
|
.PP
|
|
Set HTTP headers for all transactions.
|
|
.PP
|
|
Use this to set additional HTTP headers for all transactions
|
|
.PP
|
|
The input format is comma separated list of key,value pairs.
|
|
Standard CSV encoding (https://godoc.org/encoding/csv) may be used.
|
|
.PP
|
|
For example, to set a Cookie use \[aq]Cookie,name=value\[aq], or
|
|
\[aq]\[dq]Cookie\[dq],\[dq]name=value\[dq]\[aq].
|
|
.PP
|
|
You can set multiple headers, e.g.
|
|
\[aq]\[dq]Cookie\[dq],\[dq]name=value\[dq],\[dq]Authorization\[dq],\[dq]xxx\[dq]\[aq].
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: headers
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_WEBDAV_HEADERS
|
|
.IP \[bu] 2
|
|
Type: CommaSepList
|
|
.IP \[bu] 2
|
|
Default:
|
|
.SS Provider notes
|
|
.PP
|
|
See below for notes on specific providers.
|
|
.SS Owncloud
|
|
.PP
|
|
Click on the settings cog in the bottom right of the page and this will
|
|
show the WebDAV URL that rclone needs in the config step.
|
|
It will look something like
|
|
\f[C]https://example.com/remote.php/webdav/\f[R].
|
|
.PP
|
|
Owncloud supports modified times using the \f[C]X-OC-Mtime\f[R] header.
|
|
.SS Nextcloud
|
|
.PP
|
|
This is configured in an identical way to Owncloud.
|
|
Note that Nextcloud initially did not support streaming of files
|
|
(\f[C]rcat\f[R]) whereas Owncloud did, but
|
|
this (https://github.com/nextcloud/nextcloud-snap/issues/365) seems to
|
|
be fixed as of 2020-11-27 (tested with rclone v1.53.1 and Nextcloud
|
|
Server v19).
|
|
.SS Sharepoint Online
|
|
.PP
|
|
Rclone can be used with Sharepoint provided by OneDrive for Business or
|
|
Office365 Education Accounts.
|
|
This feature is only needed for a few of these Accounts, mostly
|
|
Office365 Education ones.
|
|
These accounts are sometimes not verified by the domain owner
|
|
github#1975 (https://github.com/rclone/rclone/issues/1975)
|
|
.PP
|
|
This means that these accounts can\[aq]t be added using the official API
|
|
(other Accounts should work with the \[dq]onedrive\[dq] option).
|
|
However, it is possible to access them using webdav.
|
|
.PP
|
|
To use a sharepoint remote with rclone, add it like this: First, you
|
|
need to get your remote\[aq]s URL:
|
|
.IP \[bu] 2
|
|
Go here (https://onedrive.live.com/about/en-us/signin/) to open your
|
|
OneDrive or to sign in
|
|
.IP \[bu] 2
|
|
Now take a look at your address bar, the URL should look like this:
|
|
\f[C]https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/_layouts/15/onedrive.aspx\f[R]
|
|
.PP
|
|
You\[aq]ll only need this URL up to the email address.
|
|
After that, you\[aq]ll most likely want to add \[dq]/Documents\[dq].
|
|
That subdirectory contains the actual data stored on your OneDrive.
|
|
.PP
|
|
Add the remote to rclone like this: Configure the \f[C]url\f[R] as
|
|
\f[C]https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents\f[R]
|
|
and use your normal account email and password for \f[C]user\f[R] and
|
|
\f[C]pass\f[R].
|
|
If you have 2FA enabled, you have to generate an app password.
|
|
Set the \f[C]vendor\f[R] to \f[C]sharepoint\f[R].
|
|
.PP
|
|
Your config file should look like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[sharepoint]
|
|
type = webdav
|
|
url = https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents
|
|
vendor = sharepoint
|
|
user = YourEmailAddress
|
|
pass = encryptedpassword
|
|
\f[R]
|
|
.fi
|
|
.SS Sharepoint with NTLM Authentication
|
|
.PP
|
|
Use this option in case your (hosted) Sharepoint is not tied to OneDrive
|
|
accounts and uses NTLM authentication.
|
|
.PP
|
|
To get the \f[C]url\f[R] configuration, similarly to the above, first
|
|
navigate to the desired directory in your browser to get the URL, then
|
|
strip everything after the name of the opened directory.
|
|
.PP
|
|
Example: If the URL is:
|
|
https://example.sharepoint.com/sites/12345/Documents/Forms/AllItems.aspx
|
|
.PP
|
|
The configuration to use would be:
|
|
https://example.sharepoint.com/sites/12345/Documents
|
|
.PP
|
|
Set the \f[C]vendor\f[R] to \f[C]sharepoint-ntlm\f[R].
|
|
.PP
|
|
NTLM uses domain and user name combination for authentication, set
|
|
\f[C]user\f[R] to \f[C]DOMAIN\[rs]username\f[R].
|
|
.PP
|
|
Your config file should look like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[sharepoint]
|
|
type = webdav
|
|
url = https://[YOUR-DOMAIN]/some-path-to/Documents
|
|
vendor = sharepoint-ntlm
|
|
user = DOMAIN\[rs]user
|
|
pass = encryptedpassword
|
|
\f[R]
|
|
.fi
|
|
.SS Required Flags for SharePoint
|
|
.PP
|
|
As SharePoint does some special things with uploaded documents, you
|
|
won\[aq]t be able to use the documents size or the documents hash to
|
|
compare if a file has been changed since the upload / which file is
|
|
newer.
|
|
.PP
|
|
For Rclone calls copying files (especially Office files such as .docx,
|
|
\&.xlsx, etc.) from/to SharePoint (like copy, sync, etc.), you should
|
|
append these flags to ensure Rclone uses the \[dq]Last Modified\[dq]
|
|
datetime property to compare your documents:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
--ignore-size --ignore-checksum --update
|
|
\f[R]
|
|
.fi
|
|
.SS dCache
|
|
.PP
|
|
dCache is a storage system that supports many protocols and
|
|
authentication/authorisation schemes.
|
|
For WebDAV clients, it allows users to authenticate with username and
|
|
password (BASIC), X.509, Kerberos, and various bearer tokens, including
|
|
Macaroons (https://www.dcache.org/manuals/workshop-2017-05-29-Umea/000-Final/anupam_macaroons_v02.pdf)
|
|
and OpenID-Connect (https://en.wikipedia.org/wiki/OpenID_Connect) access
|
|
tokens.
|
|
.PP
|
|
Configure as normal using the \f[C]other\f[R] type.
|
|
Don\[aq]t enter a username or password, instead enter your Macaroon as
|
|
the \f[C]bearer_token\f[R].
|
|
.PP
|
|
The config will end up looking something like this.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[dcache]
|
|
type = webdav
|
|
url = https://dcache...
|
|
vendor = other
|
|
user =
|
|
pass =
|
|
bearer_token = your-macaroon
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
There is a
|
|
script (https://github.com/sara-nl/GridScripts/blob/master/get-macaroon)
|
|
that obtains a Macaroon from a dCache WebDAV endpoint, and creates an
|
|
rclone config file.
|
|
.PP
|
|
Macaroons may also be obtained from the dCacheView
|
|
web-browser/JavaScript client that comes with dCache.
|
|
.SS OpenID-Connect
|
|
.PP
|
|
dCache also supports authenticating with OpenID-Connect access tokens.
|
|
OpenID-Connect is a protocol (based on OAuth 2.0) that allows services
|
|
to identify users who have authenticated with some central service.
|
|
.PP
|
|
Support for OpenID-Connect in rclone is currently achieved using another
|
|
software package called
|
|
oidc-agent (https://github.com/indigo-dc/oidc-agent).
|
|
This is a command-line tool that facilitates obtaining an access token.
|
|
Once installed and configured, an access token is obtained by running
|
|
the \f[C]oidc-token\f[R] command.
|
|
The following example shows a (shortened) access token obtained from the
|
|
\f[I]XDC\f[R] OIDC Provider.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
paul\[at]celebrimbor:\[ti]$ oidc-token XDC
|
|
eyJraWQ[...]QFXDt0
|
|
paul\[at]celebrimbor:\[ti]$
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]Note\f[R] Before the \f[C]oidc-token\f[R] command will work, the
|
|
refresh token must be loaded into the oidc agent.
|
|
This is done with the \f[C]oidc-add\f[R] command (e.g.,
|
|
\f[C]oidc-add XDC\f[R]).
|
|
This is typically done once per login session.
|
|
Full details on this and how to register oidc-agent with your OIDC
|
|
Provider are provided in the oidc-agent
|
|
documentation (https://indigo-dc.gitbooks.io/oidc-agent/).
|
|
.PP
|
|
The rclone \f[C]bearer_token_command\f[R] configuration option is used
|
|
to fetch the access token from oidc-agent.
|
|
.PP
|
|
Configure as a normal WebDAV endpoint, using the \[aq]other\[aq] vendor,
|
|
leaving the username and password empty.
|
|
When prompted, choose to edit the advanced config and enter the command
|
|
to get a bearer token (e.g., \f[C]oidc-agent XDC\f[R]).
|
|
.PP
|
|
The following example config shows a WebDAV endpoint that uses
|
|
oidc-agent to supply an access token from the \f[I]XDC\f[R] OIDC
|
|
Provider.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[dcache]
|
|
type = webdav
|
|
url = https://dcache.example.org/
|
|
vendor = other
|
|
bearer_token_command = oidc-token XDC
|
|
\f[R]
|
|
.fi
|
|
.SH Yandex Disk
|
|
.PP
|
|
Yandex Disk (https://disk.yandex.com) is a cloud storage solution
|
|
created by Yandex (https://yandex.com).
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of making a yandex configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
n/s> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Yandex Disk
|
|
\[rs] \[dq]yandex\[dq]
|
|
[snip]
|
|
Storage> yandex
|
|
Yandex Client Id - leave blank normally.
|
|
client_id>
|
|
Yandex Client Secret - leave blank normally.
|
|
client_secret>
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes
|
|
n) No
|
|
y/n> y
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
--------------------
|
|
[remote]
|
|
client_id =
|
|
client_secret =
|
|
token = {\[dq]access_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]token_type\[dq]:\[dq]OAuth\[dq],\[dq]expiry\[dq]:\[dq]2016-12-29T12:27:11.362788025Z\[dq]}
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Yandex Disk.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[R] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
See top level directories
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Make a new directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote path, deleting any
|
|
excess files in the path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory remote:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Yandex paths may be as deep as required, e.g.
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Modified time
|
|
.PP
|
|
Modified times are supported and are stored accurate to 1 ns in custom
|
|
metadata called \f[C]rclone_modified\f[R] in RFC3339 with nanoseconds
|
|
format.
|
|
.SS MD5 checksums
|
|
.PP
|
|
MD5 checksums are natively supported by Yandex Disk.
|
|
.SS Emptying Trash
|
|
.PP
|
|
If you wish to empty your trash you can use the
|
|
\f[C]rclone cleanup remote:\f[R] command which will permanently delete
|
|
all your trashed files.
|
|
This command does not take any path arguments.
|
|
.SS Quota information
|
|
.PP
|
|
To view your current quota you can use the
|
|
\f[C]rclone about remote:\f[R] command which will display your usage
|
|
limit (quota) and the current usage.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
The default restricted characters
|
|
set (https://rclone.org/overview/#restricted-characters) are replaced.
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be used in JSON strings.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to yandex (Yandex Disk).
|
|
.SS --yandex-client-id
|
|
.PP
|
|
OAuth Client Id.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_YANDEX_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --yandex-client-secret
|
|
.PP
|
|
OAuth Client Secret.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_YANDEX_CLIENT_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to yandex (Yandex Disk).
|
|
.SS --yandex-token
|
|
.PP
|
|
OAuth Access Token as a JSON blob.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_YANDEX_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --yandex-auth-url
|
|
.PP
|
|
Auth server URL.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_YANDEX_AUTH_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --yandex-token-url
|
|
.PP
|
|
Token server url.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_YANDEX_TOKEN_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --yandex-hard-delete
|
|
.PP
|
|
Delete files permanently rather than putting them into the trash.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: hard_delete
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_YANDEX_HARD_DELETE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --yandex-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_YANDEX_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,Del,Ctl,InvalidUtf8,Dot
|
|
.SS Limitations
|
|
.PP
|
|
When uploading very large files (bigger than about 5 GiB) you will need
|
|
to increase the \f[C]--timeout\f[R] parameter.
|
|
This is because Yandex pauses (perhaps to calculate the MD5SUM for the
|
|
entire file) before returning confirmation that the file has been
|
|
uploaded.
|
|
The default handling of timeouts in rclone is to assume a 5 minute pause
|
|
is an error and close the connection - you\[aq]ll see
|
|
\f[C]net/http: timeout awaiting response headers\f[R] errors in the logs
|
|
if this is happening.
|
|
Setting the timeout to twice the max size of file in GiB should be
|
|
enough, so if you want to upload a 30 GiB file set a timeout of
|
|
\f[C]2 * 30 = 60m\f[R], that is \f[C]--timeout 60m\f[R].
|
|
.PP
|
|
Having a Yandex Mail account is mandatory to use the Yandex.Disk
|
|
subscription.
|
|
Token generation will work without a mail account, but Rclone won\[aq]t
|
|
be able to complete any actions.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[403 - DiskUnsupportedUserAccountTypeError] User account type is not supported.
|
|
\f[R]
|
|
.fi
|
|
.SH Zoho Workdrive
|
|
.PP
|
|
Zoho WorkDrive (https://www.zoho.com/workdrive/) is a cloud storage
|
|
solution created by Zoho (https://zoho.com).
|
|
.SS Configuration
|
|
.PP
|
|
Here is an example of making a zoho configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone config
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
s) Set configuration password
|
|
n/s> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Zoho
|
|
\[rs] \[dq]zoho\[dq]
|
|
[snip]
|
|
Storage> zoho
|
|
** See help for zoho backend at: https://rclone.org/zoho/ **
|
|
|
|
OAuth Client Id
|
|
Leave blank normally.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
client_id>
|
|
OAuth Client Secret
|
|
Leave blank normally.
|
|
Enter a string value. Press Enter for the default (\[dq]\[dq]).
|
|
client_secret>
|
|
Edit advanced config? (y/n)
|
|
y) Yes
|
|
n) No (default)
|
|
y/n> n
|
|
Remote config
|
|
Use auto config?
|
|
* Say Y if not sure
|
|
* Say N if you are working on a remote or headless machine
|
|
y) Yes (default)
|
|
n) No
|
|
y/n>
|
|
If your browser doesn\[aq]t open automatically go to the following link: http://127.0.0.1:53682/auth?state=LVn0IHzxej1ZkmQw31d0wQ
|
|
Log in and authorize rclone for access
|
|
Waiting for code...
|
|
Got code
|
|
Choose a number from below, or type in your own value
|
|
1 / MyTeam
|
|
\[rs] \[dq]4u28602177065ff22426787a6745dba8954eb\[dq]
|
|
Enter a Team ID> 1
|
|
Choose a number from below, or type in your own value
|
|
1 / General
|
|
\[rs] \[dq]4u2869d2aa6fca04f4f2f896b6539243b85b1\[dq]
|
|
Enter a Workspace ID> 1
|
|
--------------------
|
|
[remote]
|
|
type = zoho
|
|
token = {\[dq]access_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]token_type\[dq]:\[dq]Zoho-oauthtoken\[dq],\[dq]refresh_token\[dq]:\[dq]xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\[dq],\[dq]expiry\[dq]:\[dq]2020-10-12T00:54:52.370275223+02:00\[dq]}
|
|
root_folder_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
|
--------------------
|
|
y) Yes this is OK (default)
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d>
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Rclone runs a webserver on your local computer to collect the
|
|
authorization token from Zoho Workdrive.
|
|
This is only from the moment your browser is opened until the token is
|
|
returned.
|
|
The webserver runs on \f[C]http://127.0.0.1:53682/\f[R].
|
|
If local port \f[C]53682\f[R] is protected by a firewall you may need to
|
|
temporarily unblock the firewall to complete authorization.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[R] like this,
|
|
.PP
|
|
See top level directories
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone lsd remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Make a new directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone mkdir remote:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone ls remote:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[R] to the remote path, deleting any
|
|
excess files in the path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/local/directory remote:directory
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Zoho paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[R].
|
|
.SS Modified time
|
|
.PP
|
|
Modified times are currently not supported for Zoho Workdrive
|
|
.SS Checksums
|
|
.PP
|
|
No checksums are supported.
|
|
.SS Usage information
|
|
.PP
|
|
To view your current quota you can use the
|
|
\f[C]rclone about remote:\f[R] command which will display your current
|
|
usage.
|
|
.SS Restricted filename characters
|
|
.PP
|
|
Only control characters and invalid UTF-8 are replaced.
|
|
In addition most Unicode full-width characters are not supported at all
|
|
and will be removed from filenames during upload.
|
|
.SS Standard options
|
|
.PP
|
|
Here are the standard options specific to zoho (Zoho).
|
|
.SS --zoho-client-id
|
|
.PP
|
|
OAuth Client Id.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_id
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ZOHO_CLIENT_ID
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --zoho-client-secret
|
|
.PP
|
|
OAuth Client Secret.
|
|
.PP
|
|
Leave blank normally.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: client_secret
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ZOHO_CLIENT_SECRET
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --zoho-region
|
|
.PP
|
|
Zoho region to connect to.
|
|
.PP
|
|
You\[aq]ll have to use the region your organization is registered in.
|
|
If not sure use the same top level domain as you connect to in your
|
|
browser.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: region
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ZOHO_REGION
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]com\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
United states / Global
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]eu\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Europe
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]in\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
India
|
|
.RE
|
|
.IP \[bu] 2
|
|
\[dq]com.au\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Australia
|
|
.RE
|
|
.RE
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to zoho (Zoho).
|
|
.SS --zoho-token
|
|
.PP
|
|
OAuth Access Token as a JSON blob.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ZOHO_TOKEN
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --zoho-auth-url
|
|
.PP
|
|
Auth server URL.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: auth_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ZOHO_AUTH_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --zoho-token-url
|
|
.PP
|
|
Token server url.
|
|
.PP
|
|
Leave blank to use the provider defaults.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: token_url
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ZOHO_TOKEN_URL
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.SS --zoho-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_ZOHO_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Del,Ctl,InvalidUtf8
|
|
.SH Local Filesystem
|
|
.PP
|
|
Local paths are specified as normal filesystem paths, e.g.
|
|
\f[C]/path/to/wherever\f[R], so
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i /home/source /tmp/destination
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Will sync \f[C]/home/source\f[R] to \f[C]/tmp/destination\f[R].
|
|
.SS Configuration
|
|
.PP
|
|
For consistencies sake one can also configure a remote of type
|
|
\f[C]local\f[R] in the config file, and access the local filesystem
|
|
using rclone remote paths, e.g.
|
|
\f[C]remote:path/to/wherever\f[R], but it is probably easier not to.
|
|
.SS Modified time
|
|
.PP
|
|
Rclone reads and writes the modified time using an accuracy determined
|
|
by the OS.
|
|
Typically this is 1ns on Linux, 10 ns on Windows and 1 Second on OS X.
|
|
.SS Filenames
|
|
.PP
|
|
Filenames should be encoded in UTF-8 on disk.
|
|
This is the normal case for Windows and OS X.
|
|
.PP
|
|
There is a bit more uncertainty in the Linux world, but new
|
|
distributions will have UTF-8 encoded files names.
|
|
If you are using an old Linux filesystem with non UTF-8 file names (e.g.
|
|
latin1) then you can use the \f[C]convmv\f[R] tool to convert the
|
|
filesystem to UTF-8.
|
|
This tool is available in most distributions\[aq] package managers.
|
|
.PP
|
|
If an invalid (non-UTF8) filename is read, the invalid characters will
|
|
be replaced with a quoted representation of the invalid bytes.
|
|
The name \f[C]gro\[rs]xdf\f[R] will be transferred as
|
|
\f[C]gro\[u201B]DF\f[R].
|
|
\f[C]rclone\f[R] will emit a debug message in this case (use
|
|
\f[C]-v\f[R] to see), e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Local file system at .: Replacing invalid UTF-8 characters in \[dq]gro\[rs]xdf\[dq]
|
|
\f[R]
|
|
.fi
|
|
.SS Restricted characters
|
|
.PP
|
|
With the local backend, restrictions on the characters that are usable
|
|
in file or directory names depend on the operating system.
|
|
To check what rclone will replace by default on your system, run
|
|
\f[C]rclone help flags local-encoding\f[R].
|
|
.PP
|
|
On non Windows platforms the following characters are replaced when
|
|
handling file names.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
NUL
|
|
T}@T{
|
|
0x00
|
|
T}@T{
|
|
\[u2400]
|
|
T}
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
.TE
|
|
.PP
|
|
When running on Windows the following characters are replaced.
|
|
This list is based on the Windows file naming
|
|
conventions (https://docs.microsoft.com/de-de/windows/desktop/FileIO/naming-a-file#naming-conventions).
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
NUL
|
|
T}@T{
|
|
0x00
|
|
T}@T{
|
|
\[u2400]
|
|
T}
|
|
T{
|
|
SOH
|
|
T}@T{
|
|
0x01
|
|
T}@T{
|
|
\[u2401]
|
|
T}
|
|
T{
|
|
STX
|
|
T}@T{
|
|
0x02
|
|
T}@T{
|
|
\[u2402]
|
|
T}
|
|
T{
|
|
ETX
|
|
T}@T{
|
|
0x03
|
|
T}@T{
|
|
\[u2403]
|
|
T}
|
|
T{
|
|
EOT
|
|
T}@T{
|
|
0x04
|
|
T}@T{
|
|
\[u2404]
|
|
T}
|
|
T{
|
|
ENQ
|
|
T}@T{
|
|
0x05
|
|
T}@T{
|
|
\[u2405]
|
|
T}
|
|
T{
|
|
ACK
|
|
T}@T{
|
|
0x06
|
|
T}@T{
|
|
\[u2406]
|
|
T}
|
|
T{
|
|
BEL
|
|
T}@T{
|
|
0x07
|
|
T}@T{
|
|
\[u2407]
|
|
T}
|
|
T{
|
|
BS
|
|
T}@T{
|
|
0x08
|
|
T}@T{
|
|
\[u2408]
|
|
T}
|
|
T{
|
|
HT
|
|
T}@T{
|
|
0x09
|
|
T}@T{
|
|
\[u2409]
|
|
T}
|
|
T{
|
|
LF
|
|
T}@T{
|
|
0x0A
|
|
T}@T{
|
|
\[u240A]
|
|
T}
|
|
T{
|
|
VT
|
|
T}@T{
|
|
0x0B
|
|
T}@T{
|
|
\[u240B]
|
|
T}
|
|
T{
|
|
FF
|
|
T}@T{
|
|
0x0C
|
|
T}@T{
|
|
\[u240C]
|
|
T}
|
|
T{
|
|
CR
|
|
T}@T{
|
|
0x0D
|
|
T}@T{
|
|
\[u240D]
|
|
T}
|
|
T{
|
|
SO
|
|
T}@T{
|
|
0x0E
|
|
T}@T{
|
|
\[u240E]
|
|
T}
|
|
T{
|
|
SI
|
|
T}@T{
|
|
0x0F
|
|
T}@T{
|
|
\[u240F]
|
|
T}
|
|
T{
|
|
DLE
|
|
T}@T{
|
|
0x10
|
|
T}@T{
|
|
\[u2410]
|
|
T}
|
|
T{
|
|
DC1
|
|
T}@T{
|
|
0x11
|
|
T}@T{
|
|
\[u2411]
|
|
T}
|
|
T{
|
|
DC2
|
|
T}@T{
|
|
0x12
|
|
T}@T{
|
|
\[u2412]
|
|
T}
|
|
T{
|
|
DC3
|
|
T}@T{
|
|
0x13
|
|
T}@T{
|
|
\[u2413]
|
|
T}
|
|
T{
|
|
DC4
|
|
T}@T{
|
|
0x14
|
|
T}@T{
|
|
\[u2414]
|
|
T}
|
|
T{
|
|
NAK
|
|
T}@T{
|
|
0x15
|
|
T}@T{
|
|
\[u2415]
|
|
T}
|
|
T{
|
|
SYN
|
|
T}@T{
|
|
0x16
|
|
T}@T{
|
|
\[u2416]
|
|
T}
|
|
T{
|
|
ETB
|
|
T}@T{
|
|
0x17
|
|
T}@T{
|
|
\[u2417]
|
|
T}
|
|
T{
|
|
CAN
|
|
T}@T{
|
|
0x18
|
|
T}@T{
|
|
\[u2418]
|
|
T}
|
|
T{
|
|
EM
|
|
T}@T{
|
|
0x19
|
|
T}@T{
|
|
\[u2419]
|
|
T}
|
|
T{
|
|
SUB
|
|
T}@T{
|
|
0x1A
|
|
T}@T{
|
|
\[u241A]
|
|
T}
|
|
T{
|
|
ESC
|
|
T}@T{
|
|
0x1B
|
|
T}@T{
|
|
\[u241B]
|
|
T}
|
|
T{
|
|
FS
|
|
T}@T{
|
|
0x1C
|
|
T}@T{
|
|
\[u241C]
|
|
T}
|
|
T{
|
|
GS
|
|
T}@T{
|
|
0x1D
|
|
T}@T{
|
|
\[u241D]
|
|
T}
|
|
T{
|
|
RS
|
|
T}@T{
|
|
0x1E
|
|
T}@T{
|
|
\[u241E]
|
|
T}
|
|
T{
|
|
US
|
|
T}@T{
|
|
0x1F
|
|
T}@T{
|
|
\[u241F]
|
|
T}
|
|
T{
|
|
/
|
|
T}@T{
|
|
0x2F
|
|
T}@T{
|
|
\[uFF0F]
|
|
T}
|
|
T{
|
|
\[dq]
|
|
T}@T{
|
|
0x22
|
|
T}@T{
|
|
\[uFF02]
|
|
T}
|
|
T{
|
|
*
|
|
T}@T{
|
|
0x2A
|
|
T}@T{
|
|
\[uFF0A]
|
|
T}
|
|
T{
|
|
:
|
|
T}@T{
|
|
0x3A
|
|
T}@T{
|
|
\[uFF1A]
|
|
T}
|
|
T{
|
|
<
|
|
T}@T{
|
|
0x3C
|
|
T}@T{
|
|
\[uFF1C]
|
|
T}
|
|
T{
|
|
>
|
|
T}@T{
|
|
0x3E
|
|
T}@T{
|
|
\[uFF1E]
|
|
T}
|
|
T{
|
|
?
|
|
T}@T{
|
|
0x3F
|
|
T}@T{
|
|
\[uFF1F]
|
|
T}
|
|
T{
|
|
\[rs]
|
|
T}@T{
|
|
0x5C
|
|
T}@T{
|
|
\[uFF3C]
|
|
T}
|
|
T{
|
|
|
|
|
T}@T{
|
|
0x7C
|
|
T}@T{
|
|
\[uFF5C]
|
|
T}
|
|
.TE
|
|
.PP
|
|
File names on Windows can also not end with the following characters.
|
|
These only get replaced if they are the last character in the name:
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c.
|
|
T{
|
|
Character
|
|
T}@T{
|
|
Value
|
|
T}@T{
|
|
Replacement
|
|
T}
|
|
_
|
|
T{
|
|
SP
|
|
T}@T{
|
|
0x20
|
|
T}@T{
|
|
\[u2420]
|
|
T}
|
|
T{
|
|
\&.
|
|
T}@T{
|
|
0x2E
|
|
T}@T{
|
|
\[uFF0E]
|
|
T}
|
|
.TE
|
|
.PP
|
|
Invalid UTF-8 bytes will also be
|
|
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
|
|
be converted to UTF-16.
|
|
.SS Paths on Windows
|
|
.PP
|
|
On Windows there are many ways of specifying a path to a file system
|
|
resource.
|
|
Local paths can be absolute, like
|
|
\f[C]C:\[rs]path\[rs]to\[rs]wherever\f[R], or relative, like
|
|
\f[C]..\[rs]wherever\f[R].
|
|
Network paths in UNC format, \f[C]\[rs]\[rs]server\[rs]share\f[R], are
|
|
also supported.
|
|
Path separator can be either \f[C]\[rs]\f[R] (as in
|
|
\f[C]C:\[rs]path\[rs]to\[rs]wherever\f[R]) or \f[C]/\f[R] (as in
|
|
\f[C]C:/path/to/wherever\f[R]).
|
|
Length of these paths are limited to 259 characters for files and 247
|
|
characters for directories, but there is an alternative extended-length
|
|
path format increasing the limit to (approximately) 32,767 characters.
|
|
This format requires absolute paths and the use of prefix
|
|
\f[C]\[rs]\[rs]?\[rs]\f[R], e.g.
|
|
\f[C]\[rs]\[rs]?\[rs]D:\[rs]some\[rs]very\[rs]long\[rs]path\f[R].
|
|
For convenience rclone will automatically convert regular paths into the
|
|
corresponding extended-length paths, so in most cases you do not have to
|
|
worry about this (read more below).
|
|
.PP
|
|
Note that Windows supports using the same prefix
|
|
\f[C]\[rs]\[rs]?\[rs]\f[R] to specify path to volumes identified by
|
|
their GUID, e.g.
|
|
\f[C]\[rs]\[rs]?\[rs]Volume{b75e2c83-0000-0000-0000-602f00000000}\[rs]some\[rs]path\f[R].
|
|
This is \f[I]not\f[R] supported in rclone, due to an
|
|
issue (https://github.com/golang/go/issues/39785) in go.
|
|
.SS Long paths
|
|
.PP
|
|
Rclone handles long paths automatically, by converting all paths to
|
|
extended-length path
|
|
format (https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation),
|
|
which allows paths up to 32,767 characters.
|
|
.PP
|
|
This conversion will ensure paths are absolute and prefix them with the
|
|
\f[C]\[rs]\[rs]?\[rs]\f[R].
|
|
This is why you will see that your paths, for instance
|
|
\f[C].\[rs]files\f[R] is shown as path
|
|
\f[C]\[rs]\[rs]?\[rs]C:\[rs]files\f[R] in the output, and
|
|
\f[C]\[rs]\[rs]server\[rs]share\f[R] as
|
|
\f[C]\[rs]\[rs]?\[rs]UNC\[rs]server\[rs]share\f[R].
|
|
.PP
|
|
However, in rare cases this may cause problems with buggy file system
|
|
drivers like EncFS (https://github.com/rclone/rclone/issues/261).
|
|
To disable UNC conversion globally, add this to your
|
|
\f[C].rclone.conf\f[R] file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[local]
|
|
nounc = true
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you want to selectively disable UNC, you can add it to a separate
|
|
entry like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[nounc]
|
|
type = local
|
|
nounc = true
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
And use rclone like this:
|
|
.PP
|
|
\f[C]rclone copy c:\[rs]src nounc:z:\[rs]dst\f[R]
|
|
.PP
|
|
This will use UNC paths on \f[C]c:\[rs]src\f[R] but not on
|
|
\f[C]z:\[rs]dst\f[R].
|
|
Of course this will cause problems if the absolute path length of a file
|
|
exceeds 259 characters on z, so only use this option if you have to.
|
|
.SS Symlinks / Junction points
|
|
.PP
|
|
Normally rclone will ignore symlinks or junction points (which behave
|
|
like symlinks under Windows).
|
|
.PP
|
|
If you supply \f[C]--copy-links\f[R] or \f[C]-L\f[R] then rclone will
|
|
follow the symlink and copy the pointed to file or directory.
|
|
Note that this flag is incompatible with \f[C]--links\f[R] /
|
|
\f[C]-l\f[R].
|
|
.PP
|
|
This flag applies to all commands.
|
|
.PP
|
|
For example, supposing you have a directory structure like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ tree /tmp/a
|
|
/tmp/a
|
|
\[u251C]\[u2500]\[u2500] b -> ../b
|
|
\[u251C]\[u2500]\[u2500] expected -> ../expected
|
|
\[u251C]\[u2500]\[u2500] one
|
|
\[u2514]\[u2500]\[u2500] two
|
|
\[u2514]\[u2500]\[u2500] three
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Then you can see the difference with and without the flag like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone ls /tmp/a
|
|
6 one
|
|
6 two/three
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
and
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -L ls /tmp/a
|
|
4174 expected
|
|
6 one
|
|
6 two/three
|
|
6 b/two
|
|
6 b/one
|
|
\f[R]
|
|
.fi
|
|
.SS --links, -l
|
|
.PP
|
|
Normally rclone will ignore symlinks or junction points (which behave
|
|
like symlinks under Windows).
|
|
.PP
|
|
If you supply this flag then rclone will copy symbolic links from the
|
|
local storage, and store them as text files, with a
|
|
\[aq].rclonelink\[aq] suffix in the remote storage.
|
|
.PP
|
|
The text file will contain the target of the symbolic link (see
|
|
example).
|
|
.PP
|
|
This flag applies to all commands.
|
|
.PP
|
|
For example, supposing you have a directory structure like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ tree /tmp/a
|
|
/tmp/a
|
|
\[u251C]\[u2500]\[u2500] file1 -> ./file4
|
|
\[u2514]\[u2500]\[u2500] file2 -> /home/user/file3
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Copying the entire directory with \[aq]-l\[aq]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone copyto -l /tmp/a/file1 remote:/tmp/a/
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The remote files are created with a \[aq].rclonelink\[aq] suffix
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone ls remote:/tmp/a
|
|
5 file1.rclonelink
|
|
14 file2.rclonelink
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The remote files will contain the target of the symbolic links
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone cat remote:/tmp/a/file1.rclonelink
|
|
\&./file4
|
|
|
|
$ rclone cat remote:/tmp/a/file2.rclonelink
|
|
/home/user/file3
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Copying them back with \[aq]-l\[aq]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone copyto -l remote:/tmp/a/ /tmp/b/
|
|
|
|
$ tree /tmp/b
|
|
/tmp/b
|
|
\[u251C]\[u2500]\[u2500] file1 -> ./file4
|
|
\[u2514]\[u2500]\[u2500] file2 -> /home/user/file3
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
However, if copied back without \[aq]-l\[aq]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone copyto remote:/tmp/a/ /tmp/b/
|
|
|
|
$ tree /tmp/b
|
|
/tmp/b
|
|
\[u251C]\[u2500]\[u2500] file1.rclonelink
|
|
\[u2514]\[u2500]\[u2500] file2.rclonelink
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that this flag is incompatible with \f[C]-copy-links\f[R] /
|
|
\f[C]-L\f[R].
|
|
.SS Restricting filesystems with --one-file-system
|
|
.PP
|
|
Normally rclone will recurse through filesystems as mounted.
|
|
.PP
|
|
However if you set \f[C]--one-file-system\f[R] or \f[C]-x\f[R] this
|
|
tells rclone to stay in the filesystem specified by the root and not to
|
|
recurse into different file systems.
|
|
.PP
|
|
For example if you have a directory hierarchy like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
root
|
|
\[u251C]\[u2500]\[u2500] disk1 - disk1 mounted on the root
|
|
\[br]\ \ \[u2514]\[u2500]\[u2500] file3 - stored on disk1
|
|
\[u251C]\[u2500]\[u2500] disk2 - disk2 mounted on the root
|
|
\[br]\ \ \[u2514]\[u2500]\[u2500] file4 - stored on disk12
|
|
\[u251C]\[u2500]\[u2500] file1 - stored on the root disk
|
|
\[u2514]\[u2500]\[u2500] file2 - stored on the root disk
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Using \f[C]rclone --one-file-system copy root remote:\f[R] will only
|
|
copy \f[C]file1\f[R] and \f[C]file2\f[R].
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -q --one-file-system ls root
|
|
0 file1
|
|
0 file2
|
|
\f[R]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$ rclone -q ls root
|
|
0 disk1/file3
|
|
0 disk2/file4
|
|
0 file1
|
|
0 file2
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
\f[B]NB\f[R] Rclone (like most unix tools such as \f[C]du\f[R],
|
|
\f[C]rsync\f[R] and \f[C]tar\f[R]) treats a bind mount to the same
|
|
device as being on the same filesystem.
|
|
.PP
|
|
\f[B]NB\f[R] This flag is only available on Unix based systems.
|
|
On systems where it isn\[aq]t supported (e.g.
|
|
Windows) it will be ignored.
|
|
.SS Advanced options
|
|
.PP
|
|
Here are the advanced options specific to local (Local Disk).
|
|
.SS --local-nounc
|
|
.PP
|
|
Disable UNC (long path names) conversion on Windows.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: nounc
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_NOUNC
|
|
.IP \[bu] 2
|
|
Type: string
|
|
.IP \[bu] 2
|
|
Required: false
|
|
.IP \[bu] 2
|
|
Examples:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\[dq]true\[dq]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Disables long file names.
|
|
.RE
|
|
.RE
|
|
.SS --copy-links / -L
|
|
.PP
|
|
Follow symlinks and copy the pointed to item.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: copy_links
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_COPY_LINKS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --links / -l
|
|
.PP
|
|
Translate symlinks to/from regular files with a \[aq].rclonelink\[aq]
|
|
extension.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: links
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_LINKS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --skip-links
|
|
.PP
|
|
Don\[aq]t warn about skipped symlinks.
|
|
.PP
|
|
This flag disables warning messages on skipped symlinks or junction
|
|
points, as you explicitly acknowledge that they should be skipped.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: skip_links
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_SKIP_LINKS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --local-zero-size-links
|
|
.PP
|
|
Assume the Stat size of links is zero (and read them instead)
|
|
(deprecated).
|
|
.PP
|
|
Rclone used to use the Stat size of links as the link size, but this
|
|
fails in quite a few places:
|
|
.IP \[bu] 2
|
|
Windows
|
|
.IP \[bu] 2
|
|
On some virtual filesystems (such ash LucidLink)
|
|
.IP \[bu] 2
|
|
Android
|
|
.PP
|
|
So rclone now always reads the link.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: zero_size_links
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_ZERO_SIZE_LINKS
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --local-unicode-normalization
|
|
.PP
|
|
Apply unicode NFC normalization to paths and filenames.
|
|
.PP
|
|
This flag can be used to normalize file names into unicode NFC form that
|
|
are read from the local filesystem.
|
|
.PP
|
|
Rclone does not normally touch the encoding of file names it reads from
|
|
the file system.
|
|
.PP
|
|
This can be useful when using macOS as it normally provides decomposed
|
|
(NFD) unicode which in some language (eg Korean) doesn\[aq]t display
|
|
properly on some OSes.
|
|
.PP
|
|
Note that rclone compares filenames with unicode normalization in the
|
|
sync routine so this flag shouldn\[aq]t normally be used.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: unicode_normalization
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_UNICODE_NORMALIZATION
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --local-no-check-updated
|
|
.PP
|
|
Don\[aq]t check to see if the files change during upload.
|
|
.PP
|
|
Normally rclone checks the size and modification time of files as they
|
|
are being uploaded and aborts with a message which starts \[dq]can\[aq]t
|
|
copy - source file is being updated\[dq] if the file changes during
|
|
upload.
|
|
.PP
|
|
However on some file systems this modification time check may fail (e.g.
|
|
Glusterfs #2206 (https://github.com/rclone/rclone/issues/2206)) so this
|
|
check can be disabled with this flag.
|
|
.PP
|
|
If this flag is set, rclone will use its best efforts to transfer a file
|
|
which is being updated.
|
|
If the file is only having things appended to it (e.g.
|
|
a log) then rclone will transfer the log file with the size it had the
|
|
first time rclone saw it.
|
|
.PP
|
|
If the file is being modified throughout (not just appended to) then the
|
|
transfer may fail with a hash check failure.
|
|
.PP
|
|
In detail, once the file has had stat() called on it for the first time
|
|
we:
|
|
.IP \[bu] 2
|
|
Only transfer the size that stat gave
|
|
.IP \[bu] 2
|
|
Only checksum the size that stat gave
|
|
.IP \[bu] 2
|
|
Don\[aq]t update the stat info for the file
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_check_updated
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_NO_CHECK_UPDATED
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --one-file-system / -x
|
|
.PP
|
|
Don\[aq]t cross filesystem boundaries (unix/macOS only).
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: one_file_system
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_ONE_FILE_SYSTEM
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --local-case-sensitive
|
|
.PP
|
|
Force the filesystem to report itself as case sensitive.
|
|
.PP
|
|
Normally the local backend declares itself as case insensitive on
|
|
Windows/macOS and case sensitive for everything else.
|
|
Use this flag to override the default choice.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: case_sensitive
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_CASE_SENSITIVE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --local-case-insensitive
|
|
.PP
|
|
Force the filesystem to report itself as case insensitive.
|
|
.PP
|
|
Normally the local backend declares itself as case insensitive on
|
|
Windows/macOS and case sensitive for everything else.
|
|
Use this flag to override the default choice.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: case_insensitive
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_CASE_INSENSITIVE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --local-no-preallocate
|
|
.PP
|
|
Disable preallocation of disk space for transferred files.
|
|
.PP
|
|
Preallocation of disk space helps prevent filesystem fragmentation.
|
|
However, some virtual filesystem layers (such as Google Drive File
|
|
Stream) may incorrectly set the actual file size equal to the
|
|
preallocated space, causing checksum and file size checks to fail.
|
|
Use this flag to disable preallocation.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_preallocate
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_NO_PREALLOCATE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --local-no-sparse
|
|
.PP
|
|
Disable sparse files for multi-thread downloads.
|
|
.PP
|
|
On Windows platforms rclone will make sparse files when doing
|
|
multi-thread downloads.
|
|
This avoids long pauses on large files where the OS zeros the file.
|
|
However sparse files may be undesirable as they cause disk fragmentation
|
|
and can be slow to work with.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_sparse
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_NO_SPARSE
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --local-no-set-modtime
|
|
.PP
|
|
Disable setting modtime.
|
|
.PP
|
|
Normally rclone updates modification time of files after they are done
|
|
uploading.
|
|
This can cause permissions issues on Linux platforms when the user
|
|
rclone is running as does not own the file uploaded, such as when
|
|
copying to a CIFS mount owned by another user.
|
|
If this option is enabled, rclone will no longer update the modtime
|
|
after copying a file.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: no_set_modtime
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_NO_SET_MODTIME
|
|
.IP \[bu] 2
|
|
Type: bool
|
|
.IP \[bu] 2
|
|
Default: false
|
|
.SS --local-encoding
|
|
.PP
|
|
The encoding for the backend.
|
|
.PP
|
|
See the encoding section in the
|
|
overview (https://rclone.org/overview/#encoding) for more info.
|
|
.PP
|
|
Properties:
|
|
.IP \[bu] 2
|
|
Config: encoding
|
|
.IP \[bu] 2
|
|
Env Var: RCLONE_LOCAL_ENCODING
|
|
.IP \[bu] 2
|
|
Type: MultiEncoder
|
|
.IP \[bu] 2
|
|
Default: Slash,Dot
|
|
.SS Backend commands
|
|
.PP
|
|
Here are the commands specific to the local backend.
|
|
.PP
|
|
Run them with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend COMMAND remote:
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The help below will explain what arguments each command takes.
|
|
.PP
|
|
See the \[dq]rclone backend\[dq]
|
|
command (https://rclone.org/commands/rclone_backend/) for more info on
|
|
how to pass options and arguments.
|
|
.PP
|
|
These can be run on a running backend using the rc command
|
|
backend/command (https://rclone.org/rc/#backend-command).
|
|
.SS noop
|
|
.PP
|
|
A null operation for testing backend commands
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone backend noop remote: [options] [<arguments>+]
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
This is a test command which has some options you can try to change the
|
|
output.
|
|
.PP
|
|
Options:
|
|
.IP \[bu] 2
|
|
\[dq]echo\[dq]: echo the input arguments
|
|
.IP \[bu] 2
|
|
\[dq]error\[dq]: return an error based on option value
|
|
.SH Changelog
|
|
.SS v1.58.0 - 2022-03-18
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.57.0...v1.58.0)
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Akamai Netstorage (Nil Alexandrov)
|
|
.IP \[bu] 2
|
|
Seagate Lyve (https://rclone.org/s3/#lyve), SeaweedFS, Storj, RackCorp
|
|
via s3 backend
|
|
.IP \[bu] 2
|
|
Storj (https://rclone.org/storj/) (renamed from Tardigrade - your old
|
|
config files will continue working)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
bisync (https://rclone.org/bisync/) - experimental bidirectional cloud
|
|
sync (Ivan Andreev, Chris Nelson)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]windows/arm64\f[R] build (\f[C]rclone mount\f[R] not supported
|
|
yet) (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Raise minimum go version to go1.15 (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
config: Allow dot in remote names and improve config editing (albertony)
|
|
.IP \[bu] 2
|
|
dedupe: Add quit as a choice in interactive mode (albertony)
|
|
.IP \[bu] 2
|
|
dlna: Change icons to the newest ones.
|
|
(Alain Nussbaumer)
|
|
.IP \[bu] 2
|
|
filter: Add \f[C]{{ regexp }}\f[R]
|
|
syntax (https://rclone.org/filtering/#regexp) to pattern matches (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
fshttp: Add prometheus metrics for HTTP status code (Micha\[/l] Matczuk)
|
|
.IP \[bu] 2
|
|
hashsum: Support creating hash from data received on stdin (albertony)
|
|
.IP \[bu] 2
|
|
librclone
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow empty string or null input instead of empty json object
|
|
(albertony)
|
|
.IP \[bu] 2
|
|
Add support for mount commands (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
operations: Add server-side moves to stats (Ole Frost)
|
|
.IP \[bu] 2
|
|
rc: Allow user to disable authentication for web gui (negative0)
|
|
.IP \[bu] 2
|
|
tree: Remove obsolete \f[C]--human\f[R] replaced by global
|
|
\f[C]--human-readable\f[R] (albertony)
|
|
.IP \[bu] 2
|
|
version: Report correct friendly-name for newer Windows 10/11 versions
|
|
(albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix ARM architecture version in .deb packages after nfpm change (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Hard fork \f[C]github.com/jlaffaye/ftp\f[R] to fix
|
|
\f[C]go get github.com/rclone/rclone\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
oauthutil: Fix crash when webrowser requests \f[C]/robots.txt\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
operations: Fix goroutine leak in case of copy retry (Ankur Gupta)
|
|
.IP \[bu] 2
|
|
rc:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix \f[C]operations/publiclink\f[R] default for \f[C]expires\f[R]
|
|
parameter (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix missing computation of \f[C]transferQueueSize\f[R] when summing up
|
|
statistics group (Carlo Mion)
|
|
.IP \[bu] 2
|
|
Fix missing \f[C]StatsInfo\f[R] fields in the computation of the group
|
|
sum (Carlo Mion)
|
|
.RE
|
|
.IP \[bu] 2
|
|
sync: Fix \f[C]--max-duration\f[R] so it doesn\[aq]t retry when the
|
|
duration is exceeded (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
touch: Fix issue where a directory is created instead of a file
|
|
(albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--devname\f[R] to set the device name sent to FUSE for mount
|
|
display (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]vfs/stats\f[R] remote control to show statistics (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix
|
|
\f[C]failed to _ensure cache internal error: downloaders is nil error\f[R]
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix handling of special characters in file names (Bumsu Hyeon)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix hash invalidation which caused errors with local crypt mount (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]base64\f[R] and \f[C]base32768\f[R] filename encoding options
|
|
(Max Sum, Sinan Tan)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]--azureblob-upload-concurrency\f[R] parameter to speed
|
|
uploads (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove 100MB upper limit on \f[C]chunk_size\f[R] as it is no longer
|
|
needed (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Raise \f[C]--azureblob-upload-concurrency\f[R] to 16 by default (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix crash with SAS URL and no container (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Compress
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix crash if metadata upload failed (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix memory leak (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Added \f[C]--drive-copy-shortcut-content\f[R] (Abhiraj)
|
|
.IP \[bu] 2
|
|
Disable OAuth OOB flow (copy a token) due to Google deprecation (Nick
|
|
Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
See the deprecation
|
|
note (https://developers.googleblog.com/2022/02/making-oauth-flows-safer.html#disallowed-oob).
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-skip-dangling-shortcuts\f[R] flag (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
When using a link type \f[C]--drive-export-formats\f[R] shows all doc
|
|
types (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Speed up directory listings by specifying 1000 items in a chunk (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Save an API request when at the root (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fichier
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implemented About functionality (Gourav T)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--ftp-ask-password\f[R] to prompt for password when needed
|
|
(Borna Butkovic)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add missing regions (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Disable OAuth OOB flow (copy a token) due to Google deprecation (Nick
|
|
Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
See the deprecation
|
|
note (https://developers.googleblog.com/2022/02/making-oauth-flows-safer.html#disallowed-oob).
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Googlephotos
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Disable OAuth OOB flow (copy a token) due to Google deprecation (Nick
|
|
Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
See the deprecation
|
|
note (https://developers.googleblog.com/2022/02/making-oauth-flows-safer.html#disallowed-oob).
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Hasher
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix crash on object not found (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Hdfs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add file (Move) and directory move (DirMove) support (Andy Jackson)
|
|
.RE
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Improved recognition of URL pointing to a single file (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Change API used by recursive list (ListR) (Kim)
|
|
.IP \[bu] 2
|
|
Add support for Tele2 Cloud (Fredric Arklid)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Koofr
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add Digistorage service as a Koofr provider.
|
|
(jaKa)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mailru
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix int32 overflow on arm32 (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add config option for oauth scope \f[C]Sites.Read.All\f[R] (Charlie
|
|
Jiang)
|
|
.IP \[bu] 2
|
|
Minor optimization of quickxorhash (Isaac Levy)
|
|
.IP \[bu] 2
|
|
Add \f[C]--onedrive-root-folder-id\f[R] flag (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Do not retry on \f[C]400 pathIsTooLong\f[R] error (ctrl-q)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for recursive list (ListR) (Niels van de Weem)
|
|
.IP \[bu] 2
|
|
Fix pre-1970 time stamps (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use \f[C]ListObjectsV2\f[R] for faster listings (Felix Bu\[u0308]nemann)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fallback to \f[C]ListObject\f[R] v1 on unsupported providers (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Use the \f[C]ETag\f[R] on multipart transfers to verify the transfer was
|
|
OK (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--s3-use-multipart-etag\f[R] provider quirk to disable this on
|
|
unsupported providers (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Providers
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
RackCorp object storage (bbabich)
|
|
.IP \[bu] 2
|
|
Seagate Lyve Cloud storage (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
SeaweedFS (Chris Lu)
|
|
.IP \[bu] 2
|
|
Storj Shared gateways (M\['a]rton Elek, Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add Wasabi AP Northeast 2 endpoint info (lindwurm)
|
|
.IP \[bu] 2
|
|
Add \f[C]GLACIER_IR\f[R] storage class (Yunhai Luo)
|
|
.IP \[bu] 2
|
|
Document \f[C]Content-MD5\f[R] workaround for object-lock enabled
|
|
buckets (Paulo Martins)
|
|
.IP \[bu] 2
|
|
Fix multipart upload with \f[C]--no-head\f[R] flag (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Simplify content length processing in s3 with download url (Logeshwaran
|
|
Murugesan)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add rclone to list of supported \f[C]md5sum\f[R]/\f[C]sha1sum\f[R]
|
|
commands to look for (albertony)
|
|
.IP \[bu] 2
|
|
Refactor so we only have one way of running remote commands (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix timeout on hashing large files by sending keepalives (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix unecessary seeking when uploading and downloading files (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update docs on how to create \f[C]known_hosts\f[R] file (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Storj
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rename tardigrade backend to storj backend (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement server side Move for files (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update docs to explain differences between s3 and this backend (Elek,
|
|
M\['a]rton)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix About so it shows info about the current container only (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Union
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix treatment of remotes with \f[C]//\f[R] in (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix deadlock when one part of a multi-upload fails (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix eplus policy returned nil (Vitor Arruda)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Yandex
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add permanent deletion support (deinferno)
|
|
.RE
|
|
.SS v1.57.0 - 2021-11-01
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.56.0...v1.57.0)
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Sia: for Sia decentralized cloud (Ian Levesque, Matthew Sevey, Ivan
|
|
Andreev)
|
|
.IP \[bu] 2
|
|
Hasher: caches hashes and enable hashes for backends that don\[aq]t
|
|
support them (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
lsjson --stat: to get info about a single file/dir and
|
|
\f[C]operations/stat\f[R] api (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
config paths: show configured paths (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
about: Make human-readable output more consistent with other commands
|
|
(albertony)
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use go1.17 for building and make go1.14 the minimum supported (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update Go to 1.16 and NDK to 22b for Android builds (x0b)
|
|
.RE
|
|
.IP \[bu] 2
|
|
config
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Support hyphen in remote name from environment variable (albertony)
|
|
.IP \[bu] 2
|
|
Make temporary directory user-configurable (albertony)
|
|
.IP \[bu] 2
|
|
Convert \f[C]--cache-dir\f[R] value to an absolute path (albertony)
|
|
.IP \[bu] 2
|
|
Do not override MIME types from OS defaults (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
docs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Toc styling and header levels cleanup (albertony)
|
|
.IP \[bu] 2
|
|
Extend documentation on valid remote names (albertony)
|
|
.IP \[bu] 2
|
|
Mention make for building and cmount tag for macos (Alex Chen)
|
|
.IP \[bu] 2
|
|
\&...and many more contributions to numerous to mention!
|
|
.RE
|
|
.IP \[bu] 2
|
|
fs: Move with \f[C]--ignore-existing\f[R] will not delete skipped files
|
|
(Nathan Collins)
|
|
.IP \[bu] 2
|
|
hashsum
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Treat hash values in sum file as case insensitive (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Don\[aq]t put \f[C]ERROR\f[R] or \f[C]UNSUPPORTED\f[R] in output (Ivan
|
|
Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
lib/encoder: Add encoding of square brackets (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
lib/file: Improve error message when attempting to create dir on
|
|
nonexistent drive on windows (albertony)
|
|
.IP \[bu] 2
|
|
lib/http: Factor password hash salt into options with default (Nolan
|
|
Woods)
|
|
.IP \[bu] 2
|
|
lib/kv: Add key-value database api (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
librclone
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]RcloneFreeString\f[R] function (albertony)
|
|
.IP \[bu] 2
|
|
Free strings in python example (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
log: Optionally print pid in logs (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
ls: Introduce \f[C]--human-readable\f[R] global option to print
|
|
human-readable sizes (albertony)
|
|
.IP \[bu] 2
|
|
ncdu: Introduce key \f[C]u\f[R] to toggle human-readable (albertony)
|
|
.IP \[bu] 2
|
|
operations: Add \f[C]rmdirs -v\f[R] output (Justin Winokur)
|
|
.IP \[bu] 2
|
|
serve sftp
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Generate an ECDSA server key as well as RSA (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Generate an Ed25519 server key as well as ECDSA and RSA (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve docker
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow to customize proxy settings of docker plugin (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Build docker plugin for multiple platforms (Thomas Stachl)
|
|
.RE
|
|
.IP \[bu] 2
|
|
size: Include human-readable count (albertony)
|
|
.IP \[bu] 2
|
|
touch: Add support for touching files in directory, with recursive
|
|
option, filtering and \f[C]--dry-run\f[R]/\f[C]-i\f[R] (albertony)
|
|
.IP \[bu] 2
|
|
tree: Option to print human-readable sizes removed in favor of global
|
|
option (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
lib/http
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix bad username check in single auth secret provider (Nolan Woods)
|
|
.IP \[bu] 2
|
|
Fix handling of SSL credentials (Nolan Woods)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve ftp: Ensure modtime is passed as UTC always to fix timezone
|
|
oddities (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
serve sftp: Fix generation of server keys on windows (albertony)
|
|
.IP \[bu] 2
|
|
serve docker: Fix octal umask (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enable rclone to be run as mount helper direct from the fstab (Ivan
|
|
Andreev)
|
|
.IP \[bu] 2
|
|
Use procfs to validate mount on linux (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Correctly daemonize for compatibility with automount (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ensure names used in cache path are legal on current OS (albertony)
|
|
.IP \[bu] 2
|
|
Ignore \f[C]ECLOSED\f[R] when truncating file handles to fix
|
|
intermittent bad file descriptor error (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Refactor default OS encoding out from local backend into shared encoder
|
|
lib (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Return wrapped object even with \f[C]--crypt-no-data-encryption\f[R]
|
|
(Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Fix uploads with \f[C]--crypt-no-data-encryption\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--azureblob-no-head-object\f[R] (Tatsuya Noyori)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make listings of heavily used directories more reliable (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
When doing cleanup delete as much as possible (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--box-list-chunk\f[R] to control listing chunk size (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Delete items in parallel in cleanup using \f[C]--checkers\f[R] threads
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--box-owned-by\f[R] to only show items owned by the login
|
|
passed (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Retry \f[C]operation_blocked_temporary\f[R] errors (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Chunker
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Md5all must create metadata if base hash is slow (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Speed up directory listings by constraining the API listing using the
|
|
current filters (fotile96, Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Fix buffering for single request upload for files smaller than
|
|
\f[C]--drive-upload-cutoff\f[R] (YenForYang)
|
|
.IP \[bu] 2
|
|
Add \f[C]-o config\f[R] option to \f[C]backend drives\f[R] to make
|
|
config for all shared drives (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--dropbox-batch-commit-timeout\f[R] to control batch timeout
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Filefabric
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make backoff exponential for error_background to fix errors (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix directory move after API change (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enable tls session cache by default (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Add option to disable tls13 (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Fix timeout after long uploads (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Add support for precise time (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Enable CI for ProFtpd, PureFtpd, VsFtpd (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Googlephotos
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use encoder for album names to fix albums with control characters (Parth
|
|
Shukla)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]SetModTime\f[R] to support modtime-only changes
|
|
(albertony)
|
|
.IP \[bu] 2
|
|
Improved error handling with \f[C]SetModTime\f[R] and corrupt files in
|
|
general (albertony)
|
|
.IP \[bu] 2
|
|
Add support for \f[C]UserInfo\f[R] (\f[C]rclone config userinfo\f[R])
|
|
feature (albertony)
|
|
.IP \[bu] 2
|
|
Return direct download link from \f[C]rclone link\f[R] command
|
|
(albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Koofr
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Create direct share link (Dmitry Bogatov)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add sha256 support (Ken Enrique Morel)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Premiumizeme
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix directory listing after API changes (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix server side move after API change (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix server side directory move after API changes (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support to use CDN URL to download the file (Logeshwaran)
|
|
.IP \[bu] 2
|
|
Add AWS Snowball Edge to providers examples (r0kk3rz)
|
|
.IP \[bu] 2
|
|
Use a combination of SDK retries and rclone retries (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix IAM Role for Service Account not working and other auth problems
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \f[C]shared_credentials_file\f[R] auth after reverting incorrect fix
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix corrupted on transfer: sizes differ 0 vs xxxx with Ceph (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Seafile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix error when not configured for 2fa (Fred)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix timeout when doing MD5SUM of large file (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update OCI URL (David Liu)
|
|
.IP \[bu] 2
|
|
Document OVH Cloud Archive (HNGamingUK)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Union
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix rename not working with union of local disk and bucket based remote
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.56.2 - 2021-10-01
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.56.1...v1.56.2)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
serve http: Re-add missing auth to http service (Nolan Woods)
|
|
.IP \[bu] 2
|
|
build: Update golang.org/x/sys to fix crash on macOS when compiled with
|
|
go1.17 (Herby Gillot)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix deadlock after failed update when concurrency=1 (Ivan Andreev)
|
|
.RE
|
|
.SS v1.56.1 - 2021-09-19
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.56.0...v1.56.1)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting: Fix maximum bwlimit by scaling scale max token bucket size
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
rc: Fix speed does not update in core/stats (negative0)
|
|
.IP \[bu] 2
|
|
selfupdate: Fix \f[C]--quiet\f[R] option, not quite quiet (yedamo)
|
|
.IP \[bu] 2
|
|
serve http: Fix \f[C]serve http\f[R] exiting directly after starting
|
|
(Cnly)
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Apply gofmt from golang 1.17 (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Update Go to 1.16 and NDK to 22b for android/any (x0b)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix \f[C]--daemon\f[R] mode (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix duplicates on rename (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix crash when truncating a just uploaded object (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix issue where empty dirs would build up in cache meta dir (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix instructions for auto config (Greg Sadetsky)
|
|
.IP \[bu] 2
|
|
Fix lsf example without drive-impersonate (Greg Sadetsky)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Handle HTTP 400 better in PublicLink (Alex Chen)
|
|
.IP \[bu] 2
|
|
Clarification of the process for creating custom client_id (Mariano
|
|
Absatz)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Return an early error when Put is called with an unknown size (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Try harder to delete a failed upload (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add Wasabi\[aq]s AP-Northeast endpoint info (hota)
|
|
.IP \[bu] 2
|
|
Fix typo in s3 documentation (Greg Sadetsky)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Seafile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix 2fa config state machine (Fred)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Remove spurious error message on
|
|
\f[C]--sftp-disable-concurrent-reads\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Sugarsync
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix initial connection after config re-arrangement (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.56.0 - 2021-07-20
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.55.0...v1.56.0)
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Uptobox (https://rclone.org/uptobox/) (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
serve docker (https://rclone.org/commands/rclone_serve_docker/) (Antoine
|
|
GIRARD) (Ivan Andreev)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
and accompanying docker volume plugin (https://rclone.org/docker/)
|
|
.RE
|
|
.IP \[bu] 2
|
|
checksum (https://rclone.org/commands/rclone_checksum/) to check files
|
|
against a file of checksums (Ivan Andreev)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this is also available as \f[C]rclone md5sum -C\f[R] etc
|
|
.RE
|
|
.IP \[bu] 2
|
|
config touch (https://rclone.org/commands/rclone_config_touch/): ensure
|
|
config exists at configured location (albertony)
|
|
.IP \[bu] 2
|
|
test
|
|
changenotify (https://rclone.org/commands/rclone_test_changenotify/):
|
|
command to help debugging changenotify (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Deprecations
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]dbhashsum\f[R]: Remove command deprecated a year ago (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
\f[C]cache\f[R]: Deprecate cache backend (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
rework config system so it can be used non-interactively via cli and rc
|
|
API.
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
See docs in config
|
|
create (https://rclone.org/commands/rclone_config_create/)
|
|
.IP \[bu] 2
|
|
This is a very big change to all the backends so may cause breakages -
|
|
please file bugs!
|
|
.RE
|
|
.IP \[bu] 2
|
|
librclone - export the rclone RC as a C library (lewisxy) (Nick
|
|
Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Link a C-API rclone shared object into your project
|
|
.IP \[bu] 2
|
|
Use the RC as an in memory interface
|
|
.IP \[bu] 2
|
|
Python example supplied
|
|
.IP \[bu] 2
|
|
Also supports Android and gomobile
|
|
.RE
|
|
.IP \[bu] 2
|
|
fs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--disable-http2\f[R] for global http2 disable (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make \f[C]--dump\f[R] imply \f[C]-vv\f[R] (Alex Chen)
|
|
.IP \[bu] 2
|
|
Use binary prefixes for size and rate units (albertony)
|
|
.IP \[bu] 2
|
|
Use decimal prefixes for counts (albertony)
|
|
.IP \[bu] 2
|
|
Add google search widget to rclone.org (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
accounting: Calculate rolling average speed (Haochen Tong)
|
|
.IP \[bu] 2
|
|
atexit: Terminate with non-zero status after receiving signal (Michael
|
|
Hanselmann)
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Only run event-based workflow scripts under rclone repo with manual
|
|
override (Mathieu Carbou)
|
|
.IP \[bu] 2
|
|
Add Android build with gomobile (x0b)
|
|
.RE
|
|
.IP \[bu] 2
|
|
check: Log the hash in use like cryptcheck does (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
version: Print os/version, kernel and bitness (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
config
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Prevent use of Windows reserved names in config file name (albertony)
|
|
.IP \[bu] 2
|
|
Create config file in windows appdata directory by default (albertony)
|
|
.IP \[bu] 2
|
|
Treat any config file paths with filename notfound as memory-only config
|
|
(albertony)
|
|
.IP \[bu] 2
|
|
Delay load config file (albertony)
|
|
.IP \[bu] 2
|
|
Replace defaultConfig with a thread-safe in-memory implementation (Chris
|
|
Macklin)
|
|
.IP \[bu] 2
|
|
Allow \f[C]config create\f[R] and friends to take \f[C]key=value\f[R]
|
|
parameters (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fixed issues with flags/options set by environment vars.
|
|
(Ole Frost)
|
|
.RE
|
|
.IP \[bu] 2
|
|
fshttp: Implement graceful DSCP error handling (Tyson Moore)
|
|
.IP \[bu] 2
|
|
lib/http - provides an abstraction for a central http server that
|
|
services can bind routes to (Nolan Woods)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--template\f[R] config and flags to serve/data (Nolan Woods)
|
|
.IP \[bu] 2
|
|
Add default 404 handler (Nolan Woods)
|
|
.RE
|
|
.IP \[bu] 2
|
|
link: Use \[dq]off\[dq] value for unset expiry (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
oauthutil: Raise fatal error if token expired without refresh token
|
|
(Alex Chen)
|
|
.IP \[bu] 2
|
|
rcat: Add \f[C]--size\f[R] flag for more efficient uploads of known size
|
|
(Nazar Mishturak)
|
|
.IP \[bu] 2
|
|
serve sftp: Add \f[C]--stdio\f[R] flag to serve via stdio (Tom)
|
|
.IP \[bu] 2
|
|
sync: Don\[aq]t warn about \f[C]--no-traverse\f[R] when
|
|
\f[C]--files-from\f[R] is set (Nick Gaya)
|
|
.IP \[bu] 2
|
|
\f[C]test makefiles\f[R]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--seed\f[R] flag and make data generated repeatable (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add log levels and speed summary (Nick Craig-Wood)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting: Fix startTime of statsGroups.sum (Haochen Tong)
|
|
.IP \[bu] 2
|
|
cmd/ncdu: Fix out of range panic in delete (buengese)
|
|
.IP \[bu] 2
|
|
config
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix issues with memory-only config file paths (albertony)
|
|
.IP \[bu] 2
|
|
Fix in memory config not saving on the fly backend config (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
fshttp: Fix address parsing for DSCP (Tyson Moore)
|
|
.IP \[bu] 2
|
|
ncdu: Update termbox-go library to fix crash (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
oauthutil: Fix old authorize result not recognised (Cnly)
|
|
.IP \[bu] 2
|
|
operations: Don\[aq]t update timestamps of files in
|
|
\f[C]--compare-dest\f[R] (Nick Gaya)
|
|
.IP \[bu] 2
|
|
selfupdate: fix archive name on macos (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Refactor before adding serve docker (Antoine GIRARD)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add cache reset for \f[C]--vfs-cache-max-size\f[R] handling at cache
|
|
poll interval (Leo Luan)
|
|
.IP \[bu] 2
|
|
Fix modtime changing when reading file into cache (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Avoid unnecessary subdir in cache path (albertony)
|
|
.IP \[bu] 2
|
|
Fix that umask option cannot be set as environment variable (albertony)
|
|
.IP \[bu] 2
|
|
Do not print notice about missing poll-interval support when set to 0
|
|
(albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Always use readlink to read symlink size for better compatibility (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--local-unicode-normalization\f[R] (and remove
|
|
\f[C]--local-no-unicode-normalization\f[R]) (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Skip entries removed concurrently with List() (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Support timestamped filenames from \f[C]--b2-versions\f[R] (Dominik
|
|
Mydlil)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t include the bucket name in public link file prefixes (Jeffrey
|
|
Tolar)
|
|
.IP \[bu] 2
|
|
Fix versions and .files with no extension (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Factor version handling into lib/version (Dominik Mydlil)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use upload preflight check to avoid listings in file uploads (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Return errors instead of calling log.Fatal with them (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Switch to the Drives API for looking up shared drives (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix some google docs being treated as files (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--dropbox-batch-mode\f[R] flag to speed up uploading (Nick
|
|
Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Read the batch mode (https://rclone.org/dropbox/#batch-mode) docs for
|
|
more info
|
|
.RE
|
|
.IP \[bu] 2
|
|
Set visibility in link sharing when \f[C]--expire\f[R] is set (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Simplify chunked uploads (Alexey Ivanov)
|
|
.IP \[bu] 2
|
|
Improve \[dq]own App IP\[dq] instructions (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fichier
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Check if more than one upload link is returned (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Support downloading password protected files and folders (Florian
|
|
Penzkofer)
|
|
.IP \[bu] 2
|
|
Make error messages report text from the API (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix move of files in the same directory (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Check that we actually got a download token and retry if we didn\[aq]t
|
|
(buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Filefabric
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix listing after change of from field from \[dq]int\[dq] to int.
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make upload error 250 indicate success (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
GCS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make compatible with gsutil\[aq]s mtime metadata (database64128)
|
|
.IP \[bu] 2
|
|
Clean up time format constants (database64128)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Photos
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix read only scope not being used properly (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Replace httplib with lib/http (Nolan Woods)
|
|
.IP \[bu] 2
|
|
Clean up Bind to better use middleware (Nolan Woods)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix legacy auth with state based config system (buengese)
|
|
.IP \[bu] 2
|
|
Fix invalid url in output from link command (albertony)
|
|
.IP \[bu] 2
|
|
Add no versions option (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]list_chunk option\f[R] (Nick Gaya)
|
|
.IP \[bu] 2
|
|
Also report root error if unable to cancel multipart upload (Cnly)
|
|
.IP \[bu] 2
|
|
Fix failed to configure: empty token found error (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make link return direct download link (Xuanchen Wu)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--s3-no-head-object\f[R] (Tatsuya Noyori)
|
|
.IP \[bu] 2
|
|
Remove WebIdentityRoleProvider to fix crash on auth (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Don\[aq]t check to see if remote is object if it ends with / (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add SeaweedFS (Chris Lu)
|
|
.IP \[bu] 2
|
|
Update Alibaba OSS endpoints (Chuan Zh)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix performance regression by re-enabling concurrent writes (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Expand tilde and environment variables in configured
|
|
\f[C]known_hosts_file\f[R] (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Tardigrade
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Upgrade to uplink v1.4.6 (Caleb Case)
|
|
.IP \[bu] 2
|
|
Use negative offset (Caleb Case)
|
|
.IP \[bu] 2
|
|
Add warning about \f[C]too many open files\f[R] (acsfer)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix sharepoint auth over http (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add headers option (Antoon Prins)
|
|
.RE
|
|
.SS v1.55.1 - 2021-04-26
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.55.0...v1.55.1)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
selfupdate
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Dont detect FUSE if build is static (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Add build tag noselfupdate (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
sync: Fix incorrect error reported by graceful cutoff (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
install.sh: fix macOS arm64 download (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
build: Fix version numbers in android branch builds (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
docs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Contributing.md: update setup instructions for go1.16 (Nick Gaya)
|
|
.IP \[bu] 2
|
|
WinFsp 2021 is out of beta (albertony)
|
|
.IP \[bu] 2
|
|
Minor cleanup of space around code section (albertony)
|
|
.IP \[bu] 2
|
|
Fixed some typos (albertony)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix a code path which allows dirty data to be removed causing data loss
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Compress
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix compressed name regexp (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix backend copyid of google doc to directory (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Don\[aq]t open browser when service account...
|
|
(Ansh Mittal)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add missing team_data.member scope for use with --impersonate (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix About after scopes changes - rclone config reconnect needed (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix Unable to decrypt returned paths from changeNotify (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix implicit TLS (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Work around for random \[dq]Unable to initialize RPS\[dq] errors
|
|
(OleFrost)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Revert sftp library to v1.12.0 from v1.13.0 to fix performance
|
|
regression (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix Update ReadFrom failed: failed to send packet: EOF errors (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Zoho
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix error when region isn\[aq]t set (buengese)
|
|
.IP \[bu] 2
|
|
Do not ask for mountpoint twice when using headless setup (buengese)
|
|
.RE
|
|
.SS v1.55.0 - 2021-03-31
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.54.0...v1.55.0)
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
selfupdate (https://rclone.org/commands/rclone_selfupdate/) (Ivan
|
|
Andreev)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allows rclone to update itself in-place or via a package (using
|
|
\f[C]--package\f[R] flag)
|
|
.IP \[bu] 2
|
|
Reads cryptographically signed signatures for non beta releases
|
|
.IP \[bu] 2
|
|
Works on all OSes.
|
|
.RE
|
|
.IP \[bu] 2
|
|
test (https://rclone.org/commands/rclone_test/) - these are test
|
|
commands - use with care!
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]histogram\f[R] - Makes a histogram of file name characters.
|
|
.IP \[bu] 2
|
|
\f[C]info\f[R] - Discovers file name or other limitations for paths.
|
|
.IP \[bu] 2
|
|
\f[C]makefiles\f[R] - Make a random file hierarchy for testing.
|
|
.IP \[bu] 2
|
|
\f[C]memory\f[R] - Load all the objects at remote:path into memory and
|
|
report memory stats.
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Connection strings (https://rclone.org/docs/#connection-strings)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Config parameters can now be passed as part of the remote name as a
|
|
connection string.
|
|
.IP \[bu] 2
|
|
For example, to do the equivalent of \f[C]--drive-shared-with-me\f[R]
|
|
use \f[C]drive,shared_with_me:\f[R]
|
|
.IP \[bu] 2
|
|
Make sure we don\[aq]t save on the fly remote config to the config file
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make sure backends with additional config have a different name for
|
|
caching (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
This work was sponsored by CERN, through the CS3MESH4EOSC
|
|
Project (https://cs3mesh4eosc.eu/).
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
CS3MESH4EOSC has received funding from the European Union\[cq]s Horizon
|
|
2020
|
|
.IP \[bu] 2
|
|
research and innovation programme under Grant Agreement no.
|
|
863353.
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update go build version to go1.16 and raise minimum go version to go1.13
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make a macOS ARM64 build to support Apple Silicon (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Install macfuse 4.x instead of osxfuse 3.x (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Use \f[C]GO386=softfloat\f[R] instead of deprecated \f[C]GO386=387\f[R]
|
|
for 386 builds (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Disable IOS builds for the time being (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Androids builds made with up to date NDK (x0b)
|
|
.IP \[bu] 2
|
|
Add an rclone user to the Docker image but don\[aq]t use it by default
|
|
(cynthia kwok)
|
|
.RE
|
|
.IP \[bu] 2
|
|
dedupe: Make largest directory primary to minimize data moved (Saksham
|
|
Khanna)
|
|
.IP \[bu] 2
|
|
config
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Wrap config library in an interface (Fionera)
|
|
.IP \[bu] 2
|
|
Make config file system pluggable (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
\f[C]--config \[dq]\[dq]\f[R] or \f[C]\[dq]/notfound\[dq]\f[R] for in
|
|
memory config only (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Clear fs cache of stale entries when altering config (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
copyurl: Add option to print resulting auto-filename (albertony)
|
|
.IP \[bu] 2
|
|
delete: Make \f[C]--rmdirs\f[R] obey the filters (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
docs - many fixes and reworks from edwardxml, albertony, pvalls, Ivan
|
|
Andreev, Evan Harris, buengese, Alexey Tabakman
|
|
.IP \[bu] 2
|
|
encoder/filename - add SCSU as tables (Klaus Post)
|
|
.IP \[bu] 2
|
|
Add multiple paths support to \f[C]--compare-dest\f[R] and
|
|
\f[C]--copy-dest\f[R] flag (K265)
|
|
.IP \[bu] 2
|
|
filter: Make \f[C]--exclude \[dq]dir/\[dq]\f[R] equivalent to
|
|
\f[C]--exclude \[dq]dir/**\[dq]\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
fshttp: Add DSCP support with \f[C]--dscp\f[R] for QoS with
|
|
differentiated services (Max Sum)
|
|
.IP \[bu] 2
|
|
lib/cache: Add Delete and DeletePrefix methods (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
lib/file
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make pre-allocate detect disk full errors and return them (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Don\[aq]t run preallocate concurrently (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Retry preallocate on EINTR (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
operations: Made copy and sync operations obey a RetryAfterError (Ankur
|
|
Gupta)
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add string alternatives for setting options over the rc (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]options/local\f[R] to see the options configured in the context
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]_config\f[R] parameter to set global config for just this rc
|
|
call (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement passing filter config with \f[C]_filter\f[R] parameter (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]fscache/clear\f[R] and \f[C]fscache/entries\f[R] to control the
|
|
fs cache (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Avoid +Inf value for speed in \f[C]core/stats\f[R] (albertony)
|
|
.IP \[bu] 2
|
|
Add a full set of stats to \f[C]core/stats\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Allow \f[C]fs=\f[R] params to be a JSON blob (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rcd: Added systemd notification during the \f[C]rclone rcd\f[R] command.
|
|
(Naveen Honest Raj)
|
|
.IP \[bu] 2
|
|
rmdirs: Make \f[C]--rmdirs\f[R] obey the filters (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
version: Show build tags and type of executable (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
install.sh: make it fail on download errors (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Fix excessive retries missing \f[C]--max-duration\f[R] timeout (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix crash when \f[C]--low-level-retries=0\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix failed token refresh on mounts created via the rc (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
fshttp: Fix bandwidth limiting after bad merge (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
lib/atexit
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Unregister interrupt handler once it has fired so users can interrupt
|
|
again (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix occasional failure to unmount with CTRL-C (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix deadlock calling Finalise while Run is running (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
lib/rest: Fix multipart uploads not stopping on context cancel (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow mounting to root directory on windows (albertony)
|
|
.IP \[bu] 2
|
|
Improved handling of relative paths on windows (albertony)
|
|
.IP \[bu] 2
|
|
Fix unicode issues with accented characters on macOS (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Docs: document the new FileSecurity option in WinFsp 2021 (albertony)
|
|
.IP \[bu] 2
|
|
Docs: add note about volume path syntax on windows (albertony)
|
|
.IP \[bu] 2
|
|
Fix caching of old directories after renaming them (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update cgofuse to the latest version to bring in macfuse 4 fix (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]--vfs-used-is-size\f[R] to report used space using recursive scan
|
|
(tYYGH)
|
|
.IP \[bu] 2
|
|
Don\[aq]t set modification time if it was already correct (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix Create causing windows explorer to truncate files on CTRL-C CTRL-V
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix modtimes not updating when writing via cache (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix modtimes changing by fractional seconds after upload (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix modtime set if \f[C]--vfs-cache-mode writes\f[R]/\f[C]full\f[R] and
|
|
no write (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Rename files in cache and cancel uploads on directory rename (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix directory renaming by renaming dirs cached in memory (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add flag \f[C]--local-no-preallocate\f[R] (David Sze)
|
|
.IP \[bu] 2
|
|
Make \f[C]nounc\f[R] an advanced option except on Windows (albertony)
|
|
.IP \[bu] 2
|
|
Don\[aq]t ignore preallocate disk full errors (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Cache
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--fs-cache-expire-duration\f[R] to control the fs cache (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add option to not encrypt data (Vesnyx)
|
|
.IP \[bu] 2
|
|
Log hash ok on upload (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add container public access level support.
|
|
(Manish Kumar)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix HTML files downloaded via cloudflare (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix transfers getting stuck on token expiry after API change (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Chunker
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Partially implement no-rename transactions (Maxwell Calman)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t stop server side copy if couldn\[aq]t read description (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Pass context on to drive SDK - to help with cancellation (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add polling for changes support (Robert Thomas)
|
|
.IP \[bu] 2
|
|
Make \f[C]--timeout 0\f[R] work properly (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Raise priority of rate limited message to INFO to make it more
|
|
noticeable (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fichier
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement copy & move (buengese)
|
|
.IP \[bu] 2
|
|
Implement public link (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement Shutdown method (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Close idle connections after \f[C]--ftp-idle-timeout\f[R] (1m by
|
|
default) (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make \f[C]--timeout 0\f[R] work properly (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--ftp-close-timeout\f[R] flag for use with awkward ftp servers
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Retry connections and logins on 421 errors (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Hdfs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix permissions for when directory is created (Lucas Messenger)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make \f[C]--timeout 0\f[R] work properly (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix \f[C]--s3-profile\f[R] which wasn\[aq]t working (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Close idle connections after \f[C]--sftp-idle-timeout\f[R] (1m by
|
|
default) (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \[dq]file not found\[dq] errors for read once servers (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix SetModTime stat failed: object not found with
|
|
\f[C]--sftp-set-modtime=false\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update github.com/ncw/swift to v2.0.0 (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement copying large objects (nguyenhuuluan434)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Union
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix crash when using epff policy (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix union attempting to update files on a read only file system (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Refactor to use fspath.SplitFs instead of fs.ParseRemote (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix initialisation broken in refactor (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for sharepoint with NTLM authentication (Rauno Ots)
|
|
.IP \[bu] 2
|
|
Make sharepoint-ntlm docs more consistent (Alex Chen)
|
|
.IP \[bu] 2
|
|
Improve terminology in sharepoint-ntlm docs (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Disable HTTP/2 for NTLM authentication (georne)
|
|
.IP \[bu] 2
|
|
Fix sharepoint-ntlm error 401 for parallel actions (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Check that purged directory really exists (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Yandex
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make \f[C]--timeout 0\f[R] work properly (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Zoho
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Replace client id - you will need to \f[C]rclone config reconnect\f[R]
|
|
after this (buengese)
|
|
.IP \[bu] 2
|
|
Add forgotten setupRegion() to NewFs - this finally fixes regions other
|
|
than EU (buengese)
|
|
.RE
|
|
.SS v1.54.1 - 2021-03-08
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.54.0...v1.54.1)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting: Fix --bwlimit when up or down is off (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
docs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix nesting of brackets and backticks in ftp docs (edwardxml)
|
|
.IP \[bu] 2
|
|
Fix broken link in sftp page (edwardxml)
|
|
.IP \[bu] 2
|
|
Fix typo in crypt.md (Romeo Kienzler)
|
|
.IP \[bu] 2
|
|
Changelog: Correct link to digitalis.io (Alex JOST)
|
|
.IP \[bu] 2
|
|
Replace #file-caching with #vfs-file-caching (Miron Veryanskiy)
|
|
.IP \[bu] 2
|
|
Convert bogus example link to code (edwardxml)
|
|
.IP \[bu] 2
|
|
Remove dead link from rc.md (edwardxml)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rc: Sync,copy,move: document createEmptySrcDirs parameter (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
lsjson: Fix unterminated JSON in the presence of errors (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix mount dropping on macOS by setting --daemon-timeout 10m (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Document simultaneous usage with the same cache shouldn\[aq]t be used
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Automatically raise upload cutoff to avoid spurious error (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix failed to create file system with application key limited to a
|
|
prefix (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Refer to Shared Drives instead of Team Drives (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add scopes to oauth request and optionally \[dq]members.read\[dq] (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix failed to create file system with folder level permissions policy
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix Wasabi HEAD requests returning stale data by using only 1 transport
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix shared_credentials_file auth (Dmitry Chepurovskiy)
|
|
.IP \[bu] 2
|
|
Add --s3-no-head to reducing costs docs (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Union
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix mkdir at root with remote:/ (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Zoho
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix custom client id\[aq]s (buengese)
|
|
.RE
|
|
.SS v1.54.0 - 2021-02-02
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.53.0...v1.54.0)
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Compression remote (experimental) (buengese)
|
|
.IP \[bu] 2
|
|
Enterprise File Fabric (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This work was sponsored by Storage Made
|
|
Easy (https://storagemadeeasy.com/)
|
|
.RE
|
|
.IP \[bu] 2
|
|
HDFS (Hadoop Distributed File System) (Yury Stankevich)
|
|
.IP \[bu] 2
|
|
Zoho workdrive (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Deglobalise the config (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Global config now read from the context
|
|
.IP \[bu] 2
|
|
This will enable passing of global config via the rc
|
|
.IP \[bu] 2
|
|
This work was sponsored by Digitalis (https://digitalis.io/)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]--bwlimit\f[R] for upload and download (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Obey bwlimit in http Transport for better limiting
|
|
.RE
|
|
.IP \[bu] 2
|
|
Enhance systemd integration (Hekmon)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
log level identification, manual activation with flag, automatic systemd
|
|
launch detection
|
|
.IP \[bu] 2
|
|
Don\[aq]t compile systemd log integration for non unix systems (Benjamin
|
|
Gustin)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add a \f[C]--download\f[R] flag to md5sum/sha1sum/hashsum to force
|
|
rclone to download and hash files locally (lostheli)
|
|
.IP \[bu] 2
|
|
Add \f[C]--progress-terminal-title\f[R] to print ETA to terminal title
|
|
(LaSombra)
|
|
.IP \[bu] 2
|
|
Make backend env vars show in help as the defaults for backend flags
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Raise minimum go version to go1.12 (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
dedupe
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--by-hash\f[R] to dedupe on content hash not file name (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--dedupe-mode list\f[R] to just list dupes, changing nothing
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add warning if used on a remote which can\[aq]t have duplicate names
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
fs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add Shutdown optional method for backends (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
When using \f[C]--files-from\f[R] check files concurrently (zhucan)
|
|
.IP \[bu] 2
|
|
Accumulate stats when using \f[C]--dry-run\f[R] (Ingo Weiss)
|
|
.IP \[bu] 2
|
|
Always show stats when using \f[C]--dry-run\f[R] or
|
|
\f[C]--interactive\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add support for flag \f[C]--no-console\f[R] on windows to hide the
|
|
console window (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
genautocomplete: Add support to output to stdout (Ingo)
|
|
.IP \[bu] 2
|
|
ncdu
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Highlight read errors instead of aborting (Claudio Bantaloukas)
|
|
.IP \[bu] 2
|
|
Add sort by average size in directory (Adam Pl\['a]nsk\['y])
|
|
.IP \[bu] 2
|
|
Add toggle option for average s3ize in directory - key \[aq]a\[aq] (Adam
|
|
Pl\['a]nsk\['y])
|
|
.IP \[bu] 2
|
|
Add empty folder flag into ncdu browser (Adam Pl\['a]nsk\['y])
|
|
.IP \[bu] 2
|
|
Add \f[C]!\f[R] (errror) and \f[C].\f[R] (unreadable) file flags to go
|
|
with \f[C]e\f[R] (empty) (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
obscure: Make \f[C]rclone osbcure -\f[R] ignore newline at end of line
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
operations
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add logs when need to upload files to set mod times (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Move and copy log name of the destination object in verbose (Adam
|
|
Pl\['a]nsk\['y])
|
|
.IP \[bu] 2
|
|
Add size if known to skipped items and JSON log (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Prefer actual listener address if using \[dq]:port\[dq] or
|
|
\[dq]addr:0\[dq] only (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add listener for finished jobs (Aleksandar Jankovic)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve ftp: Add options to enable TLS (Deepak Sah)
|
|
.IP \[bu] 2
|
|
serve http/webdav: Redirect requests to the base url without the / (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
serve restic: Implement object cache (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
stats: Add counter for deleted directories (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
sync: Only print \[dq]There was nothing to transfer\[dq] if no errors
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
webui
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Prompt user for updating webui if an update is available (Chaitanya
|
|
Bankanhal)
|
|
.IP \[bu] 2
|
|
Fix plugins initialization (negative0)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
fs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix nil pointer on copy & move operations directly to remote (Anagh
|
|
Kumar Baranwal)
|
|
.IP \[bu] 2
|
|
Fix parsing of ..
|
|
when joining remotes (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
log: Fix enabling systemd logging when using \f[C]--log-file\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
check
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make the error count match up in the log message (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
move: Fix data loss when source and destination are the same object
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
operations
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix \f[C]--cutof-mode\f[R] hard not cutting off immediately (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \f[C]--immutable\f[R] error message (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
sync
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix \f[C]--cutoff-mode\f[R] soft & cautious so it doesn\[aq]t end the
|
|
transfer early (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \f[C]--immutable\f[R] errors retrying many times (Nick Craig-Wood)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Docs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Many fixes and a rewrite of the filtering docs (edwardxml)
|
|
.IP \[bu] 2
|
|
Many spelling and grammar fixes (Josh Soref)
|
|
.IP \[bu] 2
|
|
Doc fixes for commands delete, purge, rmdir, rmdirs and mount
|
|
(albertony)
|
|
.IP \[bu] 2
|
|
And thanks to these people for many doc fixes too numerous to list
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ameer Dawood, Antoine GIRARD, Bob Bagwill, Christopher Stewart
|
|
.IP \[bu] 2
|
|
CokeMine, David, Dov Murik, Durval Menezes, Evan Harris, gtorelly
|
|
.IP \[bu] 2
|
|
Ilyess Bachiri, Janne Johansson, Kerry Su, Marcin Zelent,
|
|
.IP \[bu] 2
|
|
Martin Michlmayr, Milly, S\[u01A1]n Tr\[u1EA7]n-Nguy\[u1EC5]n
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update systemd status with cache stats (Hekmon)
|
|
.IP \[bu] 2
|
|
Disable bazil/fuse based mount on macOS (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make \f[C]rclone mount\f[R] actually run \f[C]rclone cmount\f[R] under
|
|
macOS (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Implement mknod to make NFS file creation work (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make sure we don\[aq]t call umount more than once (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
More user friendly mounting as network drive on windows (albertony)
|
|
.IP \[bu] 2
|
|
Detect if uid or gid are set in same option string: -o uid=123,gid=456
|
|
(albertony)
|
|
.IP \[bu] 2
|
|
Don\[aq]t attempt to unmount if fs has been destroyed already (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix virtual entries causing deleted files to still appear (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \[dq]file already exists\[dq] error for stale cache files (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix file leaks with \f[C]--vfs-cache-mode\f[R] full and
|
|
\f[C]--buffer-size 0\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix invalid cache path on windows when using :backend: as remote
|
|
(albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Continue listing files/folders when a circular symlink is detected
|
|
(Manish Gupta)
|
|
.IP \[bu] 2
|
|
New flag \f[C]--local-zero-size-links\f[R] to fix sync on some virtual
|
|
filesystems (Riccardo Iaconelli)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for service principals (James Lim)
|
|
.IP \[bu] 2
|
|
Add support for managed identities (Brad Ackerman)
|
|
.IP \[bu] 2
|
|
Add examples for access tier (Bob Pusateri)
|
|
.IP \[bu] 2
|
|
Utilize the streaming capabilities from the SDK for multipart uploads
|
|
(Denis Neuling)
|
|
.IP \[bu] 2
|
|
Fix setting of mime types (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix crash when listing outside a SAS URL\[aq]s root (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Delete archive tier blobs before update if
|
|
\f[C]--azureblob-archive-tier-delete\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix crash on startup (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix memory usage by upgrading the SDK to v0.13.0 and implementing a
|
|
TransferManager (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Require go1.14+ to compile due to SDK changes (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make NewObject use less expensive API calls (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This will improve \f[C]--files-from\f[R] and \f[C]restic serve\f[R] in
|
|
particular
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fixed crash on an empty file name (lluuaapp)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix NewObject for files that differ in case (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix finding directories in a case insentive way (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Chunker
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Skip long local hashing, hash in-transit (fixes) (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Set Features ReadMimeType to false as Object.MimeType not supported
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix case-insensitive NewObject, test metadata detection (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]rclone backend copyid\f[R] command for copying files by
|
|
ID (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Added flag \f[C]--drive-stop-on-download-limit\f[R] to stop transfers
|
|
when the download limit is exceeded (Anagh Kumar Baranwal)
|
|
.IP \[bu] 2
|
|
Implement CleanUp workaround for team drives (buengese)
|
|
.IP \[bu] 2
|
|
Allow shortcut resolution and creation to be retried (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Log that emptying the trash can take some time (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add xdg office icons to xdg desktop files (Pau Rodriguez-Estivill)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for viewing shared files and folders (buengese)
|
|
.IP \[bu] 2
|
|
Enable short lived access tokens (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement IDer on Objects so \f[C]rclone lsf\f[R] etc can read the IDs
|
|
(buengese)
|
|
.IP \[bu] 2
|
|
Set Features ReadMimeType to false as Object.MimeType not supported
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make malformed_path errors from too long files not retriable (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Test file name length before upload to fix upload loop (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fichier
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Set Features ReadMimeType to true as Object.MimeType is supported (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--ftp-disable-msld\f[R] option to ignore MLSD for really old
|
|
servers (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make \f[C]--tpslimit apply\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Storage class object header support (Laurens Janssen)
|
|
.IP \[bu] 2
|
|
Fix anonymous client to use rclone\[aq]s HTTP client (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix
|
|
\f[C]Entry doesn\[aq]t belong in directory \[dq]\[dq] (same as directory) - ignoring\f[R]
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Googlephotos
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New flag \f[C]--gphotos-include-archived\f[R] to show archived photos as
|
|
well (Nicolas Rueff)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t erroneously report support for writing mime types (buengese)
|
|
.IP \[bu] 2
|
|
Add support for Telia Cloud (Patrik Nordl\['e]n)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mailru
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Accept special folders eg camera-upload (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Avoid prehashing of large local files (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Fix uploads after recent changes on server (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Fix range requests after June 2020 changes on server (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Remove deprecated protocol quirks (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Memory
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix setting of mime types (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for China region operated by 21vianet and other regional
|
|
suppliers (NyaMisty)
|
|
.IP \[bu] 2
|
|
Warn on gateway timeout errors (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fall back to normal copy if server-side copy unavailable (Alex Chen)
|
|
.IP \[bu] 2
|
|
Fix server-side copy completely disabled on OneDrive for Business (Cnly)
|
|
.IP \[bu] 2
|
|
(business only) workaround to replace existing file on server-side copy
|
|
(Alex Chen)
|
|
.IP \[bu] 2
|
|
Enhance link creation with expiry, scope, type and password (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove % and # from the set of encoded characters (Alex Chen)
|
|
.IP \[bu] 2
|
|
Support addressing site by server-relative URL (kice)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Opendrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix finding directories in a case insensitive way (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix setting of mime types (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Premiumizeme
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix finding directories in a case insensitive way (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Qingstor
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix error propagation in CleanUp (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix rclone cleanup (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Added \f[C]--s3-disable-http2\f[R] to disable http/2 (Anagh Kumar
|
|
Baranwal)
|
|
.IP \[bu] 2
|
|
Complete SSE-C implementation (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix hashes on small files with AWS:KMS and SSE-C (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]--s3-no-head parameter\f[R] to minimise transactions on upload
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update docs with a Reducing Costs section (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Added error handling for error code 429 indicating too many requests
|
|
(Anagh Kumar Baranwal)
|
|
.IP \[bu] 2
|
|
Add requester pays option (kelv)
|
|
.IP \[bu] 2
|
|
Fix copy multipart with v2 auth failing with
|
|
\[aq]SignatureDoesNotMatch\[aq] (Louis Koo)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow cert based auth via optional pubkey (Stephen Harris)
|
|
.IP \[bu] 2
|
|
Allow user to optionally check server hosts key to add security (Stephen
|
|
Harris)
|
|
.IP \[bu] 2
|
|
Defer asking for user passwords until the SSH connection succeeds
|
|
(Stephen Harris)
|
|
.IP \[bu] 2
|
|
Remember entered password in AskPass mode (Stephen Harris)
|
|
.IP \[bu] 2
|
|
Implement Shutdown method (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement keyboard interactive authentication (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make \f[C]--tpslimit\f[R] apply (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement \f[C]--sftp-use-fstat\f[R] for unusual SFTP servers (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Sugarsync
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix NewObject for files that differ in case (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix finding directories in a case insentive way (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix deletion of parts of Static Large Object (SLO) (Nguy\[u1EC5]n
|
|
H\[u1EEF]u Lu\[^a]n)
|
|
.IP \[bu] 2
|
|
Ensure partially uploaded large files are uploaded unless
|
|
\f[C]--swift-leave-parts-on-error\f[R] (Nguy\[u1EC5]n H\[u1EEF]u
|
|
Lu\[^a]n)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Tardigrade
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Upgrade to uplink v1.4.1 (Caleb Case)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Updated docs to show streaming to nextcloud is working (Durval Menezes)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Yandex
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Set Features WriteMimeType to false as Yandex ignores mime types (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.SS v1.53.4 - 2021-01-20
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.53.3...v1.53.4)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting: Fix data race in Transferred() (Maciej Zimnoch)
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Stop tagged releases making a current beta (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Upgrade docker buildx action (Matteo Pietro Dazzi)
|
|
.IP \[bu] 2
|
|
Add -buildmode to cross-compile.go (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix docker build by upgrading ilteoood/docker_buildx (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Revert GitHub actions brew fix since this is now fixed (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix brew install --cask syntax for macOS build (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update nfpm syntax to fix build of .deb/.rpm packages (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix for Windows build errors (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
fs: Parseduration: fixed tests to use UTC time (Ankur Gupta)
|
|
.IP \[bu] 2
|
|
fshttp: Prevent overlap of HTTP headers in logs (Nathan Collins)
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix core/command giving 500 internal error (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add Copy method to rc.Params (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix 500 error when marshalling errors from core/command (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
plugins: Create plugins files only if webui is enabled.
|
|
(negative0)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve http: Fix serving files of unknown length (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
serve sftp: Fix authentication on one connection blocking others (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add optional \f[C]brew\f[R] tag to throw an error when using mount in
|
|
the binaries installed via Homebrew (Anagh Kumar Baranwal)
|
|
.IP \[bu] 2
|
|
Add \[dq].\[dq] and \[dq]..\[dq] to directories to match cmount and
|
|
expectations (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make cache dir absolute before using it to fix path too long errors
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Chunker
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Improve detection of incompatible metadata (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix server side copy of large objects (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix token renewer to fix long uploads (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix token refresh failed: is not a regular file error (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Only use SHA1 hashes in EU region (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Sharefile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Undo Fix backend due to API swapping integers for strings (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix Open Range requests to fix 4shared mount (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \[dq]Depth: 0\[dq] to GET requests to fix bitrix (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.53.3 - 2020-11-19
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.53.2...v1.53.3)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
random: Fix incorrect use of math/rand instead of crypto/rand
|
|
CVE-2020-28924 (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Passwords you have generated with \f[C]rclone config\f[R] may be
|
|
insecure
|
|
.IP \[bu] 2
|
|
See issue #4783 (https://github.com/rclone/rclone/issues/4783) for more
|
|
details and a checking tool
|
|
.RE
|
|
.IP \[bu] 2
|
|
random: Seed math/rand in one place with crypto strong seed (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix vfs/refresh calls with fs= parameter (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Sharefile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix backend due to API swapping integers for strings (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.53.2 - 2020-10-26
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.53.1...v1.53.2)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
acounting
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix incorrect speed and transferTime in core/stats (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Stabilize display order of transfers on Windows (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
operations
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix use of --suffix without --backup-dir (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix spurious \[dq]--checksum is in use but the source and destination
|
|
have no hashes in common\[dq] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Work around GitHub actions brew problem (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Stop using set-env and set-path in the GitHub actions (Nick Craig-Wood)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
mount2: Fix the swapped UID / GID values (Russell Cattelan)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Detect and recover from a file being removed externally from the cache
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix a deadlock vulnerability in downloaders.Close (Leo Luan)
|
|
.IP \[bu] 2
|
|
Fix a race condition in retryFailedResets (Leo Luan)
|
|
.IP \[bu] 2
|
|
Fix missed concurrency control between some item operations and reset
|
|
(Leo Luan)
|
|
.IP \[bu] 2
|
|
Add exponential backoff during ENOSPC retries (Leo Luan)
|
|
.IP \[bu] 2
|
|
Add a missed update of used cache space (Leo Luan)
|
|
.IP \[bu] 2
|
|
Fix --no-modtime to not attempt to set modtimes (as documented) (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix sizes and syncing with --links option on Windows (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Chunker
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Disable ListR to fix missing files on GDrive (workaround) (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Fix upload over crypt (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fichier
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Increase maximum file size from 100GB to 300GB (gyutw)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Remove clientSecret from config when upgrading to token based
|
|
authentication (buengese)
|
|
.IP \[bu] 2
|
|
Avoid double url escaping of device/mountpoint (albertony)
|
|
.IP \[bu] 2
|
|
Remove DirMove workaround as it\[aq]s not required anymore - also
|
|
(buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mailru
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix uploads after recent changes on server (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Fix range requests after june changes on server (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix disk usage for sharepoint (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add missing regions for AWS (Anagh Kumar Baranwal)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Seafile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix accessing libraries > 2GB on 32 bit systems (Muffin King)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Always convert the checksum to lower case (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Union
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Create root directories if none exist (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.53.1 - 2020-09-13
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.53.0...v1.53.1)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting: Remove new line from end of --stats-one-line display (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
check
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add back missing --download flag (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix docs (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
docs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Note --log-file does append (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add full stops for consistency in rclone --help (edwardxml)
|
|
.IP \[bu] 2
|
|
Add Tencent COS to s3 provider list (wjielai)
|
|
.IP \[bu] 2
|
|
Updated mount command to reflect that it requires Go 1.13 or newer (Evan
|
|
Harris)
|
|
.IP \[bu] 2
|
|
jottacloud: Mention that uploads from local disk will not need to cache
|
|
files to disk for md5 calculation (albertony)
|
|
.IP \[bu] 2
|
|
Fix formatting of rc docs page (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Include vendor tar ball in release and fix startdev (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \[dq]Illegal instruction\[dq] error for ARMv6 builds (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix architecture name in ARMv7 build (Nick Craig-Wood)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix spurious error \[dq]vfs cache: failed to _ensure cache EOF\[dq]
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Log an ERROR if we fail to set the file to be sparse (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Log an ERROR if we fail to set the file to be sparse (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Re-adds special oauth help text (Tim Gallant)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Opendrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Do not retry 400 errors (Evan Harris)
|
|
.RE
|
|
.SS v1.53.0 - 2020-09-02
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.52.0...v1.53.0)
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The VFS
|
|
layer (https://rclone.org/commands/rclone_mount/#vfs-virtual-file-system)
|
|
was heavily reworked for this release - see below for more details
|
|
.IP \[bu] 2
|
|
Interactive mode -i/--interactive (https://rclone.org/docs/#interactive)
|
|
for destructive operations (fishbullet)
|
|
.IP \[bu] 2
|
|
Add
|
|
--bwlimit-file (https://rclone.org/docs/#bwlimit-file-bandwidth-spec)
|
|
flag to limit speeds of individual file transfers (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Transfers are sorted by start time in the stats and progress output (Max
|
|
Sum)
|
|
.IP \[bu] 2
|
|
Make sure backends expand \f[C]\[ti]\f[R] and environment vars in file
|
|
names they use (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add --refresh-times (https://rclone.org/docs/#refresh-times) flag to set
|
|
modtimes on hashless backends (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Remove vendor directory in favour of Go modules (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Build with go1.15.x by default (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Drop macOS 386 build as it is no longer supported by go1.15 (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add ARMv7 to the supported builds (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Enable \f[C]rclone cmount\f[R] on macOS (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make rclone build with gccgo (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make rclone build with wasm (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Change beta numbering to be semver compatible (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add file properties and icon to Windows executable (albertony)
|
|
.IP \[bu] 2
|
|
Add experimental interface for integrating rclone into browsers (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
lib: Add file name compression (Klaus Post)
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow installation and use of plugins and test plugins with rclone-webui
|
|
(Chaitanya Bankanhal)
|
|
.IP \[bu] 2
|
|
Add reverse proxy pluginsHandler for serving plugins (Chaitanya
|
|
Bankanhal)
|
|
.IP \[bu] 2
|
|
Add \f[C]mount/listmounts\f[R] option for listing current mounts
|
|
(Chaitanya Bankanhal)
|
|
.IP \[bu] 2
|
|
Add \f[C]operations/uploadfile\f[R] to upload a file through rc using
|
|
encoding multipart/form-data (Chaitanya Bankanhal)
|
|
.IP \[bu] 2
|
|
Add \f[C]core/command\f[R] to execute rclone terminal commands.
|
|
(Chaitanya Bankanhal)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]rclone check\f[R]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add reporting of filenames for same/missing/changed (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make check command obey
|
|
\f[C]--dry-run\f[R]/\f[C]-i\f[R]/\f[C]--interactive\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make check do \f[C]--checkers\f[R] files concurrently (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Retry downloads if they fail when using the \f[C]--download\f[R] flag
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make it show stats by default (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]rclone obscure\f[R]: Allow obscure command to accept password on
|
|
STDIN (David Ibarra)
|
|
.IP \[bu] 2
|
|
\f[C]rclone config\f[R]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Set RCLONE_CONFIG_DIR for use in config files and subprocesses (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Reject remote names starting with a dash.
|
|
(jtagcat)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]rclone cryptcheck\f[R]: Add reporting of filenames for
|
|
same/missing/changed (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
\f[C]rclone dedupe\f[R]: Make it obey the \f[C]--size-only\f[R] flag for
|
|
duplicate detection (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
\f[C]rclone link\f[R]: Add \f[C]--expire\f[R] and \f[C]--unlink\f[R]
|
|
flags (Roman Kredentser)
|
|
.IP \[bu] 2
|
|
\f[C]rclone mkdir\f[R]: Warn when using mkdir on remotes which can\[aq]t
|
|
have empty directories (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
\f[C]rclone rc\f[R]: Allow JSON parameters to simplify command line
|
|
usage (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
\f[C]rclone serve ftp\f[R]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t compile on < go1.13 after dependency update (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add error message if auth proxy fails (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Use refactored goftp.io/server library for binary shrink (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]rclone serve restic\f[R]: Expose interfaces so that rclone can be
|
|
used as a library from within restic (Jack)
|
|
.IP \[bu] 2
|
|
\f[C]rclone sync\f[R]: Add \f[C]--track-renames-strategy leaf\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
\f[C]rclone touch\f[R]: Add ability to set nanosecond resolution times
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
\f[C]rclone tree\f[R]: Remove \f[C]-i\f[R] shorthand for
|
|
\f[C]--noindent\f[R] as it conflicts with
|
|
\f[C]-i\f[R]/\f[C]--interactive\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix documentation for \f[C]speed\f[R]/\f[C]speedAvg\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix elapsed time not show actual time since beginning (Chaitanya
|
|
Bankanhal)
|
|
.IP \[bu] 2
|
|
Fix deadlock in stats printing (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix file handle leak in GitHub release tool (Garrett Squire)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]rclone check\f[R]: Fix successful retries with \f[C]--download\f[R]
|
|
counting errors (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
\f[C]rclone dedupe\f[R]: Fix logging to be easier to understand (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Warn macOS users that mount implementation is changing (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
to test the new implementation use \f[C]rclone cmount\f[R] instead of
|
|
\f[C]rclone mount\f[R]
|
|
.IP \[bu] 2
|
|
this is because the library rclone uses has dropped macOS support
|
|
.RE
|
|
.IP \[bu] 2
|
|
rc interface
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add call for unmount all (Chaitanya Bankanhal)
|
|
.IP \[bu] 2
|
|
Make \f[C]mount/mount\f[R] remote control take vfsOpt option (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add mountOpt to \f[C]mount/mount\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add VFS and Mount options to \f[C]mount/listmounts\f[R] (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Catch panics in cgofuse initialization and turn into error messages
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Always supply stat information in Readdir (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add support for reading unknown length files using direct IO (Windows)
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix On Windows don\[aq]t add \f[C]-o uid/gid=-1\f[R] if user supplies
|
|
\f[C]-o uid/gid\f[R].
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix macOS losing directory contents in cmount (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix volume name broken in recent refactor (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement partial reads for \f[C]--vfs-cache-mode full\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--vfs-writeback\f[R] option to delay writes back to cloud
|
|
storage (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--vfs-read-ahead\f[R] parameter for use with
|
|
\f[C]--vfs-cache-mode full\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Restart pending uploads on restart of the cache (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Support synchronous cache space recovery upon ENOSPC (Leo Luan)
|
|
.IP \[bu] 2
|
|
Allow ReadAt and WriteAt to run concurrently with themselves (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Change modtime of file before upload to current (Rob Calistri)
|
|
.IP \[bu] 2
|
|
Recommend \f[C]--vfs-cache-modes writes\f[R] on backends which can\[aq]t
|
|
stream (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add an optional \f[C]fs\f[R] parameter to vfs rc methods (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix errors when using > 260 char files in the cache in Windows (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix renaming of items while they are being uploaded (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix very high load caused by slow directory listings (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix renamed files not being uploaded with
|
|
\f[C]--vfs-cache-mode minimal\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix directory locking caused by slow directory listings (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix saving from chrome without \f[C]--vfs-cache-mode writes\f[R] (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--local-no-updated\f[R] to provide a consistent view of
|
|
changing objects (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--local-no-set-modtime\f[R] option to prevent modtime changes
|
|
(tyhuber1)
|
|
.IP \[bu] 2
|
|
Fix race conditions updating and reading Object metadata (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Cache
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make any created backends be cached to fix rc problems (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix dedupe on caches wrapping drives (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--crypt-server-side-across-configs\f[R] flag (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make any created backends be cached to fix rc problems (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Alias
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make any created backends be cached to fix rc problems (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t compile on < go1.13 after dependency update (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement server-side copy for files > 5GB (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Cancel in progress multipart uploads and copies on rclone exit (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Note that b2\[aq]s encoding now allows \ but rclone\[aq]s hasn\[aq]t
|
|
changed (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix transfers when using download_url (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement rclone cleanup (buengese)
|
|
.IP \[bu] 2
|
|
Cancel in progress multipart uploads and copies on rclone exit (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Allow authentication with access token (David)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Chunker
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make any created backends be cached to fix rc problems (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]rclone backend drives\f[R] to list shared drives (teamdrives)
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement \f[C]rclone backend untrash\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Work around drive bug which didn\[aq]t set modtime of copied docs (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Added \f[C]--drive-starred-only\f[R] to only show starred files (Jay
|
|
McEntire)
|
|
.IP \[bu] 2
|
|
Deprecate \f[C]--drive-alternate-export\f[R] as it is no longer needed
|
|
(themylogin)
|
|
.IP \[bu] 2
|
|
Fix duplication of Google docs on server-side copy (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \[dq]panic: send on closed channel\[dq] when recycling dir entries
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add copyright detector info in limitations section in the docs (Alex
|
|
Guerrero)
|
|
.IP \[bu] 2
|
|
Fix \f[C]rclone link\f[R] by removing expires parameter (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fichier
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Detect Flood detected: IP Locked error and sleep for 30s (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add explicit TLS support (Heiko Bornholdt)
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--dump bodies\f[R] and \f[C]--dump auth\f[R] for
|
|
debugging (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix interoperation with pure-ftpd (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for anonymous access (Kai L\[:u]ke)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Bring back legacy authentication for use with whitelabel versions
|
|
(buengese)
|
|
.IP \[bu] 2
|
|
Switch to new api root - also implement a very ugly workaround for the
|
|
DirMove failures (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rework cancel of multipart uploads on rclone exit (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement rclone cleanup (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--onedrive-no-versions\f[R] flag to remove old versions (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]rclone link\f[R] for public link creation (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Qingstor
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Cancel in progress multipart uploads on rclone exit (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Preserve metadata when doing multipart copy (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Cancel in progress multipart uploads and copies on rclone exit (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]rclone link\f[R] for public link sharing (Roman Kredentser)
|
|
.IP \[bu] 2
|
|
Add \f[C]rclone backend restore\f[R] command to restore objects from
|
|
GLACIER (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]rclone cleanup\f[R] and \f[C]rclone backend cleanup\f[R] to
|
|
clean unfinished multipart uploads (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]rclone backend list-multipart-uploads\f[R] to list unfinished
|
|
multipart uploads (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--s3-max-upload-parts\f[R] support (Kamil Trzci\[u0144]ski)
|
|
.IP \[bu] 2
|
|
Add \f[C]--s3-no-check-bucket\f[R] for minimising rclone transactions
|
|
and perms (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--s3-profile\f[R] and \f[C]--s3-shared-credentials-file\f[R]
|
|
options (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Use regional s3 us-east-1 endpoint (David)
|
|
.IP \[bu] 2
|
|
Add Scaleway provider (Vincent Feltz)
|
|
.IP \[bu] 2
|
|
Update IBM COS endpoints (Egor Margineanu)
|
|
.IP \[bu] 2
|
|
Reduce the default \f[C]--s3-copy-cutoff\f[R] to < 5GB for Backblaze S3
|
|
compatibility (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix detection of bucket existing (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use the absolute path instead of the relative path for listing for
|
|
improved compatibility (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--sftp-subsystem\f[R] and \f[C]--sftp-server-command\f[R]
|
|
options (aus)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix dangling large objects breaking the listing (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix purge not deleting directory markers (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix update multipart object removing all of its own parts (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix missing hash from object returned from upload (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Tardigrade
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Upgrade to uplink v1.2.0 (Kaloyan Raev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Union
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix writing with the all policy (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix directory creation with 4shared (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.52.3 - 2020-08-07
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.52.2...v1.52.3)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
docs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Disable smart typography (e.g.
|
|
en-dash) in MANUAL.* and man page (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update install.md to reflect minimum Go version (Evan Harris)
|
|
.IP \[bu] 2
|
|
Update install from source instructions (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
make_manual: Support SOURCE_DATE_EPOCH (Morten Linderud)
|
|
.RE
|
|
.IP \[bu] 2
|
|
log: Fix --use-json-log going to stderr not --log-file on Windows (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
serve dlna: Fix file list on Samsung Series 6+ TVs (Matteo Pietro Dazzi)
|
|
.IP \[bu] 2
|
|
sync: Fix deadlock with --track-renames-strategy modtime (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Cache
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix moveto/copyto remote:file remote:file2 (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Stop using root_folder_id as a cache (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make dangling shortcuts appear in listings (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Drop \[dq]Disabling ListR\[dq] messages down to debug (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Workaround and policy for Google Drive API (Dmitry Ustalov)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add note to docs about home vs root directory selection (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix reverting to Copy when Move would have worked (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Avoid comma rendered in URL in onedrive.md (Kevin)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix oauth on European region \[dq]eapi.pcloud.com\[dq] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix bucket Region auto detection when Region unset in config (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.SS v1.52.2 - 2020-06-24
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.52.1...v1.52.2)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix docker release build action (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix custom timezone in Docker image (NoLooseEnds)
|
|
.RE
|
|
.IP \[bu] 2
|
|
check: Fix misleading message which printed errors instead of
|
|
differences (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
errors: Add WSAECONNREFUSED and more to the list of retriable Windows
|
|
errors (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
rcd: Fix incorrect prometheus metrics (Gary Kim)
|
|
.IP \[bu] 2
|
|
serve restic: Fix flags so they use environment variables (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
serve webdav: Fix flags so they use environment variables (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
sync: Fix --track-renames-strategy modtime (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix not being able to delete a directory with a trashed shortcut (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix creating a directory inside a shortcut (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix --drive-impersonate with cached root_folder_id (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix SSH key PEM loading (Zac Rubin)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Speed up deletes by not retrying segment container deletes (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Tardigrade
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Upgrade to uplink v1.1.1 (Caleb Case)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix free/used display for rclone about/df for certain backends (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.SS v1.52.1 - 2020-06-10
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.52.0...v1.52.1)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
lib/file: Fix SetSparse on Windows 7 which fixes downloads of files >
|
|
250MB (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update go.mod to go1.14 to enable -mod=vendor build (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove quicktest from Dockerfile (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Build Docker images with GitHub actions (Matteo Pietro Dazzi)
|
|
.IP \[bu] 2
|
|
Update Docker build workflows (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Set user_allow_other in /etc/fuse.conf in the Docker image (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix xgo build after go1.14 go.mod update (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
docs
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add link to source and modified time to footer of every page (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove manually set dates and use git dates instead (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Minor tense, punctuation, brevity and positivity changes for the home
|
|
page (edwardxml)
|
|
.IP \[bu] 2
|
|
Remove leading slash in page reference in footer when present (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Note commands which need obscured input in the docs (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
obscure: Write more help as we are referencing it elsewhere (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix OS vs Unix path confusion - fixes ChangeNotify on Windows (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix missing items when listing using --fast-list / ListR (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Putio
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix panic on Object.Open (Cenk Alti)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix upload of single files into buckets without create permission (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix --header-upload (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Tardigrade
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix listing bug by upgrading to v1.0.7
|
|
.IP \[bu] 2
|
|
Set UserAgent to rclone (Caleb Case)
|
|
.RE
|
|
.SS v1.52.0 - 2020-05-27
|
|
.PP
|
|
Special thanks to Martin Michlmayr for proof reading and correcting all
|
|
the docs and Edward Barker for helping re-write the front page.
|
|
.PP
|
|
See commits (https://github.com/rclone/rclone/compare/v1.51.0...v1.52.0)
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Tardigrade (https://rclone.org/tardigrade/) backend for use with
|
|
storj.io (Caleb Case)
|
|
.IP \[bu] 2
|
|
Union (https://rclone.org/union/) re-write to have multiple writable
|
|
remotes (Max Sum)
|
|
.IP \[bu] 2
|
|
Seafile for Seafile server (Fred \[at]creativeprojects)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
backend: command for backend-specific commands (see backends) (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
cachestats: Deprecate in favour of \f[C]rclone backend stats cache:\f[R]
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
dbhashsum: Deprecate in favour of \f[C]rclone hashsum DropboxHash\f[R]
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--header-download\f[R] and \f[C]--header-upload\f[R] flags for
|
|
setting HTTP headers when uploading/downloading (Tim Gallant)
|
|
.IP \[bu] 2
|
|
Add \f[C]--header\f[R] flag to add HTTP headers to every HTTP
|
|
transaction (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--check-first\f[R] to do all checking before starting transfers
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--track-renames-strategy\f[R] for configurable matching
|
|
criteria for \f[C]--track-renames\f[R] (Bernd Schoolmann)
|
|
.IP \[bu] 2
|
|
Add \f[C]--cutoff-mode\f[R] hard,soft,cautious (Shing Kit Chan &
|
|
Franklyn Tackitt)
|
|
.IP \[bu] 2
|
|
Filter flags (e.g.
|
|
\f[C]--files-from -\f[R]) can read from stdin (fishbullet)
|
|
.IP \[bu] 2
|
|
Add \f[C]--error-on-no-transfer\f[R] option (Jon Fautley)
|
|
.IP \[bu] 2
|
|
Implement \f[C]--order-by xxx,mixed\f[R] for copying some small and some
|
|
big files (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Allow \f[C]--max-backlog\f[R] to be negative meaning as large as
|
|
possible (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Added \f[C]--no-unicode-normalization\f[R] flag to allow Unicode
|
|
filenames to remain unique (Ben Zenker)
|
|
.IP \[bu] 2
|
|
Allow \f[C]--min-age\f[R]/\f[C]--max-age\f[R] to take a date as well as
|
|
a duration (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add rename statistics for file and directory renames (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add statistics output to JSON log (reddi)
|
|
.IP \[bu] 2
|
|
Make stats be printed on non-zero exit code (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
When running \f[C]--password-command\f[R] allow use of stdin
|
|
(S\['e]bastien Gross)
|
|
.IP \[bu] 2
|
|
Stop empty strings being a valid remote path (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
accounting: support WriterTo for less memory copying (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update to use go1.14 for the build (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]-trimpath\f[R] to release build for reproduceable builds (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove GOOS and GOARCH from Dockerfile (Brandon Philips)
|
|
.RE
|
|
.IP \[bu] 2
|
|
config
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fsync the config file after writing to save more reliably (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--obscure\f[R] and \f[C]--no-obscure\f[R] flags to
|
|
\f[C]config create\f[R]/\f[C]update\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make \f[C]config show\f[R] take \f[C]remote:\f[R] as well as
|
|
\f[C]remote\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
copyurl: Add \f[C]--no-clobber\f[R] flag (Denis)
|
|
.IP \[bu] 2
|
|
delete: Added \f[C]--rmdirs\f[R] flag to delete directories as well
|
|
(Kush)
|
|
.IP \[bu] 2
|
|
filter: Added \f[C]--files-from-raw\f[R] flag (Ankur Gupta)
|
|
.IP \[bu] 2
|
|
genautocomplete: Add support for fish shell (Matan Rosenberg)
|
|
.IP \[bu] 2
|
|
log: Add support for syslog LOCAL facilities (Patryk Jakuszew)
|
|
.IP \[bu] 2
|
|
lsjson: Add \f[C]--hash-type\f[R] parameter and use it in lsf to speed
|
|
up hashing (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]-o\f[R]/\f[C]--opt\f[R] and \f[C]-a\f[R]/\f[C]--arg\f[R] for
|
|
more structured input (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement \f[C]backend/command\f[R] for running backend-specific
|
|
commands remotely (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]mount/mount\f[R] command for starting \f[C]rclone mount\f[R]
|
|
via the API (Chaitanya)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rcd: Add Prometheus metrics support (Gary Kim)
|
|
.IP \[bu] 2
|
|
serve http
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Added a \f[C]--template\f[R] flag for user defined markup (calistri)
|
|
.IP \[bu] 2
|
|
Add Last-Modified headers to files and directories (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve sftp: Add support for multiple host keys by repeating
|
|
\f[C]--key\f[R] flag (Maxime Suret)
|
|
.IP \[bu] 2
|
|
touch: Add \f[C]--localtime\f[R] flag to make \f[C]--timestamp\f[R]
|
|
localtime not UTC (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Restore \[dq]Max number of stats groups reached\[dq] log line
|
|
(Micha\[/l] Matczuk)
|
|
.IP \[bu] 2
|
|
Correct exitcode on Transfer Limit Exceeded flag.
|
|
(Anuar Serdaliyev)
|
|
.IP \[bu] 2
|
|
Reset bytes read during copy retry (Ankur Gupta)
|
|
.IP \[bu] 2
|
|
Fix race clearing stats (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
copy: Only create empty directories when they don\[aq]t exist on the
|
|
remote (Ishuah Kariuki)
|
|
.IP \[bu] 2
|
|
dedupe: Stop dedupe deleting files with identical IDs (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
oauth
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use custom http client so that \f[C]--no-check-certificate\f[R] is
|
|
honored by oauth token fetch (Mark Spieth)
|
|
.IP \[bu] 2
|
|
Replace deprecated oauth2.NoContext (Lars Lehtonen)
|
|
.RE
|
|
.IP \[bu] 2
|
|
operations
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix setting the timestamp on Windows for multithread copy (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make rcat obey \f[C]--ignore-checksum\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make \f[C]--max-transfer\f[R] more accurate (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix dropped error (Lars Lehtonen)
|
|
.IP \[bu] 2
|
|
Fix misplaced http server config (Xiaoxing Ye)
|
|
.IP \[bu] 2
|
|
Disable duplicate log (ElonH)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve dlna
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Cds: don\[aq]t specify childCount at all when unknown (Dan Walters)
|
|
.IP \[bu] 2
|
|
Cds: use modification time as date in dlna metadata (Dan Walters)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve restic: Fix tests after restic project removed vendoring (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
sync
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix incorrect \[dq]nothing to transfer\[dq] message using
|
|
\f[C]--delete-before\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Only create empty directories when they don\[aq]t exist on the remote
|
|
(Ishuah Kariuki)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--async-read\f[R] flag to disable asynchronous reads (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Ignore \f[C]--allow-root\f[R] flag with a warning as it has been removed
|
|
upstream (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Warn if \f[C]--allow-non-empty\f[R] used on Windows and clarify docs
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Constrain to go1.13 or above otherwise bazil.org/fuse fails to compile
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix fail because of too long volume name (evileye)
|
|
.IP \[bu] 2
|
|
Report 1PB free for unknown disk sizes (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Map more rclone errors into file systems errors (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix disappearing cwd problem (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Use ReaddirPlus on Windows to improve directory listing performance
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Send a hint as to whether the filesystem is case insensitive or not
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add rc command \f[C]mount/types\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Change maximum leaf name length to 1024 bytes (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--vfs-read-wait\f[R] and \f[C]--vfs-write-wait\f[R] flags to
|
|
control time waiting for a sequential read/write (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Change default \f[C]--vfs-read-wait\f[R] to 20ms (it was 5ms and not
|
|
configurable) (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make \f[C]df\f[R] output more consistent on a rclone mount.
|
|
(Yves G)
|
|
.IP \[bu] 2
|
|
Report 1PB free for unknown disk sizes (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix race condition caused by unlocked reading of Dir.path (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make File lock and Dir lock not overlap to avoid deadlock (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement lock ordering between File and Dir to eliminate deadlocks
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Factor the vfs cache into its own package (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Pin the Fs in use in the Fs cache (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add SetSys() methods to Node to allow caching stuff on a node (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Ignore file not found errors from Hash in Read.Release (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix hang in read wait code (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Speed up multi thread downloads by using sparse files on Windows (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement \f[C]--local-no-sparse\f[R] flag for disabling sparse files
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement \f[C]rclone backend noop\f[R] for testing purposes (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \[dq]file not found\[dq] errors on post transfer Hash calculation
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Cache
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]rclone backend stats\f[R] command (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix Server Side Copy with Temp Upload (Brandon McNama)
|
|
.IP \[bu] 2
|
|
Remove Unused Functions (Lars Lehtonen)
|
|
.IP \[bu] 2
|
|
Disable race tests until bbolt is fixed (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Move methods used for testing into test file (greatroar)
|
|
.IP \[bu] 2
|
|
Add Pin and Unpin and canonicalised lookup (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Use proper import path go.etcd.io/bbolt (Robert-Andr\['e] Mauchin)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Calculate hashes for uploads from local disk (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This allows crypted Jottacloud uploads without using local disk
|
|
.IP \[bu] 2
|
|
This means crypted s3/b2 uploads will now have hashes
|
|
.RE
|
|
.IP \[bu] 2
|
|
Added \f[C]rclone backend decode\f[R]/\f[C]encode\f[R] commands to
|
|
replicate functionality of \f[C]cryptdecode\f[R] (Anagh Kumar Baranwal)
|
|
.IP \[bu] 2
|
|
Get rid of the unused Cipher interface as it obfuscated the code (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement streaming of unknown sized files so \f[C]rcat\f[R] is now
|
|
supported (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement memory pooling to control memory use (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--azureblob-disable-checksum\f[R] flag (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Retry \f[C]InvalidBlobOrBlock\f[R] error as it may indicate block
|
|
concurrency problems (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove unused \f[C]Object.parseTimeString()\f[R] (Lars Lehtonen)
|
|
.IP \[bu] 2
|
|
Fix permission error on SAS URL limited to container (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.IP \[bu] 2
|
|
Ignore directory markers at the root also (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Force the case of the SHA1 to lowercase (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove unused \f[C]largeUpload.clearUploadURL()\f[R] (Lars Lehtonen)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.IP \[bu] 2
|
|
Implement About to read size used (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add token renew function for jwt auth (David Bramwell)
|
|
.IP \[bu] 2
|
|
Added support for interchangeable root folder for Box backend (Sunil
|
|
Patra)
|
|
.IP \[bu] 2
|
|
Remove unnecessary iat from jws claims (David)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Follow shortcuts by default, skip with \f[C]--drive-skip-shortcuts\f[R]
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement \f[C]rclone backend shortcut\f[R] command for creating
|
|
shortcuts (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Added \f[C]rclone backend\f[R] command to change
|
|
\f[C]service_account_file\f[R] and \f[C]chunk_size\f[R] (Anagh Kumar
|
|
Baranwal)
|
|
.IP \[bu] 2
|
|
Fix missing files when using \f[C]--fast-list\f[R] and
|
|
\f[C]--drive-shared-with-me\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix duplicate items when using \f[C]--drive-shared-with-me\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Extend \f[C]--drive-stop-on-upload-limit\f[R] to respond to
|
|
\f[C]teamDriveFileLimitExceeded\f[R].
|
|
(harry)
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete files with multiple parents to avoid data loss (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Server side copy docs use default description if empty (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make error insufficient space to be fatal (harry)
|
|
.IP \[bu] 2
|
|
Add info about required redirect url (Elan Ruusam\[:a]e)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fichier
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.IP \[bu] 2
|
|
Implement custom pacer to deal with the new rate limiting (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix lockup when using concurrency limit on failed connections (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix lockup on failed upload when using concurrency limit (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix lockup on Close failures when using concurrency limit (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Work around pureftp sending spurious 150 messages (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]ARCHIVE\f[R] storage class to help (Adam Stroud)
|
|
.IP \[bu] 2
|
|
Ignore directory markers at the root (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Googlephotos
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make the start year configurable (Daven)
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.IP \[bu] 2
|
|
Create feature/favorites directory (Brandon Philips)
|
|
.IP \[bu] 2
|
|
Fix \[dq]concurrent map write\[dq] error (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Don\[aq]t put an image in error message (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Improved directory listing with new template from Caddy project
|
|
(calisro)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]--jottacloud-trashed-only\f[R] (buengese)
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.IP \[bu] 2
|
|
Use \f[C]RawURLEncoding\f[R] when decoding base64 encoded login token
|
|
(buengese)
|
|
.IP \[bu] 2
|
|
Implement cleanup (buengese)
|
|
.IP \[bu] 2
|
|
Update docs regarding cleanup, removed remains from old auth, and added
|
|
warning about special mountpoints.
|
|
(albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mailru
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Describe 2FA requirements (valery1707)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]--onedrive-server-side-across-configs\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.IP \[bu] 2
|
|
Fix occasional 416 errors on multipart uploads (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Added maximum chunk size limit warning in the docs (Harry)
|
|
.IP \[bu] 2
|
|
Fix missing drive on config (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make error \f[C]quotaLimitReached\f[R] to be fatal (harry)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Opendrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Added support for interchangeable root folder for pCloud backend (Sunil
|
|
Patra)
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.IP \[bu] 2
|
|
Fix initial config \[dq]Auth state doesn\[aq]t match\[dq] message (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Premiumizeme
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.IP \[bu] 2
|
|
Prune unused functions (Lars Lehtonen)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Putio
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make downloading files use the rclone http Client (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix parsing of remotes with leading and trailing / (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Qingstor
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make \f[C]rclone cleanup\f[R] remove pending multipart uploads older
|
|
than 24h (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Try harder to cancel failed multipart uploads (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Prune \f[C]multiUploader.list()\f[R] (Lars Lehtonen)
|
|
.IP \[bu] 2
|
|
Lint fix (Lars Lehtonen)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.IP \[bu] 2
|
|
Use memory pool for buffer allocations (Maciej Zimnoch)
|
|
.IP \[bu] 2
|
|
Add SSE-C support for AWS, Ceph, and MinIO (Jack Anderson)
|
|
.IP \[bu] 2
|
|
Fail fast multipart upload (Micha\[/l] Matczuk)
|
|
.IP \[bu] 2
|
|
Report errors on bucket creation (mkdir) correctly (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Specify that Minio supports URL encoding in listings (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Added 500 as retryErrorCode (Micha\[/l] Matczuk)
|
|
.IP \[bu] 2
|
|
Use \f[C]--low-level-retries\f[R] as the number of SDK retries
|
|
(Aleksandar Jankovi\['c])
|
|
.IP \[bu] 2
|
|
Fix multipart abort context (Aleksandar Jankovic)
|
|
.IP \[bu] 2
|
|
Replace deprecated \f[C]session.New()\f[R] with
|
|
\f[C]session.NewSession()\f[R] (Lars Lehtonen)
|
|
.IP \[bu] 2
|
|
Use the provided size parameter when allocating a new memory pool
|
|
(Joachim Brandon LeBlanc)
|
|
.IP \[bu] 2
|
|
Use rclone\[aq]s low level retries instead of AWS SDK to fix listing
|
|
retries (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Ignore directory markers at the root also (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Use single memory pool (Micha\[/l] Matczuk)
|
|
.IP \[bu] 2
|
|
Do not resize buf on put to memBuf (Micha\[/l] Matczuk)
|
|
.IP \[bu] 2
|
|
Improve docs for \f[C]--s3-disable-checksum\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Don\[aq]t leak memory or tokens in edge cases for multipart upload (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Seafile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement 2FA (Fred)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Added \f[C]--sftp-pem-key\f[R] to support inline key files (calisro)
|
|
.IP \[bu] 2
|
|
Fix post transfer copies failing with 0 size when using
|
|
\f[C]set_modtime=false\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Sharefile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Sugarsync
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix cosmetic issue in error message (Martin Michlmayr)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Union
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement multiple writable remotes (Max Sum)
|
|
.IP \[bu] 2
|
|
Fix server-side copy (Max Sum)
|
|
.IP \[bu] 2
|
|
Implement ListR (Max Sum)
|
|
.IP \[bu] 2
|
|
Enable ListR when upstreams contain local (Max Sum)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.IP \[bu] 2
|
|
Fix \f[C]X-OC-Mtime\f[R] header for Transip compatibility (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Report full and consistent usage with \f[C]about\f[R] (Yves G)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Yandex
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]--header-upload\f[R] and
|
|
\f[C]--header-download\f[R] (Tim Gallant)
|
|
.RE
|
|
.SS v1.51.0 - 2020-02-01
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Memory (https://rclone.org/memory/) (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Sugarsync (https://rclone.org/sugarsync/) (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Adjust all backends to have \f[C]--backend-encoding\f[R] parameter (Nick
|
|
Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this enables the encoding for special characters to be adjusted or
|
|
disabled
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]--max-duration\f[R] flag to control the maximum duration of a
|
|
transfer session (boosh)
|
|
.IP \[bu] 2
|
|
Add \f[C]--expect-continue-timeout\f[R] flag, default 1s (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--no-check-dest\f[R] flag for copying without testing the
|
|
destination (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement \f[C]--order-by\f[R] flag to order transfers (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
accounting
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t show entries in both transferring and checking (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add option to delete stats (Aleksandar Jankovic)
|
|
.RE
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Compress the test builds with gzip (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement a framework for starting test servers during tests (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
cmd: Always print elapsed time to tenth place seconds in progress (Gary
|
|
Kim)
|
|
.IP \[bu] 2
|
|
config
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--password-command\f[R] to allow dynamic config password (Damon
|
|
Permezel)
|
|
.IP \[bu] 2
|
|
Give config questions default values (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Check a remote exists when creating a new one (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
copyurl: Add \f[C]--stdout\f[R] flag to write to stdout (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
dedupe: Implement keep smallest too (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
hashsum: Add flag \f[C]--base64\f[R] flag (landall)
|
|
.IP \[bu] 2
|
|
lsf: Speed up on s3/swift/etc by not reading mimetype by default (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
lsjson: Add \f[C]--no-mimetype\f[R] flag (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
rc: Add methods to turn on blocking and mutex profiling (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
rcd
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Adding group parameter to stats (Chaitanya)
|
|
.IP \[bu] 2
|
|
Move webgui apart; option to disable browser (Xiaoxing Ye)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve sftp: Add support for public key with auth proxy (Paul Tinsley)
|
|
.IP \[bu] 2
|
|
stats: Show deletes in stats and hide zero stats (anuar45)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix error counter counting multiple times (Ankur Gupta)
|
|
.IP \[bu] 2
|
|
Fix error count shown as checks (Cnly)
|
|
.IP \[bu] 2
|
|
Clear finished transfer in stats-reset (Maciej Zimnoch)
|
|
.IP \[bu] 2
|
|
Added StatsInfo locking in statsGroups sum function (Micha\[/l] Matczuk)
|
|
.RE
|
|
.IP \[bu] 2
|
|
asyncreader: Fix EOF error (buengese)
|
|
.IP \[bu] 2
|
|
check: Fix \f[C]--one-way\f[R] recursing more directories than it needs
|
|
to (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
chunkedreader: Disable hash calculation for first segment (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
config
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Do not open browser on headless on drive/gcs/google photos (Xiaoxing Ye)
|
|
.IP \[bu] 2
|
|
SetValueAndSave ignore error if config section does not exist yet
|
|
(buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
cmd: Fix completion with an encrypted config (Danil Semelenov)
|
|
.IP \[bu] 2
|
|
dbhashsum: Stop it returning UNSUPPORTED on dropbox (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
dedupe: Add missing modes to help string (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
operations
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix dedupe continuing on errors like insufficientFilePersimmon
|
|
(SezalAgrawal)
|
|
.IP \[bu] 2
|
|
Clear accounting before low level retry (Maciej Zimnoch)
|
|
.IP \[bu] 2
|
|
Write debug message when hashes could not be checked (Ole Sch\[:u]tt)
|
|
.IP \[bu] 2
|
|
Move interface assertion to tests to remove pflag dependency (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make NewOverrideObjectInfo public and factor uses (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
proxy: Replace use of bcrypt with sha256 (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
vendor
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update bazil.org/fuse to fix FreeBSD 12.1 (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update github.com/t3rm1n4l/go-mega to fix mega \[dq]illegal base64 data
|
|
at input byte 22\[dq] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update termbox-go to fix ncdu command on FreeBSD (Kuang-che Wu)
|
|
.IP \[bu] 2
|
|
Update t3rm1n4l/go-mega - fixes mega: couldn\[aq]t login: crypto/aes:
|
|
invalid key size 0 (Nick Craig-Wood)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enable async reads for a 20% speedup (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Replace use of WriteAt with Write for cache mode >= writes and O_APPEND
|
|
(Brett Dutro)
|
|
.IP \[bu] 2
|
|
Make sure we call unmount when exiting (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Don\[aq]t build on go1.10 as bazil/fuse no longer supports it (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
When setting dates discard out of range dates (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add a newly created file straight into the directory (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Only calculate one hash for reads for a speedup (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make ReadAt for non cached files work better with non-sequential reads
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix edge cases when reading ModTime from file (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make sure existing files opened for write show correct size (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Don\[aq]t cache the path in RW file objects to fix renaming (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix rename of open files when using the VFS cache (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
When renaming files in the cache, rename the cache item in memory too
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix open file renaming on drive when using
|
|
\f[C]--vfs-cache-mode writes\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix incorrect modtime for mv into mount with
|
|
\f[C]--vfs-cache-modes writes\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
On rename, rename in cache too if the file exists (Anagh Kumar Baranwal)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make source file being updated errors be NoLowLevelRetry errors (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix update of hidden files on Windows (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Cache
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Follow move of upstream library github.com/coreos/bbolt
|
|
github.com/etcd-io/bbolt (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \f[C]fatal error: concurrent map writes\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Reorder the filename encryption options (Thomas Eales)
|
|
.IP \[bu] 2
|
|
Correctly handle trailing dot (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Chunker
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Reduce length of temporary suffix (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-stop-on-upload-limit\f[R] flag to stop syncs when
|
|
upload limit reached (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-use-shared-date\f[R] to use date file was shared
|
|
instead of modified date (Garry McNulty)
|
|
.IP \[bu] 2
|
|
Make sure invalid auth for teamdrives always reports an error (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \f[C]--fast-list\f[R] when using appDataFolder (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Use multipart resumable uploads for streaming and uploads in mount (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Log an ERROR if an incomplete search is returned (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Hide dangerous config from the configurator (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Treat \f[C]insufficient_space\f[R] errors as non retriable errors (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use new auth method used by official client (buengese)
|
|
.IP \[bu] 2
|
|
Add URL to generate Login Token to config wizard (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add support whitelabel versions (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Koofr
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use rclone HTTP client.
|
|
(jaKa)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add Sites.Read.All permission (Benjamin Richter)
|
|
.IP \[bu] 2
|
|
Add support \[dq]Retry-After\[dq] header (Motonori IWAMURO)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Opendrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]--opendrive-chunk-size\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Re-implement multipart upload to fix memory issues (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--s3-copy-cutoff\f[R] for size to switch to multipart copy
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add new region Asia Pacific (Hong Kong) (Outvi V)
|
|
.IP \[bu] 2
|
|
Reduce memory usage streaming files by reducing max stream upload size
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--s3-list-chunk\f[R] option for bucket listing (Thomas
|
|
Kriechbaumer)
|
|
.IP \[bu] 2
|
|
Force path style bucket access to off for AWS deprecation (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Use AWS web identity role provider if available (Tennix)
|
|
.IP \[bu] 2
|
|
Add StackPath Object Storage Support (Dave Koston)
|
|
.IP \[bu] 2
|
|
Fix ExpiryWindow value (Aleksandar Jankovic)
|
|
.IP \[bu] 2
|
|
Fix DisableChecksum condition (Aleksandar Jankovi\['c])
|
|
.IP \[bu] 2
|
|
Fix URL decoding of NextMarker (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--sftp-skip-links\f[R] to skip symlinks and non regular files
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Retry Creation of Connection (Sebastian Brandt)
|
|
.IP \[bu] 2
|
|
Fix \[dq]failed to parse private key file: ssh: not an encrypted
|
|
key\[dq] error (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Open files for update write only to fix AWS SFTP interop (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Reserve segments of dynamic large object when delete objects in
|
|
container what was enabled versioning.
|
|
(Nguy\[u1EC5]n H\[u1EEF]u Lu\[^a]n)
|
|
.IP \[bu] 2
|
|
Fix parsing of X-Object-Manifest (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update OVH API endpoint (unbelauscht)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make nextcloud only upload SHA1 checksums (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix case of \[dq]Bearer\[dq] in Authorization: header to agree with RFC
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add Referer header to fix problems with WAFs (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.50.2 - 2019-11-19
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting: Fix memory leak on retries operations (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix listing of the root directory with drive.files scope (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix --drive-root-folder-id with team/shared drives (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.50.1 - 2019-11-02
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
hash: Fix accidentally changed hash names for \f[C]DropboxHash\f[R] and
|
|
\f[C]CRC-32\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
fshttp: Fix error reporting on tpslimit token bucket errors (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
fshttp: Don\[aq]t print token bucket errors on context cancelled (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix listings of .
|
|
on Windows (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix DirMove/Move after Onedrive change (Xiaoxing Ye)
|
|
.RE
|
|
.SS v1.50.0 - 2019-10-26
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Citrix Sharefile (https://rclone.org/sharefile/) (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Chunker (https://rclone.org/chunker/) - an overlay backend to split
|
|
files into smaller parts (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Mail.ru Cloud (https://rclone.org/mailru/) (Ivan Andreev)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
encodings (Fabian M\[:o]ller & Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
All backends now use file name encoding to ensure any file name can be
|
|
written to any backend.
|
|
.IP \[bu] 2
|
|
See the restricted file name
|
|
docs (https://rclone.org/overview/#restricted-filenames) for more info
|
|
and the local backend docs.
|
|
.IP \[bu] 2
|
|
Some file names may look different in rclone if you are using any
|
|
control characters in names or unicode FULLWIDTH
|
|
symbols (https://en.wikipedia.org/wiki/Halfwidth_and_Fullwidth_Forms_(Unicode_block)).
|
|
.RE
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update to use go1.13 for the build (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Drop support for go1.9 (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Build rclone with GitHub actions (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Convert python scripts to python3 (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Swap Azure/go-ansiterm for mattn/go-colorable (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Dockerfile fixes (Matei David)
|
|
.IP \[bu] 2
|
|
Add plugin
|
|
support (https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#writing-a-plugin)
|
|
for backends and commands (Richard Patel)
|
|
.RE
|
|
.IP \[bu] 2
|
|
config
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use alternating Red/Green in config to make more obvious (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
contrib
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add sample DLNA server Docker Compose manifest.
|
|
(pataquets)
|
|
.IP \[bu] 2
|
|
Add sample WebDAV server Docker Compose manifest.
|
|
(pataquets)
|
|
.RE
|
|
.IP \[bu] 2
|
|
copyurl
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--auto-filename\f[R] flag for using file name from URL in
|
|
destination path (Denis)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve dlna:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Many compatibility improvements (Dan Walters)
|
|
.IP \[bu] 2
|
|
Support for external srt subtitles (Dan Walters)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Added command core/quit (Saksham Khanna)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
sync
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make \f[C]--update\f[R]/\f[C]-u\f[R] not transfer files that haven\[aq]t
|
|
changed (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Free objects after they come out of the transfer pipe to save memory
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \f[C]--files-from without --no-traverse\f[R] doing a recursive scan
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
operations
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix accounting for server-side copies (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Display \[aq]All duplicates removed\[aq] only if dedupe successful
|
|
(Sezal Agrawal)
|
|
.IP \[bu] 2
|
|
Display \[aq]Deleted X extra copies\[aq] only if dedupe successful
|
|
(Sezal Agrawal)
|
|
.RE
|
|
.IP \[bu] 2
|
|
accounting
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Only allow up to 100 completed transfers in the accounting list to save
|
|
memory (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Cull the old time ranges when possible to save memory (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix panic due to server-side copy fallback (Ivan Andreev)
|
|
.IP \[bu] 2
|
|
Fix memory leak noticeable for transfers of large numbers of objects
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix total duration calculation (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
cmd
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix environment variables not setting command line flags (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make autocomplete compatible with bash\[aq]s posix mode for macOS (Danil
|
|
Semelenov)
|
|
.IP \[bu] 2
|
|
Make \f[C]--progress\f[R] work in git bash on Windows (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \[aq]compopt: command not found\[aq] on autocomplete on macOS (Danil
|
|
Semelenov)
|
|
.RE
|
|
.IP \[bu] 2
|
|
config
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix setting of non top level flags from environment variables (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Check config names more carefully and report errors (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove error: can\[aq]t use \f[C]--size-only\f[R] and
|
|
\f[C]--ignore-size\f[R] together.
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
filter: Prevent mixing options when \f[C]--files-from\f[R] is in use
|
|
(Michele Caci)
|
|
.IP \[bu] 2
|
|
serve sftp: Fix crash on unsupported operations (e.g.
|
|
Readlink) (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow files of unknown size to be read properly (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Skip tests on <= 2 CPUs to avoid lockup (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix panic on File.Open (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \[dq]mount_fusefs: -o timeout=: option not supported\[dq] on FreeBSD
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Don\[aq]t pass huge filenames (>4k) to FUSE as it can\[aq]t cope (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add flag \f[C]--vfs-case-insensitive\f[R] for windows/macOS mounts (Ivan
|
|
Andreev)
|
|
.IP \[bu] 2
|
|
Make objects of unknown size readable through the VFS (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Move writeback of dirty data out of close() method into its own method
|
|
(FlushWrites) and remove close() call from Flush() (Brett Dutro)
|
|
.IP \[bu] 2
|
|
Stop empty dirs disappearing when renamed on bucket-based remotes (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Stop change notify polling clearing so much of the directory cache (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Disable logging to the Windows event log (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Remove \f[C]unverified:\f[R] prefix on sha1 to improve interop (e.g.
|
|
with CyberDuck) (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add options to get access token via JWT auth (David)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Disable HTTP/2 by default to work around INTERNAL_ERROR problems (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make sure that drive root ID is always canonical (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \f[C]--drive-shared-with-me\f[R] from the root with lsand
|
|
\f[C]--fast-list\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix ChangeNotify polling for shared drives (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix change notify polling when using appDataFolder (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make disallowed filenames errors not retry (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix nil pointer exception on restricted files (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fichier
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix accessing files > 2GB on 32 bit systems (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow disabling EPSV mode (Jon Fautley)
|
|
.RE
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
HEAD directory entries in parallel to speedup (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--http-no-head\f[R] to stop rclone doing HEAD in listings (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Putio
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add ability to resume uploads (Cenk Alti)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix signature v2_auth headers (Anthony Rusdi)
|
|
.IP \[bu] 2
|
|
Fix encoding for control characters (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Only ask for URL encoded directory listings if we need them on Ceph
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add option for multipart failure behaviour (Aleksandar Jankovic)
|
|
.IP \[bu] 2
|
|
Support for multipart copy (\[u5E84]\[u5929]\[u7FFC])
|
|
.IP \[bu] 2
|
|
Fix nil pointer reference if no metadata returned for object (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix \f[C]--sftp-ask-password\f[R] trying to contact the ssh agent (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix hashes of files with backslashes (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Include more ciphers with \f[C]--sftp-use-insecure-cipher\f[R] (Carlos
|
|
Ferreyra)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Parse and return Sharepoint error response (Henning Surmeier)
|
|
.RE
|
|
.SS v1.49.5 - 2019-10-05
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Revert back to go1.12.x for the v1.49.x builds as go1.13.x was causing
|
|
issues (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix rpm packages by using master builds of nfpm (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix macOS build after brew changes (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.49.4 - 2019-09-29
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
cmd/rcd: Address ZipSlip vulnerability (Richard Patel)
|
|
.IP \[bu] 2
|
|
accounting: Fix file handle leak on errors (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
oauthutil: Fix security problem when running with two users on the same
|
|
machine (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix listing of an empty root returning: error dir not found (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.49.3 - 2019-09-15
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix total duration calculation (Aleksandar Jankovic)
|
|
.IP \[bu] 2
|
|
Fix \[dq]file already closed\[dq] on transfer retries (Nick Craig-Wood)
|
|
.RE
|
|
.RE
|
|
.SS v1.49.2 - 2019-09-08
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
build: Add Docker workflow support (Alfonso Montero)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting: Fix locking in Transfer to avoid deadlock with
|
|
\f[C]--progress\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
docs: Fix template argument for mktemp in install.sh (Cnly)
|
|
.IP \[bu] 2
|
|
operations: Fix \f[C]-u\f[R]/\f[C]--update\f[R] with google photos /
|
|
files of unknown size (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
rc: Fix docs for config/create /update /password (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix need for elevated permissions on SetModTime (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.49.1 - 2019-08-28
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
config: Fix generated passwords being stored as empty password (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
rcd: Added missing parameter for web-gui info logs.
|
|
(Chaitanya)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Googlephotos
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix crash on error response (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix crash on error response (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.49.0 - 2019-08-26
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
1fichier (https://rclone.org/fichier/) (Laura Hausmann)
|
|
.IP \[bu] 2
|
|
Google Photos (https://rclone.org/googlephotos/) (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Putio (https://rclone.org/putio/) (Cenk Alti)
|
|
.IP \[bu] 2
|
|
premiumize.me (https://rclone.org/premiumizeme/) (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Experimental web GUI (https://rclone.org/gui/) (Chaitanya Bankanhal)
|
|
.IP \[bu] 2
|
|
Implement \f[C]--compare-dest\f[R] & \f[C]--copy-dest\f[R] (yparitcher)
|
|
.IP \[bu] 2
|
|
Implement \f[C]--suffix\f[R] without \f[C]--backup-dir\f[R] for backup
|
|
to current dir (yparitcher)
|
|
.IP \[bu] 2
|
|
\f[C]config reconnect\f[R] to re-login (re-run the oauth login) for the
|
|
backend.
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
\f[C]config userinfo\f[R] to discover which user you are logged in as.
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
\f[C]config disconnect\f[R] to disconnect you (log out) from the
|
|
backend.
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--use-json-log\f[R] for JSON logging (justinalin)
|
|
.IP \[bu] 2
|
|
Add context propagation to rclone (Aleksandar Jankovic)
|
|
.IP \[bu] 2
|
|
Reworking internal statistics interfaces so they work with rc jobs
|
|
(Aleksandar Jankovic)
|
|
.IP \[bu] 2
|
|
Add Higher units for ETA (AbelThar)
|
|
.IP \[bu] 2
|
|
Update rclone logos to new design (Andreas Chlupka)
|
|
.IP \[bu] 2
|
|
hash: Add CRC-32 support (Cenk Alti)
|
|
.IP \[bu] 2
|
|
help showbackend: Fixed advanced option category when there are no
|
|
standard options (buengese)
|
|
.IP \[bu] 2
|
|
ncdu: Display/Copy to Clipboard Current Path (Gary Kim)
|
|
.IP \[bu] 2
|
|
operations:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Run hashing operations in parallel (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Don\[aq]t calculate checksums when using \f[C]--ignore-checksum\f[R]
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Check transfer hashes when using \f[C]--size-only\f[R] mode (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Disable multi thread copy for local to local copies (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Debug successful hashes as well as failures (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add ability to stop async jobs (Aleksandar Jankovic)
|
|
.IP \[bu] 2
|
|
Return current settings if core/bwlimit called without parameters (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Rclone-WebUI integration with rclone (Chaitanya Bankanhal)
|
|
.IP \[bu] 2
|
|
Added command line parameter to control the cross origin resource
|
|
sharing (CORS) in the rcd.
|
|
(Security Improvement) (Chaitanya Bankanhal)
|
|
.IP \[bu] 2
|
|
Add anchor tags to the docs so links are consistent (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove _async key from input parameters after parsing so later
|
|
operations won\[aq]t get confused (buengese)
|
|
.IP \[bu] 2
|
|
Add call to clear stats (Aleksandar Jankovic)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rcd
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Auto-login for web-gui (Chaitanya Bankanhal)
|
|
.IP \[bu] 2
|
|
Implement \f[C]--baseurl\f[R] for rcd and web-gui (Chaitanya Bankanhal)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve dlna
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Only select interfaces which can multicast for SSDP (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add more builtin mime types to cover standard audio/video (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix missing mime types on Android causing missing videos (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve ftp
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Refactor to bring into line with other serve commands (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement \f[C]--auth-proxy\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve http: Implement \f[C]--baseurl\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
serve restic: Implement \f[C]--baseurl\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
serve sftp
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement auth proxy (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix detection of whether server is authorized (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve webdav
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]--baseurl\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Support \f[C]--auth-proxy\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make \[dq]bad record MAC\[dq] a retriable error (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
copyurl: Fix copying files that return HTTP errors (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
march: Fix checking sub-directories when using \f[C]--no-traverse\f[R]
|
|
(buengese)
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix unmarshalable http.AuthFn in options and put in test for
|
|
marshalability (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Move job expire flags to rc to fix initialization problem (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \f[C]--loopback\f[R] with rc/list and others (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rcat: Fix slowdown on systems with multiple hashes (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
rcd: Fix permissions problems on cache directory with web gui download
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Default \f[C]--daemon-timout\f[R] to 15 minutes on macOS and FreeBSD
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update docs to show mounting from root OK for bucket-based (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove nonseekable flag from write files (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make write without cache more efficient (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \f[C]--vfs-cache-mode minimal\f[R] and \f[C]writes\f[R] ignoring
|
|
cached files (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--local-case-sensitive\f[R] and
|
|
\f[C]--local-case-insensitive\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Avoid polluting page cache when uploading local files to remote backends
|
|
(Micha\[/l] Matczuk)
|
|
.IP \[bu] 2
|
|
Don\[aq]t calculate any hashes by default (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fadvise run syscall on a dedicated go routine (Micha\[/l] Matczuk)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Azure Storage Emulator support (Sandeep)
|
|
.IP \[bu] 2
|
|
Updated config help details to remove connection string references
|
|
(Sandeep)
|
|
.IP \[bu] 2
|
|
Make all operations work from the root (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement link sharing (yparitcher)
|
|
.IP \[bu] 2
|
|
Enable server-side copy to copy between buckets (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make all operations work from the root (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix server-side copy of big files (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update API for teamdrive use (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add error for purge with \f[C]--drive-trashed-only\f[R] (ginvine)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fichier
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make FolderID int and adjust related code (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Reduce oauth scope requested as suggested by Google (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make all operations work from the root (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--http-headers\f[R] flag for setting arbitrary headers (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use new api for retrieving internal username (buengese)
|
|
.IP \[bu] 2
|
|
Refactor configuration and minor cleanup (buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Koofr
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Support setting modification times on Koofr backend.
|
|
(jaKa)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Opendrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Refactor to use existing lib/rest facilities for uploads (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Qingstor
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Upgrade to v3 SDK and fix listing loop (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make all operations work from the root (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add INTELLIGENT_TIERING storage class (Matti Niemenmaa)
|
|
.IP \[bu] 2
|
|
Make all operations work from the root (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add missing interface check and fix About (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Completely ignore all modtime checks if SetModTime=false (Jon Fautley)
|
|
.IP \[bu] 2
|
|
Support md5/sha1 with rsync.net (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Save the md5/sha1 command in use to the config file for efficiency (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Opt-in support for diffie-hellman-group-exchange-sha256
|
|
diffie-hellman-group-exchange-sha1 (Yi FU)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use FixRangeOption to fix 0 length files via the VFS (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix upload when using no_chunk to return the correct size (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make all operations work from the root (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix segments leak during failed large file uploads.
|
|
(nguyenhuuluan434)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--webdav-bearer-token-command\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Refresh token when it expires with
|
|
\f[C]--webdav-bearer-token-command\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add docs for using bearer_token_command with oidc-agent (Paul Millar)
|
|
.RE
|
|
.SS v1.48.0 - 2019-06-15
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
serve sftp: Serve an rclone remote over SFTP (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Multi threaded downloads to local storage (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
controlled with \f[C]--multi-thread-cutoff\f[R] and
|
|
\f[C]--multi-thread-streams\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
Use rclone.conf from rclone executable directory to enable portable use
|
|
(albertony)
|
|
.IP \[bu] 2
|
|
Allow sync of a file and a directory with the same name (forgems)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this is common on bucket-based remotes, e.g.
|
|
s3, gcs
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]--ignore-case-sync\f[R] for forced case insensitivity
|
|
(garry415)
|
|
.IP \[bu] 2
|
|
Implement \f[C]--stats-one-line-date\f[R] and
|
|
\f[C]--stats-one-line-date-format\f[R] (Peter Berbec)
|
|
.IP \[bu] 2
|
|
Log an ERROR for all commands which exit with non-zero status (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Use go-homedir to read the home directory more reliably (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Enable creating encrypted config through external script invocation
|
|
(Wojciech Smigielski)
|
|
.IP \[bu] 2
|
|
build: Drop support for go1.8 (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
config: Make config create/update encrypt passwords where necessary
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
copyurl: Honor \f[C]--no-check-certificate\f[R] (Stefan Breunig)
|
|
.IP \[bu] 2
|
|
install: Linux skip man pages if no mandb (didil)
|
|
.IP \[bu] 2
|
|
lsf: Support showing the Tier of the object (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
lsjson
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Added EncryptedPath to output (calisro)
|
|
.IP \[bu] 2
|
|
Support showing the Tier of the object (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add IsBucket field for bucket-based remote listing of the root (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--loopback\f[R] flag to run commands directly without a server
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add operations/fsinfo: Return information about the remote (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Skip auth for OPTIONS request (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
cmd/providers: Add DefaultStr, ValueStr and Type fields (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
jobs: Make job expiry timeouts configurable (Aleksandar Jankovic)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve dlna reworked and improved (Dan Walters)
|
|
.IP \[bu] 2
|
|
serve ftp: add \f[C]--ftp-public-ip\f[R] flag to specify public IP
|
|
(calistri)
|
|
.IP \[bu] 2
|
|
serve restic: Add support for \f[C]--private-repos\f[R] in
|
|
\f[C]serve restic\f[R] (Florian Apolloner)
|
|
.IP \[bu] 2
|
|
serve webdav: Combine serve webdav and serve http (Gary Kim)
|
|
.IP \[bu] 2
|
|
size: Ignore negative sizes when calculating total (Garry McNulty)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make move and copy individual files obey \f[C]--backup-dir\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
If \f[C]--ignore-checksum\f[R] is in effect, don\[aq]t calculate
|
|
checksum (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
moveto: Fix case-insensitive same remote move (Gary Kim)
|
|
.IP \[bu] 2
|
|
rc: Fix serving bucket-based objects with \f[C]--rc-serve\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
serve webdav: Fix serveDir not being updated with changes from webdav
|
|
(Gary Kim)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix poll interval documentation (Animosity022)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make WriteAt for non cached files work with non-sequential writes (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Only calculate the required hashes for big speedup (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Log errors when listing instead of returning an error (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix preallocate warning on Linux with ZFS (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make rclone dedupe work through crypt (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix wrapping of ChangeNotify to decrypt directories properly (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Support PublicLink (rclone link) of underlying backend (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement Optional methods SetTier, GetTier (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement server-side copy (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement SetModTime (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix move and copy from TeamDrive to GDrive (Fionera)
|
|
.IP \[bu] 2
|
|
Add notes that cleanup works in the background on drive (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-server-side-across-configs\f[R] to default back to old
|
|
server-side copy semantics by default (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-size-as-quota\f[R] to show storage quota usage for file
|
|
size (Garry McNulty)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add FTP List timeout (Jeff Quinn)
|
|
.IP \[bu] 2
|
|
Add FTP over TLS support (Gary Kim)
|
|
.IP \[bu] 2
|
|
Add \f[C]--ftp-no-check-certificate\f[R] option for FTPS (Gary Kim)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix upload errors when uploading pre 1970 files (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for selecting device and mountpoint.
|
|
(buengese)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mega
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add cleanup support (Gary Kim)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
More accurately check if root is found (Cnly)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Support S3 Accelerated endpoints with
|
|
\f[C]--s3-use-accelerate-endpoint\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add config info for Wasabi\[aq]s EU Central endpoint (Robert Marko)
|
|
.IP \[bu] 2
|
|
Make SetModTime work for GLACIER while syncing (Philip Harvey)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add About support (Gary Kim)
|
|
.IP \[bu] 2
|
|
Fix about parsing of \f[C]df\f[R] results so it can cope with -ve
|
|
results (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Send custom client version and debug server version (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Retry on 423 Locked errors (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.47.0 - 2019-04-13
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Backend for Koofr cloud storage service.
|
|
(jaKa)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Resume downloads if the reader fails in copy (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this means rclone will restart transfers if the source has an error
|
|
.IP \[bu] 2
|
|
this is most useful for downloads or cloud to cloud copies
|
|
.RE
|
|
.IP \[bu] 2
|
|
Use \f[C]--fast-list\f[R] for listing operations where it won\[aq]t use
|
|
more memory (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this should speed up the following operations on remotes which support
|
|
\f[C]ListR\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]dedupe\f[R], \f[C]serve restic\f[R] \f[C]lsf\f[R], \f[C]ls\f[R],
|
|
\f[C]lsl\f[R], \f[C]lsjson\f[R], \f[C]lsd\f[R], \f[C]md5sum\f[R],
|
|
\f[C]sha1sum\f[R], \f[C]hashsum\f[R], \f[C]size\f[R], \f[C]delete\f[R],
|
|
\f[C]cat\f[R], \f[C]settier\f[R]
|
|
.IP \[bu] 2
|
|
use \f[C]--disable ListR\f[R] to get old behaviour if required
|
|
.RE
|
|
.IP \[bu] 2
|
|
Make \f[C]--files-from\f[R] traverse the destination unless
|
|
\f[C]--no-traverse\f[R] is set (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this fixes \f[C]--files-from\f[R] with Google drive and excessive API
|
|
use in general.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Make server-side copy account bytes and obey \f[C]--max-transfer\f[R]
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--create-empty-src-dirs\f[R] flag and default to not creating
|
|
empty dirs (ishuah)
|
|
.IP \[bu] 2
|
|
Add client side TLS/SSL flags
|
|
\f[C]--ca-cert\f[R]/\f[C]--client-cert\f[R]/\f[C]--client-key\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement \f[C]--suffix-keep-extension\f[R] for use with
|
|
\f[C]--suffix\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
build:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Switch to semver compliant version tags to be go modules compliant (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update to use go1.12.x for the build (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve dlna: Add connection manager service description to improve
|
|
compatibility (Dan Walters)
|
|
.IP \[bu] 2
|
|
lsf: Add \[aq]e\[aq] format to show encrypted names and \[aq]o\[aq] for
|
|
original IDs (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
lsjson: Added \f[C]--files-only\f[R] and \f[C]--dirs-only\f[R] flags
|
|
(calistri)
|
|
.IP \[bu] 2
|
|
rc: Implement operations/publiclink the equivalent of
|
|
\f[C]rclone link\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting: Fix total ETA when \f[C]--stats-unit bits\f[R] is in effect
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Bash TAB completion
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use private custom func to fix clash between rclone and kubectl (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix for remotes with underscores in their names (Six)
|
|
.IP \[bu] 2
|
|
Fix completion of remotes (Florian Gamb\[:o]ck)
|
|
.IP \[bu] 2
|
|
Fix autocompletion of remote paths with spaces (Danil Semelenov)
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve dlna: Fix root XML service descriptor (Dan Walters)
|
|
.IP \[bu] 2
|
|
ncdu: Fix display corruption with Chinese characters (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add SIGTERM to signals which run the exit handlers on unix (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
rc: Reload filter when the options are set via the rc (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS / Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix FreeBSD: Ignore Truncate if called with no readers and already the
|
|
correct size (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Read directory and check for a file before mkdir (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Shorten the locking window for vfs/refresh (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enable MD5 checksums when uploading files bigger than the
|
|
\[dq]Cutoff\[dq] (Dr.Rx)
|
|
.IP \[bu] 2
|
|
Fix SAS URL support (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow manual configuration of backblaze downloadUrl (Vince)
|
|
.IP \[bu] 2
|
|
Ignore already_hidden error on remove (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Ignore malformed \f[C]src_last_modified_millis\f[R] (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--skip-checksum-gphotos\f[R] to ignore incorrect checksums on
|
|
Google Photos (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Allow server-side move/copy between different remotes.
|
|
(Fionera)
|
|
.IP \[bu] 2
|
|
Add docs on team drives and \f[C]--fast-list\f[R] eventual consistency
|
|
(Nestar47)
|
|
.IP \[bu] 2
|
|
Fix imports of text files (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix range requests on 0 length files (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix creation of duplicates with server-side copy (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Retry blank errors to fix long listings (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--ftp-concurrency\f[R] to limit maximum number of connections
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fall back to default application credentials (marcintustin)
|
|
.IP \[bu] 2
|
|
Allow bucket policy only buckets (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--http-no-slash\f[R] for websites with directories with no
|
|
slashes (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove duplicates from listings (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix socket leak on 404 errors (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix token refresh (Sebastian B\[:u]nger)
|
|
.IP \[bu] 2
|
|
Add device registration (Oliver Heyme)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement graceful cancel of multipart uploads if rclone is interrupted
|
|
(Cnly)
|
|
.IP \[bu] 2
|
|
Always add trailing colon to path when addressing items, (Cnly)
|
|
.IP \[bu] 2
|
|
Return errors instead of panic for invalid uploads (Fabian M\[:o]ller)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \[dq]Glacier Deep Archive\[dq] storage class (Manu)
|
|
.IP \[bu] 2
|
|
Update Dreamhost endpoint (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Note incompatibility with CEPH Jewel (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow custom ssh client config (Alexandru Bumbacea)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Obey Retry-After to enable OVH restore from cold storage (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Work around token expiry on CEPH (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow IsCollection property to be integer or boolean (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix race when creating directories (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix About/df when reading the available/total returns 0 (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.SS v1.46 - 2019-02-09
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Support Alibaba Cloud (Aliyun) OSS via the s3 backend (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
serve dlna: serves a remove via DLNA for the local network (nicolov)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
copy, move: Restore deprecated \f[C]--no-traverse\f[R] flag (Nick
|
|
Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This is useful for when transferring a small number of files into a
|
|
large destination
|
|
.RE
|
|
.IP \[bu] 2
|
|
genautocomplete: Add remote path completion for bash completion
|
|
(Christopher Peterson & Danil Semelenov)
|
|
.IP \[bu] 2
|
|
Buffer memory handling reworked to return memory to the OS better (Nick
|
|
Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Buffer recycling library to replace sync.Pool
|
|
.IP \[bu] 2
|
|
Optionally use memory mapped memory for better memory shrinking
|
|
.IP \[bu] 2
|
|
Enable with \f[C]--use-mmap\f[R] if having memory problems - not default
|
|
yet
|
|
.RE
|
|
.IP \[bu] 2
|
|
Parallelise reading of files specified by \f[C]--files-from\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
check: Add stats showing total files matched.
|
|
(Dario Guzik)
|
|
.IP \[bu] 2
|
|
Allow rename/delete open files under Windows (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
lsjson: Use exactly the correct number of decimal places in the seconds
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add cookie support with cmdline switch \f[C]--use-cookies\f[R] for all
|
|
HTTP based remotes (qip)
|
|
.IP \[bu] 2
|
|
Warn if \f[C]--checksum\f[R] is set but there are no hashes available
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Rework rate limiting (pacer) to be more accurate and allow bursting
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Improve error reporting for too many/few arguments in commands (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
listremotes: Remove \f[C]-l\f[R] short flag as it conflicts with the new
|
|
global flag (weetmuts)
|
|
.IP \[bu] 2
|
|
Make http serving with auth generate INFO messages on auth fail (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix layout of stats (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix \f[C]--progress\f[R] crash under Windows Jenkins (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix transfer of google/onedrive docs by calling Rcat in Copy when size
|
|
is -1 (Cnly)
|
|
.IP \[bu] 2
|
|
copyurl: Fix checking of \f[C]--dry-run\f[R] (Denis Skovpen)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Check that mountpoint and local directory to mount don\[aq]t overlap
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix mount size under 32 bit Windows (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement renaming of directories for backends without DirMove (Nick
|
|
Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
now all backends except b2 support renaming directories
|
|
.RE
|
|
.IP \[bu] 2
|
|
Implement \f[C]--vfs-cache-max-size\f[R] to limit the total size of the
|
|
cache (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--dir-perms\f[R] and \f[C]--file-perms\f[R] flags to set
|
|
default permissions (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix deadlock on concurrent operations on a directory (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix deadlock between RWFileHandle.close and File.Remove (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix renaming/deleting open files with cache mode \[dq]writes\[dq] under
|
|
Windows (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix panic on rename with \f[C]--dry-run\f[R] set (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix vfs/refresh with recurse=true needing the \f[C]--fast-list\f[R] flag
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for \f[C]-l\f[R]/\f[C]--links\f[R] (symbolic link
|
|
translation) (yair\[at]unicorn)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this works by showing links as \f[C]link.rclonelink\f[R] - see local
|
|
backend docs for more info
|
|
.IP \[bu] 2
|
|
this errors if used with \f[C]-L\f[R]/\f[C]--copy-links\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix renaming/deleting open files on Windows (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Check for maximum length before decrypting filename to fix panic (Garry
|
|
McNulty)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow building azureblob backend on *BSD (themylogin)
|
|
.IP \[bu] 2
|
|
Use the rclone HTTP client to support \f[C]--dump headers\f[R],
|
|
\f[C]--tpslimit\f[R], etc.
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Use the s3 pacer for 0 delay in non error conditions (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Ignore directory markers (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Stop Mkdir attempting to create existing containers (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
cleanup: will remove unfinished large files >24hrs old (Garry McNulty)
|
|
.IP \[bu] 2
|
|
For a bucket limited application key check the bucket name (Nick
|
|
Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
before this, rclone would use the authorised bucket regardless of what
|
|
you put on the command line
|
|
.RE
|
|
.IP \[bu] 2
|
|
Added \f[C]--b2-disable-checksum\f[R] flag (Wojciech Smigielski)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this enables large files to be uploaded without a SHA-1 hash for speed
|
|
reasons
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Set default pacer to 100ms for 10 tps (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This fits the Google defaults much better and reduces the 403 errors
|
|
massively
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-pacer-min-sleep\f[R] and \f[C]--drive-pacer-burst\f[R]
|
|
to control the pacer
|
|
.RE
|
|
.IP \[bu] 2
|
|
Improve ChangeNotify support for items with multiple parents (Fabian
|
|
M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Fix ListR for items with multiple parents - this fixes oddities with
|
|
\f[C]vfs/refresh\f[R] (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Fix using \f[C]--drive-impersonate\f[R] and appfolders (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix google docs in rclone mount for some (not all) applications (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Retry-After support for Dropbox backend (Mathieu Carbou)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Wait for 60 seconds for a connection to Close then declare it dead (Nick
|
|
Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
helps with indefinite hangs on some FTP servers
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update google cloud storage endpoints (weetmuts)
|
|
.RE
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add an example with username and password which is supported but
|
|
wasn\[aq]t documented (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix backend with \f[C]--files-from\f[R] and non-existent files (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Hubic
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make error message more informative if authentication fails (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Resume and deduplication support (Oliver Heyme)
|
|
.IP \[bu] 2
|
|
Use token auth for all API requests Don\[aq]t store password anymore
|
|
(Sebastian B\[:u]nger)
|
|
.IP \[bu] 2
|
|
Add support for 2-factor authentication (Sebastian B\[:u]nger)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mega
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement v2 account login which fixes logins for newer Mega accounts
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Return error if an unknown length file is attempted to be uploaded (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add new error codes for better error reporting (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix broken support for \[dq]shared with me\[dq] folders (Alex Chen)
|
|
.IP \[bu] 2
|
|
Fix root ID not normalised (Cnly)
|
|
.IP \[bu] 2
|
|
Return err instead of panic on unknown-sized uploads (Cnly)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Qingstor
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix go routine leak on multipart upload errors (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add upload chunk size/concurrency/cutoff control (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Default \f[C]--qingstor-upload-concurrency\f[R] to 1 to work around bug
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]--s3-upload-cutoff\f[R] for single part uploads below
|
|
this (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Change \f[C]--s3-upload-concurrency\f[R] default to 4 to increase
|
|
performance (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--s3-bucket-acl\f[R] to control bucket ACL (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Auto detect region for buckets on operation failure (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add GLACIER storage class (William Cocker)
|
|
.IP \[bu] 2
|
|
Add Scaleway to s3 documentation (R\['e]my L\['e]one)
|
|
.IP \[bu] 2
|
|
Add AWS endpoint eu-north-1 (weetmuts)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for PEM encrypted private keys (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Add option to force the usage of an ssh-agent (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Perform environment variable expansion on key-file (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Fix rmdir on Windows based servers (e.g.
|
|
CrushFTP) (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix rmdir deleting directory contents on some SFTP servers (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix error on dangling symlinks (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--swift-no-chunk\f[R] to disable segmented uploads in
|
|
rcat/mount (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Introduce application credential auth support (kayrus)
|
|
.IP \[bu] 2
|
|
Fix memory usage by slimming Object (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix extra requests on upload (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix reauth on big files (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Union
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix poll-interval not working (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Support About which means rclone mount will show the correct disk size
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Support MD5 and SHA1 hashes with Owncloud and Nextcloud (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fail soft on time parsing errors (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix infinite loop on failed directory creation (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix identification of directories for Bitrix Site Manager (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix upload of 0 length files on some servers (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix if MKCOL fails with 423 Locked assume the directory exists (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.SS v1.45 - 2018-11-24
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The Yandex backend was re-written - see below for details (Sebastian
|
|
B\[:u]nger)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
rcd: New command just to serve the remote control API (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The remote control API (rc) was greatly expanded to allow full control
|
|
over rclone (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
sensitive operations require authorization or the \f[C]--rc-no-auth\f[R]
|
|
flag
|
|
.IP \[bu] 2
|
|
config/* operations to configure rclone
|
|
.IP \[bu] 2
|
|
options/* for reading/setting command line flags
|
|
.IP \[bu] 2
|
|
operations/* for all low level operations, e.g.
|
|
copy file, list directory
|
|
.IP \[bu] 2
|
|
sync/* for sync, copy and move
|
|
.IP \[bu] 2
|
|
\f[C]--rc-files\f[R] flag to serve files on the rc http server
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this is for building web native GUIs for rclone
|
|
.RE
|
|
.IP \[bu] 2
|
|
Optionally serving objects on the rc http server
|
|
.IP \[bu] 2
|
|
Ensure rclone fails to start up if the \f[C]--rc\f[R] port is in use
|
|
already
|
|
.IP \[bu] 2
|
|
See the rc docs (https://rclone.org/rc/) for more info
|
|
.RE
|
|
.IP \[bu] 2
|
|
sync/copy/move
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make \f[C]--files-from\f[R] only read the objects specified and
|
|
don\[aq]t scan directories (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This is a huge speed improvement for destinations with lots of files
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
filter: Add \f[C]--ignore-case\f[R] flag (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
ncdu: Add remove function (\[aq]d\[aq] key) (Henning Surmeier)
|
|
.IP \[bu] 2
|
|
rc command
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--json\f[R] flag for structured JSON input (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--user\f[R] and \f[C]--pass\f[R] flags and interpret
|
|
\f[C]--rc-user\f[R], \f[C]--rc-pass\f[R], \f[C]--rc-addr\f[R] (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Require go1.8 or later for compilation (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Enable softfloat on MIPS arch (Scott Edlund)
|
|
.IP \[bu] 2
|
|
Integration test framework revamped with a better report and better
|
|
retries (Nick Craig-Wood)
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
cmd: Make \f[C]--progress\f[R] update the stats correctly at the end
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
config: Create config directory on save if it is missing (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
dedupe: Check for existing filename before renaming a dupe file (ssaqua)
|
|
.IP \[bu] 2
|
|
move: Don\[aq]t create directories with \f[C]--dry-run\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
operations: Fix Purge and Rmdirs when dir is not the root (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
serve http/webdav/restic: Ensure rclone exits if the port is in use
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make \f[C]--volname\f[R] work for Windows and macOS (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Avoid context deadline exceeded error by setting a large TryTimeout
|
|
value (brused27)
|
|
.IP \[bu] 2
|
|
Fix erroneous Rmdir error \[dq]directory not empty\[dq] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Wait for up to 60s to create a just deleted container (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add dropbox impersonate support (Jake Coggiano)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix bug in \f[C]--fast-list\f[R] handing of empty folders (albertony)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Opendrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix transfer of files with \f[C]+\f[R] and \f[C]&\f[R] in (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix retries of upload chunks (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Set ACL for server-side copies to that provided by the user (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix role_arn, credential_source, ...
|
|
(Erik Swanson)
|
|
.IP \[bu] 2
|
|
Add config info for Wasabi\[aq]s US-West endpoint (Henry Ptasinski)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ensure file hash checking is really disabled (Jon Fautley)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add pacer for retries to make swift more reliable (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add Content-Type to PUT requests (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix config parsing so \f[C]--webdav-user\f[R] and
|
|
\f[C]--webdav-pass\f[R] flags work (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add RFC3339 date format (Ralf Hemberger)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Yandex
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The yandex backend was re-written (Sebastian B\[:u]nger)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This implements low level retries (Sebastian B\[:u]nger)
|
|
.IP \[bu] 2
|
|
Copy, Move, DirMove, PublicLink and About optional interfaces (Sebastian
|
|
B\[:u]nger)
|
|
.IP \[bu] 2
|
|
Improved general error handling (Sebastian B\[:u]nger)
|
|
.IP \[bu] 2
|
|
Removed ListR for now due to inconsistent behaviour (Sebastian
|
|
B\[:u]nger)
|
|
.RE
|
|
.RE
|
|
.SS v1.44 - 2018-10-15
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
serve ftp: Add ftp server (Antoine GIRARD)
|
|
.IP \[bu] 2
|
|
settier: perform storage tier changes on supported remotes (sandeepkru)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Reworked command line help
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make default help less verbose (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Split flags up into global and backend flags (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Implement specialised help for flags and backends (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Show URL of backend help page when starting config (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
stats: Long names now split in center (Joanna Marek)
|
|
.IP \[bu] 2
|
|
Add \f[C]--log-format\f[R] flag for more control over log output (dcpu)
|
|
.IP \[bu] 2
|
|
rc: Add support for OPTIONS and basic CORS (frenos)
|
|
.IP \[bu] 2
|
|
stats: show FatalErrors and NoRetryErrors in stats (C\['e]dric Connes)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix -P not ending with a new line (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
config: don\[aq]t create default config dir when user supplies
|
|
\f[C]--config\f[R] (albertony)
|
|
.IP \[bu] 2
|
|
Don\[aq]t print non-ASCII characters with \f[C]--progress\f[R] on
|
|
windows (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Correct logs for excluded items (ssaqua)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Remove EXPERIMENTAL tags (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix race condition detected by serve ftp tests (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add vfs/poll-interval rc command (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Enable rename for nearly all remotes using server-side Move or Copy
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Reduce directory cache cleared by poll-interval (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Remove EXPERIMENTAL tags (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Skip bad symlinks in dir listing with -L enabled (C\['e]dric Connes)
|
|
.IP \[bu] 2
|
|
Preallocate files on Windows to reduce fragmentation (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Preallocate files on linux with fallocate(2) (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Cache
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add cache/fetch rc function (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Fix worker scale down (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Improve performance by not sending info requests for cached chunks
|
|
(dcpu)
|
|
.IP \[bu] 2
|
|
Fix error return value of cache/fetch rc method (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Documentation fix for cache-chunk-total-size (Anagh Kumar Baranwal)
|
|
.IP \[bu] 2
|
|
Preserve leading / in wrapped remote path (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Add plex_insecure option to skip certificate validation (Fabian
|
|
M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Remove entries that no longer exist in the source (dcpu)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Preserve leading / in wrapped remote path (Fabian M\[:o]ller)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Alias
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix handling of Windows network paths (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--azureblob-list-chunk\f[R] parameter (Santiago Rodr\['i]guez)
|
|
.IP \[bu] 2
|
|
Implemented settier command support on azureblob remote.
|
|
(sandeepkru)
|
|
.IP \[bu] 2
|
|
Work around SDK bug which causes errors for chunk-sized files (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement link sharing.
|
|
(Sebastian B\[:u]nger)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-import-formats\f[R] - google docs can now be imported
|
|
(Fabian M\[:o]ller)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rewrite mime type and extension handling (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Add document links (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Add support for multipart document extensions (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Add support for apps-script to json export (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Fix escaped chars in documents during list (Fabian M\[:o]ller)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-v2-download-min-size\f[R] a workaround for slow
|
|
downloads (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Improve directory notifications in ChangeNotify (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
When listing team drives in config, continue on failure (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add a small pause after failed upload before deleting file (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix service_account_file being ignored (Fabian M\[:o]ller)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Minor improvement in quota info (omit if unlimited) (albertony)
|
|
.IP \[bu] 2
|
|
Add \f[C]--fast-list\f[R] support (albertony)
|
|
.IP \[bu] 2
|
|
Add permanent delete support: \f[C]--jottacloud-hard-delete\f[R]
|
|
(albertony)
|
|
.IP \[bu] 2
|
|
Add link sharing support (albertony)
|
|
.IP \[bu] 2
|
|
Fix handling of reserved characters.
|
|
(Sebastian B\[:u]nger)
|
|
.IP \[bu] 2
|
|
Fix socket leak on Object.Remove (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rework to support Microsoft Graph (Cnly)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[B]NB\f[R] this will require re-authenticating the remote
|
|
.RE
|
|
.IP \[bu] 2
|
|
Removed upload cutoff and always do session uploads (Oliver Heyme)
|
|
.IP \[bu] 2
|
|
Use single-part upload for empty files (Cnly)
|
|
.IP \[bu] 2
|
|
Fix new fields not saved when editing old config (Alex Chen)
|
|
.IP \[bu] 2
|
|
Fix sometimes special chars in filenames not replaced (Alex Chen)
|
|
.IP \[bu] 2
|
|
Ignore OneNote files by default (Alex Chen)
|
|
.IP \[bu] 2
|
|
Add link sharing support (jackyzy823)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use custom pacer, to retry operations when reasonable (Craig Miskell)
|
|
.IP \[bu] 2
|
|
Use configured server-side-encryption and storage class options when
|
|
calling CopyObject() (Paul Kohout)
|
|
.IP \[bu] 2
|
|
Make \f[C]--s3-v2-auth\f[R] flag (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix v2 auth on files with spaces (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Union
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement union backend which reads from multiple backends (Felix
|
|
Brucker)
|
|
.IP \[bu] 2
|
|
Implement optional interfaces (Move, DirMove, Copy, etc.) (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix ChangeNotify to support multiple remotes (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Fix \f[C]--backup-dir\f[R] on union backend (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add another time format (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add a small pause after failed upload before deleting file (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add workaround for missing mtime (buergi)
|
|
.IP \[bu] 2
|
|
Sharepoint: Renew cookies after 12hrs (Henning Surmeier)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Yandex
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Remove redundant nil checks (teresy)
|
|
.RE
|
|
.SS v1.43.1 - 2018-09-07
|
|
.PP
|
|
Point release to fix hubic and azureblob backends.
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
ncdu: Return error instead of log.Fatal in Show (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
cmd: Fix crash with \f[C]--progress\f[R] and \f[C]--stats 0\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
docs: Tidy website display (Anagh Kumar Baranwal)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix multi-part uploads.
|
|
(sandeepkru)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Hubic
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix uploads (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Retry auth fetching if it fails to make hubic more reliable (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.SS v1.43 - 2018-09-01
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Jottacloud (Sebastian B\[:u]nger)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
copyurl: copies a URL to a remote (Denis)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Reworked config for backends (Nick Craig-Wood)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
All backend config can now be supplied by command line, env var or
|
|
config file
|
|
.IP \[bu] 2
|
|
Advanced section in the config wizard for the optional items
|
|
.IP \[bu] 2
|
|
A large step towards rclone backends being usable in other go software
|
|
.IP \[bu] 2
|
|
Allow on the fly remotes with :backend: syntax
|
|
.RE
|
|
.IP \[bu] 2
|
|
Stats revamp
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--progress\f[R]/\f[C]-P\f[R] flag to show interactive progress
|
|
(Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Show the total progress of the sync in the stats (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--stats-one-line\f[R] flag for single line stats (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Added weekday schedule into \f[C]--bwlimit\f[R] (Mateusz)
|
|
.IP \[bu] 2
|
|
lsjson: Add option to show the original object IDs (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
serve webdav: Make Content-Type without reading the file and add
|
|
\f[C]--etag-hash\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
build
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Build macOS with native compiler (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Update to use go1.11 for the build (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Added core/stats to return the stats (reddi1)
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]version --check\f[R]: Prints the current release and beta versions
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
accounting
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix time to completion estimates (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix moving average speed for file stats (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
config: Fix error reading password from piped input (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
move: Fix \f[C]--delete-empty-src-dirs\f[R] flag to delete all empty
|
|
dirs on move (ishuah)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]--daemon-timeout\f[R] flag for OSXFUSE (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix mount \f[C]--daemon\f[R] not working with encrypted config (Alex
|
|
Chen)
|
|
.IP \[bu] 2
|
|
Clip the number of blocks to 2\[ha]32-1 on macOS - fixes borg backup
|
|
(Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enable vfs-read-chunk-size by default (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Add the vfs/refresh rc command (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Add non recursive mode to vfs/refresh rc command (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Try to seek buffer on read only files (Fabian M\[:o]ller)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix crash when deprecated \f[C]--local-no-unicode-normalization\f[R] is
|
|
supplied (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix mkdir error when trying to copy files to the root of a drive on
|
|
windows (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Cache
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix nil pointer deref when using lsjson on cached directory (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix nil pointer deref for occasional crash on playback (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix accounting when checking hashes on upload (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Cloud Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make very clear in the docs that rclone has no ACD keys (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azure Blob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add connection string and SAS URL auth (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
List the container to see if it exists (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Port new Azure Blob Storage SDK (sandeepkru)
|
|
.IP \[bu] 2
|
|
Added blob tier, tier between Hot, Cool and Archive.
|
|
(sandeepkru)
|
|
.IP \[bu] 2
|
|
Remove leading / from paths (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Support Application Keys (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove leading / from paths (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix upload of > 2GB files on 32 bit platforms (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Make \f[C]--box-commit-retries\f[R] flag defaulting to 100 to fix large
|
|
uploads (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-keep-revision-forever\f[R] flag (lewapm)
|
|
.IP \[bu] 2
|
|
Handle gdocs when filtering file names in list (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Support using \f[C]--fast-list\f[R] for large speedups (Fabian
|
|
M\[:o]ller)
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix Put mkParentDir failed: 521 for BunnyCDN (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix index out of range error with \f[C]--fast-list\f[R] (Nick
|
|
Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Jottacloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix MD5 error check (Oliver Heyme)
|
|
.IP \[bu] 2
|
|
Handle empty time values (Martin Polden)
|
|
.IP \[bu] 2
|
|
Calculate missing MD5s (Oliver Heyme)
|
|
.IP \[bu] 2
|
|
Docs, fixes and tests for MD5 calculation (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add optional MimeTyper interface.
|
|
(Sebastian B\[:u]nger)
|
|
.IP \[bu] 2
|
|
Implement optional About interface (for \f[C]df\f[R] support).
|
|
(Sebastian B\[:u]nger)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mega
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Wait for events instead of arbitrary sleeping (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--mega-hard-delete\f[R] flag (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix failed logins with upper case chars in email (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Shared folder support (Yoni Jah)
|
|
.IP \[bu] 2
|
|
Implement DirMove (Cnly)
|
|
.IP \[bu] 2
|
|
Fix rmdir sometimes deleting directories with contents (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Delete half uploaded files on upload error (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Qingstor
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Remove leading / from paths (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix index out of range error with \f[C]--fast-list\f[R] (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add \f[C]--s3-force-path-style\f[R] (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add support for KMS Key ID (bsteiss)
|
|
.IP \[bu] 2
|
|
Remove leading / from paths (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]storage_policy\f[R] (Ruben Vandamme)
|
|
.IP \[bu] 2
|
|
Make it so just \f[C]storage_url\f[R] or \f[C]auth_token\f[R] can be
|
|
overridden (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Fix server-side copy bug for unusual file names (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Remove leading / from paths (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ensure we call MKCOL with a URL with a trailing / for QNAP interop (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
If root ends with / then don\[aq]t check if it is a file (Nick
|
|
Craig-Wood)
|
|
.IP \[bu] 2
|
|
Don\[aq]t accept redirects when reading metadata (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Add bearer token (Macaroon) support for dCache (Nick Craig-Wood)
|
|
.IP \[bu] 2
|
|
Document dCache and Macaroons (Onno Zweers)
|
|
.IP \[bu] 2
|
|
Sharepoint recursion with different depth (Henning)
|
|
.IP \[bu] 2
|
|
Attempt to remove failed uploads (Nick Craig-Wood)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Yandex
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix listing/deleting files in the root (Nick Craig-Wood)
|
|
.RE
|
|
.SS v1.42 - 2018-06-16
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
OpenDrive (Oliver Heyme, Jakub Karlicek, ncw)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
deletefile command (Filip Bartodziej)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
copy, move: Copy single files directly, don\[aq]t use
|
|
\f[C]--files-from\f[R] work-around
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes them much more efficient
|
|
.RE
|
|
.IP \[bu] 2
|
|
Implement \f[C]--max-transfer\f[R] flag to quit transferring at a limit
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
make exit code 8 for \f[C]--max-transfer\f[R] exceeded
|
|
.RE
|
|
.IP \[bu] 2
|
|
copy: copy empty source directories to destination (Ishuah Kariuki)
|
|
.IP \[bu] 2
|
|
check: Add \f[C]--one-way\f[R] flag (Kasper Byrdal Nielsen)
|
|
.IP \[bu] 2
|
|
Add siginfo handler for macOS for ctrl-T stats (kubatasiemski)
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
add core/gc to run a garbage collection on demand
|
|
.IP \[bu] 2
|
|
enable go profiling by default on the \f[C]--rc\f[R] port
|
|
.IP \[bu] 2
|
|
return error from remote on failure
|
|
.RE
|
|
.IP \[bu] 2
|
|
lsf
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--absolute\f[R] flag to add a leading / onto path names
|
|
.IP \[bu] 2
|
|
Add \f[C]--csv\f[R] flag for compliant CSV output
|
|
.IP \[bu] 2
|
|
Add \[aq]m\[aq] format specifier to show the MimeType
|
|
.IP \[bu] 2
|
|
Implement \[aq]i\[aq] format for showing object ID
|
|
.RE
|
|
.IP \[bu] 2
|
|
lsjson
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add MimeType to the output
|
|
.IP \[bu] 2
|
|
Add ID field to output to show Object ID
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]--retries-sleep\f[R] flag (Benjamin Joseph Dag)
|
|
.IP \[bu] 2
|
|
Oauth tidy up web page and error handling (Henning Surmeier)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Password prompt output with \f[C]--log-file\f[R] fixed for unix (Filip
|
|
Bartodziej)
|
|
.IP \[bu] 2
|
|
Calculate ModifyWindow each time on the fly to fix various problems
|
|
(Stefan Breunig)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Only print \[dq]File.rename error\[dq] if there actually is an error
|
|
(Stefan Breunig)
|
|
.IP \[bu] 2
|
|
Delay rename if file has open writers instead of failing outright
|
|
(Stefan Breunig)
|
|
.IP \[bu] 2
|
|
Ensure atexit gets run on interrupt
|
|
.IP \[bu] 2
|
|
macOS enhancements
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make \f[C]--noappledouble\f[R] \f[C]--noapplexattr\f[R]
|
|
.IP \[bu] 2
|
|
Add \f[C]--volname\f[R] flag and remove special chars from it
|
|
.IP \[bu] 2
|
|
Make Get/List/Set/Remove xattr return ENOSYS for efficiency
|
|
.IP \[bu] 2
|
|
Make \f[C]--daemon\f[R] work for macOS without CGO
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--vfs-read-chunk-size\f[R] and
|
|
\f[C]--vfs-read-chunk-size-limit\f[R] (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Fix ChangeNotify for new or changed folders (Fabian M\[:o]ller)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix symlink/junction point directory handling under Windows
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[B]NB\f[R] you will need to add \f[C]-L\f[R] to your command line to
|
|
copy files with reparse points
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Cache
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add non cached dirs on notifications (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Allow root to be expired from rc (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Clean remaining empty folders from temp upload path (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Cache lists using batch writes (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Use secure websockets for HTTPS Plex addresses (John Clayton)
|
|
.IP \[bu] 2
|
|
Reconnect plex websocket on failures (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Fix panic when running without plex configs (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Fix root folder caching (Remus Bunduc)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Check the crypted hash of files when uploading for extra data security
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make Dropbox for business folders accessible using an initial
|
|
\f[C]/\f[R] in the path
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Low level retry all operations if necessary
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-acknowledge-abuse\f[R] to download flagged files
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-alternate-export\f[R] to fix large doc export
|
|
.IP \[bu] 2
|
|
Don\[aq]t attempt to choose Team Drives when using rclone config create
|
|
.IP \[bu] 2
|
|
Fix change list polling with team drives
|
|
.IP \[bu] 2
|
|
Fix ChangeNotify for folders (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Fix about (and df on a mount) for team drives
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Errorhandler for onedrive for business requests (Henning Surmeier)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Adjust upload concurrency with \f[C]--s3-upload-concurrency\f[R]
|
|
(themylogin)
|
|
.IP \[bu] 2
|
|
Fix \f[C]--s3-chunk-size\f[R] which was always using the minimum
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--ssh-path-override\f[R] flag (Piotr Oleszczyk)
|
|
.IP \[bu] 2
|
|
Fix slow downloads for long latency connections
|
|
.RE
|
|
.IP \[bu] 2
|
|
Webdav
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add workarounds for biz.mail.ru
|
|
.IP \[bu] 2
|
|
Ignore Reason-Phrase in status line to fix 4shared (Rodrigo)
|
|
.IP \[bu] 2
|
|
Better error message generation
|
|
.RE
|
|
.SS v1.41 - 2018-04-28
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Mega support added
|
|
.IP \[bu] 2
|
|
Webdav now supports SharePoint cookie authentication (hensur)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
link: create public link to files and folders (Stefan Breunig)
|
|
.IP \[bu] 2
|
|
about: gets quota info from a remote (a-roussos, ncw)
|
|
.IP \[bu] 2
|
|
hashsum: a generic tool for any hash to produce md5sum like output
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
lsd: Add -R flag and fix and update docs for all ls commands
|
|
.IP \[bu] 2
|
|
ncdu: added a \[dq]refresh\[dq] key - CTRL-L (Keith Goldfarb)
|
|
.IP \[bu] 2
|
|
serve restic: Add append-only mode (Steve Kriss)
|
|
.IP \[bu] 2
|
|
serve restic: Disallow overwriting files in append-only mode (Alexander
|
|
Neumann)
|
|
.IP \[bu] 2
|
|
serve restic: Print actual listener address (Matt Holt)
|
|
.IP \[bu] 2
|
|
size: Add --json flag (Matthew Holt)
|
|
.IP \[bu] 2
|
|
sync: implement --ignore-errors (Mateusz Pabian)
|
|
.IP \[bu] 2
|
|
dedupe: Add dedupe largest functionality (Richard Yang)
|
|
.IP \[bu] 2
|
|
fs: Extend SizeSuffix to include TB and PB for rclone about
|
|
.IP \[bu] 2
|
|
fs: add --dump goroutines and --dump openfiles for debugging
|
|
.IP \[bu] 2
|
|
rc: implement core/memstats to print internal memory usage info
|
|
.IP \[bu] 2
|
|
rc: new call rc/pid (Michael P.
|
|
Dubner)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Compile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Drop support for go1.6
|
|
.RE
|
|
.IP \[bu] 2
|
|
Release
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix \f[C]make tarball\f[R] (Chih-Hsuan Yen)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
filter: fix --min-age and --max-age together check
|
|
.IP \[bu] 2
|
|
fs: limit MaxIdleConns and MaxIdleConnsPerHost in transport
|
|
.IP \[bu] 2
|
|
lsd,lsf: make sure all times we output are in local time
|
|
.IP \[bu] 2
|
|
rc: fix setting bwlimit to unlimited
|
|
.IP \[bu] 2
|
|
rc: take note of the --rc-addr flag too as per the docs
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use About to return the correct disk total/used/free (e.g.
|
|
in \f[C]df\f[R])
|
|
.IP \[bu] 2
|
|
Set \f[C]--attr-timeout default\f[R] to \f[C]1s\f[R] - fixes:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
rclone using too much memory
|
|
.IP \[bu] 2
|
|
rclone not serving files to samba
|
|
.IP \[bu] 2
|
|
excessive time listing directories
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix \f[C]df -i\f[R] (upstream fix)
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Filter files \f[C].\f[R] and \f[C]..\f[R] from directory listing
|
|
.IP \[bu] 2
|
|
Only make the VFS cache if --vfs-cache-mode > Off
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add --local-no-check-updated to disable updated file checks
|
|
.IP \[bu] 2
|
|
Retry remove on Windows sharing violation error
|
|
.RE
|
|
.IP \[bu] 2
|
|
Cache
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Flush the memory cache after close
|
|
.IP \[bu] 2
|
|
Purge file data on notification
|
|
.IP \[bu] 2
|
|
Always forget parent dir for notifications
|
|
.IP \[bu] 2
|
|
Integrate with Plex websocket
|
|
.IP \[bu] 2
|
|
Add rc cache/stats (seuffert)
|
|
.IP \[bu] 2
|
|
Add info log on notification
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix failure reading large directories - parse file/directory size as
|
|
float
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix crypt+obfuscate on dropbox
|
|
.IP \[bu] 2
|
|
Fix repeatedly uploading the same files
|
|
.RE
|
|
.IP \[bu] 2
|
|
FTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Work around strange response from box FTP server
|
|
.IP \[bu] 2
|
|
More workarounds for FTP servers to fix mkParentDir error
|
|
.IP \[bu] 2
|
|
Fix no error on listing non-existent directory
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add service_account_credentials (Matt Holt)
|
|
.IP \[bu] 2
|
|
Detect bucket presence by listing it - minimises permissions needed
|
|
.IP \[bu] 2
|
|
Ignore zero length directory markers
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add service_account_credentials (Matt Holt)
|
|
.IP \[bu] 2
|
|
Fix directory move leaving a hardlinked directory behind
|
|
.IP \[bu] 2
|
|
Return proper google errors when Opening files
|
|
.IP \[bu] 2
|
|
When initialized with a filepath, optional features used incorrect root
|
|
path (Stefan Breunig)
|
|
.RE
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix sync for servers which don\[aq]t return Content-Length in HEAD
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add QuickXorHash support for OneDrive for business
|
|
.IP \[bu] 2
|
|
Fix socket leak in multipart session upload
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Look in S3 named profile files for credentials
|
|
.IP \[bu] 2
|
|
Add \f[C]--s3-disable-checksum\f[R] to disable checksum uploading (Chris
|
|
Redekop)
|
|
.IP \[bu] 2
|
|
Hierarchical configuration support (Giri Badanahatti)
|
|
.IP \[bu] 2
|
|
Add in config for all the supported S3 providers
|
|
.IP \[bu] 2
|
|
Add One Zone Infrequent Access storage class (Craig Rachel)
|
|
.IP \[bu] 2
|
|
Add --use-server-modtime support (Peter Baumgartner)
|
|
.IP \[bu] 2
|
|
Add --s3-chunk-size option to control multipart uploads
|
|
.IP \[bu] 2
|
|
Ignore zero length directory markers
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update docs to match code, fix typos and clarify disable_hashcheck
|
|
prompt (Michael G.
|
|
Noll)
|
|
.IP \[bu] 2
|
|
Update docs with Synology quirks
|
|
.IP \[bu] 2
|
|
Fail soft with a debug on hash failure
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add --use-server-modtime support (Peter Baumgartner)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Webdav
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Support SharePoint cookie authentication (hensur)
|
|
.IP \[bu] 2
|
|
Strip leading and trailing / off root
|
|
.RE
|
|
.SS v1.40 - 2018-03-19
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Alias backend to create aliases for existing remote names (Fabian
|
|
M\[:o]ller)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[R]: list for parsing purposes (Jakub Tasiemski)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
by default this is a simple non recursive list of files and directories
|
|
.IP \[bu] 2
|
|
it can be configured to add more info in an easy to parse way
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]serve restic\f[R]: for serving a remote as a Restic REST endpoint
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This enables restic to use any backends that rclone can access
|
|
.IP \[bu] 2
|
|
Thanks Alexander Neumann for help, patches and review
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]rc\f[R]: enable the remote control of a running rclone
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The running rclone must be started with --rc and related flags.
|
|
.IP \[bu] 2
|
|
Currently there is support for bwlimit, and flushing for mount and
|
|
cache.
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]--max-delete\f[R] flag to add a delete threshold (Bj\[/o]rn Erik
|
|
Pedersen)
|
|
.IP \[bu] 2
|
|
All backends now support RangeOption for ranged Open
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]cat\f[R]: Use RangeOption for limited fetches to make more
|
|
efficient
|
|
.IP \[bu] 2
|
|
\f[C]cryptcheck\f[R]: make reading of nonce more efficient with
|
|
RangeOption
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve http/webdav/restic
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
support SSL/TLS
|
|
.IP \[bu] 2
|
|
add \f[C]--user\f[R] \f[C]--pass\f[R] and \f[C]--htpasswd\f[R] for
|
|
authentication
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]copy\f[R]/\f[C]move\f[R]: detect file size change during copy/move
|
|
and abort transfer (ishuah)
|
|
.IP \[bu] 2
|
|
\f[C]cryptdecode\f[R]: added option to return encrypted file names.
|
|
(ishuah)
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[R]: add \f[C]--encrypted\f[R] to show encrypted name
|
|
(Jakub Tasiemski)
|
|
.IP \[bu] 2
|
|
Add \f[C]--stats-file-name-length\f[R] to specify the printed file name
|
|
length for stats (Will Gunn)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Compile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Code base was shuffled and factored
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
backends moved into a backend directory
|
|
.IP \[bu] 2
|
|
large packages split up
|
|
.IP \[bu] 2
|
|
See the CONTRIBUTING.md doc for info as to what lives where now
|
|
.RE
|
|
.IP \[bu] 2
|
|
Update to using go1.10 as the default go version
|
|
.IP \[bu] 2
|
|
Implement daily full integration
|
|
tests (https://pub.rclone.org/integration-tests/)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Release
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Include a source tarball and sign it and the binaries
|
|
.IP \[bu] 2
|
|
Sign the git tags as part of the release process
|
|
.IP \[bu] 2
|
|
Add .deb and .rpm packages as part of the build
|
|
.IP \[bu] 2
|
|
Make a beta release for all branches on the main repo (but not pull
|
|
requests)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
config: fixes errors on non existing config by loading config file only
|
|
on first access
|
|
.IP \[bu] 2
|
|
config: retry saving the config after failure (Mateusz)
|
|
.IP \[bu] 2
|
|
sync: when using \f[C]--backup-dir\f[R] don\[aq]t delete files if we
|
|
can\[aq]t set their modtime
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this fixes odd behaviour with Dropbox and \f[C]--backup-dir\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
fshttp: fix idle timeouts for HTTP connections
|
|
.IP \[bu] 2
|
|
\f[C]serve http\f[R]: fix serving files with : in - fixes
|
|
.IP \[bu] 2
|
|
Fix \f[C]--exclude-if-present\f[R] to ignore directories which it
|
|
doesn\[aq]t have permission for (Iakov Davydov)
|
|
.IP \[bu] 2
|
|
Make accounting work properly with crypt and b2
|
|
.IP \[bu] 2
|
|
remove \f[C]--no-traverse\f[R] flag because it is obsolete
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--attr-timeout\f[R] flag to control attribute caching in kernel
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this now defaults to 0 which is correct but less efficient
|
|
.IP \[bu] 2
|
|
see the mount
|
|
docs (https://rclone.org/commands/rclone_mount/#attribute-caching) for
|
|
more info
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]--daemon\f[R] flag to allow mount to run in the background
|
|
(ishuah)
|
|
.IP \[bu] 2
|
|
Fix: Return ENOSYS rather than EIO on attempted link
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This fixes FileZilla accessing an rclone mount served over sftp.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix setting modtime twice
|
|
.IP \[bu] 2
|
|
Mount tests now run on CI for Linux (mount & cmount)/Mac/Windows
|
|
.IP \[bu] 2
|
|
Many bugs fixed in the VFS layer - see below
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Many fixes for \f[C]--vfs-cache-mode\f[R] writes and above
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update cached copy if we know it has changed (fixes stale data)
|
|
.IP \[bu] 2
|
|
Clean path names before using them in the cache
|
|
.IP \[bu] 2
|
|
Disable cache cleaner if \f[C]--vfs-cache-poll-interval=0\f[R]
|
|
.IP \[bu] 2
|
|
Fill and clean the cache immediately on startup
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix Windows opening every file when it stats the file
|
|
.IP \[bu] 2
|
|
Fix applying modtime for an open Write Handle
|
|
.IP \[bu] 2
|
|
Fix creation of files when truncating
|
|
.IP \[bu] 2
|
|
Write 0 bytes when flushing unwritten handles to avoid race conditions
|
|
in FUSE
|
|
.IP \[bu] 2
|
|
Downgrade \[dq]poll-interval is not supported\[dq] message to Info
|
|
.IP \[bu] 2
|
|
Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Downgrade \[dq]invalid cross-device link: trying copy\[dq] to debug
|
|
.IP \[bu] 2
|
|
Make DirMove return fs.ErrorCantDirMove to allow fallback to Copy for
|
|
cross device
|
|
.IP \[bu] 2
|
|
Fix race conditions updating the hashes
|
|
.RE
|
|
.IP \[bu] 2
|
|
Cache
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for polling - cache will update when remote changes on
|
|
supported backends
|
|
.IP \[bu] 2
|
|
Reduce log level for Plex api
|
|
.IP \[bu] 2
|
|
Fix dir cache issue
|
|
.IP \[bu] 2
|
|
Implement \f[C]--cache-db-wait-time\f[R] flag
|
|
.IP \[bu] 2
|
|
Improve efficiency with RangeOption and RangeSeek
|
|
.IP \[bu] 2
|
|
Fix dirmove with temp fs enabled
|
|
.IP \[bu] 2
|
|
Notify vfs when using temp fs
|
|
.IP \[bu] 2
|
|
Offline uploading
|
|
.IP \[bu] 2
|
|
Remote control support for path flushing
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon cloud drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rclone no longer has any working keys - disable integration tests
|
|
.IP \[bu] 2
|
|
Implement DirChangeNotify to notify cache/vfs/mount of changes
|
|
.RE
|
|
.IP \[bu] 2
|
|
Azureblob
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presence if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.IP \[bu] 2
|
|
Improve accounting for chunked uploads
|
|
.RE
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presence if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Improve accounting for chunked uploads
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix custom oauth client parameters
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presence if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Migrate to api v3 (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Add scope configuration and root folder selection
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-impersonate\f[R] for service accounts
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
thanks to everyone who tested, explored and contributed docs
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-use-created-date\f[R] to use created date as modified
|
|
date (nbuchanan)
|
|
.IP \[bu] 2
|
|
Request the export formats only when required
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This makes rclone quicker when there are no google docs
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix finding paths with latin1 chars (a workaround for a drive bug)
|
|
.IP \[bu] 2
|
|
Fix copying of a single Google doc file
|
|
.IP \[bu] 2
|
|
Fix \f[C]--drive-auth-owner-only\f[R] to look in all directories
|
|
.RE
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix handling of directories with & in
|
|
.RE
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Removed upload cutoff and always do session uploads
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this stops the creation of multiple versions on business onedrive
|
|
.RE
|
|
.IP \[bu] 2
|
|
Overwrite object size value with real size when reading file.
|
|
(Victor)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this fixes oddities when onedrive misreports the size of images
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Remove unused chunked upload flag and code
|
|
.RE
|
|
.IP \[bu] 2
|
|
Qingstor
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presence if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Support hashes for multipart files (Chris Redekop)
|
|
.IP \[bu] 2
|
|
Initial support for IBM COS (S3) (Giri Badanahatti)
|
|
.IP \[bu] 2
|
|
Update docs to discourage use of v2 auth with CEPH and others
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presence if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix server-side copy and set modtime on files with + in
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add option to disable remote hash check command execution (Jon Fautley)
|
|
.IP \[bu] 2
|
|
Add \f[C]--sftp-ask-password\f[R] flag to prompt for password when
|
|
needed (Leo R.
|
|
Lundgren)
|
|
.IP \[bu] 2
|
|
Add \f[C]set_modtime\f[R] configuration option
|
|
.IP \[bu] 2
|
|
Fix following of symlinks
|
|
.IP \[bu] 2
|
|
Fix reading config file outside of Fs setup
|
|
.IP \[bu] 2
|
|
Fix reading $USER in username fallback not $HOME
|
|
.IP \[bu] 2
|
|
Fix running under crontab - Use correct OS way of reading username
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix refresh of authentication token
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
in v1.39 a bug was introduced which ignored new tokens - this fixes it
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix extra HEAD transaction when uploading a new file
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presence if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Webdav
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add new time formats to support mydrive.ch and others
|
|
.RE
|
|
.SS v1.39 - 2017-12-23
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
tested with nextcloud, owncloud, put.io and others!
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.IP \[bu] 2
|
|
cache - wraps a cache around other backends (Remus Bunduc)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
useful in combination with mount
|
|
.IP \[bu] 2
|
|
NB this feature is in beta so use with care
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
serve command with subcommands:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
serve webdav: this implements a webdav server for any rclone remote.
|
|
.IP \[bu] 2
|
|
serve http: command to serve a remote over HTTP
|
|
.RE
|
|
.IP \[bu] 2
|
|
config: add sub commands for full config file management
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
create/delete/dump/edit/file/password/providers/show/update
|
|
.RE
|
|
.IP \[bu] 2
|
|
touch: to create or update the timestamp of a file (Jakub Tasiemski)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
curl install for rclone (Filip Bartodziej)
|
|
.IP \[bu] 2
|
|
--stats now shows percentage, size, rate and ETA in condensed form
|
|
(Ishuah Kariuki)
|
|
.IP \[bu] 2
|
|
--exclude-if-present to exclude a directory if a file is present (Iakov
|
|
Davydov)
|
|
.IP \[bu] 2
|
|
rmdirs: add --leave-root flag (lewapm)
|
|
.IP \[bu] 2
|
|
move: add --delete-empty-src-dirs flag to remove dirs after move (Ishuah
|
|
Kariuki)
|
|
.IP \[bu] 2
|
|
Add --dump flag, introduce --dump requests, responses and remove
|
|
--dump-auth, --dump-filters
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Obscure X-Auth-Token: from headers when dumping too
|
|
.RE
|
|
.IP \[bu] 2
|
|
Document and implement exit codes for different failure modes (Ishuah
|
|
Kariuki)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Compile
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Retry lots more different types of errors to make multipart transfers
|
|
more reliable
|
|
.IP \[bu] 2
|
|
Save the config before asking for a token, fixes disappearing oauth
|
|
config
|
|
.IP \[bu] 2
|
|
Warn the user if --include and --exclude are used together (Ernest
|
|
Borowski)
|
|
.IP \[bu] 2
|
|
Fix duplicate files (e.g.
|
|
on Google drive) causing spurious copies
|
|
.IP \[bu] 2
|
|
Allow trailing and leading whitespace for passwords (Jason Rose)
|
|
.IP \[bu] 2
|
|
ncdu: fix crashes on empty directories
|
|
.IP \[bu] 2
|
|
rcat: fix goroutine leak
|
|
.IP \[bu] 2
|
|
moveto/copyto: Fix to allow copying to the same name
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
--vfs-cache mode to make writes into mounts more reliable.
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this requires caching files on the disk (see --cache-dir)
|
|
.IP \[bu] 2
|
|
As this is a new feature, use with care
|
|
.RE
|
|
.IP \[bu] 2
|
|
Use sdnotify to signal systemd the mount is ready (Fabian M\[:o]ller)
|
|
.IP \[bu] 2
|
|
Check if directory is not empty before mounting (Ernest Borowski)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add error message for cross file system moves
|
|
.IP \[bu] 2
|
|
Fix equality check for times
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rework multipart upload
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
buffer the chunks when uploading large files so they can be retried
|
|
.IP \[bu] 2
|
|
change default chunk size to 48MB now we are buffering them in memory
|
|
.IP \[bu] 2
|
|
retry every error after the first chunk is done successfully
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix error when renaming directories
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix crash on bad authentication
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add service account support (Tim Cooijmans)
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make it work properly with Digital Ocean Spaces (Andrew
|
|
Starr-Bochicchio)
|
|
.IP \[bu] 2
|
|
Fix crash if a bad listing is received
|
|
.IP \[bu] 2
|
|
Add support for ECS task IAM roles (David Minor)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix multipart upload retries
|
|
.IP \[bu] 2
|
|
Fix --hard-delete to make it work 100% of the time
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow authentication with storage URL and auth key (Giovanni Pizzi)
|
|
.IP \[bu] 2
|
|
Add new fields for swift configuration to support IBM Bluemix Swift
|
|
(Pierre Carlson)
|
|
.IP \[bu] 2
|
|
Add OS_TENANT_ID and OS_USER_ID to config
|
|
.IP \[bu] 2
|
|
Allow configs with user id instead of user name
|
|
.IP \[bu] 2
|
|
Check if swift segments container exists before creating (John Leach)
|
|
.IP \[bu] 2
|
|
Fix memory leak in swift transfers (upstream fix)
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add option to enable the use of aes128-cbc cipher (Jon Fautley)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon cloud drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix download of large files failing with \[dq]Only one auth mechanism
|
|
allowed\[dq]
|
|
.RE
|
|
.IP \[bu] 2
|
|
crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Option to encrypt directory names or leave them intact
|
|
.IP \[bu] 2
|
|
Implement DirChangeNotify (Fabian M\[:o]ller)
|
|
.RE
|
|
.IP \[bu] 2
|
|
onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add option to choose resourceURL during setup of OneDrive Business
|
|
account if more than one is available for user
|
|
.RE
|
|
.SS v1.38 - 2017-09-30
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Azure Blob Storage (thanks Andrei Dragomir)
|
|
.IP \[bu] 2
|
|
Box
|
|
.IP \[bu] 2
|
|
Onedrive for Business (thanks Oliver Heyme)
|
|
.IP \[bu] 2
|
|
QingStor from QingCloud (thanks wuyu)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]rcat\f[R] - read from standard input and stream upload
|
|
.IP \[bu] 2
|
|
\f[C]tree\f[R] - shows a nicely formatted recursive listing
|
|
.IP \[bu] 2
|
|
\f[C]cryptdecode\f[R] - decode crypted file names (thanks ishuah)
|
|
.IP \[bu] 2
|
|
\f[C]config show\f[R] - print the config file
|
|
.IP \[bu] 2
|
|
\f[C]config file\f[R] - print the config file location
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Empty directories are deleted on \f[C]sync\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]dedupe\f[R] - implement merging of duplicate directories
|
|
.IP \[bu] 2
|
|
\f[C]check\f[R] and \f[C]cryptcheck\f[R] made more consistent and use
|
|
less memory
|
|
.IP \[bu] 2
|
|
\f[C]cleanup\f[R] for remaining remotes (thanks ishuah)
|
|
.IP \[bu] 2
|
|
\f[C]--immutable\f[R] for ensuring that files don\[aq]t change (thanks
|
|
Jacob McNamee)
|
|
.IP \[bu] 2
|
|
\f[C]--user-agent\f[R] option (thanks Alex McGrath Kraak)
|
|
.IP \[bu] 2
|
|
\f[C]--disable\f[R] flag to disable optional features
|
|
.IP \[bu] 2
|
|
\f[C]--bind\f[R] flag for choosing the local addr on outgoing
|
|
connections
|
|
.IP \[bu] 2
|
|
Support for zsh auto-completion (thanks bpicode)
|
|
.IP \[bu] 2
|
|
Stop normalizing file names but do a normalized compare in
|
|
\f[C]sync\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
Compile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update to using go1.9 as the default go version
|
|
.IP \[bu] 2
|
|
Remove snapd build due to maintenance problems
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Improve retriable error detection which makes multipart uploads better
|
|
.IP \[bu] 2
|
|
Make \f[C]check\f[R] obey \f[C]--ignore-size\f[R]
|
|
.IP \[bu] 2
|
|
Fix bwlimit toggle in conjunction with schedules (thanks cbruegg)
|
|
.IP \[bu] 2
|
|
\f[C]config\f[R] ensures newly written config is on the same mount
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Revert to copy when moving file across file system boundaries
|
|
.IP \[bu] 2
|
|
\f[C]--skip-links\f[R] to suppress symlink warnings (thanks Zhiming
|
|
Wang)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Re-use \f[C]rcat\f[R] internals to support uploads from all remotes
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix \[dq]entry doesn\[aq]t belong in directory\[dq] error
|
|
.IP \[bu] 2
|
|
Stop using deprecated API methods
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix server-side copy to empty container with \f[C]--fast-list\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Change the default for \f[C]--drive-use-trash\f[R] to \f[C]true\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Set session token when using STS (thanks Girish Ramakrishnan)
|
|
.IP \[bu] 2
|
|
Glacier docs and error messages (thanks Jan Varho)
|
|
.IP \[bu] 2
|
|
Read 1000 (not 1024) items in dir listings to fix Wasabi
|
|
.RE
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix SHA1 mismatch when downloading files with no SHA1
|
|
.IP \[bu] 2
|
|
Calculate missing hashes on the fly instead of spooling
|
|
.IP \[bu] 2
|
|
\f[C]--b2-hard-delete\f[R] to permanently delete (not hide) files
|
|
(thanks John Papandriopoulos)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Hubic
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix creating containers - no longer have to use the \f[C]default\f[R]
|
|
container
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Optionally configure from a standard set of OpenStack environment vars
|
|
.IP \[bu] 2
|
|
Add \f[C]endpoint_type\f[R] config
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix bucket creation to work with limited permission users
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement connection pooling for multiple ssh connections
|
|
.IP \[bu] 2
|
|
Limit new connections per second
|
|
.IP \[bu] 2
|
|
Add support for MD5 and SHA1 hashes where available (thanks Christian
|
|
Br\[:u]ggemann)
|
|
.RE
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix URL encoding issues
|
|
.IP \[bu] 2
|
|
Fix directories with \f[C]:\f[R] in
|
|
.IP \[bu] 2
|
|
Fix panic with URL encoded content
|
|
.RE
|
|
.SS v1.37 - 2017-07-22
|
|
.IP \[bu] 2
|
|
New backends
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
FTP - thanks to Antonio Messina
|
|
.IP \[bu] 2
|
|
HTTP - thanks to Vasiliy Tolstov
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
rclone ncdu - for exploring a remote with a text based user interface.
|
|
.IP \[bu] 2
|
|
rclone lsjson - for listing with a machine-readable output
|
|
.IP \[bu] 2
|
|
rclone dbhashsum - to show Dropbox style hashes of files (local or
|
|
Dropbox)
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement --fast-list flag
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This allows remotes to list recursively if they can
|
|
.IP \[bu] 2
|
|
This uses less transactions (important if you pay for them)
|
|
.IP \[bu] 2
|
|
This may or may not be quicker
|
|
.IP \[bu] 2
|
|
This will use more memory as it has to hold the listing in memory
|
|
.IP \[bu] 2
|
|
--old-sync-method deprecated - the remaining uses are covered by
|
|
--fast-list
|
|
.IP \[bu] 2
|
|
This involved a major re-write of all the listing code
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add --tpslimit and --tpslimit-burst to limit transactions per second
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this is useful in conjunction with \f[C]rclone mount\f[R] to limit
|
|
external apps
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add --stats-log-level so can see --stats without -v
|
|
.IP \[bu] 2
|
|
Print password prompts to stderr - Hraban Luyat
|
|
.IP \[bu] 2
|
|
Warn about duplicate files when syncing
|
|
.IP \[bu] 2
|
|
Oauth improvements
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
allow auth_url and token_url to be set in the config file
|
|
.IP \[bu] 2
|
|
Print redirection URI if using own credentials.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Don\[aq]t Mkdir at the start of sync to save transactions
|
|
.RE
|
|
.IP \[bu] 2
|
|
Compile
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update build to go1.8.3
|
|
.IP \[bu] 2
|
|
Require go1.6 for building rclone
|
|
.IP \[bu] 2
|
|
Compile 386 builds with \[dq]GO386=387\[dq] for maximum compatibility
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix menu selection when no remotes
|
|
.IP \[bu] 2
|
|
Config saving reworked to not kill the file if disk gets full
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete remote if name does not change while renaming
|
|
.IP \[bu] 2
|
|
moveto, copyto: report transfers and checks as per move and copy
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add --local-no-unicode-normalization flag - Bob Potter
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Now supported on Windows using cgofuse and WinFsp - thanks to Bill
|
|
Zissimopoulos for much help
|
|
.IP \[bu] 2
|
|
Compare checksums on upload/download via FUSE
|
|
.IP \[bu] 2
|
|
Unmount when program ends with SIGINT (Ctrl+C) or SIGTERM -
|
|
J\['e]r\[^o]me Vizcaino
|
|
.IP \[bu] 2
|
|
On read only open of file, make open pending until first read
|
|
.IP \[bu] 2
|
|
Make --read-only reject modify operations
|
|
.IP \[bu] 2
|
|
Implement ModTime via FUSE for remotes that support it
|
|
.IP \[bu] 2
|
|
Allow modTime to be changed even before all writers are closed
|
|
.IP \[bu] 2
|
|
Fix panic on renames
|
|
.IP \[bu] 2
|
|
Fix hang on errored upload
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Report the name:root as specified by the user
|
|
.IP \[bu] 2
|
|
Add an \[dq]obfuscate\[dq] option for filename encryption - Stephen
|
|
Harris
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix initialization order for token renewer
|
|
.IP \[bu] 2
|
|
Remove revoked credentials, allow oauth proxy config and update docs
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Reduce minimum chunk size to 5MB
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add team drive support
|
|
.IP \[bu] 2
|
|
Reduce bandwidth by adding fields for partial responses - Martin
|
|
Kristensen
|
|
.IP \[bu] 2
|
|
Implement --drive-shared-with-me flag to view shared with me files -
|
|
Danny Tsai
|
|
.IP \[bu] 2
|
|
Add --drive-trashed-only to read only the files in the trash
|
|
.IP \[bu] 2
|
|
Remove obsolete --drive-full-list
|
|
.IP \[bu] 2
|
|
Add missing seek to start on retries of chunked uploads
|
|
.IP \[bu] 2
|
|
Fix stats accounting for upload
|
|
.IP \[bu] 2
|
|
Convert / in names to a unicode equivalent (\[uFF0F])
|
|
.IP \[bu] 2
|
|
Poll for Google Drive changes when mounted
|
|
.RE
|
|
.IP \[bu] 2
|
|
OneDrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix the uploading of files with spaces
|
|
.IP \[bu] 2
|
|
Fix initialization order for token renewer
|
|
.IP \[bu] 2
|
|
Display speeds accurately when uploading - Yoni Jah
|
|
.IP \[bu] 2
|
|
Swap to using http://localhost:53682/ as redirect URL - Michael Ledin
|
|
.IP \[bu] 2
|
|
Retry on token expired error, reset upload body on retry - Yoni Jah
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add ability to specify location and storage class via config and command
|
|
line - thanks gdm85
|
|
.IP \[bu] 2
|
|
Create container if necessary on server-side copy
|
|
.IP \[bu] 2
|
|
Increase directory listing chunk to 1000 to increase performance
|
|
.IP \[bu] 2
|
|
Obtain a refresh token for GCS - Steven Lu
|
|
.RE
|
|
.IP \[bu] 2
|
|
Yandex
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix the name reported in log messages (was empty)
|
|
.IP \[bu] 2
|
|
Correct error return for listing empty directory
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rewritten to use the v2 API
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Now supports ModTime
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Can only set by uploading the file again
|
|
.IP \[bu] 2
|
|
If you uploaded with an old rclone, rclone may upload everything again
|
|
.IP \[bu] 2
|
|
Use \f[C]--size-only\f[R] or \f[C]--checksum\f[R] to avoid this
|
|
.RE
|
|
.IP \[bu] 2
|
|
Now supports the Dropbox content hashing scheme
|
|
.IP \[bu] 2
|
|
Now supports low level retries
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Work around eventual consistency in bucket creation
|
|
.IP \[bu] 2
|
|
Create container if necessary on server-side copy
|
|
.IP \[bu] 2
|
|
Add us-east-2 (Ohio) and eu-west-2 (London) S3 regions - Zahiar Ahmed
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift, Hubic
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix zero length directory markers showing in the subdirectory listing
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this caused lots of duplicate transfers
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix paged directory listings
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this caused duplicate directory errors
|
|
.RE
|
|
.IP \[bu] 2
|
|
Create container if necessary on server-side copy
|
|
.IP \[bu] 2
|
|
Increase directory listing chunk to 1000 to increase performance
|
|
.IP \[bu] 2
|
|
Make sensible error if the user forgets the container
|
|
.RE
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for using ssh key files
|
|
.IP \[bu] 2
|
|
Fix under Windows
|
|
.IP \[bu] 2
|
|
Fix ssh agent on Windows
|
|
.IP \[bu] 2
|
|
Adapt to latest version of library - Igor Kharin
|
|
.RE
|
|
.SS v1.36 - 2017-03-18
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
SFTP remote (Jack Schmidt)
|
|
.IP \[bu] 2
|
|
Re-implement sync routine to work a directory at a time reducing memory
|
|
usage
|
|
.IP \[bu] 2
|
|
Logging revamped to be more inline with rsync - now much quieter * -v
|
|
only shows transfers * -vv is for full debug * --syslog to log to syslog
|
|
on capable platforms
|
|
.IP \[bu] 2
|
|
Implement --backup-dir and --suffix
|
|
.IP \[bu] 2
|
|
Implement --track-renames (initial implementation by Bj\[/o]rn Erik
|
|
Pedersen)
|
|
.IP \[bu] 2
|
|
Add time-based bandwidth limits (Lukas Loesche)
|
|
.IP \[bu] 2
|
|
rclone cryptcheck: checks integrity of crypt remotes
|
|
.IP \[bu] 2
|
|
Allow all config file variables and options to be set from environment
|
|
variables
|
|
.IP \[bu] 2
|
|
Add --buffer-size parameter to control buffer size for copy
|
|
.IP \[bu] 2
|
|
Make --delete-after the default
|
|
.IP \[bu] 2
|
|
Add --ignore-checksum flag (fixed by Hisham Zarka)
|
|
.IP \[bu] 2
|
|
rclone check: Add --download flag to check all the data, not just hashes
|
|
.IP \[bu] 2
|
|
rclone cat: add --head, --tail, --offset, --count and --discard
|
|
.IP \[bu] 2
|
|
rclone config: when choosing from a list, allow the value to be entered
|
|
too
|
|
.IP \[bu] 2
|
|
rclone config: allow rename and copy of remotes
|
|
.IP \[bu] 2
|
|
rclone obscure: for generating encrypted passwords for rclone\[aq]s
|
|
config (T.C.
|
|
Ferguson)
|
|
.IP \[bu] 2
|
|
Comply with XDG Base Directory specification (Dario Giovannetti)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this moves the default location of the config file in a backwards
|
|
compatible way
|
|
.RE
|
|
.IP \[bu] 2
|
|
Release changes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ubuntu snap support (Dedsec1)
|
|
.IP \[bu] 2
|
|
Compile with go 1.8
|
|
.IP \[bu] 2
|
|
MIPS/Linux big and little endian support
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix copyto copying things to the wrong place if the destination dir
|
|
didn\[aq]t exist
|
|
.IP \[bu] 2
|
|
Fix parsing of remotes in moveto and copyto
|
|
.IP \[bu] 2
|
|
Fix --delete-before deleting files on copy
|
|
.IP \[bu] 2
|
|
Fix --files-from with an empty file copying everything
|
|
.IP \[bu] 2
|
|
Fix sync: don\[aq]t update mod times if --dry-run set
|
|
.IP \[bu] 2
|
|
Fix MimeType propagation
|
|
.IP \[bu] 2
|
|
Fix filters to add ** rules to directory rules
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement -L, --copy-links flag to allow rclone to follow symlinks
|
|
.IP \[bu] 2
|
|
Open files in write only mode so rclone can write to an rclone mount
|
|
.IP \[bu] 2
|
|
Fix unnormalised unicode causing problems reading directories
|
|
.IP \[bu] 2
|
|
Fix interaction between -x flag and --max-depth
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement proper directory handling (mkdir, rmdir, renaming)
|
|
.IP \[bu] 2
|
|
Make include and exclude filters apply to mount
|
|
.IP \[bu] 2
|
|
Implement read and write async buffers - control with --buffer-size
|
|
.IP \[bu] 2
|
|
Fix fsync on for directories
|
|
.IP \[bu] 2
|
|
Fix retry on network failure when reading off crypt
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add --crypt-show-mapping to show encrypted file mapping
|
|
.IP \[bu] 2
|
|
Fix crypt writer getting stuck in a loop
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[B]IMPORTANT\f[R] this bug had the potential to cause data corruption
|
|
when
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
reading data from a network based remote and
|
|
.IP \[bu] 2
|
|
writing to a crypt on Google Drive
|
|
.RE
|
|
.IP \[bu] 2
|
|
Use the cryptcheck command to validate your data if you are concerned
|
|
.IP \[bu] 2
|
|
If syncing two crypt remotes, sync the unencrypted remote
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix panics on Move (rename)
|
|
.IP \[bu] 2
|
|
Fix panic on token expiry
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix inconsistent listings and rclone check
|
|
.IP \[bu] 2
|
|
Fix uploading empty files with go1.8
|
|
.IP \[bu] 2
|
|
Constrain memory usage when doing multipart uploads
|
|
.IP \[bu] 2
|
|
Fix upload url not being refreshed properly
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix Rmdir on directories with trashed files
|
|
.IP \[bu] 2
|
|
Fix \[dq]Ignoring unknown object\[dq] when downloading
|
|
.IP \[bu] 2
|
|
Add --drive-list-chunk
|
|
.IP \[bu] 2
|
|
Add --drive-skip-gdocs (K\['a]roly Ol\['a]h)
|
|
.RE
|
|
.IP \[bu] 2
|
|
OneDrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement Move
|
|
.IP \[bu] 2
|
|
Fix Copy
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix overwrite detection in Copy
|
|
.IP \[bu] 2
|
|
Fix waitForJob to parse errors correctly
|
|
.RE
|
|
.IP \[bu] 2
|
|
Use token renewer to stop auth errors on long uploads
|
|
.IP \[bu] 2
|
|
Fix uploading empty files with go1.8
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix depth 1 directory listings
|
|
.RE
|
|
.IP \[bu] 2
|
|
Yandex
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix single level directory listing
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Normalise the case for single level directory listings
|
|
.IP \[bu] 2
|
|
Fix depth 1 listing
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Added ca-central-1 region (Jon Yergatian)
|
|
.RE
|
|
.SS v1.35 - 2017-01-02
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
moveto and copyto commands for choosing a destination name on copy/move
|
|
.IP \[bu] 2
|
|
rmdirs command to recursively delete empty directories
|
|
.IP \[bu] 2
|
|
Allow repeated --include/--exclude/--filter options
|
|
.IP \[bu] 2
|
|
Only show transfer stats on commands which transfer stuff
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
show stats on any command using the \f[C]--stats\f[R] flag
|
|
.RE
|
|
.IP \[bu] 2
|
|
Allow overlapping directories in move when server-side dir move is
|
|
supported
|
|
.IP \[bu] 2
|
|
Add --stats-unit option - thanks Scott McGillivray
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix the config file being overwritten when two rclone instances are
|
|
running
|
|
.IP \[bu] 2
|
|
Make rclone lsd obey the filters properly
|
|
.IP \[bu] 2
|
|
Fix compilation on mips
|
|
.IP \[bu] 2
|
|
Fix not transferring files that don\[aq]t differ in size
|
|
.IP \[bu] 2
|
|
Fix panic on nil retry/fatal error
|
|
.RE
|
|
.IP \[bu] 2
|
|
Mount
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Retry reads on error - should help with reliability a lot
|
|
.IP \[bu] 2
|
|
Report the modification times for directories from the remote
|
|
.IP \[bu] 2
|
|
Add bandwidth accounting and limiting (fixes --bwlimit)
|
|
.IP \[bu] 2
|
|
If --stats provided will show stats and which files are transferring
|
|
.IP \[bu] 2
|
|
Support R/W files if truncate is set.
|
|
.IP \[bu] 2
|
|
Implement statfs interface so df works
|
|
.IP \[bu] 2
|
|
Note that write is now supported on Amazon Drive
|
|
.IP \[bu] 2
|
|
Report number of blocks in a file - thanks Stefan Breunig
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Prevent the user pointing crypt at itself
|
|
.IP \[bu] 2
|
|
Fix failed to authenticate decrypted block errors
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
these will now return the underlying unexpected EOF instead
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for server-side move and directory move - thanks Stefan
|
|
Breunig
|
|
.IP \[bu] 2
|
|
Fix nil pointer deref on size attribute
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use new prefix and delimiter parameters in directory listings
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This makes --max-depth 1 dir listings as used in mount much faster
|
|
.RE
|
|
.IP \[bu] 2
|
|
Reauth the account while doing uploads too - should help with token
|
|
expiry
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make DirMove more efficient and complain about moving the root
|
|
.IP \[bu] 2
|
|
Create destination directory on Move()
|
|
.RE
|
|
.SS v1.34 - 2016-11-06
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Stop single file and \f[C]--files-from\f[R] operations iterating through
|
|
the source bucket.
|
|
.IP \[bu] 2
|
|
Stop removing failed upload to cloud storage remotes
|
|
.IP \[bu] 2
|
|
Make ContentType be preserved for cloud to cloud copies
|
|
.IP \[bu] 2
|
|
Add support to toggle bandwidth limits via SIGUSR2 - thanks Marco
|
|
Paganini
|
|
.IP \[bu] 2
|
|
\f[C]rclone check\f[R] shows count of hashes that couldn\[aq]t be
|
|
checked
|
|
.IP \[bu] 2
|
|
\f[C]rclone listremotes\f[R] command
|
|
.IP \[bu] 2
|
|
Support linux/arm64 build - thanks Fredrik Fornwall
|
|
.IP \[bu] 2
|
|
Remove \f[C]Authorization:\f[R] lines from \f[C]--dump-headers\f[R]
|
|
output
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ignore files with control characters in the names
|
|
.IP \[bu] 2
|
|
Fix \f[C]rclone move\f[R] command
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Delete src files which already existed in dst
|
|
.IP \[bu] 2
|
|
Fix deletion of src file when dst file older
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix \f[C]rclone check\f[R] on crypted file systems
|
|
.IP \[bu] 2
|
|
Make failed uploads not count as \[dq]Transferred\[dq]
|
|
.IP \[bu] 2
|
|
Make sure high level retries show with \f[C]-q\f[R]
|
|
.IP \[bu] 2
|
|
Use a vendor directory with godep for repeatable builds
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]rclone mount\f[R] - FUSE
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement FUSE mount options
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]--no-modtime\f[R], \f[C]--debug-fuse\f[R], \f[C]--read-only\f[R],
|
|
\f[C]--allow-non-empty\f[R], \f[C]--allow-root\f[R],
|
|
\f[C]--allow-other\f[R]
|
|
.IP \[bu] 2
|
|
\f[C]--default-permissions\f[R], \f[C]--write-back-cache\f[R],
|
|
\f[C]--max-read-ahead\f[R], \f[C]--umask\f[R], \f[C]--uid\f[R],
|
|
\f[C]--gid\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]--dir-cache-time\f[R] to control caching of directory entries
|
|
.IP \[bu] 2
|
|
Implement seek for files opened for read (useful for video players)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
with \f[C]-no-seek\f[R] flag to disable
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix crash on 32 bit ARM (alignment of 64 bit counter)
|
|
.IP \[bu] 2
|
|
\&...and many more internal fixes and improvements!
|
|
.RE
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t show encrypted password in configurator to stop confusion
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New wait for upload option \f[C]--acd-upload-wait-per-gb\f[R]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
upload timeouts scale by file size and can be disabled
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add 502 Bad Gateway to list of errors we retry
|
|
.IP \[bu] 2
|
|
Fix overwriting a file with a zero length file
|
|
.IP \[bu] 2
|
|
Fix ACD file size warning limit - thanks Felix B\[:u]nemann
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Unix: implement \f[C]-x\f[R]/\f[C]--one-file-system\f[R] to stay on a
|
|
single file system
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
thanks Durval Menezes and Luiz Carlos Rumbelsperger Viana
|
|
.RE
|
|
.IP \[bu] 2
|
|
Windows: ignore the symlink bit on files
|
|
.IP \[bu] 2
|
|
Windows: Ignore directory-based junction points
|
|
.RE
|
|
.IP \[bu] 2
|
|
B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make sure each upload has at least one upload slot - fixes strange
|
|
upload stats
|
|
.IP \[bu] 2
|
|
Fix uploads when using crypt
|
|
.IP \[bu] 2
|
|
Fix download of large files (sha1 mismatch)
|
|
.IP \[bu] 2
|
|
Return error when we try to create a bucket which someone else owns
|
|
.IP \[bu] 2
|
|
Update B2 docs with Data usage, and Crypt section - thanks Tomasz Mazur
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Command line and config file support for
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Setting/overriding ACL - thanks Radek Senfeld
|
|
.IP \[bu] 2
|
|
Setting storage class - thanks Asko Tamm
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make exponential backoff work exactly as per Google specification
|
|
.IP \[bu] 2
|
|
add \f[C].epub\f[R], \f[C].odp\f[R] and \f[C].tsv\f[R] as export
|
|
formats.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t read metadata for directory marker objects
|
|
.RE
|
|
.SS v1.33 - 2016-08-24
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement encryption
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
data encrypted in NACL secretbox format
|
|
.IP \[bu] 2
|
|
with optional file name encryption
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
rclone mount - implements FUSE mounting of remotes (EXPERIMENTAL)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
works on Linux, FreeBSD and OS X (need testers for the last 2!)
|
|
.RE
|
|
.IP \[bu] 2
|
|
rclone cat - outputs remote file or files to the terminal
|
|
.IP \[bu] 2
|
|
rclone genautocomplete - command to make a bash completion script for
|
|
rclone
|
|
.RE
|
|
.IP \[bu] 2
|
|
Editing a remote using \f[C]rclone config\f[R] now goes through the
|
|
wizard
|
|
.IP \[bu] 2
|
|
Compile with go 1.7 - this fixes rclone on macOS Sierra and on 386
|
|
processors
|
|
.IP \[bu] 2
|
|
Use cobra for sub commands and docs generation
|
|
.RE
|
|
.IP \[bu] 2
|
|
drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Document how to make your own client_id
|
|
.RE
|
|
.IP \[bu] 2
|
|
s3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
User-configurable Amazon S3 ACL (thanks Radek \[vS]enfeld)
|
|
.RE
|
|
.IP \[bu] 2
|
|
b2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix stats accounting for upload - no more jumping to 100% done
|
|
.IP \[bu] 2
|
|
On cleanup delete hide marker if it is the current file
|
|
.IP \[bu] 2
|
|
New B2 API endpoint (thanks Per Cederberg)
|
|
.IP \[bu] 2
|
|
Set maximum backoff to 5 Minutes
|
|
.RE
|
|
.IP \[bu] 2
|
|
onedrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix URL escaping in file names - e.g.
|
|
uploading files with \f[C]+\f[R] in them.
|
|
.RE
|
|
.IP \[bu] 2
|
|
amazon cloud drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix token expiry during large uploads
|
|
.IP \[bu] 2
|
|
Work around 408 REQUEST_TIMEOUT and 504 GATEWAY_TIMEOUT errors
|
|
.RE
|
|
.IP \[bu] 2
|
|
local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix filenames with invalid UTF-8 not being uploaded
|
|
.IP \[bu] 2
|
|
Fix problem with some UTF-8 characters on OS X
|
|
.RE
|
|
.SS v1.32 - 2016-07-13
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix upload of files large files not in root
|
|
.RE
|
|
.SS v1.31 - 2016-07-13
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Reduce memory on sync by about 50%
|
|
.IP \[bu] 2
|
|
Implement --no-traverse flag to stop copy traversing the destination
|
|
remote.
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This can be used to reduce memory usage down to the smallest possible.
|
|
.IP \[bu] 2
|
|
Useful to copy a small number of files into a large destination folder.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Implement cleanup command for emptying trash / removing old versions of
|
|
files
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Currently B2 only
|
|
.RE
|
|
.IP \[bu] 2
|
|
Single file handling improved
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Now copied with --files-from
|
|
.IP \[bu] 2
|
|
Automatically sets --no-traverse when copying a single file
|
|
.RE
|
|
.IP \[bu] 2
|
|
Info on using installing with ansible - thanks Stefan Weichinger
|
|
.IP \[bu] 2
|
|
Implement --no-update-modtime flag to stop rclone fixing the remote
|
|
modified times.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix move command - stop it running for overlapping Fses - this was
|
|
causing data loss.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix incomplete hashes - this was causing problems for B2.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Rename Amazon Cloud Drive to Amazon Drive - no changes to config file
|
|
needed.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for non-default project domain - thanks Antonio Messina.
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add instructions on how to use rclone with minio.
|
|
.IP \[bu] 2
|
|
Add ap-northeast-2 (Seoul) and ap-south-1 (Mumbai) regions.
|
|
.IP \[bu] 2
|
|
Skip setting the modified time for objects > 5GB as it isn\[aq]t
|
|
possible.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add --b2-versions flag so old versions can be listed and retrieved.
|
|
.IP \[bu] 2
|
|
Treat 403 errors (e.g.
|
|
cap exceeded) as fatal.
|
|
.IP \[bu] 2
|
|
Implement cleanup command for deleting old file versions.
|
|
.IP \[bu] 2
|
|
Make error handling compliant with B2 integrations notes.
|
|
.IP \[bu] 2
|
|
Fix handling of token expiry.
|
|
.IP \[bu] 2
|
|
Implement --b2-test-mode to set \f[C]X-Bz-Test-Mode\f[R] header.
|
|
.IP \[bu] 2
|
|
Set cutoff for chunked upload to 200MB as per B2 guidelines.
|
|
.IP \[bu] 2
|
|
Make upload multi-threaded.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t retry 461 errors.
|
|
.RE
|
|
.SS v1.30 - 2016-06-18
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Directory listing code reworked for more features and better error
|
|
reporting (thanks to Klaus Post for help).
|
|
This enables
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Directory include filtering for efficiency
|
|
.IP \[bu] 2
|
|
--max-depth parameter
|
|
.IP \[bu] 2
|
|
Better error reporting
|
|
.IP \[bu] 2
|
|
More to come
|
|
.RE
|
|
.IP \[bu] 2
|
|
Retry more errors
|
|
.IP \[bu] 2
|
|
Add --ignore-size flag - for uploading images to onedrive
|
|
.IP \[bu] 2
|
|
Log -v output to stdout by default
|
|
.IP \[bu] 2
|
|
Display the transfer stats in more human-readable form
|
|
.IP \[bu] 2
|
|
Make 0 size files specifiable with \f[C]--max-size 0b\f[R]
|
|
.IP \[bu] 2
|
|
Add \f[C]b\f[R] suffix so we can specify bytes in --bwlimit, --min-size,
|
|
etc.
|
|
.IP \[bu] 2
|
|
Use \[dq]password:\[dq] instead of \[dq]password>\[dq] prompt - thanks
|
|
Klaus Post and Leigh Klotz
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix retry doing one too many retries
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix problems with OS X and UTF-8 characters
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Check a file exists before uploading to help with 408 Conflict errors
|
|
.IP \[bu] 2
|
|
Reauth on 401 errors - this has been causing a lot of problems
|
|
.IP \[bu] 2
|
|
Work around spurious 403 errors
|
|
.IP \[bu] 2
|
|
Restart directory listings on error
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Check a file exists before uploading to help with duplicates
|
|
.IP \[bu] 2
|
|
Fix retry of multipart uploads
|
|
.RE
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement large file uploading
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add AES256 server-side encryption for - thanks Justin R.
|
|
Wilson
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make sure we don\[aq]t use conflicting content types on upload
|
|
.IP \[bu] 2
|
|
Add service account support - thanks Michal Witkowski
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add auth version parameter
|
|
.IP \[bu] 2
|
|
Add domain option for openstack (v3 auth) - thanks Fabian Ruff
|
|
.RE
|
|
.SS v1.29 - 2016-04-18
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]-I, --ignore-times\f[R] for unconditional upload
|
|
.IP \[bu] 2
|
|
Improve \f[C]dedupe\f[R]command
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Now removes identical copies without asking
|
|
.IP \[bu] 2
|
|
Now obeys \f[C]--dry-run\f[R]
|
|
.IP \[bu] 2
|
|
Implement \f[C]--dedupe-mode\f[R] for non interactive running
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode interactive\f[R] - interactive the default.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode skip\f[R] - removes identical files then skips
|
|
anything left.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode first\f[R] - removes identical files then keeps the
|
|
first one.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode newest\f[R] - removes identical files then keeps the
|
|
newest one.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode oldest\f[R] - removes identical files then keeps the
|
|
oldest one.
|
|
.IP \[bu] 2
|
|
\f[C]--dedupe-mode rename\f[R] - removes identical files then renames
|
|
the rest to be different.
|
|
.RE
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make rclone check obey the \f[C]--size-only\f[R] flag.
|
|
.IP \[bu] 2
|
|
Use \[dq]application/octet-stream\[dq] if discovered mime type is
|
|
invalid.
|
|
.IP \[bu] 2
|
|
Fix missing \[dq]quit\[dq] option when there are no remotes.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Increase default chunk size to 8 MB - increases upload speed of big
|
|
files
|
|
.IP \[bu] 2
|
|
Speed up directory listings and make more reliable
|
|
.IP \[bu] 2
|
|
Add missing retries for Move and DirMove - increases reliability
|
|
.IP \[bu] 2
|
|
Preserve mime type on file update
|
|
.RE
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Enable mod time syncing
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This means that B2 will now check modification times
|
|
.IP \[bu] 2
|
|
It will upload new files to update the modification times
|
|
.IP \[bu] 2
|
|
(there isn\[aq]t an API to just set the mod time.)
|
|
.IP \[bu] 2
|
|
If you want the old behaviour use \f[C]--size-only\f[R].
|
|
.RE
|
|
.IP \[bu] 2
|
|
Update API to new version
|
|
.IP \[bu] 2
|
|
Fix parsing of mod time when not in metadata
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift/Hubic
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t return an MD5SUM for static large objects
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix uploading files bigger than 50GB
|
|
.RE
|
|
.SS v1.28 - 2016-03-01
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Configuration file encryption - thanks Klaus Post
|
|
.IP \[bu] 2
|
|
Improve \f[C]rclone config\f[R] adding more help and making it easier to
|
|
understand
|
|
.IP \[bu] 2
|
|
Implement \f[C]-u\f[R]/\f[C]--update\f[R] so creation times can be used
|
|
on all remotes
|
|
.IP \[bu] 2
|
|
Implement \f[C]--low-level-retries\f[R] flag
|
|
.IP \[bu] 2
|
|
Optionally disable gzip compression on downloads with
|
|
\f[C]--no-gzip-encoding\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t make directories if \f[C]--dry-run\f[R] set
|
|
.IP \[bu] 2
|
|
Fix and document the \f[C]move\f[R] command
|
|
.IP \[bu] 2
|
|
Fix redirecting stderr on unix-like OSes when using \f[C]--log-file\f[R]
|
|
.IP \[bu] 2
|
|
Fix \f[C]delete\f[R] command to wait until all finished - fixes missing
|
|
deletes.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use one upload URL per go routine fixes
|
|
\f[C]more than one upload using auth token\f[R]
|
|
.IP \[bu] 2
|
|
Add pacing, retries and reauthentication - fixes token expiry problems
|
|
.IP \[bu] 2
|
|
Upload without using a temporary file from local (and remotes which
|
|
support SHA1)
|
|
.IP \[bu] 2
|
|
Fix reading metadata for all files when it shouldn\[aq]t have been
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix listing drive documents at root
|
|
.IP \[bu] 2
|
|
Disable copy and move for Google docs
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix uploading of chunked files with non ASCII characters
|
|
.IP \[bu] 2
|
|
Allow setting of \f[C]storage_url\f[R] in the config - thanks Xavier
|
|
Lucas
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow IAM role and credentials from environment variables - thanks Brian
|
|
Stengaard
|
|
.IP \[bu] 2
|
|
Allow low privilege users to use S3 (check if directory exists during
|
|
Mkdir) - thanks Jakub Gedeon
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Retry on more things to make directory listings more reliable
|
|
.RE
|
|
.SS v1.27 - 2016-01-31
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Easier headless configuration with \f[C]rclone authorize\f[R]
|
|
.IP \[bu] 2
|
|
Add support for multiple hash types - we now check SHA1 as well as MD5
|
|
hashes.
|
|
.IP \[bu] 2
|
|
\f[C]delete\f[R] command which does obey the filters (unlike
|
|
\f[C]purge\f[R])
|
|
.IP \[bu] 2
|
|
\f[C]dedupe\f[R] command to deduplicate a remote.
|
|
Useful with Google Drive.
|
|
.IP \[bu] 2
|
|
Add \f[C]--ignore-existing\f[R] flag to skip all files that exist on
|
|
destination.
|
|
.IP \[bu] 2
|
|
Add \f[C]--delete-before\f[R], \f[C]--delete-during\f[R],
|
|
\f[C]--delete-after\f[R] flags.
|
|
.IP \[bu] 2
|
|
Add \f[C]--memprofile\f[R] flag to debug memory use.
|
|
.IP \[bu] 2
|
|
Warn the user about files with same name but different case
|
|
.IP \[bu] 2
|
|
Make \f[C]--include\f[R] rules add their implicit exclude * at the end
|
|
of the filter list
|
|
.IP \[bu] 2
|
|
Deprecate compiling with go1.3
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix download of files > 10 GB
|
|
.IP \[bu] 2
|
|
Fix directory traversal (\[dq]Next token is expired\[dq]) for large
|
|
directory listings
|
|
.IP \[bu] 2
|
|
Remove 409 conflict from error codes we will retry - stops very long
|
|
pauses
|
|
.RE
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
SHA1 hashes now checked by rclone core
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-auth-owner-only\f[R] to only consider files owned by
|
|
the user - thanks Bj\[:o]rn Harrtell
|
|
.IP \[bu] 2
|
|
Export Google documents
|
|
.RE
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make file exclusion error controllable with -q
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix upload from unprivileged user.
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix updating of mod times of files with \f[C]+\f[R] in.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add local file system option to disable UNC on Windows.
|
|
.RE
|
|
.SS v1.26 - 2016-01-02
|
|
.IP \[bu] 2
|
|
New Features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Yandex storage backend - thank you Dmitry Burdeev (\[dq]dibu\[dq])
|
|
.IP \[bu] 2
|
|
Implement Backblaze B2 storage backend
|
|
.IP \[bu] 2
|
|
Add --min-age and --max-age flags - thank you Adriano Aur\['e]lio
|
|
Meirelles
|
|
.IP \[bu] 2
|
|
Make ls/lsl/md5sum/size/check obey includes and excludes
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix crash in http logging
|
|
.IP \[bu] 2
|
|
Upload releases to github too
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix sync for chunked files
|
|
.RE
|
|
.IP \[bu] 2
|
|
OneDrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Re-enable server-side copy
|
|
.IP \[bu] 2
|
|
Don\[aq]t mask HTTP error codes with JSON decode error
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix corrupting Content-Type on mod time update (thanks Joseph Spurrier)
|
|
.RE
|
|
.SS v1.25 - 2015-11-14
|
|
.IP \[bu] 2
|
|
New features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement Hubic storage system
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix deletion of some excluded files without --delete-excluded
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This could have deleted files unexpectedly on sync
|
|
.IP \[bu] 2
|
|
Always check first with \f[C]--dry-run\f[R]!
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Stop SetModTime losing metadata (e.g.
|
|
X-Object-Manifest)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This could have caused data loss for files > 5GB in size
|
|
.RE
|
|
.IP \[bu] 2
|
|
Use ContentType from Object to avoid lookups in listings
|
|
.RE
|
|
.IP \[bu] 2
|
|
OneDrive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
disable server-side copy as it seems to be broken at Microsoft
|
|
.RE
|
|
.SS v1.24 - 2015-11-07
|
|
.IP \[bu] 2
|
|
New features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add support for Microsoft OneDrive
|
|
.IP \[bu] 2
|
|
Add \f[C]--no-check-certificate\f[R] option to disable server
|
|
certificate verification
|
|
.IP \[bu] 2
|
|
Add async readahead buffer for faster transfer of big files
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow spaces in remotes and check remote names for validity at creation
|
|
time
|
|
.IP \[bu] 2
|
|
Allow \[aq]&\[aq] and disallow \[aq]:\[aq] in Windows filenames.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ignore directory marker objects where appropriate - allows working with
|
|
Hubic
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete the container if fs wasn\[aq]t at root
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete the bucket if fs wasn\[aq]t at root
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete the bucket if fs wasn\[aq]t at root
|
|
.RE
|
|
.SS v1.23 - 2015-10-03
|
|
.IP \[bu] 2
|
|
New features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement \f[C]rclone size\f[R] for measuring remotes
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix headless config for drive and gcs
|
|
.IP \[bu] 2
|
|
Tell the user they should try again if the webserver method failed
|
|
.IP \[bu] 2
|
|
Improve output of \f[C]--dump-headers\f[R]
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Allow anonymous access to public buckets
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Stop chunked operations logging \[dq]Failed to read info: Object Not
|
|
Found\[dq]
|
|
.IP \[bu] 2
|
|
Use Content-Length on uploads for extra reliability
|
|
.RE
|
|
.SS v1.22 - 2015-09-28
|
|
.IP \[bu] 2
|
|
Implement rsync like include and exclude flags
|
|
.IP \[bu] 2
|
|
swift
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Support files > 5GB - thanks Sergey Tolmachev
|
|
.RE
|
|
.SS v1.21 - 2015-09-22
|
|
.IP \[bu] 2
|
|
New features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Display individual transfer progress
|
|
.IP \[bu] 2
|
|
Make lsl output times in localtime
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix allowing user to override credentials again in Drive, GCS and ACD
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement compliant pacing scheme
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make directory reads concurrent for increased speed.
|
|
.RE
|
|
.SS v1.20 - 2015-09-15
|
|
.IP \[bu] 2
|
|
New features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Amazon Drive support
|
|
.IP \[bu] 2
|
|
Oauth support redone - fix many bugs and improve usability
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use \[dq]golang.org/x/oauth2\[dq] as oauth library of choice
|
|
.IP \[bu] 2
|
|
Improve oauth usability for smoother initial signup
|
|
.IP \[bu] 2
|
|
drive, googlecloudstorage: optionally use auto config for the oauth
|
|
token
|
|
.RE
|
|
.IP \[bu] 2
|
|
Implement --dump-headers and --dump-bodies debug flags
|
|
.IP \[bu] 2
|
|
Show multiple matched commands if abbreviation too short
|
|
.IP \[bu] 2
|
|
Implement server-side move where possible
|
|
.RE
|
|
.IP \[bu] 2
|
|
local
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Always use UNC paths internally on Windows - fixes a lot of bugs
|
|
.RE
|
|
.IP \[bu] 2
|
|
dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
force use of our custom transport which makes timeouts work
|
|
.RE
|
|
.IP \[bu] 2
|
|
Thanks to Klaus Post for lots of help with this release
|
|
.SS v1.19 - 2015-08-28
|
|
.IP \[bu] 2
|
|
New features
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Server side copies for s3/swift/drive/dropbox/gcs
|
|
.IP \[bu] 2
|
|
Move command - uses server-side copies if it can
|
|
.IP \[bu] 2
|
|
Implement --retries flag - tries 3 times by default
|
|
.IP \[bu] 2
|
|
Build for plan9/amd64 and solaris/amd64 too
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make a current version download with a fixed URL for scripting
|
|
.IP \[bu] 2
|
|
Ignore rmdir in limited fs rather than throwing error
|
|
.RE
|
|
.IP \[bu] 2
|
|
dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Increase chunk size to improve upload speeds massively
|
|
.IP \[bu] 2
|
|
Issue an error message when trying to upload bad file name
|
|
.RE
|
|
.SS v1.18 - 2015-08-17
|
|
.IP \[bu] 2
|
|
drive
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]--drive-use-trash\f[R] flag so rclone trashes instead of
|
|
deletes
|
|
.IP \[bu] 2
|
|
Add \[dq]Forbidden to download\[dq] message for files with no
|
|
downloadURL
|
|
.RE
|
|
.IP \[bu] 2
|
|
dropbox
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Remove datastore
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This was deprecated and it caused a lot of problems
|
|
.IP \[bu] 2
|
|
Modification times and MD5SUMs no longer stored
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix uploading files > 2GB
|
|
.RE
|
|
.IP \[bu] 2
|
|
s3
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
use official AWS SDK from github.com/aws/aws-sdk-go
|
|
.IP \[bu] 2
|
|
\f[B]NB\f[R] will most likely require you to delete and recreate remote
|
|
.IP \[bu] 2
|
|
enable multipart upload which enables files > 5GB
|
|
.IP \[bu] 2
|
|
tested with Ceph / RadosGW / S3 emulation
|
|
.IP \[bu] 2
|
|
many thanks to Sam Liston and Brian Haymore at the Utah Center for High
|
|
Performance Computing (https://www.chpc.utah.edu/) for a Ceph test
|
|
account
|
|
.RE
|
|
.IP \[bu] 2
|
|
misc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Show errors when reading the config file
|
|
.IP \[bu] 2
|
|
Do not print stats in quiet mode - thanks Leonid Shalupov
|
|
.IP \[bu] 2
|
|
Add FAQ
|
|
.IP \[bu] 2
|
|
Fix created directories not obeying umask
|
|
.IP \[bu] 2
|
|
Linux installation instructions - thanks Shimon Doodkin
|
|
.RE
|
|
.SS v1.17 - 2015-06-14
|
|
.IP \[bu] 2
|
|
dropbox: fix case insensitivity issues - thanks Leonid Shalupov
|
|
.SS v1.16 - 2015-06-09
|
|
.IP \[bu] 2
|
|
Fix uploading big files which was causing timeouts or panics
|
|
.IP \[bu] 2
|
|
Don\[aq]t check md5sum after download with --size-only
|
|
.SS v1.15 - 2015-06-06
|
|
.IP \[bu] 2
|
|
Add --checksum flag to only discard transfers by MD5SUM - thanks Alex
|
|
Couper
|
|
.IP \[bu] 2
|
|
Implement --size-only flag to sync on size not checksum & modtime
|
|
.IP \[bu] 2
|
|
Expand docs and remove duplicated information
|
|
.IP \[bu] 2
|
|
Document rclone\[aq]s limitations with directories
|
|
.IP \[bu] 2
|
|
dropbox: update docs about case insensitivity
|
|
.SS v1.14 - 2015-05-21
|
|
.IP \[bu] 2
|
|
local: fix encoding of non utf-8 file names - fixes a duplicate file
|
|
problem
|
|
.IP \[bu] 2
|
|
drive: docs about rate limiting
|
|
.IP \[bu] 2
|
|
google cloud storage: Fix compile after API change in
|
|
\[dq]google.golang.org/api/storage/v1\[dq]
|
|
.SS v1.13 - 2015-05-10
|
|
.IP \[bu] 2
|
|
Revise documentation (especially sync)
|
|
.IP \[bu] 2
|
|
Implement --timeout and --conntimeout
|
|
.IP \[bu] 2
|
|
s3: ignore etags from multipart uploads which aren\[aq]t md5sums
|
|
.SS v1.12 - 2015-03-15
|
|
.IP \[bu] 2
|
|
drive: Use chunked upload for files above a certain size
|
|
.IP \[bu] 2
|
|
drive: add --drive-chunk-size and --drive-upload-cutoff parameters
|
|
.IP \[bu] 2
|
|
drive: switch to insert from update when a failed copy deletes the
|
|
upload
|
|
.IP \[bu] 2
|
|
core: Log duplicate files if they are detected
|
|
.SS v1.11 - 2015-03-04
|
|
.IP \[bu] 2
|
|
swift: add region parameter
|
|
.IP \[bu] 2
|
|
drive: fix crash on failed to update remote mtime
|
|
.IP \[bu] 2
|
|
In remote paths, change native directory separators to /
|
|
.IP \[bu] 2
|
|
Add synchronization to ls/lsl/lsd output to stop corruptions
|
|
.IP \[bu] 2
|
|
Ensure all stats/log messages to go stderr
|
|
.IP \[bu] 2
|
|
Add --log-file flag to log everything (including panics) to file
|
|
.IP \[bu] 2
|
|
Make it possible to disable stats printing with --stats=0
|
|
.IP \[bu] 2
|
|
Implement --bwlimit to limit data transfer bandwidth
|
|
.SS v1.10 - 2015-02-12
|
|
.IP \[bu] 2
|
|
s3: list an unlimited number of items
|
|
.IP \[bu] 2
|
|
Fix getting stuck in the configurator
|
|
.SS v1.09 - 2015-02-07
|
|
.IP \[bu] 2
|
|
windows: Stop drive letters (e.g.
|
|
C:) getting mixed up with remotes (e.g.
|
|
drive:)
|
|
.IP \[bu] 2
|
|
local: Fix directory separators on Windows
|
|
.IP \[bu] 2
|
|
drive: fix rate limit exceeded errors
|
|
.SS v1.08 - 2015-02-04
|
|
.IP \[bu] 2
|
|
drive: fix subdirectory listing to not list entire drive
|
|
.IP \[bu] 2
|
|
drive: Fix SetModTime
|
|
.IP \[bu] 2
|
|
dropbox: adapt code to recent library changes
|
|
.SS v1.07 - 2014-12-23
|
|
.IP \[bu] 2
|
|
google cloud storage: fix memory leak
|
|
.SS v1.06 - 2014-12-12
|
|
.IP \[bu] 2
|
|
Fix \[dq]Couldn\[aq]t find home directory\[dq] on OSX
|
|
.IP \[bu] 2
|
|
swift: Add tenant parameter
|
|
.IP \[bu] 2
|
|
Use new location of Google API packages
|
|
.SS v1.05 - 2014-08-09
|
|
.IP \[bu] 2
|
|
Improved tests and consequently lots of minor fixes
|
|
.IP \[bu] 2
|
|
core: Fix race detected by go race detector
|
|
.IP \[bu] 2
|
|
core: Fixes after running errcheck
|
|
.IP \[bu] 2
|
|
drive: reset root directory on Rmdir and Purge
|
|
.IP \[bu] 2
|
|
fs: Document that Purger returns error on empty directory, test and fix
|
|
.IP \[bu] 2
|
|
google cloud storage: fix ListDir on subdirectory
|
|
.IP \[bu] 2
|
|
google cloud storage: re-read metadata in SetModTime
|
|
.IP \[bu] 2
|
|
s3: make reading metadata more reliable to work around eventual
|
|
consistency problems
|
|
.IP \[bu] 2
|
|
s3: strip trailing / from ListDir()
|
|
.IP \[bu] 2
|
|
swift: return directories without / in ListDir
|
|
.SS v1.04 - 2014-07-21
|
|
.IP \[bu] 2
|
|
google cloud storage: Fix crash on Update
|
|
.SS v1.03 - 2014-07-20
|
|
.IP \[bu] 2
|
|
swift, s3, dropbox: fix updated files being marked as corrupted
|
|
.IP \[bu] 2
|
|
Make compile with go 1.1 again
|
|
.SS v1.02 - 2014-07-19
|
|
.IP \[bu] 2
|
|
Implement Dropbox remote
|
|
.IP \[bu] 2
|
|
Implement Google Cloud Storage remote
|
|
.IP \[bu] 2
|
|
Verify Md5sums and Sizes after copies
|
|
.IP \[bu] 2
|
|
Remove times from \[dq]ls\[dq] command - lists sizes only
|
|
.IP \[bu] 2
|
|
Add add \[dq]lsl\[dq] - lists times and sizes
|
|
.IP \[bu] 2
|
|
Add \[dq]md5sum\[dq] command
|
|
.SS v1.01 - 2014-07-04
|
|
.IP \[bu] 2
|
|
drive: fix transfer of big files using up lots of memory
|
|
.SS v1.00 - 2014-07-03
|
|
.IP \[bu] 2
|
|
drive: fix whole second dates
|
|
.SS v0.99 - 2014-06-26
|
|
.IP \[bu] 2
|
|
Fix --dry-run not working
|
|
.IP \[bu] 2
|
|
Make compatible with go 1.1
|
|
.SS v0.98 - 2014-05-30
|
|
.IP \[bu] 2
|
|
s3: Treat missing Content-Length as 0 for some ceph installations
|
|
.IP \[bu] 2
|
|
rclonetest: add file with a space in
|
|
.SS v0.97 - 2014-05-05
|
|
.IP \[bu] 2
|
|
Implement copying of single files
|
|
.IP \[bu] 2
|
|
s3 & swift: support paths inside containers/buckets
|
|
.SS v0.96 - 2014-04-24
|
|
.IP \[bu] 2
|
|
drive: Fix multiple files of same name being created
|
|
.IP \[bu] 2
|
|
drive: Use o.Update and fs.Put to optimise transfers
|
|
.IP \[bu] 2
|
|
Add version number, -V and --version
|
|
.SS v0.95 - 2014-03-28
|
|
.IP \[bu] 2
|
|
rclone.org: website, docs and graphics
|
|
.IP \[bu] 2
|
|
drive: fix path parsing
|
|
.SS v0.94 - 2014-03-27
|
|
.IP \[bu] 2
|
|
Change remote format one last time
|
|
.IP \[bu] 2
|
|
GNU style flags
|
|
.SS v0.93 - 2014-03-16
|
|
.IP \[bu] 2
|
|
drive: store token in config file
|
|
.IP \[bu] 2
|
|
cross compile other versions
|
|
.IP \[bu] 2
|
|
set strict permissions on config file
|
|
.SS v0.92 - 2014-03-15
|
|
.IP \[bu] 2
|
|
Config fixes and --config option
|
|
.SS v0.91 - 2014-03-15
|
|
.IP \[bu] 2
|
|
Make config file
|
|
.SS v0.90 - 2013-06-27
|
|
.IP \[bu] 2
|
|
Project named rclone
|
|
.SS v0.00 - 2012-11-18
|
|
.IP \[bu] 2
|
|
Project started
|
|
.SH Bugs and Limitations
|
|
.SS Limitations
|
|
.SS Directory timestamps aren\[aq]t preserved
|
|
.PP
|
|
Rclone doesn\[aq]t currently preserve the timestamps of directories.
|
|
This is because rclone only really considers objects when syncing.
|
|
.SS Rclone struggles with millions of files in a directory/bucket
|
|
.PP
|
|
Currently rclone loads each directory/bucket entirely into memory before
|
|
using it.
|
|
Since each rclone object takes 0.5k-1k of memory this can take a very
|
|
long time and use a large amount of memory.
|
|
.PP
|
|
Millions of files in a directory tends to occur on bucket-based remotes
|
|
(e.g.
|
|
S3 buckets) since those remotes do not segregate subdirectories within
|
|
the bucket.
|
|
.SS Bucket-based remotes and folders
|
|
.PP
|
|
Bucket-based remotes (e.g.
|
|
S3/GCS/Swift/B2) do not have a concept of directories.
|
|
Rclone therefore cannot create directories in them which means that
|
|
empty directories on a bucket-based remote will tend to disappear.
|
|
.PP
|
|
Some software creates empty keys ending in \f[C]/\f[R] as directory
|
|
markers.
|
|
Rclone doesn\[aq]t do this as it potentially creates more objects and
|
|
costs more.
|
|
This ability may be added in the future (probably via a flag/option).
|
|
.SS Bugs
|
|
.PP
|
|
Bugs are stored in rclone\[aq]s GitHub project:
|
|
.IP \[bu] 2
|
|
Reported
|
|
bugs (https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug)
|
|
.IP \[bu] 2
|
|
Known
|
|
issues (https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Known+Problem%22)
|
|
.SH Frequently Asked Questions
|
|
.SS Do all cloud storage systems support all rclone commands
|
|
.PP
|
|
Yes they do.
|
|
All the rclone commands (e.g.
|
|
\f[C]sync\f[R], \f[C]copy\f[R], etc.) will work on all the remote
|
|
storage systems.
|
|
.SS Can I copy the config from one machine to another
|
|
.PP
|
|
Sure! Rclone stores all of its config in a single file.
|
|
If you want to find this file, run \f[C]rclone config file\f[R] which
|
|
will tell you where it is.
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for more
|
|
info.
|
|
.SS How do I configure rclone on a remote / headless box with no browser?
|
|
.PP
|
|
This has now been documented in its own remote setup
|
|
page (https://rclone.org/remote_setup/).
|
|
.SS Can rclone sync directly from drive to s3
|
|
.PP
|
|
Rclone can sync between two remote cloud storage systems just fine.
|
|
.PP
|
|
Note that it effectively downloads the file and uploads it again, so the
|
|
node running rclone would need to have lots of bandwidth.
|
|
.PP
|
|
The syncs would be incremental (on a file by file basis).
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone sync -i drive:Folder s3:bucket
|
|
\f[R]
|
|
.fi
|
|
.SS Using rclone from multiple locations at the same time
|
|
.PP
|
|
You can use rclone from multiple places at the same time if you choose
|
|
different subdirectory for the output, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Server A> rclone sync -i /tmp/whatever remote:ServerA
|
|
Server B> rclone sync -i /tmp/whatever remote:ServerB
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you sync to the same directory then you should use rclone copy
|
|
otherwise the two instances of rclone may delete each other\[aq]s files,
|
|
e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Server A> rclone copy /tmp/whatever remote:Backup
|
|
Server B> rclone copy /tmp/whatever remote:Backup
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The file names you upload from Server A and Server B should be different
|
|
in this case, otherwise some file systems (e.g.
|
|
Drive) may make duplicates.
|
|
.SS Why doesn\[aq]t rclone support partial transfers / binary diffs like rsync?
|
|
.PP
|
|
Rclone stores each file you transfer as a native object on the remote
|
|
cloud storage system.
|
|
This means that you can see the files you upload as expected using
|
|
alternative access methods (e.g.
|
|
using the Google Drive web interface).
|
|
There is a 1:1 mapping between files on your hard disk and objects
|
|
created in the cloud storage system.
|
|
.PP
|
|
Cloud storage systems (at least none I\[aq]ve come across yet) don\[aq]t
|
|
support partially uploading an object.
|
|
You can\[aq]t take an existing object, and change some bytes in the
|
|
middle of it.
|
|
.PP
|
|
It would be possible to make a sync system which stored binary diffs
|
|
instead of whole objects like rclone does, but that would break the 1:1
|
|
mapping of files on your hard disk to objects in the remote cloud
|
|
storage system.
|
|
.PP
|
|
All the cloud storage systems support partial downloads of content, so
|
|
it would be possible to make partial downloads work.
|
|
However to make this work efficiently this would require storing a
|
|
significant amount of metadata, which breaks the desired 1:1 mapping of
|
|
files to objects.
|
|
.SS Can rclone do bi-directional sync?
|
|
.PP
|
|
No, not at present.
|
|
rclone only does uni-directional sync from A -> B.
|
|
It may do in the future though since it has all the primitives - it just
|
|
requires writing the algorithm to do it.
|
|
.SS Can I use rclone with an HTTP proxy?
|
|
.PP
|
|
Yes.
|
|
rclone will follow the standard environment variables for proxies,
|
|
similar to cURL and other programs.
|
|
.PP
|
|
In general the variables are called \f[C]http_proxy\f[R] (for services
|
|
reached over \f[C]http\f[R]) and \f[C]https_proxy\f[R] (for services
|
|
reached over \f[C]https\f[R]).
|
|
Most public services will be using \f[C]https\f[R], but you may wish to
|
|
set both.
|
|
.PP
|
|
The content of the variable is \f[C]protocol://server:port\f[R].
|
|
The protocol value is the one used to talk to the proxy server, itself,
|
|
and is commonly either \f[C]http\f[R] or \f[C]socks5\f[R].
|
|
.PP
|
|
Slightly annoyingly, there is no \f[I]standard\f[R] for the name; some
|
|
applications may use \f[C]http_proxy\f[R] but another one
|
|
\f[C]HTTP_PROXY\f[R].
|
|
The \f[C]Go\f[R] libraries used by \f[C]rclone\f[R] will try both
|
|
variations, but you may wish to set all possibilities.
|
|
So, on Linux, you may end up with code similar to
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
export http_proxy=http://proxyserver:12345
|
|
export https_proxy=$http_proxy
|
|
export HTTP_PROXY=$http_proxy
|
|
export HTTPS_PROXY=$http_proxy
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The \f[C]NO_PROXY\f[R] allows you to disable the proxy for specific
|
|
hosts.
|
|
Hosts must be comma separated, and can contain domains or parts.
|
|
For instance \[dq]foo.com\[dq] also matches \[dq]bar.foo.com\[dq].
|
|
.PP
|
|
e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
export no_proxy=localhost,127.0.0.0/8,my.host.name
|
|
export NO_PROXY=$no_proxy
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
Note that the ftp backend does not support \f[C]ftp_proxy\f[R] yet.
|
|
.SS Rclone gives x509: failed to load system roots and no roots provided error
|
|
.PP
|
|
This means that \f[C]rclone\f[R] can\[aq]t find the SSL root
|
|
certificates.
|
|
Likely you are running \f[C]rclone\f[R] on a NAS with a cut-down Linux
|
|
OS, or possibly on Solaris.
|
|
.PP
|
|
Rclone (via the Go runtime) tries to load the root certificates from
|
|
these places on Linux.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\[dq]/etc/ssl/certs/ca-certificates.crt\[dq], // Debian/Ubuntu/Gentoo etc.
|
|
\[dq]/etc/pki/tls/certs/ca-bundle.crt\[dq], // Fedora/RHEL
|
|
\[dq]/etc/ssl/ca-bundle.pem\[dq], // OpenSUSE
|
|
\[dq]/etc/pki/tls/cacert.pem\[dq], // OpenELEC
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
So doing something like this should fix the problem.
|
|
It also sets the time which is important for SSL to work properly.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
mkdir -p /etc/ssl/certs/
|
|
curl -o /etc/ssl/certs/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt
|
|
ntpclient -s -h pool.ntp.org
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
The two environment variables \f[C]SSL_CERT_FILE\f[R] and
|
|
\f[C]SSL_CERT_DIR\f[R], mentioned in the x509
|
|
package (https://godoc.org/crypto/x509), provide an additional way to
|
|
provide the SSL root certificates.
|
|
.PP
|
|
Note that you may need to add the \f[C]--insecure\f[R] option to the
|
|
\f[C]curl\f[R] command line if it doesn\[aq]t work without.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl --insecure -o /etc/ssl/certs/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt
|
|
\f[R]
|
|
.fi
|
|
.SS Rclone gives Failed to load config file: function not implemented error
|
|
.PP
|
|
Likely this means that you are running rclone on Linux version not
|
|
supported by the go runtime, ie earlier than version 2.6.23.
|
|
.PP
|
|
See the system requirements section in the go install
|
|
docs (https://golang.org/doc/install) for full details.
|
|
.SS All my uploaded docx/xlsx/pptx files appear as archive/zip
|
|
.PP
|
|
This is caused by uploading these files from a Windows computer which
|
|
hasn\[aq]t got the Microsoft Office suite installed.
|
|
The easiest way to fix is to install the Word viewer and the Microsoft
|
|
Office Compatibility Pack for Word, Excel, and PowerPoint 2007 and later
|
|
versions\[aq] file formats
|
|
.SS tcp lookup some.domain.com no such host
|
|
.PP
|
|
This happens when rclone cannot resolve a domain.
|
|
Please check that your DNS setup is generally working, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
# both should print a long list of possible IP addresses
|
|
dig www.googleapis.com # resolve using your default DNS
|
|
dig www.googleapis.com \[at]8.8.8.8 # resolve with Google\[aq]s DNS server
|
|
\f[R]
|
|
.fi
|
|
.PP
|
|
If you are using \f[C]systemd-resolved\f[R] (default on Arch Linux),
|
|
ensure it is at version 233 or higher.
|
|
Previous releases contain a bug which causes not all domains to be
|
|
resolved properly.
|
|
.PP
|
|
Additionally with the \f[C]GODEBUG=netdns=\f[R] environment variable the
|
|
Go resolver decision can be influenced.
|
|
This also allows to resolve certain issues with DNS resolution.
|
|
See the name resolution section in the go
|
|
docs (https://golang.org/pkg/net/#hdr-Name_Resolution).
|
|
.SS The total size reported in the stats for a sync is wrong and keeps changing
|
|
.PP
|
|
It is likely you have more than 10,000 files that need to be synced.
|
|
By default, rclone only gets 10,000 files ahead in a sync so as not to
|
|
use up too much memory.
|
|
You can change this default with the
|
|
--max-backlog (https://rclone.org/docs/#max-backlog-n) flag.
|
|
.SS Rclone is using too much memory or appears to have a memory leak
|
|
.PP
|
|
Rclone is written in Go which uses a garbage collector.
|
|
The default settings for the garbage collector mean that it runs when
|
|
the heap size has doubled.
|
|
.PP
|
|
However it is possible to tune the garbage collector to use less memory
|
|
by setting GOGC (https://dave.cheney.net/tag/gogc) to a lower value, say
|
|
\f[C]export GOGC=20\f[R].
|
|
This will make the garbage collector work harder, reducing memory size
|
|
at the expense of CPU usage.
|
|
.PP
|
|
The most common cause of rclone using lots of memory is a single
|
|
directory with thousands or millions of files in.
|
|
Rclone has to load this entirely into memory as rclone objects.
|
|
Each rclone object takes 0.5k-1k of memory.
|
|
.SS Rclone changes fullwidth Unicode punctuation marks in file names
|
|
.PP
|
|
For example: On a Windows system, you have a file with name
|
|
\f[C]Test\[uFF1A]1.jpg\f[R], where \f[C]\[uFF1A]\f[R] is the Unicode
|
|
fullwidth colon symbol.
|
|
When using rclone to copy this to your Google Drive, you will notice
|
|
that the file gets renamed to \f[C]Test:1.jpg\f[R], where \f[C]:\f[R] is
|
|
the regular (halfwidth) colon.
|
|
.PP
|
|
The reason for such renames is the way rclone handles different
|
|
restricted filenames (https://rclone.org/overview/#restricted-filenames)
|
|
on different cloud storage systems.
|
|
It tries to avoid ambiguous file names as much and allow moving files
|
|
between many cloud storage systems transparently, by replacing invalid
|
|
characters with similar looking Unicode characters when transferring to
|
|
one storage system, and replacing back again when transferring to a
|
|
different storage system where the original characters are supported.
|
|
When the same Unicode characters are intentionally used in file names,
|
|
this replacement strategy leads to unwanted renames.
|
|
Read more
|
|
here (https://rclone.org/overview/#restricted-filenames-caveats).
|
|
.SH License
|
|
.PP
|
|
This is free software under the terms of MIT the license (check the
|
|
COPYING file included with the source code).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Copyright (C) 2019 by Nick Craig-Wood https://www.craig-wood.com/nick/
|
|
|
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
of this software and associated documentation files (the \[dq]Software\[dq]), to deal
|
|
in the Software without restriction, including without limitation the rights
|
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
copies of the Software, and to permit persons to whom the Software is
|
|
furnished to do so, subject to the following conditions:
|
|
|
|
The above copyright notice and this permission notice shall be included in
|
|
all copies or substantial portions of the Software.
|
|
|
|
THE SOFTWARE IS PROVIDED \[dq]AS IS\[dq], WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
THE SOFTWARE.
|
|
\f[R]
|
|
.fi
|
|
.SH Authors and contributors
|
|
.SS Authors
|
|
.IP \[bu] 2
|
|
Nick Craig-Wood <nick@craig-wood.com>
|
|
.SS Contributors
|
|
.PP
|
|
{{< rem
|
|
\f[C]email addresses removed from here need to be addeed to bin/.ignore-emails to make sure update-authors.py doesn\[aq]t immediately put them back in again.\f[R]
|
|
>}}
|
|
.IP \[bu] 2
|
|
Alex Couper <amcouper@gmail.com>
|
|
.IP \[bu] 2
|
|
Leonid Shalupov <leonid@shalupov.com> <shalupov@diverse.org.ru>
|
|
.IP \[bu] 2
|
|
Shimon Doodkin <helpmepro1@gmail.com>
|
|
.IP \[bu] 2
|
|
Colin Nicholson <colin@colinn.com>
|
|
.IP \[bu] 2
|
|
Klaus Post <klauspost@gmail.com>
|
|
.IP \[bu] 2
|
|
Sergey Tolmachev <tolsi.ru@gmail.com>
|
|
.IP \[bu] 2
|
|
Adriano Aur\['e]lio Meirelles <adriano@atinge.com>
|
|
.IP \[bu] 2
|
|
C.
|
|
Bess <cbess@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Dmitry Burdeev <dibu28@gmail.com>
|
|
.IP \[bu] 2
|
|
Joseph Spurrier <github@josephspurrier.com>
|
|
.IP \[bu] 2
|
|
Bj\[:o]rn Harrtell <bjorn@wololo.org>
|
|
.IP \[bu] 2
|
|
Xavier Lucas <xavier.lucas@corp.ovh.com>
|
|
.IP \[bu] 2
|
|
Werner Beroux <werner@beroux.com>
|
|
.IP \[bu] 2
|
|
Brian Stengaard <brian@stengaard.eu>
|
|
.IP \[bu] 2
|
|
Jakub Gedeon <jgedeon@sofi.com>
|
|
.IP \[bu] 2
|
|
Jim Tittsler <jwt@onjapan.net>
|
|
.IP \[bu] 2
|
|
Michal Witkowski <michal@improbable.io>
|
|
.IP \[bu] 2
|
|
Fabian Ruff <fabian.ruff@sap.com>
|
|
.IP \[bu] 2
|
|
Leigh Klotz <klotz@quixey.com>
|
|
.IP \[bu] 2
|
|
Romain Lapray <lapray.romain@gmail.com>
|
|
.IP \[bu] 2
|
|
Justin R.
|
|
Wilson <jrw972@gmail.com>
|
|
.IP \[bu] 2
|
|
Antonio Messina <antonio.s.messina@gmail.com>
|
|
.IP \[bu] 2
|
|
Stefan G.
|
|
Weichinger <office@oops.co.at>
|
|
.IP \[bu] 2
|
|
Per Cederberg <cederberg@gmail.com>
|
|
.IP \[bu] 2
|
|
Radek \[vS]enfeld <rush@logic.cz>
|
|
.IP \[bu] 2
|
|
Fredrik Fornwall <fredrik@fornwall.net>
|
|
.IP \[bu] 2
|
|
Asko Tamm <asko@deekit.net>
|
|
.IP \[bu] 2
|
|
xor-zz <xor@gstocco.com>
|
|
.IP \[bu] 2
|
|
Tomasz Mazur <tmazur90@gmail.com>
|
|
.IP \[bu] 2
|
|
Marco Paganini <paganini@paganini.net>
|
|
.IP \[bu] 2
|
|
Felix B\[:u]nemann <buenemann@louis.info>
|
|
.IP \[bu] 2
|
|
Durval Menezes <jmrclone@durval.com>
|
|
.IP \[bu] 2
|
|
Luiz Carlos Rumbelsperger Viana <maxd13_luiz_carlos@hotmail.com>
|
|
.IP \[bu] 2
|
|
Stefan Breunig <stefan-github@yrden.de>
|
|
.IP \[bu] 2
|
|
Alishan Ladhani <ali-l@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
0xJAKE <0xJAKE@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Thibault Molleman <thibaultmol@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Scott McGillivray <scott.mcgillivray@gmail.com>
|
|
.IP \[bu] 2
|
|
Bj\[/o]rn Erik Pedersen <bjorn.erik.pedersen@gmail.com>
|
|
.IP \[bu] 2
|
|
Lukas Loesche <lukas@mesosphere.io>
|
|
.IP \[bu] 2
|
|
emyarod <allllaboutyou@gmail.com>
|
|
.IP \[bu] 2
|
|
T.C.
|
|
Ferguson <tcf909@gmail.com>
|
|
.IP \[bu] 2
|
|
Brandur <brandur@mutelight.org>
|
|
.IP \[bu] 2
|
|
Dario Giovannetti <dev@dariogiovannetti.net>
|
|
.IP \[bu] 2
|
|
K\['a]roly Ol\['a]h <okaresz@aol.com>
|
|
.IP \[bu] 2
|
|
Jon Yergatian <jon@macfanatic.ca>
|
|
.IP \[bu] 2
|
|
Jack Schmidt <github@mowsey.org>
|
|
.IP \[bu] 2
|
|
Dedsec1 <Dedsec1@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Hisham Zarka <hzarka@gmail.com>
|
|
.IP \[bu] 2
|
|
J\['e]r\[^o]me Vizcaino <jerome.vizcaino@gmail.com>
|
|
.IP \[bu] 2
|
|
Mike Tesch <mjt6129@rit.edu>
|
|
.IP \[bu] 2
|
|
Marvin Watson <marvwatson@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Danny Tsai <danny8376@gmail.com>
|
|
.IP \[bu] 2
|
|
Yoni Jah <yonjah+git@gmail.com> <yonjah+github@gmail.com>
|
|
.IP \[bu] 2
|
|
Stephen Harris <github@spuddy.org> <sweharris@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Ihor Dvoretskyi <ihor.dvoretskyi@gmail.com>
|
|
.IP \[bu] 2
|
|
Jon Craton <jncraton@gmail.com>
|
|
.IP \[bu] 2
|
|
Hraban Luyat <hraban@0brg.net>
|
|
.IP \[bu] 2
|
|
Michael Ledin <mledin89@gmail.com>
|
|
.IP \[bu] 2
|
|
Martin Kristensen <me@azgul.com>
|
|
.IP \[bu] 2
|
|
Too Much IO <toomuchio@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Anisse Astier <anisse@astier.eu>
|
|
.IP \[bu] 2
|
|
Zahiar Ahmed <zahiar@live.com>
|
|
.IP \[bu] 2
|
|
Igor Kharin <igorkharin@gmail.com>
|
|
.IP \[bu] 2
|
|
Bill Zissimopoulos <billziss@navimatics.com>
|
|
.IP \[bu] 2
|
|
Bob Potter <bobby.potter@gmail.com>
|
|
.IP \[bu] 2
|
|
Steven Lu <tacticalazn@gmail.com>
|
|
.IP \[bu] 2
|
|
Sjur Fredriksen <sjurtf@ifi.uio.no>
|
|
.IP \[bu] 2
|
|
Ruwbin <hubus12345@gmail.com>
|
|
.IP \[bu] 2
|
|
Fabian M\[:o]ller <fabianm88@gmail.com> <f.moeller@nynex.de>
|
|
.IP \[bu] 2
|
|
Edward Q.
|
|
Bridges <github@eqbridges.com>
|
|
.IP \[bu] 2
|
|
Vasiliy Tolstov <v.tolstov@selfip.ru>
|
|
.IP \[bu] 2
|
|
Harshavardhana <harsha@minio.io>
|
|
.IP \[bu] 2
|
|
sainaen <sainaen@gmail.com>
|
|
.IP \[bu] 2
|
|
gdm85 <gdm85@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Yaroslav Halchenko <debian@onerussian.com>
|
|
.IP \[bu] 2
|
|
John Papandriopoulos <jpap@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Zhiming Wang <zmwangx@gmail.com>
|
|
.IP \[bu] 2
|
|
Andy Pilate <cubox@cubox.me>
|
|
.IP \[bu] 2
|
|
Oliver Heyme <olihey@googlemail.com> <olihey@users.noreply.github.com>
|
|
<de8olihe@lego.com>
|
|
.IP \[bu] 2
|
|
wuyu <wuyu@yunify.com>
|
|
.IP \[bu] 2
|
|
Andrei Dragomir <adragomi@adobe.com>
|
|
.IP \[bu] 2
|
|
Christian Br\[:u]ggemann <mail@cbruegg.com>
|
|
.IP \[bu] 2
|
|
Alex McGrath Kraak <amkdude@gmail.com>
|
|
.IP \[bu] 2
|
|
bpicode <bjoern.pirnay@googlemail.com>
|
|
.IP \[bu] 2
|
|
Daniel Jagszent <daniel@jagszent.de>
|
|
.IP \[bu] 2
|
|
Josiah White <thegenius2009@gmail.com>
|
|
.IP \[bu] 2
|
|
Ishuah Kariuki <kariuki@ishuah.com> <ishuah91@gmail.com>
|
|
.IP \[bu] 2
|
|
Jan Varho <jan@varho.org>
|
|
.IP \[bu] 2
|
|
Girish Ramakrishnan <girish@cloudron.io>
|
|
.IP \[bu] 2
|
|
LingMan <LingMan@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Jacob McNamee <jacobmcnamee@gmail.com>
|
|
.IP \[bu] 2
|
|
jersou <jertux@gmail.com>
|
|
.IP \[bu] 2
|
|
thierry <thierry@substantiel.fr>
|
|
.IP \[bu] 2
|
|
Simon Leinen <simon.leinen@gmail.com> <ubuntu@s3-test.novalocal>
|
|
.IP \[bu] 2
|
|
Dan Dascalescu <ddascalescu+github@gmail.com>
|
|
.IP \[bu] 2
|
|
Jason Rose <jason@jro.io>
|
|
.IP \[bu] 2
|
|
Andrew Starr-Bochicchio <a.starr.b@gmail.com>
|
|
.IP \[bu] 2
|
|
John Leach <john@johnleach.co.uk>
|
|
.IP \[bu] 2
|
|
Corban Raun <craun@instructure.com>
|
|
.IP \[bu] 2
|
|
Pierre Carlson <mpcarl@us.ibm.com>
|
|
.IP \[bu] 2
|
|
Ernest Borowski <er.borowski@gmail.com>
|
|
.IP \[bu] 2
|
|
Remus Bunduc <remus.bunduc@gmail.com>
|
|
.IP \[bu] 2
|
|
Iakov Davydov <iakov.davydov@unil.ch> <dav05.gith@myths.ru>
|
|
.IP \[bu] 2
|
|
Jakub Tasiemski <tasiemski@gmail.com>
|
|
.IP \[bu] 2
|
|
David Minor <dminor@saymedia.com>
|
|
.IP \[bu] 2
|
|
Tim Cooijmans <cooijmans.tim@gmail.com>
|
|
.IP \[bu] 2
|
|
Laurence <liuxy6@gmail.com>
|
|
.IP \[bu] 2
|
|
Giovanni Pizzi <gio.piz@gmail.com>
|
|
.IP \[bu] 2
|
|
Filip Bartodziej <filipbartodziej@gmail.com>
|
|
.IP \[bu] 2
|
|
Jon Fautley <jon@dead.li>
|
|
.IP \[bu] 2
|
|
lewapm <32110057+lewapm@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Yassine Imounachen <yassine256@gmail.com>
|
|
.IP \[bu] 2
|
|
Chris Redekop <chris-redekop@users.noreply.github.com>
|
|
<chris.redekop@gmail.com>
|
|
.IP \[bu] 2
|
|
Jon Fautley <jon@adenoid.appstal.co.uk>
|
|
.IP \[bu] 2
|
|
Will Gunn <WillGunn@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Lucas Bremgartner <lucas@bremis.ch>
|
|
.IP \[bu] 2
|
|
Jody Frankowski <jody.frankowski@gmail.com>
|
|
.IP \[bu] 2
|
|
Andreas Roussos <arouss1980@gmail.com>
|
|
.IP \[bu] 2
|
|
nbuchanan <nbuchanan@utah.gov>
|
|
.IP \[bu] 2
|
|
Durval Menezes <rclone@durval.com>
|
|
.IP \[bu] 2
|
|
Victor <vb-github@viblo.se>
|
|
.IP \[bu] 2
|
|
Mateusz <pabian.mateusz@gmail.com>
|
|
.IP \[bu] 2
|
|
Daniel Loader <spicypixel@gmail.com>
|
|
.IP \[bu] 2
|
|
David0rk <davidork@gmail.com>
|
|
.IP \[bu] 2
|
|
Alexander Neumann <alexander@bumpern.de>
|
|
.IP \[bu] 2
|
|
Giri Badanahatti <gbadanahatti@us.ibm.com@Giris-MacBook-Pro.local>
|
|
.IP \[bu] 2
|
|
Leo R.
|
|
Lundgren <leo@finalresort.org>
|
|
.IP \[bu] 2
|
|
wolfv <wolfv6@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Dave Pedu <dave@davepedu.com>
|
|
.IP \[bu] 2
|
|
Stefan Lindblom <lindblom@spotify.com>
|
|
.IP \[bu] 2
|
|
seuffert <oliver@seuffert.biz>
|
|
.IP \[bu] 2
|
|
gbadanahatti <37121690+gbadanahatti@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Keith Goldfarb <barkofdelight@gmail.com>
|
|
.IP \[bu] 2
|
|
Steve Kriss <steve@heptio.com>
|
|
.IP \[bu] 2
|
|
Chih-Hsuan Yen <yan12125@gmail.com>
|
|
.IP \[bu] 2
|
|
Alexander Neumann <fd0@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Matt Holt <mholt@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Eri Bastos <bastos.eri@gmail.com>
|
|
.IP \[bu] 2
|
|
Michael P.
|
|
Dubner <pywebmail@list.ru>
|
|
.IP \[bu] 2
|
|
Antoine GIRARD <sapk@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Mateusz Piotrowski <mpp302@gmail.com>
|
|
.IP \[bu] 2
|
|
Animosity022 <animosity22@users.noreply.github.com>
|
|
<earl.texter@gmail.com>
|
|
.IP \[bu] 2
|
|
Peter Baumgartner <pete@lincolnloop.com>
|
|
.IP \[bu] 2
|
|
Craig Rachel <craig@craigrachel.com>
|
|
.IP \[bu] 2
|
|
Michael G.
|
|
Noll <miguno@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
hensur <me@hensur.de>
|
|
.IP \[bu] 2
|
|
Oliver Heyme <de8olihe@lego.com>
|
|
.IP \[bu] 2
|
|
Richard Yang <richard@yenforyang.com>
|
|
.IP \[bu] 2
|
|
Piotr Oleszczyk <piotr.oleszczyk@gmail.com>
|
|
.IP \[bu] 2
|
|
Rodrigo <rodarima@gmail.com>
|
|
.IP \[bu] 2
|
|
NoLooseEnds <NoLooseEnds@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Jakub Karlicek <jakub@karlicek.me>
|
|
.IP \[bu] 2
|
|
John Clayton <john@codemonkeylabs.com>
|
|
.IP \[bu] 2
|
|
Kasper Byrdal Nielsen <byrdal76@gmail.com>
|
|
.IP \[bu] 2
|
|
Benjamin Joseph Dag <bjdag1234@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
themylogin <themylogin@gmail.com>
|
|
.IP \[bu] 2
|
|
Onno Zweers <onno.zweers@surfsara.nl>
|
|
.IP \[bu] 2
|
|
Jasper Lievisse Adriaanse <jasper@humppa.nl>
|
|
.IP \[bu] 2
|
|
sandeepkru <sandeep.ummadi@gmail.com>
|
|
<sandeepkru@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
HerrH <atomtigerzoo@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Andrew <4030760+sparkyman215@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
dan smith <XX1011@gmail.com>
|
|
.IP \[bu] 2
|
|
Oleg Kovalov <iamolegkovalov@gmail.com>
|
|
.IP \[bu] 2
|
|
Ruben Vandamme <github-com-00ff86@vandamme.email>
|
|
.IP \[bu] 2
|
|
Cnly <minecnly@gmail.com>
|
|
.IP \[bu] 2
|
|
Andres Alvarez <1671935+kir4h@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
reddi1 <xreddi@gmail.com>
|
|
.IP \[bu] 2
|
|
Matt Tucker <matthewtckr@gmail.com>
|
|
.IP \[bu] 2
|
|
Sebastian B\[:u]nger <buengese@gmail.com> <buengese@protonmail.com>
|
|
.IP \[bu] 2
|
|
Martin Polden <mpolden@mpolden.no>
|
|
.IP \[bu] 2
|
|
Alex Chen <Cnly@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Denis <deniskovpen@gmail.com>
|
|
.IP \[bu] 2
|
|
bsteiss <35940619+bsteiss@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
C\['e]dric Connes <cedric.connes@gmail.com>
|
|
.IP \[bu] 2
|
|
Dr.
|
|
Tobias Quathamer <toddy15@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
dcpu <42736967+dcpu@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Sheldon Rupp <me@shel.io>
|
|
.IP \[bu] 2
|
|
albertony <12441419+albertony@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
cron410 <cron410@gmail.com>
|
|
.IP \[bu] 2
|
|
Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Felix Brucker <felix@felixbrucker.com>
|
|
.IP \[bu] 2
|
|
Santiago Rodr\['i]guez <scollazo@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Craig Miskell <craig.miskell@fluxfederation.com>
|
|
.IP \[bu] 2
|
|
Antoine GIRARD <sapk@sapk.fr>
|
|
.IP \[bu] 2
|
|
Joanna Marek <joanna.marek@u2i.com>
|
|
.IP \[bu] 2
|
|
frenos <frenos@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
ssaqua <ssaqua@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
xnaas <me@xnaas.info>
|
|
.IP \[bu] 2
|
|
Frantisek Fuka <fuka@fuxoft.cz>
|
|
.IP \[bu] 2
|
|
Paul Kohout <pauljkohout@yahoo.com>
|
|
.IP \[bu] 2
|
|
dcpu <43330287+dcpu@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
jackyzy823 <jackyzy823@gmail.com>
|
|
.IP \[bu] 2
|
|
David Haguenauer <ml@kurokatta.org>
|
|
.IP \[bu] 2
|
|
teresy <hi.teresy@gmail.com>
|
|
.IP \[bu] 2
|
|
buergi <patbuergi@gmx.de>
|
|
.IP \[bu] 2
|
|
Florian Gamboeck <mail@floga.de>
|
|
.IP \[bu] 2
|
|
Ralf Hemberger <10364191+rhemberger@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Scott Edlund <sedlund@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Erik Swanson <erik@retailnext.net>
|
|
.IP \[bu] 2
|
|
Jake Coggiano <jake@stripe.com>
|
|
.IP \[bu] 2
|
|
brused27 <brused27@noemailaddress>
|
|
.IP \[bu] 2
|
|
Peter Kaminski <kaminski@istori.com>
|
|
.IP \[bu] 2
|
|
Henry Ptasinski <henry@logout.com>
|
|
.IP \[bu] 2
|
|
Alexander <kharkovalexander@gmail.com>
|
|
.IP \[bu] 2
|
|
Garry McNulty <garrmcnu@gmail.com>
|
|
.IP \[bu] 2
|
|
Mathieu Carbou <mathieu.carbou@gmail.com>
|
|
.IP \[bu] 2
|
|
Mark Otway <mark@otway.com>
|
|
.IP \[bu] 2
|
|
William Cocker <37018962+WilliamCocker@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Fran\[,c]ois Leurent <131.js@cloudyks.org>
|
|
.IP \[bu] 2
|
|
Arkadius Stefanski <arkste@gmail.com>
|
|
.IP \[bu] 2
|
|
Jay <dev@jaygoel.com>
|
|
.IP \[bu] 2
|
|
andrea rota <a@xelera.eu>
|
|
.IP \[bu] 2
|
|
nicolov <nicolov@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Dario Guzik <dario@guzik.com.ar>
|
|
.IP \[bu] 2
|
|
qip <qip@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
yair\[at]unicorn <yair@unicorn>
|
|
.IP \[bu] 2
|
|
Matt Robinson <brimstone@the.narro.ws>
|
|
.IP \[bu] 2
|
|
kayrus <kay.diam@gmail.com>
|
|
.IP \[bu] 2
|
|
R\['e]my L\['e]one <remy.leone@gmail.com>
|
|
.IP \[bu] 2
|
|
Wojciech Smigielski <wojciech.hieronim.smigielski@gmail.com>
|
|
.IP \[bu] 2
|
|
weetmuts <oehrstroem@gmail.com>
|
|
.IP \[bu] 2
|
|
Jonathan <vanillajonathan@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
James Carpenter <orbsmiv@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Vince <vince0villamora@gmail.com>
|
|
.IP \[bu] 2
|
|
Nestar47 <47841759+Nestar47@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Six <brbsix@gmail.com>
|
|
.IP \[bu] 2
|
|
Alexandru Bumbacea <alexandru.bumbacea@booking.com>
|
|
.IP \[bu] 2
|
|
calisro <robert.calistri@gmail.com>
|
|
.IP \[bu] 2
|
|
Dr.Rx <david.rey@nventive.com>
|
|
.IP \[bu] 2
|
|
marcintustin <marcintustin@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
jaKa Mo\[u010D]nik <jaka@koofr.net>
|
|
.IP \[bu] 2
|
|
Fionera <fionera@fionera.de>
|
|
.IP \[bu] 2
|
|
Dan Walters <dan@walters.io>
|
|
.IP \[bu] 2
|
|
Danil Semelenov <sgtpep@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
xopez <28950736+xopez@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Ben Boeckel <mathstuf@gmail.com>
|
|
.IP \[bu] 2
|
|
Manu <manu@snapdragon.cc>
|
|
.IP \[bu] 2
|
|
Kyle E.
|
|
Mitchell <kyle@kemitchell.com>
|
|
.IP \[bu] 2
|
|
Gary Kim <gary@garykim.dev>
|
|
.IP \[bu] 2
|
|
Jon <jonathn@github.com>
|
|
.IP \[bu] 2
|
|
Jeff Quinn <jeffrey.quinn@bluevoyant.com>
|
|
.IP \[bu] 2
|
|
Peter Berbec <peter@berbec.com>
|
|
.IP \[bu] 2
|
|
didil <1284255+didil@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
id01 <gaviniboom@gmail.com>
|
|
.IP \[bu] 2
|
|
Robert Marko <robimarko@gmail.com>
|
|
.IP \[bu] 2
|
|
Philip Harvey <32467456+pharveybattelle@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
JorisE <JorisE@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
garry415 <garry.415@gmail.com>
|
|
.IP \[bu] 2
|
|
forgems <forgems@gmail.com>
|
|
.IP \[bu] 2
|
|
Florian Apolloner <florian@apolloner.eu>
|
|
.IP \[bu] 2
|
|
Aleksandar Jankovi\['c] <office@ajankovic.com>
|
|
<ajankovic@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Maran <maran@protonmail.com>
|
|
.IP \[bu] 2
|
|
nguyenhuuluan434 <nguyenhuuluan434@gmail.com>
|
|
.IP \[bu] 2
|
|
Laura Hausmann <zotan@zotan.pw> <laura@hausmann.dev>
|
|
.IP \[bu] 2
|
|
yparitcher <y@paritcher.com>
|
|
.IP \[bu] 2
|
|
AbelThar <abela.tharen@gmail.com>
|
|
.IP \[bu] 2
|
|
Matti Niemenmaa <matti.niemenmaa+git@iki.fi>
|
|
.IP \[bu] 2
|
|
Russell Davis <russelldavis@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Yi FU <yi.fu@tink.se>
|
|
.IP \[bu] 2
|
|
Paul Millar <paul.millar@desy.de>
|
|
.IP \[bu] 2
|
|
justinalin <justinalin@qnap.com>
|
|
.IP \[bu] 2
|
|
EliEron <subanimehd@gmail.com>
|
|
.IP \[bu] 2
|
|
justina777 <chiahuei.lin@gmail.com>
|
|
.IP \[bu] 2
|
|
Chaitanya Bankanhal <bchaitanya15@gmail.com>
|
|
.IP \[bu] 2
|
|
Micha\[/l] Matczuk <michal@scylladb.com>
|
|
.IP \[bu] 2
|
|
Macavirus <macavirus@zoho.com>
|
|
.IP \[bu] 2
|
|
Abhinav Sharma <abhi18av@outlook.com>
|
|
.IP \[bu] 2
|
|
ginvine <34869051+ginvine@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Patrick Wang <mail6543210@yahoo.com.tw>
|
|
.IP \[bu] 2
|
|
Cenk Alti <cenkalti@gmail.com>
|
|
.IP \[bu] 2
|
|
Andreas Chlupka <andy@chlupka.com>
|
|
.IP \[bu] 2
|
|
Alfonso Montero <amontero@tinet.org>
|
|
.IP \[bu] 2
|
|
Ivan Andreev <ivandeex@gmail.com>
|
|
.IP \[bu] 2
|
|
David Baumgold <david@davidbaumgold.com>
|
|
.IP \[bu] 2
|
|
Lars Lehtonen <lars.lehtonen@gmail.com>
|
|
.IP \[bu] 2
|
|
Matei David <matei.david@gmail.com>
|
|
.IP \[bu] 2
|
|
David <david.bramwell@endemolshine.com>
|
|
.IP \[bu] 2
|
|
Anthony Rusdi <33247310+antrusd@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Richard Patel <me@terorie.dev>
|
|
.IP \[bu] 2
|
|
\[u5E84]\[u5929]\[u7FFC] <zty0826@gmail.com>
|
|
.IP \[bu] 2
|
|
SwitchJS <dev@switchjs.com>
|
|
.IP \[bu] 2
|
|
Raphael <PowershellNinja@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Sezal Agrawal <sezalagrawal@gmail.com>
|
|
.IP \[bu] 2
|
|
Tyler <TylerNakamura@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Brett Dutro <brett.dutro@gmail.com>
|
|
.IP \[bu] 2
|
|
Vighnesh SK <booterror99@gmail.com>
|
|
.IP \[bu] 2
|
|
Arijit Biswas <dibbyo456@gmail.com>
|
|
.IP \[bu] 2
|
|
Michele Caci <michele.caci@gmail.com>
|
|
.IP \[bu] 2
|
|
AlexandrBoltris <ua2fgb@gmail.com>
|
|
.IP \[bu] 2
|
|
Bryce Larson <blarson@saltstack.com>
|
|
.IP \[bu] 2
|
|
Carlos Ferreyra <crypticmind@gmail.com>
|
|
.IP \[bu] 2
|
|
Saksham Khanna <sakshamkhanna@outlook.com>
|
|
.IP \[bu] 2
|
|
dausruddin <5763466+dausruddin@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
zero-24 <zero-24@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Xiaoxing Ye <ye@xiaoxing.us>
|
|
.IP \[bu] 2
|
|
Barry Muldrey <barry@muldrey.net>
|
|
.IP \[bu] 2
|
|
Sebastian Brandt <sebastian.brandt@friday.de>
|
|
.IP \[bu] 2
|
|
Marco Molteni <marco.molteni@mailbox.org>
|
|
.IP \[bu] 2
|
|
Ankur Gupta <7876747+ankur0493@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Maciej Zimnoch <maciej@scylladb.com>
|
|
.IP \[bu] 2
|
|
anuar45 <serdaliyev.anuar@gmail.com>
|
|
.IP \[bu] 2
|
|
Fernando <ferferga@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
David Cole <david.cole@sohonet.com>
|
|
.IP \[bu] 2
|
|
Wei He <git@weispot.com>
|
|
.IP \[bu] 2
|
|
Outvi V <19144373+outloudvi@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Thomas Kriechbaumer <thomas@kriechbaumer.name>
|
|
.IP \[bu] 2
|
|
Tennix <tennix@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Ole Sch\[:u]tt <ole@schuett.name>
|
|
.IP \[bu] 2
|
|
Kuang-che Wu <kcwu@csie.org>
|
|
.IP \[bu] 2
|
|
Thomas Eales <wingsuit@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Paul Tinsley <paul.tinsley@vitalsource.com>
|
|
.IP \[bu] 2
|
|
Felix Hungenberg <git@shiftgeist.com>
|
|
.IP \[bu] 2
|
|
Benjamin Richter <github@dev.telepath.de>
|
|
.IP \[bu] 2
|
|
landall <cst_zf@qq.com>
|
|
.IP \[bu] 2
|
|
thestigma <thestigma@gmail.com>
|
|
.IP \[bu] 2
|
|
jtagcat <38327267+jtagcat@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Damon Permezel <permezel@me.com>
|
|
.IP \[bu] 2
|
|
boosh <boosh@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
unbelauscht <58393353+unbelauscht@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Motonori IWAMURO <vmi@nifty.com>
|
|
.IP \[bu] 2
|
|
Benjapol Worakan <benwrk@live.com>
|
|
.IP \[bu] 2
|
|
Dave Koston <dave.koston@stackpath.com>
|
|
.IP \[bu] 2
|
|
Durval Menezes <DurvalMenezes@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Tim Gallant <me@timgallant.us>
|
|
.IP \[bu] 2
|
|
Frederick Zhang <frederick888@tsundere.moe>
|
|
.IP \[bu] 2
|
|
valery1707 <valery1707@gmail.com>
|
|
.IP \[bu] 2
|
|
Yves G <theYinYeti@yalis.fr>
|
|
.IP \[bu] 2
|
|
Shing Kit Chan <chanshingkit@gmail.com>
|
|
.IP \[bu] 2
|
|
Franklyn Tackitt <franklyn@tackitt.net>
|
|
.IP \[bu] 2
|
|
Robert-Andr\['e] Mauchin <zebob.m@gmail.com>
|
|
.IP \[bu] 2
|
|
evileye <48332831+ibiruai@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Joachim Brandon LeBlanc <brandon@leblanc.codes>
|
|
.IP \[bu] 2
|
|
Patryk Jakuszew <patryk.jakuszew@gmail.com>
|
|
.IP \[bu] 2
|
|
fishbullet <shindu666@gmail.com>
|
|
.IP \[bu] 2
|
|
greatroar <\[at]>
|
|
.IP \[bu] 2
|
|
Bernd Schoolmann <mail@quexten.com>
|
|
.IP \[bu] 2
|
|
Elan Ruusam\[:a]e <glen@pld-linux.org>
|
|
.IP \[bu] 2
|
|
Max Sum <max@lolyculture.com>
|
|
.IP \[bu] 2
|
|
Mark Spieth <mspieth@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
harry <me@harry.plus>
|
|
.IP \[bu] 2
|
|
Samantha McVey <samantham@posteo.net>
|
|
.IP \[bu] 2
|
|
Jack Anderson <jack.anderson@metaswitch.com>
|
|
.IP \[bu] 2
|
|
Michael G <draget@speciesm.net>
|
|
.IP \[bu] 2
|
|
Brandon Philips <brandon@ifup.org>
|
|
.IP \[bu] 2
|
|
Daven <dooven@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Martin Stone <martin@d7415.co.uk>
|
|
.IP \[bu] 2
|
|
David Bramwell <13053834+dbramwell@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Sunil Patra <snl_su@live.com>
|
|
.IP \[bu] 2
|
|
Adam Stroud <adam.stroud@gmail.com>
|
|
.IP \[bu] 2
|
|
Kush <kushsharma@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Matan Rosenberg <matan129@gmail.com>
|
|
.IP \[bu] 2
|
|
gitch1 <63495046+gitch1@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
ElonH <elonhhuang@gmail.com>
|
|
.IP \[bu] 2
|
|
Fred <fred@creativeprojects.tech>
|
|
.IP \[bu] 2
|
|
S\['e]bastien Gross <renard@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Maxime Suret <11944422+msuret@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Caleb Case <caleb@storj.io> <calebcase@gmail.com>
|
|
.IP \[bu] 2
|
|
Ben Zenker <imbenzenker@gmail.com>
|
|
.IP \[bu] 2
|
|
Martin Michlmayr <tbm@cyrius.com>
|
|
.IP \[bu] 2
|
|
Brandon McNama <bmcnama@pagerduty.com>
|
|
.IP \[bu] 2
|
|
Daniel Slyman <github@skylayer.eu>
|
|
.IP \[bu] 2
|
|
Alex Guerrero <guerrero@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Matteo Pietro Dazzi <matteopietro.dazzi@gft.com>
|
|
.IP \[bu] 2
|
|
edwardxml <56691903+edwardxml@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Roman Kredentser <shareed2k@gmail.com>
|
|
.IP \[bu] 2
|
|
Kamil Trzci\[u0144]ski <ayufan@ayufan.eu>
|
|
.IP \[bu] 2
|
|
Zac Rubin <z-0@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Vincent Feltz <psycho@feltzv.fr>
|
|
.IP \[bu] 2
|
|
Heiko Bornholdt <bornholdt@informatik.uni-hamburg.de>
|
|
.IP \[bu] 2
|
|
Matteo Pietro Dazzi <matteopietro.dazzi@gmail.com>
|
|
.IP \[bu] 2
|
|
jtagcat <gitlab@c7.ee>
|
|
.IP \[bu] 2
|
|
Petri Salminen <petri@salminen.dev>
|
|
.IP \[bu] 2
|
|
Tim Burke <tim.burke@gmail.com>
|
|
.IP \[bu] 2
|
|
Kai L\[:u]ke <kai@kinvolk.io>
|
|
.IP \[bu] 2
|
|
Garrett Squire <github@garrettsquire.com>
|
|
.IP \[bu] 2
|
|
Evan Harris <eharris@puremagic.com>
|
|
.IP \[bu] 2
|
|
Kevin <keyam@microsoft.com>
|
|
.IP \[bu] 2
|
|
Morten Linderud <morten@linderud.pw>
|
|
.IP \[bu] 2
|
|
Dmitry Ustalov <dmitry.ustalov@gmail.com>
|
|
.IP \[bu] 2
|
|
Jack <196648+jdeng@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
kcris <cristian.tarsoaga@gmail.com>
|
|
.IP \[bu] 2
|
|
tyhuber1 <68970760+tyhuber1@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
David Ibarra <david.ibarra@realty.com>
|
|
.IP \[bu] 2
|
|
Tim Gallant <tim@lilt.com>
|
|
.IP \[bu] 2
|
|
Kaloyan Raev <kaloyan@storj.io>
|
|
.IP \[bu] 2
|
|
Jay McEntire <jay.mcentire@gmail.com>
|
|
.IP \[bu] 2
|
|
Leo Luan <leoluan@us.ibm.com>
|
|
.IP \[bu] 2
|
|
aus <549081+aus@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Aaron Gokaslan <agokaslan@fb.com>
|
|
.IP \[bu] 2
|
|
Egor Margineanu <egmar@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Lucas Kanashiro <lucas.kanashiro@canonical.com>
|
|
.IP \[bu] 2
|
|
WarpedPixel <WarpedPixel@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Sam Edwards <sam@samedwards.ca>
|
|
.IP \[bu] 2
|
|
wjielai <gouki0123@gmail.com>
|
|
.IP \[bu] 2
|
|
Muffin King <jinxz_k@live.com>
|
|
.IP \[bu] 2
|
|
Christopher Stewart <6573710+1f47a@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Russell Cattelan <cattelan@digitalelves.com>
|
|
.IP \[bu] 2
|
|
gyutw <30371241+gyutw@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Hekmon <edouardhur@gmail.com>
|
|
.IP \[bu] 2
|
|
LaSombra <lasombra@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Dov Murik <dov.murik@gmail.com>
|
|
.IP \[bu] 2
|
|
Ameer Dawood <ameer1234567890@gmail.com>
|
|
.IP \[bu] 2
|
|
Dan Hipschman <dan.hipschman@opendoor.com>
|
|
.IP \[bu] 2
|
|
Josh Soref <jsoref@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
David <david@staron.nl>
|
|
.IP \[bu] 2
|
|
Ingo <ingo@hoffmann.cx>
|
|
.IP \[bu] 2
|
|
Adam Pl\['a]nsk\['y] <adamplansky@users.noreply.github.com>
|
|
<adamplansky@gmail.com>
|
|
.IP \[bu] 2
|
|
Manish Gupta <manishgupta.ait@gmail.com>
|
|
.IP \[bu] 2
|
|
Deepak Sah <sah.sslpu@gmail.com>
|
|
.IP \[bu] 2
|
|
Marcin Zelent <marcin@zelent.net>
|
|
.IP \[bu] 2
|
|
zhucan <zhucan.k8s@gmail.com>
|
|
.IP \[bu] 2
|
|
James Lim <james.lim@samsara.com>
|
|
.IP \[bu] 2
|
|
Laurens Janssen <BD69BM@insim.biz>
|
|
.IP \[bu] 2
|
|
Bob Bagwill <bobbagwill@gmail.com>
|
|
.IP \[bu] 2
|
|
Nathan Collins <colli372@msu.edu>
|
|
.IP \[bu] 2
|
|
lostheli
|
|
.IP \[bu] 2
|
|
kelv <kelvin@acks.org>
|
|
.IP \[bu] 2
|
|
Milly <milly.ca@gmail.com>
|
|
.IP \[bu] 2
|
|
gtorelly <gtorelly@gmail.com>
|
|
.IP \[bu] 2
|
|
Brad Ackerman <brad@facefault.org>
|
|
.IP \[bu] 2
|
|
Mitsuo Heijo <mitsuo.heijo@gmail.com>
|
|
.IP \[bu] 2
|
|
Claudio Bantaloukas <rockdreamer@gmail.com>
|
|
.IP \[bu] 2
|
|
Benjamin Gustin <gustin.ben@gmail.com>
|
|
.IP \[bu] 2
|
|
Ingo Weiss <ingo@redhat.com>
|
|
.IP \[bu] 2
|
|
Kerry Su <me@sshockwave.net>
|
|
.IP \[bu] 2
|
|
Ilyess Bachiri <ilyess.bachiri@sonder.com>
|
|
.IP \[bu] 2
|
|
Yury Stankevich <urykhy@gmail.com>
|
|
.IP \[bu] 2
|
|
kice <wslikerqs@gmail.com>
|
|
.IP \[bu] 2
|
|
Denis Neuling <denisneuling@gmail.com>
|
|
.IP \[bu] 2
|
|
Janne Johansson <icepic.dz@gmail.com>
|
|
.IP \[bu] 2
|
|
Patrik Nordl\['e]n <patriki@gmail.com>
|
|
.IP \[bu] 2
|
|
CokeMine <aptx4561@gmail.com>
|
|
.IP \[bu] 2
|
|
S\[u01A1]n Tr\[u1EA7]n-Nguy\[u1EC5]n <github@sntran.com>
|
|
.IP \[bu] 2
|
|
lluuaapp <266615+lluuaapp@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Zach Kipp <kipp.zach@gmail.com>
|
|
.IP \[bu] 2
|
|
Riccardo Iaconelli <riccardo@kde.org>
|
|
.IP \[bu] 2
|
|
Sakuragawa Misty <gyc990326@gmail.com>
|
|
.IP \[bu] 2
|
|
Nicolas Rueff <nicolas@rueff.fr>
|
|
.IP \[bu] 2
|
|
Pau Rodriguez-Estivill <prodrigestivill@gmail.com>
|
|
.IP \[bu] 2
|
|
Bob Pusateri <BobPusateri@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Alex JOST <25005220+dimejo@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Alexey Tabakman <samosad.ru@gmail.com>
|
|
.IP \[bu] 2
|
|
David Sze <sze.david@gmail.com>
|
|
.IP \[bu] 2
|
|
cynthia kwok <cynthia.m.kwok@gmail.com>
|
|
.IP \[bu] 2
|
|
Miron Veryanskiy <MironVeryanskiy@gmail.com>
|
|
.IP \[bu] 2
|
|
K265 <k.265@qq.com>
|
|
.IP \[bu] 2
|
|
Vesnyx <Vesnyx@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Dmitry Chepurovskiy <me@dm3ch.net>
|
|
.IP \[bu] 2
|
|
Rauno Ots <rauno.ots@cgi.com>
|
|
.IP \[bu] 2
|
|
Georg Neugschwandtner <georg.neugschwandtner@gmx.net>
|
|
.IP \[bu] 2
|
|
pvalls <polvallsrue@gmail.com>
|
|
.IP \[bu] 2
|
|
Robert Thomas <31854736+wolveix@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Romeo Kienzler <romeo.kienzler@gmail.com>
|
|
.IP \[bu] 2
|
|
tYYGH <tYYGH@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
georne <77802995+georne@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Maxwell Calman <mcalman@MacBook-Pro.local>
|
|
.IP \[bu] 2
|
|
Naveen Honest Raj <naveendurai19@gmail.com>
|
|
.IP \[bu] 2
|
|
Lucas Messenger <lmesseng@cisco.com>
|
|
.IP \[bu] 2
|
|
Manish Kumar <krmanish260@gmail.com>
|
|
.IP \[bu] 2
|
|
x0b <x0bdev@gmail.com>
|
|
.IP \[bu] 2
|
|
CERN through the CS3MESH4EOSC Project
|
|
.IP \[bu] 2
|
|
Nick Gaya <nicholasgaya+github@gmail.com>
|
|
.IP \[bu] 2
|
|
Ashok Gelal <401055+ashokgelal@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Dominik Mydlil <dominik.mydlil@outlook.com>
|
|
.IP \[bu] 2
|
|
Nazar Mishturak <nazarmx@gmail.com>
|
|
.IP \[bu] 2
|
|
Ansh Mittal <iamAnshMittal@gmail.com>
|
|
.IP \[bu] 2
|
|
noabody <noabody@yahoo.com>
|
|
.IP \[bu] 2
|
|
OleFrost <82263101+olefrost@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Kenny Parsons <kennyparsons93@gmail.com>
|
|
.IP \[bu] 2
|
|
Jeffrey Tolar <tolar.jeffrey@gmail.com>
|
|
.IP \[bu] 2
|
|
jtagcat <git-514635f7@jtag.cat>
|
|
.IP \[bu] 2
|
|
Tatsuya Noyori <63089076+public-tatsuya-noyori@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
lewisxy <lewisxy@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Nolan Woods <nolan_w@sfu.ca>
|
|
.IP \[bu] 2
|
|
Gautam Kumar <25435568+gautamajay52@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Chris Macklin <chris.macklin@10xgenomics.com>
|
|
.IP \[bu] 2
|
|
Antoon Prins <antoon.prins@surfsara.nl>
|
|
.IP \[bu] 2
|
|
Alexey Ivanov <rbtz@dropbox.com>
|
|
.IP \[bu] 2
|
|
Serge Pouliquen <sp31415@free.fr>
|
|
.IP \[bu] 2
|
|
acsfer <carlos@reendex.com>
|
|
.IP \[bu] 2
|
|
Tom <tom@tom-fitzhenry.me.uk>
|
|
.IP \[bu] 2
|
|
Tyson Moore <tyson@tyson.me>
|
|
.IP \[bu] 2
|
|
database64128 <free122448@hotmail.com>
|
|
.IP \[bu] 2
|
|
Chris Lu <chrislusf@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Reid Buzby <reid@rethink.software>
|
|
.IP \[bu] 2
|
|
darrenrhs <darrenrhs@gmail.com>
|
|
.IP \[bu] 2
|
|
Florian Penzkofer <fp@nullptr.de>
|
|
.IP \[bu] 2
|
|
Xuanchen Wu <117010292@link.cuhk.edu.cn>
|
|
.IP \[bu] 2
|
|
partev <petrosyan@gmail.com>
|
|
.IP \[bu] 2
|
|
Dmitry Sitnikov <fo2@inbox.ru>
|
|
.IP \[bu] 2
|
|
Haochen Tong <i@hexchain.org>
|
|
.IP \[bu] 2
|
|
Michael Hanselmann <public@hansmi.ch>
|
|
.IP \[bu] 2
|
|
Chuan Zh <zhchuan7@gmail.com>
|
|
.IP \[bu] 2
|
|
Antoine GIRARD <antoine.girard@sapk.fr>
|
|
.IP \[bu] 2
|
|
Justin Winokur (Jwink3101) <Jwink3101@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Mariano Absatz (git) <scm@baby.com.ar>
|
|
.IP \[bu] 2
|
|
Greg Sadetsky <lepetitg@gmail.com>
|
|
.IP \[bu] 2
|
|
yedamo <logindaveye@gmail.com>
|
|
.IP \[bu] 2
|
|
hota <lindwurm.q@gmail.com>
|
|
.IP \[bu] 2
|
|
vinibali <vinibali1@gmail.com>
|
|
.IP \[bu] 2
|
|
Ken Enrique Morel <ken.morel.santana@gmail.com>
|
|
.IP \[bu] 2
|
|
Justin Hellings <justin.hellings@gmail.com>
|
|
.IP \[bu] 2
|
|
Parth Shukla <pparth@pparth.net>
|
|
.IP \[bu] 2
|
|
wzl <wangzl31@outlook.com>
|
|
.IP \[bu] 2
|
|
HNGamingUK <connor@earnshawhome.co.uk>
|
|
.IP \[bu] 2
|
|
Jonta <359397+Jonta@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
YenForYang <YenForYang@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Joda St\[:o]\[ss]er <stoesser@yay-digital.de> <services+github@simjo.st>
|
|
.IP \[bu] 2
|
|
Logeshwaran <waranlogesh@gmail.com>
|
|
.IP \[bu] 2
|
|
Rajat Goel <rajat@dropbox.com>
|
|
.IP \[bu] 2
|
|
r0kk3rz <r0kk3rz@gmail.com>
|
|
.IP \[bu] 2
|
|
Matthew Sevey <mjsevey@gmail.com>
|
|
.IP \[bu] 2
|
|
Filip Rysavy <fil@siasky.net>
|
|
.IP \[bu] 2
|
|
Ian Levesque <ian@ianlevesque.org>
|
|
.IP \[bu] 2
|
|
Thomas Stachl <thomas@stachl.me>
|
|
.IP \[bu] 2
|
|
Dmitry Bogatov <git#v1@kaction.cc>
|
|
.IP \[bu] 2
|
|
thomae <4493560+thomae@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
trevyn <trevyn-git@protonmail.com>
|
|
.IP \[bu] 2
|
|
David Liu <david.yx.liu@oracle.com>
|
|
.IP \[bu] 2
|
|
Chris Nelson <stuff@cjnaz.com>
|
|
.IP \[bu] 2
|
|
Felix Bu\[u0308]nemann <felix.buenemann@gmail.com>
|
|
.IP \[bu] 2
|
|
At\['i]lio Ant\[^o]nio <atiliodadalto@hotmail.com>
|
|
.IP \[bu] 2
|
|
Roberto Ricci <ricci@disroot.org>
|
|
.IP \[bu] 2
|
|
Carlo Mion <mion00@gmail.com>
|
|
.IP \[bu] 2
|
|
Chris Lu <chris.lu@gmail.com>
|
|
.IP \[bu] 2
|
|
Vitor Arruda <vitor.pimenta.arruda@gmail.com>
|
|
.IP \[bu] 2
|
|
bbabich <bbabich@datamossa.com>
|
|
.IP \[bu] 2
|
|
David <dp.davide.palma@gmail.com>
|
|
.IP \[bu] 2
|
|
Borna Butkovic <borna@favicode.net>
|
|
.IP \[bu] 2
|
|
Fredric Arklid <fredric.arklid@consid.se>
|
|
.IP \[bu] 2
|
|
Andy Jackson <Andrew.Jackson@bl.uk>
|
|
.IP \[bu] 2
|
|
Sinan Tan <i@tinytangent.com>
|
|
.IP \[bu] 2
|
|
deinferno <14363193+deinferno@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
rsapkf <rsapkfff@pm.me>
|
|
.IP \[bu] 2
|
|
Will Holtz <wholtz@gmail.com>
|
|
.IP \[bu] 2
|
|
GGG KILLER <gggkiller2@gmail.com>
|
|
.IP \[bu] 2
|
|
Logeshwaran Murugesan <logeshwaran@testpress.in>
|
|
.IP \[bu] 2
|
|
Lu Wang <coolwanglu@gmail.com>
|
|
.IP \[bu] 2
|
|
Bumsu Hyeon <ksitht@gmail.com>
|
|
.IP \[bu] 2
|
|
Shmz Ozggrn <98463324+ShmzOzggrn@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Kim <kim@jotta.no>
|
|
.IP \[bu] 2
|
|
Niels van de Weem <n.van.de.weem@smile.nl>
|
|
.IP \[bu] 2
|
|
Koopa <codingkoopa@gmail.com>
|
|
.IP \[bu] 2
|
|
Yunhai Luo <yunhai-luo@hotmail.com>
|
|
.IP \[bu] 2
|
|
Charlie Jiang <w@chariri.moe>
|
|
.IP \[bu] 2
|
|
Alain Nussbaumer <alain.nussbaumer@alleluia.ch>
|
|
.IP \[bu] 2
|
|
Vanessasaurus <814322+vsoch@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Isaac Levy <isaac.r.levy@gmail.com>
|
|
.IP \[bu] 2
|
|
Gourav T <workflowautomation@protonmail.com>
|
|
.IP \[bu] 2
|
|
Paulo Martins <paulo.pontes.m@gmail.com>
|
|
.IP \[bu] 2
|
|
viveknathani <viveknathani2402@gmail.com>
|
|
.IP \[bu] 2
|
|
Eng Zer Jun <engzerjun@gmail.com>
|
|
.IP \[bu] 2
|
|
Abhiraj <abhiraj.official15@gmail.com>
|
|
.IP \[bu] 2
|
|
M\['a]rton Elek <elek@apache.org> <elek@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Vincent Murphy <vdm@vdm.ie>
|
|
.IP \[bu] 2
|
|
ctrl-q <34975747+ctrl-q@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Nil Alexandrov <nalexand@akamai.com>
|
|
.SH Contact the rclone project
|
|
.SS Forum
|
|
.PP
|
|
Forum for questions and general discussion:
|
|
.IP \[bu] 2
|
|
https://forum.rclone.org
|
|
.SS GitHub repository
|
|
.PP
|
|
The project\[aq]s repository is located at:
|
|
.IP \[bu] 2
|
|
https://github.com/rclone/rclone
|
|
.PP
|
|
There you can file bug reports or contribute with pull requests.
|
|
.SS Twitter
|
|
.PP
|
|
You can also follow me on twitter for rclone announcements:
|
|
.IP \[bu] 2
|
|
[\[at]njcw](https://twitter.com/njcw)
|
|
.SS Email
|
|
.PP
|
|
Or if all else fails or you want to ask something private or
|
|
confidential email Nick Craig-Wood (mailto:nick@craig-wood.com).
|
|
Please don\[aq]t email me requests for help - those are better directed
|
|
to the forum.
|
|
Thanks!
|
|
.SH AUTHORS
|
|
Nick Craig-Wood.
|