mirror of
https://github.com/rclone/rclone.git
synced 2025-07-19 05:23:32 +02:00
Before this change, rclone had to load an entire directory into RAM in order to sort it so it could be synced. With directories with millions of entries, this used too much memory. This fixes the probem by using an on disk sort when there are more than --list-cutoff entries in a directory. Fixes #7974
269 lines
11 KiB
Markdown
269 lines
11 KiB
Markdown
---
|
||
title: "FAQ"
|
||
description: "Rclone Frequently Asked Questions"
|
||
---
|
||
|
||
# Frequently Asked Questions
|
||
|
||
### Do all cloud storage systems support all rclone commands ###
|
||
|
||
Yes they do. All the rclone commands (e.g. `sync`, `copy`, etc.) will
|
||
work on all the remote storage systems.
|
||
|
||
### Can I copy the config from one machine to another ###
|
||
|
||
Sure! Rclone stores all of its config in a single file. If you want
|
||
to find this file, run `rclone config file` which will tell you where
|
||
it is.
|
||
|
||
See the [remote setup docs](/remote_setup/) for more info.
|
||
|
||
### How do I configure rclone on a remote / headless box with no browser? ###
|
||
|
||
This has now been documented in its own [remote setup page](/remote_setup/).
|
||
|
||
### Can rclone sync directly from drive to s3 ###
|
||
|
||
Rclone can sync between two remote cloud storage systems just fine.
|
||
|
||
Note that it effectively downloads the file and uploads it again, so
|
||
the node running rclone would need to have lots of bandwidth.
|
||
|
||
The syncs would be incremental (on a file by file basis).
|
||
|
||
e.g.
|
||
|
||
rclone sync --interactive drive:Folder s3:bucket
|
||
|
||
|
||
### Using rclone from multiple locations at the same time ###
|
||
|
||
You can use rclone from multiple places at the same time if you choose
|
||
different subdirectory for the output, e.g.
|
||
|
||
```
|
||
Server A> rclone sync --interactive /tmp/whatever remote:ServerA
|
||
Server B> rclone sync --interactive /tmp/whatever remote:ServerB
|
||
```
|
||
|
||
If you sync to the same directory then you should use rclone copy
|
||
otherwise the two instances of rclone may delete each other's files, e.g.
|
||
|
||
```
|
||
Server A> rclone copy /tmp/whatever remote:Backup
|
||
Server B> rclone copy /tmp/whatever remote:Backup
|
||
```
|
||
|
||
The file names you upload from Server A and Server B should be
|
||
different in this case, otherwise some file systems (e.g. Drive) may
|
||
make duplicates.
|
||
|
||
### Why doesn't rclone support partial transfers / binary diffs like rsync? ###
|
||
|
||
Rclone stores each file you transfer as a native object on the remote
|
||
cloud storage system. This means that you can see the files you
|
||
upload as expected using alternative access methods (e.g. using the
|
||
Google Drive web interface). There is a 1:1 mapping between files on
|
||
your hard disk and objects created in the cloud storage system.
|
||
|
||
Cloud storage systems (at least none I've come across yet) don't
|
||
support partially uploading an object. You can't take an existing
|
||
object, and change some bytes in the middle of it.
|
||
|
||
It would be possible to make a sync system which stored binary diffs
|
||
like rsync does, instead of whole objects, but that would break the
|
||
1:1 mapping of files on your hard disk to objects in the remote cloud
|
||
storage system.
|
||
|
||
All the cloud storage systems support partial downloads of content, so
|
||
it would be possible to make partial downloads work. However to make
|
||
this work efficiently this would require storing a significant amount
|
||
of metadata, which breaks the desired 1:1 mapping of files to objects.
|
||
|
||
### Can rclone do bi-directional sync? ###
|
||
|
||
Yes, since rclone v1.58.0, [bidirectional cloud sync](/bisync/) is
|
||
available.
|
||
|
||
### Can I use rclone with an HTTP proxy? ###
|
||
|
||
Yes. rclone will follow the standard environment variables for
|
||
proxies, similar to cURL and other programs.
|
||
|
||
In general the variables are called `http_proxy` (for services reached
|
||
over `http`) and `https_proxy` (for services reached over `https`). Most
|
||
public services will be using `https`, but you may wish to set both.
|
||
|
||
The content of the variable is `protocol://server:port`. The protocol
|
||
value is the one used to talk to the proxy server, itself, and is commonly
|
||
either `http` or `socks5`.
|
||
|
||
Slightly annoyingly, there is no _standard_ for the name; some applications
|
||
may use `http_proxy` but another one `HTTP_PROXY`. The `Go` libraries
|
||
used by `rclone` will try both variations, but you may wish to set all
|
||
possibilities. So, on Linux, you may end up with code similar to
|
||
|
||
export http_proxy=http://proxyserver:12345
|
||
export https_proxy=$http_proxy
|
||
export HTTP_PROXY=$http_proxy
|
||
export HTTPS_PROXY=$http_proxy
|
||
|
||
|
||
Note: If the proxy server requires a username and password, then use
|
||
|
||
export http_proxy=http://username:password@proxyserver:12345
|
||
export https_proxy=$http_proxy
|
||
export HTTP_PROXY=$http_proxy
|
||
export HTTPS_PROXY=$http_proxy
|
||
|
||
The `NO_PROXY` allows you to disable the proxy for specific hosts.
|
||
Hosts must be comma separated, and can contain domains or parts.
|
||
For instance "foo.com" also matches "bar.foo.com".
|
||
|
||
e.g.
|
||
|
||
export no_proxy=localhost,127.0.0.0/8,my.host.name
|
||
export NO_PROXY=$no_proxy
|
||
|
||
Note that the FTP backend does not support `ftp_proxy` yet.
|
||
|
||
### Rclone gives x509: failed to load system roots and no roots provided error ###
|
||
|
||
This means that `rclone` can't find the SSL root certificates. Likely
|
||
you are running `rclone` on a NAS with a cut-down Linux OS, or
|
||
possibly on Solaris.
|
||
|
||
Rclone (via the Go runtime) tries to load the root certificates from
|
||
these places on Linux.
|
||
|
||
"/etc/ssl/certs/ca-certificates.crt", // Debian/Ubuntu/Gentoo etc.
|
||
"/etc/pki/tls/certs/ca-bundle.crt", // Fedora/RHEL
|
||
"/etc/ssl/ca-bundle.pem", // OpenSUSE
|
||
"/etc/pki/tls/cacert.pem", // OpenELEC
|
||
|
||
So doing something like this should fix the problem. It also sets the
|
||
time which is important for SSL to work properly.
|
||
|
||
```
|
||
mkdir -p /etc/ssl/certs/
|
||
curl -o /etc/ssl/certs/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt
|
||
ntpclient -s -h pool.ntp.org
|
||
```
|
||
|
||
The two environment variables `SSL_CERT_FILE` and `SSL_CERT_DIR`, mentioned in the [x509 package](https://godoc.org/crypto/x509),
|
||
provide an additional way to provide the SSL root certificates.
|
||
|
||
Note that you may need to add the `--insecure` option to the `curl` command line if it doesn't work without.
|
||
|
||
```
|
||
curl --insecure -o /etc/ssl/certs/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt
|
||
```
|
||
|
||
### Rclone gives Failed to load config file: function not implemented error ###
|
||
|
||
Likely this means that you are running rclone on Linux version not
|
||
supported by the go runtime, ie earlier than version 2.6.23.
|
||
|
||
See the [system requirements section in the go install
|
||
docs](https://golang.org/doc/install) for full details.
|
||
|
||
### All my uploaded docx/xlsx/pptx files appear as archive/zip ###
|
||
|
||
This is caused by uploading these files from a Windows computer which
|
||
hasn't got the Microsoft Office suite installed. The easiest way to
|
||
fix is to install the Word viewer and the Microsoft Office
|
||
Compatibility Pack for Word, Excel, and PowerPoint 2007 and later
|
||
versions' file formats
|
||
|
||
### tcp lookup some.domain.com no such host ###
|
||
|
||
This happens when rclone cannot resolve a domain. Please check that
|
||
your DNS setup is generally working, e.g.
|
||
|
||
```
|
||
# both should print a long list of possible IP addresses
|
||
dig www.googleapis.com # resolve using your default DNS
|
||
dig www.googleapis.com @8.8.8.8 # resolve with Google's DNS server
|
||
```
|
||
|
||
If you are using `systemd-resolved` (default on Arch Linux), ensure it
|
||
is at version 233 or higher. Previous releases contain a bug which
|
||
causes not all domains to be resolved properly.
|
||
|
||
|
||
The Go resolver decision can be influenced with the `GODEBUG=netdns=...`
|
||
environment variable. This also allows to resolve certain issues with
|
||
DNS resolution. On Windows or MacOS systems, try forcing use of the
|
||
internal Go resolver by setting `GODEBUG=netdns=go` at runtime. On
|
||
other systems (Linux, \*BSD, etc) try forcing use of the system
|
||
name resolver by setting `GODEBUG=netdns=cgo` (and recompile rclone
|
||
from source with CGO enabled if necessary). See the
|
||
[name resolution section in the go docs](https://golang.org/pkg/net/#hdr-Name_Resolution).
|
||
|
||
### Failed to start auth webserver on Windows ###
|
||
```
|
||
Error: config failed to refresh token: failed to start auth webserver: listen tcp 127.0.0.1:53682: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
|
||
...
|
||
yyyy/mm/dd hh:mm:ss Fatal error: config failed to refresh token: failed to start auth webserver: listen tcp 127.0.0.1:53682: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
|
||
```
|
||
|
||
This is sometimes caused by the Host Network Service causing issues with opening the port on the host.
|
||
|
||
A simple solution may be restarting the Host Network Service with eg. Powershell
|
||
```
|
||
Restart-Service hns
|
||
```
|
||
|
||
### The total size reported in the stats for a sync is wrong and keeps changing
|
||
|
||
It is likely you have more than 10,000 files that need to be
|
||
synced. By default, rclone only gets 10,000 files ahead in a sync so as
|
||
not to use up too much memory. You can change this default with the
|
||
[--max-backlog](/docs/#max-backlog-n) flag.
|
||
|
||
### Rclone is using too much memory or appears to have a memory leak
|
||
|
||
Rclone is written in Go which uses a garbage collector. The default
|
||
settings for the garbage collector mean that it runs when the heap
|
||
size has doubled.
|
||
|
||
However it is possible to tune the garbage collector to use less
|
||
memory by [setting GOGC](https://dave.cheney.net/tag/gogc) to a lower
|
||
value, say `export GOGC=20`. This will make the garbage collector
|
||
work harder, reducing memory size at the expense of CPU usage.
|
||
|
||
The most common cause of rclone using lots of memory is a single
|
||
directory with millions of files in.
|
||
|
||
Before rclone v1.70 has to load this entirely into memory as rclone
|
||
objects. Each rclone object takes 0.5k-1k of memory. There is
|
||
[a workaround for this](https://github.com/rclone/rclone/wiki/Big-syncs-with-millions-of-files)
|
||
which involves a bit of scripting.
|
||
|
||
However with rclone v1.70 and later rclone will automatically save
|
||
directory entries to disk when a directory with more than
|
||
[`--list-cutoff`](/docs/#list-cutoff) (1,000,000 by default) entries
|
||
is detected.
|
||
|
||
From v1.70 rclone also has the [--max-buffer-memory](/docs/#max-buffer-memory)
|
||
flag which helps particularly when multi-thread transfers are using
|
||
too much memory.
|
||
|
||
### Rclone changes fullwidth Unicode punctuation marks in file names
|
||
|
||
For example: On a Windows system, you have a file with name `Test:1.jpg`,
|
||
where `:` is the Unicode fullwidth colon symbol. When using rclone
|
||
to copy this to your Google Drive, you will notice that the file
|
||
gets renamed to `Test:1.jpg`, where `:` is the regular (halfwidth) colon.
|
||
|
||
The reason for such renames is the way rclone handles different
|
||
[restricted filenames](/overview/#restricted-filenames) on different
|
||
cloud storage systems. It tries to avoid ambiguous file names as
|
||
much and allow moving files between many cloud storage systems
|
||
transparently, by replacing invalid characters with similar looking
|
||
Unicode characters when transferring to one storage system, and replacing
|
||
back again when transferring to a different storage system where the
|
||
original characters are supported. When the same Unicode characters
|
||
are intentionally used in file names, this replacement strategy leads
|
||
to unwanted renames. Read more [here](/overview/#restricted-filenames-caveats).
|