when multi-thread downloading is enabled, rclone used
to send a write to disk after every read, resulting in a lot
of small writes to different locations of the file.
depending on the underlying filesystem or device, it can be more
efficient to send bigger writes.
When copying to a backend which has the PartialUploads feature flag
set and can Move files the file is copied into a temporary name first.
Once the copy is complete, the file is renamed to the real
destination.
This prevents other processes from seeing partially downloaded copies
of files being downloaded and prevents overwriting the old file until
the new one is complete.
This also adds --inplace flag that can be used to disable the partial
file copy/rename feature.
See #3770
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
Before this patch, files or directories with unknown modtime would
appear as the current date.
When mounted some systems look at modification dates of directories to
see if they change and having them change whenever they drop out of
the directory cache is not optimal.
See #6986
* fs: add TerminalColorMode type
* fs: add new config(flags) for TerminalColorMode
* lib/terminal: use TerminalColorMode to determine how to handle colors
* Add documentation for '--terminal-color-mode'
* tree: remove obsolete --color replaced by global --color
This changes the default behaviour of tree. It now displays colors by
default instead of only displaying them when the flag -C/--color was
active. Old behaviour (no color) can be achieved by setting --color to
'never'.
Fixes: #6604
A very common mistake for new users of rclone is to use a remote name
without a colon. This can be on the command line or in the config when
setting up a crypt backend.
This change checks to see if the user uses a path which matches a
remote name and gives an NOTICE like this if they do
NOTICE: "remote" refers to a local folder, use "remote:" to refer to your remote or "./remote" to hide this warning
See: https://forum.rclone.org/t/sync-to-onedrive-personal-lands-file-in-localfilesystem-but-not-in-onedrive/32956
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.
This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
Some environment variables didn’t behave like their corresponding
command line flags. The affected flags were --stats, --log-level,
--separator, --multi-tread-streams, --rc-addr, --rc-user and --rc-pass.
Example:
RCLONE_STATS='10s'
rclone check remote: remote: --progress
# Expected: rclone check remote: remote: --progress –-stats=10s
# Actual: rclone check remote: remote: --progress
Remote specific options set by environment variables was overruled by
less specific backend options set by environment variables. Example:
RCLONE_DRIVE_USE_TRASH='false'
RCLONE_CONFIG_MYDRIVE_USE_TRASH='true'
rclone deletefile myDrive:my-test-file
# Expected: my-test-file is recoverable in the trash folder
# Actual: my-test-file is permanently deleted (not recoverable)
Backend specific options set by environment variables was overruled by
general backend options set by environment variables. Example:
RCLONE_SKIP_LINKS='true'
RCLONE_LOCAL_SKIP_LINKS='false'
rclone lsd local:
# Expected result: Warnings when symlinks are skipped
# Actual result: No warnings when symlinks are skipped
# That is RCLONE_SKIP_LINKS takes precedence
The above issues have been fixed.
The debug logging (-vv) has been enhanced to show when flags are set by
environment variables.
The documentation has been enhanced with details on the precedence of
configuration options.
See pull request #5341 for more information.
This also factors the config questions into a state based mechanism so
a backend can be configured using the same dialog as rclone config but
remotely.
This commit makes the previously statically configured fs cache configurable.
It introduces two parameters `--fs-cache-expire-duration` and
`--fs-cache-expire-interval` to control the caching of the items.
It also adds new interfaces to lib/cache to set these.
Users have noticed that backends created via the rc have been failing
to refresh their tokens with this error:
Token refresh failed try 1/5: context canceled
This is because the rc server cancels the context used to make the
backend when the request has finished. This same context is used to
refresh the token and the oauth library checks to see if the context
has been cancelled.
This patch creates a new context for the cached backends and copies
the global and filter config into the new context.
See: https://forum.rclone.org/t/google-drive-token-refresh-failed/22283
This is done by making fs.Config private and attaching it to the
context instead.
The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
Adds a flag, --progress-terminal-title, that when used with --progress,
will print the string `ETA: %s` to the terminal title.
This also adds WriteTerminalTitle to lib/terminal
If your filenames contain two near-identical Unicode characters,
rclone will normalize these, making them identical. This flag
gives you the ability to keep them unique. This might
create unintended side effects, such as duplicating files that
contain certain Unicode characters, when downloading them from
certain cloud providers to a macOS filesystem.
Fixes#4228
This allows rclone to exit with a non-zero return code if no files are
transferred. This is useful when calling rclone as part of a workflow/script
pipeline as it allows the end user to stop processing if no files have been
transferred.
NB: Enabling this option will return in rclone exiting non-zero if there are no
transfers. Depending on how your're currently using rclone in your scripts,
this may break existing setups!
This commit adds the `--track-renames-strategy` flag which allows the
user to choose the strategy for tracking renames when using the
`--track-renames` flag.
This can be "hash" or "modtime" or both currently.
This, when used with `--track-renames-strategy modtime` enables
support for tracking renames in encrypted remotes.
Fixes#3696Fixes#2721
This gives you more control over how long rclone will run for, making
it easier to script backups, e.g. via cron. Once the `--max-duration`
time limit is reached, no new transfers will be initiated, but those
already in-flight will be allowed to complete.
Fixes#985
Before this change the expect/continue timeout was set to
--conntimeout which was 60s by default which is too long to wait.
This was noticed when using s3 with a proxy which apparently didn't
support expect / continue properly.
Set --expect-continue-timeout 0 to disable expect/continue.
