- Changes
- Rename `--s3-authkey` to `--auth-key` to get it out of the s3 backend namespace
- Enable `Content-MD5` integrity checks
- Remove locking after code audit
- Documentation
- Factor out documentation into seperate file
- Add Quickstart to docs
- Add Bugs section to docs
- Add experimental tag to docs
- Add rclone provider to s3 backend docs
- Fixes
- Correct quirks in s3 backend
- Change fmt.Printlns into fs.Logs
- Make metadata storage per backend not global
- Log on startup if anonymous access is enabled
- Coding style fixes
- rename fs to vfs to save confusion with the rest of rclone code
- rename db to b for *s3Backend
Fixes#7062
- add context to log and fallthrough to error log level
- test: use rclone random lib to generate random strings
- calculate hash from vfs cache if file is uploading
- add server started log with server url
- remove md5 hasher
The upstream library rclone uses for rclone mount no longer supports
freebsd. Not only is it broken, but it no longer compiles.
This patch disables rclone mount for freebsd.
However all is not lost for freebsd users - compiling rclone with the
`cmount` tag, so `go install -tags cmount` will install a working
`rclone mount` command which uses cgofuse and the libfuse C library
directly.
Note that the binaries from rclone.org will not have mount support as
we don't have a freebsd build machine in CI and it is very hard to
cross compile cmount.
See: https://github.com/bazil/fuse/issues/280Fixes#5843
ncdu stores the position that it was in for each directory. However
doing a rescan can cause those positions to be out of range if the
number of files decreased in a directory. When re-entering the
directory, this causes an index out of range error.
This fixes the problem by detecting the index out of range and
flushing the saved directory position.
See: https://forum.rclone.org/t/slice-bounds-out-of-range-during-ncdu/42492/
This was caused by a change to the upstream library
ProtonMail/go-crypto checking the flags on the keys more strictly.
However the signing key for rclone is very old and does not have those
flags. Adding those flags using `gpg --edit-key` and then the
`change-usage` subcommand to remove, save, quite then re-add, save
quit the signing capabilities caused the key to work.
This also adds tests for the verification and adds the selfupdate
tests into the integration test harness as they had been disabled on
CI because they rely on external sources and are sometimes unreliable.
Fixes#7373
With automount the target mount drive appears twice in /proc/self/mountinfo.
379 27 0:70 / /mnt/rclone rw,relatime shared:433 - autofs systemd-1 rw,fd=57,...
566 379 0:90 / /mnt/rclone rw,nosuid,nodev,relatime shared:488 - fuse.rclone remote: rw,...
Before this fix we only looked for the mount once in
/proc/self/mountinfo. It finds the automount line and since this
doesn't have fs type rclone it concludes the mount isn't ready yet.
This patch makes rclone look through all the mounts and if any of them
have fs type rclone it concludes the mount is ready.
See: https://forum.rclone.org/t/systemd-mount-works-but-automount-does-not/42287/
Before this change, if a hardlink command was issued, rclone would
just ignore it and not return an error.
This changes any unknown operations (including hardlink) to return an
unsupported error.
Summary:
In cases where cmount is not available in macOS, we alias nfsmount to mount command and transparently start the NFS server and mount it to the target dir.
The NFS server is started on localhost on a random port so it is reasonably secure.
Test Plan:
```
go run rclone.go mount --http-url https://beta.rclone.org :http: nfs-test
```
Added mount tests:
```
go test ./cmd/nfsmount
```
Summary:
Adding a new command to serve any remote over NFS. This is only useful for new macOS versions where FUSE mounts are not available.
* Added willscot/go-nfs dependency and updated go.mod and go.sum
Test Plan:
```
go run rclone.go serve nfs --http-url https://beta.rclone.org :http:
```
Test that it is serving correctly by mounting the NFS directory.
```
mkdir nfs-test
mount -oport=58654,mountport=58654 localhost: nfs-test
```
Then we can list the mounted directory to see it is working.
```
ls nfs-test
```
This was always the intention, it was just implemented wrong.
This shortens the s3 docs by 1369 bringing them down to half the size
just about.
Fixes#7325
Before this change, bisync ignored the dryRun parameter (only when specified
via the rc.)
This change fixes the issue, so that the dryRun rc parameter is equivalent to
the --dry-run flag.
If the server returns the MIME type as application/octet-stream we
assume it doesn't really know what the MIME type. This patch tries
matching the MIME type from the file extension instead in this case.
This enables the use of servers (like OneDrive for Business) which
don't allow the setting of MIME types on upload and have a poor
selection of mime types.
Fixes#7259
In this commit we introduced a race condition when using the auth
proxy.
94a320f23c serve ftp: update to goftp.io/server v2.0.1
This was due to the re-organisation of the upstream library which made
the driver be a singleton rather than per session.
This means that when using the auth proxy we need to keep track of
which VFS to use by based on which FTP user is connected.
This also adjusts the locking so that the methods will run
concurrently.
Before this change uploading files with rclone to:
rclone serve sftp --vfs-cache-mode full
Would return the error:
command "md5sum XXX" failed with error: unexpected non file
This patch detects that the file is still in the VFS cache and reads
the MD5SUM from there rather from the remote.
Fixes#7241
This adds an additional parameter to the creation of each flag. This
specifies one or more flag groups. This **must** be set for global
flags and **must not** be set for local flags.
This causes flags.md to be built with sections to aid comprehension
and it causes the documentation pages for each command (and the
`--help`) to be built showing the flags groups as specified in the
`groups` annotation on the command.
See: https://forum.rclone.org/t/make-docs-for-mortals-not-only-rclone-gurus/39476/
This introduces a new fs.Option flag, Sensitive and uses this along
with IsPassword to redact the info in the config file for support
purposes.
It adds this flag into backends where appropriate. It was necessary to
add oauthutil.SharedOptions to some backends as they were missing
them.
Fixes#5209
When using `rclone cat` to print the contents of several files, the
user may want to inject some separator between the files, such as a
comma or a newline. This patch adds a `--separator` option to the `cat`
command to make that possible. The default value remains an empty
string, `""`, maintaining the prior behavior of `rclone cat`.
Closes#6968
In this commit we accidentally removed the global --rc flags.
0df7466d2b cmd/rcd: Fix command docs to include command specific prefix (#6675)
This re-instates them.
Before this change if both --progress and --interactive were set then
the screen display could become muddled.
This change makes --progress and --interactive use the same lock so
while rclone is asking for interactive questions, the progress will be
paused.
Fixes#6755
This change addresses two issues with commands that re-used
flags from common packages:
1) cobra.Command definitions did not include the command specific
prefix in doc strings.
2) Command specific flag prefixes were added after generating
command doc strings.
Before this change if logs were not redirected, logging would
corrupt the terminal screen.
This commit stores the logs (max ~100 lines) in an array and
print them when the program exits.
This ensures the virtual terminal processing mode is enabled on the rclone process
for Windows 10 consoles (by using Windows Console API functions GetConsoleMode/SetConsoleMode
and flag ENABLE_VIRTUAL_TERMINAL_PROCESSING), which adds native support for ANSI/VT100
escape sequences. This mode is default in many cases, e.g. when using the Windows
Terminal application, but in other cases it is not, and the default can also be
controlled with registry setting (see below), and therefore configuring it on the process
seem to be the only reliable way of ensuring it is enabled when supported.
[HKEY_CURRENT_USER\Console]
"VirtualTerminalLevel"=dword:00000001
Since rclone version 1.61.0 the tree command uses ANSI color sequences in output by
default, but this lead to issues in Windows terminals that were not handling these (#6668).
This commit ensures the tree command uses the terminal package for output. It relies on
go-colorable to properly handle ANSI color sequences: If stdout is connected to a terminal
the escape sequences are decoded and the text are written with color formatting using
Windows Console API. If stdout is not connected to a terminal, e.g. redirected to file,
the escape sequences are stripped off. The tree command has its own method for writing
directly to a file, specified with flag --output, and then the output is not passed
through the terminal package and must therefore be written without ansi codes.
Since version 3 of fuse libfuse no longer does anything when given the
nonempty option and it's default is to allow mounting over non empty
directories like normal mount does.
Some versions of libfuse give an error when using `--allow-non-empty`
which is annoying for the user.
We now do this check ourselves so we no longer need to pass the option
to libfuse.
Fixes#3562
