Commit Graph

858 Commits

Author SHA1 Message Date
Ole Frost
58c99427b3 config: fixed issues with flags/options set by environment vars.
Some environment variables didn’t behave like their corresponding
command line flags. The affected flags were --stats, --log-level,
--separator, --multi-tread-streams, --rc-addr, --rc-user and --rc-pass.
Example:

    RCLONE_STATS='10s'
    rclone check remote: remote: --progress
    # Expected: rclone check remote: remote: --progress –-stats=10s
    # Actual: rclone check remote: remote: --progress

Remote specific options set by environment variables was overruled by
less specific backend options set by environment variables. Example:

    RCLONE_DRIVE_USE_TRASH='false'
    RCLONE_CONFIG_MYDRIVE_USE_TRASH='true'
    rclone deletefile myDrive:my-test-file
    # Expected: my-test-file is recoverable in the trash folder
    # Actual: my-test-file is permanently deleted (not recoverable)

Backend specific options set by environment variables was overruled by
general backend options set by environment variables. Example:

    RCLONE_SKIP_LINKS='true'
    RCLONE_LOCAL_SKIP_LINKS='false'
    rclone lsd local:
    # Expected result: Warnings when symlinks are skipped
    # Actual result: No warnings when symlinks are skipped
    # That is RCLONE_SKIP_LINKS takes precedence

The above issues have been fixed.

The debug logging (-vv) has been enhanced to show when flags are set by
environment variables.

The documentation has been enhanced with details on the precedence of
configuration options.

See pull request #5341 for more information.
2021-07-05 16:38:20 +01:00
albertony
318fa4472b docs: fix incorrect syntax in config update example 2021-06-10 08:59:18 +02:00
Tom
04a1f673f0
serve sftp: add --stdio flag to serve via stdio - fixes #5311 2021-05-28 13:40:32 +01:00
Ivan Andreev
5b6f637461
fs/hash: align hashsum names and update documentation (#5339)
- Unify all hash names as lowercase alphanumerics without punctuation.
- Legacy names continue to work but disappear from docs, they can be depreciated or dropped later.
- Make rclone hashsum print supported hash list in case of wrong spelling.
- Update documentation.

Fixes #5071
Fixes #4841
2021-05-21 17:32:33 +03:00
Nick Craig-Wood
054b467f32 check: log the hash in use like cryptcheck does
See: https://forum.rclone.org/t/does-a-rclone-check-on-similar-remotes-still-compute-hashes/24288/15
2021-05-18 16:21:19 +01:00
Ivan Andreev
23da913d03
dbhashsum: drop command deprecated a year ago - #4837 (#5336)
dbhashsum was deprecated in rclone 1.52 on 2020-05-27
this patch drops the command completely since rclone 1.56
2021-05-18 12:27:17 +03:00
Nick Craig-Wood
82f1f7d2c4 config: expand docs on config protocol #3455 2021-05-17 12:10:58 +01:00
Nick Craig-Wood
635d1e10ae config create: add --state and --result parameters #3455 2021-05-14 14:07:44 +01:00
Nick Craig-Wood
296ceadda6 fs: add --all to rclone config create/update to ask all the config questions #3455
This also factors the config questions into a state based mechanism so
a backend can be configured using the same dialog as rclone config but
remotely.
2021-05-14 14:07:44 +01:00
Nick Craig-Wood
99caf79ffe config: allow config create and friends to take key=value parameters #3455 2021-05-14 14:07:44 +01:00
Nick Craig-Wood
095cf9e4be config create: add --non-interactive and --continue parameters #3455
This adds a mechanism to add external interfaces to rclone using the
state based configuration.
2021-05-14 14:07:44 +01:00
Nick Craig-Wood
94dbfa4ea6 fs: change Config callback into state based callback #3455
This is a very large change which turns the post Config function in
backends into a state based call and response system so that
alternative user interfaces can be added.

The existing config logic has been converted, but it is quite
complicated and folloup commits will likely be needed to fix it!

Follow up commits will add a command line and API based way of using
this configuration system.
2021-05-14 14:07:44 +01:00
Ivan Andreev
119bddc10b selfupdate: fix archive name on macos 2021-05-13 22:35:39 +03:00
Nolan Woods
ab7dfe0c87 http: clean up Bind to better use middleware 2021-05-02 11:31:01 +01:00
Gautam Kumar
9e8cd6bff9 docs: fixed some typos 2021-04-28 22:55:27 +01:00
Nolan Woods
5d2e327b6f http: Replace httplib with lib/http 2021-04-28 22:54:15 +01:00
Nolan Woods
77221d7528 httplib: Deprecate package 2021-04-28 22:54:15 +01:00
Nolan Woods
1971c1ef87 httplib: Move httplib/serve/data to ../serve/http/data 2021-04-28 22:54:15 +01:00
Nolan Woods
7e7dbe16c2 httplib: Add --template config and flags to serve/data 2021-04-28 22:54:15 +01:00
Nolan Woods
002d323c94 lib/http: Move HTTP object serialization logic to lib/http 2021-04-28 22:54:15 +01:00
Nick Craig-Wood
b78c9a65fa backends: remove log.Fatal and replace with error returns #5234
This changes the Config interface so that it returns an error.
2021-04-27 18:18:08 +01:00
albertony
08a2df51be Use decimal prefixes for counts
Fixes #5126
2021-04-27 02:25:52 +03:00
albertony
2925e1384c Use binary prefixes for size and rate units
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
2021-04-27 02:25:52 +03:00
albertony
98579608ec docs: cleanup spelling of size and rate units 2021-04-27 02:25:52 +03:00
albertony
f8d56bebaf
config: delay load config file (#5258)
Restructuring of config code in v1.55 resulted in config
file being loaded early at process startup. If configuration
file is encrypted this means user will need to supply the password,
even when running commands that does not use config.
This also lead to an issue where mount with --deamon failed to
decrypt the config file when it had to prompt user for passord.

Fixes #5236
Fixes #5228
2021-04-26 23:37:49 +02:00
Nick Craig-Wood
4a9469a3dc test changenotify: add command to help debugging changenotify 2021-04-21 10:52:05 +01:00
buengese
4a5cbf2a19 cmd/ncdu: fix out of range panic in delete 2021-04-16 23:20:03 +02:00
albertony
6366d3dfc5 docs: extend description of drive mount access on windows 2021-04-13 22:33:19 +02:00
albertony
60d376c323 docs: add guide to configuring autorun in install documentation 2021-04-13 22:33:19 +02:00
albertony
7b1ca716bf config: add touch command to ensure config exists at configured location (#5226)
A new command `rclone config touch` which calls config.SaveConfig().
Useful during testing of configuration location things.
It will ensure the config file exists and test that it is writable.
2021-04-13 19:25:09 +03:00
Nazar Mishturak
964c3e0732
rcat: add --size flag for more efficient uploads of known size - fixes #4403
This allows preallocating space at remote end with RcatSize.
2021-04-13 12:25:47 +01:00
Nick Craig-Wood
a46a3c0811 test makefiles: add log levels and speed summary 2021-04-12 18:14:01 +01:00
Nick Craig-Wood
60dcafe04d test makefiles: add --seed flag and make data generated repeatable #5214 2021-04-12 18:14:01 +01:00
albertony
23a0d4a1e6 config: fix issues with memory-only config file paths
Fixes #5222
2021-04-12 18:17:19 +02:00
Nick Craig-Wood
9393225a1d link: use "off" value for unset expiry 2021-04-07 13:58:37 +01:00
albertony
3d3ff61f74
docs: minor cleanup of space around code section 2021-04-07 08:47:29 +02:00
albertony
d98f192425
docs: WinFsp 2021 is out of beta 2021-04-07 08:13:40 +02:00
Ivan Andreev
3859fe2f52
cmd/version: print os/version, kernel and bitness (#5204)
Related to #5121

Note: OpenBSD is stub yet. This will be fixed after upstream PR gets resolved
https://github.com/shirou/gopsutil/pull/993
2021-04-05 21:53:09 +03:00
Ivan Andreev
9eab258ffb build: add build tag noselfupdate
Allow downstream packaging to build rclone without selfupdate command:
$ go build -tags noselfupdate

Fixes #5187
2021-04-04 11:22:09 +03:00
Ivan Andreev
93353c431b selfupdate: dont detect FUSE if build is static
Before this patch selfupdate detected ANY build with cmount tag as a build
having libFUSE capabilities. However, only dynamic builds really have it.
The official linux builds are static and have the cmount tag as of the time
of this writing. This results in inability to update official linux binaries.
This patch fixes that. The build can be fixed independently.
2021-04-03 21:54:15 +03:00
Ivan Andreev
1fe2460e38 selfupdate: abort if updating would discard fuse semantics 2021-03-22 22:55:24 +03:00
Ivan Andreev
ef5c212f9b version: show build tags and type of executable
This patch modifies the output of `rclone version`.
The `os/arch` line is split into `os/type` and `os/arch`.
The `go version` line is now tagged as `go/version` for consistency.

Additionally the `go/linking` line tells whether the rclone
was linked as a static or dynamic executable.
The new `go/tags` line shows a space separated list of build tags.

The info about linking and build tags is also added to the output
of the `core/version` RC endpoint.
2021-03-22 22:55:24 +03:00
Nick Craig-Wood
4cc2a7f342 mount: fix caching of old directories after renaming them
It was discovered `rclone mount` (but not `rclone cmount`) cached
directories after rename which it shouldn't have done.

This caused IO errors when trying to access files in renamed
directories on bucket based file systems.

This turned out to be the kernel caching the directories as basil/fuse
sets their expiry time to 60s for some reason.

This fix invalidates the relevant kernel cache entries in the for the
directories which fixes the problem.

Fixes: #4977
See: https://forum.rclone.org/t/after-a-directory-renmane-using-mv-files-are-not-visible-any-longer/22797
2021-03-22 09:07:01 +00:00
Ivan Andreev
2bdbf00fa3
selfupdate: add instructions on reverting the update (#5141) 2021-03-18 23:11:16 +03:00
albertony
ce182adf46
copyurl: add option to print resulting auto-filename (#5095)
Fixes #5094
2021-03-18 10:04:59 +01:00
Nick Craig-Wood
28f6efe955 cmd: refactor to use fspath.SplitFs instead of fs.ParseRemote #4996 2021-03-15 19:22:07 +00:00
Nick Craig-Wood
3cf6ea848b config: remove log.Fatal and replace with error passing where possible 2021-03-14 16:03:35 +00:00
Evan Harris
83368998be docs: Updated sync and dedupe command docs #4429 2021-03-13 15:01:32 +00:00
Ivan Andreev
6fa74340a0
cmd: rclone selfupdate (#5080)
Implements self-update command
Fixes #548
Fixes #5076
2021-03-11 22:39:30 +03:00
Nick Craig-Wood
a7fd65bf2d config: move account initialisation out into accounting
Before this change the initialisation for the accounting package was
done in the config package for some strange historical reason.
2021-03-11 17:29:26 +00:00
Nick Craig-Wood
1fed2d910c config: make config file system pluggable
If you are using rclone a library you can decide to use the rclone
config file system or not by calling

    configfile.LoadConfig(ctx)

If you don't you will need to set `config.Data` to an implementation
of `config.Storage`.

Other changes
- change interface of config.FileGet to remove unused default
- remove MustValue from config.Storage interface
- change GetValue to return string or bool like elsewhere in rclone
- implement a default config file system which panics with helpful error
- implement getWithDefault to replace the removed MustValue
- don't embed goconfig.ConfigFile so we can change the methods
2021-03-11 17:29:26 +00:00
Naveen Honest Raj
02a5d350f9 rcd: Added systemd notification during the 'rclone rcd' command call. This also fixes #5073.
Signed-off-by: Naveen Honest Raj <naveendurai19@gmail.com>
2021-03-11 17:12:14 +00:00
Nick Craig-Wood
04a8859d29 cmount: fix mount dropping on macOS by setting --daemon-timeout 10m
Previously rclone set --daemon-timeout to 15m by default. However
osxfuse seems to be ignoring that value since it is above the maximum
value of 10m. This is conjecture since the source of osxfuse is no
longer available.

Setting the value to 10m seems to resolve the problem.

See: https://forum.rclone.org/t/rclone-mount-frequently-drops-when-using-plex/22352
2021-02-21 12:56:19 +00:00
albertony
cd6fd4be4b
mount: docs: document the new FileSecurity option in WinFsp 2021 (#5002) 2021-02-17 03:44:28 +03:00
Nick Craig-Wood
5b84adf3b9 test: add "rclone test histogram" for file name distribution stats 2021-02-13 14:24:43 +00:00
Nick Craig-Wood
f890965020 test: add makefiles test command (converted from script) 2021-02-13 14:24:43 +00:00
Nick Craig-Wood
f88a5542cf test: move test commands under "rclone test" and make them visible 2021-02-13 14:24:43 +00:00
Miron Veryanskiy
fd94b3a473 docs: replace #file-caching with #vfs-file-caching
The documentation had dead links pointing to #file-caching. They've been
moved to point to #vfs-file-caching.
2021-02-13 12:56:25 +00:00
Nick Craig-Wood
92187a3b33 cmount: fix unicode issues with accented characters on macOS
This adds

    -o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC

To the mount options if it isn't already present which fixes mounting
issues on macOS with accented characters in the finder.
2021-02-11 15:13:19 +00:00
albertony
cc5a1e90d8 mount: improved handling of relative paths on windows 2021-02-08 20:55:23 +00:00
albertony
b39fa54ab2 mount: allow mounting to root directory on windows 2021-02-08 20:55:23 +00:00
Nick Craig-Wood
8574a7bd67 rc: factor async/sync job handing into rc/jobs from rc/rcserver
This fixes async jobs with `rclone rc --loopback` which isn't very
important but sets the stage for _config setting.
2021-02-07 14:56:41 +00:00
Nick Craig-Wood
1545ace8f2 build: remove go1.13 build constraints 2021-02-03 17:46:08 +00:00
Nick Craig-Wood
bcac8fdc83 Use http.NewRequestWithContext where possible after go1.13 minimum version 2021-02-03 17:41:27 +00:00
Nick Craig-Wood
1415666074 lsjson: fix unterminated JSON in the presence of errors
See: https://forum.rclone.org/t/rclone-lsjson-invalid-json-produced-no-at-the-end/22046
2021-02-02 17:46:56 +00:00
Nick Craig-Wood
2f67681e3b cmount: don't attempt to unmount if fs has been destroyed already #4957 2021-01-30 17:19:28 +00:00
Nick Craig-Wood
8f6f4b053c obscure: make rclone osbcure - ignore newline at end of line
See: https://forum.rclone.org/t/authentification-issues-with-webdav-server/21891
2021-01-28 15:54:41 +00:00
albertony
7a496752f3 fs: add support for flag --no-console on windows to hide the console window 2021-01-27 18:44:35 +00:00
Martin Michlmayr
cd075f1703 docs: fix markup of arguments #4276
Command line arguments have to be marked as code.
2021-01-25 22:40:46 +03:00
Nick Craig-Wood
3e986cdf54 dedup: add warning if used on a remote which can't have duplicate names 2021-01-22 15:25:35 +00:00
Nick Craig-Wood
14c0d8a93e cryptdecode: fix formatting 2021-01-21 10:39:51 +00:00
Evan Harris
768ad4de2a docs: Updated crypt/cryptdecode docs with additional info 2021-01-21 09:55:20 +00:00
Nick Craig-Wood
c553ad5158 serve sftp: fix authentication on one connection blocking others - fixes #4882
Before this change, if one connection was authenticating this would
block any others from authenticating.

This was due to ssh.NewServerConn not being called in a go routine
after the Accept call.

This is fixed by running the ssh authentication in a go routine.

Thanks to @FiloSottile for advice on how to fix this.

See: https://github.com/golang/go/issues/43521
2021-01-06 15:34:07 +00:00
Ilyess Bachiri
5d1f947f32
docs: fix a typo in the dedupe docs 2021-01-05 15:46:57 +00:00
Nick Craig-Wood
2af7b61fc3 cmd: fix wording for no retries message
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326/
2021-01-02 11:20:02 +00:00
Nick Craig-Wood
5601652d65 ncdu: add ! (errror) and . (unreadable) file flags to go with e (empty) 2020-12-28 17:25:46 +00:00
Adam Plánský
b218bc5bed ncdu: add empty folder flag into ncdu browser
Implemented empty folder flag for ncdu browser interface. If there is
empty folder in the list the flag e is prepended before size. If there
is no empty folder this flag is ommited. It has the same behaviour as
original ncdu browser. (https://dev.yorhel.nl/ncdu/man)
2020-12-28 17:25:46 +00:00
Claudio Bantaloukas
f7fe1d766b
cmd/ncdu: highlight read errors instead of aborting - fixes #4014
When a directory cannot be walk-ed because of a permissions error
- or any error for that matter -, ncdu mode keeps track of the error
and highlights directories that could not be read.

Previously, the error would cause ncdu to abort.

Now, directories with unreadable sub-directories are displayed in yellow and
a message warns that the total may be underestimated.

Unreadable directories themselves are displayed in red along with the error message
2020-12-28 14:08:12 +00:00
albertony
55aec19389 mount: docs: add section about windows filesystem permissions 2020-12-28 13:59:34 +00:00
albertony
f50e15c77c mount: docs: add note about mounted file system size 2020-12-28 13:59:34 +00:00
albertony
e3191d096f mount: just set default options without checking if customized by user, because it will be overridden anyway 2020-12-28 13:59:34 +00:00
albertony
07c40780b3 mount: also detect if uid or gid are set in same option string: -o uid=123,gid=456 2020-12-28 13:59:34 +00:00
albertony
67b82b4a28 mount: docs: update documentation according to new syntax on windows 2020-12-28 13:59:34 +00:00
albertony
5f47e1e034 mount: cleanup os specific option handling and documentation 2020-12-28 13:59:34 +00:00
albertony
e92cb9e8f8 mount: more user friendly mounting as network drive on windows
Add --network-mode option to activate mounting as network drive without having to set volume prefix.
Add support for automatic drive letter assignment (not specific to network drive mounting).
Allow full network share unc path in --volname, which will also implicitely activate network drive mounting.
Allow full network share unc path as mountpoint, which will also implicitely activate network drive mounting, and the specified path will be used as volume prefix and the remote will be mounted on an automatically assigned drive letter instead.
2020-12-28 13:59:34 +00:00
Nick Craig-Wood
629c0d0f65 serve http: fix serving files of unknown length
Before this change serving files of unknown length were always
returned as 0 length files.

This change serves them correctly, but does not support Range:
requests on them.

See: https://forum.rclone.org/t/serve-http-behavior-when-the-size-is-unknown/21319
2020-12-27 22:01:41 +00:00
lostheli
c8cfa43ccc Add a download flag to hashsum and related commands to force rclone to download and hash files locally
This commit modifies the operations.hashSum function by adding an alternate code path. This code path is triggered by passing downloadFlag = True. When activated, rclone will download files from the remote and hash them locally. downloadFlag = False preserves the existing behavior of using the remote to retrieve the hash.

This commit modifies HashLister to support the new hashSum method as well as consolidating the roles of HashLister, HashListerBase64, Md5sum, and Sha1sum.  The printing of hashes from the function defined in HashLister has been revised to work with --progress.  There are light changes to operations.syncFprintf and cmd.startProgress for this.

The unit test operations_test.TestHashSums is modified to support this change and test the download functionality.

The command functions hashsum, md5sum, sha1sum, and dbhashsum are modified to support this change.  A download flag has been added and an output-file flag has been added.  The output-file flag writes hashes to a file instead of stdout to avoid the need to redirect stdout.
2020-12-27 15:40:44 +00:00
Nick Craig-Wood
73bb9322f5 rc: prefer actual listener address if using ":port" or "addr:0" only
Before this change rclone would turn `localhost:8888` into
`127.0.0.1:8888` which apparently does not work with some browsers.

See: https://github.com/rclone/rclone-webui-react/issues/117
2020-12-26 16:28:54 +00:00
Nick Craig-Wood
e45716cac2 mount: add "." and ".." to directories to match cmount and expectations
See: https://forum.rclone.org/t/empty-directorys-size-for-a-mounted-crypt-remote/21077
2020-12-17 12:14:22 +00:00
Nick Craig-Wood
c98dd8755c log: fix enabling systemd logging when using --log-file
This also moves all the systemd logging decisions to fs/log
2020-12-17 11:55:27 +00:00
Nick Craig-Wood
86014cebd7 dedupe: add --dedupe-mode list to just list dupes, changing nothing 2020-12-02 16:52:12 +00:00
Nick Craig-Wood
507f861c67 dedupe: add --by-hash to dedupe on hash not file name - fixes #1674 2020-12-02 16:52:12 +00:00
Nick Craig-Wood
8bf4697dc2 serve http/webdav: redirect requests to the base url without the /
When using `--baseurl` before this patch, if a request was made to the
base URL without a trailing / then rclone would return a 404 error.

Unfortunately GVFS / Nautilus makes the request without the /
regardless of what the user put in.

This patch redirects the request to the base URL with a /. So if the
user was using `--baseurl rclone` then a request to
http://localhost/rclone would be redirected with a 308 response to
http://localhost/rclone/

Fixes #4814
2020-12-01 18:00:07 +00:00
Nick Craig-Wood
f7d9b15707 cmount: don't call host.Umount if a signal has been received
Before this change cgofuse and libatexit would race to see who could
unmount the file system with unpredicatable results. On Linux it could
report an error or not, depending.

This change checks to see if umount is beng called from a signal and
if so leaves the unmounting to cgofuse/libfuse.

See #4804
2020-11-29 17:44:00 +00:00
Nick Craig-Wood
1cfce703b2 mountlib: make sure we don't call umount more than once #4804
Before this change when using CTRL-C with rclone cmount the
mount would be unmounted twice.
2020-11-29 17:44:00 +00:00
Nick Craig-Wood
c22d04aa30 filter: deglobalise to put filter config into the context #4685 2020-11-27 17:28:42 +00:00
Nick Craig-Wood
354b4f19ec rc: add context to flag Reload function #4685 2020-11-27 17:28:42 +00:00
edwardxml
19a8b66cee
docs: update rclone about docs
Create a full loop of documentation for rclone about, backends overview
and individual backend pages.

Discussion:
https://github.com/rclone/rclone/pull/4774 relates

Previously pull was requested, in part, under ref
https://github.com/rclone/rclone/pull/4801

Notes:
Introduce a tentative draft see-link format the end of sections to try
rather than lots of in-para links.

Update about.go incl link to list of backends not supporting about.

In list of backends not supporting about, include link to about command
reference.

I appreciate there may be decisions to make going forward about whether
command links should be code formatted, and using proper pretty url
links, but I have fudged that for now.

Update backend pages that do not support about with wording used
previously for ftp - it is in passive voice but I can live with it. (my
own wording and fault). The note is applied to a limitations section. If
one does not already exist it is created (even if there are other
limitations with their own sections)
2020-11-27 14:08:52 +00:00
Anagh Kumar Baranwal
07dee18d6b cmount: Add optional brew tag to throw an error when using mount in the
binaries installed via Homebrew - Fixes #4775

Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-11-26 22:56:59 +00:00
Nick Craig-Wood
9d574c0d63 fshttp: read config from ctx not passed in ConfigInfo #4685 2020-11-26 16:40:12 +00:00