Commit Graph

5804 Commits

Author SHA1 Message Date
Nick Craig-Wood
1fed2d910c config: make config file system pluggable
If you are using rclone a library you can decide to use the rclone
config file system or not by calling

    configfile.LoadConfig(ctx)

If you don't you will need to set `config.Data` to an implementation
of `config.Storage`.

Other changes
- change interface of config.FileGet to remove unused default
- remove MustValue from config.Storage interface
- change GetValue to return string or bool like elsewhere in rclone
- implement a default config file system which panics with helpful error
- implement getWithDefault to replace the removed MustValue
- don't embed goconfig.ConfigFile so we can change the methods
2021-03-11 17:29:26 +00:00
Fionera
c95b580478 config: Wrap config library in an interface 2021-03-11 17:29:26 +00:00
Nick Craig-Wood
2be310cd6e build: fix dependencies for docs build 2021-03-11 17:29:26 +00:00
Naveen Honest Raj
02a5d350f9 rcd: Added systemd notification during the 'rclone rcd' command call. This also fixes #5073.
Signed-off-by: Naveen Honest Raj <naveendurai19@gmail.com>
2021-03-11 17:12:14 +00:00
albertony
18cd2064ec mount: docs: add note about volume path syntax on windows 2021-03-11 17:09:22 +00:00
buengese
59ed70ca91 fichier: implement public link 2021-03-11 00:44:26 +01:00
Nick Craig-Wood
6df56c55b0 Changelog updates from Version v1.54.1 2021-03-08 11:06:11 +00:00
Nick Craig-Wood
94e34cb783 build: fix nfpm install by using the released binary 2021-03-07 16:42:22 +00:00
Robert Thomas
c3e2392f2b
dropbox: fix polling support for scoped apps - fixes #5089 (#5092)
This fixes the polling implementation for Dropbox, particularly
when using a scoped app. This also adds a lower end check for the
timeout, as I forgot to include that in the original implementation.
2021-03-05 17:44:47 +00:00
Nick Craig-Wood
f7e3115955 s3: fix Wasabi HEAD requests returning stale data by using only 1 transport
In this commit

fc5b14b620 s3: Added `--s3-disable-http2` to disable http/2

We created our own transport so we could disable http/2. However the
added function is called twice meaning that we create two HTTP
transports. This didn't happen with the original code because the
default transport is cached by fshttp.

Rclone normally does a PUT followed by a HEAD request to check an
upload has been successful.

With the two transports, the PUT and the HEAD were being done on
different HTTP transports. This means that it wasn't re-using the same
HTTP connection, so the HEAD request showed the previous object value.
This caused rclone to declare the upload was corrupted, delete the
object and try again.

This patch makes sure we only create one transport and use it for both
PUT and HEAD requests which fixes the problem with Wasabi.

See: https://forum.rclone.org/t/each-time-rclone-is-run-1-3-fails-2-3-succeeds/22545
2021-03-05 15:34:56 +00:00
Nick Craig-Wood
e01e8010a0 Add Maxwell Calman to contributors 2021-03-05 15:34:56 +00:00
Ivan Andreev
75056dc9b2
ftp: update dependency jlaffaye/ftp (#5097) 2021-03-05 15:58:04 +03:00
Ivan Andreev
7aa7acd926
address stringent ineffectual assignment check in golangci-lint (#5093) 2021-03-04 14:26:48 +03:00
Nick Craig-Wood
0ad38dd6fa dropbox,ftp,onedrive,yandex: make --timeout 0 work properly
See: https://forum.rclone.org/t/an-issue-about-ftp-backend-in-2-different-systems/22551
2021-03-01 12:08:58 +00:00
Maxwell Calman
9cc8ff4dd4 chunker: partially implement no-rename transactions (#4675)
Some storage providers e.g. S3 don't have an efficient rename operation.
Before this change, when chunker finished an upload, the server-side copy
and delete operations that renamed temporary chunks to their final names
could take a significant amount of time.
This PR records transaction identifier (versioning) in the metadata of
chunker composite objects striving to remove the need for rename
operations on such backends.
This approach will be triggered be the new "transactions" configuration
option, which can be "rename" (the default) or "norename".
We implement the new approach for uploads (Put operations).
The chunker Move operation still uses the rename operation of
underlying backend. Filling this gap is left for a later PR.

Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
2021-02-28 10:49:17 +00:00
Nick Craig-Wood
b029fb591f s3: fix failed to create file system with folder level permissions policy
Before this change, if folder level access permissions policy was in
use, with trailing `/` marking the folders then rclone would HEAD the
path without a trailing `/` to work out if it was a file or a folder.
This returned a permission denied error, which rclone returned to the
user.

    Failed to create file system for "s3:bucket/path/": Forbidden: Forbidden
        status code: 403, request id: XXXX, host id:

Previous to this change

53aa03cc44 s3: complete sse-c implementation

rclone would assume any errors when HEAD-ing the object implied it
didn't exist and this test would not fail.

This change reverts the functionality of the test to work as it did
before, meaning any errors on HEAD will make rclone assume the object
does not exist and the path is referring to a directory.

Fixes #4990
2021-02-24 20:35:44 +00:00
Nick Craig-Wood
95e9c4e7f1 Add georne to contributors 2021-02-24 20:35:44 +00:00
Nick Craig-Wood
c40bafb72c Add tYYGH to contributors 2021-02-24 20:35:44 +00:00
Nick Craig-Wood
eac77b06ab Add Romeo Kienzler to contributors 2021-02-24 20:35:44 +00:00
Yaroslav Halchenko
0355d6daf2 CONTRIBUTING.md: recommend to push feature branch with -u + minor tuneups 2021-02-24 20:24:59 +00:00
buengese
c4b8df6903 fichier: implement copy & move 2021-02-24 21:05:41 +01:00
Ivan Andreev
0dd3ae5e0d Add Robert Thomas to contributors 2021-02-24 19:40:54 +03:00
Robert Thomas
e5aa92c922
dropbox: add polling support - fixes #2949
This implements polling support for the Dropbox backend. The Dropbox SDK dependency had to be updated due to an auth issue, which was fixed on Jan 12 2021. A secondary internal Dropbox service was created to handle unauthorized SDK requests, as is necessary when using the ListFolderLongpoll function/endpoint. The config variable was renamed to cfg to avoid potential conflicts with the imported config package.
2021-02-24 09:33:31 +00:00
Ivan Andreev
f6265fbeff Add pvalls to contributors 2021-02-24 03:35:24 +03:00
Ivan Andreev
1397b85214 Add Georg Neugschwandtner to contributors 2021-02-24 03:28:15 +03:00
Ivan Andreev
86a0dae632 Add Rauno Ots to contributors 2021-02-24 03:27:16 +03:00
Ivan Andreev
076ff96f6b webdav: check that purged directory really exists (#2921)
Sharepoint 2016 returns status 204 to the purge request
even if the directory to purge does not really exist.
This change adds an extra check to detect this condition
and returns a proper error code.
2021-02-23 23:27:30 +00:00
Ivan Andreev
985011e73b webdav: fix sharepoint-ntlm error 401 for parallel actions (#2921)
The go-ntlmssp NTLM negotiator has to try various authentication methods.
Intermediate responses from Sharepoint have status code 401, only the
final one is different. When rclone runs a large operation in parallel
goroutines according to --checkers or --transfers, one of threads can
receive intermediate 401 response targeted for another one and returns
the 401 authentication error to the user.
This patch fixes that.
2021-02-23 23:27:30 +00:00
Ivan Andreev
9ca6bf59c6 webdav: enforce encoding to fix errors with sharepoint-ntlm (#2921)
On-premises Sharepoint returns HTTP errors 400 or 500 in
reply to attempts to use file names with special characters
like hash, percent, tilde, invalid UTF-7 and so on.
This patch activates transparent encoding of such characters.
2021-02-23 23:27:30 +00:00
georne
e5d5ae9ab7 webdav: disable HTTP/2 for NTLM authentication (#2921)
As per Microsoft documentation, Windows authentication
(NTLM/Kerberos/Negotiate) is not supported with HTTP/2.
This patch disables transparent HTTP/2 support when the
vendor setting is "sharepoint-ntlm". Otherwise connections
to IIS/10.0 can fail with HTTP_1_1_REQUIRED.

Co-authored-by: Georg Neugschwandtner <georg.neugschwandtner@gmx.net>
2021-02-23 23:27:30 +00:00
Ivan Andreev
ac6bb222f9 webdav: improve terminology in sharepoint-ntlm docs (#2921)
The most popular keyword for the Sharepoint in-house or company
installations is "On-Premises".
"Microsoft OneDrive account" is in fact just a Microsoft account.

Co-authored-by: Georg Neugschwandtner <georg.neugschwandtner@gmx.net>
2021-02-23 23:27:30 +00:00
Alex Chen
62d5876eb4 webdav: make sharepoint-ntlm docs more consistent (#2921)
Clarify difference between Sharepoint Online and
hosted Sharepoint with NTLM authentication.
2021-02-23 23:27:30 +00:00
Rauno Ots
9808a53416 webdav: add support for sharepoint with NTLM authentication (#2921)
Add new option option "sharepoint-ntlm" for the vendor setting.
Use it when your hosted Sharepoint is not tied to the OneDrive
accounts and uses NTLM authentication.
Also add documentation and integration test.

Fixes: #2171
2021-02-23 23:27:30 +00:00
pvalls
cc08f66dc1 docs: singular/plural duplicity for MByte{s} 2021-02-23 11:34:32 +00:00
pvalls
6b8da24eb8 docs: uppercase for MBytes
MBytes is written as Mbytes and MBytes interchangeably.
Use uppercase consistently across all docs.md
2021-02-23 11:34:32 +00:00
buengese
333faa6c68 zoho: fix custom client id's 2021-02-23 11:27:05 +00:00
Nick Craig-Wood
1b92e4636e rc: implement passing filter config with _filter parameter 2021-02-23 10:54:40 +00:00
Nick Craig-Wood
c5a299d5b1 rc: fix options/local to return the filter options 2021-02-23 10:33:03 +00:00
Nick Craig-Wood
04a8859d29 cmount: fix mount dropping on macOS by setting --daemon-timeout 10m
Previously rclone set --daemon-timeout to 15m by default. However
osxfuse seems to be ignoring that value since it is above the maximum
value of 10m. This is conjecture since the source of osxfuse is no
longer available.

Setting the value to 10m seems to resolve the problem.

See: https://forum.rclone.org/t/rclone-mount-frequently-drops-when-using-plex/22352
2021-02-21 12:56:19 +00:00
Nick Craig-Wood
4b5fe3adad delete,rmdirs: make --rmdirs obey the filters
See: https://forum.rclone.org/t/a-problem-with-rclone-delete-from-list/22143
2021-02-19 10:32:28 +00:00
edwardxml
7db68b72f1
docs: directory filter rules 2021-02-18 12:11:56 +01:00
edwardxml
9c667be2a1
docs: remove dead link from rc.md (#5038) 2021-02-18 01:37:17 +03:00
tYYGH
c0cf54067a
vfs: --vfs-used-is-size to report used space using recursive scan (#4043)
Some backends, most notably S3, do not report the amount of bytes used.
This patch introduces a new flag that allows instead of relying on the
backend, use recursive scan similar to `rclone size` to compute the total
used space. However, this is ineffective and should be used as a last resort.

Co-authored-by: Yves G <theYinYeti@yalis.fr>
2021-02-17 23:36:13 +03:00
Romeo Kienzler
297ca23abd
docs: fix typo in crypt.md (#5037) 2021-02-17 19:11:57 +03:00
Nick Craig-Wood
d809930e1d union: fix mkdir at root with remote:/
Before the this fix if you specified remote:/ then the union backend
would fail to notice the root directory existed.

This was fixed by stripping the trailing / from the root.

See: https://forum.rclone.org/t/upgraded-from-1-45-to-1-54-now-cant-create-new-directory-within-union-mount/22284/
2021-02-17 12:11:34 +00:00
Nick Craig-Wood
fdc0528bd5 Add Dmitry Chepurovskiy to contributors 2021-02-17 12:11:34 +00:00
Nick Craig-Wood
a0320d6e94 Add Vesnyx to contributors 2021-02-17 12:11:34 +00:00
Nick Craig-Wood
89bf036e15 Add K265 to contributors 2021-02-17 12:11:34 +00:00
Dmitry Chepurovskiy
1605f9e14d
s3: Fix shared_credentials_file auth
S3 backend shared_credentials_file option wasn't working neither from
config option nor from command line option. This was caused cause
shared_credentials_file_provider works as part of chain provider, but in
case user haven't specified access_token and access_key we had removed
(set nil) to credentials field, that may contain actual credentials got
from ChainProvider.

AWS_SHARED_CREDENTIALS_FILE env varible as far as i understood worked,
cause aws_sdk code handles it as one of default auth options, when
there's not configured credentials.
2021-02-17 12:04:26 +00:00
albertony
cd6fd4be4b
mount: docs: document the new FileSecurity option in WinFsp 2021 (#5002) 2021-02-17 03:44:28 +03:00