Commit Graph

5107 Commits

Author SHA1 Message Date
Nick Craig-Wood
019667170f Add Zach Kipp to contributors 2021-01-28 15:54:41 +00:00
albertony
7a496752f3 fs: add support for flag --no-console on windows to hide the console window 2021-01-27 18:44:35 +00:00
Yury Stankevich
b569dc11a0 hdfs: support kerberos authentication #42 2021-01-27 18:16:58 +00:00
Riccardo Iaconelli
df4e6079f1 local: new flag --local-zero-size-links to fix sync on some virtual filesystems
Assume the Stat size of links is zero (and read them instead)

On some virtual filesystems (such ash LucidLink), reading a link size via a
Stat call always returns 0.
However, on unix it reads as the length of the text in the link. This may
cause errors like this when syncing:

    Failed to copy: corrupted on transfer: sizes differ 0 vs 13

Setting this flag causes rclone to read the link and use that as the size of
the link instead of 0 which in most cases fixes the problem.

Fixes #4950

Signed-off-by: Riccardo Iaconelli <riccardo@kde.org>
2021-01-27 18:13:16 +00:00
Zach Kipp
6156f90601 Fix test failure in different local time zones
TestParseDuration relied on an elapsed time calculation which
would vary based on the system local time. Fix the test by not relying
on the system time location. Also make the test more deterministic
by injecting time in tests rather than using system time.

Fixes #4529.
2021-01-27 15:05:35 +00:00
Louis Koo
cdaea62932
s3: fix copy multipart with v2 auth failing with 'SignatureDoesNotMatch'
Signed-off-by: zhuc <zhucan.k8s@gmail.com>
2021-01-27 14:43:02 +00:00
Nick Craig-Wood
78afe01d15 filefabric: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood
4eac88babf premiumizeme: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood
b4217fabd3 opendrive: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood
92b9dabf3c fstests: only test with ASCII uppercase for case insensitive tests
Upper/lower case for multilingual is provider dependent so we don't go
there in the tests.
2021-01-27 14:28:17 +00:00
Antoine GIRARD
4323ff8a63
docs: fix broken link in serve sftp/ftp #4968 2021-01-27 13:41:32 +03:00
Nick Craig-Wood
3e188495f5 sugarsync: fix finding directories in a case insentive way #4830 2021-01-26 14:48:33 +00:00
Nick Craig-Wood
acb9e17eb3 box: fix finding directories in a case insentive way #4830 2021-01-26 14:48:33 +00:00
Nick Craig-Wood
c8ab4f1d02 fstests: add a test for finding a directory in a case insensitive way #4830 2021-01-26 14:48:06 +00:00
Nick Craig-Wood
e776a1b122 fstests: don't run encoding tests on -short 2021-01-26 14:46:23 +00:00
Nick Craig-Wood
c57af26de9 docs: add note about --check-first being useful with --order-by 2021-01-26 12:12:26 +00:00
Nick Craig-Wood
7d89912666 Add lluuaapp to contributors 2021-01-26 12:12:26 +00:00
Martin Michlmayr
cd075f1703 docs: fix markup of arguments #4276
Command line arguments have to be marked as code.
2021-01-25 22:40:46 +03:00
lluuaapp
35b2ca642c b2: fixed possible crash when accessing Backblaze b2 remote 2021-01-25 17:48:40 +00:00
edwardxml
127f48e8ad docs: Rewrite rclone filtering documentation
This is an attempt at rewriting the rclone filter documentation page.

I have drawn largely from what appears to be the strong original
structure of the page; existing text, and forum comments.

The term flag is used throughout rather than differentiating `--`
options with more complex arguments. That diverges from some standard
practice but is consistent with messages in the rclone binary and `go`
documentation.

The term directory not folder is used throughout.

I tried referring to objects more broadly rather than files and it
just did not seem to work. Apart from a note at the top the
explanations refer entirely to paths, directories and files. My
justification is that bucket store users understand the concept of
files. Not all users of directory aware storage are so familiar with
objects, keys and metadata.

Many of the changes I have made involve moving issues into what seemed
to me to be more relevant parts of the original page structure. I
still find the content repetitious and overly long but that may be
inevitable when users can only be expected to read the section of the
page they think most relevant.

I have eliminated the rsync section from the original structure. It is
hard enough explaining how rclone filters work without also setting
out how they do not. Comment on sync is instead relegated to a
paragraph in the directory filter section.

The structure of the page is intended to work with a hugo toc card
from html Header2 to Header3.

My original intention was to establish a separate examples section. I
have instead retained examples in each section, added to them and
tried to make clear what is documentation and what example.

The changes draw on Github and Forum issues too numerous to mention.
for instance:

https://forum.rclone.org/t/certain-exclusion-flags-seem-to-be-ignored/20049/2

I am **especially** grateful for
https://forum.rclone.org/t/object-key-remote-directory-filter-clarification/20386/2
for making sense of directory filters for me.

@ncw has a fun (and useful) online filter app at
https://filterdemo.rclone.org/ I have not referred to it at this stage
though I particularly like the fact that it is tied to the same
codebase as an rclone version.

I have added cautions about mixing the `--filter...` flags with
`--exclude...` or `--include...`. The same issues seem to arise as
already recognised between the latter two.

The formal summary of glob syntax introduced at the top of the page is
shamelessly stolen from https://godoc.org/github.com/gobwas/glob

I have tried not to alter too many header descriptions and thereby
break existing links to them.

The reference to 'lass' in the example has been retained to confuse
all those not of Scottish or Yorkshire heritage.

Some of my activity was to remove ambiguity and I anticipate
suggestions to roll that back where it has become overly complex.

I tried particularly to bring together and make clear material about
directory filters. It was previously scattered throughout the page and
I couldn't understand it. I am particularly grateful for the
explanations I received about directory filters though any remaining
errors are entirely my own.

Removed erroneous references to non existent `--filter...` flags.

In some ways the best person to write this page would be one with no
knowledge whatsoever of how rclone filters work. The further I got
into it the better qualified I found myself to be.

E&OE
2021-01-22 16:59:22 +00:00
Nick Craig-Wood
3e986cdf54 dedup: add warning if used on a remote which can't have duplicate names 2021-01-22 15:25:35 +00:00
Nick Craig-Wood
b80d498304 vfs: fix file leaks with --vfs-cache-mode full and --buffer-size 0
Before this change using --vfs-cache-mode full and --buffer-size 0
together caused the vfs downloader to open more and more downloaders.

This is fixed by introducing a minimum size of 1M for the window to
look for an existing downloader.

Fixes #4892
2021-01-21 18:35:04 +00:00
Nick Craig-Wood
757e696a6b Add Sơn Trần-Nguyễn to contributors 2021-01-21 18:35:04 +00:00
Nick Craig-Wood
e3979131f2 Add CokeMine to contributors 2021-01-21 18:35:04 +00:00
Nick Craig-Wood
a774f6bfdb qingstor: fix rclone cleanup
This patch changes to using the default page limit for listing
unfinished multpart uploads rather than 1000. 1000 is the maximum
specified in the docs, but setting anything larger than 200 gives an
error.
2021-01-21 17:35:31 +00:00
Nick Craig-Wood
d7cd35e2ca qingstor: fix error propagation in CleanUp
Before this change errors cleaning multiple buckets were passing silently
2021-01-21 17:35:31 +00:00
Aleksandar Jankovic
38e70f1797 rc/jobs: add listener for finished jobs
Add jobs.OnFinish method to register listener that will trigger when job
is finished.

Includes fix for stopping listeners.
2021-01-21 13:43:41 +00:00
albertony
3b49440c25 crypt: docs: extended description 2021-01-21 13:40:12 +00:00
Sơn Trần-Nguyễn
7c0287b824 docs: fix references to upstreams in union example 2021-01-21 13:33:16 +00:00
edwardxml
f97c2c85bd
docs: update ftp with note about active mode
See: https://forum.rclone.org/t/copy-or-sync-from-ftp-server-results-in-source-directory-not-found/20636/4
2021-01-21 13:21:31 +00:00
Nick Craig-Wood
14c0d8a93e cryptdecode: fix formatting 2021-01-21 10:39:51 +00:00
Evan Harris
768ad4de2a docs: Updated crypt/cryptdecode docs with additional info 2021-01-21 09:55:20 +00:00
CokeMine
817987dfc4 docs: Update Onedrive max file size limit 2021-01-21 09:50:48 +00:00
buengese
eb090d3544 compress: check type assertion in SetTier - fixes #4941 2021-01-20 22:59:14 +00:00
Nick Craig-Wood
4daf8b7083 Changelog updates from Version v1.53.4 2021-01-20 22:34:04 +00:00
Nick Craig-Wood
0be69018b8 drive: log that emptying the trash can take some time - fixes #4915 2021-01-19 18:09:36 +00:00
Nick Craig-Wood
9b9ab5f3e8 gcs: Fix Entry doesn't belong in directory "" (same as directory) - ignoring
This change allows directory markers to be non-zero in size.

See: https://forum.rclone.org/t/public-gcs-bucket-and-entry-doesnt-belong-in-directory-same-as-directory/21753/
2021-01-19 16:50:37 +00:00
Nick Craig-Wood
072464cbdb gcs: fix anonymous client to use rclone's HTTP client 2021-01-19 16:50:37 +00:00
Nick Craig-Wood
b0491dec88 Add new email address for buengese to contributors 2021-01-19 16:50:28 +00:00
Nick Craig-Wood
ccfefedb47 Add Patrik Nordlén to contributors 2021-01-19 16:48:37 +00:00
Nick Craig-Wood
2fffcf9e7f Add Janne Johansson to contributors 2021-01-19 16:48:37 +00:00
buengese
a39a5d261c docs/compress: but a warning at the top 2021-01-18 21:42:58 +01:00
buengese
45b57822d5 compress: improve testing 2021-01-18 21:42:58 +01:00
buengese
d8984cd37f compress: correctly handle wrapping of remotes without PutStream
Also fixes ObjectInfo wrapping for Hash and Size - fixes #4928
2021-01-18 21:42:58 +01:00
Patrik Nordlén
80e63af470
jottacloud: Add support for Telia Cloud (#4930) 2021-01-17 02:38:57 +01:00
Janne Johansson
db2c38b21b
docs: fix small spelling nit, file -> find in FAQ 2021-01-13 17:36:57 +00:00
Nick Craig-Wood
cef51d58ac jottacloud: fix token refresh failed: is not a regular file error
Before this change the jottacloud token renewer would run and give the
error:

    Token refresh failed: is not a regular file

This is because the refresh runs on the root and it isn't a file.

This was fixed by ignoring that specific error.

See: https://forum.rclone.org/t/jottacloud-crypt-3-gb-copy-runs-for-a-week-without-completing/21173
2021-01-12 17:09:44 +00:00
Nick Craig-Wood
e0b5a13a13 jottacloud: fix token renewer to fix long uploads
See: https://forum.rclone.org/t/jottacloud-crypt-3-gb-copy-runs-for-a-week-without-completing/21173
2021-01-11 16:44:11 +00:00
Nick Craig-Wood
de21356154 Add Denis Neuling to contributors 2021-01-11 16:44:11 +00:00
Ivan Andreev
35a4de2030 chunker: fix case-insensitive NewObject, test metadata detection #4902
- fix test case FsNewObjectCaseInsensitive (PR #4830)
- continue PR #4917, add comments in metadata detection code
- add warning about metadata detection in user documentation
- change metadata size limits, make room for future development
- hide critical chunker parameters from command line
2021-01-10 22:29:24 +03:00