Commit Graph

4832 Commits

Author SHA1 Message Date
Nick Craig-Wood
7de13fc426 build: add -buildmode to cross-compile.go
This builds on

768e4c4735 build: Temporary fix for Windows build errors

But passes the -buildmode flag down to the cross-compile.go command
too.
2021-01-13 16:48:18 +00:00
Ivan Andreev
c2f6d48d45 build: Temporary fix for Windows build errors
Applies a temporary fix similar to https://github.com/grafana/grafana/pull/28557
before go 1.15.6+ fixes https://github.com/golang/go/issues/40795
2021-01-13 16:48:18 +00:00
Nick Craig-Wood
9d9999d17b serve sftp: fix authentication on one connection blocking others - fixes #4882
Before this change, if one connection was authenticating this would
block any others from authenticating.

This was due to ssh.NewServerConn not being called in a go routine
after the Accept call.

This is fixed by running the ssh authentication in a go routine.

Thanks to @FiloSottile for advice on how to fix this.

See: https://github.com/golang/go/issues/43521
2021-01-13 16:48:18 +00:00
Nick Craig-Wood
15f31d3ca4 webdav: add "Depth: 0" to GET requests to fix bitrix
See: https://forum.rclone.org/t/bitrix24-de-remote-support/21112/
2021-01-13 16:48:18 +00:00
Nick Craig-Wood
0ea51f74a1 webdav: fix Open Range requests to fix 4shared mount
Before this change the webdav backend didn't truncate Range requests
to the size of the object. Most webdav providers are OK with this (it
is RFC compliant), but it causes 4shared to return 500 internal error.

Because Range requests are used in mounting, this meant that mounting
didn't work for 4shared.

This change truncates the Range request to the size of the object.

See: https://forum.rclone.org/t/cant-copy-use-files-on-webdav-mount-4shared-that-have-foreign-characters/21334/
2021-01-13 16:48:18 +00:00
Nick Craig-Wood
6cd360233d serve http: fix serving files of unknown length
Before this change serving files of unknown length were always
returned as 0 length files.

This change serves them correctly, but does not support Range:
requests on them.

See: https://forum.rclone.org/t/serve-http-behavior-when-the-size-is-unknown/21319
2021-01-13 16:48:18 +00:00
negative0
687d2d495b plugins: Create plugins files only if webui is enabled. Fixes #4592. May fix #4600. 2021-01-13 16:48:18 +00:00
Nathan Collins
50a107a5f3 fshttp: prevent overlap of HTTP headers in logs 2021-01-13 16:48:18 +00:00
Nick Craig-Wood
2ed2861d09 build: update nfpm syntax to fix build of .deb/.rpm packages 2021-01-13 16:48:18 +00:00
Nick Craig-Wood
e2cd449c62 build: fix brew install --cask syntax for macOS build 2021-01-13 16:48:18 +00:00
Nick Craig-Wood
98dbbc78ab build: revert GitHub actions brew fix since this is now fixed
Revert "build: work around GitHub actions brew problem"

This reverts commit a2fa1370c5.
2021-01-13 16:48:18 +00:00
Nick Craig-Wood
53c4191350 mount: add "." and ".." to directories to match cmount and expectations
See: https://forum.rclone.org/t/empty-directorys-size-for-a-mounted-crypt-remote/21077
2021-01-13 16:48:18 +00:00
Nick Craig-Wood
e4ece15e68 vfs: make cache dir absolute before using it to fix path too long errors
If --cache-dir is passed in as a relative path, then rclone will not
be able to turn it into a UNC path under Windows, which means that
file names longer than 260 chars will fail when stored in the cache.

This patch makes the --cache-dir path absolute before using it.

See: https://forum.rclone.org/t/handling-of-long-paths-on-windows-260-characters/20913
2021-01-13 16:48:18 +00:00
Matteo Pietro Dazzi
fbf46908bf build: upgrade docker buildx action 2021-01-13 16:48:18 +00:00
Nick Craig-Wood
a96539eeec gcs: fix server side copy of large objects - fixes #3724
Before this change rclone was using the copy endpoint to copy large objects.

This can fail for large objects with this error:

    Error 413: Copy spanning locations and/or storage classes could
    not complete within 30 seconds. Please use the Rewrite method

This change makes Copy use the Rewrite method as suggested by the
error message which should be good for any size of copy.
2021-01-13 16:48:18 +00:00
Anagh Kumar Baranwal
86cd5230d7 cmount: Add optional brew tag to throw an error when using mount in the
binaries installed via Homebrew - Fixes #4775

Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2021-01-13 16:48:18 +00:00
Maciej Zimnoch
716019cf7d accounting: fix data race in Transferred()
startedTransfers is accessed by multiple threads, and it wasn't
protected by the mutex call in Transferred() func.

Fixes #4799
2021-01-13 16:48:18 +00:00
Nick Craig-Wood
c59fe40795 pcloud: only use SHA1 hashes in EU region
Apparently only SHA1 hashes are supported in the EU region for
pcloud. This has been confirmed by pCloud support. The EU regions also
support SHA256 hashes which we don't support yet.

https://forum.rclone.org/t/pcloud-to-local-no-hashes-in-common/19440
2021-01-13 16:48:18 +00:00
Ankur Gupta
ecd60f2430 fs: parseduration: fixed tests to use UTC time 2021-01-13 16:48:18 +00:00
Nick Craig-Wood
d2a5640c3a Revert "sharefile: fix backend due to API swapping integers for strings"
The API seems to have reverted to what it was before

This reverts commit 095c7bd801.
2021-01-13 16:24:19 +00:00
Ivan Andreev
8d3acfb38c chunker: improve detection of incompatible metadata #4917
Before this patch chunker required that there is at least one
data chunk to start checking for a composite object.

Now if chunker finds at least one potential temporary or control
chunk, it marks found files as a suspected composite object.
When later rclone tries a concrete operation on the object,
it performs postponed metadata read and decides: is this a native
composite object, incompatible metadata or just garbage.
2021-01-10 21:59:25 +03:00
Nick Craig-Wood
200de46249 build: Stop tagged releases making a current beta - fixes #4789
Before this change stable releases updated the current beta which mean
confusingly the current beta release would jump backwards from
1.54.0-beta to 1.53.3-beta say.

This commit stops any tagged build making a current beta release. They
will still make beta releases, they just won't update the
rclone*current*.zip and version.txt files.

This also means that a .0 release will not make a current beta like it
does at the moment.
2020-11-25 13:56:53 +00:00
Nick Craig-Wood
cee618bc03 build: attempt to fix docker build by upgrading ilteoood/docker_buildx 2020-11-25 13:53:42 +00:00
Nick Craig-Wood
db2aa771dc Start v1.53.4-DEV development 2020-11-19 17:50:08 +00:00
Nick Craig-Wood
55bd60019e Version v1.53.3 2020-11-19 17:01:56 +00:00
Nick Craig-Wood
c8b11d27e1 random: seed math/rand in one place with crypto strong seed #4783
This shouldn't be read as encouraging the use of math/rand instead of
crypto/rand in security sensitive contexts, rather as a safer default
if that does happen by accident.
2020-11-19 16:51:26 +00:00
Nick Craig-Wood
4c215cc81e random: fix incorrect use of math/rand instead of crypto/rand CVE-2020-28924
For implications see the linked issue.

Fixes #4783
2020-11-19 16:49:51 +00:00
Nick Craig-Wood
4df333255a sharefile: fix backend due to API swapping integers for strings
For some reason the API started returning some integers as strings in
JSON. This is probably OK in Javascript but it upsets Go.

This is easily fixed with the `json:"name,size"` struct tag.
2020-11-18 15:50:12 +00:00
Nick Craig-Wood
843d684568 vfs: fix vfs/refresh calls with fs= parameter
Before this change rclone gave an error when the fs parameter was
provided.

This change removes the fs parameter from the parameters once it has
been read which avoids the error.

See: https://forum.rclone.org/t/precaching-with-vfs-refresh-fails-with-an-error-when-having-multiple-cloud-drives/20267
2020-11-07 14:27:47 +00:00
Nick Craig-Wood
46ea3d93b5 Start v1.53.3-DEV development 2020-10-26 15:40:37 +00:00
Nick Craig-Wood
89f2d43f17 Version v1.53.2 2020-10-26 13:26:58 +00:00
Nick Craig-Wood
cfc5d76fca build: stop using set-env and set-path in the GitHub actions
A security problem was discovered when using set-env and
set-path. This has been deprecated by GitHub and a new mechanism
introduced.

This patch switches to using the new mechanism which will stop GitHub
warning about the use of the old mechanism.

See: https://github.com/actions/toolkit/security/advisories/GHSA-mfwh-5m23-j46w
2020-10-26 12:51:07 +00:00
Nick Craig-Wood
0af493f693 build: work around GitHub actions brew problem
Brew was failing with

    fatal: 'origin' does not appear to be a git repository
    fatal: Could not read from remote repository.

See: https://github.com/actions/virtual-environments/issues/1811
See: https://github.com/actions/virtual-environments/issues/1869
2020-10-26 10:37:42 +00:00
Nick Craig-Wood
51b3ee9a97 test: add ListRetries config parameter to integration tests
Occasionally the b2 tests fail because the integration tests don't
retry hard enough with their new setting of -list-retries 3. Override
this setting to 5 for the b2 tests only.
2020-10-25 16:15:41 +00:00
Nick Craig-Wood
6a4b49479d union: create root directories if none exist
This fixes the TestUnion: integration test if the /tmp/union[123] dirs
don't exist.
2020-10-25 16:02:52 +00:00
Nick Craig-Wood
4b03ee0f99 test: remove TestS3Ceph: and TestSwiftCeph: from integration tests
Unfortunately we don't have access to this server any more
2020-10-25 08:48:23 +00:00
albertony
2f6231f7ac jottacloud: avoid double url escaping of device/mountpoint - fixes #4697 2020-10-22 16:52:41 +01:00
Nick Craig-Wood
c0e6f54f01 local: fix sizes and syncing with --links option on Windows - fixes #4581
Before this change rclone returned the size from the Stat call of the
link. On Windows this reads as 0 always, however on unix it reads as
the length of the text in the link. This caused errors like this when
syncing:

    Failed to copy: corrupted on transfer: sizes differ 0 vs 13

This change causes Windows platforms to read the link and use that as
the size of the link instead of 0 which fixes the problem.
2020-10-17 10:42:02 +01:00
Nick Craig-Wood
def7b77d0f vfs: Fix --no-modtime to not attempt to set modtimes (as documented)
See: https://forum.rclone.org/t/rclone-mount-with-azure-blob-archive-tier/19414
2020-10-09 17:02:13 +01:00
Nick Craig-Wood
51b18a4a26 onedrive: fix disk usage for sharepoint
Some onedrive sharepoints appear to return all 0s for quota

    "quota":{"deleted":0,"remaining":0,"total":0,"used":0}

This commit detects this and returns unknown for all quota parts.

See: https://forum.rclone.org/t/zero-size-volume-when-mounting-onedrive-sharepoint/19597
2020-10-09 14:12:29 +01:00
buengese
7cb76f9054 jottacloud: remove clientSecret from config when upgrading to token based authentication - #4645 2020-10-08 11:53:27 +02:00
Anagh Kumar Baranwal
00ccc93482 s3: Add missing regions for AWS
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-10-06 16:55:34 +01:00
Dov Murik
f9fe494d93 docs: Box: explain about the backslash-like unicode character
Add the full name of the backslash-lookalike unicode character.
2020-10-06 16:55:31 +01:00
edwardxml
4a0c266787 crypt: update docs
Mostly tense, clarity and point of view proposed changes.

There is still some duplication and benefits that would accrue from further examples.
2020-10-06 16:43:06 +01:00
gyutw
f48d0a518c fichier: increase maximum file size from 100GB to 300GB - fixes #4634 2020-10-06 16:43:06 +01:00
Russell Cattelan
99ff594773 cmd/mount2: fix the swapped UID / GID values 2020-10-06 16:43:06 +01:00
Christopher Stewart
6c140705e3 s3: fix spelling mistake
Fix spelling mistake "patific" => "pacific"
2020-10-06 16:43:06 +01:00
Leo Luan
e76963a971 vfs: Add a missed update of used cache space
The missed update can cause incorrect before-cleaning cache stats
and a pre-mature condition broadcast in purgeOld before the cache
space use is reduced below the quota.
2020-10-06 16:43:06 +01:00
Leo Luan
43ad7b10a2 vfs: Add exponential backoff during ENOSPC retries
Add an exponentially increasing delay during retries up ENOSPC error
to avoid exhausting the 10 retries too soon when the cache space
recovery from item resets is not available from the file system yet
or consumed by other large cache writes.
2020-10-06 16:43:06 +01:00
Leo Luan
f6970c65dd vfs: Fix missed concurrency control between some item operations and reset
Item reset is invoked by cache cleaner for synchronous recovery
from ENOSPC errors. The reset operation removes the cache file and
closes/reopens the downloaders.  Although most parts of reset and
other item operations are done with the item mutex held, the mutex
is released during fd.WriteAt and downloaders calls. We used preAccess
and postAccess calls to serialize Reset, ReadAt, and Open, but missed
some other item operations. The patch adds preAccess/postAccess
calls in Sync, Truncate, Close, WriteAt, and rename.
2020-10-06 16:43:06 +01:00