Commit Graph

7929 Commits

Author SHA1 Message Date
nielash
51582e36e8 onedrive: set all metadata permissions and return error summary
Before this change when setting permissions from the metadata rclone
would stop on the first error.

This change causes rclone to attempt to set all the permissions and
return an error summary at the end.
2024-04-13 19:57:30 +01:00
Kyle Reynolds
47cbddbd27 fs rc: fixes incorrect Content-Type in HTTP API - fixes #7726 2024-04-13 19:56:34 +01:00
nielash
5323a21898 operations: fix move when dst is nil and fdst is case-insensitive
Before this change, the MoveCaseInsensitive logic in operations.move made the
assumption that dst != nil && remote != "". After this change, it should work
correctly when either one is present without the other.
2024-04-13 19:28:09 +01:00
Nick Craig-Wood
f2e693f722 sync: fix case normalisation on s3
Before this change when the sync routine attempted to normalise a
case, say from "FiLe.txt" to "file.txt" this caused a 400 Bad Request
error:

> This copy request is illegal because it is trying to copy an object
> to itself without changing the object's metadata, storage class,
> website redirect location or encryption attributes.

This was caused by passing the same object as the source and
destination to the move routine, whereas the destination object had a
different case and didn't exist, so should have been passed as nil.

See: https://github.com/rclone/rclone/pull/7743#discussion_r1557345906
2024-04-13 19:28:09 +01:00
Nick Craig-Wood
93955b755f operations: fix retries downloading too much data with certain backends
Before this fix if more than one retry happened on a file that rclone
had opened for read with a backend that uses fs.FixRangeOption then
rclone would read too much data and the transfer would fail.

Backends affected:

- azureblob, azurefiles, b2, box, dropbox, fichier, filefabric
- googlecloudstorage, hidrive, imagekit, jottacloud, koofr, netstorage
- onedrive, opendrive, oracleobjectstorage, pikpak, premiumizeme
- protondrive, qingstor, quatrix, s3, sharefile, sugarsync, swift
- uptobox, webdav, zoho

This was because rclone was emitting Range requests for the wrong data
range on the second and subsequent retries.

This was caused by fs.FixRangeOption modifying the options and the
reopen code relying on them not being modified.

This fix makes a copy of the fs.FixRangeOption in the reopen code to
fix the problem.

In future it might be best to change fs.FixRangeOption so it returns a
new options slice.

Fixes #7759
2024-04-13 19:25:15 +01:00
Nick Craig-Wood
a4fc5edc5e operations: add more assertions to ReOpen tests to check seek positions 2024-04-13 19:25:15 +01:00
Nick Craig-Wood
8b73dcb95d Add static-moonlight to contributors 2024-04-13 19:25:15 +01:00
static-moonlight
3ba57cabce doc: add example how to run serve s3 2024-04-13 19:22:26 +01:00
Nick Craig-Wood
c87097109b serve s3: adjust to move of Mikubill/gofakes3 to rclone/gofakes3
This also updates the interface which has gained a ctx parameter in
the mean time.
2024-04-13 18:25:41 +01:00
Nick Craig-Wood
ae76498a38 Add guangwu to contributors 2024-04-13 18:25:41 +01:00
Nick Craig-Wood
10f730c49f Add jakzoe to contributors 2024-04-13 18:25:41 +01:00
albertony
88d96d133b Add go mod and sum to gitattributes for consistent line endings 2024-04-13 11:16:42 +02:00
nielash
2c7680050b bisync: rename extended_char_paths test
The .lck file filename length needs to be less than 255 bytes (not symbols) on
linux, and it was still too long on this test, because of the
subdir=測試_Русский_{spc}_{spc}_ě_áñ
2024-04-11 16:27:20 +01:00
nielash
fe6c9aa4da chunker: fix case-insensitive comparison on local without metadata
Before this change, chunker would erroneously consider two different paths to be
equal if, due to special characters, they normalized to equal-folding strings in
Standard Encoding, but not otherwise. This caused base objects to get moved when
they should not have been. This change fixes the issue, which was discovered on
the bisync integration tests.

Ideally it should also be fixed when the base Fs is non-local, but there's not an
easy way at the moment to reference the wrapped Fs's encoding, at least without
breaking encapsulation.
2024-04-11 16:27:20 +01:00
nielash
8524afa9ce chunker: fix NewFs when root points to composite multi-chunk file without metadata
Before this change, calling NewFs on a composite multi-chunk file with
--chunker-meta-format "none"
would fail due to f.base pointing to the wrong Fs. This change fixes the issue,
which was discovered on the bisync integration tests.
2024-04-11 16:27:20 +01:00
nielash
21f3ba13f6 bisync: more fixes for integration tests
-use fs.ConfigStringFull instead of bilib.StripHexString to properly reverse
connection string remotes
2024-04-11 16:27:20 +01:00
nielash
04128f97ee bisync: fix endless loop if lockfile decoder errors
Before this change, the decoder looked only for `io.EOF`, and if any other error
was returned, it could cause an infinite loop. This change fixes the issue by
breaking for any non-nil error.
2024-04-10 16:33:05 +01:00
nielash
bef9fd0bc3 bisync: make tempDir path shorter
to avoid exceeding linux filename length limits
2024-04-10 16:33:05 +01:00
guangwu
2ab2ec29f9 fix: close cpu profile
Signed-off-by: guoguangwu <guoguangwug@gmail.com>
2024-04-09 11:23:55 +01:00
jakzoe
8817ee25ae
docs: fix typo in filtering.md
Fix typo: moved misplaced double quotation mark.
2024-04-09 11:16:59 +01:00
Nick Craig-Wood
46b3854330 drive: set all metadata permissions and return error summary
Before this change when setting permissions from the metadata rclone
would stop on the first error.

This change causes rclone to attempt to set all the permissions and
return an error summary at the end.
2024-04-08 17:23:22 +01:00
Nick Craig-Wood
efbaca3a95 crypt: fix max suggested length of filenames 2024-04-08 17:23:22 +01:00
nielash
f995ece64d bisync: fix io.PipeWriter not getting closed on tests 2024-04-07 21:55:26 -04:00
jumbi77
68c2ba74dd pikpak: fix a typo in a comment
Last still open fix from PR #6970
2024-04-06 11:27:24 +01:00
albertony
e739ee2c27 docs: ensure empty line between text and a following heading 2024-04-05 21:39:44 +02:00
Dan McArdle
05e5712bc4 .github/workflows: Upgrade deprecated macos-11 to macos-latest
See https://github.com/actions/runner-images
2024-04-05 18:01:39 +01:00
Dan McArdle
a2e38e9883 cmd/gitannex: Downgrade to protocol version 1
This enables compatibility with versions of git-annex currently
available on GitHub's "ubuntu-latest" image, aka Ubuntu 22.04 Jammy.
Currently, Jammy is shipping git-annex 8.20210223-2ubuntu2.
https://packages.ubuntu.com/jammy/git-annex

Issue #7625
2024-04-05 18:01:39 +01:00
Dan McArdle
ef42c32cc6 cmd/gitannex: Replace e2e test script with Go test
This commit implements milestone 2.1 for the gitannex subcommand:
https://github.com/rclone/rclone/issues/7625#issuecomment-1951403856

This rewrite makes a few improvements over the old shell script:

(1) It no longer uses the system's rclone.conf. Now, it writes the
    rclone.conf file in an ephemeral directory.

(2) It no longer makes any assumptions about the contents of /tmp.

However, it now assumes that an rclone built from the HEAD commit is on
the PATH. It makes a best-effort attempt to verify this assumption, but
I'm not sure it's bulletproof.

I'm hoping that writing this in Go will enable more cross-platform
support in the future, but for now we're still restricted to Unixy
systems due to reliance on the HOME environment variable.

Issue #7625
2024-04-05 18:01:39 +01:00
Nick Craig-Wood
6a5c0065ef docs: clarify option syntax
See: https://forum.rclone.org/t/seeming-documentation-problem-rclones-syntax-a-problem-with-the-categories-on-this-forum/45395/
2024-04-05 15:59:32 +01:00
Nick Craig-Wood
6da27db844 build: fix CVE-2023-45288 by upgrading golang.org/x/net
See: https://pkg.go.dev/vuln/GO-2024-2687
2024-04-05 15:59:32 +01:00
Nick Craig-Wood
c0497d46d5 ulozto: remove use of github.com/pkg/errors 2024-04-05 15:59:32 +01:00
Nick Craig-Wood
df3df06d2e Add Pieter van Oostrum to contributors 2024-04-05 15:59:32 +01:00
Pieter van Oostrum
1e3ab7acfd docs: fix MANUAL formatting problems
1) Missing closing code backticks (```) in s3.md causes formatting problems
2) Pandoc requires blank lines before ATX headings
2024-04-05 10:41:47 +02:00
Kyle Reynolds
339bc1d1a3 backend koofr: remove trailing bracket - fixes #7600 2024-04-04 20:03:26 +01:00
nielash
71069ed5c1 webdav: fix SetModTime erasing checksums on owncloud and nextcloud
Before this change, calling SetModTime on owncloud and nextcloud would
inadvertently erase the object's stored hashes. This change fixes the issue,
which was discovered by the bisync integration tests.
2024-04-03 16:43:11 -04:00
nielash
75df38f6ee bisync: use fstest.RandomRemote on tests
- use fstest.RandomRemote to create the root level directory
- fix parsing of canonical name for connection string remotes
2024-04-03 16:43:11 -04:00
nielash
ce4064aabf hdfs: fix f.String() not including subpath 2024-04-03 16:43:11 -04:00
Nick Craig-Wood
7c9f1b8917 local: disable unreliable test
In this commit we merged an unreliable test

e053c8a1c0 copy: fix nil pointer dereference when corrupted on transfer with nil dst

It is a good idea but very hard to implement so it always works.

Hence this disables it for the moment.
2024-04-02 18:48:34 +01:00
Nick Craig-Wood
e71c95a554 docs: update warp sponsorship 2024-04-02 16:32:24 +01:00
nielash
e053c8a1c0 copy: fix nil pointer dereference when corrupted on transfer with nil dst 2024-04-02 15:34:58 +01:00
Nick Craig-Wood
c2d96113ac Add Erisa A to contributors 2024-04-02 15:34:58 +01:00
Nick Craig-Wood
5ff961d2ea Add yoelvini to contributors 2024-04-02 15:34:58 +01:00
Nick Craig-Wood
eedeaf7cbb Add Alexandre Lavigne to contributors 2024-04-02 15:34:58 +01:00
Kyle Reynolds
f6e716543a
test info: improve cleanup of temp files - fixes #7209
Co-authored-by: Kyle Reynolds <kyle.reynolds@bridgerphotonics.com>
2024-04-02 15:03:33 +01:00
nielash
998df26ceb onedrive: fix --metadata-mapper called twice if writing permissions
Before this change, the --metadata-mapper was called twice if an object was
uploaded via multipart upload with --metadata and --onedrive-metadata-permissions
"write" or "read,write". This change fixes the issue.
2024-04-02 14:57:43 +01:00
Pat Patterson
93c960df59 b2: Add tests for new cleanup and cleanup-hidden backend commands. 2024-04-02 12:36:43 +01:00
Nikita Shoshin
92368f6d2b rcserver: set ModTime for dirs and files served by --rc-serve 2024-04-02 12:10:45 +01:00
Erisa A
08bf5228a7 docs: Add R2 note about no_check_bucket 2024-04-02 12:08:56 +01:00
yoelvini
76f3eb3ed2
s3: add new AWS region il-central-1 Tel Aviv 2024-04-01 18:17:16 +01:00
nielash
0d43da7655 bisync: more fixes for integration tests
- fix parsing of connection string remotes (comma in name)
- skip remotes that can't upload empty files
- Mkdir the test case subdir before cache.Get-ing it
	(only storj seems to need this... bug?)
2024-03-31 17:56:28 -04:00