Commit Graph

7305 Commits

Author SHA1 Message Date
Nick Craig-Wood
38309f2df2 Add Bjørn Smith to contributors 2023-09-01 18:09:44 +01:00
NoLooseEnds
7487d34c33
jotta: added Telia Sky whitelabel (Norway)
Duplicated Telia Cloud (Sweden) changed the URLs and added teliase and teliano
(instead of just telia) to differentiate.

See: #5153 #5016
2023-09-01 14:55:32 +01:00
kapitainsky
e45cb4fc75
docs: single character remote names in Windows
Clarify how single character remote names are interpreted in Windows (as drive letters)

See: https://forum.rclone.org/t/issue-with-single-character-configuration-on-windows-with-rclone/
2023-09-01 11:00:14 +01:00
Bjørn Smith
21008b4cd5
docs: sftp: add note regarding format of server_command
Elaborate exactly how server_command should be used in the configuration file
2023-09-01 10:57:42 +01:00
Nick Craig-Wood
cffe85e6c5 fshttp: fix --bind 0.0.0.0 allowing IPv6 and --bind ::0 allowing IPv4
Due to a bug/misfeature in the go standard library as described here:
https://github.com/golang/go/issues/48723 the go standard library
binds to both IPv4 and IPv6 when passed 0.0.0.0 or ::0.

This patch detects the bind address and forces the correct IP
protocol.

Fixes #6124
Fixes #6244
See: https://forum.rclone.org/t/issues-with-bind-0-0-0-0-and-onedrive-getting-etag-mismatch-when-using-ipv6/41379/
2023-09-01 10:47:39 +01:00
Nick Craig-Wood
d12a92eac9 box: fix unhelpful decoding of error messages into decimal numbers
Before this change the box backend could make errors like

    Error "not_found" (404): On-Behalf-Of User not found ([123 34 105 110
    118 97 108 105 100 95 117 115 101 114 95 105 100 34 58 123 34 105 100
    34 58 34 48 48 48 48 48 48 48 48 48 48 48 34 125 125])

This fixes it to produce this instead

    Error "not_found" (404): On-Behalf-Of User not found ({"invalid_user_id":{"id":"00000000000"}})
2023-08-31 23:03:27 +01:00
eNV25
11eeaaf792
cmd/ncdu: fix add keybinding to rescan filesystem 2023-08-30 14:29:46 +01:00
David Sze
a603efeaf4 box: add polling support 2023-08-30 09:25:00 +01:00
eNV25
0bd0a992a4 cmd/ncdu: add keybinding to rescan filesystem
Fixes #7255
2023-08-30 09:05:58 +01:00
Justin Hellings
82c8d78a44
docs: may not -> might not, to remove ambiguity
"may not" can be interpreted as "is not allowed".
Replaced with "might not" in both cases to remove ambiguity.
2023-08-29 15:10:50 +01:00
Nick Craig-Wood
a83fec756b build: fix lint errors when re-enabling revive var-naming 2023-08-29 13:03:49 +01:00
Nick Craig-Wood
e953598987 build: fix lint errors when re-enabling revive exported & package-comments 2023-08-29 13:03:13 +01:00
Nick Craig-Wood
feaa20d885 build: re-enable revive linters
In this commit:

75dfdbf211 ci: revert revive settings back to fix lint

We accidentally disabled all the revive linters. Unfortunately setting
the rules clears the default set of rules so it is necessary to
mention all rules that we need.
2023-08-29 13:01:15 +01:00
Nick Craig-Wood
967fc6d7f4 lib/multipart: fix accounting for multipart transfers
This change makes sure the accouting is done when bytes are taken out
of the buffer rather than put in.

See: https://forum.rclone.org/t/improve-transfer-stats-calculation-for-multipart-uploads/41172
2023-08-27 23:10:58 +01:00
Nick Craig-Wood
b95bda1e92 s3: fix purging of root directory with --s3-directory-markers - fixes #7247 2023-08-25 17:39:16 +01:00
Nick Craig-Wood
9c14562850 fstests: add backend integration test for purging root directory #7247 2023-08-25 17:39:07 +01:00
Nick Craig-Wood
f992742404 s3: fix accounting for multpart uploads 2023-08-25 16:31:31 +01:00
Nick Craig-Wood
f2467d07aa oracleobjectstorage: fix accounting for multpart uploads 2023-08-25 16:31:31 +01:00
Nick Craig-Wood
d69cdb79f7 b2: fix accounting for multpart uploads 2023-08-25 16:31:31 +01:00
Nick Craig-Wood
df5d92d709 operations: fix terminology in multi-thread copy 2023-08-25 16:31:31 +01:00
Nick Craig-Wood
1b5b36523b operations: fix accounting for multi-thread transfers
This stops the accounting moving in large chunks and gets it to move
as the data is read out of the buffer.
2023-08-24 17:51:36 +01:00
Nick Craig-Wood
2f424ceecf operations: don't buffer when a backend implements OpenWriterAt
In this case we don't seek on errors so no need for seeking.
2023-08-24 17:51:36 +01:00
Nick Craig-Wood
bc986b44b2 lib/pool: add DelayAccounting() to fix accounting when reading hashes 2023-08-24 16:42:09 +01:00
Nick Craig-Wood
f4b1a51af6 lib/pool: add SetAccounting to RW 2023-08-24 15:28:40 +01:00
Manoj Ghosh
25703ad20e oracleobjectstorage: implement OpenChunkWriter and multi-thread uploads #7056 2023-08-24 12:39:28 +01:00
Nick Craig-Wood
ab803d1278 b2: implement OpenChunkWriter and multi-thread uploads #7056
This implements the OpenChunkWriter interface for b2 which
enables multi-thread uploads.

This makes the memory controls of the s3 backend inoperative; they are
replaced with the global ones.

    --b2-memory-pool-flush-time
    --b2-memory-pool-use-mmap

By using the buffered reader this fixes excessive memory use when
uploading large files as it will share memory pages between all
readers.
2023-08-24 12:39:27 +01:00
Nick Craig-Wood
0427177857 azureblob: implement OpenChunkWriter and multi-thread uploads #7056
This implements the OpenChunkWriter interface for azureblob which
enables multi-thread uploads.

This makes the memory controls of the s3 backend inoperative; they are
replaced with the global ones.

    --azureblob-memory-pool-flush-time
    --azureblob-memory-pool-use-mmap

By using the buffered reader this fixes excessive memory use when
uploading large files as it will share memory pages between all
readers.
2023-08-24 12:39:27 +01:00
Nick Craig-Wood
3dfcfc2caa operations: document multi-thread copy and tweak defaults 2023-08-24 12:39:27 +01:00
Nick Craig-Wood
d4cff1ae19 operations: add abort on exit to multithread copy 2023-08-24 12:39:27 +01:00
Nick Craig-Wood
f5753369e4 operations: multipart: don't buffer transfers to local disk #7056 2023-08-24 12:39:27 +01:00
Nick Craig-Wood
4c76fac594 s3: factor generic multipart upload into lib/multipart #7056
This makes the memory controls of the s3 backend inoperative and
replaced with the global ones.

    --s3-memory-pool-flush-time
    --s3-memory-pool-use-mmap

By using the buffered reader this fixes excessive memory use when
uploading large files as it will share memory pages between all
readers.

Fixes #7141
2023-08-24 12:39:27 +01:00
Nick Craig-Wood
0d0bcdac31 fs: add context.Ctx to ChunkWriter methods
WriteChunk in particular needs a different context from that which
OpenChunkWriter was used with so add it to all the methods.
2023-08-24 12:39:27 +01:00
Nick Craig-Wood
f3bd02f0ef operations: fix and tidy multithread code
- fix docs and error messages for multithread
- use sync/errgroup built in concurrency limiting
- re-arrange multithread code
- don't continue multi-thread uploads if one part fails
2023-08-24 12:39:27 +01:00
Nick Craig-Wood
e6fde67491 s3: fix retry logic, logging and error reporting for chunk upload
- move retries into correct place into lowest level functions
- fix logging and error reporting
2023-08-24 12:39:27 +01:00
Nick Craig-Wood
b4e3332e02 fs: introduces aliases for OpenWriterAtFn and OpenChunkWriterFn 2023-08-24 12:39:27 +01:00
Nick Craig-Wood
0dea83a4aa pool: add page backed reader/writer for multi thread uploads 2023-08-24 12:39:27 +01:00
Nick Craig-Wood
e8f3f98aa0 lib/readers: add NoSeeker to adapt io.Reader to io.ReadSeeker 2023-08-24 12:39:27 +01:00
Nick Craig-Wood
d61328e459 serve ftp: fix race condition when using the auth proxy
In this commit we introduced a race condition when using the auth
proxy.

94a320f23c serve ftp: update to goftp.io/server v2.0.1

This was due to the re-organisation of the upstream library which made
the driver be a singleton rather than per session.

This means that when using the auth proxy we need to keep track of
which VFS to use by based on which FTP user is connected.

This also adjusts the locking so that the methods will run
concurrently.
2023-08-23 15:11:47 +01:00
r-ricci
9844704567
docs: remove contributor's old email 2023-08-23 12:31:48 +01:00
Nick Craig-Wood
94a320f23c serve ftp: update to goftp.io/server v2.0.1 - fixes #7237 2023-08-22 17:24:05 +01:00
Nick Craig-Wood
7fc573db27 serve sftp: fix hash calculations with --vfs-cache-mode full
Before this change uploading files with rclone to:

    rclone serve sftp --vfs-cache-mode full

Would return the error:

    command "md5sum XXX" failed with error: unexpected non file

This patch detects that the file is still in the VFS cache and reads
the MD5SUM from there rather from the remote.

Fixes #7241
2023-08-22 13:18:36 +01:00
Nick Craig-Wood
af95616122 Add Roberto Ricci to contributors 2023-08-22 13:18:29 +01:00
Roberto Ricci
72f9f1e9c0 vfs: make sure struct field is aligned for atomic access 2023-08-22 12:52:13 +01:00
Roberto Ricci
91b8152321 vfs: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
552b6c47ff lib: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
01a155fb00 fs: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
50d0597d56 cmount: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
123a030441 smb: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
28ceb323ee sftp: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
c624dd5c3a seafile: use atomic types 2023-08-22 12:52:13 +01:00