Nick Craig-Wood
27b281ef69
chunkedreader: add --vfs-read-chunk-streams to parallel read chunks
...
This converts the ChunkedReader into an interface and provides two
implementations one sequential and one parallel.
This can be used to improve the performance of the VFS on high
bandwidth or high latency links.
Fixes #4760
2024-08-14 21:13:09 +01:00
Nick Craig-Wood
cc634213a5
fs: define the optional interface SetMetadata and implement it in wrapping backends
...
This also implements backend integration tests for the feature
2024-05-14 12:51:03 +01:00
Lewis Hook
bf494d48d6
Improve error messages when objects have been corrupted on transfer - fixes #5268
2024-03-23 12:35:35 +00:00
Nick Craig-Wood
39db8caff1
cache,chunker,combine,compress,crypt,hasher,union: implement MkdirMetadata and related Features
2024-02-28 16:09:04 +00:00
nielash
0297542f6b
cache,chunker,combine,compress,crypt,hasher,union: implement DirSetModTime (if supported by wrapped remote)
2024-02-28 16:09:04 +00:00
Nick Craig-Wood
c69eb84573
chunker,compress,crypt,hasher,union: fix rclone move a file over itself deleting the file
...
This fixes the Root() returned by the backend when it has returned
fs.ErrorIsFile.
Before this change it returned a root which included the file path.
Because Root() was wrong this caused the detection of the file being
moved over itself check to fail.
This adds an integration test to check it for all backends.
See: https://forum.rclone.org/t/rclone-move-chunker-dir-file-chunker-dir-deletes-all-file-chunks/43333/
2023-12-10 22:29:57 +00:00
Nick Craig-Wood
3ea1c5c4d2
compress: fix ChangeNotify
...
ChangeNotify has been broken on the compress backend for a long time!
Before this change it was wrapping the file names received rather than
unwrapping them to discover the original names.
It is likely ChangeNotify was working adequately though for users as
the VFS just uses the directories rather than the file names.
2023-09-05 17:22:36 +01:00
Nick Craig-Wood
4fbe0652c9
compress: fix integration tests by adding missing OpenChunkWriter exclude
2023-09-04 19:26:14 +01:00
Vitor Gomes
f36ca0cd25
features: add new interfaces OpenChunkWriter and ChunkWriter #7056
2023-08-12 17:55:01 +01:00
Nick Craig-Wood
3faa84b47c
combine,compress,crypt,hasher,union: support wrapping backends with PartialUploads
...
This means that, for example, wrapping a sftp backend with crypt will
upload to a temporary name and then rename unless disabled with
--inplace.
See: https://forum.rclone.org/t/backup-versioning/38978/7
2023-06-14 10:52:03 +01:00
albertony
5d6b8141ec
Replace deprecated ioutil
...
As of Go 1.16, the same functionality is now provided by package io or
package os, and those implementations should be preferred in new code.
2022-11-07 11:41:47 +00:00
Nick Craig-Wood
5f13d84135
compress: add extra debugging in case we have a repeat of #6434
2022-09-18 11:31:11 +01:00
Nick Craig-Wood
07efdb55fa
compress: fix error handling to not use or return nil objects #6434
2022-09-18 11:31:11 +01:00
Nick Craig-Wood
fb6ddd680c
compress: fix crash due to nil metadata #6434
...
Before this fix, if an error ocurred reading the metadata, it could be
set as nil and then used, causing a crash.
This fix changes the readMetadata function so it returns an error, and
the error is always set if the metadata returned is nil.
2022-09-18 11:31:11 +01:00
Josh Soref
ce3b65e6dc
all: fix spelling across the project
...
* abcdefghijklmnopqrstuvwxyz
* accounting
* additional
* allowed
* almost
* already
* appropriately
* arise
* bandwidth
* behave
* bidirectional
* brackets
* cached
* characters
* cloud
* committing
* concatenating
* configured
* constructs
* current
* cutoff
* deferred
* different
* directory
* disposition
* dropbox
* either way
* error
* excess
* experiments
* explicitly
* externally
* files
* github
* gzipped
* hierarchies
* huffman
* hyphen
* implicitly
* independent
* insensitive
* integrity
* libraries
* literally
* metadata
* mimics
* missing
* modification
* multipart
* multiple
* nightmare
* nonexistent
* number
* obscure
* ourselves
* overridden
* potatoes
* preexisting
* priority
* received
* remote
* replacement
* represents
* reproducibility
* response
* satisfies
* sensitive
* separately
* separator
* specifying
* string
* successful
* synchronization
* syncing
* šenfeld
* take
* temporarily
* testcontents
* that
* the
* themselves
* throttling
* timeout
* transaction
* transferred
* unnecessary
* using
* webbrowser
* which
* with
* workspace
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-08-30 11:16:26 +02:00
Nick Craig-Wood
6fd9e3d717
build: reformat comments to pass go1.19 vet
...
See: https://go.dev/doc/go1.19#go-doc
2022-08-05 16:35:41 +01:00
Nick Craig-Wood
c198700812
compress: support metadata
2022-06-29 17:30:37 +01:00
Nick Craig-Wood
866c873daa
backend: allow wrapping backend tests to run in make quicktest
2022-06-29 17:30:37 +01:00
albertony
fdd2f8e6d2
Error strings should not be capitalized
...
Reported by staticcheck 2022.1.2 (v0.3.2)
See: staticcheck.io
2022-06-23 23:26:02 +02:00
albertony
ec117593f1
Fix lint issues reported by staticcheck
...
Used staticcheck 2022.1.2 (v0.3.2)
See: staticcheck.io
2022-06-13 21:13:50 +02:00
albertony
8697f0bd26
about: improved error message
2022-05-13 12:08:10 +01:00
Nick Craig-Wood
e0fbca02d4
compress: fix memory leak - fixes #6013
...
Before this change we forgot to close the compressor when checking to
see if an object was compressible.
2022-03-03 17:10:21 +00:00
Nick Craig-Wood
71a784cfa2
compress: fix crash if metadata upload failed - fixes #5994
...
Before this changed the backend attempted to delete a nil object if
the metadata upload failed.
2022-02-28 19:47:52 +00:00
Nick Craig-Wood
e43b5ce5e5
Remove github.com/pkg/errors and replace with std library version
...
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.
This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
2021-11-07 11:53:30 +00:00
albertony
e2f47ecdeb
docs: punctuation cleanup
...
See #5538
2021-10-20 22:56:19 +02:00
Nick Craig-Wood
a98e3ea6f1
build: replace the deprecated golint linter with revive
...
This fixes up a small number of new lint items also
2021-10-15 12:51:31 +01:00
Alfonso Montero
19fc1b2a95
docs/compress: minor improvements
2021-10-09 18:22:38 +02:00
albertony
2925e1384c
Use binary prefixes for size and rate units
...
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
2021-04-27 02:25:52 +03:00
buengese
efdee3a5fe
compress: fix compressed name regexp
2021-04-22 18:38:38 +02:00
buengese
eb090d3544
compress: check type assertion in SetTier - fixes #4941
2021-01-20 22:59:14 +00:00
buengese
45b57822d5
compress: improve testing
2021-01-18 21:42:58 +01:00
buengese
d8984cd37f
compress: correctly handle wrapping of remotes without PutStream
...
Also fixes ObjectInfo wrapping for Hash and Size - fixes #4928
2021-01-18 21:42:58 +01:00
buengese
886b3abac1
compress: fix broken tests
2020-12-02 16:30:02 +01:00
Nick Craig-Wood
47aada16a0
fs: add Shutdown optional method for backends
2020-11-27 17:35:01 +00:00
buengese
bc4282e49e
compress: added experimental compression remote - implements #2098 , #1356 , #675
...
This remote implements transparent compression using gzip. Uses JSON as a for storing metadata.
Co-authored-by: id01 <gaviniboom@gmail.com>
2020-11-13 02:31:59 +01:00
