Nick Craig-Wood
3a6f1f5cd7
filter: add metadata filters --metadata-include/exclude/filter and friends
...
Fixes #6353
2022-12-17 10:21:11 +00:00
Eng Zer Jun
8e507075d1
test: replace defer cleanup with t.Cleanup
...
Reference: https://pkg.go.dev/testing#T.Cleanup
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-12-09 14:38:05 +00:00
Nick Craig-Wood
dd71f5d968
fs: move operations.NewOverrideRemote to fs.NewOverrideRemote
2022-11-21 08:02:09 +00:00
Nick Craig-Wood
617c5d5e1b
rcat: preserve metadata when Copy falls back to Rcat
...
Before this change if we copied files of unknown size, then they lost
their metadata.
This was particularly noticeable using --s3-decompress.
This change adds metadata to Rcat and RcatSized and changes Copy to
pass the metadata in when it calls Rcat for an unknown sized input.
Fixes #6546
2022-11-10 12:04:35 +00:00
Nick Craig-Wood
be6f29930b
dedupe: make dedupe obey the filters
...
See: https://forum.rclone.org/t/dial-tcp-lookup-api-pcloud-com-no-such-host/33910/
2022-11-10 09:56:02 +00:00
Nick Craig-Wood
653bc23728
dedupe: count Checks in the stats while scanning for duplicates
...
This allows the user to see rclone has not hung.
See: https://forum.rclone.org/t/dial-tcp-lookup-api-pcloud-com-no-such-host/33910/
2022-11-10 09:56:02 +00:00
albertony
5d6b8141ec
Replace deprecated ioutil
...
As of Go 1.16, the same functionality is now provided by package io or
package os, and those implementations should be preferred in new code.
2022-11-07 11:41:47 +00:00
Josh Soref
ce3b65e6dc
all: fix spelling across the project
...
* abcdefghijklmnopqrstuvwxyz
* accounting
* additional
* allowed
* almost
* already
* appropriately
* arise
* bandwidth
* behave
* bidirectional
* brackets
* cached
* characters
* cloud
* committing
* concatenating
* configured
* constructs
* current
* cutoff
* deferred
* different
* directory
* disposition
* dropbox
* either way
* error
* excess
* experiments
* explicitly
* externally
* files
* github
* gzipped
* hierarchies
* huffman
* hyphen
* implicitly
* independent
* insensitive
* integrity
* libraries
* literally
* metadata
* mimics
* missing
* modification
* multipart
* multiple
* nightmare
* nonexistent
* number
* obscure
* ourselves
* overridden
* potatoes
* preexisting
* priority
* received
* remote
* replacement
* represents
* reproducibility
* response
* satisfies
* sensitive
* separately
* separator
* specifying
* string
* successful
* synchronization
* syncing
* šenfeld
* take
* temporarily
* testcontents
* that
* the
* themselves
* throttling
* timeout
* transaction
* transferred
* unnecessary
* using
* webbrowser
* which
* with
* workspace
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-08-30 11:16:26 +02:00
Nick Craig-Wood
6fd9e3d717
build: reformat comments to pass go1.19 vet
...
See: https://go.dev/doc/go1.19#go-doc
2022-08-05 16:35:41 +01:00
Nick Craig-Wood
a875320e37
sync,operations: optimise --copy-dest and --compare-dest
...
Before this change --compare-dest and --copy-dest would check to see
if the compare/copy object existed first, before seeing if the
destination object was present.
This is inefficient, because in most --copy-dest syncs the destination
will be present and the compare/copy object need never be tested.
--compare-dest syncs may also be speeded up if they are done to the
same directory repeatedly.
This fixes the problem by re-arranging the logic so if the transfer is
not needed then the compare/copy object is never tested.
See: https://forum.rclone.org/t/union-with-copy-dest-enabled-is-slower-than-expected/32172
2022-08-03 17:44:47 +01:00
Nick Naumann
11be920e90
sync: add filter-sensitivity to --backup-dir option
...
The old Overlapping function and corresponding tests have been removed, as it has been completely replaced by the OverlappingFilterCheck function.
2022-07-13 16:04:53 +01:00
Nick Craig-Wood
55bbff6346
operations: add --server-side-across-configs global flag for any backend
2022-07-11 18:17:42 +01:00
Nick Craig-Wood
2e54b56a01
rcat: check checksums by default like copy does #6305
...
Before this change we were calculating the checksum for an rcat
transfer but never checking it.
See: https://forum.rclone.org/t/optimize-rclone-on-raspberry-pi-4-8gb/31741
2022-07-07 16:09:09 +01:00
albertony
5b579cea47
staticcheck: use golang.org/x/text/cases instead of deprecated strings.Title
...
strings.Title has been deprecated since Go 1.18 and an alternative has been
available since Go 1.0. The rule Title uses for word boundaries does not handle
Unicode punctuation properly. Use golang.org/x/text/cases instead.
2022-07-04 11:24:59 +02:00
albertony
0772cae314
staticcheck: use result of type assertion to simplify cases
2022-07-04 11:24:59 +02:00
zzr93
060c8dfff0
operations: use correct src/dst in some log messages
...
Most of the time this will make no difference to user logs, however
the difference may be visible in JSON logs and on the rare occasions
src and dst are pointing to different file names.
2022-07-04 10:18:04 +01:00
Nick Craig-Wood
776a083892
lsf: add metadata support with M flag
2022-06-29 14:29:36 +01:00
Nick Craig-Wood
d823a38ce5
lsjson: add --metadata/-M flag
...
Note that this removes the `-M` flag from `--encrypted` as it
conflicted with the global flag and adds it to `--metadata`.
2022-06-29 14:29:36 +01:00
Nick Craig-Wood
c4451bc43a
fs: add --metadata-set flag to specify metadata for uploads
2022-06-29 14:29:36 +01:00
Nick Craig-Wood
0652ec95db
fs: implement MetadataInfo to show info about metadata in help and rc
...
Info about this will appear in operations/fsinfo and in the backend
help (`rclone help backend s3`).
2022-06-29 14:29:36 +01:00
Nick Craig-Wood
6a0e021dac
fs: implement optional Metadata interface for Objects #111
...
This implements integration tests for the feature also.
2022-06-29 11:21:29 +01:00
Nick Craig-Wood
461d041c4d
fstest: remove spurious contents return from PutTestContents and friends
2022-06-29 11:18:02 +01:00
albertony
fdd2f8e6d2
Error strings should not be capitalized
...
Reported by staticcheck 2022.1.2 (v0.3.2)
See: staticcheck.io
2022-06-23 23:26:02 +02:00
albertony
de5ccaab8e
docs: cross link doc pages for related commands
2022-06-21 23:43:00 +02:00
buengese
ac0dc9922e
copyurl: add tests for the option to honor the HTTP header filename directive
2022-06-20 12:06:09 +01:00
J-P Treen
8b8802a078
copyurl: Add option to honor the HTTP header filename directive.
...
Implemented --header-filename for use with copyurl.
For specifically setting preferred download filenames for HTTP requests, RFC 6226
specifies a 'filename' directive, available within 'Content-Disposition'
header. We can handle with 'mime.ParseMediaType'.
See below for details:
https://httpwg.org/specs/rfc6266.html#disposition.parameter.filename
https://httpwg.org/specs/rfc6266.html#advice.generating
Co-authored-by: buengese <buengese@protonmail.com>
2022-06-20 12:06:09 +01:00
Nick Craig-Wood
115f1c2cc9
operations: speed up hash checking by aborting the other hash if first returns nothing
...
This speeds up hash checks when a Hash() function returns "" - this
means that the hash can be canceled for the other side.
In the common case of local hash vs remote hash empty this saves a lot
of time.
See: https://forum.rclone.org/t/rclone-s3-backend-copy-is-2x-slower-than-aws-s3-cp/27321/9
2022-06-16 22:13:50 +01:00
Nick Craig-Wood
4f94b27800
check: implement --no-traverse and --no-unicode-normalization
...
See: https://forum.rclone.org/t/rclone-check-head-or-list-object-from-source/30400
2022-06-16 22:13:50 +01:00
Nick Craig-Wood
cfe0911e0d
sync: fix tests for overlapping with filter
...
In commit
3ccf222acb
2022-06-09 14:26:43 +01:00
albertony
f4f0e444bf
filter: allow multiple --exclude-if-present flags - fixes #6219
2022-06-08 17:11:52 +02:00
Nick
3ccf222acb
sync: overlap check is now filter-sensitive
...
Previously, the overlap check was based on simple prefix checks of the source and destination paths. Now it actually checks whether the destination is excluded via any filter rule or a "--exclude-if-present"-file.
2022-06-01 18:24:54 +01:00
Eng Zer Jun
4f0ddb60e7
refactor: replace strings.Replace with strings.ReplaceAll
...
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-05-17 11:08:37 +01:00
albertony
86bd5f6922
size: warn about inaccurate results when objects with unknown size
2022-04-26 19:37:15 +02:00
albertony
07f53aebdc
touch: fix issue where directory is created instead of file
...
Detected on ftp, sftp and Dropbox backends.
Fixes #5952
2022-01-28 20:29:12 +01:00
albertony
237daa8aaf
dedupe: add quit as a choice in interactive mode
...
Fixes #5881
2022-01-14 19:57:48 +01:00
Ole Frost
3c2252b7c0
fs/operations: add server-side moves to stats
...
Fixes #5430
2021-11-18 12:20:56 +00:00
Nick Craig-Wood
ba6730720d
Fix repeated error messages after pkg/errors removal
2021-11-15 17:58:40 +00:00
albertony
29abbd2032
hashsum: support creating hash from data received on stdin
...
See #5781
2021-11-11 15:16:45 +01:00
Nick Craig-Wood
f36d6d01b5
rc: fix operations/publiclink default for expires parameter
...
Before this change the expires parameter was defaulting to 0 if not
provided.
This change makes it default to fs.DurationOff which is the same as
the `rclone link` command.
See: https://forum.rclone.org/t/operations-publiclink-from-dropbox-error-not-autorized/27374
2021-11-11 11:14:22 +00:00
database64128
a7a8372976
🧪 fstest: fix time tests on Windows and add convenience methods to check local and remote fs with precision
...
Previously only the fs being checked on gets passed to
GetModifyWindow(). However, in most tests, the test files are
generated in the local fs and transferred to the remote fs. So the
local fs time precision has to be taken into account.
This meant that on Windows the time tests failed because the
local fs has a time precision of 100ns. Checking remote items uploaded
from local fs on Windows also requires a modify window of 100ns.
2021-11-09 11:43:36 +00:00
Nick Craig-Wood
e43b5ce5e5
Remove github.com/pkg/errors and replace with std library version
...
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.
This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
2021-11-07 11:53:30 +00:00
Atílio Antônio
c08d48a50d
docs: improve grammar and fix typos ( #5361 )
...
This alters some comments in source files, but is interested mainly in documentation files and help messages.
2021-11-04 12:50:43 +01:00
albertony
4f05ece39e
test: fix touchdir test on backends without modtime support
2021-10-22 13:37:34 +02:00
albertony
a70c20fe6b
touch: improve error message from recursive touch
2021-10-22 13:37:34 +02:00
albertony
e2f47ecdeb
docs: punctuation cleanup
...
See #5538
2021-10-20 22:56:19 +02:00
albertony
41876dd669
touch: add support for touching files in directory, with options for recursive, filtering and dry-run/interactive
...
Fixes #5301
2021-10-20 19:24:57 +02:00
Nick Craig-Wood
54da6154c4
operations: fix lsjson --stat on root directories of bucket based backends
2021-10-16 10:38:24 +01:00
Nick Craig-Wood
edf8978d15
operations: fix HashSum tests after removing ERROR and UNSUPPORTED
...
This was caused by
7a1cab57b6
2021-10-15 10:51:08 +01:00
Nick Craig-Wood
f529c02446
lsjson: add --stat flag and operations/stat api
...
This enables information about single files to be efficiently
retrieved.
2021-10-14 17:15:50 +01:00
Ivan Andreev
16fb608bee
hashsum: treat hash values in sum file as case insensitive
...
Also warn duplicate file paths in sum files.
Fixes https://forum.rclone.org/t/rclone-check-sum/25566/45
2021-10-13 18:21:34 +03:00
