extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-11-08 01:25:14 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,011 Commits 178 Branches 113 Tags 301 MiB
537b62917f
Commit Graph

77 Commits

Author SHA1 Message Date
Nick Craig-Wood
b91c349cd5 local: fix hash invalidation which caused errors with local crypt mount 
Before this fix if a file was updated, but to the same length and
timestamp then the local backend would return the wrong (cached)
hashes for the object.

This happens regularly on a crypted local disk mount when the VFS
thinks files have been changed but actually their contents are
identical to that written previously. This is because when files are
uploaded their nonce changes so the contents of the file changes but
the timestamp and size remain the same because the file didn't
actually change.

This causes errors like this:

    ERROR: file: Failed to copy: corrupted on transfer: md5 crypted
    hash differ "X" vs "Y"

This turned out to be because the local backend wasn't clearing its
cache of hashes when the file was updated.

This fix clears the hash cache for Update and Remove.

It also puts a src and destination in the crypt message to make future
debugging easier.

Fixes #4031
 2021-11-24 12:09:34 +00:00
Sinan Tan
8c61a09be2 crypt: add test cases and documentation for base64 and base32768 filename encoding #5801 2021-11-15 17:57:02 +00:00
Max Sum
c217145cae crypt: add base64 and base32768 filename encoding options #5801 2021-11-15 17:57:02 +00:00
Nick Craig-Wood
e43b5ce5e5 Remove github.com/pkg/errors and replace with std library version 
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.

This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
 2021-11-07 11:53:30 +00:00
Atílio Antônio
c08d48a50d
docs: improve grammar and fix typos (#5361) 
This alters some comments in source files, but is interested mainly in documentation files and help messages.
 2021-11-04 12:50:43 +01:00
albertony
e2f47ecdeb docs: punctuation cleanup 
See #5538
 2021-10-20 22:56:19 +02:00
Nick Craig-Wood
1cb31e8cc7 crypt: fix uploads with --crypt-no-data-encryption 
Before this change, when uploading to a crypt, the ObjectInfo
accidentally used the encrypted size, not the unencrypted size when
--crypt-no-data-encryption was set.

Fixes #5498
 2021-10-12 17:12:41 +01:00
Ivan Andreev
7190c058a7 crypt: return wrapped object even with no-data-encryption 
In presence of no_data_encryption the Crypt's Put method used to over-optimize
and returned base object. This patch makes it return Crypt-wrapped object now.

Fixes #5498
 2021-10-12 00:41:12 +03:00
Jonta
d8d621c175
docs: grammar/readability (#5633) 2021-09-30 15:34:00 +02:00
Dominik Mydlil
3fe2aaf96c crypt: support timestamped filenames from --b2-versions 
With the file version format standardized in lib/version, `crypt` can
now treat the version strings separately from the encrypted/decrypted
file names. This allows --b2-versions to work with `crypt`.

Fixes #1627

Co-authored-by: Luc Ritchie <luc.ritchie@gmail.com>
 2021-04-12 15:59:18 +01:00
albertony
12cd322643
crypt: log hash ok on upload 2021-03-23 18:36:51 +01:00
Vesnyx
4ea7c7aa47
crypt: add option to not encrypt data #1077 (#2981) 
Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
 2021-02-17 03:40:37 +03:00
Nick Craig-Wood
47aada16a0 fs: add Shutdown optional method for backends 2020-11-27 17:35:01 +00:00
Nick Craig-Wood
8b96933e58 fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
d846210978 fs: Add context to NewFs #3257 #4685 
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
 2020-11-09 18:05:54 +00:00
Josh Soref
e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Dan Hipschman
70f92fd6b3 crypt: small simplification, no functionality change 2020-10-12 17:20:39 +01:00
Nick Craig-Wood
70c8566cb8 fs: Pin created backends until parents are finalized 
This attempts to solve the backend lifecycle problem by

- Pinning backends mentioned on the command line into the cache
  indefinitely

- Unpinning backends when the containing structure (VFS, wrapping
  backend) is destroyed

See: https://forum.rclone.org/t/rclone-rc-backend-command-not-working-as-expected/18834
 2020-09-01 18:21:03 +01:00
Nick Craig-Wood
0d066bdf46 alias,cache,chunker,crypt: make any created backends be cached to fix rc problems 
Before this change, when the above backends created a new backend they
didn't put it into the backend cache.

This meant that rc commands acting on those backends did not work.

This was fixed by making sure the backends use the backend cache.

See: https://forum.rclone.org/t/rclone-rc-backend-command-not-working-as-expected/18834
 2020-09-01 18:21:03 +01:00
Nick Craig-Wood
3f0d54daae crypt: fix purge bug introduced by refactor #1891 
In this commit

a2afa9aadd fs: Add directory to optional Purge interface

We failed to encrypt the directory name so the Purge failed.

This was spotted by the integration tests.
 2020-09-01 15:16:14 +01:00
Nick Craig-Wood
a2afa9aadd fs: Add directory to optional Purge interface - fixes #1891 
- add a directory to the optional Purge interface
- fix up all the backends
- add an additional integration test to test for the feature
- use the new feature in operations.Purge

Many of the backends had been prepared in advance for this so the
change was trivial for them.
 2020-07-31 17:43:17 +01:00
Nick Craig-Wood
61ff7306ae crypt: add --crypt-server-side-across-configs flag 
This can be used for changing filename encryption mode without
re-uploading data.

See: https://forum.rclone.org/t/revert-filename-encryption-method/17454/
 2020-06-27 11:40:15 +01:00
Roman Kredentser
55ad1354b6 link: Add --expire and --unlink flags 
This adds expire and unlink fields to the PublicLink interface.

This fixes up the affected backends and removes unlink parameters
where they are present.
 2020-06-05 14:51:05 +01:00
Martin Michlmayr
041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Martin Michlmayr
4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Nick Craig-Wood
52b7337d28 crypt: change backend encode/decode to output a plain list 
This commit changes the output of the rclone backend encode crypt: and
decode commands to output a plain list of decoded or encoded file
names.

This makes the command much more useful for command line scripting.
 2020-05-13 18:11:45 +01:00
Anagh Kumar Baranwal
856c2b565f crypt: Added decode/encode commands to replicate functionality of cryptdecode 
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
 2020-05-04 16:23:33 +00:00
Nick Craig-Wood
1648c1a0f3 crypt: calculate hashes for uploads from local disk 
Before this change crypt would not calculate hashes for files it was
uploading. This is because, in the general case, they have to be
downloaded, encrypted and hashed which is too resource intensive.

However this causes backends which need the hash first before
uploading (eg s3/b2 when uploading chunked files) not to have a hash
of the file. This causes cryptcheck to complain about missing hashes
on large files uploaded via s3/b2.

This change calculates hashes for the upload if the upload is coming
from a local filesystem. It does this by encrypting and hashing the
local file re-using the code used by cryptcheck. For a local disk this
is not a lot more intensive than calculating the hash.

See: https://forum.rclone.org/t/strange-output-for-cryptcheck/15437
Fixes: #2809
 2020-04-22 11:33:48 +01:00
Nick Craig-Wood
44b1a591a8 crypt: get rid of the unused Cipher interface as it obfuscated the code 2020-04-22 11:33:48 +01:00
Nick Craig-Wood
cd3c699f28 lib/readers: factor ErrorReader from multiple sources 2020-04-19 15:18:49 +01:00
Nick Craig-Wood
7a5a74cecb crypt: clarify that directory_name_encryption depends on filename_encryption 
See: https://forum.rclone.org/t/directory-name-encryption-is-set-to-always-false-when-choosing-filename-encryption-off/14600
 2020-02-28 16:26:45 +00:00
buengese
9c858c3228 crypt: correctly handle trailing dot 2020-01-22 01:40:04 +01:00
Thomas Eales
42de601fa6 crypt: reorder the filename encryption options 
This brings the default `standard` to the top of the list to replace `off`
 2020-01-12 14:23:35 +00:00
Nick Craig-Wood
3f7af64316 config: give config questions default values - fixes #3672 2019-11-05 11:53:44 +00:00
Fabian Möller
6c0a749a42 crypt: remove checkValidString 
Remove the usage of checkValidString in decryptSegment to allow all
paths which can be created by encryptSegment to be decryptable.
 2019-09-30 14:05:49 +01:00
Nick Craig-Wood
ffa1dac10b build: apply gofmt from go1.13 to change case of number literals 2019-09-05 13:59:06 +01:00
Nick Craig-Wood
3ebcb555f4 fs: add optional features UserInfo and Disconnect 2019-08-12 13:49:23 +01:00
Nick Craig-Wood
57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Aleksandar Jankovic
f78cd1e043 Add context propagation to rclone 
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
 2019-06-19 11:59:46 +01:00
Nick Craig-Wood
99bc013c0a crypt: remove stray debug in ChangeNotify 2019-05-12 16:50:03 +01:00
Nick Craig-Wood
7c4fe3eb75 local: define OpenWriterAt interface and test and implement it #2252 
This will enable multipart downloads in future commits
 2019-05-11 23:35:19 +01:00
Nick Craig-Wood
eeab7a0a43 crypt: Implement Optional methods SetTier, GetTier - fixes #2895 
This implements optional methods on Object
- ID
- SetTier
- GetTier

And declares that it will not implement MimeType for the FsCheck test.
 2019-04-16 13:33:10 +01:00
Nick Craig-Wood
ac8d1db8d3 crypt: support PublicLink (rclone link) of underlying backend - fixes #3042 2019-04-16 13:33:10 +01:00
Nick Craig-Wood
cdf12b1fc8 crypt: Fix wrapping of ChangeNotify to decrypt directories properly 
Also change the way it is added to make the FsCheckWrap test pass
 2019-04-16 13:33:10 +01:00
Nick Craig-Wood
7981e450a4 crypt: make rclone dedupe work through crypt 
Implement these optional methods:

- WrapFs
- SetWrapper
- MergeDirs
- DirCacheFlush

Fixes #2233 Fixes #2689
 2019-04-16 13:33:10 +01:00
Fabian Möller
a0d4c04687
backend: fix misspellings 2019-02-07 19:51:03 +01:00
Nick Craig-Wood
d966cef14c build: fix problems found with unconvert 2019-02-02 11:45:07 +00:00
Garry McNulty
9cb3a68c38 crypt: check for maximum length before decrypting filename 
The EME Transform() method will panic if the input data is larger than
2048 bytes.

Fixes #2826
 2018-12-19 11:51:44 +00:00