37 Commits

Author	SHA1	Message	Date
Tomasz Raganowicz	c1ad0291c6	crypt: Introduce cipherVersion on the object level ... Introduce cipherVersion on the object level, so it can then be used in Size() function to determine the correct decrypted size based on explicit object version; Modify DecryptDataSeek and newDecrypterSeek, to include 'o *Object' parameter, so it can then be passed to newDecrypter where version detection happens based on magic bytes.	2024-10-07 17:12:43 +02:00
Tomasz Raganowicz	0538517cee	crypt: Provide migration path to V2 cipher; Make cipher version handling more strict and explicit; Introduce --crypt-exact-size parameter to support in-place migration from V1 to V2; Update documentation	2024-09-28 18:45:03 +02:00
Tomasz Raganowicz	2266c3cfca	crypt: Implement backend 'decrypt' command which can decrypt remote file locally once we provide its encrypted name and unwrapped content encryption key (CEK). These can be obtained using 'backend show-cek' and 'backend encode' respectively.	2024-09-26 15:19:57 +02:00
Tomasz Raganowicz	a5d6af71d0	crypt: Implement V2 cipher version with MD5 hash support ... In V2 version each file encrypted using separate CEK (wrapped with AES Keywrap RFC3394). Truncation protection uses last byte of nonce to indicate last encryption block. Added MD5 hash support for crypt.go (only if V2 cipher is used), MD5 of plaintext is encrypted and authenticated at the end of file. Added 4 reserved bytes at the beginning for future use. Added 1 hash type byte in the footer to allow upgrade to different than MD5 hash. cipher_version setting determines hash version used for writes. For reads we determine cipher version by reading header magic bytes, that's regardless of cipher_version setting. A Added TestNewEncrypterV2 test to demonstrate the outputs and make sure that outputs match the expected hardcoded values. Modified other tests to satisfy newly added params (e.g. cek). Related to: https://github.com/rclone/rclone/issues/7192	2024-09-26 15:12:37 +02:00
albertony	4454ed9d3b	build: fix gocritic lint issue underef	2024-08-15 22:08:34 +01:00
albertony	c6d3714e73	build: fix gocritic lint issue assignop	2024-08-15 22:08:34 +01:00
Nick Craig-Wood	bb6cfe109d	crypt: fix reading 0 length files ... In an earlier patch d5afcf9e34 crypt: try not to return "unexpected EOF" error This introduced a bug for 0 length files which this fixes which only manifests if the io.Reader returns data and EOF which not all readers do. This was failing in the integration tests.	2023-04-24 16:54:40 +01:00
jladbrook	281a007b1a	crypt: add suffix option to set a custom suffix for encrypted files - fixes #6392	2023-04-20 17:28:13 +01:00
albertony	9172c9b3dd	crypt: reduce allocations ... This changes crypt's use of sync.Pool: Instead of storing slices it now stores pointers pointers fixed sized arrays. This issue was reported by staticcheck: SA6002 - Storing non-pointer values in sync.Pool allocates memory A sync.Pool is used to avoid unnecessary allocations and reduce the amount of work the garbage collector has to do. When passing a value that is not a pointer to a function that accepts an interface, the value needs to be placed on the heap, which means an additional allocation. Slices are a common thing to put in sync.Pools, and they're structs with 3 fields (length, capacity, and a pointer to an array). In order to avoid the extra allocation, one should store a pointer to the slice instead. See: https://staticcheck.io/docs/checks#SA6002	2023-03-26 14:28:15 +02:00
Dimitri Papadopoulos	bfe272bf67	backend: fix typos found by codespell	2023-03-24 11:34:14 +00:00
Nick Craig-Wood	d0810b602a	crypt: add --crypt-pass-bad-blocks to allow corrupted file output	2023-03-22 16:23:37 +00:00
Nick Craig-Wood	d5afcf9e34	crypt: try not to return "unexpected EOF" error ... Before this change the code wasn't taking into account the error io.ErrUnexpectedEOF that io.ReadFull can return properly. Sometimes that error was being returned instead of a more specific and useful error. To fix this, io.ReadFull was replaced with the simpler readers.ReadFill which is much easier to use correctly.	2023-03-22 16:23:37 +00:00
João Henrique Franco	85eb9776bd	crypt: fix typo in comment ... strign -> string	2022-08-22 10:43:54 +02:00
Nick Craig-Wood	6fd9e3d717	build: reformat comments to pass go1.19 vet ... See: https://go.dev/doc/go1.19#go-doc	2022-08-05 16:35:41 +01:00
albertony	fdd2f8e6d2	Error strings should not be capitalized ... Reported by staticcheck 2022.1.2 (v0.3.2) See: staticcheck.io	2022-06-23 23:26:02 +02:00
Max Sum	c217145cae	crypt: add base64 and base32768 filename encoding options #5801	2021-11-15 17:57:02 +00:00
Nick Craig-Wood	e43b5ce5e5	Remove github.com/pkg/errors and replace with std library version ... This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.	2021-11-07 11:53:30 +00:00
Atílio Antônio	c08d48a50d	docs: improve grammar and fix typos (#5361) ... This alters some comments in source files, but is interested mainly in documentation files and help messages.	2021-11-04 12:50:43 +01:00
Dominik Mydlil	3fe2aaf96c	crypt: support timestamped filenames from --b2-versions ... With the file version format standardized in lib/version, `crypt` can now treat the version strings separately from the encrypted/decrypted file names. This allows --b2-versions to work with `crypt`. Fixes #1627 Co-authored-by: Luc Ritchie <luc.ritchie@gmail.com>	2021-04-12 15:59:18 +01:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Dan Hipschman	70f92fd6b3	crypt: small simplification, no functionality change	2020-10-12 17:20:39 +01:00
Martin Michlmayr	041b201abd	doc: fix typos throughout docs and code	2020-05-25 11:23:58 +01:00
Martin Michlmayr	4aee962233	doc: fix typos throughout docs and code	2020-05-20 15:54:51 +01:00
Nick Craig-Wood	1648c1a0f3	crypt: calculate hashes for uploads from local disk ... Before this change crypt would not calculate hashes for files it was uploading. This is because, in the general case, they have to be downloaded, encrypted and hashed which is too resource intensive. However this causes backends which need the hash first before uploading (eg s3/b2 when uploading chunked files) not to have a hash of the file. This causes cryptcheck to complain about missing hashes on large files uploaded via s3/b2. This change calculates hashes for the upload if the upload is coming from a local filesystem. It does this by encrypting and hashing the local file re-using the code used by cryptcheck. For a local disk this is not a lot more intensive than calculating the hash. See: https://forum.rclone.org/t/strange-output-for-cryptcheck/15437 Fixes: #2809	2020-04-22 11:33:48 +01:00
Nick Craig-Wood	44b1a591a8	crypt: get rid of the unused Cipher interface as it obfuscated the code	2020-04-22 11:33:48 +01:00
Fabian Möller	6c0a749a42	crypt: remove checkValidString ... Remove the usage of checkValidString in decryptSegment to allow all paths which can be created by encryptSegment to be decryptable.	2019-09-30 14:05:49 +01:00
Nick Craig-Wood	57d5de6fba	build: fix up package paths after repo move ... git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g' goimports -w `find . -name \*.go`	2019-07-28 18:47:38 +01:00
Aleksandar Jankovic	f78cd1e043	Add context propagation to rclone ... - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.	2019-06-19 11:59:46 +01:00
Fabian Möller	a0d4c04687	backend: fix misspellings	2019-02-07 19:51:03 +01:00
Nick Craig-Wood	d966cef14c	build: fix problems found with unconvert	2019-02-02 11:45:07 +00:00
Garry McNulty	9cb3a68c38	crypt: check for maximum length before decrypting filename ... The EME Transform() method will panic if the input data is larger than 2048 bytes. Fixes #2826	2018-12-19 11:51:44 +00:00
Nick Craig-Wood	ade252f13b	build: fixup code formatting after goimports change	2018-10-13 22:47:12 +01:00
Nick Craig-Wood	e5be471ce0	Use io.SeekStart/End/Current constants now for go1.7+ #2154	2018-04-07 11:42:36 +01:00
Fabian Möller	00adf40f9f	cryptdecode: use Cipher instead of NewFs (#2087) ... * crypt: extract NewCipher out of NewFs * cryptdecode: make use of crypt.NewCipher Fixes #2075	2018-02-25 12:57:14 +01:00
Nick Craig-Wood	c929de9dc4	crypt: Implement RangeOption #1825	2018-02-19 15:45:24 +00:00
Nick Craig-Wood	d0d6b83a7a	fs/accounting: rework to enable accounting to work with crypt and b2 ... This removes the old system of part accounting and replaces it with a system of popping off the accounting reader and wrapping up new ones as necessary. This makes it much easier to carry the context down the chain of wrapped readers and get the limiting as near as possible to the output. This makes the accounting more accurate and the bandwidth limiting smoother. Fixes #2029 and Fixes #1443	2018-02-02 15:14:41 +00:00
Nick Craig-Wood	b8b620f5c2	Move all backends into backend directory	2018-01-12 17:07:38 +00:00