13 Commits

Author	SHA1	Message	Date
Nick Craig-Wood	93955b755f	operations: fix retries downloading too much data with certain backends ... Before this fix if more than one retry happened on a file that rclone had opened for read with a backend that uses fs.FixRangeOption then rclone would read too much data and the transfer would fail. Backends affected: - azureblob, azurefiles, b2, box, dropbox, fichier, filefabric - googlecloudstorage, hidrive, imagekit, jottacloud, koofr, netstorage - onedrive, opendrive, oracleobjectstorage, pikpak, premiumizeme - protondrive, qingstor, quatrix, s3, sharefile, sugarsync, swift - uptobox, webdav, zoho This was because rclone was emitting Range requests for the wrong data range on the second and subsequent retries. This was caused by fs.FixRangeOption modifying the options and the reopen code relying on them not being modified. This fix makes a copy of the fs.FixRangeOption in the reopen code to fix the problem. In future it might be best to change fs.FixRangeOption so it returns a new options slice. Fixes #7759	2024-04-13 19:25:15 +01:00
Nick Craig-Wood	c0fb9ebfce	operations: make Open() return an io.ReadSeekCloser #7350 ... As part of reducing memory usage in rclone, we need to have a raw handle to an object we can seek with.	2023-11-20 18:07:05 +00:00
Nick Craig-Wood	1f9c962183	operations: reopen downloads on error when using check --download and cat ... Before this change, some parts of operations called the Open method on objects directly, and some called NewReOpen to make an object which can re-open itself on errors. This adds a new function operations.Open which should be called instead of fs.Object.Open to open a reliable stream of data and changes all call sites to use that. This means `rclone check --download` and `rclone cat` will re-open files on failures. See: https://forum.rclone.org/t/does-rclone-support-retries-for-check-when-using-download-flag/38641	2023-06-10 14:42:29 +01:00
Nick Craig-Wood	6fd9e3d717	build: reformat comments to pass go1.19 vet ... See: https://go.dev/doc/go1.19#go-doc	2022-08-05 16:35:41 +01:00
albertony	0772cae314	staticcheck: use result of type assertion to simplify cases	2022-07-04 11:24:59 +02:00
Nick Craig-Wood	e43b5ce5e5	Remove github.com/pkg/errors and replace with std library version ... This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.	2021-11-07 11:53:30 +00:00
Tim Gallant	9bf3d3da4c	fs: add UploadHeaders, DownloadHeaders to Update/Put/Open options	2020-04-23 11:07:21 +01:00
Nick Craig-Wood	7f15cc9556	operations: make ReOpen and NewReOpen public for re-use elsewhere	2020-04-19 15:18:49 +01:00
Nick Craig-Wood	4537d9b5cf	operations: make reopen code error on NoLowLevelRetry errors - fixes #3777	2019-12-06 10:54:03 +00:00
Nick Craig-Wood	57d5de6fba	build: fix up package paths after repo move ... git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g' goimports -w `find . -name \*.go`	2019-07-28 18:47:38 +01:00
Aleksandar Jankovic	f78cd1e043	Add context propagation to rclone ... - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.	2019-06-19 11:59:46 +01:00
Nick Craig-Wood	0655738da6	operations: re-work reopen framework so it can take a RangeOption #2252 ... This is in preparation for multipart downloads.	2019-05-11 23:35:19 +01:00
Nick Craig-Wood	076d3da825	operations: resume downloads if the reader fails in copy - fixes #2108 ... This puts a shim on the reader opened by Copy so that if an error is returned, the reader is re-opened at the correct seek point. This should make downloading very large files more reliable.	2019-02-12 11:47:57 +00:00