19 Commits

Author	SHA1	Message	Date
Nick Craig-Wood	4f8ee736b1	vfs: make cache dir absolute before using it to fix path too long errors ... If --cache-dir is passed in as a relative path, then rclone will not be able to turn it into a UNC path under Windows, which means that file names longer than 260 chars will fail when stored in the cache. This patch makes the --cache-dir path absolute before using it. See: https://forum.rclone.org/t/handling-of-long-paths-on-windows-260-characters/20913	2020-12-11 10:00:51 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Leo Luan	c5c56cda02	vfs: Add a missed update of used cache space ... The missed update can cause incorrect before-cleaning cache stats and a pre-mature condition broadcast in purgeOld before the cache space use is reduced below the quota.	2020-10-06 16:35:23 +01:00
Leo Luan	2295123cad	vfs: Add exponential backoff during ENOSPC retries ... Add an exponentially increasing delay during retries up ENOSPC error to avoid exhausting the 10 retries too soon when the cache space recovery from item resets is not available from the file system yet or consumed by other large cache writes.	2020-10-06 16:35:23 +01:00
Leo Luan	64d736a57b	vfs: Fix a race condition in retryFailedResets ... A failed item reset is saved in the errItems for retryFailedResets to process. If the item gets closed before the retry, the item may have been removed from the c.item array. Previous code did not account for this condition. This patch adds the check for the exitence of the retry items in retryFailedResets.	2020-10-06 16:35:23 +01:00
Hekmon	66def93373	mount cmd: update systemd status with cache stats	2020-10-06 16:21:30 +01:00
Sam Edwards	23b2c58018	vfs: Quiet removeNotInUse logging to debug when not removing	2020-09-02 11:55:20 +01:00
Leo Luan	c665201b85	vfs: support synchronous cache space recovery upon ENOSPC ... This patch provides the support of synchronous cache space recovery to allow read threads to recover from ENOSPC errors when cache space can be recovered from cache items that are not in use or safe to be reset/emptied . The patch complements the existing cache cleaning process in two ways. Firstly, the existing cache cleaning process is time-driven that runs periodically. The cache space can run out while the cache cleaner thread is still waiting for its next scheduled run. The io threads encountering ENOSPC return an internal error to the applications in this case even when cache space can be recovered to avoid this error. This patch addresses this problem by having the read threads kick the cache cleaner thread in this condition to recover cache space preventing unnecessary ENOSPC errors from being seen by the applications. Secondly, this patch enhances the cache cleaner to support cache item reset. Currently the cache purge process removes cache items that are not in use. This may not be sufficient when the total size of the working set exceeds the cache directory's capacity. Like in the current code, this patch starts the purge process by removing cache files that are not in use. Cache items whose access times are older than vfs-cache-max-age are removed first. After that, other not-in-use items are removed in LRU order until vfs-cache-max-size is reached. If the vfs-cache-max-size (the quota) is still not reached at this time, this patch adds a cache reset step to reset/empty cache files that are still in use but not dirtied. This enables application processes to continue without seeing an error even when the working set depletes the cache space as long as there is not a large write working set hoarding the entire cache space. By design this patch does not add ENOSPC error recovery for write IOs. Rclone does not empty a write cache item until the file data is written back to the backend upon close. Allowing more cache space to be consumed by dirty cache items when the cache space is already running low would increase the risk of exhausting the cache space in a way that the vfs mount becomes unreadable.	2020-08-25 21:12:06 +01:00
Nick Craig-Wood	b2f4f52b64	vfs cache: make logging consistent and remove some debug logging	2020-07-06 17:32:53 +01:00
Nick Craig-Wood	15402e46c9	vfs: Add recovered items on cache reload to directory listings ... Before this change, if we restarted an upload after a restart then the file would get uploaded but never added to the directory listings. This change makes sure we add virtual items to the directory cache when reloading the cache so that they show up properly.	2020-06-30 12:03:39 +01:00
Nick Craig-Wood	ee04732cbb	vfs: factor writeback and downloaders into their own packages	2020-06-30 12:03:39 +01:00
Nick Craig-Wood	fa472a340e	vfs: fix writeback deadlocks and other bugs ... - fix deadlock when cancelling upload - fix double upload and panic after cancelled upload - fix cancelation strategy of uploading files - don't cancel uploads if we don't modify the file - cancel uploads if we do modify the file - fix deadlock between Item and writeback - fix confusion about whether writeback item was being uploaded - fix cornercases in cancelling uploads and removing files	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	58a7faa281	vfs: Make tests run reliably ... On file Remove - cancel any writebacks in progress - ignore error message deleting non existent file if file was in the process of being uploaded Writeback - Don't transfer the file if it has disappeared in the meantime - Take our own copy of the file name to avoid deadlocks - Fix delayed retry logic - Wait for upload to finish when cancelling upload Fix race condition in item saving Fix race condition in vfscache test Make sure we delete the file on the error path - this makes cascading failures much less likely	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	496a87a665	vfs: restart pending uploads on restart of the cache	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	e4e53a2e61	vfs: add --vfs-writeback option to delay writes back to cloud storage ... This is enabled by default and can be disabled with --vfs-writeback 0	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	28255f1bac	vfs: fix errors when using > 260 char files in the cache in Windows ... This makes the cache use UNC paths on Windows. This stops the cache exploding when using > 260 character paths	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	917cb4acb3	vfs: implement partial reads in --vfs-cache-mode full ... This allows reads to only read part of the file and it keeps on disk a cache of what parts of each file have been loaded. File data itself is kept in sparse files.	2020-06-30 12:01:36 +01:00