mirror of https://github.com/rclone/rclone.git synced 2024-12-23 15:38:57 +01:00

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

azure-blob azure-blob-storage azure-files backblaze-b2 cloud-storage dropbox encryption ftp fuse-filesystem go golang google-cloud-storage google-drive onedrive openstack-swift rclone s3 sftp sync webdav

Go to file

nielash 4025f42bd9 bisync: Graceful Shutdown, --recover from interruptions without --resync - fixes #7470 Before this change, bisync had no mechanism to gracefully cancel a sync early and exit in a clean state. Additionally, there was no way to recover on the next run -- any interruption at all would cause bisync to require a --resync, which made bisync more difficult to use as a scheduled background process. This change introduces a "Graceful Shutdown" mode and --recover flag to robustly recover from even un-graceful shutdowns. If --recover is set, in the event of a sudden interruption or other un-graceful shutdown, bisync will attempt to automatically recover on the next run, instead of requiring --resync. Bisync is able to recover robustly by keeping one "backup" listing at all times, representing the state of both paths after the last known successful sync. Bisync can then compare the current state with this snapshot to determine which changes it needs to retry. Changes that were synced after this snapshot (during the run that was later interrupted) will appear to bisync as if they are "new or changed on both sides", but in most cases this is not a problem, as bisync will simply do its usual "equality check" and learn that no action needs to be taken on these files, since they are already identical on both sides. In the rare event that a file is synced successfully during a run that later aborts, and then that same file changes AGAIN before the next run, bisync will think it is a sync conflict, and handle it accordingly. (From bisync's perspective, the file has changed on both sides since the last trusted sync, and the files on either side are not currently identical.) Therefore, --recover carries with it a slightly increased chance of having conflicts -- though in practice this is pretty rare, as the conditions required to cause it are quite specific. This risk can be reduced by using bisync's "Graceful Shutdown" mode (triggered by sending SIGINT or Ctrl+C), when you have the choice, instead of forcing a sudden termination. --recover and --resilient are similar, but distinct -- the main difference is that --resilient is about _retrying_, while --recover is about _recovering_. Most users will probably want both. --resilient allows retrying when bisync has chosen to abort itself due to safety features such as failing --check-access or detecting a filter change. --resilient does not cover external interruptions such as a user shutting down their computer in the middle of a sync -- that is what --recover is for. "Graceful Shutdown" mode is activated by sending SIGINT or pressing Ctrl+C during a run. Once triggered, bisync will use best efforts to exit cleanly before the timer runs out. If bisync is in the middle of transferring files, it will attempt to cleanly empty its queue by finishing what it has started but not taking more. If it cannot do so within 30 seconds, it will cancel the in-progress transfers at that point and then give itself a maximum of 60 seconds to wrap up, save its state for next time, and exit. With the -vP flags you will see constant status updates and a final confirmation of whether or not the graceful shutdown was successful. At any point during the "Graceful Shutdown" sequence, a second SIGINT or Ctrl+C will trigger an immediate, un-graceful exit, which will leave things in a messier state. Usually a robust recovery will still be possible if using --recover mode, otherwise you will need to do a --resync. If you plan to use Graceful Shutdown mode, it is recommended to use --resilient and --recover, and it is important to NOT use --inplace, otherwise you risk leaving partially-written files on one side, which may be confused for real files on the next run. Note also that in the event of an abrupt interruption, a lock file will be left behind to block concurrent runs. You will need to delete it before you can proceed with the next run (or wait for it to expire on its own, if using --max-lock.)		2024-01-20 16:31:28 -05:00
.github	build(deps): bump actions/cache from 3 to 4	2024-01-19 17:19:08 +00:00
backend	local: fix cleanRootPath on Windows after go1.21.4 stdlib update	2024-01-20 14:50:08 -05:00
bin	amazonclouddrive: remove Amazon Drive backend code and docs #7539	2024-01-04 17:05:54 +00:00
cmd	bisync: Graceful Shutdown, --recover from interruptions without --resync - fixes #7470	2024-01-20 16:31:28 -05:00
cmdtest	cmdtest: fix typos found by codespell	2023-03-24 20:44:25 +00:00
contrib	docker volume plugin: add missing fuse3 dependency #6844	2023-03-15 15:57:53 +00:00
docs	bisync: Graceful Shutdown, --recover from interruptions without --resync - fixes #7470	2024-01-20 16:31:28 -05:00
fs	bisync: full support for comparing checksum, size, modtime - fixes #5679 fixes #5683 fixes #5684 fixes #5675	2024-01-20 16:08:06 -05:00
fstest	build: remove random.Seed since random generator is seeded automatically in go1.20	2024-01-15 16:22:07 +00:00
graphics	build: Add file properties and icon to Windows executable (fixes #4304 )	2020-06-11 09:26:14 +01:00
lib	build: remove random.Seed since random generator is seeded automatically in go1.20	2024-01-15 16:22:07 +00:00
librclone	docs/librclone: the newer and recommended ucrt64 subsystem of msys2 can now be used for building on windows	2024-01-01 21:56:45 +01:00
vfs	build: remove random.Seed since random generator is seeded automatically in go1.20	2024-01-15 16:22:07 +00:00
.gitattributes	build: add azure pipelines build	2019-08-06 10:31:32 +01:00
.gitignore	serve s3: let rclone act as an S3 compatible server	2023-11-16 16:59:55 +00:00
.golangci.yml	build: re-enable revive linters	2023-08-29 13:01:15 +01:00
CONTRIBUTING.md	Refresh CONTRIBUTING.md	2023-11-19 12:48:27 +00:00
COPYING	Initial commit - some small parts working	2012-11-18 17:32:31 +00:00
Dockerfile	build: fix docker build on arm/v6	2024-01-03 17:43:23 +00:00
go.mod	operations: add logger to log list of sync results -- fixes #7282	2024-01-20 14:50:08 -05:00
go.sum	operations: add logger to log list of sync results -- fixes #7282	2024-01-20 14:50:08 -05:00
MAINTAINERS.md	Add @nielash as bisync maintainer	2023-09-05 04:05:39 -04:00
Makefile	build: use API when fetching golangci-lint as it is more reliable	2024-01-15 16:22:07 +00:00
MANUAL.html	Version v1.65.0	2023-11-26 16:07:39 +00:00
MANUAL.md	Version v1.65.0	2023-11-26 16:07:39 +00:00
MANUAL.txt	Version v1.65.0	2023-11-26 16:07:39 +00:00
notes.txt	docs: spelling: etc.	2020-10-28 18:16:23 +00:00
rclone.1	Version v1.65.0	2023-11-26 16:07:39 +00:00
rclone.go	lib: add plugin support	2019-09-29 11:05:10 +01:00
README.md	docs: add bisync to index	2024-01-20 14:50:08 -05:00
RELEASE.md	build: fix docker build on arm/v6	2024-01-03 17:43:23 +00:00
VERSION	Start v1.66.0-DEV development	2023-11-26 17:14:38 +00:00