Nick Craig-Wood 48cdedc97b Re-implement sync routine to work a directory at a time
Multiple directories (up to --checkers worth) are scanned at once.

This uses much less memory than the previous scheme - only the amount
of memory needed to hold an entire directory listing of objects.

For directory based remotes the speed is unchanged.

For bucket based remotes, instead of doing one API call to list the
whole bucket, it does multiple calls, one for each pseudo directory.
However these are done in parallel so in practice this seems to speed
up directory listings.

This replaces the existing sync method as it performs faster and uses
less memory.

The old sync method is available with the temporary --old-sync-method
flag.

Fixes #517
Fixes #439
Fixes #236
Fixes #1067
2017-02-04 10:30:25 +00:00
2017-02-04 10:29:46 +00:00
2017-02-04 10:29:46 +00:00
2017-02-02 21:14:10 +00:00
2016-10-04 11:37:18 +01:00
2016-12-09 17:12:11 +00:00
2017-02-04 10:29:46 +00:00
2017-01-02 15:33:06 +00:00

Logo

Website | Documentation | Contributing | Changelog | Installation | Forum G+

Build Status Windows Build Status GoDoc

Rclone is a command line program to sync files and directories to and from

  • Google Drive
  • Amazon S3
  • Openstack Swift / Rackspace cloud files / Memset Memstore
  • Dropbox
  • Google Cloud Storage
  • Amazon Drive
  • Microsoft One Drive
  • Hubic
  • Backblaze B2
  • Yandex Disk
  • SFTP
  • The local filesystem

Features

  • MD5/SHA1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Check mode to check for file hash equality
  • Can sync to and from network, eg two different cloud accounts
  • Optional encryption (Crypt)
  • Optional FUSE mount

See the home page for installation, usage, documentation, changelog and configuration walkthroughs.

License

This is free software under the terms of MIT the license (check the COPYING file included in this package).

Description
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Readme 340 MiB
Languages
Go 98.5%
Shell 0.4%
Python 0.4%
HTML 0.3%
JavaScript 0.2%
Other 0.1%