extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2025-01-05 05:49:33 +01:00
Code Issues Packages Projects Releases Wiki Activity
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
7,598 Commits 183 Branches 114 Tags 339 MiB
Go 98.6%
HTML 0.4%
Shell 0.3%
Python 0.3%
JavaScript 0.2%
Other 0.1%
98e81a6c2b
Go to file
Nick Craig-Wood 98e81a6c2b azureblob: fix data corruption bug #7590 
It was reported that rclone copy occasionally uploaded corrupted data
to azure blob.

This turned out to be a race condition updating the block count which
caused blocks to be duplicated.

This bug was introduced in this commit in v1.64.0 and will be fixed in v1.65.2

0427177857 azureblob: implement OpenChunkWriter and multi-thread uploads #7056

This race only seems to happen if `--checksum` is used but can happen otherwise.

Unfortunately Azure blob does not check the MD5 that we send them so
despite sending incorrect data this corruption is not detected. The
corruption is detected when rclone tries to download the file, so
attempting to copy the files back to local disk will result in errors
such as:

    ERROR : file.pokosuf5.partial: corrupted on transfer: md5 hash differ "XXX" vs "YYY"

This adds a check to test the blocklist we upload is as we expected
which would have caught the problem had it been in place earlier.
2024-01-24 11:46:47 +00:00
.github build(deps): bump actions/cache from 3 to 4 2024-01-24 11:46:47 +00:00
backend azureblob: fix data corruption bug #7590 2024-01-24 11:46:47 +00:00
bin docs: update contributor email 2024-01-05 16:19:19 +00:00
cmd docs: Fix broken test_proxy.py link again 2024-01-08 10:57:13 +00:00
cmdtest cmdtest: fix typos found by codespell 2023-03-24 20:44:25 +00:00
contrib docker volume plugin: add missing fuse3 dependency #6844 2023-03-15 15:57:53 +00:00
docs docs: add bisync to index 2024-01-24 11:46:47 +00:00
fs Start v1.65.2-DEV development 2024-01-08 11:52:14 +00:00
fstest build: add -race flag to integration tester test_all 2024-01-24 11:46:47 +00:00
graphics build: Add file properties and icon to Windows executable (fixes #4304) 2020-06-11 09:26:14 +01:00
lib oauthutil: avoid panic when *token and *ts.token are the same 2024-01-05 16:19:19 +00:00
librclone docs/librclone: the newer and recommended ucrt64 subsystem of msys2 can now be used for building on windows 2024-01-05 16:19:19 +00:00
vfs vfs: fix stale data when using --vfs-cache-mode full 2024-01-24 11:46:47 +00:00
.gitattributes build: add azure pipelines build 2019-08-06 10:31:32 +01:00
.gitignore serve s3: let rclone act as an S3 compatible server 2023-11-16 16:59:55 +00:00
.golangci.yml build: re-enable revive linters 2023-08-29 13:01:15 +01:00
CONTRIBUTING.md Refresh CONTRIBUTING.md 2023-11-19 12:48:27 +00:00
COPYING Initial commit - some small parts working 2012-11-18 17:32:31 +00:00
Dockerfile build: fix docker build on arm/v6 2024-01-05 16:19:19 +00:00
go.mod build(deps): bump github.com/cloudflare/circl from 1.3.6 to 1.3.7 2024-01-24 11:46:10 +00:00
go.sum build(deps): bump github.com/cloudflare/circl from 1.3.6 to 1.3.7 2024-01-24 11:46:10 +00:00
MAINTAINERS.md Add @nielash as bisync maintainer 2023-09-05 04:05:39 -04:00
Makefile Makefile: use POSIX compatible install arguments 2023-11-20 15:01:26 +00:00
MANUAL.html Version v1.65.1 2024-01-08 10:58:32 +00:00
MANUAL.md Version v1.65.1 2024-01-08 10:58:32 +00:00
MANUAL.txt Version v1.65.1 2024-01-08 10:58:32 +00:00
notes.txt docs: spelling: etc. 2020-10-28 18:16:23 +00:00
rclone.1 Version v1.65.1 2024-01-08 10:58:32 +00:00
rclone.go lib: add plugin support 2019-09-29 11:05:10 +01:00
README.md docs: add bisync to index 2024-01-24 11:46:47 +00:00
RELEASE.md build: fix docker build on arm/v6 2024-01-05 16:19:19 +00:00
VERSION Start v1.65.2-DEV development 2024-01-08 11:52:14 +00:00

README.md

rclone logo rclone logo

Website | Documentation | Download | Contributing | Changelog | Installation | Forum

Build Status Go Report Card GoDoc Docker Pulls

Rclone

Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers.

Storage providers

Please see the full list of all storage providers and their features

Virtual storage providers

These backends adapt or modify other storage providers

  • Alias: rename existing remotes 📄
  • Cache: cache remotes (DEPRECATED) 📄
  • Chunker: split large files 📄
  • Combine: combine multiple remotes into a directory tree 📄
  • Compress: compress files 📄
  • Crypt: encrypt files 📄
  • Hasher: hash files 📄
  • Union: join multiple remotes to work together 📄

Features

  • MD5/SHA-1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Bisync (two way) to keep two directories in sync bidirectionally
  • Check mode to check for file hash equality
  • Can sync to and from network, e.g. two different cloud accounts
  • Optional large file chunking (Chunker)
  • Optional transparent compression (Compress)
  • Optional encryption (Crypt)
  • Optional FUSE mount (rclone mount)
  • Multi-threaded downloads to local disk
  • Can serve local or remote files over HTTP/WebDAV/FTP/SFTP/DLNA

Installation & documentation

Please see the rclone website for:

Downloads

License

This is free software under the terms of the MIT license (check the COPYING file included in this package).