mirror of
https://github.com/rclone/rclone.git
synced 2024-12-04 22:34:48 +01:00
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
azure-blobazure-blob-storageazure-filesbackblaze-b2cloud-storagedropboxencryptionftpfuse-filesystemgogolanggoogle-cloud-storagegoogle-driveonedriveopenstack-swiftrclones3sftpsyncwebdav
40111ba5e1
This changes the OpenWriterAt implementation to make client/fd handling atomic. This PR stabilizes the situation of bigger files and multi-threaded uploads. The root cause boils down to the old "fun" property of pclouds fileops API: sessions are bound to TCP connections. This forces us to use a http client with only a single connection underneath. With large files, we reuse the same connection for each chunk. If that connection interrupts (e.g. because we are talking through the internet), all chunks will fail. The probability for latter one increases with larger files. As the point of the whole multi-threaded feature was to speed-up large files in the first place, this change pulls the client creation (and hence connection handling) into each chunk. This should stabilize the situation, as each chunk (and retry) gets its own connection. |
||
|---|---|---|
| .github | ||
| backend | ||
| bin | ||
| cmd | ||
| cmdtest | ||
| contrib | ||
| docs | ||
| fs | ||
| fstest | ||
| graphics | ||
| lib | ||
| librclone | ||
| vfs | ||
| .gitattributes | ||
| .gitignore | ||
| .golangci.yml | ||
| CONTRIBUTING.md | ||
| COPYING | ||
| Dockerfile | ||
| go.mod | ||
| go.sum | ||
| MAINTAINERS.md | ||
| Makefile | ||
| MANUAL.html | ||
| MANUAL.md | ||
| MANUAL.txt | ||
| notes.txt | ||
| rclone.1 | ||
| rclone.go | ||
| README.md | ||
| RELEASE.md | ||
| VERSION | ||
Special thanks to our sponsor:
Warp is a modern, Rust-based terminal with AI built in so you and your team can build great software, faster.
Visit warp.dev to learn more.
Website | Documentation | Download | Contributing | Changelog | Installation | Forum
Rclone
Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers.
Storage providers
- 1Fichier 📄
- Akamai Netstorage 📄
- Alibaba Cloud (Aliyun) Object Storage System (OSS) 📄
- Amazon S3 📄
- ArvanCloud Object Storage (AOS) 📄
- Backblaze B2 📄
- Box 📄
- Ceph 📄
- China Mobile Ecloud Elastic Object Storage (EOS) 📄
- Cloudflare R2 📄
- Citrix ShareFile 📄
- DigitalOcean Spaces 📄
- Digi Storage 📄
- Dreamhost 📄
- Dropbox 📄
- Enterprise File Fabric 📄
- Fastmail Files 📄
- Files.com 📄
- FTP 📄
- GoFile 📄
- Google Cloud Storage 📄
- Google Drive 📄
- Google Photos 📄
- HDFS (Hadoop Distributed Filesystem) 📄
- Hetzner Storage Box 📄
- HiDrive 📄
- HTTP 📄
- Huawei Cloud Object Storage Service(OBS) 📄
- iCloud Drive 📄
- ImageKit 📄
- Internet Archive 📄
- Jottacloud 📄
- IBM COS S3 📄
- IONOS Cloud 📄
- Koofr 📄
- Leviia Object Storage 📄
- Liara Object Storage 📄
- Linkbox 📄
- Linode Object Storage 📄
- Magalu Object Storage 📄
- Mail.ru Cloud 📄
- Memset Memstore 📄
- Mega 📄
- Memory 📄
- Microsoft Azure Blob Storage 📄
- Microsoft Azure Files Storage 📄
- Microsoft OneDrive 📄
- Minio 📄
- Nextcloud 📄
- OVH 📄
- Blomp Cloud Storage 📄
- OpenDrive 📄
- OpenStack Swift 📄
- Oracle Cloud Storage 📄
- Oracle Object Storage 📄
- Outscale 📄
- ownCloud 📄
- pCloud 📄
- Petabox 📄
- PikPak 📄
- Pixeldrain 📄
- premiumize.me 📄
- put.io 📄
- Proton Drive 📄
- QingStor 📄
- Qiniu Cloud Object Storage (Kodo) 📄
- Quatrix 📄
- Rackspace Cloud Files 📄
- RackCorp Object Storage 📄
- rsync.net 📄
- Scaleway 📄
- Seafile 📄
- SeaweedFS 📄
- Selectel Object Storage 📄
- SFTP 📄
- SMB / CIFS 📄
- StackPath 📄
- Storj 📄
- SugarSync 📄
- Synology C2 Object Storage 📄
- Tencent Cloud Object Storage (COS) 📄
- Uloz.to 📄
- Wasabi 📄
- WebDAV 📄
- Yandex Disk 📄
- Zoho WorkDrive 📄
- The local filesystem 📄
Please see the full list of all storage providers and their features
Virtual storage providers
These backends adapt or modify other storage providers
- Alias: rename existing remotes 📄
- Cache: cache remotes (DEPRECATED) 📄
- Chunker: split large files 📄
- Combine: combine multiple remotes into a directory tree 📄
- Compress: compress files 📄
- Crypt: encrypt files 📄
- Hasher: hash files 📄
- Union: join multiple remotes to work together 📄
Features
- MD5/SHA-1 hashes checked at all times for file integrity
- Timestamps preserved on files
- Partial syncs supported on a whole file basis
- Copy mode to just copy new/changed files
- Sync (one way) mode to make a directory identical
- Bisync (two way) to keep two directories in sync bidirectionally
- Check mode to check for file hash equality
- Can sync to and from network, e.g. two different cloud accounts
- Optional large file chunking (Chunker)
- Optional transparent compression (Compress)
- Optional encryption (Crypt)
- Optional FUSE mount (rclone mount)
- Multi-threaded downloads to local disk
- Can serve local or remote files over HTTP/WebDAV/FTP/SFTP/DLNA
Installation & documentation
Please see the rclone website for:
Downloads
License
This is free software under the terms of the MIT license (check the COPYING file included in this package).