mirror of
https://github.com/rclone/rclone.git
synced 2025-05-29 06:28:05 +02:00
269f90c1e4
I discovered that `rclone` always upload in chunks of 16MiB whenever
uploading a file smaller than `--drive-upload-cutoff`. This is
undesirable since the purpose of the flag `--drive-upload-cutoff` is
to *prevent* chunking below a certain file size.
I realized that it wasn't `rclone` forcing the 16MiB chunks. The
`google-api-go-client` forces a chunk size default of
[`googleapi.DefaultUploadChunkSize`](32bf29c2e1/googleapi/googleapi.go (L55-L57))
bytes for resumable type uploads. This means that all requests that
use `*drive.Service` directly for upload without specifying a
`googleapi.ChunkSize` will be forced to use a *`resumable`*
`uploadType` (rather than `multipart`) for files less than
`googleapi.DefaultUploadChunkSize`. This is also noted directly in the
Drive API client documentation [here](https://pkg.go.dev/google.golang.org/api/drive/v3@v0.44.0#FilesUpdateCall.Media).
This fixes the problem by passing `googleapi.ChunkSize(0)` to
`Media()` method calls, which is the only way to disable chunking
completely. This is mentioned in the API docs
[here](https://pkg.go.dev/google.golang.org/api/googleapi@v0.44.0#ChunkSize).
The other alternative would be to pass
`googleapi.ChunkSize(f.opt.ChunkSize)` -- however, I'm *strongly* in
favor of *not* doing this for performance reasons. By not explicitly
passing a `googleapi.ChunkSize(0)`, we effectively allow
[`PrepareUpload()`](https://pkg.go.dev/google.golang.org/api/internal/gensupport@v0.44.0#PrepareUpload)
to create a
[`NewMediaBuffer`](https://pkg.go.dev/google.golang.org/api/internal/gensupport@v0.44.0#NewMediaBuffer)
that copies the original `io.Reader` passed to `Media()` in order to
check that its size is less than `ChunkSize`, which will unnecessarily
consume time and memory.
`minChunkSize` is also changed to be `googleapi.MinUploadChunkSize`,
as it is something specified we have no control over.
Website | Documentation | Download | Contributing | Changelog | Installation | Forum
Rclone
Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.
Storage providers
- 1Fichier 📄
- Alibaba Cloud (Aliyun) Object Storage System (OSS) 📄
- Amazon Drive 📄 (See note)
- Amazon S3 📄
- Backblaze B2 📄
- Box 📄
- Ceph 📄
- Citrix ShareFile 📄
- DigitalOcean Spaces 📄
- Dreamhost 📄
- Dropbox 📄
- Enterprise File Fabric 📄
- FTP 📄
- GetSky 📄
- Google Cloud Storage 📄
- Google Drive 📄
- Google Photos 📄
- HDFS (Hadoop Distributed Filesystem) 📄
- HTTP 📄
- Hubic 📄
- Jottacloud 📄
- IBM COS S3 📄
- Koofr 📄
- Mail.ru Cloud 📄
- Memset Memstore 📄
- Mega 📄
- Memory 📄
- Microsoft Azure Blob Storage 📄
- Microsoft OneDrive 📄
- Minio 📄
- Nextcloud 📄
- OVH 📄
- OpenDrive 📄
- OpenStack Swift 📄
- Oracle Cloud Storage 📄
- ownCloud 📄
- pCloud 📄
- premiumize.me 📄
- put.io 📄
- QingStor 📄
- Rackspace Cloud Files 📄
- Scaleway 📄
- Seafile 📄
- SeaweedFS 📄
- SFTP 📄
- StackPath 📄
- SugarSync 📄
- Tardigrade 📄
- Tencent Cloud Object Storage (COS) 📄
- Wasabi 📄
- WebDAV 📄
- Yandex Disk 📄
- Zoho WorkDrive 📄
- The local filesystem 📄
Please see the full list of all storage providers and their features
Features
- MD5/SHA-1 hashes checked at all times for file integrity
- Timestamps preserved on files
- Partial syncs supported on a whole file basis
- Copy mode to just copy new/changed files
- Sync (one way) mode to make a directory identical
- Check mode to check for file hash equality
- Can sync to and from network, e.g. two different cloud accounts
- Optional large file chunking (Chunker)
- Optional transparent compression (Compress)
- Optional encryption (Crypt)
- Optional FUSE mount (rclone mount)
- Multi-threaded downloads to local disk
- Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna
Installation & documentation
Please see the rclone website for:
Downloads
License
This is free software under the terms of MIT the license (check the COPYING file included in this package).