extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-12-22 23:22:08 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,383 Commits 182 Branches 114 Tags 334 MiB
f6b9fdf7c6
Commit Graph

7383 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood
f6b9fdf7c6 build: fix docker beta build running out of space 
This removes some unused SDKs from the build machine to free some
space up before building. It also adds some lines to show the free
space.
 2023-10-11 12:13:55 +01:00
Nick Craig-Wood
c9fe2f75a8 oracleobjectstorage: fix OpenOptions being ignored in uploadMultipart with chunkWriter 2023-10-11 12:13:55 +01:00
Vitor Gomes
340a67c012 s3: fix OpenOptions being ignored in uploadMultipart with chunkWriter 2023-10-11 12:13:55 +01:00
Saleh Dindar
264b3f0c90 vfs: [bugfix] Update dir modification time 
A subtle bug where dir modification time is not updated when the dir already exists
in the cache. It is only noticeable when some clients use dir modification time to
invalidate cache.
 2023-10-11 12:13:55 +01:00
Nick Craig-Wood
a7978cea56 operations: close file in TestUploadFile test so it can be deleted on Windows 2023-10-11 12:13:55 +01:00
Nick Craig-Wood
bebd82c586 b2: reduce default --b2-upload-concurrency to 4 to reduce memory usage 
In v1.63 memory usage in the b2 backend was limited to `--transfers` *
`--b2-chunk-size`

However in v1.64 this was raised to `--transfers` * `--b2-chunk-size`
* `--b2-upload-concurrency`.

The default value for this was accidently set quite high at 16 which
means by default rclone could use up to 6.4GB of memory!

The new default sets a more reasonable (but still high) max memory of 1.6GB.
 2023-10-11 12:13:55 +01:00
Nick Craig-Wood
af02c3b2a7 b2: fix locking window when getting mutipart upload URL 
Before this change, the lock was held while the upload URL was being
fetched from the server.

This meant that any other threads were blocked from getting upload
URLs unecessarily.

It also increased the potential for deadlock.
 2023-10-11 12:13:55 +01:00
Nick Craig-Wood
77dfe5f1fd pacer: fix b2 deadlock by defaulting max connections to unlimited 
Before this change, the maximum number of connections was set to 10.

This means that b2 could deadlock while uploading multipart uploads
due to a lock being held longer than it should have been.
 2023-10-11 12:13:55 +01:00
Nick Craig-Wood
e9a95a78de s3: fix slice bounds out of range error when listing 
In this commit:

5f938fb9ed s3: fix "Entry doesn't belong in directory" errors when using directory markers

We checked that the remote has the prefix and then changed the remote
before removing the prefix. This sometimes causes:

    panic: runtime error: slice bounds out of range [56:55]

The fix is to do the modification of the remote after removing the
prefix.

See: https://forum.rclone.org/t/cryptcheck-panic-runtime-error-slice-bounds-out-of-range/41977
 2023-10-11 12:13:55 +01:00
Nick Craig-Wood
82ca5295f4 docs: fix backend doc generator to not output duplicate config names 
This was always the intention, it was just implemented wrong.

This shortens the s3 docs by 1369 bringing them down to half the size
just about.

Fixes #7325
 2023-10-11 12:13:55 +01:00
Dimitri Papadopoulos Orfanos
9d8a40b813 docs: fix typos found by codespell in docs and code comments 2023-10-11 12:13:55 +01:00
Nick Craig-Wood
12d80c5219 onedrive: fix the configurator to allow /teams/ID in the config 
See: https://forum.rclone.org/t/sharepoint-to-google/41548/
 2023-10-11 12:08:40 +01:00
Nick Craig-Wood
038a87c569 lsjson: make sure we set the global metadata flag too 2023-10-11 12:08:40 +01:00
Nick Craig-Wood
3ef97993ad b2: fix multipart upload: corrupted on transfer: sizes differ XXX vs 0 
Before this change the b2 backend wasn't writing the metadata to the
object properly after a multipart upload.

The symptom of this was that sometimes it would give the error:

    corrupted on transfer: sizes differ XXX vs 0

This was fixed by returning the metadata in the chunk writer and setting it in Update.

See: https://forum.rclone.org/t/multipart-upload-to-b2-sometimes-failing-with-corrupted-on-transfer-sizes-differ/41829
 2023-10-11 12:08:40 +01:00
Nick Craig-Wood
04bba67cd5 azureblob: fix "fatal error: concurrent map writes" 
Before this change, the metadata map could be accessed from multiple
goroutines at once, sometimes causing this error.

This fix adds a global mutex for adjusting the metadata map to make
all accesses safe.

See: https://forum.rclone.org/t/azure-blob-storage-with-vfs-cache-concurrent-map-writes-exception/41686
 2023-10-11 12:08:40 +01:00
dependabot[bot]
29dd29b9f3 build(deps): bump docker/setup-qemu-action from 2 to 3 
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 2 to 3.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-10-11 12:08:40 +01:00
dependabot[bot]
532248352b build(deps): bump docker/setup-buildx-action from 2 to 3 
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2 to 3.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-10-11 12:08:40 +01:00
Kaloyan Raev
ab803942de storj: update storj.io/uplink to v1.12.0 
The improved upload logic is active by default in uplink v1.12.0, so the
`testuplink.WithConcurrentSegmentUploadsDefaultConfig(ctx)` is not
required anymore.

See https://github.com/rclone/rclone/pull/7198
 2023-10-11 12:08:40 +01:00
Nick Craig-Wood
f933e80258 docs: add notes on how to update the website between releases 2023-10-11 12:08:40 +01:00
Nick Craig-Wood
1c6f0101a5 docs: remove minio sponsor box for the moment 2023-10-11 12:08:40 +01:00
Nick Craig-Wood
c6f161de90 docs: update Storj partner link 2023-10-11 12:08:40 +01:00
Herby Gillot
bdcf7fe28c docs: add MacPorts install info 
https://ports.macports.org/port/rclone/
 2023-10-11 12:08:40 +01:00
dependabot[bot]
776dc47eb8 build(deps): bump docker/metadata-action from 4 to 5 
Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 4 to 5.
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Upgrade guide](https://github.com/docker/metadata-action/blob/master/UPGRADE.md)
- [Commits](https://github.com/docker/metadata-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: docker/metadata-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-10-11 12:08:40 +01:00
dependabot[bot]
167046e21a build(deps): bump docker/login-action from 2 to 3 
Bumps [docker/login-action](https://github.com/docker/login-action) from 2 to 3.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-10-11 12:08:40 +01:00
dependabot[bot]
98d50d545a build(deps): bump docker/build-push-action from 4 to 5 
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 4 to 5.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-10-11 12:08:40 +01:00
Manoj Ghosh
48242c5357 fix overview of oracle object storage as it supports multithreaded 2023-10-11 12:08:40 +01:00
Pat Patterson
e437e6c209 operations: ensure concurrency is no greater than the number of chunks - fixes #7299 2023-10-11 12:08:40 +01:00
Nick Craig-Wood
b813a01718 Start v1.64.1-DEV development 2023-10-11 11:55:24 +01:00
Nick Craig-Wood
77f7bb08af Version v1.64.0 2023-09-11 15:59:44 +01:00
Nick Craig-Wood
a5a61f4874 protondrive: make cached keys rclone style and not show with rclone config redacted 2023-09-11 15:57:08 +01:00
Nick Craig-Wood
e8879f3e77 docs: document release signing and verification 
Fixes #7257
 2023-09-11 12:28:23 +01:00
Nick Craig-Wood
2a6675cffd docs: fix typo in rc docs - fixes #7287 2023-09-11 09:34:22 +01:00
Nick Craig-Wood
fa4d171f62 protondrive: complete docs with all references to Proton Drive 2023-09-10 11:57:39 +01:00
Nick Craig-Wood
d4d530bd8e drive: add --drive-fast-list-bug-fix to control ListR bug workaround 
See: https://forum.rclone.org/t/how-to-list-empty-directories-recursively/40995/12
 2023-09-09 17:46:03 +01:00
Nick Craig-Wood
f4b011e4e4 s3: add rclone backend restore-status command 
This command shows the restore status of objects being retrieved from GLACIER.

See: https://forum.rclone.org/t/aws-s3-glacier-monitor-restore-status-command-for-glacier-restoring-process/41373/7
 2023-09-09 17:44:36 +01:00
Nick Craig-Wood
d80890bf32 Add Drew Stinnett to contributors 2023-09-09 17:44:36 +01:00
Nick Craig-Wood
39392d70dd Add David Pedersen to contributors 2023-09-09 17:44:36 +01:00
Drew Stinnett
643386f026 rc: Add operations/settier to API 2023-09-09 17:41:02 +01:00
Chun-Hung Tseng
ed755bf04f
protondrive: implement two-password mode (#7279) 2023-09-08 22:54:46 +02:00
David Pedersen
071c3f28e5 vfs: Update parent directory modtimes on vfs actions 
This isn't written back to the storage so might change on remount but
makes the VFS just a little more POSIX compatible.
 2023-09-08 17:19:52 +01:00
Nick Craig-Wood
7453b7d5f3 hdfs: fix retry "replication in progress" errors when uploading 
Before this change uploaded files could return the error "replication
in progress".

This error is harmless though and means the Close should be retried
which is what this patch does.
 2023-09-08 15:35:50 +01:00
Nick Craig-Wood
c9350149d8 hdfs: fix uploading to the wrong object on Update with overriden remote name 
In this commit we discovered a problem with objects being uploaded to
the incorrect object name. It added an integration test for the
problem.

65b2e378e0 drive: fix incorrect remote after Update on object

This test was tripped by the hdfs backend and this patch fixes the
problem.
 2023-09-08 15:35:50 +01:00
Nick Craig-Wood
08789a5815 test_all: remove filefabric from integration tests 
The filefabric test server doesn't seem to be working at all so remove
it from the integration tests pending revitalization.
 2023-09-08 15:35:50 +01:00
Nick Craig-Wood
4037af9c1a Add Oksana and Volodymyr Kit to contributors 2023-09-08 15:35:50 +01:00
Oksana
628ff8e524
quatrix: add backend to support Quatrix 
Co-authored-by: Volodymyr Kit <v.kit@maytech.net>
 2023-09-08 14:31:29 +01:00
Chun-Hung Tseng
578c75cb1e
protondrive: fix signature verification logic by accounting for legacy signing scheme (#7278) 2023-09-08 16:00:34 +08:00
Nick Craig-Wood
63ab250817 vfs: add --vfs-cache-min-free-space to control minimum free space on the disk containing the cache 
See: https://forum.rclone.org/t/rclone-fails-to-control-disk-usage-and-its-filling-the-disk-to-100/41494/
 2023-09-07 15:57:45 +01:00
Nick Craig-Wood
39f910a65d rc: add core/du to measure local disk usage 2023-09-07 15:57:45 +01:00
Nick Craig-Wood
0fb36562dd Add lib/diskusage to measure used/free on disks 2023-09-07 15:57:45 +01:00
Nick Craig-Wood
8c25a15a40 Add zjx20 to contributors 2023-09-07 15:57:45 +01:00