Commit Graph

4479 Commits

Author SHA1 Message Date
Gary Kim
763944f673
rcd: fix incorrect prometheus metrics - fixes #4341
This was caused by using the stats group from the context passed in by the rcd
rather than the global stats group.

Signed-off-by: Gary Kim <gary@garykim.dev>
2020-06-14 10:09:24 +01:00
Vincent Feltz
f4d7e41f24 s3: add Scaleway provider - fixes #4338 2020-06-13 11:55:37 +01:00
Zac Rubin
f9306218f8 sftp: Fix SSH key PEM loading
For SSH authentication, `key_pem` should both override `key_file`
and not require other SSH authentication methods to be set.

Prior to this fix, rclone would attempt to use an ssh-agent
when `key_pem` was the only SSH authentication method set.

Fixes #4240
2020-06-12 22:46:33 +01:00
Nick Craig-Wood
fb06427c69 sync: fix --track-renames-strategy modtime
Before this change `--track-renames-strategy` was broken. The hashing
method it used could declare times that were very close together to be
different.

The time hash was discarded and instead we check the modification time
window on every hash match.

Provided that the user doesn't use `--track-renames-strategy` on a
huge number of identically sized files this will perform just fine.

See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992/5
2020-06-12 15:38:35 +01:00
Nick Craig-Wood
93bd601149 touch: add ability to set nanosecond resolution times 2020-06-12 15:38:35 +01:00
Nick Craig-Wood
848c5b78e1 drive: fix not being able to delete a directory with a trashed shortcut
When we resolve the shortcut we now propagate the trashed status of
the shortcut into the resolved item which fixes the issue.
2020-06-12 15:10:35 +01:00
buengese
84d5df3c84 jottacloud: bring back legacy authentification for use with whitelabel versions - fixes #4299 2020-06-12 12:08:27 +02:00
Nick Craig-Wood
63e6d9d2d1 serve webdav,serve restic: Fix flags so they use environment variables
See: https://forum.rclone.org/t/serve-restic-append-only-environment-variable/17050
2020-06-11 19:28:51 +01:00
albertony
6a2b7b97d7
build: Add file properties and icon to Windows executable (fixes #4304) 2020-06-11 09:26:14 +01:00
Chaitanya
d8d19072c5 mount: Add call for unmount all
mount: handle locking through single mutex.
2020-06-10 22:19:34 +01:00
Chaitanya
830ab37371 rc: Add mount list option for listing current mounts 2020-06-10 22:19:34 +01:00
Nick Craig-Wood
7e48ee8758 cache: fix dedupe on caches wrapping drives - fixes #4320
This implements the MergeDirs optional method.
2020-06-10 21:52:52 +01:00
Nick Craig-Wood
d55053098f check: make check do --checkers files concurrently - fixes #4318 2020-06-10 17:20:54 +01:00
Nick Craig-Wood
63cf0b1cdd check: make check command obey --dry-run/-i/--interactive - fixes #4325 2020-06-10 17:20:54 +01:00
Nick Craig-Wood
5866b1b017 bin: test-repeat.sh - script to run tests many times with individual logs 2020-06-10 17:08:31 +01:00
Nick Craig-Wood
8493f3939c bin: not-in-stable.go - script to help with merging fixes to the stable branch 2020-06-10 17:07:43 +01:00
Nick Craig-Wood
095f4e9b9d build: fix docker release build action 2020-06-10 17:00:33 +01:00
Nick Craig-Wood
a1382a03aa Start v1.52.1-DEV development 2020-06-10 16:49:55 +01:00
Nick Craig-Wood
844b903595 docs: promote the use of -i/--interactive and "rclone sync -i" everywhere #1574 2020-06-10 12:33:53 +01:00
Nick Craig-Wood
a3b3e1f646 tree: remove -i shorthand for --noindent as it conflicts with -i/--interactive 2020-06-10 12:33:53 +01:00
Nick Craig-Wood
b23cf58a41 operations: Add skip all, do all, quit operations to --interactive - fixes #3886
This also adds SkipDestructive into all the remaing places --dry-run
was used and adds documentation.
2020-06-10 12:33:53 +01:00
fishbullet
ba5eb230fb operations: interactive mode -i/--interactive for destructive operations #3886 2020-06-10 12:33:53 +01:00
Nick Craig-Wood
2ea15a72bc s3: fix --header-upload - Fixes #4303
Before this change we were setting the headers on the PUT
request for normal and multipart uploads. For normal uploads this caused the error

    403 Forbidden: There were headers present in the request which were not signed

After this fix we set the headers in the object upload request itself
as the s3 SDK expects.

This means that we only support a limited range of headers

- Cache-Control
- Content-Disposition
- Content-Encoding
- Content-Language
- Content-Type
- X-Amz-Tagging
- X-Amz-Meta-

Note for the last of those are for setting custom metadata in the form
"X-Amz-Meta-Key: value".

This now works for multipart uploads and single part uploads

See also #59
2020-06-10 12:28:48 +01:00
Nick Craig-Wood
b5c654a100 lib/structs: factor reflection based structure manipulation into a library 2020-06-10 12:28:48 +01:00
Nick Craig-Wood
6807b0e42f Add Kamil Trzciński to contributors 2020-06-10 12:16:27 +01:00
Cenk Alti
16422a6b78 putio: fix panic on Object.Open #4315 2020-06-10 12:16:09 +01:00
Rob Calistri
b2ded6212b vfs: Change modtime of file before upload to current
Previously files before they get uploaded will inherit the directory modtime.  This changes that to use the current time instead.
2020-06-10 12:10:50 +01:00
Nick Craig-Wood
88df5927f9 vfs: funnel all read/write calls through ReadAt/WriteAt
This is in preparation for partial reads for read/write files
2020-06-09 18:07:41 +01:00
Nick Craig-Wood
8c37262e05 vfs: don't use embedded methods for read/write handles for clarity 2020-06-09 18:07:23 +01:00
Nick Craig-Wood
3c14a893fb asyncreader: Make StopBuffer as well as Abandon and fix confusion in callers 2020-06-09 18:05:12 +01:00
Nick Craig-Wood
05bc19c331 vfs: Remove uneeded locking from read write handle String() 2020-06-09 18:04:50 +01:00
Caleb Case
40fe97e946 backend/tardigrade: Set UserAgent to rclone
This provides two things:

* It gives Storj insight into which uplink clients are using the
  network.
* It facilitate rclone participating in the Tardigrade Open Source
  Partner Program https://tardigrade.io/partner/
2020-06-09 14:20:28 +01:00
Kamil Trzciński
7458d37d2a
s3: add max_upload_parts support - fixes #4159
* s3: add `max_upload_parts` support

This allows to configure a maximum amount of chunks used to upload file:

- Support Scaleway which has a limit of 1k chunks currently
- Reduce a cost on S3 when each request costs some money at the expense of memory used

Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2020-06-08 18:22:34 +01:00
Nick Craig-Wood
c4110780bf lib/file: fix SetSparse on Windows 7 which fixes downloads of files > 250MB
Before this change we passed both lpOverlapped and lpBytesReturned as NULL.

> If lpOverlapped is NULL, lpBytesReturned cannot be NULL. Even when
> an operation produces no output data, and lpOutBuffer can be NULL,
> the DeviceIoControl function makes use of the variable pointed to by
> lpBytesReturned. After such an operation, the value of the variable
> is without meaning.

After this change we set lpBytesReturned to a valid pointer.

See: https://forum.rclone.org/t/errors-when-downloading-any-file-over-250mb-from-google-drive-windows-sparse-files/16889
2020-06-06 13:13:15 +01:00
Nick Craig-Wood
d729004554 Add Roman Kredentser to contributors 2020-06-05 14:51:52 +01:00
Roman Kredentser
c0521791db s3: implement link sharing with PublicLink 2020-06-05 14:51:05 +01:00
Roman Kredentser
55ad1354b6 link: Add --expire and --unlink flags
This adds expire and unlink fields to the PublicLink interface.

This fixes up the affected backends and removes unlink parameters
where they are present.
2020-06-05 14:51:05 +01:00
Nick Craig-Wood
fb61ed8506 b2: Implement server side copy for files > 5GB - fixes #3991
This factors copy out of SetModTime and Copy so it can be called from
both places.

This also reworks all the multipart uploading to use sync.Errgroup and
memory pooling like the other backends. This makes it more memory
efficient and handle errors better.

See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/10
2020-06-05 13:27:53 +01:00
Nick Craig-Wood
4c7f7582fd obscure: write more help as we are referencing it elsewhere 2020-06-05 12:48:28 +01:00
Caleb Case
a4f1f3d4e8 backend/tardigrade: Upgrade uplink to v1.0.7
This fixes a regression in the rclone tests from the v1.0.6 upgrade of
uplink. The failure was due to an improperly converted error resulting
in the wrong type of error.
2020-06-05 10:51:33 +01:00
Nick Craig-Wood
973e3d6a7b backends: make sure backends expand ~ and environment vars in file names they use
See: https://forum.rclone.org/t/relative-path-in-rclone-config-service-account-json/16693
2020-06-03 17:39:08 +01:00
Nick Craig-Wood
b62d08d136 config: set RCLONE_CONFIG_DIR for use in config files and subprocesses
See: https://forum.rclone.org/t/relative-path-in-rclone-config-service-account-json/16693
2020-06-03 17:39:08 +01:00
Nick Craig-Wood
50e31c6636 vfs: fix OS vs Unix path confusion - fixes ChangeNotify on Windows
See: https://forum.rclone.org/t/windows-mount-polling-not-recognising-all-changes-made-by-another-box/16708
2020-06-03 17:05:58 +01:00
Nick Craig-Wood
151f03378f s3: fix upload of single files into buckets without create permission
Before this change, attempting to upload a single file into an s3
bucket which did not have create permission gave AccessDenied: Access
Denied error when it tried to create the bucket.

This was masked until e2bf91452a was
fixed.

This fix marks the bucket as OK if a fetch on an object indicates it
is OK. This stops rclone thinking it has to create the bucket in the
first place.

Fixes #4297
2020-06-02 14:33:21 +01:00
Nick Craig-Wood
26fb9007da build: fix xgo build after go1.14 go.mod update
Before this change xgo was getting added to go.mod - the build then failed with

    go: inconsistent vendoring in /usr/src/rclone:
    github.com/karalabe/xgo@v0.0.0-20191115072854-c5ccff8648a7: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt

This change gets xgo in GOPATH mode to avoid it getting added to go.mod
2020-06-02 13:41:54 +01:00
Nick Craig-Wood
3b20335d2a docs: remove leading slash in page reference in footer when present 2020-06-02 12:09:45 +01:00
Nick Craig-Wood
8d55367a6a build: set user_allow_other in /etc/fuse.conf in the Docker image
This allows non root mounts to use the --allow-other flag

See: https://forum.rclone.org/t/trying-utilize-docker-for-the-first-time-having-some-issues-with-an-rclone-mount-user-allow-other-error-etc-fuse-conf-has-been-updated-to-allow/16393
2020-06-01 21:33:48 +01:00
Nick Craig-Wood
187ee62e3d Add edwardxml to contributors 2020-06-01 21:33:48 +01:00
Nick Craig-Wood
10e2ec1fbb Add Matteo Pietro Dazzi to contributors 2020-06-01 21:33:48 +01:00
edwardxml
83999cd1d1 docs: minor tense, punctuation, brevity and positivity changes for the home page 2020-06-01 16:32:27 +01:00