Commit Graph

927 Commits

Author SHA1 Message Date
Nick Craig-Wood
101f82c6b3 drive: drop "Disabling ListR" messages down to debug
This was causing unecessary anguish for users since these messages are
harmless and really only interesting for debugging.

See: https://forum.rclone.org/t/rclone-gdrive-error/18098
2020-07-25 16:50:55 +01:00
Nick Craig-Wood
d35673efc6 webdav: fix directory creation with 4shared - fixes #4428
When we run MKCOL on 4shared on a directory that already exists, this
returns a 409/Conflict error. However this error code usually means
that the intermediate collections need creating.

The actual error code to return when trying to create a directory that
already exists isn't specified in the RFC, only that an error MUST be
returned and there are already 3 statuses checked in the code.

However using 409 makes rclone's usual strategy for making directories
fail and return the 409 error.

This patch tries the MKCOL and if it returns an unrecognised error
code, then calls PROPFIND on the directory to discover whether the
directory really exists or not.

This should also cover other WebDAV servers returning other error
messages we haven't accounted for in the code yet.
2020-07-24 17:26:42 +01:00
Nick Craig-Wood
8f9d5af26d cache: remove mount tests as they aren't being run and cause maintenance issues
Before this change the cache backend contained its own routines for
mounting testing on that mount.

These tests are never run on the CI and cause a maintenance burden.

This commit removes the tests.
2020-07-24 11:57:49 +01:00
Nick Craig-Wood
0272a7f405 mount: change interface of mount commands to take mount options
This is in preparation of being able to pass mount options to the rc
command "mount/mount"
2020-07-24 10:48:51 +01:00
Nick Craig-Wood
2871268505 mount: change interface of mount commands to take VFS
This is in preparation of being able to pass options to the rc command
"mount/mount"
2020-07-23 12:30:41 +01:00
Nick Craig-Wood
d2efb4b29b ftp: add support for --dump bodies and --dump auth
See: https://forum.rclone.org/t/rclone-copy-gives-error-connection-reset-by-peer-using-ftp/17934/27
2020-07-21 16:26:31 +01:00
Nick Craig-Wood
80d2f38192 s3: fix bucket Region auto detection when Region unset in config #2915
Previous to this fix if Region was not set and Endpoint was not set
then we set the endpoint to "https://s3.amazonaws.com/".

This is unecessary because if the Region alone isn't set then we set
it to "us-east-1" which has the same endpoint.

Having the endpoint set breaks the bucket region auto detection with
the error "Failed to update region for bucket: can't set region to
"xxx" as endpoint is set".

This fix removes that check.
2020-07-10 17:16:59 +01:00
Nick Craig-Wood
0792f4722c swift: fix purge not deleting directory markers
At some point Purge stopped deleting directory markers. We don't have
an integration test for this so it went unnoticed.

This patch fixes the problem but doesn't introduce an integration test
as we don't have a framework for making directory markers yet.
2020-07-10 15:16:11 +01:00
Nick Craig-Wood
db37360a1d swift: fix dangling large objects breaking the listing
Before this change, large objects which had had their contents deleted
would return "Object not found" and break the listing.

This change makes these objects appear as 0 sized entities so they can
be listed and deleted.
2020-07-10 11:03:08 +01:00
Nick Craig-Wood
d4b2709fb0 pcloud: fix oauth on European region "eapi.pcloud.com"
Pcloud appears to have opened up a new region and they are returning
the hostname in the oauth callback, thus

    GET /?code=XXX&locationid=1&hostname=api.pcloud.com&state=XXX HTTP/1.1
    GET /?code=XXX&locationid=2&hostname=eapi.pcloud.com&state=XXX HTTP/1.1

This isn't documented yet, however pCloud have confirmed that this is
the correct interpretation.

Rclone now reads the "hostname" parameter in the oauth callback and
stores it in the config file. It uses it for all subequent API calls.
2020-07-03 20:38:42 +01:00
Nick Craig-Wood
e6fdc3a932 drive: make dangling shortcuts appear in listings
Previous to this a dangling shortcut would error the directory
listing.

This patch makes dangling shortcuts appear as 0 sized objects in the
directory listing so they can be deleted. These objects can't be read
though.
2020-07-02 22:12:44 +01:00
Nick Craig-Wood
50e36fb482 onedrive: Fix reverting to Copy when Move would have worked
For some objects the onedrive backend has been doing a server side
copy and a delete when a server side move would have worked OK.

This was caused by not detecting the home drive correctly (when it was
an empty string) and assuming that these transfers were cross drive.

This is fixed by comparing canonicalizing drive IDs before comparing them.
2020-07-02 10:55:36 +01:00
Kai Lüke
54f2587c1e gcs: add support for anonymous access
Currently credentials are required to download a public bucket file
which is not really necessary and makes automated usage more complex.
Add a new option "anonymous" which when enabled configures the gcs
backend to use an anonymous HTTP client. This of course only works
for read access and trying to write will lead to errors like that:
"googleapi: Error 401: Anonymous caller does not not have
storage.objects.create access to the Google Cloud Storage object.",
as expected. By default the anonymous access option is disabled so that
the GCS Application Default Credentials are still used by default as
before and an error is given if they can't be found.
2020-07-01 20:54:49 +01:00
Nick Craig-Wood
fefcbf60fa sftp: use the absolute path instead of the relative path
Before this change rclone used the relative path from the current
working directory.

It appears that WS FTP doesn't like this and the openssh sftp tool
also uses absolute paths which is a good reason for switching to
absolute paths.

This change reads the current working directory at startup and bases
all file requests from there.

See: https://forum.rclone.org/t/sftp-ssh-fx-failure-directory-not-found/17436
2020-06-30 16:07:23 +01:00
Nick Craig-Wood
20f4fda3c9 local: fix race conditions updating and reading Object metadata 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
7622506fe2 local: factor UNCPath into lib/file 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
c820576329 fs: define SlowModTime and SlowHash features in the relevant backends 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
2a3b377d34 azureblob: don't compile on < go1.13 after dependency update 2020-06-29 14:45:39 +01:00
Nick Craig-Wood
61ff7306ae crypt: add --crypt-server-side-across-configs flag
This can be used for changing filename encryption mode without
re-uploading data.

See: https://forum.rclone.org/t/revert-filename-encryption-method/17454/
2020-06-27 11:40:15 +01:00
Nick Craig-Wood
0bcf4769fe local: make --local-no-updated provide a consistent view of the objects
Before this change the --local-no-updated flag would not error if the
files changed in size during the transfer. The file could still be
read beyond the size advertised though which caused problems with
certain backends.

After this change we attempt to provide a consistent view of the file
once it has been opened.

Once the file has had stat() called on it for the first time we

- Only transfer the size that stat gave
- Only checksum the size that stat gave
- Don't update the stat info for the file

This means that files that are extending can be transferred - rclone
will transfer the length it saw the first time it listed the file.

See: https://forum.rclone.org/t/transport-connection-broken/16494/21
2020-06-27 10:00:43 +01:00
David
9058ec32e1 s3: Use regional s3 us-east-1 endpoint 2020-06-26 16:25:52 +01:00
Nick Craig-Wood
61e4b4db42 drive: Allow the use of --drive-impersonate with the root_folder_id "appDataFolder"
In this commit

5c5ad6220 drive: fix --drive-impersonate with cached root_folder_id

We disabled the use of root_folder_id with --drive-impersonate to fix
a problem with a cached root_folder_id giving the wrong results.

This, alas, broke one users setup with a root_folder_id of
appDataFolder. Since this is identifiable and definitely couldn't have
been cached, we can safely skip this check in this case.

See: https://forum.rclone.org/t/rclone-gdrive-no-longer-returning-anything/17215/10
2020-06-25 21:43:11 +01:00
Nick Craig-Wood
fd7c63bc78 s3: add backend restore command to restore objects from GLACIER
See: https://forum.rclone.org/t/rclone-settier-fails-with-scaleway-entitytoolarge/17384
2020-06-25 21:33:23 +01:00
Nick Craig-Wood
49a7d08a40 qingstor: cancel in progress multipart uploads on rclone exit #4300 2020-06-25 15:22:53 +01:00
Nick Craig-Wood
2c10ce64aa onedrive: rework cancel of multipart uploads on rclone exit #4300
This now uses the atexit.OnError framework rather than a home grown one.
2020-06-25 15:22:53 +01:00
Nick Craig-Wood
a41a294e1d box: cancel in progress multipart uploads and copies on rclone exit #4300 2020-06-25 15:22:53 +01:00
Nick Craig-Wood
47b17dc1bb b2: cancel in progress multipart uploads and copies on rclone exit #4300 2020-06-25 15:22:53 +01:00
Nick Craig-Wood
5f75444ef6 s3: cancel in progress multipart uploads and copies on rclone exit #4300 2020-06-25 12:55:56 +01:00
Nick Craig-Wood
2121c0fa23 dircache: factor DirMove code out of backends into dircache
Before this change there was lots of duplicated code in all the
dircache using backends to support DirMove.

This change factors this code into the dircache library.
2020-06-25 09:41:36 +01:00
Nick Craig-Wood
a8652e2252 dircache: simplify interface, fix corner cases and apply to backends
Dircache was changed to:

- Remove special cases for the root directory
- Remove Fatal errors
- Call FindRoot on behalf of the user wherever possible
- Bring up to modern Go standards

Backends were changed to:

- Remove calls to FindRoot
- Change calls to FindRootAndPath to FindPath
- Don't make special cases for the root

This fixes several corner cases, for example removing a non existent
directory if FindRoot hasn't been called.
2020-06-25 09:41:36 +01:00
Nick Craig-Wood
81151523af drive: fix shortcut tests 2020-06-24 15:52:02 +01:00
Nick Craig-Wood
0dba7b8a46 swift: speed up deletes by not retrying segment container deletes
Before this fix rclone would continually try to delete non empty
segment containers which made deleting lots of files very slow.

This fix makes rclone just try the delete once and then carry on which
was the original intent of the code before the retry logic got put in.
2020-06-24 10:01:24 +01:00
buengese
e247811db5 jottacloud: remove debug Printf accidentally left in 2020-06-23 13:16:23 +02:00
buengese
ce767bc3cf pcloud: implement PublicLink 2020-06-21 17:22:56 +02:00
Nick Craig-Wood
a55d882b7b webdav: Fix free/used display for rclone about/df for certain backends - fixes #4348
Before this change if the server sent us xml like this

```
<D:propstat>
<D:prop>
<g0:quota-available-bytes/>
<g0:quota-used-bytes/>
</D:prop>
<D:status>HTTP/1.1 404 Not Found</D:status>
</D:propstat>
```

Rclone would read the empty XML items as containing 0

After this fix we make sure that we have a value before using it.
2020-06-20 15:15:15 +01:00
Nick Craig-Wood
5c5ad62208 drive: fix --drive-impersonate with cached root_folder_id
Before this fix rclone v1.51 and 1.52 would incorrectly use the cached
root_folder_id when the --drive-impersonate flag was in use. This
meant that rclone could be looking up the wrong directory ID with
unpredictable results - usually all files apparently being missing.

This fix makes rclone look up the root_folder_id always when using
--drive-impersonate. It does this by clearing the root_folder_id and
making a NOTICE message that it is ignoring the cached value.

It also stops rclone caching the root_folder_id when using
--drive-impersonate.

See: https://forum.rclone.org/t/rclone-gdrive-no-longer-returning-anything/17215
2020-06-20 15:01:37 +01:00
buengese
b6b8958fb4 box: implement CleanUp - fixes #4326 2020-06-18 23:39:59 +02:00
Nick Craig-Wood
d8eea0e397 build: run gofmt -s to simplify the code: suggested by Go Report Card 2020-06-18 18:45:39 +01:00
Nick Craig-Wood
df9c930581 dropbox: fix public link by removing expires parameter
Adding the expires parameter gives settings_error/not_authorized/.. errors.

The expires setting isn't in the documentation so this commit removes
it for now.
2020-06-18 18:40:33 +01:00
Nick Craig-Wood
85bcacac90 s3: Cap expiry duration to 1 Week and return error when sharing dir 2020-06-18 17:50:50 +01:00
Nick Craig-Wood
5e6f4ab281 drive: fix creating a directory inside a shortcut
See: https://forum.rclone.org/t/cant-create-new-directory-on-google-drive-remote/17208
2020-06-17 11:32:28 +01:00
buengese
2c4f7b61c1 jottacloud: switch to new api root - fixes #4295
- also implement a very ugly workaround for the DirMove failures
2020-06-16 15:44:34 +02:00
Heiko Bornholdt
17d5a72416 ftp: add explicit tls support
Add support for explicit FTP over TLS.

Fixes #4100
2020-06-16 09:13:50 +01:00
Nick Craig-Wood
b58bb03e95 test: Don't run unreliable tests on CI #4171 2020-06-15 21:34:37 +01:00
Vincent Feltz
f4d7e41f24 s3: add Scaleway provider - fixes #4338 2020-06-13 11:55:37 +01:00
Zac Rubin
f9306218f8 sftp: Fix SSH key PEM loading
For SSH authentication, `key_pem` should both override `key_file`
and not require other SSH authentication methods to be set.

Prior to this fix, rclone would attempt to use an ssh-agent
when `key_pem` was the only SSH authentication method set.

Fixes #4240
2020-06-12 22:46:33 +01:00
Nick Craig-Wood
848c5b78e1 drive: fix not being able to delete a directory with a trashed shortcut
When we resolve the shortcut we now propagate the trashed status of
the shortcut into the resolved item which fixes the issue.
2020-06-12 15:10:35 +01:00
buengese
84d5df3c84 jottacloud: bring back legacy authentification for use with whitelabel versions - fixes #4299 2020-06-12 12:08:27 +02:00
Nick Craig-Wood
7e48ee8758 cache: fix dedupe on caches wrapping drives - fixes #4320
This implements the MergeDirs optional method.
2020-06-10 21:52:52 +01:00
Nick Craig-Wood
2ea15a72bc s3: fix --header-upload - Fixes #4303
Before this change we were setting the headers on the PUT
request for normal and multipart uploads. For normal uploads this caused the error

    403 Forbidden: There were headers present in the request which were not signed

After this fix we set the headers in the object upload request itself
as the s3 SDK expects.

This means that we only support a limited range of headers

- Cache-Control
- Content-Disposition
- Content-Encoding
- Content-Language
- Content-Type
- X-Amz-Tagging
- X-Amz-Meta-

Note for the last of those are for setting custom metadata in the form
"X-Amz-Meta-Key: value".

This now works for multipart uploads and single part uploads

See also #59
2020-06-10 12:28:48 +01:00
Cenk Alti
16422a6b78 putio: fix panic on Object.Open #4315 2020-06-10 12:16:09 +01:00
Caleb Case
40fe97e946 backend/tardigrade: Set UserAgent to rclone
This provides two things:

* It gives Storj insight into which uplink clients are using the
  network.
* It facilitate rclone participating in the Tardigrade Open Source
  Partner Program https://tardigrade.io/partner/
2020-06-09 14:20:28 +01:00
Kamil Trzciński
7458d37d2a
s3: add max_upload_parts support - fixes #4159
* s3: add `max_upload_parts` support

This allows to configure a maximum amount of chunks used to upload file:

- Support Scaleway which has a limit of 1k chunks currently
- Reduce a cost on S3 when each request costs some money at the expense of memory used

Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2020-06-08 18:22:34 +01:00
Roman Kredentser
c0521791db s3: implement link sharing with PublicLink 2020-06-05 14:51:05 +01:00
Roman Kredentser
55ad1354b6 link: Add --expire and --unlink flags
This adds expire and unlink fields to the PublicLink interface.

This fixes up the affected backends and removes unlink parameters
where they are present.
2020-06-05 14:51:05 +01:00
Nick Craig-Wood
fb61ed8506 b2: Implement server side copy for files > 5GB - fixes #3991
This factors copy out of SetModTime and Copy so it can be called from
both places.

This also reworks all the multipart uploading to use sync.Errgroup and
memory pooling like the other backends. This makes it more memory
efficient and handle errors better.

See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/10
2020-06-05 13:27:53 +01:00
Nick Craig-Wood
973e3d6a7b backends: make sure backends expand ~ and environment vars in file names they use
See: https://forum.rclone.org/t/relative-path-in-rclone-config-service-account-json/16693
2020-06-03 17:39:08 +01:00
Nick Craig-Wood
151f03378f s3: fix upload of single files into buckets without create permission
Before this change, attempting to upload a single file into an s3
bucket which did not have create permission gave AccessDenied: Access
Denied error when it tried to create the bucket.

This was masked until e2bf91452a was
fixed.

This fix marks the bucket as OK if a fetch on an object indicates it
is OK. This stops rclone thinking it has to create the bucket in the
first place.

Fixes #4297
2020-06-02 14:33:21 +01:00
Nick Craig-Wood
cbf3d43561 drive: fix missing items when listing using --fast-list / ListR
This is caused by a bug in Google drive where, in some circumstances
querying for "(A in parents) or (B in parents)" returns nothing
whereas querying for "A in parents" and "B in parents" separately
works fine.

This has been reported here:

https://issuetracker.google.com/issues/149522397

This workaround detects this condition by seeing if a listing for more
than one directory at once returns nothing.

If it does then it retries each one individually.

This can potentially have a false positive if the user has multiple
empty directories which are queried at once. The consequence of this
will be that ListR is disabled for a while until the directories are
found to be actually empty in which case it will be re-enabled.

Fixes #3114 and Fixes #4289
2020-05-31 11:44:15 +01:00
Nick Craig-Wood
74b8cbfb84 docs: set unsafe HTML parsing to false and fix raw HTML insertion
This means that markdown files can't contain <thing> any more.
2020-05-27 17:31:09 +01:00
Nick Craig-Wood
78ca08ba8a pcloud: fix initial config "Auth state doesn't match" message #4210
pCloud should be passing back the state parameter that rclone passed
in on config but it seems to have got lost somewhere.

This sets a work-around for the pCloud backend allowing an empty state
parameter.

See: https://forum.rclone.org/t/cannot-connect-to-pcloud/16592
See: https://forum.rclone.org/t/cannot-create-pcloud-config-file-on-osx/16583
2020-05-26 11:27:01 +01:00
Nick Craig-Wood
49ba4eeb86 oauthutil: tidy interface to Config to add Options struct
The interface was getting so that a new function was needed for every
Config variant. Adding an Options struct fixes this.
2020-05-26 11:27:01 +01:00
Nick Craig-Wood
c08617c70f box: Calculate Free amount in About call 2020-05-25 16:47:34 +01:00
Martin Michlmayr
041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Nick Craig-Wood
9db8ecbc32 box: implement About to read size used - fixes #4264 2020-05-23 18:46:44 +01:00
Martin Michlmayr
a36ef8582f doc: use consistent capitalization 2020-05-20 15:54:51 +01:00
Martin Michlmayr
f34a40a709 swift: fix cosmetic issue in error message 2020-05-20 15:54:51 +01:00
Martin Michlmayr
4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Fred
5f71d186b2 seafile: implement 2FA 2020-05-20 15:46:35 +01:00
Nick Craig-Wood
cf5d0f5c1f Revert "drive: server side copy docs use default description if empty"
This reverts commit 9e4b68a364.

This does not work as intended - it only changes docs files and to
make it change drive files would take an extra roundtrip.

I think the sematics of server side copy are now correct - additional
features should be added with a new flag.

See #4230
2020-05-19 16:48:02 +01:00
Nick Craig-Wood
bdafbad61e cache: fix tests writing to empty path
This meant the tests were writing to the current directory instead of
a temporary directory.
2020-05-19 16:01:35 +01:00
Brandon McNama
19ff7c9302 cache: Fix Server Side Copy with Temp Upload
When wrapping a backend that supports Server Side Copy (e.g. `b2`, `s3`)
and configuring the `tmp_upload_path` option, the `cache` backend would
erroneously report that Server Side Copy/Move was not supported, causing
operations such as file moves to fail. This change fixes this issue
under these circumstances such that Server Side Copy will now be used
when the wrapped backend supports it.

Fixes #3206
2020-05-19 12:17:40 +01:00
Martin Michlmayr
fb169a8b54
doc: fix typos throughout docs 2020-05-19 12:02:44 +01:00
calisro
bcbfad1482
sft[: added --sftp-pem-key to support inline key files 2020-05-19 11:55:38 +01:00
Nick Craig-Wood
610f40f700 local: implement --local-no-sparse flag for disabling sparse files #2469
This also introduces a one time warning for sparse files and updates
the docs to warn about them.
2020-05-19 10:16:43 +01:00
Brandon Philips
633f50cd3e
googlephotos: create feature/favorites directory - Fixes #4189
Enable access “Favorite” images on Google Photos backend.

This adds a “feature/favorites” folder in the Google Photos backend
and uses the Feature Filter API:

https://developers.google.com/photos/library/reference/rest/v1/mediaItems/search#Filters
2020-05-18 17:55:16 +01:00
Nick Craig-Wood
e4f1e19127 sftp: fix post transfer copies failing with 0 size when using set_modtime=false
Before this change we early exited the SetModTime call which means we
skipped reading the info about the file.

This change reads info about the file in the SetModTime call even if
we are skipping setting the modtime.

See: https://forum.rclone.org/t/sftp-and-set-modtime-false-error/16362
2020-05-14 17:30:01 +01:00
Nick Craig-Wood
4a1b644bfb azureblob: implement streaming of unknown sized files
See: https://forum.rclone.org/t/rclone-rcat-azure-blob-container-sas-token-403-error/16286/3
2020-05-14 11:56:15 +01:00
Nick Craig-Wood
8c9c86c3d6 putio: fix parsing of remotes with leading and trailing /
See: https://forum.rclone.org/t/unable-to-copy-from-remote-but-mount-works/16351/
2020-05-14 11:52:43 +01:00
Nick Craig-Wood
8a58e0235d s3: don't leak memory or tokens in edge cases for multipart upload 2020-05-14 07:48:18 +01:00
Nick Craig-Wood
52b7337d28 crypt: change backend encode/decode to output a plain list
This commit changes the output of the rclone backend encode crypt: and
decode commands to output a plain list of decoded or encoded file
names.

This makes the command much more useful for command line scripting.
2020-05-13 18:11:45 +01:00
Max Sum
33d9310c49
union: enable ListR when upstreams contain local
Enable fast list functions for union backend when:

- at least one of the upstreams supports fast list
- upstreams only consist of backends that support fast list and local backend.

Fixes #3000
2020-05-13 13:10:35 +01:00
Nick Craig-Wood
9e4b68a364 drive: server side copy docs use default description if empty
When server side copying Google docs files we attempt to preserve the
description.

This patch makes it so that we use the default description if the
original description was empty.

See: 6fdd7149c1 (commitcomment-38008638)
2020-05-13 12:31:37 +01:00
Nick Craig-Wood
d342f9f942 azureblob: fix permission error on SAS URL limited to container
Before this change, for some operations, eg rcat or copyto (of a file)
rclone would attempt to create the container when using a SAS URL
limited to a container.

After this change we assume the container does not need creating when
using a container SAS URL.

See: https://forum.rclone.org/t/rclone-rcat-azure-blob-container-sas-token-403-error/16286
2020-05-13 09:11:51 +01:00
Nick Craig-Wood
8ddb3fbb2e drive: fix using list recursive on shortcuts to directories 2020-05-12 17:08:05 +01:00
Nick Craig-Wood
b91e01fd22 drive: strip trailing slashes in shortcut command #4098
This also fixes typo in the name of the function, and allows making
shortcuts from the root directory which are useful in cross drive
shortcut creation.

This also adds a basic suite of tests for creating listing, removing
shortcuts.
2020-05-12 17:08:05 +01:00
Caleb Case
0ce662faad Tardigrade Backend 2020-05-12 15:56:50 +00:00
Max Sum
54b16bd054 union: implement ListR 2020-05-10 17:57:03 +00:00
Max Sum
f21e97001b union: fix server-side copy 2020-05-10 17:56:18 +00:00
Nick Craig-Wood
bb65974e2f drive: implement backend shortcut command for creating shortcuts #4098 2020-05-09 15:16:15 +01:00
Nick Craig-Wood
bc0f487369 drive: look for dirs as well as files on NewObject
This means that we can return ErrorNotAFile when there is an object
with the same name as a directory rather than potentially creating a
duplicate name.
2020-05-09 15:16:15 +01:00
Fred
c754e89906 seafile: New backend for seafile server 2020-05-06 17:33:22 +00:00
Nick Craig-Wood
afde340c9e gcs: fix --header-upload - #59
Before this code we were settig the headers on the PUT request. However this isn't where GCS needs them.

After this fix we set the headers in the object upload request itself.

This means that we only support a limited range of headers

- Cache-Control
- Content-Disposition
- Content-Encoding
- Content-Language
- Content-Type
- X-Goog-Meta-

Note for the last of those are for setting custom metadata in the form
"X-Goog-Meta-Key: value".
2020-05-06 17:34:23 +01:00
Anagh Kumar Baranwal
a86196a156 drive: Added command to change service_account_file and chunk_size
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-05-04 16:23:33 +00:00
Anagh Kumar Baranwal
856c2b565f crypt: Added decode/encode commands to replicate functionality of cryptdecode
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-05-04 16:23:33 +00:00
Nick Craig-Wood
14cab0fff0 local: fix "file not found" errors on post transfer Hash calculation
Before this change the local backend was returning file not found
errors for post transfer hashes for files which were moved. This was
caused by the routine which checks for the object being changed.

After this change we ignore file not found errors while checking to
see if the object has changed. If the hash has to be computed then a
file not found error will be thrown when it is opened, otherwise the
cached hash will be returned.
2020-05-04 12:17:46 +01:00
Nick Craig-Wood
f2b1fedc4f drive: follow shortcuts by default, skip with --drive-skip-shortcuts
Before this change rclone would skip all shortcuts with a message

    Ignoring unknown document type "application/vnd.google-apps.shortcut"

After this message rclone resolves the shortcuts by default to the
actual files that they point to. See the docs for more info.

The --drive-skip-shortcuts flag can be used to skip shortcuts.
2020-05-02 18:28:38 +01:00
Nick Craig-Wood
b52a39a84e drive: fix merge breakage
In 2f5a2d3c48 an incorrect merge caused compilation to fail
2020-05-01 13:02:32 +01:00
Nick Craig-Wood
2f5a2d3c48 drive: Don't return nil Object with nil error from newObject* functions.
Before this change the newObject* functions could return object=nil
with err=nil.  The result of these functions are passed outside of the
backend code (eg in Copy, Move) and returning a nil object with a nil
error leads to crashes elsewhere as it breaks expectations.

After this change we return (nil, fs.ErrorObjectNotFound) in these
cases. The one place this is actually needd internally (when turning
items into listings) we detect that error and use it to mean skip the
directory item.

This problem was noticed while testing the shortcuts code. It
shouldn't happen normally but it is conceivable it could.
2020-04-30 17:11:36 +01:00
Nick Craig-Wood
74d9dabdff b2: force the case of the SHA1 to lowercase - fixes #4162
Apparently some tools (eg duplicati) upload the SHA1 in uppercase to
b2 to be stored in the `large_file_sha1` metadata. This patch forces
it to lower case.
2020-04-29 17:08:21 +01:00