Commit Graph

636 Commits

Author SHA1 Message Date
Ankur Gupta
75a6c49f87 Fix error counter - fixes #3650
For few commands, RClone counts a error multiple times. This was fixed by
creating a new error type which keeps a flag to remember if the error has
already been counted or not. The CountError function now wraps the original
error eith the above new error type and returns it.
2019-11-18 14:13:02 +00:00
Nick Craig-Wood
19229b1215 drive: fix --drive-root-folder-id with team/shared drives
Before this change rclone used the team_drive ID as the root if set
even if the root_folder_id was set too.

This change uses the root_folder_id in preference over the team_drive
which restores the functionality.

This problem was introduced by ba7c2ac443

Fixes #3742
2019-11-16 18:38:21 +00:00
Nick Craig-Wood
479c803fd9 vendor: update all dependencies 2019-11-14 21:51:34 +00:00
Nick Craig-Wood
3dcf1e61cf cache: follow move of upstream library github.com/coreos/bbolt github.com/etcd-io/bbolt 2019-11-14 21:51:34 +00:00
Sebastian Brandt
f158a398f3 sftp: Retry Creation of Connection - fixes #3656
Removes the existing rate limiter because it is implicitly included in
the pacer.
2019-11-14 12:50:01 +00:00
jaKa
acefa5c40d koofr: use rclone HTTP client. 2019-11-14 11:36:44 +00:00
Nick Craig-Wood
1e423d21e1 drive: fix listing of the root directory with drive.files scope
We attempt to find the ID of the root folder by doing a GET on the
folder ID "root". With scope "drive.files" this fails with a 404
message.

After this change if we get the 404 message, we just carry on using
"root" as the root folder ID and we cache that for future lookups.

This means that changenotify messages will not work correctly in the
root folder but otherwise has minor consequences.

See: https://forum.rclone.org/t/fresh-raspberry-pi-build-google-drive-404-error-failed-to-ls-googleapi-error-404-file-not-found/12791
2019-11-11 09:07:34 +00:00
Nick Craig-Wood
9b5308144f s3: Reduce memory usage streaming files by reducing max stream upload size
Before this change rclone would allow the user to stream (eg with
rclone mount, rclone rcat or uploading google photos or docs) 5TB
files.  This meant that rclone allocated 4 * 525 MB buffers per
transfer which is way too much memory by default.

This change makes rclone use the configured chunk size for streamed
uploads.  This is 5MB by default which means that rclone can stream
upload files up to 48GB by default staying below the 10,000 chunks
limit.

This can be increased with --s3-chunk-size if necessary.

If rclone detects that a file is being streamed to s3 it will make a
single NOTICE level log stating the limitation.

This fixes the enormous memory usage.

Fixes #3568
See: https://forum.rclone.org/t/how-much-memory-does-rclone-need/12743
2019-11-09 15:55:19 +00:00
Aleksandar Jankovic
4b20afa94a backend/s3: fix ExpiryWindow value
ExpiryWindow accepts duration but it was set to value 3.
This changes it to 3 * time.Minute since default is 5 min.
2019-11-05 13:55:55 +00:00
Nick Craig-Wood
3f7af64316 config: give config questions default values - fixes #3672 2019-11-05 11:53:44 +00:00
Nick Craig-Wood
7bf056316f local: fix listings of . on Windows - fixes #3676 2019-10-30 16:00:18 +00:00
Nick Craig-Wood
1ce1ea34aa hash: fix hash names for DropboxHash and CRC-32
These were unintentionally renamed as part of 1dc8bcd48c

Fixes #3679
2019-10-30 12:20:10 +00:00
Xiaoxing Ye
191cfb79d1 onedrive: no trailing slash reading metadata...
No trailing slash when reading metadata of an item given item ID.

This should fix #3664.
2019-10-29 13:33:11 +00:00
Nick Craig-Wood
ab895390f4 s3: fix nil pointer reference if no metadata returned for object
Fixes #3651 Fixes #3652
2019-10-25 13:45:47 +01:00
Nick Craig-Wood
a3a5857874 drive: fix change notify polling when using appDataFolder
See: https://forum.rclone.org/t/remote-changes-arent-picked-up/12520
2019-10-24 12:51:01 +01:00
Nick Craig-Wood
0f0079ff71 b2: remove unverified: prefix on sha1 - fixes #3654 2019-10-23 08:41:56 +01:00
dausruddin
7eee2f904a drive: fix typo 2019-10-21 22:28:28 +01:00
Nick Craig-Wood
3ef0c73826 drive: fix ChangeNotify polling for shared drives
Before this fix we neglected to add the shared drive ID to the request
when asking for an initial change notify token and this caused a lot
more results to be returned than was necessary.
2019-10-21 20:51:11 +01:00
Nick Craig-Wood
2bbfcc74e9 drive: fix --drive-shared-with-me from the root with ls and --fast-list
When we changed recursive lists to use --fast-list by default this
broke listing with --drive-shared-with-me from the root.

This turned out to be an unwarranted assumption in the ListR code that
all items would have a parent folder that we had searched for - this
isn't true for shared with me items.

This was fixed when using --drive-shared-with-me to give items that
didn't have any parents a synthetic parent.

Fixes #3639
2019-10-21 12:16:01 +01:00
Nick Craig-Wood
ba7c2ac443 drive: make sure that drive root ID is always canonical
Before this change we used the id "root" as an alias for the root drive ID.

However this causes problems when we receive IDs back from drive which
are not in this format and have been expanded to their canonical ID.

This change looks up the ID "root" and stores it in the
"drive_folder_id" parameter in the config file.

This helps with
- Notifying changes at the root
- Files shared with me at the root

See #3639
2019-10-21 12:16:01 +01:00
Nick Craig-Wood
2d9b8cb981 azureblob: disable logging to the Windows event log
See: https://forum.rclone.org/t/event-log-warning/12430
2019-10-21 11:50:31 +01:00
Carlos Ferreyra
9cb549a227 sftp: include more ciphers with use_insecure_cipher 2019-10-17 14:58:31 +01:00
Nick Craig-Wood
38652d046d drive: disable HTTP/2 by default to work around INTERNAL_ERROR problems
Before this change when rclone was compiled with go1.13 it used HTTP/2
to contact drive by default.

This causes lockups and INTERNAL_ERRORs from the HTTP/2 code.

This is a workaround disabling the HTTP/2 code on an option.

It can be re-enabled with `--drive-disable-http2=false`

See #3631
2019-10-16 11:26:08 +01:00
Cenk Alti
929f275ae5 putio: add ability to resume uploads 2019-10-14 20:01:16 +01:00
Ivan Andreev
77b42aa33a chunker: fix integration tests and hashsum issues 2019-10-13 10:43:46 +01:00
Ivan Andreev
910c80bd02 chunker: option to hash all files 2019-10-13 10:43:46 +01:00
Ivan Andreev
9049bb62ca chunker: prevent chunk corruption, survive meta-like input 2019-10-13 10:43:46 +01:00
Ivan Andreev
7aa2b4191c chunker: reservations for future extensions 2019-10-13 10:43:46 +01:00
Alex Chen
41ed33b08e docs: update onedrive/sharepoint docs on some known issues 2019-10-12 12:08:22 +01:00
Nick Craig-Wood
65a82fe77d dropbox: fix nil pointer exception on restricted files
See: https://forum.rclone.org/t/issues-syncing-dropbox/12233
2019-10-11 16:21:24 +01:00
Jon Fautley
5d33236050 ftp: allow disabling EPSV mode 2019-10-10 21:00:41 +01:00
Nick Craig-Wood
6abaa9e22c fstests: allow skipping of the broken UTF-8 test for the cache backend 2019-10-10 10:36:18 +01:00
Nick Craig-Wood
8c1edf410c dropbox: make disallowed filenames return no retry error - fixes #3569
Before this change we silently skipped uploads to dropbox of
disallowed file names.  However this then caused "corrupted on
transfer" errors because the sizes were wrong.

After this change we return an no retry error which will mean that the
sync fails (as it should - not all files were uploaded) but no
unecessary retries happened.
2019-10-08 19:59:47 +01:00
Henning Surmeier
eff11b44cf webdav: parse and return sharepoint error response
fixes #3176
2019-10-06 20:17:13 +01:00
Nick Craig-Wood
5271fe3b3f yandex: use lib/encoder 2019-10-05 10:22:43 +01:00
庄天翼
7b1274e29a s3: support for multipart copy
Fixes #2375 Fixes #3579
2019-10-04 16:49:06 +01:00
Nick Craig-Wood
d21ddf280c mailru: comment out some debugging statements 2019-10-02 20:10:01 +01:00
Nick Craig-Wood
135717e12b mailru: use lib/encoder 2019-10-02 20:10:01 +01:00
Aleksandar Jankovic
6b55b8b133 s3: add option for multipart failiure behaviour
This is needed for resuming uploads across different sessions.
2019-10-02 16:49:16 +01:00
Nick Craig-Wood
b94b2a3723 mega: fix after lib/encoder change 2019-10-02 12:41:52 +01:00
Nick Craig-Wood
fd51f24906 putio: use lib/encoder
And in the process
- fix a bug with + and & in file name
- fix NewObject returning directories as files
2019-10-02 11:34:08 +01:00
Nick Craig-Wood
4615343b73 premiumizeme: use lib/encoder 2019-10-02 11:34:08 +01:00
Fionera
1dc8bcd48c Remove backend dependency from fs/hash 2019-10-01 16:29:58 +01:00
Nick Craig-Wood
77a520c97c fichier: fix accessing files > 2GB on 32 bit systems - fixes #3581 2019-10-01 16:03:49 +01:00
Nick Craig-Wood
04eb96b50b fichier: fix NewObject after lib/encoder changes
This bug was introduced as part of the lib/encoder changes in
8d8fad724b.  It caused NewObject not to work for a file with
escaped characters in it.
2019-10-01 15:30:51 +01:00
Fabian Möller
b9bd15a8c9 koofr: use lib/encoder
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2019-09-30 22:00:25 +01:00
Nick Craig-Wood
b581f2de26 sharefile: use lib/encoder 2019-09-30 22:00:25 +01:00
Nick Craig-Wood
8d8fad724b ficher: use lib/encoder 2019-09-30 22:00:25 +01:00
Nick Craig-Wood
d122d1d191 qingstor: use lib/encoder 2019-09-30 22:00:25 +01:00
Nick Craig-Wood
35d6ff89bf azureblob: use lib/encoder 2019-09-30 22:00:24 +01:00