Commit Graph

935 Commits

Author SHA1 Message Date
Nick Craig-Wood
5271fe3b3f yandex: use lib/encoder 2019-10-05 10:22:43 +01:00
Nick Craig-Wood
135717e12b mailru: use lib/encoder 2019-10-02 20:10:01 +01:00
Nick Craig-Wood
fd51f24906 putio: use lib/encoder
And in the process
- fix a bug with + and & in file name
- fix NewObject returning directories as files
2019-10-02 11:34:08 +01:00
Nick Craig-Wood
4615343b73 premiumizeme: use lib/encoder 2019-10-02 11:34:08 +01:00
Fionera
1dc8bcd48c Remove backend dependency from fs/hash 2019-10-01 16:29:58 +01:00
Nick Craig-Wood
b581f2de26 sharefile: use lib/encoder 2019-09-30 22:00:25 +01:00
Nick Craig-Wood
8d8fad724b ficher: use lib/encoder 2019-09-30 22:00:25 +01:00
Nick Craig-Wood
d122d1d191 qingstor: use lib/encoder 2019-09-30 22:00:25 +01:00
Nick Craig-Wood
35d6ff89bf azureblob: use lib/encoder 2019-09-30 22:00:24 +01:00
Nick Craig-Wood
53bec33027 swift: use lib/encoder 2019-09-30 22:00:24 +01:00
Fabian Möller
3304bb7a56 googlecloudstorage: use lib/encoder
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2019-09-30 22:00:24 +01:00
Fabian Möller
33f129fbbc s3: use lib/encoder
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2019-09-30 22:00:24 +01:00
Nick Craig-Wood
6ae7bd7914 local: encode invalid UTF-8 on macOS 2019-09-30 22:00:24 +01:00
Nick Craig-Wood
32af4cd6f3 ftp: use lib/encoder 2019-09-30 22:00:24 +01:00
Fabian Möller
00b2c02bf4 pcloud: use lib/encoder
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2019-09-30 22:00:24 +01:00
Fabian Möller
c8d3e57418 encodings: add . and .. to all backends, except Drive 2019-09-30 14:05:49 +01:00
Fabian Möller
4837bc3546 jottacloud: use lib/encoder 2019-09-30 14:05:49 +01:00
Fabian Möller
c09b62a088 encodings: add all known backend encodings 2019-09-30 14:05:49 +01:00
Fabian Möller
97a218903c fstest: remove WinPath from fstest.Item 2019-09-30 14:05:48 +01:00
Nick Craig-Wood
262ba28dec config: add ReadNonEmptyLine utility function 2019-09-30 11:23:21 +01:00
Nick Craig-Wood
74f6300875 Start v1.49.4-DEV development 2019-09-29 19:47:59 +01:00
Ivan Andreev
ccecfa9cb1 chunker: finish meta-format before release
changes:
- chunker: remove GetTier and SetTier
- remove wdmrcompat metaformat
- remove fastopen strategy
- make hash_type option non-advanced
- adverise hash support when possible
- add metadata field "ver", run strict checks
- describe internal behavior in comments
- improve documentation

note:
wdmrcompat used to write file name in the metadata, so maximum metadata
size was 1K; removing it allows to cap size by 200 bytes now.
2019-09-25 11:03:33 +01:00
Ivan Andreev
c41812fc88 tests: bring memory hungry tests close to end 2019-09-24 12:45:12 +01:00
Ivan Andreev
d98d1be3fe accounting: fix panic due to server-side copy fallback 2019-09-24 12:45:12 +01:00
Ivan Andreev
661dc568f3 fstest: let backends advertise maximum file size 2019-09-24 12:45:12 +01:00
Ivan Andreev
1e4691f951 tests/sync: adjust transfer counts for chunker 2019-09-24 12:45:12 +01:00
Nick Craig-Wood
557edecd40 log: add Stack() function for debugging who calls what 2019-09-20 11:53:08 +01:00
Nick Craig-Wood
b242b0a078 lib/cache,rc/jobs: make tests more reliable 2019-09-20 11:53:08 +01:00
Nick Craig-Wood
56544bb2fd accounting: fix file handle leak on errors - fixes #3547
In 53a1a0e3ef we introduced a problem where if there was an
error on the file being transferred then the file was re-opened and
the old one wasn't closed.

This was partially fixed in bfbddab46b however this didn't
address the case of the old file being closed.

This is now fixed by
- marking the file as open again in UpdateReader
- moving the stopping the accounting machinery to a new method Done
2019-09-19 16:20:07 +01:00
Nick Craig-Wood
b8b12a4000 Start v1.49.3-DEV development 2019-09-15 18:10:08 +01:00
Nick Craig-Wood
ef7bfd3f03 fs: Make prefix free backend config read prefix free env var also
Before this change you could only configure the local backend flags
which don't have the local prefix (eg `--copy-links`) with
`RCLONE_LOCAL_COPY_LINKS`.

This change makes `RCLONE_COPY_LINKS` valid too which is much more
logical for the users.

Fixes #3534
2019-09-14 18:26:07 +01:00
Nick Craig-Wood
6959c997e2 config: remove error: can't use --size-only and --ignore-size together.
As sometimes they are useful together, for example when the remote
changes the sizes of the uploaded files.

See: https://forum.rclone.org/t/files-upload-will-be-auto-compress-how-do-i-sync-a-file-to-remote/10578
2019-09-14 10:13:44 +01:00
Nick Craig-Wood
bfbddab46b fs/accounting: Fix "file already closed" on transfer retries
This was caused by the recent reworking of the accounting interface.
The Transfer object was recycling the Accounting object without
resetting the stream.

See: https://forum.rclone.org/t/error-file-already-closed/11469/
See: https://forum.rclone.org/t/rclone-b2-sync-post-error-method-not-supported/11718/
2019-09-13 18:35:02 +01:00
Aleksandar Jankovic
934440a9df accounting: fix total duration calculation
Fixes: #3498
2019-09-12 10:29:40 +01:00
Ivan Andreev
bdcd0b4c64 Add mailru hash (mrhash) 2019-09-09 21:34:15 +01:00
Nick Craig-Wood
94eb9a4014 Start v1.49.2-DEV development 2019-09-08 17:36:21 +01:00
Nick Craig-Wood
f1347139fa config: check config names more carefully and report errors - fixes #3506
Before this change it was possible to make a remote with an invalid
name in the config file, either manually or with `rclone config
create` (but not with `rclone config`).

When this remote was used, because it was invalid, rclone would
presume this remote name was a local directory for a very suprising
user experience!

This change checks remote names more carefully and returns errors
- when the user tries to use an invalid remote name on the command line
- when an invalid remote name is used in `rclone config create/update/password`
- when the user tries to enter an invalid remote name in `rclone config`

This does not prevent the user entering a remote name with invalid
characters in the config manually, but such a remote will fail
immediately when it is used on the command line.
2019-09-06 12:07:09 +01:00
Nick Craig-Wood
cf9b973fe4 accounting: fix locking in Transfer to avoid deadlock with --progress
Before this change, using -P occasionally deadlocked on the Transfer
mutex when Transfer.Done() was called with a non nil error and the
StatsInfo mutex since they mutually call each other.

This was fixed by making sure that the Transfer mutex is always
released before calling any StatsInfo methods.

This improves on: 6f87267b34

Fixes #3505
2019-09-06 10:00:44 +01:00
Nick Craig-Wood
ffa1dac10b build: apply gofmt from go1.13 to change case of number literals 2019-09-05 13:59:06 +01:00
Nick Craig-Wood
530ba66d35 operations: fix -u/--update with google photos / files of unknown size
Before this change if -u/--update was in effect we compared the size
of the files to see if the transfer should go ahead.  This was
comparing -1 with an actual size so the transfer always proceeded.

After this change we use the existing `sizeDiffers` function which
does the correct comparison with -1 for files of unknown length.

See: https://forum.rclone.org/t/sync-with-google-photos-to-local-drive-will-result-in-recoping/11605
2019-09-04 17:31:17 +01:00
Nick Craig-Wood
89b6d89077 build: drop support for go1.9 2019-09-04 10:23:48 +01:00
Denis
b71ac141cc copyurl: add --auto-filename flag for using file name from url in destination path (#3451) 2019-09-03 17:25:19 +01:00
Nick Craig-Wood
5932acfee3 rc: fix docs for config/create /update /password 2019-09-03 08:34:15 +01:00
Nick Craig-Wood
8d296d0e1d Start v1.49.1-DEV development 2019-09-02 13:10:47 +01:00
Nick Craig-Wood
50a4ed8fc4 operations: fix accounting for server side copies
See: https://forum.rclone.org/t/b2-server-side-copy-doesnt-show-cumulative-progress/11154
2019-09-02 09:26:33 +01:00
Nick Craig-Wood
693112d57e config: Fix generated passwords being stored as empty password - Fixes #3492 2019-08-28 12:11:03 +01:00
Nick Craig-Wood
d1a39dcc4b Start v1.49.0-DEV development 2019-08-26 17:44:09 +01:00
Nick Craig-Wood
a6387e1f81 Version v1.49.0 2019-08-26 15:25:20 +01:00
Nick Craig-Wood
193c30d570 Review random string/password generation
- factor password generation into lib/random.Password
- call from appropriate places
- choose appropriate use of random.String vs random.Password
2019-08-25 11:09:19 +01:00
Cenk Alti
8159658e67 hash: add CRC-32 support 2019-08-23 12:11:36 +01:00