Commit Graph

3656 Commits

Author SHA1 Message Date
Fabian Möller
a56c9ab61d docs: add section for restricted filenames 2019-09-30 14:05:48 +01:00
Fabian Möller
97a218903c fstest: remove WinPath from fstest.Item 2019-09-30 14:05:48 +01:00
Nick Craig-Wood
4627ac5709 New backend for Citrix Sharefile - Fixes #1543
Many thanks to Bob Droog for organizing a test account and extensive
testing.
2019-09-30 12:28:33 +01:00
Nick Craig-Wood
1e7144eb63 docs: Add more notes on making backend docs 2019-09-30 12:27:03 +01:00
Nick Craig-Wood
f29e5b6e7d lib/oauthutil: refactor web server and allow an auth callback 2019-09-30 11:34:30 +01:00
Nick Craig-Wood
25a0e7e8aa lib/oauthutil: add a new redirect URL for oauth.rclone.org
This is for use with oauth providers which won't accept http: links.
2019-09-30 11:23:21 +01:00
Nick Craig-Wood
262ba28dec config: add ReadNonEmptyLine utility function 2019-09-30 11:23:21 +01:00
Nick Craig-Wood
74f6300875 Start v1.49.4-DEV development 2019-09-29 19:47:59 +01:00
Nick Craig-Wood
86dcb54c38 fstests: make tests pass when using -remote :backend: 2019-09-29 17:25:54 +01:00
Nick Craig-Wood
25a0703b45 Add Richard Patel to contributors 2019-09-29 11:14:33 +01:00
Richard Patel
32d5af8fb6 cmd/rcd: Address ZipSlip vulnerability
Don't create files outside of target
directory while unzipping.

Fixes #3529 reported by Nico Waisman at Semmle Security Team
2019-09-29 11:14:21 +01:00
Richard Patel
44b603d2bd lib: add plugin support
This enables loading plugins from RCLONE_PLUGIN_PATH if set.
2019-09-29 11:05:10 +01:00
Nick Craig-Wood
349112df6b oauthutil: fix security problem when running with two users on the same machine
Before this change two users could run `rclone config` for the same
backend on the same machine at the same time.

User A would get as far as starting the web server.  User B would then
fail to start the webserver, but it would open the browser on the
/auth URL which would redirect the user to the login.  This would then
cause user B to authenticate to user A's rclone.

This changes fixes the problem in two ways.

Firstly it passes the state to the /auth call before redirecting and
checks it there, erroring with a 403 error if it doesn't match.  This
would have fixed the problem on its own.

Secondly it delays the opening of the web browser until after the auth
webserver has started which prevents the user entering the credentials
if another auth server is running.

Fixes #3573
2019-09-29 10:42:02 +01:00
Nick Craig-Wood
fef8b98be2 ftp: fix listing of an empty root returning: error dir not found
Before this change if rclone listed an empty root directory then it
would return an error dir not found.

After this change we assume the root directory exists and don't
attempt to check it which was failing before.

See: https://forum.rclone.org/t/ftp-empty-directory-yields-directory-not-found-error/12069/
2019-09-28 18:01:12 +01:00
Nick Craig-Wood
6750af6167 build: make VERSION file be master of the last release - fixes #3570
Prior to this beta releases would appear to be older than the point
release, eg v1.49.0-096-gc41812fc which was released after v1.49.3 and
contains all the patches from v1.49.3.
2019-09-26 16:51:44 +01:00
Nick Craig-Wood
8681ef36d6 build: replace Circle CI build and make GitHub actions the default CI 2019-09-25 16:38:10 +01:00
Nick Craig-Wood
ec9914205f build: remove Appveyor, Circle CI, Travis and Pkgr builds 2019-09-25 16:38:10 +01:00
Ivan Andreev
ccecfa9cb1 chunker: finish meta-format before release
changes:
- chunker: remove GetTier and SetTier
- remove wdmrcompat metaformat
- remove fastopen strategy
- make hash_type option non-advanced
- adverise hash support when possible
- add metadata field "ver", run strict checks
- describe internal behavior in comments
- improve documentation

note:
wdmrcompat used to write file name in the metadata, so maximum metadata
size was 1K; removing it allows to cap size by 200 bytes now.
2019-09-25 11:03:33 +01:00
Ivan Andreev
c41812fc88 tests: bring memory hungry tests close to end 2019-09-24 12:45:12 +01:00
Ivan Andreev
d98d1be3fe accounting: fix panic due to server-side copy fallback 2019-09-24 12:45:12 +01:00
Ivan Andreev
661dc568f3 fstest: let backends advertise maximum file size 2019-09-24 12:45:12 +01:00
Ivan Andreev
1e4691f951 tests/sync: adjust transfer counts for chunker 2019-09-24 12:45:12 +01:00
Ivan Andreev
be674faff1 tests/config: integration tests for chunker
Recommended `rclone.conf` snippets for this `config.yaml`:
```
[TestChunkerLocal]
type = chunker
meta_format = simplejson
remote = /tmp/rclone-chunker-test

[TestChunkerChunk3bLocal]
type = chunker
chunk_size = 3b
meta_format = simplejson
remote = /tmp/rclone-chunker-test

[TestChunkerNometaLocal]
type = chunker
meta_format = none
remote = /tmp/rclone-chunker-test

[TestChunkerChunk3bNometaLocal]
type = chunker
chunk_size = 3b
meta_format = none
remote = /tmp/rclone-chunker-test

[TestChunkerCompatLocal]
type = chunker
meta_format = wdmrcompat
remote = /tmp/rclone-chunker-test
```
2019-09-24 12:45:12 +01:00
Ivan Andreev
c68c919cea docs: chunker documentation 2019-09-24 12:45:12 +01:00
Ivan Andreev
59dba1de88 chunker: implementation + required fstest patch
Note: chunker implements many irrelevant methods (UserInfo, Disconnect etc),
but they are required by TestIntegration/FsCheckWrap and cannot be removed.

Dropped API methods: MergeDirs DirCacheFlush PublicLink UserInfo Disconnect OpenWriterAt

Meta formats:
- renamed old simplejson format to wdmrcompat.
- new simplejson format supports hash sums and verification of chunk size/count.

Change list:
- split-chunking overlay for mailru
- add to all
- fix linter errors
- fix integration tests
- support chunks without meta object
- fix package paths
- propagate context
- fix formatting
- implement new required wrapper interfaces
- also test large file uploads
- simplify options
- user friendly name pattern
- set default chunk size 2G
- fix building with golang 1.9
- fix ci/cd on a separate branch
- fix updated object name (SyncUTFNorm failed)
- fix panic in Box overlay
- workaround: Box rename failed if name taken
- enhance comments in unit test
- fix formatting
- embed wrapped remote rather than inherit
- require wrapped remote to support move (or copy)
- implement 3 (keep fstest)
- drop irrelevant file system interfaces
- factor out Object.mainChunk
- refactor TestLargeUpload as InternalTest
- add unit test for chunk name formats
- new improved simplejson meta format
- tricky case in test FsIsFile (fix+ignore)
- remove debugging print
- hide temporary objects from listings
- fix bugs in chunking reader:
  - return EOF immediately when all data is sent
  - handle case when wrapped remote puts by hash (bug detected by TestRcat)
- chunked file hashing (feature)
- server-side copy across configs (feature)
- robust cleanup of temporary chunks in Put
- linear download strategy (no read-ahead, feature)
- fix unexpected EOF in the box multipart uploader
- throw error if destination ignores data
2019-09-24 12:45:12 +01:00
Fionera
49d6d6425c serve/httplib: Write the template to a buffer to catch render errors
Fixes #3559
2019-09-22 21:31:11 +01:00
Nick Craig-Wood
28cc2009d4 Add Anthony Rusdi to contributors 2019-09-21 14:39:03 +01:00
Nick Craig-Wood
dd4fe9ff60 Add David to contributors 2019-09-21 14:39:03 +01:00
Anthony Rusdi
899f285319 s3: fix signature v2_auth headers
When used with v2_auth = true, PresignRequest doesn't return
signed headers, so remote dest authentication would be fail.
This commit copying back HTTPRequest.Header to headers.

Tested with RiakCS v2.1.0.

Signed-off-by: Anthony Rusdi <33247310+antrusd@users.noreply.github.com>
2019-09-21 14:38:51 +01:00
David
4788545b05 box: add options to get access token via JWT auth 2019-09-20 17:15:16 +01:00
David
1934426789 jwtutil: functionality to get an access token via JWT authentication 2019-09-20 17:15:16 +01:00
David
643192b347 vendor: add pkcs8 helpers for decrypting encrypted private keys 2019-09-20 17:15:16 +01:00
Nick Craig-Wood
1031bcfc5a build: remove azure pipelines build 2019-09-20 16:08:18 +01:00
Nick Craig-Wood
ce00c0a0d9 build: build rclone with github actions 2019-09-20 16:08:18 +01:00
Nick Craig-Wood
1164eed2af lib/pacer: make tests more reliable 2019-09-20 16:07:55 +01:00
Nick Craig-Wood
557edecd40 log: add Stack() function for debugging who calls what 2019-09-20 11:53:08 +01:00
Nick Craig-Wood
b242b0a078 lib/cache,rc/jobs: make tests more reliable 2019-09-20 11:53:08 +01:00
Nick Craig-Wood
08b86cc94b mount: skip tests on <= 2 CPUs to avoid lockup in #3154 2019-09-20 11:53:08 +01:00
Nick Craig-Wood
56544bb2fd accounting: fix file handle leak on errors - fixes #3547
In 53a1a0e3ef we introduced a problem where if there was an
error on the file being transferred then the file was re-opened and
the old one wasn't closed.

This was partially fixed in bfbddab46b however this didn't
address the case of the old file being closed.

This is now fixed by
- marking the file as open again in UpdateReader
- moving the stopping the accounting machinery to a new method Done
2019-09-19 16:20:07 +01:00
Matei David
70e043e641 Fixed typo in Docker image doc. 2019-09-19 16:17:57 +01:00
Dan Walters
c49a71f438 dlna: move root descriptor xml template to the static assets
Reduce binary size.
2019-09-17 12:52:32 +01:00
Dan Walters
5f07bbf8ce dlna: fake out implementation of X_MS_MediaReceiverRegistrar
Using the same responses as minidlna.

Fixes #3502.
2019-09-17 12:52:02 +01:00
Dan Walters
2f10472df3 dlna: count the number of children in the response to BrowseMetadata 2019-09-17 12:28:20 +01:00
Matei David
ab89e93968 Add Matei David to contributors 2019-09-17 10:12:32 +01:00
Matei David
070a8bfcd8 Dockerfile fixes
- ref: https://forum.rclone.org/t/run-docker-container-in-userspace/11734/7
- enable userspace operation
- enable Docker userspace mount exposed to the host
- add more Docker image usage documentation
2019-09-17 10:12:32 +01:00
Ivan Andreev
8fe87c8157 mailru: skip extra http request if data fits in hash 2019-09-17 10:04:51 +01:00
Ivan Andreev
8fb44a822d mailru: fix rare nil pointer panic 2019-09-17 10:04:51 +01:00
Nick Craig-Wood
3cff258577 sftp: fix --sftp-ask-password trying to contact the ssh agent
See: https://forum.rclone.org/t/rclone-command-line/11766
2019-09-16 11:16:27 +01:00
Nick Craig-Wood
66347aff2a fstest: calculate hashes for uploaded test files to fix minio integration tests
Before this change we didn't calculate any hashes for test files
created in the Run framework.

This means that files were uploaded to S3 without a `Content-MD5`
header.  This in turn caused minio to disengage `--compat` mode which
in turn caused the `TestSyncAfterChangingModtimeOnlyWithNoUpdateModTime`
test to fail in `fs/sync`.

After this change we supply all hashes supported by the destination Fs
on the upload object.

This means that the `Content-MD5` is set and minio engages `--compat`
mode to fix the problem.  Using `--compat` on the command line also
fixes the problem.

This much better replicates how objects are actually uploaded with
operations.Copy so should improve the integration tests.
2019-09-16 10:59:01 +01:00
Nick Craig-Wood
b8b12a4000 Start v1.49.3-DEV development 2019-09-15 18:10:08 +01:00