Commit Graph

159 Commits

Author SHA1 Message Date
negative0
6272ca74bc plugins: Move plugins cache path initialization to initPluginsOrError.
Fixes #4951.
2021-01-28 16:58:23 +00:00
Aleksandar Jankovic
38e70f1797 rc/jobs: add listener for finished jobs
Add jobs.OnFinish method to register listener that will trigger when job
is finished.

Includes fix for stopping listeners.
2021-01-21 13:43:41 +00:00
Nick Craig-Wood
a7689d7023 rcserver: fix 500 error when marshalling errors from core/command
Before this change attempting to return an error from core/command
failed with a 500 error and a message about unmarshable types.

This is because it was attempting to marshal the input parameters
which get _response added to them which contains an unmarshalable
field.

This was fixed by using the original parameters in the error response
rather than the one modified during the error handling.

This also adds end to end tests for the streaming facilities as used
in core/command.
2021-01-10 16:34:46 +00:00
Nick Craig-Wood
847a44e7ad fs/rc: add Copy method to rc.Params 2021-01-10 16:34:46 +00:00
Nick Craig-Wood
b3710c962e rc: fix core/command giving 500 internal error - fixes #4914
Before this change calling core/command gave the error

    error: response object is required expecting *http.ResponseWriter value for key "_response" (was *http.response)

This was because the http.ResponseWriter is an interface not an object.

Removing the `*` fixes the problem.

This also indicates that this bit of code wasn't properly tested.
2021-01-10 16:34:46 +00:00
negative0
ed7af3f370 plugins: Create plugins files only if webui is enabled. Fixes #4592. May fix #4600. 2020-12-27 15:05:41 +00:00
Nick Craig-Wood
354b4f19ec rc: add context to flag Reload function #4685 2020-11-27 17:28:42 +00:00
Nick Craig-Wood
2e21c58e6a fs: deglobalise the config #4685
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
2020-11-26 16:40:12 +00:00
Nick Craig-Wood
1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
d69b96a94c test: Add context to mockfs.NewFs #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
d846210978 fs: Add context to NewFs #3257 #4685
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
2020-11-09 18:05:54 +00:00
Josh Soref
e4a87f772f docs: spelling: e.g.
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Chaitanya Bankanhal
c8a719ae0d webui: Prompt user for updating webui if an update is available 2020-09-07 16:45:00 +01:00
Nick Craig-Wood
344bce7e2a docs: fix formatting of rc docs page
See: https://forum.rclone.org/t/rclone-1-53-release/18880/24
2020-09-03 11:53:24 +01:00
Nick Craig-Wood
75d54d720c version: replace internal code with github.com/coreos/go-semver
We were already importing go-semver so it makes sense to remove the
duplicated semver parsing code and just use go-semver
2020-08-31 13:55:04 +01:00
Nick Craig-Wood
cc0421cb9e rc/webgui: skip AddPlugin and RemovePlugin tests if download fails 2020-08-31 13:45:06 +01:00
Nick Craig-Wood
9c01ac9894 rc/webgui: improve error handling on web fetches 2020-08-31 13:45:06 +01:00
Chaitanya Bankanhal
20300d1f61 plugins: Change failing plugin test to new repo rclone/rclone-test-plugin 2020-08-31 13:45:06 +01:00
Chaitanya Bankanhal
6231beefc5 webui: Fix broken webui because of plugins redirection 2020-08-31 13:45:06 +01:00
Aaron Gokaslan
bf685f600e webgui: fixes previously unhandled error in JSON marshall in fs/rc/webgui/plugins.go:writeToFile 2020-08-30 17:15:03 +01:00
Chaitanya Bankanhal
d6996e3347 plugins: Add url query params to regex for referrer path 2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal
dffcc99373 plugins: Create availablePlugins config file if it does not exist. 2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal
09b79679cd plugins: restructure and add tests for pluginsctl/* calls 2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal
cf68e61f40 Add redirection for plugin urls 2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal
22674d1146 plugins: Add reverse proxy pluginsHandler for serving plugins 2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal
f9ee0dc3f2 plugins: allow installation and use of plugins and test plugins with rclone-webui 2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal
65fa6a946a webui: Expose webui downloader and other utility for use with plugins 2020-08-24 10:56:04 +01:00
Chaitanya
4cf82118d9 rc: add plugins support 2020-08-24 10:56:04 +01:00
Chaitanya
5f56611a76 webgui: Move to new package fs/rc/webgui. 2020-08-24 10:56:04 +01:00
Chaitanya Bankanhal
6308153ae7 rc: pass response writer when needsResponse is set instead of pointer
rc: Fix failing tests for *http.ResponseWriter
2020-08-17 17:09:31 +01:00
Chaitanya Bankanhal
a9713cd0ed core/command: Add streaming output for long running commands. 2020-08-17 17:09:31 +01:00
Chaitanya Bankanhal
1cae4152f9 rc: add NeedsResponse for rc calls 2020-08-17 17:09:31 +01:00
Nick Craig-Wood
4884bee8ba core/command: pretend to be "rclone version" to make tests pass 2020-08-17 17:09:31 +01:00
Chaitanya Bankanhal
54fc2821cd core/command: Add version command instead of ls 2020-08-17 17:09:31 +01:00
Chaitanya Bankanhal
5549fd25fc core/command: Allow rc to execute rclone terminal commands.
Allow command parameter to be skipped.
2020-08-17 17:09:31 +01:00
Nick Craig-Wood
aab9aa8a2e js: add experimental interface for integrating rclone into browsers
This works by compiling rclone to wasm and exporting the RC api to
javascript.
2020-08-10 17:32:21 +01:00
Nick Craig-Wood
744828a4de rc: allow JSON parameters to simplify command line usage
If the parameter being passed is an object then it can be passed as a
JSON string rather than using the `--json` flag which simplifies the
command line.

rclone rc operations/list fs=/tmp remote=test opt='{"showHash": true}'

Rather than

rclone rc operations/list --json '{"fs": "/tmp", "remote": "test", "opt": {"showHash": true}}'
2020-07-22 18:40:52 +01:00
Chaitanya Bankanhal
fa43d02874 rc: Add operations/uploadfile to upload a file through rc using encoding multipart/form-data 2020-06-25 11:38:24 +01:00
Chaitanya
d0de39ebcd rc: add NeedsRequest to call. 2020-06-25 11:38:24 +01:00
Nick Craig-Wood
b58bb03e95 test: Don't run unreliable tests on CI #4171 2020-06-15 21:34:37 +01:00
Roman Kredentser
55ad1354b6 link: Add --expire and --unlink flags
This adds expire and unlink fields to the PublicLink interface.

This fixes up the affected backends and removes unlink parameters
where they are present.
2020-06-05 14:51:05 +01:00
Martin Michlmayr
041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Martin Michlmayr
4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
calisro
c80b6d96dd
http: improved directory listing with new template from Caddy project
This includes a new directory listing template which was originally
from the Caddy project (used with permission and copyright attribution).

This is used whenever we serve directory listings so `rclone serve
http`, `rclone serve webdav` and `rclone rcd --rc-serve`

This also modifies the tests so they work with the original template which
is easier to debug.
2020-05-08 16:15:21 +01:00
ElonH
d119bfd934 rcd: disable duplicate log
if running `rclone rcd --rc-user=admin --rc-pass=admin
--rc-allow-origin="*"`, lots of duplicate warnings apperent in log

Warning: Allow origin set to *. This can cause serious security problems.
Warning: Allow origin set to *. This can cause serious security problems.
....

This is not conducive to analyzing debugging info.

Therefore, let's show it only once.
2020-05-05 13:47:25 +00:00
Nick Craig-Wood
195d152785 rc: add GetStructMissingOK 2020-04-29 09:42:31 +01:00
Xiaoxing Ye
c4572ebc91
rc: fix misplaced http server config - fixes #4130 2020-04-23 20:22:47 +01:00
Gary Kim
38a4d50e73 rcd: Add Prometheus metrics support - fixes #3858
Signed-off-by: Gary Kim <gary@garykim.dev>
2020-03-01 09:58:34 +00:00
Lars Lehtonen
fe6f4135b4 fs/rc: fix dropped error 2020-02-04 11:31:06 +00:00
Xiaoxing Ye
ccaca04a5d rcd: move webgui apart; option to disable browser
Fix #3601, #3785
2020-01-20 12:27:55 +00:00
Nick Craig-Wood
199ac61bde rc: add methods to turn on blocking and mutex profiling 2019-10-28 22:11:38 +00:00
Nick Craig-Wood
1868c77e16 rc: fix formatting of docs 2019-10-27 10:43:40 +00:00
Saksham Khanna
0c265713fd rc: added command core/quit 2019-10-17 15:04:22 +01:00
Nick Craig-Wood
b242b0a078 lib/cache,rc/jobs: make tests more reliable 2019-09-20 11:53:08 +01:00
Nick Craig-Wood
ffa1dac10b build: apply gofmt from go1.13 to change case of number literals 2019-09-05 13:59:06 +01:00
Michał Matczuk
3247e69cf5 fs/rc/jobs: ExecuteJob propagate the error returned by function
Without this patch the resulting error is first converted to string and then recreated.
This makes it impossible to use the defined error types to figure out the cause of the error,
and may result in invalid HTTP status codes.

This patch adds a test TestExecuteJobErrorPropagation to validate that the errors are
properly propagated.
2019-08-22 16:10:48 +01:00
Chaitanya
159f2e29a8 rcd: prefix patch for rcd and web-gui 2019-08-22 08:36:10 +01:00
Chaitanya
efd826ad4b rcd: auto-login for web-gui
rcd: auto use authentication if none is provided for web-gui
2019-08-22 08:36:10 +01:00
Michal Matczuk
5d6593de4f * rc/jobs: Add SetInitialJobID function that allows for setting the jobID 2019-08-21 11:01:39 +01:00
Nick Craig-Wood
3ecbd603ab rc: move job expire flags to rc to fix initalization problem
See: https://forum.rclone.org/t/rc-rc-job-expire-interval-bug/11188

rclone was ignoring the --rc-job-expire-duration and --rc-job-interval
flags.  This turned out to be an initialization order problem and was
fixed by moving those flags out of global config into rc config.
2019-08-10 17:12:22 +01:00
Nick Craig-Wood
0693deea1c rc: fix unmarshalable http.AuthFn in options and put in test for marshalability 2019-08-10 16:22:17 +01:00
Chaitanya
33677ff367 rc: Added command line parameter to control the cross origin resource sharing (CORS) in the rcd. (Security Improvement)
rc: Import statements


Fixing the problem with test
2019-08-09 15:14:52 +01:00
negative0
27a075e9fc rcd: Removed the shorthand for webgui. Shorthand is reserved for rsync compatibility. 2019-08-06 12:50:31 +01:00
Nick Craig-Wood
5065c422b4 lib/random: unify random string generation into random.String
This was factored from fstest as we were including the testing
enviroment into the main binary because of it.

This was causing opening the browser to fail because of 8243ff8bc8.
2019-08-06 12:44:08 +01:00
Chaitanya Bankanhal
d51a970932 rcd: Change URL after webgui move to rclone organization 2019-08-05 16:22:40 +01:00
Chaitanya Bankanhal
84e2806c4b rc: Rclone-WebUI integration with rclone
This adds experimental support for web gui integration so that rclone can fetch and run a web based GUI using the --rc-web-ui and related flags.

It downloads and caches a webui zip file which it then unpacks and opens in the browser.
2019-08-04 12:32:37 +01:00
Aleksandar Jankovic
6a3e301303 accounting: add call to clear stats
- Make calls more consistent by changing path to kebab case.
- Add stacktrace information to job panics
2019-08-02 16:56:19 +01:00
Nick Craig-Wood
57d5de6fba build: fix up package paths after repo move
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
2019-07-28 18:47:38 +01:00
Aleksandar Jankovic
8243ff8bc8 accounting: isolate stats to groups
Introduce stats groups that will isolate accounting for logically
different transferring operations. That way multiple accounting
operations can be done in parallel without interfering with each other
stats.

Using groups is optional. There is dedicated global stats that will be
used by default if no group is specified. This is operating mode for CLI
usage which is just fire and forget operation.

For running rclone as rc http server each request will create it's own
group. Also there is an option to specify your own group.
2019-07-28 14:48:19 +01:00
buengese
19ae053168 rcserver: remove _async key from input parameters after parsing so later operations won't get confused - fixes #3346 2019-07-20 19:35:10 +02:00
Aleksandar Jankovic
5935cb0a29 jobs: add ability to stop async jobs
Depends on #3257
2019-06-19 14:17:41 +01:00
Aleksandar Jankovic
f78cd1e043 Add context propagation to rclone
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
2019-06-19 11:59:46 +01:00
Aleksandar Jankovic
93207ead9c rc/jobs: make job expiry timeouts configurable 2019-06-15 09:55:32 +01:00
Nick Craig-Wood
f681d32996 rc: Fix serving bucket based objects with --rc-serve
Before this change serving bucket based objects
`[remote:bucket]/path/to/object` would fail with 404 not found.

This was because the leading `/` in `/path/to/object` was being passed
to NewObject.
2019-06-10 11:59:06 +01:00
Nick Craig-Wood
206e1caa99 fs/cache: factor Fs caching from fs/rc into its own package 2019-05-27 16:14:55 +01:00
Nick Craig-Wood
577fda059d rc: fix race in tests 2019-05-01 16:09:50 +01:00
Nick Craig-Wood
9df322e889 tests: make test servers choose a random port to make more reliable
Tests have been randomly failing with messages like

    listen tcp 127.0.0.1:51778: bind: address already in use

Rework all the test servers so they choose a random free port on
startup and use that for the tests to avoid.
2019-05-01 16:09:50 +01:00
Fabian Möller
48c09608ea
fix spelling 2019-04-30 14:12:18 +02:00
Nick Craig-Wood
16d8014cbb build: drop support for go1.8 2019-04-15 21:49:58 +01:00
Nick Craig-Wood
0d475958c7 Fix errors discovered with go vet nilness tool 2019-03-18 11:23:00 +00:00
Nick Craig-Wood
cccadfa7ae rc: add ability for options blocks to register reload functions 2019-03-10 13:09:44 +00:00
Nick Craig-Wood
6e153781a7 rc: add help to show how to set log level with options/set 2019-02-03 11:47:57 +00:00
Nick Craig-Wood
97752ca8fb build: fix problems found with ineffasign linter 2019-02-02 11:45:07 +00:00
Jay
082a7065b1 Use vfsgen for static HTML templates 2018-12-26 15:07:21 +00:00
Nick Craig-Wood
57b85b8155 rc: fix job tests on Windows 2018-11-06 13:03:48 +00:00
Nick Craig-Wood
bc17ca7ed9 rc: implement core/obscure 2018-11-05 15:44:40 +00:00
Nick Craig-Wood
1916410316 rc: add core/version and put definitions next to implementations 2018-11-05 15:44:40 +00:00
Nick Craig-Wood
75a88de55c rc/rcserver: with --rc-files if auth set, pass on to URL opened
If `--rc-user` or `--rc-pass` is set then the URL that is opened with
`--rc-files` will have the authorization in the URL in the
`http://user:pass@localhost/` style.
2018-11-05 15:44:40 +00:00
Nick Craig-Wood
fa0a9653d2 rc: methods marked as AuthRequired need auth unless --rc-no-auth
Methods which can read or mutate external storage will require
authorisation - enforce this.  This can be overidden by `--rc-no-auth`.
2018-11-04 20:42:57 +00:00
Nick Craig-Wood
75e8ea383c rc: implement rc.PutCachedFs for prefilling the remote cache 2018-11-04 20:42:57 +00:00
Nick Craig-Wood
8c8b58a7de rc: expire remote cache and fix tests under race detector 2018-11-04 20:42:57 +00:00
Nick Craig-Wood
b961e07c57 rc: ensure rclone fails to start up if the --rc port is in use already 2018-11-04 15:11:51 +00:00
Nick Craig-Wood
89550e7121 rcserver: serve directories as well as files 2018-11-04 15:11:51 +00:00
Nick Craig-Wood
b972dcb0ae rc: implement options/blocks,get,set and register options 2018-11-03 11:32:00 +00:00
Nick Craig-Wood
0bfa9811f7 rc: factor server code into rcserver and implement serving objects
If a GET or HEAD request is receivied with a URL parameter of fs then
it will be served from that remote.
2018-11-03 11:32:00 +00:00
Nick Craig-Wood
75252e4a89 rc: add --rc-files flag to serve files on the rc http server
This enables building a browser based UI for rclone
2018-11-03 11:32:00 +00:00
Nick Craig-Wood
2089405e1b fs/rc: add more infrastructure to help writing rc functions
- Fs cache for rc commands
- Helper functions for parsing the input
- Reshape command for manipulating JSON blobs
- Background Job starting, control, query and expiry
2018-11-02 17:32:20 +00:00
Nick Craig-Wood
ade252f13b build: fixup code formatting after goimports change 2018-10-13 22:47:12 +01:00
frenos
382a6863b5 rc: add support for OPTIONS and basic CORS - #2575 2018-09-24 17:04:47 +01:00
Nick Craig-Wood
8c0335a176 build: fix for goimports format change
See https://github.com/golang/go/issues/23709
2018-07-03 22:33:15 +01:00