1346 Commits

Author	SHA1	Message	Date
Nick Craig-Wood	b167d30420	Add client side TLS/SSL flags --ca-cert/--client-cert/--client-key ... Fixes #2966	2019-02-12 12:26:19 +00:00
Nick Craig-Wood	076d3da825	operations: resume downloads if the reader fails in copy - fixes #2108 ... This puts a shim on the reader opened by Copy so that if an error is returned, the reader is re-opened at the correct seek point. This should make downloading very large files more reliable.	2019-02-12 11:47:57 +00:00
Nick Craig-Wood	d29c545627	Start v1.46-DEV development	2019-02-09 12:21:57 +00:00
Nick Craig-Wood	eb85ecc9c4	Version v1.46	2019-02-09 10:42:57 +00:00
Nick Craig-Wood	8ec55ae20b	Fix broken flag type tests ... Introduced in fc1bf5f931	2019-02-07 16:42:26 +00:00
Nick Craig-Wood	fc1bf5f931	Make flags show up with their proper names, eg SizeSuffix rather than int	2019-02-07 11:57:26 +00:00
Nick Craig-Wood	bdc638530e	walk: make NewDirTree always use ListR #2946 ... This fixes vfs/refresh with recurse=true needing the --fast-list flag	2019-02-04 10:37:27 +00:00
Nick Craig-Wood	2135879dda	lsjson: use exactly the correct number of decimal places in the seconds	2019-02-03 20:03:23 +00:00
Cnly	318d1bb6f9	fs: clarify behaviour of Put() and Upload() for unknown-sized objects	2019-02-03 11:47:57 +00:00
Cnly	b8b53901e8	operations: call Rcat in Copy when size is -1 - #2832	2019-02-03 11:47:57 +00:00
Nick Craig-Wood	6e153781a7	rc: add help to show how to set log level with options/set	2019-02-03 11:47:57 +00:00
Nick Craig-Wood	eb91356e28	fs/asyncreader: optionally user mmap for memory allocation with --use-mmap #2200 ... This replaces the `sync.Pool` allocator with lib/pool. This implements a pool of buffers of up to 64MB which can be re-used but is flushed every 5 seconds. If `--use-mmap` is set then rclone will use mmap for memory allocations which is much better at returning memory to the OS.	2019-02-02 14:35:56 +00:00
Nick Craig-Wood	d966cef14c	build: fix problems found with unconvert	2019-02-02 11:45:07 +00:00
Nick Craig-Wood	97752ca8fb	build: fix problems found with ineffasign linter	2019-02-02 11:45:07 +00:00
Nick Craig-Wood	bbd03f49a4	operations: Implement DirMove for moving a directory #2539 ... This does the equivalent of sync.Move but is specialised for moving files in one backend.	2019-01-27 21:26:56 +00:00
Nick Craig-Wood	a30e80564d	config: when using auto confirm make user interaction configurable ... * drive: don't run teamdrive config if auto confirm set * onedrive: don't run extra config if auto confirm set * make Confirm results customisable by config Fixes #1010	2019-01-18 14:46:26 +00:00
Nick Craig-Wood	5990573ccd	accounting: fix layout of stats - fixes #2910 ... This fixes several things wrong with the layout of the stats. Transfers which haven't started are printed in the same format as those which have so the stats with `--progress` don't show horrible artifacts. Checkers and transfers now get a ": checkers" and ": transfers" label on the end of the stats line. Transfers will have the transfer stats when the transfer has started instead of this. There was a bug in the routine which shortened the file names (it always produces strings 1 too long). This is now fixed with a test. The formatting string was wrong with a fixed width of 45 - this is now replaces with the value of `--stats-file-name-length`. This also meant that there were unecessary leading spaces in the file names. So the default `--stats-file-name-length` was raised to 45 from 40.	2019-01-14 16:12:39 +00:00
qip	f471a7e3f5	fshttp: Add cookie support with cmdline switch --use-cookies ... Cookies are handled by cookiejar in memory with fshttp module through the entire session. One useful scenario is, with HTTP storage system where index server adds authentication cookie while redirecting to CDN for actual files. Also, it can be helpful to reuse fshttp in other storage systems requiring cookie.	2019-01-11 17:35:29 +00:00
Dario Guzik	7782eda88e	check: Add stats showing total files matched.	2019-01-11 14:13:48 +00:00
Nick Craig-Wood	ff72059a94	operations: warn if --checksum is set but there are no hashes available ... Also caveat the help of --checksum Fixes #2903	2019-01-10 11:07:10 +00:00
Denis Skovpen	2d2533a08a	cmd/copyurl: fix checking of --dry-run	2019-01-08 11:28:05 +00:00
Nick Craig-Wood	2d01a65e36	oauthutil: read a fresh token config file before using the refresh token. ... This means that rclone will pick up tokens from concurrently running rclones. This helps for Box which only allows each refresh token to be used once. Without this fix, rclone caches the refresh token at the start of the run, then when the token expires the refresh token may have been used already by a concurrently running rclone. This also will retry the oauth up to 5 times at 1 second intervals. See: https://forum.rclone.org/t/box-token-refresh-timing/8175	2019-01-08 11:01:30 +00:00
Jay	082a7065b1	Use vfsgen for static HTML templates	2018-12-26 15:07:21 +00:00
Nick Craig-Wood	5ee1816a71	filter: parallelise reading of --files-from - fixes #2835 ... Before this change rclone would read the list of files from the files-from parameter and check they existed one at a time. This could take a very long time for lots of files. After this change, rclone will check up to --checkers in parallel.	2018-12-13 13:22:30 +00:00
Nick Craig-Wood	107293c80e	copy,move: restore --no-traverse flag ... The --no-traverse flag was not implemented when the new sync routines (using the march package) was implemented. This re-implements --no-traverse in march by trying to find a match for each object with NewObject rather than from a directory listing.	2018-12-02 20:28:13 +00:00
Nick Craig-Wood	e3c4ebd59a	march: factor calling parameters into a structure	2018-12-02 18:07:26 +00:00
Nick Craig-Wood	01fcad9b9c	rc: fix docs for sync/{sync,copy,move} and operations/{copy,move}file	2018-11-29 11:11:30 +00:00
Nick Craig-Wood	a6c28a5faa	Start v1.45-DEV development	2018-11-24 15:20:24 +00:00
Nick Craig-Wood	d35bd15762	Version v1.45	2018-11-24 13:44:25 +00:00
Peter Kaminski	63a2a935fc	fix typos in original files, per #2727 review request	2018-11-14 22:48:58 +00:00
Nick Craig-Wood	f92beb4e14	fstest: Fix TestPurge causing errors with subsequent tests on azure ... Before this change TestPurge would remove a container and subsequent tests would fail because the container was still being deleted so couldn't be created. This was fixed by introducing an fstest.NewRunIndividual() test runner for TestPurge which causes the test to be run on a new container.	2018-11-14 17:14:02 +00:00
Nick Craig-Wood	16f797a7d7	filter: add --ignore-case flag - fixes #502 ... The --ignore-case flag causes the filtering of file names to be case insensitive. The flag name comes from GNU tar.	2018-11-12 14:29:37 +00:00
Nick Craig-Wood	4ad08794c9	fserrors: add "server closed idle connection" to retriable errors ... This seems to be related to this go issue: https://github.com/golang/go/issues/19943 See: https://forum.rclone.org/t/copy-from-dropbox-to-google-drive-yields-failed-to-copy-failed-to-open-source-object-server-closed-idle-connection-error/7460	2018-11-08 11:12:25 +00:00
Nick Craig-Wood	c6786eeb2d	move: don't create directories with --dry-run - fixes #2676	2018-11-06 13:34:15 +00:00
Nick Craig-Wood	57b85b8155	rc: fix job tests on Windows	2018-11-06 13:03:48 +00:00
Nick Craig-Wood	e6dd121f52	config: add rc operations for config	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	e600217666	config: create config directory on save if it is missing	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	bc17ca7ed9	rc: implement core/obscure	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	1916410316	rc: add core/version and put definitions next to implementations	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	dddfbec92a	cmd/version: factor version number parsing routines into fs/version	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	75a88de55c	rc/rcserver: with --rc-files if auth set, pass on to URL opened ... If `--rc-user` or `--rc-pass` is set then the URL that is opened with `--rc-files` will have the authorization in the URL in the `http://user:pass@localhost/` style.	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	2466f4d152	sync: add rc commands for sync/copy/move	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	39283c8a35	operations: implement operations remote control commands	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	46c2f55545	copyurl: factor code into operations and write test	2018-11-04 20:42:57 +00:00
Nick Craig-Wood	fc2afcbcbd	lsjson: factor internals of lsjson command into operations	2018-11-04 20:42:57 +00:00
Nick Craig-Wood	fa0a9653d2	rc: methods marked as AuthRequired need auth unless --rc-no-auth ... Methods which can read or mutate external storage will require authorisation - enforce this. This can be overidden by `--rc-no-auth`.	2018-11-04 20:42:57 +00:00
Nick Craig-Wood	75e8ea383c	rc: implement rc.PutCachedFs for prefilling the remote cache	2018-11-04 20:42:57 +00:00
Nick Craig-Wood	8c8b58a7de	rc: expire remote cache and fix tests under race detector	2018-11-04 20:42:57 +00:00
Nick Craig-Wood	b961e07c57	rc: ensure rclone fails to start up if the --rc port is in use already	2018-11-04 15:11:51 +00:00
Nick Craig-Wood	89550e7121	rcserver: serve directories as well as files	2018-11-04 15:11:51 +00:00
Nick Craig-Wood	b972dcb0ae	rc: implement options/blocks,get,set and register options	2018-11-03 11:32:00 +00:00
Nick Craig-Wood	0bfa9811f7	rc: factor server code into rcserver and implement serving objects ... If a GET or HEAD request is receivied with a URL parameter of fs then it will be served from that remote.	2018-11-03 11:32:00 +00:00
Nick Craig-Wood	75252e4a89	rc: add --rc-files flag to serve files on the rc http server ... This enables building a browser based UI for rclone	2018-11-03 11:32:00 +00:00
Nick Craig-Wood	2089405e1b	fs/rc: add more infrastructure to help writing rc functions ... - Fs cache for rc commands - Helper functions for parsing the input - Reshape command for manipulating JSON blobs - Background Job starting, control, query and expiry	2018-11-02 17:32:20 +00:00
ssaqua	3d81b75f44	dedupe: check for existing filename before renaming a dupe file	2018-11-02 16:51:52 +00:00
Nick Craig-Wood	948a5d25c2	operations: Fix Purge and Rmdirs when dir is not empty ... Before this change, Purge on the fallback path would try to delete directories starting from the root rather than the dir passed in. Rmdirs would also attempt to delete the root.	2018-10-27 11:51:17 +01:00
Nick Craig-Wood	543e37f662	Require go1.8 for compilation	2018-10-25 17:06:33 +01:00
Nick Craig-Wood	c5ac96e9e7	Make --files-from only read the objects specified and don't scan directories ... Before this change using --files-from would scan all the directories that the files could possibly be in causing rclone to do more work that was necessary. After this change, rclone constructs an in memory tree using the --fast-list mechanism but from all of the files in the --files-from list and without scanning any directories. Any objects that are not found in the --files-from list are ignored silently. This mechanism is used for sync/copy/move (march) and all of the listing commands ls/lsf/md5sum/etc (walk).	2018-10-20 18:13:31 +01:00
Nick Craig-Wood	67703a73de	Start v1.44-DEV development	2018-10-15 12:33:27 +01:00
Nick Craig-Wood	f96ce5674b	Version v1.44	2018-10-15 11:03:08 +01:00
Nick Craig-Wood	ade252f13b	build: fixup code formatting after goimports change	2018-10-13 22:47:12 +01:00
Fabian Möller	57273d364b	fstests: add TestFsPutChunked	2018-10-11 14:47:58 +01:00
Fabian Möller	1a40bceb1d	backend: unify NewFs path handling for wrapping remotes ... Use the same function to join the root paths for the wrapping remotes alias, cache and crypt. The new function fspath.JoinRootPath is equivalent to path.Join, but if the first non empty element starts with "//", this is preserved to allow Windows network path to be used in these remotes.	2018-10-10 17:50:27 +01:00
Nick Craig-Wood	6709084e2f	config: Show URL of backend help page when starting config	2018-10-06 11:47:46 +01:00
Nick Craig-Wood	6b1f915ebc	fs: Implement RegInfo.FileName to return the on disk filename for a backend ... Use it in make_backend_docs.py	2018-10-06 11:47:46 +01:00
Nick Craig-Wood	1557287c64	fs: Make Option.GetValue() public #2541	2018-10-06 11:47:45 +01:00
Nick Craig-Wood	e7e467fb3a	cmd: factor FlagName into fs.Option #2541	2018-10-06 11:47:45 +01:00
Nick Craig-Wood	59e14c25df	vfs: enable rename for nearly all remotes using server side Move or Copy ... Before this change remotes without server side Move (eg swift, s3, gcs) would not be able to rename files. After it means nearly all remotes will be able to rename files on rclone mount with the notable exceptions of b2 and yandex. This changes checks to see if the remote can do Move or Copy then calls `operations.Move` to do the actual move. This will do a server side Move or Copy but won't download and re-upload the file. It also checks to see if the destination exists first which avoids conflicts or duplicates. Fixes #1965 Fixes #2569	2018-09-29 14:56:20 +01:00
Fabian Möller	1eec59e091	fs: update ChangeNotifier interface ... This introduces a channel to the ChangeNotify function, which can be used to update the poll-interval and cleanly exit the polling function.	2018-09-25 14:01:13 +02:00
ssaqua	ae63e4b4f0	list: change debug logs for excluded items	2018-09-24 17:08:35 +01:00
frenos	382a6863b5	rc: add support for OPTIONS and basic CORS - #2575	2018-09-24 17:04:47 +01:00
sandeepkru	e0c5f7ff1b	fs - Remove unreferenced ListTierer optional interface	2018-09-18 13:56:09 +01:00
Joanna Marek	e022ffce0f	accounting: change too long names cutting mechanism - fixes #2490	2018-09-18 10:27:23 +01:00
sandeepkru	7accd30da8	cmd and fs: Added new command settier which performs storage tier changes on ... supported remotes	2018-09-12 21:09:08 +01:00
sandeepkru	5716a58413	fs: Added new optional interfaces SetTierer, GetTierer and ListTierer, these are to ... perform object tier changes on supported remotes	2018-09-12 21:09:08 +01:00
Cédric Connes	1b95ca2852	stats: handle FatalError and NoRetryError when reported to stats	2018-09-07 14:44:50 +01:00
albertony	09d7bd2d40	config: don't create default config dir when user supplies --config ... Avoid creating empty default configuration directory when user supplies path to config file. Fixes #2514.	2018-09-03 15:30:53 +01:00
dcpu	f4aaec9ce5	log: Add --log-format flag - fixes #2424	2018-09-02 18:11:09 +01:00
Nick Craig-Wood	543d75723b	Start v1.43-DEV development	2018-09-01 18:37:48 +01:00
Nick Craig-Wood	20c55a6829	Version v1.43	2018-09-01 12:58:00 +01:00
Nick Craig-Wood	7d74686698	fs/accounting: increase maximum burst size of token bucket ... This stops occasional errors when using --bwlimit which look like this Token bucket error: rate: Wait(n=2255475) exceeds limiter's burst 2097152	2018-08-30 17:24:08 +01:00
Denis	1c578ced1c	cmd: add copyurl command - Fixes #1320	2018-08-30 16:45:41 +01:00
Nick Craig-Wood	de6ec8056f	fs/accounting: fix moving average speed for file stats ... Before this change the moving average for the individual file stats would start at 0 and only converge to the correct value over 15-30 seconds. This change starts the weighting period as 1 and moves it up once per sample which gets the average to a better value instantly.	2018-08-28 22:55:51 +01:00
Nick Craig-Wood	66fe4a2523	fs/accounting: fix stats display which was missing transferring data ... Before this change the total stats were ignoring the in flight checking, transferring and bytes.	2018-08-28 22:21:17 +01:00
Nick Craig-Wood	f5617dadf3	fs/accounting: factor out eta and percent calculations and write tests	2018-08-28 22:21:17 +01:00
Nick Craig-Wood	ca44fb1fba	accounting: fix time to completion estimates ... Previous to this change package used for this github.com/VividCortex/ewma took a 0 average to mean reset the statistics. This happens quite often when transferring files though a buffer. Replace that implementation with a simple home grown one (with about the same constant), without that feature.	2018-08-26 22:00:48 +01:00
Nick Craig-Wood	94950258a4	fs: allow backends to be named using their Name or Prefix #2449 ... This means that, for example Google Cloud Storage can be known as `:gcs:bucket` on the command line, as well as `:google cloud storage:bucket`.	2018-08-26 17:59:31 +01:00
Nick Craig-Wood	8656bd2bb0	fs: Allow on the fly remotes with :backend: syntax - fixes #2449 ... This change allows remotes to be created on the fly without a config file by using the remote type prefixed with a : as the remote name, Eg :s3: to make an s3 remote. This assumes the user is supplying the backend config via command line flags or environment variables.	2018-08-26 17:59:31 +01:00
Nick Craig-Wood	b6db90cc32	cmd: add --progress/-P flag to show progress ... Fixes #2347 Fixes #1210	2018-08-26 17:20:38 +01:00
Nick Craig-Wood	9f671c5dd0	fs: fix tests for *SepList	2018-08-21 10:58:59 +01:00
Alex Chen	c6c74cb869	mountlib: fix mount --daemon not working with encrypted config - fixes #2473 ... This passes the configKey to the child process as an Obscured temporary file with an environment variable to the	2018-08-21 09:41:16 +01:00
Nick Craig-Wood	1b8f4b616c	fs: move CommaSepList and SpaceSepList here from config ... fs can't import config so having them there means they are not usable by rclone core.	2018-08-20 17:52:05 +01:00
Fabian Möller	f818df52b8	config: add List type	2018-08-20 17:38:51 +01:00
Nick Craig-Wood	7712a0e111	fs/asyncreader: skip some tests to work around race detector bug ... The race detector currently detects a race with len(chan) against close(chan). See: https://github.com/golang/go/issues/27070 Skip the tests which trip this bug under the race detector.	2018-08-20 12:34:29 +01:00
Fabian Möller	6dc5aa7454	docs: clearify buffer-size is per transfer/filehandle	2018-08-17 18:11:40 +01:00
Fabian Möller	552eb8e06b	vfs: try to seek buffer on read only files	2018-08-17 18:10:28 +01:00
Nick Craig-Wood	6390dec7db	fs/accounting: add --stats-one-line flag for single line stats	2018-08-17 17:58:00 +01:00
Nick Craig-Wood	80a3db34a8	fs/accounting: show the total progress of the sync in the stats #379	2018-08-17 17:58:00 +01:00
Nick Craig-Wood	cb7a461287	sync: add a buffer for checks, uploads and renames #379 ... --max-backlog controls the queue length. Add statistics for the check/upload/rename queues. This means that checking can complete before the uploads which will give rclone the ability to show exactly what is outstanding.	2018-08-17 17:58:00 +01:00
Andres Alvarez	990919f268	Add disclaimer about generated passwords being stored in an obscured format	2018-08-11 15:07:50 +01:00
Sebastian Bünger	dd3e912731	fs/OpenOptions: Make FixRangeOption clamp range to filesize.	2018-08-10 11:14:34 +01:00
Nick Craig-Wood	e979cd62c1	rc: fix formatting in docs	2018-08-07 21:05:21 +01:00
reddi1	6bfdabab6d	rc: added core/stats to return the stats - fixes #2405	2018-08-07 20:56:40 +01:00
Andres Alvarez	c1f5add049	Add tests for reveal functions	2018-08-07 20:51:50 +01:00
Oleg Kovalov	06c9f76cd2	all: fix go-critic linter suggestions	2018-08-06 21:14:03 +01:00
Fabian Möller	9815b09d90	fs: add multipliers for SizeSuffix	2018-07-29 18:17:05 +01:00
Fabian Möller	9c90b5e77c	stats: use appropriate Lock func's	2018-07-22 11:33:19 +02:00
Nick Craig-Wood	ab78eb13e4	sync: correct help for --delete-during and --delete-after	2018-07-18 19:30:14 +01:00
ishuah	dcc74fa404	move: fix delete-empty-src-dirs flag to delete all empty dirs on move - fixes #2372	2018-07-17 10:34:34 +01:00
Nick Craig-Wood	4d7d240c12	config: Add advanced section to the config editor	2018-07-16 21:20:47 +01:00
Nick Craig-Wood	d046402d80	config: Make sure Required values are entered	2018-07-16 21:20:47 +01:00
Nick Craig-Wood	9bdf465c10	config: make config wizard understand types and defaults	2018-07-16 21:20:47 +01:00
Nick Craig-Wood	f3f48d7d49	Implement new backend config system ... This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet	2018-07-16 21:20:47 +01:00
Nick Craig-Wood	3c89406886	config: Make fs.ConfigFileGet return an exists flag	2018-07-16 08:50:52 +01:00
Nick Craig-Wood	85d09729f2	fs: factor OptionToEnv and ConfigToEnv into fs	2018-07-16 08:50:52 +01:00
Nick Craig-Wood	b3bd2d1c9e	config: add configstruct parser to parse maps into config structures	2018-07-16 08:50:52 +01:00
Nick Craig-Wood	4c586a9264	config: add configmap package to manage config in a generic way	2018-07-16 08:50:52 +01:00
Nick Craig-Wood	1c80e84f8a	fs: Implement Scan method for SizeSuffix and Duration	2018-07-16 08:50:52 +01:00
Nick Craig-Wood	dbb4b2c900	fs/config: don't print errors about --config if supplied - fixes #2397 ... Before this change if the rclone was running in an environment which couldn't find the HOME directory, it would print a warning about supplying a --config flag even if the user had done so.	2018-07-15 12:39:11 +01:00
Nick Craig-Wood	a3d9a38f51	fs/fserrors: make sure Cause never returns nil	2018-07-13 10:31:40 +01:00
Nick Craig-Wood	144c1a04d4	fs: Fix parsing of paths under Windows - fixes #2353 ... Before this copyto would parse windows paths incorrectly. This change moves the parsing code into fspath and makes sure fspath.Split calls fspath.Parse which does the parsing correctly for This also renames fspath.RemoteParse to fspath.Parse for consistency	2018-07-06 23:16:43 +01:00
Nick Craig-Wood	8c0335a176	build: fix for goimports format change ... See https://github.com/golang/go/issues/23709	2018-07-03 22:33:15 +01:00
Nick Craig-Wood	947e10eb2b	config: fix error reading password from piped input - fixes #1308	2018-06-28 11:54:15 +01:00
Nick Craig-Wood	935533e57f	filter: raise --include and --exclude warning to ERROR so it appears without -v	2018-06-22 22:18:55 +01:00
Mateusz	56e1e82005	fs: added weekday schedule into --bwlimit - fixes #1822	2018-06-17 18:38:09 +01:00
Nick Craig-Wood	7110349547	Start v1.42-DEV development	2018-06-16 21:25:58 +01:00
Nick Craig-Wood	a9adb43896	Version v1.42	2018-06-16 18:21:09 +01:00
Nick Craig-Wood	1c75581959	sync: fix TestCopyRedownload after ModifyWindow changes #2310	2018-06-10 17:34:00 +01:00
Filip Bartodziej	ce109ed9c0	log: password prompt output fixed for unix - partially fixes #2220	2018-06-10 12:57:45 +01:00
Nick Craig-Wood	ba43acb6aa	sync: fix TestCopyEmptyDirectories after ModifyWindow changes #2310	2018-06-04 21:41:25 +01:00
ishuah	d758e1908e	copy: create (pseudo copy) empty source directories to destination - fixes #1837	2018-06-04 11:01:14 +01:00
ishuah	737aed8412	Ensure items in srcEmptyDirs are actually empty	2018-06-04 11:01:14 +01:00
Stefan	4009fb67c8	fs: calculate ModifyWindow each time on the fly instead of relying on global state - see #2319, #2328	2018-06-03 20:45:34 +02:00
Nick Craig-Wood	3ef938ebde	lsf: add --absolute flag to add a leading / onto path names	2018-06-03 10:42:34 +01:00
Kasper Byrdal Nielsen	aad75e6720	check: Add one-way argument ... --one-way argument will check that all files on source matches the files on detination, but not the other way. For example files present on destination but not on source will not trigger an error. Fixes: #1526	2018-05-31 09:42:16 +01:00
Nick Craig-Wood	3a2248aa5f	rc: add core/gc to run a garbage collection on demand	2018-05-24 15:10:28 +01:00
Nick Craig-Wood	573ef4c8ee	rc: enable go profiling by default on the --rc port ... This means you can use the pprof tool on a running rclone, eg go tool pprof http://localhost:5572/debug/pprof/heap	2018-05-24 15:10:28 +01:00
Nick Craig-Wood	512f4b4487	Update error checking on fmt.Fprint* after errcheck update ... Now we need to check or ignore errors on fmt.Fprint* explicitly - previously errcheck just ignored them for us.	2018-05-22 09:41:13 +01:00
Nick Craig-Wood	8e625e0bc3	config: add ConfirmWithDefault to change the default on AutoConfig	2018-05-16 09:09:41 +01:00
Nick Craig-Wood	e56be0dfd8	lsf: Add --csv flag for compliant CSV output	2018-05-13 12:18:21 +01:00
Nick Craig-Wood	2a32e2d838	operations: turn ListFormatted into a Format method on ListFormat	2018-05-13 12:17:55 +01:00
Nick Craig-Wood	f77efc7649	lsf: Add 'm' format specifier to show the MimeType	2018-05-13 12:17:55 +01:00
Nick Craig-Wood	aadbcce486	fs: Add MimeTypeDirEntry to return the MimeType of a DirEntry	2018-05-13 12:17:55 +01:00
Nick Craig-Wood	909c3a92d6	lsf: implement 'i' format for showing object ID - fixes #1476	2018-05-13 12:17:55 +01:00
Nick Craig-Wood	826975c341	fs: add Optional ID() method to Object and implement it in backends ... ID() shows the internal ID of the Object if available.	2018-05-13 12:17:55 +01:00
Nick Craig-Wood	e80ae4e09c	build: remove unused struct fields spotted by structcheck	2018-05-05 17:32:41 +01:00
Nick Craig-Wood	1320e84bc2	build: remove unused code spotted by the deadcode linter	2018-05-05 17:32:41 +01:00
Nick Craig-Wood	cb5bd47e61	build: fix errors spotted by ineffassign linter ... These were mostly caused by shadowing err and a good fraction of them will have caused errors not to be propagated properly.	2018-05-05 17:32:41 +01:00
Nick Craig-Wood	f95835d613	fserrors: Look deeper into errors for Fatal/Retry/NoRetry errors. ... Before this change fatal errors which were wrapped in a system error (eg a URLError) were not recognised as fatal errors.	2018-05-05 12:58:28 +01:00
Nick Craig-Wood	be79b47a7a	sync: log when we abandon the sync due to a fatal error	2018-05-05 12:58:28 +01:00
Nick Craig-Wood	be22735609	fs/accounting: fix deadlock on GetBytes ... A deadlock could occur since we have now put a mutex on GetBytes from StatsInfo.String (s.mu) - progress (acc.statmu) and read (acc.statmu) - GetBytes (s.mu). Fix this by giving stringSet its own locking and excluding the call which caused the deadlock from the mutex in StatsInfo.String.	2018-05-05 12:58:28 +01:00
Nick Craig-Wood	1b1b3c13cd	sync: add a test for aborting on max upload	2018-05-05 12:58:28 +01:00
Nick Craig-Wood	5c128272fd	Implement --max-transfer flag to quit transferring at a limit #1655	2018-05-05 12:58:28 +01:00
Nick Craig-Wood	d178233e74	sync,march: check the cancel context on every channel send and receive ... This fixes a deadlock on sync when all the copying channels receive a Fatal Error.	2018-05-05 12:58:28 +01:00
Fabian Möller	9fdf273614	fs: improve ChunkedReader ... - make Close permanent and return errors afterwards - use RangeSeek from the wrapped reader if present - add a limit to chunk growth - correct RangeSeek interface behavior - add tests	2018-05-05 12:49:42 +01:00
Nick Craig-Wood	55674c0bfc	Start v1.41-DEV development	2018-04-28 12:37:55 +01:00
Nick Craig-Wood	e4c380b2a8	Version v1.41	2018-04-28 11:46:27 +01:00
Nick Craig-Wood	74cbdea0ef	Revert "copy: create (pseudo copy) empty source directories to destination" ... Unfortunately this commit attempts to create every directory rather than just the empty ones, so will need re-working. Removing this feature for the 1.41 release This reverts commit 0daced29db.	2018-04-28 10:02:32 +01:00
ishuah	0daced29db	copy: create (pseudo copy) empty source directories to destination - fixes #1837	2018-04-27 16:15:32 +01:00
Richard Yang	a81ec00a8c	dedupe: Add dedupe largest functionality - fixes #2269	2018-04-26 16:21:07 +01:00
Nick Craig-Wood	9f04ce282e	rc: fix setting bwlimit to unlimited	2018-04-26 12:21:29 +01:00
Nick Craig-Wood	764440068e	filter: fix --min-age and --max-age together check ... Somehow in the code reorganisation of 11da2a6c9b the check for --min-age and --max-age got switched around. This commit fixes that and means you can use --min-age and --max-age together.	2018-04-26 09:17:22 +01:00
Nick Craig-Wood	a703216286	filter: take double negatives out of filter flag help	2018-04-26 09:17:13 +01:00
Nick Craig-Wood	9ab2521ef2	rc: autogenerate and tidy the docs and commands ... * Rename rc/pid -> core/pid * Sort the output of `rc list` * Make a script to autogenerate the docs * Tidy docs	2018-04-23 20:57:17 +01:00
Nick Craig-Wood	21a10e58c9	rc: implement core/memstats to print internal memory usage info	2018-04-23 20:49:36 +01:00
Nick Craig-Wood	24980d7123	config: fix typo in error message #2268	2018-04-21 22:49:30 +01:00
Nick Craig-Wood	bcdb7719c6	fs/hash: install QuickXorHash as a supported rclone hash type #2262	2018-04-20 21:02:57 +01:00
Nick Craig-Wood	c51d97c752	hashsum: make generic tool for any hash to produce md5sum like output	2018-04-20 21:02:37 +01:00
Nick Craig-Wood	ef3bcec76c	fs: Extend SizeSuffix to include TB and PB for rclone about	2018-04-17 21:53:42 +01:00
Nick Craig-Wood	1ac6dacf0f	about: complete other providers and re-work internals ... * Implement about for: * local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox * Implement `--json` and `---full` flag for `rclone about` * change About interface to return a Usage structure * Remove operations.About as it is too thin an interface * Implement Integration test Relates to #1138 and #1564	2018-04-17 21:53:27 +01:00
a-roussos	94e277d759	about: add new command 'about' to get quota info from a remote ... Implemented for drive only. Relates to #1138 and #1564.	2018-04-17 21:50:14 +01:00
Nick Craig-Wood	8c3740c2c5	config: Improve the Provider matching to have a negated match #2140 ... This makes it easier to make classes of provider in the config.	2018-04-13 16:06:37 +01:00
Giri Badanahatti	acd5d4377e	config,s3: hierarchical configuration support #2140 ... This introduces a method of making provider specific configuration within a remote. This is useful particularly in s3. This commit does the basic configuration in S3 for IBM COS.	2018-04-13 16:05:35 +01:00
Peter Baumgartner	1db68571fd	s3,swift: Add --use-server-modtime ... `--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.	2018-04-13 13:32:17 +01:00
Nick Craig-Wood	d97fe3b824	fs/operations: make dedupe work with mega ... * factor into its own files * remove assumptions about having a given hash type * make tests work if the remote has no hash	2018-04-13 13:23:55 +01:00
Nick Craig-Wood	dcf8334673	fs: add --dump goroutines and --dump openfiles ... These are developer flags useful for tracking down resource leaks.	2018-04-11 20:55:58 +01:00
Nick Craig-Wood	37be78705d	fs/fshttp: limit MaxIdleConns and MaxIdleConnsPerHost ... Before this change mega (which uses a different host per download) would open too many sockets.	2018-04-11 20:51:28 +01:00
Michael P. Dubner	92b266d361	rc: new call rc/pid - closes #2211	2018-04-09 13:33:04 +01:00
Nick Craig-Wood	cbec59146a	lsf: make sure we use localtime in tests - fixes Box integration tests ... This problem was introduced with eca99b33c0. It seems Box is the only remote which converts time zones, so if you give it a GMT time zone, it returns a PST time zone which represents the same instant.	2018-04-09 11:46:49 +01:00
Nick Craig-Wood	be54fd8f70	Remove builds conditional on go1.7 since that is now guaranteed #2154 ... Old fallback code was deleted and the go1.7 style code inlined where appropriate.	2018-04-07 11:42:55 +01:00
Nick Craig-Wood	e5be471ce0	Use io.SeekStart/End/Current constants now for go1.7+ #2154	2018-04-07 11:42:36 +01:00
Nick Craig-Wood	80588a5a6b	Replace "golang.org/x/net/context" with "context" for go1.7+ #2154	2018-04-07 11:42:08 +01:00
Nick Craig-Wood	67023f0040	Require go1.7 for compilation #2154 ... * Update the travis tests to exclude go1.6 * Update the compile check to require go1.7+ * Update misc go1.6 workarounds marked in the source	2018-04-06 20:18:14 +01:00
Matt Holt	e13f65b953	serve restic: Print actual listener address	2018-04-04 14:56:26 +01:00
Nick Craig-Wood	eca99b33c0	lsd,lsf: make sure all times we output are in local time - fixes #2183 ... Previous to this change times from lsd/lsf were output in whatever timezone they were in whereas times from lsl were converted to localtime.	2018-04-01 15:40:04 +01:00
Stefan	a8267d1628	link: allow creating public link to files and folders - closes #1562	2018-03-29 09:10:19 +02:00
Mateusz Pabian	91b068ad3a	sync: implement --ignore-errors - fixes #642	2018-03-23 22:01:10 +00:00
Nick Craig-Wood	940df88eb2	Start v1.40-DEV development	2018-03-19 14:20:48 +00:00
Nick Craig-Wood	26f1c55987	Version v1.40	2018-03-19 10:06:13 +00:00
Stefan	86892467d9	config: load config file only on first access (closes #1659, closes #2096) (#2147)	2018-03-17 12:36:30 +01:00
Nick Craig-Wood	4295428a0f	fs/accounting: add remote control of bwlimit	2018-03-17 10:34:02 +00:00
Nick Craig-Wood	5bf639048f	sync: log an error that --track-renames doesn't work with sync or move ... Fixes #2008	2018-03-17 10:34:02 +00:00
remusb	97b48cf988	rc: add support for Go 1.6	2018-03-14 22:58:20 +02:00
Nick Craig-Wood	86e5a35491	Implement Remote Control for rclone #2111 ... This implements a remote control protocol activated with the --rc flag and a new command `rclone rc` to use that interface. Still to do * docs - need finishing * tests	2018-03-14 22:58:20 +02:00
Nick Craig-Wood	dfd0f4c5a4	sync: when using --backup-dir don't delete files if we can't set their modtime ... This is a problem when syncing a file which just needed its modtime set with dropbox which can't set the mod time of a file without re-uploading it. Before this change we would delete the file, then the server side move would fail moving the file to the backup-dir because it no longer existed. After this change the destination file is moved to the backup-dir instead of being deleted and the new file is uploaded. Fixes #2134	2018-03-13 16:05:06 +00:00
Nick Craig-Wood	0c9dc006c5	fs: make display of default values of --min-age/--max-age be off - Fixes #2121	2018-03-13 09:06:07 +00:00
Remus Bunduc	70f07fd3ac	fs: add ChangeNotify and backend support for it (#2094) ... * fs: rename DirChangeNotify to ChangeNotify * cache: switch to ChangeNotify * ChangeNotify: keep order of notifications	2018-03-08 22:03:34 +02:00
Nick Craig-Wood	e0eb666dbf	fs/walk: fix new golint warning about unused variables in range	2018-03-02 17:01:58 +00:00
Nick Craig-Wood	abc736df1d	cat: Use RangeOption for limited fetches to make more efficient #1825	2018-02-19 18:00:39 +00:00
Nick Craig-Wood	ab0d06eb16	fs: Make RangeOption mandatory #1825	2018-02-19 18:00:39 +00:00
Mateusz	afc963ed92	config: retry saving the config after failure - fixes #2060	2018-02-19 17:59:27 +00:00
Fabian Möller	451cd6d971	fs: add ChunkedReader	2018-02-19 15:45:24 +00:00
Fabian Möller	a647c54888	fs: add RangeSeeker interface	2018-02-19 15:45:24 +00:00
Nick Craig-Wood	a5c3bcc9c7	fshttp: fix idle timeouts for HTTP connections #2057 ... Now we only nudge on the idle timeout after a successful Read or Write which returns some bytes and no errors.	2018-02-16 10:35:41 +00:00
Bjørn Erik Pedersen	ab8c0a81fa	Add a delete threshold to sync (--max-delete) ... Fixes #959	2018-02-12 11:29:58 +00:00
Nick Craig-Wood	d0d6b83a7a	fs/accounting: rework to enable accounting to work with crypt and b2 ... This removes the old system of part accounting and replaces it with a system of popping off the accounting reader and wrapping up new ones as necessary. This makes it much easier to carry the context down the chain of wrapped readers and get the limiting as near as possible to the output. This makes the accounting more accurate and the bandwidth limiting smoother. Fixes #2029 and Fixes #1443	2018-02-02 15:14:41 +00:00
Nick Craig-Wood	bea02fcf52	fs/accounting: factor into separate files without changing functionality	2018-02-02 15:14:40 +00:00
Nick Craig-Wood	6fb868e00c	config: fix --log-level flag after code reorganization - fixes #2043	2018-02-02 14:07:44 +00:00
Nick Craig-Wood	8df78f2b6d	operations: ignore size of objects when they are < 0 #320 ... This allows google docs to be transferred and checked correctly.	2018-01-31 16:22:05 +00:00
Nick Craig-Wood	2eb5cfb7ad	fs: Formalize the ObjectUnWrapper interface	2018-01-31 16:21:41 +00:00
Nick Craig-Wood	fe52502f19	fs: Adjust RangeOption.Decode to return -1 for read to end ... A Range request can never request 0 bytes however this change was made to make a clearer signal that the limit means read to the end. Add test and more documentation and fixup uses	2018-01-27 14:31:29 +00:00
Nick Craig-Wood	9a73688e3a	fs: Add ParseRangeOption to parse incoming Range: requests	2018-01-27 13:16:37 +00:00
Nick Craig-Wood	bc3ee977f4	fs/hash: move interface assertion to tests so it doesn't pull in spf13/flag	2018-01-26 14:35:18 +00:00
Nick Craig-Wood	a6833b68ca	local: factor RangeOption code to Decode() method and readers.LimitedReadCloser #1825	2018-01-22 19:44:00 +00:00
Nick Craig-Wood	d876392d15	onedrive: Factor code into fs.FixRangeOption	2018-01-22 17:05:00 +00:00
Nick Craig-Wood	84bc4dc142	Clarify RangeOption semantics	2018-01-21 09:51:28 +00:00
Nick Craig-Wood	ee66419a27	fs/fserrors: Add test for error from #1964	2018-01-19 17:07:40 +00:00
Nick Craig-Wood	5ad226ab54	fs: Add dir option to fs.Purge #1891 ... Purge optional interface needs fixing too.	2018-01-19 11:45:50 +00:00
Nick Craig-Wood	97c414f025	config/hash: rename more symbols after factoring into own package	2018-01-18 20:27:52 +00:00
Nick Craig-Wood	71722b5b95	config: factor Obscure and Reveal into its own package	2018-01-18 20:19:55 +00:00
Ernest Borowski	5cce74d630	flags: remove --no-traverse flag because it is obsolete - fixes #1813 ... Signed-off-by: Ernest Borowski <er.borowski@gmail.com>	2018-01-18 11:00:25 +00:00
Iakov Davydov	acd55a8f65	local, fs: --exclude-if-present ignores directories which it doesn't have permission for - fixes #1959	2018-01-16 20:00:16 +00:00
Nick Craig-Wood	11da2a6c9b	Break the fs package up into smaller parts. ... The purpose of this is to make it easier to maintain and eventually to allow the rclone backends to be re-used in other projects without having to use the rclone configuration system. The new code layout is documented in CONTRIBUTING.	2018-01-15 17:51:14 +00:00
Nick Craig-Wood	b8b620f5c2	Move all backends into backend directory	2018-01-12 17:07:38 +00:00
Nick Craig-Wood	712e6a8085	lsf: fix integration tests	2018-01-11 13:52:15 +00:00
Will Gunn	5c7e8d5a2b	fs: Add --stats-file-name-length to specify the printed file name length for stats ... Fixes #1206	2018-01-10 20:32:36 +00:00
Nick Craig-Wood	c74c3b37da	lsf: add option to print hashes	2018-01-06 17:53:37 +00:00
Nick Craig-Wood	7c71ee1a5b	fs: fix TestListFormat on remotes which return 0 as dir size not -1	2018-01-06 17:47:42 +00:00
Jakub Tasiemski	0d041602cf	cmd: new command lsf	2018-01-06 14:39:31 +00:00
Nick Craig-Wood	b412c745a1	Start v1.39-DEV development	2017-12-23 13:40:28 +00:00
Nick Craig-Wood	f34a9116d4	Version v1.39	2017-12-23 13:07:45 +00:00
Nick Craig-Wood	7e585cda96	fs: fix TestRmdirsLeaveRoot test	2017-12-14 08:57:28 +00:00
lewapm	9c242edc10	rmdirs: add --leave-root flag - fixes #1874	2017-12-13 10:23:54 +00:00
ishuah	b05e472d2e	stats: condensed transfer output to fit 80x25 terminals	2017-12-09 10:48:36 +03:00
remusb	25b073c767	fs: add Wrap feature for FS to identify their parent FS (#1884)	2017-12-06 17:14:34 +02:00
Nick Craig-Wood	fa4a25a73b	fs: only test one level of cache ... Can't test multiple caches at once as can only have 1 DB open at once	2017-12-04 15:50:59 +00:00
Ernest Borowski	934df67aef	filter: warn the user if he use --include and --exclude together fixes #1764 ... Signed-off-by: Ernest Borowski <er.borowski@gmail.com>	2017-12-04 14:20:01 +00:00
Nick Craig-Wood	4b185355df	fs: rcat - use in memory object and Copy for more reliable transfers	2017-12-03 12:14:15 +00:00
Nick Craig-Wood	7d15c33e42	fs: make Copy and Move return the destination object if possible	2017-12-03 12:14:15 +00:00
Nick Craig-Wood	11332a19a0	fs: make an in memory object for short transfers	2017-12-03 12:14:15 +00:00
Nick Craig-Wood	639e812789	fs: integration tests: add SUMMARY heading for log scraping	2017-11-29 15:55:37 +00:00
Nick Craig-Wood	1c6cad2252	fs: integration tests: add 30 minute timeout per test	2017-11-29 13:51:17 +00:00
ishuah	aab8051f50	move: add --delete-empty-src-dirs flag - fixes #1854	2017-11-28 11:38:19 +03:00
Nick Craig-Wood	fdb01437d8	fs: Allow the http Transport to have an optional filter request function	2017-11-24 09:07:56 +00:00
Nick Craig-Wood	02ffd43572	fs: Save the config before asking for a token - fixes #1220 ... Before this if the client_id/client_secret was edited it would disappear when asking for the new token. This means the post config is done after the user has confirmed the config is OK which can't be helped.	2017-11-23 14:01:32 +00:00
Nick Craig-Wood	e53892f53b	fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use ... RepeatableReaderSized has a pre-allocated buffer which should help with memory usage - before it grew the buffer. Since we know the size of the chunks, pre-allocating it should be much more efficient. RepeatableReaderBuffer uses the buffer passed in. RepeatableLimit* are convenience funcitions for wrapping a reader in an io.LimitReader and then a RepeatableReader with the same buffer size.	2017-11-23 13:53:46 +00:00
ishuah	6c62fced60	move: fixed root source directories getting deleted after move - fixes #1849	2017-11-23 12:01:35 +03:00
Nick Craig-Wood	159fce0106	fs: fix --cache-dir to have some effect	2017-11-22 17:05:02 +00:00
Nick Craig-Wood	cbe5d7ce64	fs: Remove X-Auth-Token: from headers when dumping for swift	2017-11-21 17:32:07 +00:00
Nick Craig-Wood	1a65a4e769	fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters ... Now --dump-flag is written as --dump flag. This is a comma separated list which can contain * headers - HTTP headers as before * bodies - HTTP bodies as before * requests - HTTP request bodies * responses - HTTP response bodies * auth - HTTP auth * filters - Filter rexeps Leave --dump-headers and --dump-bodies for the time being but remove the other --dump-* flags as they aren't used very often.	2017-11-21 17:32:07 +00:00
ishuah	b4aa920a3d	stats: show the amount of data transferred in kb/mb - fixes #1167	2017-11-21 12:40:02 +03:00
Nick Craig-Wood	8951875c21	vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables	2017-11-20 12:46:21 +00:00
Nick Craig-Wood	e946a8eab0	fs: Add CacheDir config variable	2017-11-20 12:00:32 +00:00
Nick Craig-Wood	46947b3b9b	rcat: fix goroutine leak ... This was leaking goroutines in the short file case beause it wasn't calling Close() on the Account object. This became apparent when testing with mount.	2017-11-20 12:00:32 +00:00
Iakov Davydov	7d3abdc463	tests for --exclude-if-present	2017-11-17 21:51:11 +00:00
Iakov Davydov	538246f6c3	support exclude file in --fast-list mode	2017-11-17 21:51:11 +00:00
Iakov Davydov	557dd8f031	ListDirSorted check for excludefile	2017-11-17 21:51:11 +00:00
Iakov Davydov	37aaa19f3a	new option: --exclude-if-present	2017-11-17 21:51:11 +00:00
Iakov Davydov	cef2e3bf83	path -> startPath in walkRDirTree (we need the path package)	2017-11-17 21:51:11 +00:00
Iakov Davydov	a3a436ce16	WalkRDirTree: return error if unknown item type	2017-11-17 21:51:11 +00:00
Iakov Davydov	5d05df3124	ListContainsExcludeFile: checks for exclude file in the list	2017-11-17 21:51:11 +00:00
Iakov Davydov	421ba84e12	DirTree.Prune: deletes several directories	2017-11-17 21:51:11 +00:00
Iakov Davydov	7ae7080824	FileExists check if a file exists	2017-11-17 21:51:11 +00:00
Remus Bunduc	016abf825e	cache: first version	2017-11-15 15:23:21 +00:00
ishuah	aec2265be0	rclone: implement exit codes - #1136	2017-11-15 17:48:37 +03:00
Substantiel	2423fa40e2	config: add password sub command for setting obscured passwords	2017-11-15 14:44:45 +00:00
ishuah	c65734ee69	move: delete source directory after successful move - fixes #1642	2017-11-07 22:21:38 +00:00
Substantiel	7749157596	Add --auto-confirm flag	2017-11-05 21:56:50 +00:00
Nick Craig-Wood	cc7ed13b9b	fs: factor test running code into fstest/run.go	2017-11-03 12:59:58 +00:00
ishuah	10f27e2ff2	allow trailing+leading whitespace for passwords - #1717 ... warn users when they enter passwords with leading/trailing whitespaces Updated config_test.go, removing deprecated test case and updated TestReveal	2017-10-20 15:46:17 +01:00
Nick Craig-Wood	115053930e	Make error messages less crypting when revealing an unobscured password - fixes #1743	2017-10-16 22:03:06 +01:00
Nick Craig-Wood	edfab09eb9	config: add sub commands for full config file management ... Previously config sub commands were manually parsed rather than using cobra. Make config command have the following sub commands: * create Create a new remote with name, type and options. * delete Delete an existing remote <name>. * dump Dump the config file as JSON. * edit Enter an interactive configuration session. * file Show path of configuration file in use. * providers List in JSON format all the providers and options. * show Print (decrypted) config file, or the config for a single remote. * update Update options in an existing remote. The following changes were made to existing commands * listproviders was renamed to providers * listoptions was removed in favour of providing the output in providers * jsonconfig was renamed to create * an optional parameter was added to the show command	2017-10-14 11:50:41 +01:00
thierry	0575623dff	Add config listproviders, listoptions, jsonconfig for automated config ... Addition of a method listing the providers, a method listing the options of a provider and method of manual configuration.	2017-10-13 17:17:36 +01:00
Nick Craig-Wood	fc8b13c993	moveto/copyto: Fix to allow copying to the same name - fixes #1736	2017-10-12 20:45:36 +01:00
Nick Craig-Wood	43ced30f11	fs: Add more errors to retry - fixes #1733	2017-10-10 19:51:02 +01:00
Nick Craig-Wood	3eb7f52e39	fs: Add "unexpected EOF reading trailer" as a retriable error - fixes #1730	2017-10-09 17:29:16 +01:00
Nick Craig-Wood	7f3dc9b5c4	Implement WebDAV remote #580 ... This has special knowledge of Owncloud and Nextcloud to enable more functionality such as mod times.	2017-10-09 16:19:37 +01:00
Nick Craig-Wood	00fe6d95da	fs: fix duplicate files causing spurious copies ... Before this fix duplicate files (on Google Drive) caused the next file to be spuriously copied. `rclone dedupe` worked around the problem.	2017-10-02 16:52:53 +01:00
Nick Craig-Wood	9e9297838f	Implement pcloud remote - #418	2017-10-01 11:37:35 +01:00
Nick Craig-Wood	1c10497b68	Start v1.38-DEV development	2017-09-30 15:16:09 +01:00
Nick Craig-Wood	d96e45ba5b	Version v1.38	2017-09-30 14:20:43 +01:00
Nick Craig-Wood	657b3a674d	fs: fix test_all -clean to run just one cleaning thread per remote	2017-09-30 11:07:09 +01:00
Nick Craig-Wood	2857ed5c35	fs: fix --immutable tests on remotes which don't have modtime	2017-09-28 08:56:30 +01:00
Jacob McNamee	2d8e75cab4	Implement --immutable option	2017-09-24 20:00:00 +01:00
Stefan	85877f3adc	config: add show/file subcommands which print the config/its path (fixes #1086)	2017-09-19 17:59:19 +02:00
Nick Craig-Wood	87335de8a8	fs: fix filename normalization issues in the tests when running on OS X	2017-09-17 15:31:22 +01:00
Stefan Breunig	12405f9f41	fuse: re-use rcat to support uploads for all remotes (fixes #1672)	2017-09-16 22:49:08 +02:00
Stefan	234bfae0d5	b2: implement streaming upload of files with unknown length (see #1614) (closes #1686)	2017-09-16 22:43:48 +02:00
Nick Craig-Wood	4ac9a65049	fs: stop normalizing file names but do a normalized compare in the sync ... This works by using a transform function to transform file names when doing a compare when matching file names in a directory. rclone now UTF-8 normalizes the file names and does a case insensitive compare if the destination remote is case insensitive. This deprecates the --local-no-unicode-normalization flag. Fixes #1477	2017-09-16 19:49:31 +01:00
Nick Craig-Wood	a8e41f081c	fs: re-implement check and cryptcheck using the same traversal as sync ... This makes them 100% consistent with sync and also make them use less memory as they no longer build the whole tree in memory first. Fixes #1657	2017-09-16 19:49:31 +01:00
Nick Craig-Wood	261c7ad9e4	fs: make syncCopyMove use context for go routine cancellation	2017-09-16 19:49:31 +01:00
Nick Craig-Wood	fe96d5cf0a	fs: factor multiple directory traverse out of sync	2017-09-16 19:49:31 +01:00
Nick Craig-Wood	6df12b3f00	fs: improve retriable error detection	2017-09-16 19:48:49 +01:00
Stefan Breunig	9d3f11b493	amazonclouddrive, rcat: ensure rcat integration test passes even with AmazonCloudDrive (fixes: #1680)	2017-09-15 18:09:04 +02:00
Nick Craig-Wood	798502b204	fs: add more errors to be considered temporary errors ... This makes a framework for adding temporary errors identified by syscall number or by error string. Fixes #1660	2017-09-14 18:01:43 +01:00
Stefan Breunig	56dedc49e3	rcat: properly report if the upload fails	2017-09-13 20:21:52 +02:00
Girish Ramakrishnan	4c45cbea18	copy: error out if dst could not be listed	2017-09-12 09:29:44 +01:00
Stefan Breunig	a91448c83a	rcat: honor --dry-run even for small files	2017-09-11 22:28:16 +02:00
Stefan Breunig	80b1f2a494	rcat: configurable small files cutoff and implement proper upload verification	2017-09-11 08:26:53 +02:00
Stefan Breunig	57817397a0	rcat: directly upload small files without streaming them	2017-09-11 08:25:34 +02:00
Nick Craig-Wood	cbea06026a	Make check obey --ignore-size - fixes #1643	2017-09-01 11:20:41 +01:00
Nick Craig-Wood	2c8d6e86cc	fs: fix gofmt	2017-08-31 16:01:19 +01:00
cbruegg	bb6300b032	Fix bwlimit toggle in conjunction with schedules (Fixes #1607)	2017-08-31 15:33:29 +01:00
Stefan Breunig	a122b9fa7a	yandex: implement streaming uploads (see #1614)	2017-08-19 14:07:23 +02:00
Alex McGrath Kraak	2cc9071791	http: add --user-agent option. close #1557	2017-08-17 06:49:27 +02:00
Nick Craig-Wood	265fb8a5e2	fs: Manage empty directories - fixes #100 ... During the sync we collect a list of directories which should be empty and attempt to rmdir them at the end of the sync. If the directories are not empty then the rmdir will fail, logging a message but not erroring the sync.	2017-08-09 21:07:00 +01:00
Nick Craig-Wood	8a1a900733	fstest: use Feature.CanHaveEmptyDirectories to sharpen tests ... Now we actually test whether the directories are present or not, filtering out empty directories in the test using the CanHaveEmptyDirectories flag.	2017-08-09 20:55:08 +01:00
Nick Craig-Wood	20ae7d562b	fs: Add CanHaveEmptyDirectories and BucketBased feature flags to all remotes	2017-08-09 20:55:08 +01:00
Nick Craig-Wood	ec2ea37ad2	fs: Add --disable flag to disable optional features - fixes #1551 ... Eg to disable server side copy use `--disable copy`, to see a list of what you can disable, `--disable help`.	2017-08-07 21:34:45 +01:00
Nick Craig-Wood	bfe812ea6b	dedupe: implement merging of duplicate directories - fixes #1243	2017-08-07 15:36:41 +01:00
Nick Craig-Wood	db1995e63a	Add MergeDirs optional interface and implement it for drive	2017-08-07 15:32:47 +01:00
Nick Craig-Wood	81a2ab599f	fs: add optional ID to fs.Directory and set it in the remotes which care	2017-08-07 15:31:22 +01:00
Stefan Breunig	40f24e0ea3	config: use absolute ConfigPath to ensure newly written config is on the same mount - fixes #1569	2017-08-05 12:13:25 +02:00
Nick Craig-Wood	92d2e1f8d7	azureblob: rework and complete #801 ... * Fixup bitrot (rclone and Azure library) * Implement Copy * Add modtime to metadata under mtime key as RFC3339Nano * Make multipart upload work * Make it pass the integration tests * Fix uploading of zero length blobs * Rename to azureblob as it seems likely we will do azurefile * Add docs	2017-08-04 22:56:16 +01:00
Andrei Dragomir	036fd61a50	Added Azure Blob storage support #801	2017-08-04 22:54:27 +01:00
wuyu	ec5b72f8d5	Add new QingStor remote ... Add new package qingstor to support QingStor API. Add new unit test for its and tested through; But I commented on some tests case because of some of the features of QingStor. Add new docs for it.	2017-08-04 17:25:47 +01:00
Nick Craig-Wood	f682002b84	fs: Make tests create a new bucket rather than purging the old one ... This enables QingStor to pass the tests as it has a 2 minute lockout on deleting the old bucket then creating it again.	2017-08-04 17:09:28 +01:00
Stefan Breunig	28a18303f3	implement rcat – fixes #230, fixes #1001	2017-08-03 21:42:35 +02:00
Nick Craig-Wood	3e3a59768e	fs/test_all: fix after fstest factorisation	2017-08-03 20:01:05 +01:00
Nick Craig-Wood	e01741b557	fs: Cleaning up directories in test is no longer needed ... ..as it is done in the finalise method.	2017-08-03 20:01:05 +01:00
Nick Craig-Wood	71fe046937	fs: Add Find method to DirTree	2017-07-26 16:38:53 +01:00
Nick Craig-Wood	d5ff7104e5	fs: Implement NewDirTree for non --fast-list	2017-07-26 16:38:44 +01:00
Nick Craig-Wood	cd4895690a	fstest: Factor test initialisation into Initialise()	2017-07-26 16:38:33 +01:00
Nick Craig-Wood	1ecf2bcbd5	fs: fix typo in --bind description	2017-07-23 23:08:33 +01:00
Nick Craig-Wood	c3d6cc91ec	Fix --bind flag changes under go1.6 ... Correcting 9f24639568	2017-07-23 22:36:32 +01:00
Nick Craig-Wood	9f24639568	Add --bind flag for choosing the local addr on outgoing connections - fixes #1087 ... Supported by all remotes except FTP.	2017-07-23 16:27:39 +01:00
Nick Craig-Wood	7b81f12dad	box: add docs ... * reorder remotes so they are in alphabetical order by full name everywhere * update CONTRIBUTING doc	2017-07-23 11:32:34 +01:00
Nick Craig-Wood	d279161cee	Implement box storage remote - #97	2017-07-23 11:32:34 +01:00
Nick Craig-Wood	db6009126d	Fix test failure with new stretchr/testify - fixes #1550	2017-07-23 08:59:07 +01:00
Nick Craig-Wood	c8b5ee1e54	Start v1.37-DEV development	2017-07-22 20:43:06 +01:00
Nick Craig-Wood	a73ecec11f	Version v1.37	2017-07-22 20:04:29 +01:00
Nick Craig-Wood	0cf19ef66a	Make ListDirSorted check for subdirectories and write test	2017-07-19 09:36:27 +01:00
Nick Craig-Wood	655891170f	Check in ListDirSorted that the directory entries all belong	2017-07-18 23:39:42 +01:00
Nick Craig-Wood	6f71260acf	Add --tpslimit and --tpslimit-burst to limit transactions per second for HTTP ... This is useful if you are being rate limited or banned by your cloud storage provider.	2017-07-16 17:25:39 +01:00
Nick Craig-Wood	62e28d0a72	Replace token bucket limiter github.com/tsenart/tb with golang.org/x/time/rate ... In tests tsenart/tb has proved inaccurate at low rates.	2017-07-16 16:14:44 +01:00
Nick Craig-Wood	72add5ab27	sync: state whether duplicates are objects are directories	2017-07-08 15:42:18 +01:00
Nick Craig-Wood	69ff009264	Use a stable sort for sorting directory entries ... This is useful if there are duplicates. Assuming the remote delivers the entries in a consistent order, this will give the best user experience in syncing as it will consistently use the first entry for the sync comparison.	2017-07-06 14:07:26 +01:00
Nick Craig-Wood	27b157580e	Move make_test_files.go into bin	2017-07-06 11:54:57 +01:00
Nick Craig-Wood	f0512d1a52	Fix missing fs.Dir -> fs.Directory	2017-07-06 11:31:36 +01:00
Nick Craig-Wood	575e779b55	Warn about duplicate files when syncing - fixes #1506 ... Error about unsorted directories and test thoroughly	2017-06-30 21:24:13 +01:00
Nick Craig-Wood	e7d04fc103	Create fs.Directory interface and use it everywhere	2017-06-30 14:26:59 +01:00
Nick Craig-Wood	e2d7d413ef	fs: rename BasicInfo to DirEntry	2017-06-30 14:26:58 +01:00
Nick Craig-Wood	e7e9aa0dfa	fs: Remove unused ListFser interface	2017-06-30 14:26:58 +01:00
Nick Craig-Wood	f88300a153	Don't Mkdir at the start of sync - fixes #1131 ... This is possible now that the bucket based remotes will create the buckets on demand (9c1e703777).	2017-06-29 12:31:53 +01:00
Nick Craig-Wood	e54087ece1	Fix config tests to save configData which fixes subsequent tests	2017-06-29 12:31:53 +01:00
Nick Craig-Wood	54561fd2bc	s3: work around eventual consistency in bucket creation ... Deleting a bucket then testing its existence can give the wrong result. Work around by keeping a flag as to whether we have deleted the bucket.	2017-06-29 12:31:52 +01:00
Nick Craig-Wood	f3c7e1a9dd	Debug directory creation and removal - fixes #1192	2017-06-27 22:19:35 +01:00
Nick Craig-Wood	aa20486485	Add --stats-log-level so can see --stats without -v - fixes #1180 ... The most common use for this flag is likely to be showing the stats without using -v by using `--stats-log-level NOTICE`.	2017-06-26 22:50:37 +01:00
Stefan Breunig	52b042971a	keep file permissions and try to keep user/group on supported systems (fixes #1467)	2017-06-25 09:05:24 +02:00
Stefan Breunig	2d2778eabf	don't delete remote if name does not change while renaming (fixes #1495)	2017-06-25 08:55:54 +02:00
Vasiliy Tolstov	6b89e6c381	add new http remote filesystem ... Signed-off-by: Vasiliy Tolstov <v.tolstov@selfip.ru>	2017-06-20 21:27:14 +01:00
Nick Craig-Wood	b76cd4abd2	Fix Range header option	2017-06-20 21:27:14 +01:00
Nick Craig-Wood	4fd9570332	fs: Use an in place filter in ListDirSorted	2017-06-20 21:27:14 +01:00
Nick Craig-Wood	5250fcdf08	core: fix data race in walk ... This was detected by the race detector when the client of Walk() sorted entries.	2017-06-15 16:46:32 +01:00
Nick Craig-Wood	64662bef8d	Deprecate --old-sync-method it is replaced with --fast-list ... Remove old sync method code.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	0b8d9084fc	test_all: print command line so it can be cut and pasted into bash	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	8a6a8b9623	Change List interface and add ListR optional interface ... This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	6fc88ff32e	Use --fast-list flag for sync/copy/move - fixes #1277 ... Redo test framework to take a -fast-list flag and test remotes with that flag.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	50928a5027	Implement --fast-list flag. ... This is supported remotes which can do a recursive listing. It will use more memory. This is related to #1277 but doesn't fix that issue yet.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	53c3e5f0ab	Add placeholder support for ListR interface. ... The ListR interface will be implemented by remotes that can do a recursive directory listing more efficiently than just recursing through the directories. These include the bucket based remotes.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	0edb025257	Fixup tests with dirs vs bucket based fs	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	7e20e16cff	core: Implement Walk directory listing and use in place of Lister ... This is in preparation for removing the Lister code and replacing the fundamental operation in the Fs with listing a single directory.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	68333d34a1	dropbox: make setting mod time on existing files work properly ... This is a fix left over from the v2 conversion. Dropbox ignores the client modification on an incoming file if it was identical to the existing file. This change deletes the existing file first before re-uploading the new one.	2017-06-13 13:58:39 +01:00
Nick Craig-Wood	740b3f6ae2	Fix problems found with ineffassign	2017-06-13 11:52:36 +01:00
Nick Craig-Wood	9a11d3efd9	Revert "Start Cat tests from 2 as onedrive doesn't support ranging from 1" ... Now that https://github.com/OneDrive/onedrive-api-docs/issues/543 is fixed, this can be reverted. This reverts commit 320c53eab0.	2017-06-10 13:48:00 +01:00
Nick Craig-Wood	ee14efd3c2	config: fix menu selection when no remotes	2017-06-10 11:39:40 +01:00
Nick Craig-Wood	52e1bfae2a	oauth: Allow auth_url and token_url to be set in the config file ... If set in the config file, these override the ones configured into the remote. This enables alternative oauth servers to be used for all oauth remotes. This can only be altered by editing the config file for the moment.	2017-06-08 20:35:32 +01:00
Nick Craig-Wood	b49821956a	Fix copyto/moveto test error (see #1261)	2017-06-07 14:08:46 +01:00
Nick Craig-Wood	a61ba1e7c4	moveto, copyto: report transfers and checks as per move and copy	2017-06-07 13:02:21 +01:00
Nick Craig-Wood	d30cc1e119	Factor RemoteSplit into fs	2017-06-07 12:27:33 +01:00
Nick Craig-Wood	74a3dfc4e1	Fix TestHashSums	2017-06-06 23:21:47 +01:00
Nick Craig-Wood	b047402294	config: Fix save of temp file under Windows - fixes #1458	2017-06-01 16:38:19 +01:00
Stefan	9cede6b372	fully write new config file before moving to target location (fixes #1287) ... * fully write new config file before moving to target location (fixes #1287) * do not fail if there is no previous config; print temporary config path on failure	2017-06-01 08:57:10 +01:00
Stefan Breunig	decd960867	make moveto/copyto no-ops when source and destination are the same (fixes #1261)	2017-05-30 23:01:19 +01:00
Nick Craig-Wood	20da3e6352	Add options to Put, PutUnchecked and Update, add HashOption and speed up local ... * Add options to Put, PutUnchecked and Update for all Fses * Use these to create HashOption * Implement this in local * Pass the option in fs.Copy This has the effect that we only calculate hashes we need to in the local Fs which speeds up transfers significantly.	2017-05-29 12:04:52 +01:00
Nick Craig-Wood	6381959850	dropbox: support Dropbox content hashing scheme - fixes #1302 ... * add support to hashing module * add dbhashsum to list the hashes * add support to dropbox module This means objects up and downloaded to/from Dropbox will have their hashes checked. Note after this change local objects are calculating MD5, SHA1 and DBHASH which is excessive and needs to be fixed.	2017-05-29 12:04:44 +01:00
Stefan Breunig	a2e3af0523	poll for Google Drive changes when mounted	2017-05-28 17:54:52 +01:00
Nick Craig-Wood	88ef475629	config: allow keys to be deleted from the config file	2017-05-25 10:15:22 +01:00
Nick Craig-Wood	ae9f8304fa	Attempt to make async buffer test more reliable	2017-05-24 16:24:06 +01:00
Nick Craig-Wood	e0ba1a2cd2	ftp: fix bitrot	2017-05-18 20:49:36 +01:00
Antonio Messina	c72fca2711	Add ftp backend - fixes #540	2017-05-18 20:49:36 +01:00
Nick Craig-Wood	50e79bc087	fs: Implement fs.Trace for tracing entry and exit of functions	2017-05-15 16:56:45 +01:00
Nick Craig-Wood	17633f5460	Require go1.6 for building rclone ... This is required because google.golang.org/grpc needs it.	2017-05-11 17:07:49 +01:00
Yoni Jah	cdeeff988e	Added RepetableReader to fs. used in OneDrive with io.LimitedReader to display accurate speed	2017-05-02 22:31:05 +01:00
Hraban Luyat	66e8c1600e	Print password prompts to stderr ... This makes rclone with encrypted config better suited for use in pipelines. E.g.: $ rclone lsl mydrive:Some/Dir | sort -k 4 If the password prompt ("Enter configuration password") is printed to stdout, it will be swallowed by sort. By printing it to stderr, you still see the prompt, without sacrificing compatibility with the unix pipeline.	2017-04-22 09:38:39 +01:00
Nick Craig-Wood	bc25190fc7	Fix misleading log message with --dry-run - fixes #1309	2017-04-10 16:07:22 +01:00
marvwatson	4810aa65a4	Update references from HTTP to HTTPS where possible	2017-03-29 05:38:34 -07:00
Nick Craig-Wood	503cd84919	Start v1.36-DEV development	2017-03-18 11:30:59 +00:00
Nick Craig-Wood	118e26f8e2	Version v1.36	2017-03-18 11:16:43 +00:00
Nick Craig-Wood	9b07d32c02	onedrive, drive, amazonclouddrive: make sure we find the root ... This fixes copyto copying things to the wrong place - fixes #1231	2017-03-16 09:42:49 +00:00
Nick Craig-Wood	99f7fe736a	onedrive: implement Move #197	2017-03-15 14:01:08 +00:00
Nick Craig-Wood	320c53eab0	Start Cat tests from 2 as onedrive doesn't support ranging from 1 ... This has been reported here: https://github.com/OneDrive/onedrive-api-docs/issues/543	2017-03-12 14:24:33 +00:00
Nick Craig-Wood	4d5b73df85	Fix TestListDirSorted eventual consistency listing problems	2017-03-12 14:00:22 +00:00
Nick Craig-Wood	b2a4ea9304	fs/buffer: Fix panic on concurrent Read/Close - fixes #1213	2017-03-06 19:22:17 +00:00
Nick Craig-Wood	10e532bce9	Fix --files-from with an empty file copying everything - fixes #1196	2017-03-04 10:12:54 +00:00
Nick Craig-Wood	4ab7e05e02	Fix MimeType propagation ... In fs.Copy, don't wrap objects if possible, and if not, then add a MimeType method into the wrapped object.	2017-03-04 10:10:55 +00:00
Nick Craig-Wood	79e3c67bbd	local, yandex, dropbox: fix NewObject suceeding on a directory #1079 ... Add tests to make it consistent across all remotes	2017-02-25 11:09:57 +00:00
Nick Craig-Wood	e2f0feef3c	Add debugging to print hash values on failed hash comparison	2017-02-23 11:23:19 +00:00
Nick Craig-Wood	e59dc81658	Stop --track-renames deleting case folded source files - fixes #1094 ... What was happening is that when Move was implemented as Copy + Delete, MoveFile was seeing the files didn't need transferring (because they were identical) then deleted the source. The fix uses Move instead and patches onedrive to disallow a case folded identical copy (which errors with 500 error)	2017-02-22 19:28:22 +00:00
Nick Craig-Wood	6b0f2ef4bd	Fix --delete-before deleting files on copy - fixes #1166	2017-02-22 13:17:38 +00:00
Hisham Zarka	b3a526814e	fix --ignore-checksum	2017-02-18 13:13:53 +04:00
Nick Craig-Wood	033d1eb7af	Refactor Account interface	2017-02-17 09:15:24 +00:00
Nick Craig-Wood	ac62ef430d	Prevent double closes on async buffer	2017-02-17 08:55:24 +00:00
Nick Craig-Wood	928be0f1fd	mount: fix seek with buffering to use correct interface ... Stop pre-cache before seeking which stops lots of excess data transfer	2017-02-17 08:55:24 +00:00
Nick Craig-Wood	6f75290678	Make async buffering start slowly to improve seek performance	2017-02-17 08:26:14 +00:00
Nick Craig-Wood	f3c5745468	Add srcRemote and dstRemote parameters to DirMove #954	2017-02-16 17:42:37 +00:00
Nick Craig-Wood	dac4bb22d3	mount: Make include and exclude filters apply to mount - fixes #1060	2017-02-15 23:28:53 +00:00
Nick Craig-Wood	b52c80e85c	sync: don't update mod times if --dry-run set - fixes #1100	2017-02-15 23:09:44 +00:00
Nick Craig-Wood	f15c6b68b6	Re-add the async buffer on seek - fixes #1137	2017-02-15 22:54:21 +00:00
Nick Craig-Wood	3f778d70f7	Add sync.Pool to async reader	2017-02-15 22:37:58 +00:00
Nick Craig-Wood	7fa687b3e1	fs: Async buffer: use ReadFill to fill the chunks and increase to 1MB	2017-02-14 22:36:37 +00:00
Nick Craig-Wood	493da54113	Add --buffer-size parameter to control buffer size for copy	2017-02-14 22:36:37 +00:00
Nick Craig-Wood	541929258b	check: Add --download flag to check all the data, not just hashes	2017-02-13 10:48:26 +00:00
Nick Craig-Wood	7047c67a5e	sync: Fix log message containing <nil>	2017-02-13 09:23:21 +00:00
Nick Craig-Wood	01c747e7db	Add cryptcheck command to check integrity of crypt remotes #1102	2017-02-12 16:30:18 +00:00
Nick Craig-Wood	666dae4229	Add --syslog flag to optionally log to syslog on capable platforms	2017-02-11 20:27:46 +00:00
Nick Craig-Wood	ac1c041377	Redo log level flags ... * -vv or --log-level DEBUG * -v or --log-level INFO * --log-level NOTICE (default) * -q --log-level ERROR Replace Config.Verbose and Config.Quiet with Config.LogLevel Fixes #739 Fixes #1108 Fixes #1000	2017-02-11 20:22:42 +00:00
Nick Craig-Wood	0366ea39c5	Reassign some logging levels	2017-02-11 17:56:05 +00:00
Nick Craig-Wood	80f53176d9	Rename log functions and factor into own file	2017-02-11 17:54:50 +00:00
Nick Craig-Wood	50e190ff54	cat: don't allocate buffers if not needed to reduce memory usage	2017-02-09 11:46:53 +00:00
Nick Craig-Wood	dd20a297d6	cat: Fix go routine leak	2017-02-09 11:25:36 +00:00
Nick Craig-Wood	c0ad29c06c	Clarify logging and docs for --no-traverse incompatibilities - fixes #1059	2017-02-08 22:35:12 +00:00
Nick Craig-Wood	d091d4a8bb	rclone cat: add --head, --tail, --offset, --count and --discard ... Fixes #819	2017-02-08 08:09:41 +00:00
Nick Craig-Wood	48cdedc97b	Re-implement sync routine to work a directory at a time ... Multiple directories (up to --checkers worth) are scanned at once. This uses much less memory than the previous scheme - only the amount of memory needed to hold an entire directory listing of objects. For directory based remotes the speed is unchanged. For bucket based remotes, instead of doing one API call to list the whole bucket, it does multiple calls, one for each pseudo directory. However these are done in parallel so in practice this seems to speed up directory listings. This replaces the existing sync method as it performs faster and uses less memory. The old sync method is available with the temporary --old-sync-method flag. Fixes #517 Fixes #439 Fixes #236 Fixes #1067	2017-02-04 10:30:25 +00:00
Nick Craig-Wood	7c6cd3a9e1	Make --delete-after the default and refactor --delete-{before,during,after} parsing	2017-02-04 10:30:25 +00:00
Nick Craig-Wood	bcdd73369f	Ignore --delete-before with --track-renames	2017-02-04 10:30:25 +00:00
Nick Craig-Wood	86bec20b56	sync: factor accumulating the rename checks	2017-02-04 10:30:25 +00:00
Nick Craig-Wood	c3b2b89473	Add ListDirSorted function to list a directory ... * fix error return of readFilesFn also	2017-02-04 10:30:25 +00:00
Nick Craig-Wood	85f05c57d1	Clean empty directories between test runs	2017-02-04 10:30:25 +00:00
Nick Craig-Wood	726cb43be9	Complete SFTP remote #521 ... * Add unit tests * Fix up remote so it passes tests * Add docs	2017-02-04 10:29:46 +00:00
Nick Craig-Wood	288302c2cf	Make fallback purge delete empty directories too. ... This was implemented to make the SFTP unit tests pass.	2017-02-04 10:29:46 +00:00
Jack Schmidt	b9a8315696	Basic SFTP support, Issue #521	2017-02-04 10:29:18 +00:00
Nick Craig-Wood	9d331ce04b	Implement --ignore-checksum flag ... Fixes #793 Fixes #863 Fixes #981	2017-02-03 08:11:10 +00:00
Nick Craig-Wood	2192805360	rclone config: when choosing from a list, allow the value to be entered	2017-01-29 15:51:26 +00:00
Nick Craig-Wood	db0b93c0ad	rclone config: allow rename and copy of remotes - fixes #641	2017-01-29 15:37:44 +00:00
Dario Giovannetti	9d36258923	Comply with XDG Base Directory specification ... Fixes #868	2017-01-26 10:22:08 +00:00
Nick Craig-Wood	b6848a3edb	Fix race in Lister.Finished which was causing the tests to be unreliable	2017-01-19 20:11:17 +00:00
Nick Craig-Wood	e2bf9eb8e9	Implement --suffix for use with --backup-dir only #98 ... This also makes sure we remove files we are about to override in the --backup-dir properly.	2017-01-19 20:11:17 +00:00
Nick Craig-Wood	1fa258c2b4	Define a new Features() method for Fs ... Optional interfaces are becoming more important in rclone, --track-renames and --backup-dir both rely on them. Up to this point rclone has used interface upgrades to define optional behaviour on Fs objects. However when one Fs object wraps another it is very difficult for this scheme to work accurately. rclone has relied on specific error messages being returned when the interface isn't supported - this is unsatisfactory because it means you have to call the interface to see whether it is supported. This change enables accurate detection of optional interfaces by use of a Features struct as returned by an obligatory Fs.Features() method. The Features struct contains flags and function pointers which can be tested against nil to see whether they can be used. As a result crypt and hubic can accurately reflect the capabilities of the underlying Fs they are wrapping.	2017-01-16 17:33:25 +00:00
ncw	3745c526f1	Implement --backup-dir - fixes #98 ... The parameter of backup-dir specifies a remote that all deleted or overwritten files will be copied to.	2017-01-16 17:33:25 +00:00
Nick Craig-Wood	c123c702ab	Fix fs.Overlapping and factor fs.SameConfig	2017-01-14 09:55:53 +00:00
ncw	4aae7bcca6	Factor server side move detection	2017-01-14 09:55:53 +00:00
Brandur	3b1e0b66bb	Return error on not found from ListFn ... This changes `ListFn`'s implementation so that if it encounters a not found error, instead of sending a fatal error to log, it coordinates the return of the error between checker goroutines and sends it back to the caller. The main impetus here is that it allows an external program compiling against rclone as a package to handle a not found, where it currently it cannot. This does change error output on a not found a little bit, we go from this: 2017/01/09 21:14:03 directory not found To this: 2017/01/09 21:13:44 Failed to ls: directory not found	2017-01-12 10:07:59 +00:00
Nick Craig-Wood	78c1f2839e	Fix filters to add ** rules to directory rules ... This fixes `--exclude ".*{,/**}"` to exclude all . files and . directories.	2017-01-05 19:33:49 +00:00
Nick Craig-Wood	de2d967abd	Stop --track-renames hashing matching files - fixes #984 ... Also only hash files of the correct size. This speeds it up a lot.	2017-01-05 17:58:01 +00:00
Marco Paganini	6611d92e21	Only start bandwidth ticker when necessary. ... - Only start the token ticker when the timetable entry has more than one entry. - This fixes the "Scheduled bandwidth change" log message when no bwlimit is specified. - Fixes #987	2017-01-04 19:03:49 -08:00
Nick Craig-Wood	f73ee5eade	Make all config file variables be settable in the environment ... These are set in the form RCLONE_CONFIG_remote_option where remote is the uppercased remote name and option is the uppercased config file option name. Note that RCLONE_CONFIG_remote_TYPE must be set if defining a new remote. Fixes #616	2017-01-03 22:42:47 +00:00
Nick Craig-Wood	0d75d2585f	Allow all options to be set from environment variables ... The option names are munged changing - to _ making upper case and prepending RCLONE_. The values are as parsed by pflag.	2017-01-03 22:42:47 +00:00
Marco Paganini	3b0f944e23	Add time-based bandwidth limits. ... - Change the --bwlimit command line parameter to accept both a limit (as before) or a full timetable (formatted as "hh:mm,limit hh:mm,limit...") - The timetable is checked once a minute by a ticker function. A new tokenBucket is created every time a bandwidth change is necessary. - This change is compatible with the SIGUSR2 change to toggle bandwidth limits. This resolves #221.	2017-01-03 21:00:38 +00:00
Nick Craig-Wood	f1221b510b	Change --track-renames to use the length,hash pair stored in a map ... This makes it much faster in the case of many files and use less memory. This also detects use of --no-traverse and disables it.	2017-01-03 20:37:06 +00:00
Nick Craig-Wood	274ab349f4	sync: Only allow --track-renames if have a common hash	2017-01-03 20:35:05 +00:00
Bjørn Erik Pedersen	47d3a450a4	sync: Track and perform server-side renames ... This commits adds support for tracking of file renames if `track-renames` flag is set, and it then performs server-side renames for remotes that support it, i.e. remotes that implement either the `Mover` or the `Copier` interface. Fixes #888	2017-01-03 20:35:05 +00:00
Nick Craig-Wood	5b8b379feb	Version v1.35	2017-01-02 15:33:06 +00:00
Nick Craig-Wood	4dd5428b13	Fix rmdirs test and integration tests which depend on each other	2017-01-02 14:15:07 +00:00
Nick Craig-Wood	64ec220d5d	Fix --no-update-modtime test on remotes which don't support hashes	2016-12-31 15:19:26 +00:00
Nick Craig-Wood	cbfec0d281	Fix tests for missing config file	2016-12-20 15:05:08 +00:00
Nick Craig-Wood	1b2dda8c4c	oauthutil: Reload config file off disk before updating token ... This fixes the config file being overwritten when two rclones are running. Fixes #682	2016-12-19 15:04:07 +00:00
Nick Craig-Wood	4482e75f38	Fix golint	2016-12-15 21:02:41 +00:00
Nick Craig-Wood	0117aeafbf	mount: this removes the async buffering as it was killing seek performance	2016-12-15 17:08:52 +00:00
Nick Craig-Wood	8083804575	Make sure wrapped retry/fatal errors are never nil to avoid panic	2016-12-13 16:02:14 +00:00
Nick Craig-Wood	fafbcc8e2f	Make server side move more obvious in debug	2016-12-09 16:57:07 +00:00
Nick Craig-Wood	48a2e3844d	Add optional interface DirCacheFlush for making the tests more reliable ... This is defined for the users of dircache drive, onedrive, and acd. This helps fix the DirMove tests on acd.	2016-12-09 16:57:07 +00:00
Nick Craig-Wood	dcf53a1d12	Allows multiple --include/--exclude/--filter options - fixes #875	2016-12-07 13:37:40 +00:00
Nick Craig-Wood	3bdfa284a9	Make rclone lsd obey the filters properly	2016-12-07 11:16:36 +00:00
Nick Craig-Wood	be4fd51289	fuse: Add bandwidth accounting and buffering ... This fixes rclone mount ignoring bwlimit and increases buffering which should speed up transfers greatly. Fixes #796 Fixes #690	2016-12-04 16:57:47 +00:00
Nick Craig-Wood	c265f451f2	Implement moveto and copyto commands for choosing a destination name on copy/move ... Fixes #227 Fixes #476	2016-12-03 23:43:52 +00:00
Nick Craig-Wood	2058652fa4	Allow overlapping remotes in move when DirMove is supported	2016-12-03 09:08:40 +00:00
Nick Craig-Wood	50b3cfccb1	Factor Move out of sync.go and add remote parameter to Move and Copy	2016-12-03 09:08:40 +00:00
Nick Craig-Wood	2756900749	Fix not transferring files that don't differ in size - fixes #911 ... Due to a logic error files stored on remotes which support modtime but not hashes weren't being transferred when updating with a file of the same size but different modtime. Instead the modtime of the remote file was being set to that of the local file. In practice this affected crypt with all remotes except Amazon Drive and Dropbox.	2016-11-28 17:08:15 +00:00
Nick Craig-Wood	539853df36	Fix rmdirs test	2016-11-28 12:23:24 +00:00
Scott McGillivray	f9df545e3c	add --stats-unit option and improve alignment for --stats output	2016-11-28 12:18:30 +00:00
Nick Craig-Wood	f3365dd251	Make rclone rmdirs command to delete empty directories - fixes #831	2016-11-27 11:49:31 +00:00
Nick Craig-Wood	aaa1370a36	Add directory parameter to Rmdir and Mkdir #100 #831 ... This will enable rclone to manage directories properly in the future.	2016-11-26 12:02:53 +00:00
Nick Craig-Wood	d95288175f	Version v1.34	2016-11-06 10:18:30 +00:00
Nick Craig-Wood	01be5bff02	Fix ogier/pflag vs spf13/pflag	2016-11-05 18:18:08 +00:00
Nick Craig-Wood	e825df6448	Fix Check on crypted file systems	2016-11-05 18:17:21 +00:00
Nick Craig-Wood	ff41b0d435	Improve error message when source remote isn't found in sync #848	2016-11-05 18:03:55 +00:00
Nick Craig-Wood	441951a93b	Stop removing failed upload to cloud storage remotes - fixes #559 ... We do remove a partially written file on local so we don't have corrupted files lying around.	2016-11-04 21:34:25 +00:00
Nick Craig-Wood	318e335137	Remove Authorization: headers from --dump-headers output ... Add in `--dump-auth` flag to put it back.	2016-11-02 15:53:43 +00:00
Nick Craig-Wood	d1759fdfa9	Add request ID to HTTP debugging to make it easier to trace concurrent flows	2016-10-31 12:01:28 +00:00
Nick Craig-Wood	e65059e431	Fix non-windows/non-unix builds for bwlimit/SIGUSR2 feature and add a mutex ... The race detector complained whenever SIGUSR2 was sent to rclone so this adds a mutex to prevent concurrent access.	2016-10-30 19:20:16 +00:00
Nick Craig-Wood	5454f2abd0	Fix race in checkServerTime	2016-10-30 19:16:27 +00:00
Marco Paganini	cc4f5ba7ba	Add support to toggle bandwidth limits via SIGUSR2. ... Sending rclone a SIGUSR2 signal will toggle the limiter between off and the limit set with the --bwlimit command-line option.	2016-10-30 17:46:59 +00:00
Nick Craig-Wood	de73063977	Fix output of crypt objects in logs	2016-10-20 17:46:51 +01:00
Nick Craig-Wood	b7875fc02a	rclone check: show count of hashes that couldn't be checked #700	2016-10-17 16:48:11 +01:00
Nick Craig-Wood	8a56a6836a	Check server time against local time #654	2016-10-08 14:00:50 +01:00
Nick Craig-Wood	83849e0a36	Don't show encrypted password to stop confusion - fixes #656	2016-10-08 11:26:14 +01:00
Nick Craig-Wood	d42b38699b	Make ResponseHeaderTimeout be --timeout not --contimeout fixes #766 ... This was causing a problem with Amazon Drive which often pauses for a long time after uploads before returning the response.	2016-10-08 10:12:19 +01:00
Nick Craig-Wood	d033e92234	Stop single file and --files-from operations iterating through the source bucket. ... This works by making sure directory listings that use a filter only iterate the files provided in the filter (if any). Single file copies now don't iterate the source or destination buckets. Note that this could potentially slow down very long `--files-from` lists - this is easy to fix (with another flag probably) if it causes anyone a problem. Fixes #610 Fixes #769	2016-10-07 11:39:39 +01:00
Nick Craig-Wood	e94850f322	Fix timeouts not working when set to 0 and firing too often - #766	2016-10-06 10:23:23 +01:00
Nick Craig-Wood	aef2ac5c04	Add options for Open and implement Range for all remotes	2016-10-05 21:03:56 +01:00
Nick Craig-Wood	9ea20bac42	Fix accidentally committed test in move code	2016-10-03 20:16:41 +01:00
Nick Craig-Wood	945f49ab5e	Make ContentType be preserved for cloud -> cloud copies - fixes #733	2016-10-03 20:02:04 +01:00
Nick Craig-Wood	6c9a258d82	Fix move command ... * Delete src files which already existed in dst - fixes #751 * Fix deletion of src file when dst file older	2016-10-03 19:58:44 +01:00
Nick Craig-Wood	f2eeb4301c	Make --dump-bodies imply --dump-headers	2016-09-22 08:40:37 +01:00
Nick Craig-Wood	0caac70994	Fix build for < go1.7	2016-09-13 11:36:14 +01:00
Nick Craig-Wood	83ba59749f	Make failed uploads not count as "Transferred" - fixes #708	2016-09-12 18:15:58 +01:00
Nick Craig-Wood	bbf819e2d1	Move versioncheck so it happens earlier in the compile process.	2016-09-12 17:50:19 +01:00
Nick Craig-Wood	0cb9bb3b54	Redo http Transport code ... * Insert User-Agent in Transport - fixes #199 * Update timeouts to use Context * Modernise transport	2016-09-12 17:50:19 +01:00
Nick Craig-Wood	2eaac80c86	b2 with crypt: fix crash when uploading large files - fixes #673	2016-09-05 18:10:01 +01:00
Nick Craig-Wood	3996bbb8cb	Version v1.33	2016-08-24 23:02:05 +01:00
Nick Craig-Wood	059743a1b0	crypt: add to integration tests	2016-08-23 17:45:37 +01:00
Nick Craig-Wood	a54806e5c1	Fix Move when underlying remote returns ErrorCantMove	2016-08-23 17:45:37 +01:00
Nick Craig-Wood	226c2a0d83	Implement crypt for encrypted remotes - #219	2016-08-23 17:45:37 +01:00
Nick Craig-Wood	9985fc40f4	Make Password parameters obey Optional flag and offer to generate random ones	2016-08-23 17:45:37 +01:00
Nick Craig-Wood	b1de4c8cba	Implement password Option and re-implement editing ... Editing now shows all the options for the fs and asks one at a time whether they should be changed.	2016-08-23 17:45:37 +01:00
Nick Craig-Wood	6a4e424630	Re-implement Obscure/Reveal so they use AES-CTR encryption	2016-08-23 17:45:37 +01:00
Nick Craig-Wood	ebb67c135e	Fix listToChan passing nil objects to DeleteFile	2016-08-23 17:45:37 +01:00
Nick Craig-Wood	037a000cc8	b2: fix stats accounting for upload - fixes #602	2016-08-22 21:19:38 +01:00
Nick Craig-Wood	84eb7031bb	Implement the rclone cat command	2016-08-18 22:45:32 +01:00
Nick Craig-Wood	d7b79b4481	Mark the compiled from source version with -DEV - fixes #627	2016-08-18 21:31:10 +01:00
Nick Craig-Wood	b5faaf7116	Fix double close of abort channel - fixes #592	2016-08-18 18:56:57 +01:00
Nick Craig-Wood	0805ec051f	Add BasicInfo interface shared between Dir and Object	2016-08-05 17:45:27 +01:00
Nick Craig-Wood	ae56df7d4f	Add --dedupe-mode only to dedupe command	2016-08-04 21:47:14 +01:00
Nick Craig-Wood	67562081f7	Version v1.32	2016-07-13 17:32:39 +01:00
Nick Craig-Wood	63f6827a0d	Version v1.31	2016-07-13 12:28:01 +01:00
Nick Craig-Wood	ac3c83f966	Fix integration tests for drive	2016-07-12 21:38:15 +01:00
Nick Craig-Wood	b9c8e61d39	Explicitly check the state in tests after writing files ... ...otherwise Amazon Drive will fail.	2016-07-12 21:36:39 +01:00
Nick Craig-Wood	a6056408dd	Fix move command - stop it running for overlapping fses - fixes #577 ... * Make move command check for overlapping remotes and refuse to run * Do copy/delete rather than all the copies then all the deletes * Doesn't purge the source - this was unexpected behaviour see #512 and #416 * Add -list-retries flag to test suite to control retries This changes the semantics of `move` slightly. However it now errs on the side of not deleting stuff.	2016-07-12 10:49:37 +01:00
Nick Craig-Wood	b9479cf7ab	Implement --no-update-modtime flag - fixes #511	2016-07-12 10:46:45 +01:00
Nick Craig-Wood	a20d80565b	Tidy stats output - fixes #541	2016-07-11 13:04:30 +01:00
Nick Craig-Wood	4bd9932703	Fix wording in verbose copy logs - fixes #574	2016-07-09 10:11:57 +01:00
Nick Craig-Wood	13591c7c00	Redo error handling for sync/copy/move ... * Factor sync/copy/move into its own file * Make fatal errors abort the sync * Make Copy return errors * Make Sync/Copy/Move return the last Copy error if there was one * Prioritise returning Fatal errors * NoRetry errors are returned if no other types of errors	2016-07-04 13:45:53 +01:00
Nick Craig-Wood	28f4061892	Add two more classes of error Fatal and NoRetry ... These are for remotes to signal that they have a fatal error and don't want to continue (eg cap exceeded) or that a particular file shouldn't be retried for some reason.	2016-07-04 13:45:52 +01:00
Nick Craig-Wood	0a43ff9c13	Modify interface for accounting to take a string not an fs.Object	2016-07-02 16:58:50 +01:00
Nick Craig-Wood	9aae143833	Implement cleanup command for emptying trash / removing old versions of files	2016-07-01 16:35:36 +01:00
Nick Craig-Wood	70dc97231e	Convert more tests to use assert/require	2016-06-30 15:45:30 +01:00
Nick Craig-Wood	af4ef8ad8d	Implement --no-traverse flag to stop copy traversing the destination remote. ... Refactor sync/copy/move * Don't load the src listing unless doing a sync and --delete-before * Don't load the dst listing if doing copy/move and --no-traverse is set `rclone --no-traverse copy src dst` now won't load either of the listings into memory so will use the minimum amount of memory. This change will reduce the amount of memory rclone uses dramatically too as in normal operations (copy without --notraverse or sync) as it no longer loads the source file listing into memory at all. Fixes #8 Fixes #544 Fixes #546	2016-06-29 17:38:50 +01:00
Nick Craig-Wood	13797a1fb8	Make retry logs be debug in main copy routine	2016-06-28 08:51:57 +01:00
Nick Craig-Wood	3ad8fb8634	Make DeleteFile and DeleteFiles return errors	2016-06-28 08:51:57 +01:00
Nick Craig-Wood	ab43005422	Make NewObject return an error ... * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error	2016-06-28 08:51:57 +01:00
Nick Craig-Wood	b1f131964e	Rename NewFsObject to NewObject	2016-06-28 08:51:57 +01:00
Nick Craig-Wood	1a87b69376	Get rid of LimitedFs - FIXME needs docs on copying single files ... If remote:path points to a file make NewFs return a sentinel error fs.ErrorIsFile and an Fs which points to the parent. Use this to remove the LimitedFs and just add this file to the --files-from list. This means that server side operations can be used also. Fixes #518 Fixes #545	2016-06-28 08:51:43 +01:00
Nick Craig-Wood	5a3b109e25	Fix issues identified by go vet -shadow - fixes #530	2016-06-21 21:17:52 +01:00
Klaus Post	e0aa4bb492	Fix incomplete local hashes. ... Fixes #533	2016-06-19 16:51:49 +02:00
Nick Craig-Wood	bd0227450e	Version v1.30	2016-06-18 16:41:46 +01:00
Nick Craig-Wood	f438f1e9ef	Fix stats print	2016-06-18 16:41:46 +01:00
Nick Craig-Wood	d3dd672640	Document recursion requirements for Fses	2016-06-18 14:12:47 +01:00
Nick Craig-Wood	2a46be8cf3	b2: implement large file uploading - fixes #456	2016-06-18 13:38:05 +01:00
Nick Craig-Wood	1b4370bde1	Rework retry logic when copying objects ... * Fix off by one retry logic - fixes #406 * Retry any retriable errors * Restructure code	2016-06-18 10:55:58 +01:00
Nick Craig-Wood	2cfb3834f2	Log errors with %v	2016-06-18 09:36:47 +01:00
Nick Craig-Wood	46135d830e	Add --ignore-size flag - fixes #399	2016-06-17 17:20:08 +01:00
Nick Craig-Wood	df1092ef33	Change Fs.Put so that it must cope with existing files ... This should fix duplicate files on drive and 409 errors on amazonclouddrive however it will slow down the upload slightly as another roundtrip will be needed. None of the other Fses needed adjusting. Fixes #483	2016-06-13 19:29:10 +01:00
Nick Craig-Wood	4c5b2833b3	Convert to using github.com/pkg/errors everywhere	2016-06-13 17:43:03 +01:00
Nick Craig-Wood	7fe653c350	Unwrap errors properly for patform specific connection retry code. ... Include more possible errors for Windows. For #442	2016-06-10 13:48:41 +01:00
Nick Craig-Wood	f17cb1bf50	Fix retry of Windows wsaend errors #442 ... Make the test for wsaend error less specific	2016-06-09 15:34:13 +01:00
Nick Craig-Wood	67d0375b98	Audit use of log.Print and change to Debug, Log, or ErrorLog as appropriate	2016-06-06 21:23:54 +01:00
Nick Craig-Wood	108760e17b	Log -v output to stdout by default - fixes #228	2016-06-04 18:49:27 +01:00
Nick Craig-Wood	e2788aa729	Display the transfer stats in more human readable form - fixes #428	2016-06-03 22:49:50 +01:00
Nick Craig-Wood	772f99fd74	Make SizeSuffix output without b suffix for more useful printouts	2016-06-03 22:49:14 +01:00
Nick Craig-Wood	a21cc161de	Make 0 size files specifiable with --max-size 0b - fixes #450	2016-06-03 21:54:27 +01:00
Nick Craig-Wood	e818b7c206	Represent -1 as "off" for SIZE values	2016-06-03 21:51:39 +01:00
Nick Craig-Wood	5723d788a4	Add b suffix so we can specify bytes in --bwlimit, --min-size etc ... Fixes #449	2016-06-03 21:16:48 +01:00
Nick Craig-Wood	1d6698a754	Build tweaks - fixes #484 ... * disable CGO for static builds everywhere * override Version in release build script * don't output symbol table in release binaries	2016-06-03 20:34:19 +01:00
Nick Craig-Wood	ccdd1ea6c4	Add --max-depth parameter ... This will apply to ls/lsd/sync/copy etc Fixes #412 Fixes #213	2016-06-03 17:05:39 +01:00
Nick Craig-Wood	ad85f6e413	Implement directory include filtering for efficiency ... Fixes #395	2016-05-16 17:14:04 +01:00
Nick Craig-Wood	ac9c20b048	Make IsRetryError function	2016-05-14 17:11:19 +01:00
Nick Craig-Wood	2db35f0ce7	Dump out unexpected state in integration test	2016-05-07 21:19:26 +01:00
Nick Craig-Wood	dbfa7031d2	Factor Lister into own file, write tests and fix	2016-05-07 17:17:43 +01:00
Nick Craig-Wood	c2d0e86431	Add more tests for List() and fix resulting problems	2016-05-07 14:50:35 +01:00
Nick Craig-Wood	68ec6a9f5b	Add a directory parameter to Fs.List()	2016-05-06 16:52:34 +01:00
Nick Craig-Wood	753b0717be	Refactor the List and ListDir interface ... Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316	2016-05-06 16:52:34 +01:00
Nick Craig-Wood	1752ee3c8b	Retry errors which indicate the connection closed prematurely. ... See discussion in #442	2016-04-29 17:29:34 +01:00
Nick Craig-Wood	5c2d8ffe33	Retry only the failing tests in the integration tests	2016-04-26 10:20:07 +01:00
Klaus Post	6c0c9abd57	Use "password:" instead of "password>" prompt ... Fixes #410	2016-04-21 19:39:46 +01:00
Nick Craig-Wood	157d7d45f5	Version v1.29	2016-04-18 18:30:29 +01:00
Nick Craig-Wood	b5cba73cc3	Make test more reliable	2016-04-18 17:48:52 +01:00
Nick Craig-Wood	0f8e7c3843	Make rclone check obey the --size-only flag - fixes #419	2016-04-07 15:01:45 +01:00
Nick Craig-Wood	b835330714	Use "application/octet-stream" if mime.TypeByExtension returns invalid type ... Fixes #424	2016-04-07 14:32:01 +01:00
Klaus Post	8f4d6973fb	Fix missing "quit" option when there are no remotes.	2016-03-23 16:57:56 +00:00
Nick Craig-Wood	ced3a4bc19	Implement -I, --ignore-times for unconditional upload - fixes #311	2016-03-22 17:02:27 +00:00
Nick Craig-Wood	cb22583212	b2: Enable mod time syncing - fixes #348	2016-03-22 15:56:44 +00:00
Nick Craig-Wood	414b35ea56	Change the interface of SetModTime to return an error - #348	2016-03-22 15:56:44 +00:00
Nick Craig-Wood	8ef551bf9c	Make dedupe remove identical copies without asking and add non interactive mode - fixes #338 ... * Now removes identical copies without asking * Now obeys `--dry-run` * Implement `--dedupe-mode` for non interactive running * `--dedupe-mode interactive` - interactive the default. * `--dedupe-mode skip` - removes identical files then skips anything left. * `--dedupe-mode first` - removes identical files then keeps the first one. * `--dedupe-mode newest` - removes identical files then keeps the newest one. * `--dedupe-mode oldest` - removes identical files then keeps the oldest one. * `--dedupe-mode rename` - removes identical files then renames the rest to be different. * Add tests which will only run on Google Drive.	2016-03-06 18:15:01 +00:00
Nick Craig-Wood	874a64e5f6	A script to make a directory heirarchy for testing	2016-03-05 20:26:15 +00:00
Nick Craig-Wood	3c7f80f58f	Version v1.28	2016-03-01 09:00:01 +00:00
Nick Craig-Wood	229ea3f86c	Stop --update tests running on remotes which don't do mod time	2016-03-01 07:26:33 +00:00
Nick Craig-Wood	41eb386063	Reset password/config path in config tests to fix other tests	2016-02-29 21:43:37 +00:00
Nick Craig-Wood	dfc7cd97a3	Optionally disable gzip compression on downloads with --no-gzip-encoding - fixes #353	2016-02-29 19:48:54 +00:00
Nick Craig-Wood	280ac26464	Implement -u/--update so creation times can be used on all remotes - #226	2016-02-29 17:46:40 +00:00
Nick Craig-Wood	88cca8a6eb	Simplify literals (after running gofmt -s over the code)	2016-02-29 16:57:23 +00:00
Nick Craig-Wood	3343c1afa4	Don't make directories if --dry-run set - fixes #342	2016-02-28 19:56:50 +00:00
Nick Craig-Wood	867a26fe4f	Implement --low-level-retries flag - fixes #266	2016-02-25 22:58:21 +00:00
Nick Craig-Wood	3890105cdc	Add -run-only flag to run_all test	2016-02-25 22:05:57 +00:00
Nick Craig-Wood	d2219a800a	Fix and document the move command - fixes #334 ... * Don't attempt to use server side Move unless they are on the same Fs * Fix move in the presense of filters	2016-02-25 20:05:34 +00:00
Nick Craig-Wood	ccb59480bd	Add InActive method to Filter to detect when no fiters are in use.	2016-02-25 19:58:00 +00:00
Nick Craig-Wood	95e4d837ef	Make config chooser easier to understand	2016-02-21 13:40:16 +00:00
Nick Craig-Wood	e08e35984c	Add help to remote chooser in rclone config - fixes #43	2016-02-21 13:40:16 +00:00
Nick Craig-Wood	700e47d6e2	Stub out ReadPassword on plan9 and solaris to fix compilation	2016-02-21 10:31:53 +00:00
klauspost	ef06371c93	Create separate interface for object information. ... Take out read-only information about a Fs in a separate struct to limit access. See discussion at #282.	2016-02-19 13:31:09 +00:00
klauspost	84b00b362f	Change back to original goconfig package. ... Add documentation for `--ask-password`.	2016-02-17 11:45:05 +01:00
klauspost	bfd7601cf9	Add configuration file encryption ... See #317 for details. Use `rclone config` to add/change/remove password. Tests that loads the default configuration will now fail with a better error message, and add a switch that makes it possible to disable password prompts and fail instead. Make it possible to use the "RCLONE_CONFIG_PASS" environment variable as password for configuration.	2016-02-16 16:32:05 +01:00
Nick Craig-Wood	a6320bbad3	Fix delete command to wait until all finished - fixes missing deletes. ... This also could affect deletes at the end of the sync command.	2016-02-15 16:43:59 +00:00
Nick Craig-Wood	85d99de26b	Fix typo in error strings	2016-02-09 16:15:50 +00:00
Nick Craig-Wood	e62c0a58a7	Version 1.27	2016-01-31 17:50:13 +00:00
Nick Craig-Wood	0f73129ab7	dedupe command to deduplicate a remote. Useful with google drive - fixes #41	2016-01-31 16:09:42 +00:00
Nick Craig-Wood	1373efaa39	Delete command which does obey the filters - fixes #327	2016-01-31 16:06:04 +00:00
Nick Craig-Wood	5c37b777fc	Make the --dry-run warnings into logs so they appear without the -v flag	2016-01-31 16:06:04 +00:00
Nick Craig-Wood	8c211fc8df	Warn the user about files with same name but different case ... Relates to #107 & #119.	2016-01-26 16:57:09 +00:00
Nick Craig-Wood	74a71f7824	Add tests for --delete-before, --delete-during and --delete-after	2016-01-26 16:57:09 +00:00
Nick Craig-Wood	12b51c5eb8	Remove duplicate check for filter IncludeObject	2016-01-26 16:57:09 +00:00
klauspost	14069fd8e6	Implement --delete-before, --delete-during, --delete-after - fixes #252.	2016-01-26 16:57:09 +00:00
Nick Craig-Wood	cd62f41606	Reduce number of logs and show hash type where appropriate	2016-01-24 18:06:57 +00:00
Nick Craig-Wood	109d4ee490	Prefix all test remotes with rclone-test- and make names more pronouncable	2016-01-24 12:37:46 +00:00
Nick Craig-Wood	18ebec8276	Check remote is empty between integration tests	2016-01-24 12:37:19 +00:00
Nick Craig-Wood	eda4130703	Fix integration tests so they can be run independently and out of order - fixes #291 ... * Make all integration tests start with an empty remote * Add an -individual flag so this can be a different bucket/container/directory * Fix up tests after changing the hashers * Add sha1sum test * Make directory checking in tests sleep more to fix acd inconsistencies * Factor integration tests to make more maintainable * Ensure remote writes have a fstest.CheckItems() before use * this fixes eventual consistency on the directory listings later * Call fs.Stats.ResetCounters() before every fs.Sync() Note that the tests shouldn't be run concurrently as fs.Config is global state.	2016-01-23 17:02:09 +00:00
Nick Craig-Wood	27f67edb1a	Fix formatting problem in sha1sum	2016-01-17 13:56:42 +00:00
Nick Craig-Wood	3ffea738e6	Make hash constants start from 1 not 2	2016-01-17 10:47:24 +00:00
Nick Craig-Wood	e8daab7971	Fix integration tests for remotes with unsupported hash schemes	2016-01-16 09:45:15 +00:00
klauspost	78c3a5ccfa	Add support for multiple hash types. ... Add support for multiple hash types with negotiation of common hash types for comparison. Manually rebased version of #277 (see discussion there)	2016-01-11 13:39:33 +01:00
Nick Craig-Wood	af5f4ee724	Make --include rules add their implict exclude * at the end of the filter list ... This means you can mix `--include` and `--include-from` with the other filters (eg `--exclude`) but you must include all the files you want in the include statement. Fixes #280	2016-01-10 11:42:53 +00:00
klauspost	25f22ec561	Add "--ignore-existing" flag. ... Add option to completely ignore existing files and not consider them for transfer. Fixes #274	2016-01-08 08:20:04 +00:00
Nick Craig-Wood	5189231a34	Tweaks to rclone authorize ... * Document the headless / remote setup procedure * Move Config constants into fs * Parse arguments in main for Authorize	2016-01-07 20:31:23 +00:00
klauspost	bcbd30bb8a	Add easier headless configuration. ... This will allow setting up a remote with copy&paste of values to a headless machine. It will allow copy+pasting a token into the configuration. This requires rclone to be on a machine with a proper browser. Custom client id and secrets are supported. To test token generation, use `rclone auth "fs type"`.	2016-01-07 20:31:23 +00:00
Nick Craig-Wood	1ce3673006	Add -clean flag to test_all.go to clean left over test directories	2016-01-03 21:49:26 +00:00
Nick Craig-Wood	3423de65fa	Make canonical place for all fs in fs/all/all.go	2016-01-03 14:12:45 +00:00
Nick Craig-Wood	77fb8ac240	Version 1.26	2016-01-02 12:04:32 +00:00
dibu28	3ac4407b88	Implement Yandex storage backend - fixes #234	2015-12-30 12:11:46 +00:00
Nick Craig-Wood	8ea0d5212f	Add -verbose flag to test_all and fix tries count	2015-12-30 11:34:22 +00:00
Nick Craig-Wood	33b3eea6ec	Implement Backblaze B2 - fixes #224	2015-12-30 10:05:07 +00:00
Nick Craig-Wood	113624691a	Add -dump-headers and -dump-bodies flags for operations test debugging	2015-12-30 09:35:35 +00:00
Nick Craig-Wood	ddf39f2d57	Replace test_all.sh with test_all.go which is cross platform and parallel	2015-12-30 09:26:34 +00:00
Nick Craig-Wood	e69e181090	Fix --min-age and --max-age when only one is present	2015-12-17 14:22:43 +00:00
Nick Craig-Wood	ac65d8369e	Make fs.CheckClose public to stop duplication	2015-12-17 13:30:58 +00:00
Nick Craig-Wood	8057d668bb	Fix crash in http logging - fixes #223 ... A nil-pointer exception was caused if the http transaction ever resulted in a go error while using `--dump-bodies`. Now don't ignore the error and log it instead of the http body.	2015-12-17 13:30:58 +00:00
Nick Craig-Wood	36f1bc4a8a	Make ls/lsl/md5sum/size/check obey includes and excludes - fixes #169 ... * run check directory listings concurrently	2015-12-17 13:30:58 +00:00
Adriano Aurélio Meirelles	3cbd57d9ad	Add support to filter files based on their age	2015-12-17 09:52:38 -02:00
Nick Craig-Wood	7f0020a407	Version v1.25	2015-11-14 13:06:39 +00:00
Nick Craig-Wood	fcea3777c0	Implement Hubic storage system - fixes #200	2015-11-14 08:08:52 +00:00
Nick Craig-Wood	bed01a303f	Add UnWrapper interface and implement in LimitedFs	2015-11-14 08:08:51 +00:00
Nick Craig-Wood	2c2cb84ca7	Make it so optional interface Purge can fail so it can be wrapped	2015-11-14 08:08:51 +00:00
Nick Craig-Wood	e9dda25c60	Implement Move in limited fs	2015-11-14 08:08:51 +00:00
Nick Craig-Wood	80ffbade22	Fix deletion of some excluded files without --delete-excluded #205 ... This only happened if the destination file was present but the source file was missing.	2015-11-12 11:46:04 +00:00
Nick Craig-Wood	4276abc58b	Version v1.24	2015-11-07 16:23:12 +00:00
Nick Craig-Wood	d42cb11b84	Fix tests to run all tests again and add onedrive	2015-11-07 11:21:15 +00:00
Nick Craig-Wood	365b4babae	Make filter test files pass errcheck	2015-11-07 10:27:47 +00:00
Nick Craig-Wood	2fcb8f5db7	Add support for Microsoft One Drive - fixes #10 ... * Still to do * Copy * Move * MoveDir	2015-11-07 10:19:10 +00:00
Nick Craig-Wood	ff16e0f6df	Factor common error handling into fs module	2015-11-06 15:26:58 +00:00
Nick Craig-Wood	ed72c678f8	Protect accounting from being closed twice	2015-11-06 15:26:58 +00:00
klauspost	b85ddc4e4f	Extend CI tests to include formatting checks. ... CI tests now tests 'go vet', 'go fmt' (via goimports) and golint. Adds Travis experimental OSX support.	2015-11-03 13:50:29 +01:00
klauspost	b872ff0237	Add option to disable server certificate verification. ... The option name mirrors the 'wget' option (also `--no-check-certificate`). The cURL equivalent is called `--insecure`, which is a bit unclear. Put in the "developers" section in documentation with proper warnings. Fixes #168	2015-10-29 16:42:25 +01:00
Nick Craig-Wood	1b95718460	Fix typos in filter docs and unit test assertions	2015-10-20 09:16:47 +01:00
Nick Craig-Wood	4e952af614	Allow spaces in remotes and check remote names for validity at creation time - fixes #171	2015-10-12 17:54:09 +01:00
Klaus Post	6344c3051c	Add async readahead buffer ... This adds an async read buffer of 4x4MB when copying files >10MB. This fixes #164 and reduces the number of IO operations for copy/move.	2015-10-12 08:30:27 +01:00
Nick Craig-Wood	3a900e5bb7	Version v1.23	2015-10-03 16:24:07 +01:00
Nick Craig-Wood	b4d7741611	Improve output of --dump-headers	2015-10-03 16:04:51 +01:00
Nick Craig-Wood	177dbbc29a	Implement rclone size for measuring remotes - fixes #153	2015-10-03 16:04:21 +01:00
Nick Craig-Wood	57aad81b68	Version v1.22	2015-09-28 19:38:20 +01:00
Nick Craig-Wood	a91bcaaeb0	Implement rsync like include and exclude - fixes #27 ... * Implement include/exclude * Implement rsync compatible file globbing * Implement command line filtering flags * --delete-excluded - Delete files on dest excluded from sync * --filter - Add a file-filtering rule * --filter-from - Read filtering patterns from a file * --exclude - Exclude files matching pattern * --exclude-from - Read exclude patterns from file * --include - Include files matching pattern * --include-from - Read include patterns from file * --files-from - Read list of source-file nam * --min-size - Don't transfer any file smaller than this in k or suffix k|M|G * --max-size - Don't transfer any file larger than this in k or suffix k|M|G * Document	2015-09-28 19:18:21 +01:00
Nick Craig-Wood	5d4b149884	Version v1.21	2015-09-22 21:05:11 +01:00