Nick Craig-Wood
7d74686698
fs/accounting: increase maximum burst size of token bucket
...
This stops occasional errors when using --bwlimit which look like this
Token bucket error: rate: Wait(n=2255475) exceeds limiter's burst 2097152
2018-08-30 17:24:08 +01:00
Denis
1c578ced1c
cmd: add copyurl command - Fixes #1320
2018-08-30 16:45:41 +01:00
Nick Craig-Wood
de6ec8056f
fs/accounting: fix moving average speed for file stats
...
Before this change the moving average for the individual file stats
would start at 0 and only converge to the correct value over 15-30
seconds.
This change starts the weighting period as 1 and moves it up once per
sample which gets the average to a better value instantly.
2018-08-28 22:55:51 +01:00
Nick Craig-Wood
66fe4a2523
fs/accounting: fix stats display which was missing transferring data
...
Before this change the total stats were ignoring the in flight
checking, transferring and bytes.
2018-08-28 22:21:17 +01:00
Nick Craig-Wood
f5617dadf3
fs/accounting: factor out eta and percent calculations and write tests
2018-08-28 22:21:17 +01:00
Nick Craig-Wood
ca44fb1fba
accounting: fix time to completion estimates
...
Previous to this change package used for this
github.com/VividCortex/ewma took a 0 average to mean reset the
statistics. This happens quite often when transferring files though a
buffer.
Replace that implementation with a simple home grown one (with about
the same constant), without that feature.
2018-08-26 22:00:48 +01:00
Nick Craig-Wood
94950258a4
fs: allow backends to be named using their Name or Prefix #2449
...
This means that, for example Google Cloud Storage can be known as
`:gcs:bucket` on the command line, as well as `:google cloud
storage:bucket`.
2018-08-26 17:59:31 +01:00
Nick Craig-Wood
8656bd2bb0
fs: Allow on the fly remotes with :backend: syntax - fixes #2449
...
This change allows remotes to be created on the fly without a config
file by using the remote type prefixed with a : as the remote name, Eg
:s3: to make an s3 remote.
This assumes the user is supplying the backend config via command line
flags or environment variables.
2018-08-26 17:59:31 +01:00
Nick Craig-Wood
b6db90cc32
cmd: add --progress/-P flag to show progress
...
Fixes #2347
Fixes #1210
2018-08-26 17:20:38 +01:00
Nick Craig-Wood
9f671c5dd0
fs: fix tests for *SepList
2018-08-21 10:58:59 +01:00
Alex Chen
c6c74cb869
mountlib: fix mount --daemon not working with encrypted config - fixes #2473
...
This passes the configKey to the child process as an Obscured temporary file with an environment variable to the
2018-08-21 09:41:16 +01:00
Nick Craig-Wood
1b8f4b616c
fs: move CommaSepList and SpaceSepList here from config
...
fs can't import config so having them there means they are not usable
by rclone core.
2018-08-20 17:52:05 +01:00
Fabian Möller
f818df52b8
config: add List type
2018-08-20 17:38:51 +01:00
Nick Craig-Wood
7712a0e111
fs/asyncreader: skip some tests to work around race detector bug
...
The race detector currently detects a race with len(chan) against
close(chan).
See: https://github.com/golang/go/issues/27070
Skip the tests which trip this bug under the race detector.
2018-08-20 12:34:29 +01:00
Fabian Möller
6dc5aa7454
docs: clearify buffer-size is per transfer/filehandle
2018-08-17 18:11:40 +01:00
Fabian Möller
552eb8e06b
vfs: try to seek buffer on read only files
2018-08-17 18:10:28 +01:00
Nick Craig-Wood
6390dec7db
fs/accounting: add --stats-one-line flag for single line stats
2018-08-17 17:58:00 +01:00
Nick Craig-Wood
80a3db34a8
fs/accounting: show the total progress of the sync in the stats #379
2018-08-17 17:58:00 +01:00
Nick Craig-Wood
cb7a461287
sync: add a buffer for checks, uploads and renames #379
...
--max-backlog controls the queue length.
Add statistics for the check/upload/rename queues.
This means that checking can complete before the uploads which will
give rclone the ability to show exactly what is outstanding.
2018-08-17 17:58:00 +01:00
Andres Alvarez
990919f268
Add disclaimer about generated passwords being stored in an obscured format
2018-08-11 15:07:50 +01:00
Sebastian Bünger
dd3e912731
fs/OpenOptions: Make FixRangeOption clamp range to filesize.
2018-08-10 11:14:34 +01:00
Nick Craig-Wood
e979cd62c1
rc: fix formatting in docs
2018-08-07 21:05:21 +01:00
reddi1
6bfdabab6d
rc: added core/stats to return the stats - fixes #2405
2018-08-07 20:56:40 +01:00
Andres Alvarez
c1f5add049
Add tests for reveal functions
2018-08-07 20:51:50 +01:00
Oleg Kovalov
06c9f76cd2
all: fix go-critic linter suggestions
2018-08-06 21:14:03 +01:00
Fabian Möller
9815b09d90
fs: add multipliers for SizeSuffix
2018-07-29 18:17:05 +01:00
Fabian Möller
9c90b5e77c
stats: use appropriate Lock func's
2018-07-22 11:33:19 +02:00
Nick Craig-Wood
ab78eb13e4
sync: correct help for --delete-during and --delete-after
2018-07-18 19:30:14 +01:00
ishuah
dcc74fa404
move: fix delete-empty-src-dirs flag to delete all empty dirs on move - fixes #2372
2018-07-17 10:34:34 +01:00
Nick Craig-Wood
4d7d240c12
config: Add advanced section to the config editor
2018-07-16 21:20:47 +01:00
Nick Craig-Wood
d046402d80
config: Make sure Required values are entered
2018-07-16 21:20:47 +01:00
Nick Craig-Wood
9bdf465c10
config: make config wizard understand types and defaults
2018-07-16 21:20:47 +01:00
Nick Craig-Wood
f3f48d7d49
Implement new backend config system
...
This unifies the 3 methods of reading config
* command line
* environment variable
* config file
And allows them all to be configured in all places. This is done by
making the []fs.Option in the backend registration be the master
source of what the backend options are.
The backend changes are:
* Use the new configmap.Mapper parameter
* Use configstruct to parse it into an Options struct
* Add all config to []fs.Option including defaults and help
* Remove all uses of pflag
* Remove all uses of config.FileGet
2018-07-16 21:20:47 +01:00
Nick Craig-Wood
3c89406886
config: Make fs.ConfigFileGet return an exists flag
2018-07-16 08:50:52 +01:00
Nick Craig-Wood
85d09729f2
fs: factor OptionToEnv and ConfigToEnv into fs
2018-07-16 08:50:52 +01:00
Nick Craig-Wood
b3bd2d1c9e
config: add configstruct parser to parse maps into config structures
2018-07-16 08:50:52 +01:00
Nick Craig-Wood
4c586a9264
config: add configmap package to manage config in a generic way
2018-07-16 08:50:52 +01:00
Nick Craig-Wood
1c80e84f8a
fs: Implement Scan method for SizeSuffix and Duration
2018-07-16 08:50:52 +01:00
Nick Craig-Wood
dbb4b2c900
fs/config: don't print errors about --config if supplied - fixes #2397
...
Before this change if the rclone was running in an environment which
couldn't find the HOME directory, it would print a warning about
supplying a --config flag even if the user had done so.
2018-07-15 12:39:11 +01:00
Nick Craig-Wood
a3d9a38f51
fs/fserrors: make sure Cause never returns nil
2018-07-13 10:31:40 +01:00
Nick Craig-Wood
144c1a04d4
fs: Fix parsing of paths under Windows - fixes #2353
...
Before this copyto would parse windows paths incorrectly.
This change moves the parsing code into fspath and makes sure
fspath.Split calls fspath.Parse which does the parsing correctly for
This also renames fspath.RemoteParse to fspath.Parse for consistency
2018-07-06 23:16:43 +01:00
Nick Craig-Wood
8c0335a176
build: fix for goimports format change
...
See https://github.com/golang/go/issues/23709
2018-07-03 22:33:15 +01:00
Nick Craig-Wood
947e10eb2b
config: fix error reading password from piped input - fixes #1308
2018-06-28 11:54:15 +01:00
Nick Craig-Wood
935533e57f
filter: raise --include and --exclude warning to ERROR so it appears without -v
2018-06-22 22:18:55 +01:00
Mateusz
56e1e82005
fs: added weekday schedule into --bwlimit - fixes #1822
2018-06-17 18:38:09 +01:00
Nick Craig-Wood
7110349547
Start v1.42-DEV development
2018-06-16 21:25:58 +01:00
Nick Craig-Wood
a9adb43896
Version v1.42
2018-06-16 18:21:09 +01:00
Nick Craig-Wood
1c75581959
sync: fix TestCopyRedownload after ModifyWindow changes #2310
2018-06-10 17:34:00 +01:00
Filip Bartodziej
ce109ed9c0
log: password prompt output fixed for unix - partially fixes #2220
2018-06-10 12:57:45 +01:00
Nick Craig-Wood
ba43acb6aa
sync: fix TestCopyEmptyDirectories after ModifyWindow changes #2310
2018-06-04 21:41:25 +01:00
ishuah
d758e1908e
copy: create (pseudo copy) empty source directories to destination - fixes #1837
2018-06-04 11:01:14 +01:00
ishuah
737aed8412
Ensure items in srcEmptyDirs are actually empty
2018-06-04 11:01:14 +01:00
Stefan
4009fb67c8
fs: calculate ModifyWindow each time on the fly instead of relying on global state - see #2319 , #2328
2018-06-03 20:45:34 +02:00
Nick Craig-Wood
3ef938ebde
lsf: add --absolute flag to add a leading / onto path names
2018-06-03 10:42:34 +01:00
Kasper Byrdal Nielsen
aad75e6720
check: Add one-way argument
...
--one-way argument will check that all files on source matches the files on detination,
but not the other way. For example files present on destination but not on source will not
trigger an error.
Fixes : #1526
2018-05-31 09:42:16 +01:00
Nick Craig-Wood
3a2248aa5f
rc: add core/gc to run a garbage collection on demand
2018-05-24 15:10:28 +01:00
Nick Craig-Wood
573ef4c8ee
rc: enable go profiling by default on the --rc port
...
This means you can use the pprof tool on a running rclone, eg
go tool pprof http://localhost:5572/debug/pprof/heap
2018-05-24 15:10:28 +01:00
Nick Craig-Wood
512f4b4487
Update error checking on fmt.Fprint* after errcheck update
...
Now we need to check or ignore errors on fmt.Fprint* explicitly -
previously errcheck just ignored them for us.
2018-05-22 09:41:13 +01:00
Nick Craig-Wood
8e625e0bc3
config: add ConfirmWithDefault to change the default on AutoConfig
2018-05-16 09:09:41 +01:00
Nick Craig-Wood
e56be0dfd8
lsf: Add --csv flag for compliant CSV output
2018-05-13 12:18:21 +01:00
Nick Craig-Wood
2a32e2d838
operations: turn ListFormatted into a Format method on ListFormat
2018-05-13 12:17:55 +01:00
Nick Craig-Wood
f77efc7649
lsf: Add 'm' format specifier to show the MimeType
2018-05-13 12:17:55 +01:00
Nick Craig-Wood
aadbcce486
fs: Add MimeTypeDirEntry to return the MimeType of a DirEntry
2018-05-13 12:17:55 +01:00
Nick Craig-Wood
909c3a92d6
lsf: implement 'i' format for showing object ID - fixes #1476
2018-05-13 12:17:55 +01:00
Nick Craig-Wood
826975c341
fs: add Optional ID() method to Object and implement it in backends
...
ID() shows the internal ID of the Object if available.
2018-05-13 12:17:55 +01:00
Nick Craig-Wood
e80ae4e09c
build: remove unused struct fields spotted by structcheck
2018-05-05 17:32:41 +01:00
Nick Craig-Wood
1320e84bc2
build: remove unused code spotted by the deadcode linter
2018-05-05 17:32:41 +01:00
Nick Craig-Wood
cb5bd47e61
build: fix errors spotted by ineffassign linter
...
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
2018-05-05 17:32:41 +01:00
Nick Craig-Wood
f95835d613
fserrors: Look deeper into errors for Fatal/Retry/NoRetry errors.
...
Before this change fatal errors which were wrapped in a system error (eg a
URLError) were not recognised as fatal errors.
2018-05-05 12:58:28 +01:00
Nick Craig-Wood
be79b47a7a
sync: log when we abandon the sync due to a fatal error
2018-05-05 12:58:28 +01:00
Nick Craig-Wood
be22735609
fs/accounting: fix deadlock on GetBytes
...
A deadlock could occur since we have now put a mutex on GetBytes from
StatsInfo.String (s.mu) - progress (acc.statmu) and read (acc.statmu)
- GetBytes (s.mu).
Fix this by giving stringSet its own locking and excluding the call
which caused the deadlock from the mutex in StatsInfo.String.
2018-05-05 12:58:28 +01:00
Nick Craig-Wood
1b1b3c13cd
sync: add a test for aborting on max upload
2018-05-05 12:58:28 +01:00
Nick Craig-Wood
5c128272fd
Implement --max-transfer flag to quit transferring at a limit #1655
2018-05-05 12:58:28 +01:00
Nick Craig-Wood
d178233e74
sync,march: check the cancel context on every channel send and receive
...
This fixes a deadlock on sync when all the copying channels receive a
Fatal Error.
2018-05-05 12:58:28 +01:00
Fabian Möller
9fdf273614
fs: improve ChunkedReader
...
- make Close permanent and return errors afterwards
- use RangeSeek from the wrapped reader if present
- add a limit to chunk growth
- correct RangeSeek interface behavior
- add tests
2018-05-05 12:49:42 +01:00
Nick Craig-Wood
55674c0bfc
Start v1.41-DEV development
2018-04-28 12:37:55 +01:00
Nick Craig-Wood
e4c380b2a8
Version v1.41
2018-04-28 11:46:27 +01:00
Nick Craig-Wood
74cbdea0ef
Revert "copy: create (pseudo copy) empty source directories to destination"
...
Unfortunately this commit attempts to create every directory rather
than just the empty ones, so will need re-working.
Removing this feature for the 1.41 release
This reverts commit 0daced29db
.
2018-04-28 10:02:32 +01:00
ishuah
0daced29db
copy: create (pseudo copy) empty source directories to destination - fixes #1837
2018-04-27 16:15:32 +01:00
Richard Yang
a81ec00a8c
dedupe: Add dedupe largest functionality - fixes #2269
2018-04-26 16:21:07 +01:00
Nick Craig-Wood
9f04ce282e
rc: fix setting bwlimit to unlimited
2018-04-26 12:21:29 +01:00
Nick Craig-Wood
764440068e
filter: fix --min-age and --max-age together check
...
Somehow in the code reorganisation of
11da2a6c9b
the check for --min-age and
--max-age got switched around. This commit fixes that and means you
can use --min-age and --max-age together.
2018-04-26 09:17:22 +01:00
Nick Craig-Wood
a703216286
filter: take double negatives out of filter flag help
2018-04-26 09:17:13 +01:00
Nick Craig-Wood
9ab2521ef2
rc: autogenerate and tidy the docs and commands
...
* Rename rc/pid -> core/pid
* Sort the output of `rc list`
* Make a script to autogenerate the docs
* Tidy docs
2018-04-23 20:57:17 +01:00
Nick Craig-Wood
21a10e58c9
rc: implement core/memstats to print internal memory usage info
2018-04-23 20:49:36 +01:00
Nick Craig-Wood
24980d7123
config: fix typo in error message #2268
2018-04-21 22:49:30 +01:00
Nick Craig-Wood
bcdb7719c6
fs/hash: install QuickXorHash as a supported rclone hash type #2262
2018-04-20 21:02:57 +01:00
Nick Craig-Wood
c51d97c752
hashsum: make generic tool for any hash to produce md5sum like output
2018-04-20 21:02:37 +01:00
Nick Craig-Wood
ef3bcec76c
fs: Extend SizeSuffix to include TB and PB for rclone about
2018-04-17 21:53:42 +01:00
Nick Craig-Wood
1ac6dacf0f
about: complete other providers and re-work internals
...
* Implement about for:
* local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
* Implement `--json` and `---full` flag for `rclone about`
* change About interface to return a Usage structure
* Remove operations.About as it is too thin an interface
* Implement Integration test
Relates to #1138 and #1564
2018-04-17 21:53:27 +01:00
a-roussos
94e277d759
about: add new command 'about' to get quota info from a remote
...
Implemented for drive only.
Relates to #1138 and #1564 .
2018-04-17 21:50:14 +01:00
Nick Craig-Wood
8c3740c2c5
config: Improve the Provider matching to have a negated match #2140
...
This makes it easier to make classes of provider in the config.
2018-04-13 16:06:37 +01:00
Giri Badanahatti
acd5d4377e
config,s3: hierarchical configuration support #2140
...
This introduces a method of making provider specific configuration
within a remote. This is useful particularly in s3.
This commit does the basic configuration in S3 for IBM COS.
2018-04-13 16:05:35 +01:00
Peter Baumgartner
1db68571fd
s3,swift: Add --use-server-modtime
...
`--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.
2018-04-13 13:32:17 +01:00
Nick Craig-Wood
d97fe3b824
fs/operations: make dedupe work with mega
...
* factor into its own files
* remove assumptions about having a given hash type
* make tests work if the remote has no hash
2018-04-13 13:23:55 +01:00
Nick Craig-Wood
dcf8334673
fs: add --dump goroutines and --dump openfiles
...
These are developer flags useful for tracking down resource leaks.
2018-04-11 20:55:58 +01:00
Nick Craig-Wood
37be78705d
fs/fshttp: limit MaxIdleConns and MaxIdleConnsPerHost
...
Before this change mega (which uses a different host per download)
would open too many sockets.
2018-04-11 20:51:28 +01:00
Michael P. Dubner
92b266d361
rc: new call rc/pid - closes #2211
2018-04-09 13:33:04 +01:00
Nick Craig-Wood
cbec59146a
lsf: make sure we use localtime in tests - fixes Box integration tests
...
This problem was introduced with eca99b33c0
. It seems Box is the only
remote which converts time zones, so if you give it a GMT time zone,
it returns a PST time zone which represents the same instant.
2018-04-09 11:46:49 +01:00
Nick Craig-Wood
be54fd8f70
Remove builds conditional on go1.7 since that is now guaranteed #2154
...
Old fallback code was deleted and the go1.7 style code inlined where
appropriate.
2018-04-07 11:42:55 +01:00
Nick Craig-Wood
e5be471ce0
Use io.SeekStart/End/Current constants now for go1.7+ #2154
2018-04-07 11:42:36 +01:00
Nick Craig-Wood
80588a5a6b
Replace "golang.org/x/net/context" with "context" for go1.7+ #2154
2018-04-07 11:42:08 +01:00
Nick Craig-Wood
67023f0040
Require go1.7 for compilation #2154
...
* Update the travis tests to exclude go1.6
* Update the compile check to require go1.7+
* Update misc go1.6 workarounds marked in the source
2018-04-06 20:18:14 +01:00
Matt Holt
e13f65b953
serve restic: Print actual listener address
2018-04-04 14:56:26 +01:00
Nick Craig-Wood
eca99b33c0
lsd,lsf: make sure all times we output are in local time - fixes #2183
...
Previous to this change times from lsd/lsf were output in whatever
timezone they were in whereas times from lsl were converted to
localtime.
2018-04-01 15:40:04 +01:00
Stefan
a8267d1628
link: allow creating public link to files and folders - closes #1562
2018-03-29 09:10:19 +02:00
Mateusz Pabian
91b068ad3a
sync: implement --ignore-errors - fixes #642
2018-03-23 22:01:10 +00:00
Nick Craig-Wood
940df88eb2
Start v1.40-DEV development
2018-03-19 14:20:48 +00:00
Nick Craig-Wood
26f1c55987
Version v1.40
2018-03-19 10:06:13 +00:00
Stefan
86892467d9
config: load config file only on first access ( closes #1659 , closes #2096 ) ( #2147 )
2018-03-17 12:36:30 +01:00
Nick Craig-Wood
4295428a0f
fs/accounting: add remote control of bwlimit
2018-03-17 10:34:02 +00:00
Nick Craig-Wood
5bf639048f
sync: log an error that --track-renames doesn't work with sync or move
...
Fixes #2008
2018-03-17 10:34:02 +00:00
remusb
97b48cf988
rc: add support for Go 1.6
2018-03-14 22:58:20 +02:00
Nick Craig-Wood
86e5a35491
Implement Remote Control for rclone #2111
...
This implements a remote control protocol activated with the --rc flag
and a new command `rclone rc` to use that interface.
Still to do
* docs - need finishing
* tests
2018-03-14 22:58:20 +02:00
Nick Craig-Wood
dfd0f4c5a4
sync: when using --backup-dir don't delete files if we can't set their modtime
...
This is a problem when syncing a file which just needed its modtime
set with dropbox which can't set the mod time of a file without
re-uploading it.
Before this change we would delete the file, then the server side move
would fail moving the file to the backup-dir because it no longer
existed.
After this change the destination file is moved to the backup-dir
instead of being deleted and the new file is uploaded.
Fixes #2134
2018-03-13 16:05:06 +00:00
Nick Craig-Wood
0c9dc006c5
fs: make display of default values of --min-age/--max-age be off - Fixes #2121
2018-03-13 09:06:07 +00:00
Remus Bunduc
70f07fd3ac
fs: add ChangeNotify and backend support for it ( #2094 )
...
* fs: rename DirChangeNotify to ChangeNotify
* cache: switch to ChangeNotify
* ChangeNotify: keep order of notifications
2018-03-08 22:03:34 +02:00
Nick Craig-Wood
e0eb666dbf
fs/walk: fix new golint warning about unused variables in range
2018-03-02 17:01:58 +00:00
Nick Craig-Wood
abc736df1d
cat: Use RangeOption for limited fetches to make more efficient #1825
2018-02-19 18:00:39 +00:00
Nick Craig-Wood
ab0d06eb16
fs: Make RangeOption mandatory #1825
2018-02-19 18:00:39 +00:00
Mateusz
afc963ed92
config: retry saving the config after failure - fixes #2060
2018-02-19 17:59:27 +00:00
Fabian Möller
451cd6d971
fs: add ChunkedReader
2018-02-19 15:45:24 +00:00
Fabian Möller
a647c54888
fs: add RangeSeeker interface
2018-02-19 15:45:24 +00:00
Nick Craig-Wood
a5c3bcc9c7
fshttp: fix idle timeouts for HTTP connections #2057
...
Now we only nudge on the idle timeout after a successful Read or Write
which returns some bytes and no errors.
2018-02-16 10:35:41 +00:00
Bjørn Erik Pedersen
ab8c0a81fa
Add a delete threshold to sync (--max-delete)
...
Fixes #959
2018-02-12 11:29:58 +00:00
Nick Craig-Wood
d0d6b83a7a
fs/accounting: rework to enable accounting to work with crypt and b2
...
This removes the old system of part accounting and replaces it with a
system of popping off the accounting reader and wrapping up new ones
as necessary.
This makes it much easier to carry the context down the chain of
wrapped readers and get the limiting as near as possible to the
output. This makes the accounting more accurate and the bandwidth
limiting smoother.
Fixes #2029 and Fixes #1443
2018-02-02 15:14:41 +00:00
Nick Craig-Wood
bea02fcf52
fs/accounting: factor into separate files without changing functionality
2018-02-02 15:14:40 +00:00
Nick Craig-Wood
6fb868e00c
config: fix --log-level flag after code reorganization - fixes #2043
2018-02-02 14:07:44 +00:00
Nick Craig-Wood
8df78f2b6d
operations: ignore size of objects when they are < 0 #320
...
This allows google docs to be transferred and checked correctly.
2018-01-31 16:22:05 +00:00
Nick Craig-Wood
2eb5cfb7ad
fs: Formalize the ObjectUnWrapper interface
2018-01-31 16:21:41 +00:00
Nick Craig-Wood
fe52502f19
fs: Adjust RangeOption.Decode to return -1 for read to end
...
A Range request can never request 0 bytes however this change was made
to make a clearer signal that the limit means read to the end.
Add test and more documentation and fixup uses
2018-01-27 14:31:29 +00:00
Nick Craig-Wood
9a73688e3a
fs: Add ParseRangeOption to parse incoming Range: requests
2018-01-27 13:16:37 +00:00
Nick Craig-Wood
bc3ee977f4
fs/hash: move interface assertion to tests so it doesn't pull in spf13/flag
2018-01-26 14:35:18 +00:00
Nick Craig-Wood
a6833b68ca
local: factor RangeOption code to Decode() method and readers.LimitedReadCloser #1825
2018-01-22 19:44:00 +00:00
Nick Craig-Wood
d876392d15
onedrive: Factor code into fs.FixRangeOption
2018-01-22 17:05:00 +00:00
Nick Craig-Wood
84bc4dc142
Clarify RangeOption semantics
2018-01-21 09:51:28 +00:00
Nick Craig-Wood
ee66419a27
fs/fserrors: Add test for error from #1964
2018-01-19 17:07:40 +00:00
Nick Craig-Wood
5ad226ab54
fs: Add dir option to fs.Purge #1891
...
Purge optional interface needs fixing too.
2018-01-19 11:45:50 +00:00
Nick Craig-Wood
97c414f025
config/hash: rename more symbols after factoring into own package
2018-01-18 20:27:52 +00:00
Nick Craig-Wood
71722b5b95
config: factor Obscure and Reveal into its own package
2018-01-18 20:19:55 +00:00
Ernest Borowski
5cce74d630
flags: remove --no-traverse flag because it is obsolete - fixes #1813
...
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
2018-01-18 11:00:25 +00:00
Iakov Davydov
acd55a8f65
local, fs: --exclude-if-present ignores directories which it doesn't have permission for - fixes #1959
2018-01-16 20:00:16 +00:00
Nick Craig-Wood
11da2a6c9b
Break the fs package up into smaller parts.
...
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.
The new code layout is documented in CONTRIBUTING.
2018-01-15 17:51:14 +00:00
Nick Craig-Wood
b8b620f5c2
Move all backends into backend directory
2018-01-12 17:07:38 +00:00
Nick Craig-Wood
712e6a8085
lsf: fix integration tests
2018-01-11 13:52:15 +00:00
Will Gunn
5c7e8d5a2b
fs: Add --stats-file-name-length to specify the printed file name length for stats
...
Fixes #1206
2018-01-10 20:32:36 +00:00
Nick Craig-Wood
c74c3b37da
lsf: add option to print hashes
2018-01-06 17:53:37 +00:00
Nick Craig-Wood
7c71ee1a5b
fs: fix TestListFormat on remotes which return 0 as dir size not -1
2018-01-06 17:47:42 +00:00
Jakub Tasiemski
0d041602cf
cmd: new command lsf
2018-01-06 14:39:31 +00:00
Nick Craig-Wood
b412c745a1
Start v1.39-DEV development
2017-12-23 13:40:28 +00:00
Nick Craig-Wood
f34a9116d4
Version v1.39
2017-12-23 13:07:45 +00:00
Nick Craig-Wood
7e585cda96
fs: fix TestRmdirsLeaveRoot test
2017-12-14 08:57:28 +00:00
lewapm
9c242edc10
rmdirs: add --leave-root flag - fixes #1874
2017-12-13 10:23:54 +00:00
ishuah
b05e472d2e
stats: condensed transfer output to fit 80x25 terminals
2017-12-09 10:48:36 +03:00
remusb
25b073c767
fs: add Wrap feature for FS to identify their parent FS ( #1884 )
2017-12-06 17:14:34 +02:00
Nick Craig-Wood
fa4a25a73b
fs: only test one level of cache
...
Can't test multiple caches at once as can only have 1 DB open at once
2017-12-04 15:50:59 +00:00
Ernest Borowski
934df67aef
filter: warn the user if he use --include and --exclude together fixes #1764
...
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
2017-12-04 14:20:01 +00:00
Nick Craig-Wood
4b185355df
fs: rcat - use in memory object and Copy for more reliable transfers
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
7d15c33e42
fs: make Copy and Move return the destination object if possible
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
11332a19a0
fs: make an in memory object for short transfers
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
639e812789
fs: integration tests: add SUMMARY heading for log scraping
2017-11-29 15:55:37 +00:00
Nick Craig-Wood
1c6cad2252
fs: integration tests: add 30 minute timeout per test
2017-11-29 13:51:17 +00:00
ishuah
aab8051f50
move: add --delete-empty-src-dirs flag - fixes #1854
2017-11-28 11:38:19 +03:00
Nick Craig-Wood
fdb01437d8
fs: Allow the http Transport to have an optional filter request function
2017-11-24 09:07:56 +00:00
Nick Craig-Wood
02ffd43572
fs: Save the config before asking for a token - fixes #1220
...
Before this if the client_id/client_secret was edited it would
disappear when asking for the new token.
This means the post config is done after the user has confirmed the
config is OK which can't be helped.
2017-11-23 14:01:32 +00:00
Nick Craig-Wood
e53892f53b
fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use
...
RepeatableReaderSized has a pre-allocated buffer which should help
with memory usage - before it grew the buffer. Since we know the size
of the chunks, pre-allocating it should be much more efficient.
RepeatableReaderBuffer uses the buffer passed in.
RepeatableLimit* are convenience funcitions for wrapping a reader in
an io.LimitReader and then a RepeatableReader with the same buffer
size.
2017-11-23 13:53:46 +00:00
ishuah
6c62fced60
move: fixed root source directories getting deleted after move - fixes #1849
2017-11-23 12:01:35 +03:00
Nick Craig-Wood
159fce0106
fs: fix --cache-dir to have some effect
2017-11-22 17:05:02 +00:00
Nick Craig-Wood
cbe5d7ce64
fs: Remove X-Auth-Token: from headers when dumping for swift
2017-11-21 17:32:07 +00:00
Nick Craig-Wood
1a65a4e769
fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters
...
Now --dump-flag is written as --dump flag. This is a comma separated list which can contain
* headers - HTTP headers as before
* bodies - HTTP bodies as before
* requests - HTTP request bodies
* responses - HTTP response bodies
* auth - HTTP auth
* filters - Filter rexeps
Leave --dump-headers and --dump-bodies for the time being but remove
the other --dump-* flags as they aren't used very often.
2017-11-21 17:32:07 +00:00
ishuah
b4aa920a3d
stats: show the amount of data transferred in kb/mb - fixes #1167
2017-11-21 12:40:02 +03:00
Nick Craig-Wood
8951875c21
vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables
2017-11-20 12:46:21 +00:00
Nick Craig-Wood
e946a8eab0
fs: Add CacheDir config variable
2017-11-20 12:00:32 +00:00
Nick Craig-Wood
46947b3b9b
rcat: fix goroutine leak
...
This was leaking goroutines in the short file case beause it wasn't
calling Close() on the Account object. This became apparent when
testing with mount.
2017-11-20 12:00:32 +00:00
Iakov Davydov
7d3abdc463
tests for --exclude-if-present
2017-11-17 21:51:11 +00:00
Iakov Davydov
538246f6c3
support exclude file in --fast-list mode
2017-11-17 21:51:11 +00:00
Iakov Davydov
557dd8f031
ListDirSorted check for excludefile
2017-11-17 21:51:11 +00:00
Iakov Davydov
37aaa19f3a
new option: --exclude-if-present
2017-11-17 21:51:11 +00:00
Iakov Davydov
cef2e3bf83
path -> startPath in walkRDirTree (we need the path package)
2017-11-17 21:51:11 +00:00
Iakov Davydov
a3a436ce16
WalkRDirTree: return error if unknown item type
2017-11-17 21:51:11 +00:00
Iakov Davydov
5d05df3124
ListContainsExcludeFile: checks for exclude file in the list
2017-11-17 21:51:11 +00:00
Iakov Davydov
421ba84e12
DirTree.Prune: deletes several directories
2017-11-17 21:51:11 +00:00
Iakov Davydov
7ae7080824
FileExists check if a file exists
2017-11-17 21:51:11 +00:00
Remus Bunduc
016abf825e
cache: first version
2017-11-15 15:23:21 +00:00
ishuah
aec2265be0
rclone: implement exit codes - #1136
2017-11-15 17:48:37 +03:00
Substantiel
2423fa40e2
config: add password sub command for setting obscured passwords
2017-11-15 14:44:45 +00:00
ishuah
c65734ee69
move: delete source directory after successful move - fixes #1642
2017-11-07 22:21:38 +00:00
Substantiel
7749157596
Add --auto-confirm flag
2017-11-05 21:56:50 +00:00
Nick Craig-Wood
cc7ed13b9b
fs: factor test running code into fstest/run.go
2017-11-03 12:59:58 +00:00
ishuah
10f27e2ff2
allow trailing+leading whitespace for passwords - #1717
...
warn users when they enter passwords with leading/trailing whitespaces
Updated config_test.go, removing deprecated test case and updated TestReveal
2017-10-20 15:46:17 +01:00
Nick Craig-Wood
115053930e
Make error messages less crypting when revealing an unobscured password - fixes #1743
2017-10-16 22:03:06 +01:00
Nick Craig-Wood
edfab09eb9
config: add sub commands for full config file management
...
Previously config sub commands were manually parsed rather than using
cobra.
Make config command have the following sub commands:
* create Create a new remote with name, type and options.
* delete Delete an existing remote <name>.
* dump Dump the config file as JSON.
* edit Enter an interactive configuration session.
* file Show path of configuration file in use.
* providers List in JSON format all the providers and options.
* show Print (decrypted) config file, or the config for a single remote.
* update Update options in an existing remote.
The following changes were made to existing commands
* listproviders was renamed to providers
* listoptions was removed in favour of providing the output in providers
* jsonconfig was renamed to create
* an optional parameter was added to the show command
2017-10-14 11:50:41 +01:00
thierry
0575623dff
Add config listproviders, listoptions, jsonconfig for automated config
...
Addition of a method listing the providers, a method listing the
options of a provider and method of manual configuration.
2017-10-13 17:17:36 +01:00
Nick Craig-Wood
fc8b13c993
moveto/copyto: Fix to allow copying to the same name - fixes #1736
2017-10-12 20:45:36 +01:00
Nick Craig-Wood
43ced30f11
fs: Add more errors to retry - fixes #1733
2017-10-10 19:51:02 +01:00
Nick Craig-Wood
3eb7f52e39
fs: Add "unexpected EOF reading trailer" as a retriable error - fixes #1730
2017-10-09 17:29:16 +01:00
Nick Craig-Wood
7f3dc9b5c4
Implement WebDAV remote #580
...
This has special knowledge of Owncloud and Nextcloud to enable more
functionality such as mod times.
2017-10-09 16:19:37 +01:00
Nick Craig-Wood
00fe6d95da
fs: fix duplicate files causing spurious copies
...
Before this fix duplicate files (on Google Drive) caused the next file
to be spuriously copied. `rclone dedupe` worked around the problem.
2017-10-02 16:52:53 +01:00
Nick Craig-Wood
9e9297838f
Implement pcloud remote - #418
2017-10-01 11:37:35 +01:00
Nick Craig-Wood
1c10497b68
Start v1.38-DEV development
2017-09-30 15:16:09 +01:00
Nick Craig-Wood
d96e45ba5b
Version v1.38
2017-09-30 14:20:43 +01:00
Nick Craig-Wood
657b3a674d
fs: fix test_all -clean to run just one cleaning thread per remote
2017-09-30 11:07:09 +01:00
Nick Craig-Wood
2857ed5c35
fs: fix --immutable tests on remotes which don't have modtime
2017-09-28 08:56:30 +01:00
Jacob McNamee
2d8e75cab4
Implement --immutable option
2017-09-24 20:00:00 +01:00
Stefan
85877f3adc
config: add show/file subcommands which print the config/its path ( fixes #1086 )
2017-09-19 17:59:19 +02:00
Nick Craig-Wood
87335de8a8
fs: fix filename normalization issues in the tests when running on OS X
2017-09-17 15:31:22 +01:00
Stefan Breunig
12405f9f41
fuse: re-use rcat to support uploads for all remotes ( fixes #1672 )
2017-09-16 22:49:08 +02:00
Stefan
234bfae0d5
b2: implement streaming upload of files with unknown length (see #1614 ) ( closes #1686 )
2017-09-16 22:43:48 +02:00
Nick Craig-Wood
4ac9a65049
fs: stop normalizing file names but do a normalized compare in the sync
...
This works by using a transform function to transform file names when
doing a compare when matching file names in a directory. rclone now
UTF-8 normalizes the file names and does a case insensitive compare if
the destination remote is case insensitive.
This deprecates the --local-no-unicode-normalization flag.
Fixes #1477
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
a8e41f081c
fs: re-implement check and cryptcheck using the same traversal as sync
...
This makes them 100% consistent with sync and also make them use less
memory as they no longer build the whole tree in memory first.
Fixes #1657
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
261c7ad9e4
fs: make syncCopyMove use context for go routine cancellation
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
fe96d5cf0a
fs: factor multiple directory traverse out of sync
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
6df12b3f00
fs: improve retriable error detection
2017-09-16 19:48:49 +01:00
Stefan Breunig
9d3f11b493
amazonclouddrive, rcat: ensure rcat integration test passes even with AmazonCloudDrive ( fixes : #1680 )
2017-09-15 18:09:04 +02:00
Nick Craig-Wood
798502b204
fs: add more errors to be considered temporary errors
...
This makes a framework for adding temporary errors identified by
syscall number or by error string.
Fixes #1660
2017-09-14 18:01:43 +01:00
Stefan Breunig
56dedc49e3
rcat: properly report if the upload fails
2017-09-13 20:21:52 +02:00
Girish Ramakrishnan
4c45cbea18
copy: error out if dst could not be listed
2017-09-12 09:29:44 +01:00
Stefan Breunig
a91448c83a
rcat: honor --dry-run even for small files
2017-09-11 22:28:16 +02:00
Stefan Breunig
80b1f2a494
rcat: configurable small files cutoff and implement proper upload verification
2017-09-11 08:26:53 +02:00
Stefan Breunig
57817397a0
rcat: directly upload small files without streaming them
2017-09-11 08:25:34 +02:00
Nick Craig-Wood
cbea06026a
Make check obey --ignore-size - fixes #1643
2017-09-01 11:20:41 +01:00
Nick Craig-Wood
2c8d6e86cc
fs: fix gofmt
2017-08-31 16:01:19 +01:00
cbruegg
bb6300b032
Fix bwlimit toggle in conjunction with schedules ( Fixes #1607 )
2017-08-31 15:33:29 +01:00
Stefan Breunig
a122b9fa7a
yandex: implement streaming uploads (see #1614 )
2017-08-19 14:07:23 +02:00
Alex McGrath Kraak
2cc9071791
http: add --user-agent option. close #1557
2017-08-17 06:49:27 +02:00
Nick Craig-Wood
265fb8a5e2
fs: Manage empty directories - fixes #100
...
During the sync we collect a list of directories which should be empty
and attempt to rmdir them at the end of the sync. If the directories
are not empty then the rmdir will fail, logging a message but not
erroring the sync.
2017-08-09 21:07:00 +01:00
Nick Craig-Wood
8a1a900733
fstest: use Feature.CanHaveEmptyDirectories to sharpen tests
...
Now we actually test whether the directories are present or not,
filtering out empty directories in the test using the
CanHaveEmptyDirectories flag.
2017-08-09 20:55:08 +01:00
Nick Craig-Wood
20ae7d562b
fs: Add CanHaveEmptyDirectories and BucketBased feature flags to all remotes
2017-08-09 20:55:08 +01:00
Nick Craig-Wood
ec2ea37ad2
fs: Add --disable flag to disable optional features - fixes #1551
...
Eg to disable server side copy use `--disable copy`, to see a list of
what you can disable, `--disable help`.
2017-08-07 21:34:45 +01:00
Nick Craig-Wood
bfe812ea6b
dedupe: implement merging of duplicate directories - fixes #1243
2017-08-07 15:36:41 +01:00
Nick Craig-Wood
db1995e63a
Add MergeDirs optional interface and implement it for drive
2017-08-07 15:32:47 +01:00
Nick Craig-Wood
81a2ab599f
fs: add optional ID to fs.Directory and set it in the remotes which care
2017-08-07 15:31:22 +01:00
Stefan Breunig
40f24e0ea3
config: use absolute ConfigPath to ensure newly written config is on the same mount - fixes #1569
2017-08-05 12:13:25 +02:00
Nick Craig-Wood
92d2e1f8d7
azureblob: rework and complete #801
...
* Fixup bitrot (rclone and Azure library)
* Implement Copy
* Add modtime to metadata under mtime key as RFC3339Nano
* Make multipart upload work
* Make it pass the integration tests
* Fix uploading of zero length blobs
* Rename to azureblob as it seems likely we will do azurefile
* Add docs
2017-08-04 22:56:16 +01:00
Andrei Dragomir
036fd61a50
Added Azure Blob storage support #801
2017-08-04 22:54:27 +01:00
wuyu
ec5b72f8d5
Add new QingStor remote
...
Add new package qingstor to support QingStor API.
Add new unit test for its and tested through; But I commented
on some tests case because of some of the features of QingStor.
Add new docs for it.
2017-08-04 17:25:47 +01:00
Nick Craig-Wood
f682002b84
fs: Make tests create a new bucket rather than purging the old one
...
This enables QingStor to pass the tests as it has a 2 minute lockout
on deleting the old bucket then creating it again.
2017-08-04 17:09:28 +01:00
Stefan Breunig
28a18303f3
implement rcat – fixes #230 , fixes #1001
2017-08-03 21:42:35 +02:00
Nick Craig-Wood
3e3a59768e
fs/test_all: fix after fstest factorisation
2017-08-03 20:01:05 +01:00
Nick Craig-Wood
e01741b557
fs: Cleaning up directories in test is no longer needed
...
..as it is done in the finalise method.
2017-08-03 20:01:05 +01:00
Nick Craig-Wood
71fe046937
fs: Add Find method to DirTree
2017-07-26 16:38:53 +01:00
Nick Craig-Wood
d5ff7104e5
fs: Implement NewDirTree for non --fast-list
2017-07-26 16:38:44 +01:00
Nick Craig-Wood
cd4895690a
fstest: Factor test initialisation into Initialise()
2017-07-26 16:38:33 +01:00
Nick Craig-Wood
1ecf2bcbd5
fs: fix typo in --bind description
2017-07-23 23:08:33 +01:00
Nick Craig-Wood
c3d6cc91ec
Fix --bind flag changes under go1.6
...
Correcting 9f24639568
2017-07-23 22:36:32 +01:00
Nick Craig-Wood
9f24639568
Add --bind flag for choosing the local addr on outgoing connections - fixes #1087
...
Supported by all remotes except FTP.
2017-07-23 16:27:39 +01:00
Nick Craig-Wood
7b81f12dad
box: add docs
...
* reorder remotes so they are in alphabetical order by full name everywhere
* update CONTRIBUTING doc
2017-07-23 11:32:34 +01:00
Nick Craig-Wood
d279161cee
Implement box storage remote - #97
2017-07-23 11:32:34 +01:00
Nick Craig-Wood
db6009126d
Fix test failure with new stretchr/testify - fixes #1550
2017-07-23 08:59:07 +01:00
Nick Craig-Wood
c8b5ee1e54
Start v1.37-DEV development
2017-07-22 20:43:06 +01:00