Nick Craig-Wood
712e6a8085
lsf: fix integration tests
2018-01-11 13:52:15 +00:00
Will Gunn
5c7e8d5a2b
fs: Add --stats-file-name-length to specify the printed file name length for stats
...
Fixes #1206
2018-01-10 20:32:36 +00:00
Nick Craig-Wood
c74c3b37da
lsf: add option to print hashes
2018-01-06 17:53:37 +00:00
Nick Craig-Wood
7c71ee1a5b
fs: fix TestListFormat on remotes which return 0 as dir size not -1
2018-01-06 17:47:42 +00:00
Jakub Tasiemski
0d041602cf
cmd: new command lsf
2018-01-06 14:39:31 +00:00
Nick Craig-Wood
b412c745a1
Start v1.39-DEV development
2017-12-23 13:40:28 +00:00
Nick Craig-Wood
f34a9116d4
Version v1.39
2017-12-23 13:07:45 +00:00
Nick Craig-Wood
7e585cda96
fs: fix TestRmdirsLeaveRoot test
2017-12-14 08:57:28 +00:00
lewapm
9c242edc10
rmdirs: add --leave-root flag - fixes #1874
2017-12-13 10:23:54 +00:00
ishuah
b05e472d2e
stats: condensed transfer output to fit 80x25 terminals
2017-12-09 10:48:36 +03:00
remusb
25b073c767
fs: add Wrap feature for FS to identify their parent FS ( #1884 )
2017-12-06 17:14:34 +02:00
Nick Craig-Wood
fa4a25a73b
fs: only test one level of cache
...
Can't test multiple caches at once as can only have 1 DB open at once
2017-12-04 15:50:59 +00:00
Ernest Borowski
934df67aef
filter: warn the user if he use --include and --exclude together fixes #1764
...
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
2017-12-04 14:20:01 +00:00
Nick Craig-Wood
4b185355df
fs: rcat - use in memory object and Copy for more reliable transfers
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
7d15c33e42
fs: make Copy and Move return the destination object if possible
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
11332a19a0
fs: make an in memory object for short transfers
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
639e812789
fs: integration tests: add SUMMARY heading for log scraping
2017-11-29 15:55:37 +00:00
Nick Craig-Wood
1c6cad2252
fs: integration tests: add 30 minute timeout per test
2017-11-29 13:51:17 +00:00
ishuah
aab8051f50
move: add --delete-empty-src-dirs flag - fixes #1854
2017-11-28 11:38:19 +03:00
Nick Craig-Wood
fdb01437d8
fs: Allow the http Transport to have an optional filter request function
2017-11-24 09:07:56 +00:00
Nick Craig-Wood
02ffd43572
fs: Save the config before asking for a token - fixes #1220
...
Before this if the client_id/client_secret was edited it would
disappear when asking for the new token.
This means the post config is done after the user has confirmed the
config is OK which can't be helped.
2017-11-23 14:01:32 +00:00
Nick Craig-Wood
e53892f53b
fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use
...
RepeatableReaderSized has a pre-allocated buffer which should help
with memory usage - before it grew the buffer. Since we know the size
of the chunks, pre-allocating it should be much more efficient.
RepeatableReaderBuffer uses the buffer passed in.
RepeatableLimit* are convenience funcitions for wrapping a reader in
an io.LimitReader and then a RepeatableReader with the same buffer
size.
2017-11-23 13:53:46 +00:00
ishuah
6c62fced60
move: fixed root source directories getting deleted after move - fixes #1849
2017-11-23 12:01:35 +03:00
Nick Craig-Wood
159fce0106
fs: fix --cache-dir to have some effect
2017-11-22 17:05:02 +00:00
Nick Craig-Wood
cbe5d7ce64
fs: Remove X-Auth-Token: from headers when dumping for swift
2017-11-21 17:32:07 +00:00
Nick Craig-Wood
1a65a4e769
fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters
...
Now --dump-flag is written as --dump flag. This is a comma separated list which can contain
* headers - HTTP headers as before
* bodies - HTTP bodies as before
* requests - HTTP request bodies
* responses - HTTP response bodies
* auth - HTTP auth
* filters - Filter rexeps
Leave --dump-headers and --dump-bodies for the time being but remove
the other --dump-* flags as they aren't used very often.
2017-11-21 17:32:07 +00:00
ishuah
b4aa920a3d
stats: show the amount of data transferred in kb/mb - fixes #1167
2017-11-21 12:40:02 +03:00
Nick Craig-Wood
8951875c21
vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables
2017-11-20 12:46:21 +00:00
Nick Craig-Wood
e946a8eab0
fs: Add CacheDir config variable
2017-11-20 12:00:32 +00:00
Nick Craig-Wood
46947b3b9b
rcat: fix goroutine leak
...
This was leaking goroutines in the short file case beause it wasn't
calling Close() on the Account object. This became apparent when
testing with mount.
2017-11-20 12:00:32 +00:00
Iakov Davydov
7d3abdc463
tests for --exclude-if-present
2017-11-17 21:51:11 +00:00
Iakov Davydov
538246f6c3
support exclude file in --fast-list mode
2017-11-17 21:51:11 +00:00
Iakov Davydov
557dd8f031
ListDirSorted check for excludefile
2017-11-17 21:51:11 +00:00
Iakov Davydov
37aaa19f3a
new option: --exclude-if-present
2017-11-17 21:51:11 +00:00
Iakov Davydov
cef2e3bf83
path -> startPath in walkRDirTree (we need the path package)
2017-11-17 21:51:11 +00:00
Iakov Davydov
a3a436ce16
WalkRDirTree: return error if unknown item type
2017-11-17 21:51:11 +00:00
Iakov Davydov
5d05df3124
ListContainsExcludeFile: checks for exclude file in the list
2017-11-17 21:51:11 +00:00
Iakov Davydov
421ba84e12
DirTree.Prune: deletes several directories
2017-11-17 21:51:11 +00:00
Iakov Davydov
7ae7080824
FileExists check if a file exists
2017-11-17 21:51:11 +00:00
Remus Bunduc
016abf825e
cache: first version
2017-11-15 15:23:21 +00:00
ishuah
aec2265be0
rclone: implement exit codes - #1136
2017-11-15 17:48:37 +03:00
Substantiel
2423fa40e2
config: add password sub command for setting obscured passwords
2017-11-15 14:44:45 +00:00
ishuah
c65734ee69
move: delete source directory after successful move - fixes #1642
2017-11-07 22:21:38 +00:00
Substantiel
7749157596
Add --auto-confirm flag
2017-11-05 21:56:50 +00:00
Nick Craig-Wood
cc7ed13b9b
fs: factor test running code into fstest/run.go
2017-11-03 12:59:58 +00:00
ishuah
10f27e2ff2
allow trailing+leading whitespace for passwords - #1717
...
warn users when they enter passwords with leading/trailing whitespaces
Updated config_test.go, removing deprecated test case and updated TestReveal
2017-10-20 15:46:17 +01:00
Nick Craig-Wood
115053930e
Make error messages less crypting when revealing an unobscured password - fixes #1743
2017-10-16 22:03:06 +01:00
Nick Craig-Wood
edfab09eb9
config: add sub commands for full config file management
...
Previously config sub commands were manually parsed rather than using
cobra.
Make config command have the following sub commands:
* create Create a new remote with name, type and options.
* delete Delete an existing remote <name>.
* dump Dump the config file as JSON.
* edit Enter an interactive configuration session.
* file Show path of configuration file in use.
* providers List in JSON format all the providers and options.
* show Print (decrypted) config file, or the config for a single remote.
* update Update options in an existing remote.
The following changes were made to existing commands
* listproviders was renamed to providers
* listoptions was removed in favour of providing the output in providers
* jsonconfig was renamed to create
* an optional parameter was added to the show command
2017-10-14 11:50:41 +01:00
thierry
0575623dff
Add config listproviders, listoptions, jsonconfig for automated config
...
Addition of a method listing the providers, a method listing the
options of a provider and method of manual configuration.
2017-10-13 17:17:36 +01:00
Nick Craig-Wood
fc8b13c993
moveto/copyto: Fix to allow copying to the same name - fixes #1736
2017-10-12 20:45:36 +01:00
Nick Craig-Wood
43ced30f11
fs: Add more errors to retry - fixes #1733
2017-10-10 19:51:02 +01:00
Nick Craig-Wood
3eb7f52e39
fs: Add "unexpected EOF reading trailer" as a retriable error - fixes #1730
2017-10-09 17:29:16 +01:00
Nick Craig-Wood
7f3dc9b5c4
Implement WebDAV remote #580
...
This has special knowledge of Owncloud and Nextcloud to enable more
functionality such as mod times.
2017-10-09 16:19:37 +01:00
Nick Craig-Wood
00fe6d95da
fs: fix duplicate files causing spurious copies
...
Before this fix duplicate files (on Google Drive) caused the next file
to be spuriously copied. `rclone dedupe` worked around the problem.
2017-10-02 16:52:53 +01:00
Nick Craig-Wood
9e9297838f
Implement pcloud remote - #418
2017-10-01 11:37:35 +01:00
Nick Craig-Wood
1c10497b68
Start v1.38-DEV development
2017-09-30 15:16:09 +01:00
Nick Craig-Wood
d96e45ba5b
Version v1.38
2017-09-30 14:20:43 +01:00
Nick Craig-Wood
657b3a674d
fs: fix test_all -clean to run just one cleaning thread per remote
2017-09-30 11:07:09 +01:00
Nick Craig-Wood
2857ed5c35
fs: fix --immutable tests on remotes which don't have modtime
2017-09-28 08:56:30 +01:00
Jacob McNamee
2d8e75cab4
Implement --immutable option
2017-09-24 20:00:00 +01:00
Stefan
85877f3adc
config: add show/file subcommands which print the config/its path ( fixes #1086 )
2017-09-19 17:59:19 +02:00
Nick Craig-Wood
87335de8a8
fs: fix filename normalization issues in the tests when running on OS X
2017-09-17 15:31:22 +01:00
Stefan Breunig
12405f9f41
fuse: re-use rcat to support uploads for all remotes ( fixes #1672 )
2017-09-16 22:49:08 +02:00
Stefan
234bfae0d5
b2: implement streaming upload of files with unknown length (see #1614 ) ( closes #1686 )
2017-09-16 22:43:48 +02:00
Nick Craig-Wood
4ac9a65049
fs: stop normalizing file names but do a normalized compare in the sync
...
This works by using a transform function to transform file names when
doing a compare when matching file names in a directory. rclone now
UTF-8 normalizes the file names and does a case insensitive compare if
the destination remote is case insensitive.
This deprecates the --local-no-unicode-normalization flag.
Fixes #1477
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
a8e41f081c
fs: re-implement check and cryptcheck using the same traversal as sync
...
This makes them 100% consistent with sync and also make them use less
memory as they no longer build the whole tree in memory first.
Fixes #1657
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
261c7ad9e4
fs: make syncCopyMove use context for go routine cancellation
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
fe96d5cf0a
fs: factor multiple directory traverse out of sync
2017-09-16 19:49:31 +01:00
Nick Craig-Wood
6df12b3f00
fs: improve retriable error detection
2017-09-16 19:48:49 +01:00
Stefan Breunig
9d3f11b493
amazonclouddrive, rcat: ensure rcat integration test passes even with AmazonCloudDrive ( fixes : #1680 )
2017-09-15 18:09:04 +02:00
Nick Craig-Wood
798502b204
fs: add more errors to be considered temporary errors
...
This makes a framework for adding temporary errors identified by
syscall number or by error string.
Fixes #1660
2017-09-14 18:01:43 +01:00
Stefan Breunig
56dedc49e3
rcat: properly report if the upload fails
2017-09-13 20:21:52 +02:00
Girish Ramakrishnan
4c45cbea18
copy: error out if dst could not be listed
2017-09-12 09:29:44 +01:00
Stefan Breunig
a91448c83a
rcat: honor --dry-run even for small files
2017-09-11 22:28:16 +02:00
Stefan Breunig
80b1f2a494
rcat: configurable small files cutoff and implement proper upload verification
2017-09-11 08:26:53 +02:00
Stefan Breunig
57817397a0
rcat: directly upload small files without streaming them
2017-09-11 08:25:34 +02:00
Nick Craig-Wood
cbea06026a
Make check obey --ignore-size - fixes #1643
2017-09-01 11:20:41 +01:00
Nick Craig-Wood
2c8d6e86cc
fs: fix gofmt
2017-08-31 16:01:19 +01:00
cbruegg
bb6300b032
Fix bwlimit toggle in conjunction with schedules ( Fixes #1607 )
2017-08-31 15:33:29 +01:00
Stefan Breunig
a122b9fa7a
yandex: implement streaming uploads (see #1614 )
2017-08-19 14:07:23 +02:00
Alex McGrath Kraak
2cc9071791
http: add --user-agent option. close #1557
2017-08-17 06:49:27 +02:00
Nick Craig-Wood
265fb8a5e2
fs: Manage empty directories - fixes #100
...
During the sync we collect a list of directories which should be empty
and attempt to rmdir them at the end of the sync. If the directories
are not empty then the rmdir will fail, logging a message but not
erroring the sync.
2017-08-09 21:07:00 +01:00
Nick Craig-Wood
8a1a900733
fstest: use Feature.CanHaveEmptyDirectories to sharpen tests
...
Now we actually test whether the directories are present or not,
filtering out empty directories in the test using the
CanHaveEmptyDirectories flag.
2017-08-09 20:55:08 +01:00
Nick Craig-Wood
20ae7d562b
fs: Add CanHaveEmptyDirectories and BucketBased feature flags to all remotes
2017-08-09 20:55:08 +01:00
Nick Craig-Wood
ec2ea37ad2
fs: Add --disable flag to disable optional features - fixes #1551
...
Eg to disable server side copy use `--disable copy`, to see a list of
what you can disable, `--disable help`.
2017-08-07 21:34:45 +01:00
Nick Craig-Wood
bfe812ea6b
dedupe: implement merging of duplicate directories - fixes #1243
2017-08-07 15:36:41 +01:00
Nick Craig-Wood
db1995e63a
Add MergeDirs optional interface and implement it for drive
2017-08-07 15:32:47 +01:00
Nick Craig-Wood
81a2ab599f
fs: add optional ID to fs.Directory and set it in the remotes which care
2017-08-07 15:31:22 +01:00
Stefan Breunig
40f24e0ea3
config: use absolute ConfigPath to ensure newly written config is on the same mount - fixes #1569
2017-08-05 12:13:25 +02:00
Nick Craig-Wood
92d2e1f8d7
azureblob: rework and complete #801
...
* Fixup bitrot (rclone and Azure library)
* Implement Copy
* Add modtime to metadata under mtime key as RFC3339Nano
* Make multipart upload work
* Make it pass the integration tests
* Fix uploading of zero length blobs
* Rename to azureblob as it seems likely we will do azurefile
* Add docs
2017-08-04 22:56:16 +01:00
Andrei Dragomir
036fd61a50
Added Azure Blob storage support #801
2017-08-04 22:54:27 +01:00
wuyu
ec5b72f8d5
Add new QingStor remote
...
Add new package qingstor to support QingStor API.
Add new unit test for its and tested through; But I commented
on some tests case because of some of the features of QingStor.
Add new docs for it.
2017-08-04 17:25:47 +01:00
Nick Craig-Wood
f682002b84
fs: Make tests create a new bucket rather than purging the old one
...
This enables QingStor to pass the tests as it has a 2 minute lockout
on deleting the old bucket then creating it again.
2017-08-04 17:09:28 +01:00
Stefan Breunig
28a18303f3
implement rcat – fixes #230 , fixes #1001
2017-08-03 21:42:35 +02:00
Nick Craig-Wood
3e3a59768e
fs/test_all: fix after fstest factorisation
2017-08-03 20:01:05 +01:00
Nick Craig-Wood
e01741b557
fs: Cleaning up directories in test is no longer needed
...
..as it is done in the finalise method.
2017-08-03 20:01:05 +01:00
Nick Craig-Wood
71fe046937
fs: Add Find method to DirTree
2017-07-26 16:38:53 +01:00
Nick Craig-Wood
d5ff7104e5
fs: Implement NewDirTree for non --fast-list
2017-07-26 16:38:44 +01:00
Nick Craig-Wood
cd4895690a
fstest: Factor test initialisation into Initialise()
2017-07-26 16:38:33 +01:00
Nick Craig-Wood
1ecf2bcbd5
fs: fix typo in --bind description
2017-07-23 23:08:33 +01:00
Nick Craig-Wood
c3d6cc91ec
Fix --bind flag changes under go1.6
...
Correcting 9f24639568
2017-07-23 22:36:32 +01:00
Nick Craig-Wood
9f24639568
Add --bind flag for choosing the local addr on outgoing connections - fixes #1087
...
Supported by all remotes except FTP.
2017-07-23 16:27:39 +01:00
Nick Craig-Wood
7b81f12dad
box: add docs
...
* reorder remotes so they are in alphabetical order by full name everywhere
* update CONTRIBUTING doc
2017-07-23 11:32:34 +01:00
Nick Craig-Wood
d279161cee
Implement box storage remote - #97
2017-07-23 11:32:34 +01:00
Nick Craig-Wood
db6009126d
Fix test failure with new stretchr/testify - fixes #1550
2017-07-23 08:59:07 +01:00
Nick Craig-Wood
c8b5ee1e54
Start v1.37-DEV development
2017-07-22 20:43:06 +01:00
Nick Craig-Wood
a73ecec11f
Version v1.37
2017-07-22 20:04:29 +01:00
Nick Craig-Wood
0cf19ef66a
Make ListDirSorted check for subdirectories and write test
2017-07-19 09:36:27 +01:00
Nick Craig-Wood
655891170f
Check in ListDirSorted that the directory entries all belong
2017-07-18 23:39:42 +01:00
Nick Craig-Wood
6f71260acf
Add --tpslimit and --tpslimit-burst to limit transactions per second for HTTP
...
This is useful if you are being rate limited or banned by your cloud
storage provider.
2017-07-16 17:25:39 +01:00
Nick Craig-Wood
62e28d0a72
Replace token bucket limiter github.com/tsenart/tb with golang.org/x/time/rate
...
In tests tsenart/tb has proved inaccurate at low rates.
2017-07-16 16:14:44 +01:00
Nick Craig-Wood
72add5ab27
sync: state whether duplicates are objects are directories
2017-07-08 15:42:18 +01:00
Nick Craig-Wood
69ff009264
Use a stable sort for sorting directory entries
...
This is useful if there are duplicates. Assuming the remote delivers
the entries in a consistent order, this will give the best user
experience in syncing as it will consistently use the first entry for
the sync comparison.
2017-07-06 14:07:26 +01:00
Nick Craig-Wood
27b157580e
Move make_test_files.go into bin
2017-07-06 11:54:57 +01:00
Nick Craig-Wood
f0512d1a52
Fix missing fs.Dir -> fs.Directory
2017-07-06 11:31:36 +01:00
Nick Craig-Wood
575e779b55
Warn about duplicate files when syncing - fixes #1506
...
Error about unsorted directories and test thoroughly
2017-06-30 21:24:13 +01:00
Nick Craig-Wood
e7d04fc103
Create fs.Directory interface and use it everywhere
2017-06-30 14:26:59 +01:00
Nick Craig-Wood
e2d7d413ef
fs: rename BasicInfo to DirEntry
2017-06-30 14:26:58 +01:00
Nick Craig-Wood
e7e9aa0dfa
fs: Remove unused ListFser interface
2017-06-30 14:26:58 +01:00
Nick Craig-Wood
f88300a153
Don't Mkdir at the start of sync - fixes #1131
...
This is possible now that the bucket based remotes will create the
buckets on demand (9c1e703777
).
2017-06-29 12:31:53 +01:00
Nick Craig-Wood
e54087ece1
Fix config tests to save configData which fixes subsequent tests
2017-06-29 12:31:53 +01:00
Nick Craig-Wood
54561fd2bc
s3: work around eventual consistency in bucket creation
...
Deleting a bucket then testing its existence can give the wrong
result. Work around by keeping a flag as to whether we have deleted
the bucket.
2017-06-29 12:31:52 +01:00
Nick Craig-Wood
f3c7e1a9dd
Debug directory creation and removal - fixes #1192
2017-06-27 22:19:35 +01:00
Nick Craig-Wood
aa20486485
Add --stats-log-level so can see --stats without -v - fixes #1180
...
The most common use for this flag is likely to be showing the stats
without using -v by using `--stats-log-level NOTICE`.
2017-06-26 22:50:37 +01:00
Stefan Breunig
52b042971a
keep file permissions and try to keep user/group on supported systems ( fixes #1467 )
2017-06-25 09:05:24 +02:00
Stefan Breunig
2d2778eabf
don't delete remote if name does not change while renaming ( fixes #1495 )
2017-06-25 08:55:54 +02:00
Vasiliy Tolstov
6b89e6c381
add new http remote filesystem
...
Signed-off-by: Vasiliy Tolstov <v.tolstov@selfip.ru>
2017-06-20 21:27:14 +01:00
Nick Craig-Wood
b76cd4abd2
Fix Range header option
2017-06-20 21:27:14 +01:00
Nick Craig-Wood
4fd9570332
fs: Use an in place filter in ListDirSorted
2017-06-20 21:27:14 +01:00
Nick Craig-Wood
5250fcdf08
core: fix data race in walk
...
This was detected by the race detector when the client of Walk() sorted entries.
2017-06-15 16:46:32 +01:00
Nick Craig-Wood
64662bef8d
Deprecate --old-sync-method it is replaced with --fast-list
...
Remove old sync method code.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
0b8d9084fc
test_all: print command line so it can be cut and pasted into bash
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
8a6a8b9623
Change List interface and add ListR optional interface
...
This simplifies the implementation of remotes. The only required
interface is now `List` which is a simple one level directory list.
Optionally remotes may implement `ListR` if they have an efficient way
of doing a recursive list.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
6fc88ff32e
Use --fast-list flag for sync/copy/move - fixes #1277
...
Redo test framework to take a -fast-list flag and test remotes with that flag.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
50928a5027
Implement --fast-list flag.
...
This is supported remotes which can do a recursive listing. It will
use more memory.
This is related to #1277 but doesn't fix that issue yet.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
53c3e5f0ab
Add placeholder support for ListR interface.
...
The ListR interface will be implemented by remotes that can do a
recursive directory listing more efficiently than just recursing
through the directories. These include the bucket based remotes.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
0edb025257
Fixup tests with dirs vs bucket based fs
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
7e20e16cff
core: Implement Walk directory listing and use in place of Lister
...
This is in preparation for removing the Lister code and replacing the
fundamental operation in the Fs with listing a single directory.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
68333d34a1
dropbox: make setting mod time on existing files work properly
...
This is a fix left over from the v2 conversion. Dropbox ignores the
client modification on an incoming file if it was identical to the
existing file. This change deletes the existing file first before
re-uploading the new one.
2017-06-13 13:58:39 +01:00
Nick Craig-Wood
740b3f6ae2
Fix problems found with ineffassign
2017-06-13 11:52:36 +01:00
Nick Craig-Wood
9a11d3efd9
Revert "Start Cat tests from 2 as onedrive doesn't support ranging from 1"
...
Now that https://github.com/OneDrive/onedrive-api-docs/issues/543 is
fixed, this can be reverted.
This reverts commit 320c53eab0
.
2017-06-10 13:48:00 +01:00
Nick Craig-Wood
ee14efd3c2
config: fix menu selection when no remotes
2017-06-10 11:39:40 +01:00
Nick Craig-Wood
52e1bfae2a
oauth: Allow auth_url and token_url to be set in the config file
...
If set in the config file, these override the ones configured into the
remote. This enables alternative oauth servers to be used for all
oauth remotes. This can only be altered by editing the config file
for the moment.
2017-06-08 20:35:32 +01:00
Nick Craig-Wood
b49821956a
Fix copyto/moveto test error (see #1261 )
2017-06-07 14:08:46 +01:00
Nick Craig-Wood
a61ba1e7c4
moveto, copyto: report transfers and checks as per move and copy
2017-06-07 13:02:21 +01:00
Nick Craig-Wood
d30cc1e119
Factor RemoteSplit into fs
2017-06-07 12:27:33 +01:00
Nick Craig-Wood
74a3dfc4e1
Fix TestHashSums
2017-06-06 23:21:47 +01:00
Nick Craig-Wood
b047402294
config: Fix save of temp file under Windows - fixes #1458
2017-06-01 16:38:19 +01:00
Stefan
9cede6b372
fully write new config file before moving to target location ( fixes #1287 )
...
* fully write new config file before moving to target location (fixes #1287 )
* do not fail if there is no previous config; print temporary config path on failure
2017-06-01 08:57:10 +01:00
Stefan Breunig
decd960867
make moveto/copyto no-ops when source and destination are the same ( fixes #1261 )
2017-05-30 23:01:19 +01:00
Nick Craig-Wood
20da3e6352
Add options to Put, PutUnchecked and Update, add HashOption and speed up local
...
* Add options to Put, PutUnchecked and Update for all Fses
* Use these to create HashOption
* Implement this in local
* Pass the option in fs.Copy
This has the effect that we only calculate hashes we need to in the
local Fs which speeds up transfers significantly.
2017-05-29 12:04:52 +01:00
Nick Craig-Wood
6381959850
dropbox: support Dropbox content hashing scheme - fixes #1302
...
* add support to hashing module
* add dbhashsum to list the hashes
* add support to dropbox module
This means objects up and downloaded to/from Dropbox will have their
hashes checked.
Note after this change local objects are calculating MD5, SHA1 and
DBHASH which is excessive and needs to be fixed.
2017-05-29 12:04:44 +01:00
Stefan Breunig
a2e3af0523
poll for Google Drive changes when mounted
2017-05-28 17:54:52 +01:00
Nick Craig-Wood
88ef475629
config: allow keys to be deleted from the config file
2017-05-25 10:15:22 +01:00
Nick Craig-Wood
ae9f8304fa
Attempt to make async buffer test more reliable
2017-05-24 16:24:06 +01:00
Nick Craig-Wood
e0ba1a2cd2
ftp: fix bitrot
2017-05-18 20:49:36 +01:00
Antonio Messina
c72fca2711
Add ftp backend - fixes #540
2017-05-18 20:49:36 +01:00
Nick Craig-Wood
50e79bc087
fs: Implement fs.Trace for tracing entry and exit of functions
2017-05-15 16:56:45 +01:00
Nick Craig-Wood
17633f5460
Require go1.6 for building rclone
...
This is required because google.golang.org/grpc needs it.
2017-05-11 17:07:49 +01:00
Yoni Jah
cdeeff988e
Added RepetableReader to fs. used in OneDrive with io.LimitedReader to display accurate speed
2017-05-02 22:31:05 +01:00
Hraban Luyat
66e8c1600e
Print password prompts to stderr
...
This makes rclone with encrypted config better suited for use in
pipelines. E.g.:
$ rclone lsl mydrive:Some/Dir | sort -k 4
If the password prompt ("Enter configuration password") is printed to
stdout, it will be swallowed by sort. By printing it to stderr, you
still see the prompt, without sacrificing compatibility with the unix
pipeline.
2017-04-22 09:38:39 +01:00
Nick Craig-Wood
bc25190fc7
Fix misleading log message with --dry-run - fixes #1309
2017-04-10 16:07:22 +01:00
marvwatson
4810aa65a4
Update references from HTTP to HTTPS where possible
2017-03-29 05:38:34 -07:00
Nick Craig-Wood
503cd84919
Start v1.36-DEV development
2017-03-18 11:30:59 +00:00
Nick Craig-Wood
118e26f8e2
Version v1.36
2017-03-18 11:16:43 +00:00
Nick Craig-Wood
9b07d32c02
onedrive, drive, amazonclouddrive: make sure we find the root
...
This fixes copyto copying things to the wrong place - fixes #1231
2017-03-16 09:42:49 +00:00
Nick Craig-Wood
99f7fe736a
onedrive: implement Move #197
2017-03-15 14:01:08 +00:00
Nick Craig-Wood
320c53eab0
Start Cat tests from 2 as onedrive doesn't support ranging from 1
...
This has been reported here: https://github.com/OneDrive/onedrive-api-docs/issues/543
2017-03-12 14:24:33 +00:00
Nick Craig-Wood
4d5b73df85
Fix TestListDirSorted eventual consistency listing problems
2017-03-12 14:00:22 +00:00
Nick Craig-Wood
b2a4ea9304
fs/buffer: Fix panic on concurrent Read/Close - fixes #1213
2017-03-06 19:22:17 +00:00
Nick Craig-Wood
10e532bce9
Fix --files-from with an empty file copying everything - fixes #1196
2017-03-04 10:12:54 +00:00
Nick Craig-Wood
4ab7e05e02
Fix MimeType propagation
...
In fs.Copy, don't wrap objects if possible, and if not, then add a
MimeType method into the wrapped object.
2017-03-04 10:10:55 +00:00
Nick Craig-Wood
79e3c67bbd
local, yandex, dropbox: fix NewObject suceeding on a directory #1079
...
Add tests to make it consistent across all remotes
2017-02-25 11:09:57 +00:00
Nick Craig-Wood
e2f0feef3c
Add debugging to print hash values on failed hash comparison
2017-02-23 11:23:19 +00:00
Nick Craig-Wood
e59dc81658
Stop --track-renames deleting case folded source files - fixes #1094
...
What was happening is that when Move was implemented as Copy + Delete,
MoveFile was seeing the files didn't need transferring (because they
were identical) then deleted the source.
The fix uses Move instead and patches onedrive to disallow a case
folded identical copy (which errors with 500 error)
2017-02-22 19:28:22 +00:00
Nick Craig-Wood
6b0f2ef4bd
Fix --delete-before deleting files on copy - fixes #1166
2017-02-22 13:17:38 +00:00
Hisham Zarka
b3a526814e
fix --ignore-checksum
2017-02-18 13:13:53 +04:00
Nick Craig-Wood
033d1eb7af
Refactor Account interface
2017-02-17 09:15:24 +00:00
Nick Craig-Wood
ac62ef430d
Prevent double closes on async buffer
2017-02-17 08:55:24 +00:00
Nick Craig-Wood
928be0f1fd
mount: fix seek with buffering to use correct interface
...
Stop pre-cache before seeking which stops lots of excess data transfer
2017-02-17 08:55:24 +00:00
Nick Craig-Wood
6f75290678
Make async buffering start slowly to improve seek performance
2017-02-17 08:26:14 +00:00
Nick Craig-Wood
f3c5745468
Add srcRemote and dstRemote parameters to DirMove #954
2017-02-16 17:42:37 +00:00
Nick Craig-Wood
dac4bb22d3
mount: Make include and exclude filters apply to mount - fixes #1060
2017-02-15 23:28:53 +00:00
Nick Craig-Wood
b52c80e85c
sync: don't update mod times if --dry-run set - fixes #1100
2017-02-15 23:09:44 +00:00
Nick Craig-Wood
f15c6b68b6
Re-add the async buffer on seek - fixes #1137
2017-02-15 22:54:21 +00:00
Nick Craig-Wood
3f778d70f7
Add sync.Pool to async reader
2017-02-15 22:37:58 +00:00
Nick Craig-Wood
7fa687b3e1
fs: Async buffer: use ReadFill to fill the chunks and increase to 1MB
2017-02-14 22:36:37 +00:00
Nick Craig-Wood
493da54113
Add --buffer-size parameter to control buffer size for copy
2017-02-14 22:36:37 +00:00
Nick Craig-Wood
541929258b
check: Add --download flag to check all the data, not just hashes
2017-02-13 10:48:26 +00:00
Nick Craig-Wood
7047c67a5e
sync: Fix log message containing <nil>
2017-02-13 09:23:21 +00:00
Nick Craig-Wood
01c747e7db
Add cryptcheck command to check integrity of crypt remotes #1102
2017-02-12 16:30:18 +00:00
Nick Craig-Wood
666dae4229
Add --syslog flag to optionally log to syslog on capable platforms
2017-02-11 20:27:46 +00:00
Nick Craig-Wood
ac1c041377
Redo log level flags
...
* -vv or --log-level DEBUG
* -v or --log-level INFO
* --log-level NOTICE (default)
* -q --log-level ERROR
Replace Config.Verbose and Config.Quiet with Config.LogLevel
Fixes #739 Fixes #1108 Fixes #1000
2017-02-11 20:22:42 +00:00
Nick Craig-Wood
0366ea39c5
Reassign some logging levels
2017-02-11 17:56:05 +00:00
Nick Craig-Wood
80f53176d9
Rename log functions and factor into own file
2017-02-11 17:54:50 +00:00
Nick Craig-Wood
50e190ff54
cat: don't allocate buffers if not needed to reduce memory usage
2017-02-09 11:46:53 +00:00
Nick Craig-Wood
dd20a297d6
cat: Fix go routine leak
2017-02-09 11:25:36 +00:00
Nick Craig-Wood
c0ad29c06c
Clarify logging and docs for --no-traverse incompatibilities - fixes #1059
2017-02-08 22:35:12 +00:00
Nick Craig-Wood
d091d4a8bb
rclone cat: add --head, --tail, --offset, --count and --discard
...
Fixes #819
2017-02-08 08:09:41 +00:00
Nick Craig-Wood
48cdedc97b
Re-implement sync routine to work a directory at a time
...
Multiple directories (up to --checkers worth) are scanned at once.
This uses much less memory than the previous scheme - only the amount
of memory needed to hold an entire directory listing of objects.
For directory based remotes the speed is unchanged.
For bucket based remotes, instead of doing one API call to list the
whole bucket, it does multiple calls, one for each pseudo directory.
However these are done in parallel so in practice this seems to speed
up directory listings.
This replaces the existing sync method as it performs faster and uses
less memory.
The old sync method is available with the temporary --old-sync-method
flag.
Fixes #517
Fixes #439
Fixes #236
Fixes #1067
2017-02-04 10:30:25 +00:00