Nick Craig-Wood
265fb8a5e2
fs: Manage empty directories - fixes #100
...
During the sync we collect a list of directories which should be empty
and attempt to rmdir them at the end of the sync. If the directories
are not empty then the rmdir will fail, logging a message but not
erroring the sync.
2017-08-09 21:07:00 +01:00
Nick Craig-Wood
8a1a900733
fstest: use Feature.CanHaveEmptyDirectories to sharpen tests
...
Now we actually test whether the directories are present or not,
filtering out empty directories in the test using the
CanHaveEmptyDirectories flag.
2017-08-09 20:55:08 +01:00
Nick Craig-Wood
20ae7d562b
fs: Add CanHaveEmptyDirectories and BucketBased feature flags to all remotes
2017-08-09 20:55:08 +01:00
Nick Craig-Wood
ec2ea37ad2
fs: Add --disable flag to disable optional features - fixes #1551
...
Eg to disable server side copy use `--disable copy`, to see a list of
what you can disable, `--disable help`.
2017-08-07 21:34:45 +01:00
Nick Craig-Wood
bfe812ea6b
dedupe: implement merging of duplicate directories - fixes #1243
2017-08-07 15:36:41 +01:00
Nick Craig-Wood
db1995e63a
Add MergeDirs optional interface and implement it for drive
2017-08-07 15:32:47 +01:00
Nick Craig-Wood
81a2ab599f
fs: add optional ID to fs.Directory and set it in the remotes which care
2017-08-07 15:31:22 +01:00
Stefan Breunig
40f24e0ea3
config: use absolute ConfigPath to ensure newly written config is on the same mount - fixes #1569
2017-08-05 12:13:25 +02:00
Nick Craig-Wood
92d2e1f8d7
azureblob: rework and complete #801
...
* Fixup bitrot (rclone and Azure library)
* Implement Copy
* Add modtime to metadata under mtime key as RFC3339Nano
* Make multipart upload work
* Make it pass the integration tests
* Fix uploading of zero length blobs
* Rename to azureblob as it seems likely we will do azurefile
* Add docs
2017-08-04 22:56:16 +01:00
Andrei Dragomir
036fd61a50
Added Azure Blob storage support #801
2017-08-04 22:54:27 +01:00
wuyu
ec5b72f8d5
Add new QingStor remote
...
Add new package qingstor to support QingStor API.
Add new unit test for its and tested through; But I commented
on some tests case because of some of the features of QingStor.
Add new docs for it.
2017-08-04 17:25:47 +01:00
Nick Craig-Wood
f682002b84
fs: Make tests create a new bucket rather than purging the old one
...
This enables QingStor to pass the tests as it has a 2 minute lockout
on deleting the old bucket then creating it again.
2017-08-04 17:09:28 +01:00
Stefan Breunig
28a18303f3
implement rcat – fixes #230 , fixes #1001
2017-08-03 21:42:35 +02:00
Nick Craig-Wood
3e3a59768e
fs/test_all: fix after fstest factorisation
2017-08-03 20:01:05 +01:00
Nick Craig-Wood
e01741b557
fs: Cleaning up directories in test is no longer needed
...
..as it is done in the finalise method.
2017-08-03 20:01:05 +01:00
Nick Craig-Wood
71fe046937
fs: Add Find method to DirTree
2017-07-26 16:38:53 +01:00
Nick Craig-Wood
d5ff7104e5
fs: Implement NewDirTree for non --fast-list
2017-07-26 16:38:44 +01:00
Nick Craig-Wood
cd4895690a
fstest: Factor test initialisation into Initialise()
2017-07-26 16:38:33 +01:00
Nick Craig-Wood
1ecf2bcbd5
fs: fix typo in --bind description
2017-07-23 23:08:33 +01:00
Nick Craig-Wood
c3d6cc91ec
Fix --bind flag changes under go1.6
...
Correcting 9f24639568
2017-07-23 22:36:32 +01:00
Nick Craig-Wood
9f24639568
Add --bind flag for choosing the local addr on outgoing connections - fixes #1087
...
Supported by all remotes except FTP.
2017-07-23 16:27:39 +01:00
Nick Craig-Wood
7b81f12dad
box: add docs
...
* reorder remotes so they are in alphabetical order by full name everywhere
* update CONTRIBUTING doc
2017-07-23 11:32:34 +01:00
Nick Craig-Wood
d279161cee
Implement box storage remote - #97
2017-07-23 11:32:34 +01:00
Nick Craig-Wood
db6009126d
Fix test failure with new stretchr/testify - fixes #1550
2017-07-23 08:59:07 +01:00
Nick Craig-Wood
c8b5ee1e54
Start v1.37-DEV development
2017-07-22 20:43:06 +01:00
Nick Craig-Wood
a73ecec11f
Version v1.37
2017-07-22 20:04:29 +01:00
Nick Craig-Wood
0cf19ef66a
Make ListDirSorted check for subdirectories and write test
2017-07-19 09:36:27 +01:00
Nick Craig-Wood
655891170f
Check in ListDirSorted that the directory entries all belong
2017-07-18 23:39:42 +01:00
Nick Craig-Wood
6f71260acf
Add --tpslimit and --tpslimit-burst to limit transactions per second for HTTP
...
This is useful if you are being rate limited or banned by your cloud
storage provider.
2017-07-16 17:25:39 +01:00
Nick Craig-Wood
62e28d0a72
Replace token bucket limiter github.com/tsenart/tb with golang.org/x/time/rate
...
In tests tsenart/tb has proved inaccurate at low rates.
2017-07-16 16:14:44 +01:00
Nick Craig-Wood
72add5ab27
sync: state whether duplicates are objects are directories
2017-07-08 15:42:18 +01:00
Nick Craig-Wood
69ff009264
Use a stable sort for sorting directory entries
...
This is useful if there are duplicates. Assuming the remote delivers
the entries in a consistent order, this will give the best user
experience in syncing as it will consistently use the first entry for
the sync comparison.
2017-07-06 14:07:26 +01:00
Nick Craig-Wood
27b157580e
Move make_test_files.go into bin
2017-07-06 11:54:57 +01:00
Nick Craig-Wood
f0512d1a52
Fix missing fs.Dir -> fs.Directory
2017-07-06 11:31:36 +01:00
Nick Craig-Wood
575e779b55
Warn about duplicate files when syncing - fixes #1506
...
Error about unsorted directories and test thoroughly
2017-06-30 21:24:13 +01:00
Nick Craig-Wood
e7d04fc103
Create fs.Directory interface and use it everywhere
2017-06-30 14:26:59 +01:00
Nick Craig-Wood
e2d7d413ef
fs: rename BasicInfo to DirEntry
2017-06-30 14:26:58 +01:00
Nick Craig-Wood
e7e9aa0dfa
fs: Remove unused ListFser interface
2017-06-30 14:26:58 +01:00
Nick Craig-Wood
f88300a153
Don't Mkdir at the start of sync - fixes #1131
...
This is possible now that the bucket based remotes will create the
buckets on demand (9c1e703777
2017-06-29 12:31:53 +01:00
Nick Craig-Wood
e54087ece1
Fix config tests to save configData which fixes subsequent tests
2017-06-29 12:31:53 +01:00
Nick Craig-Wood
54561fd2bc
s3: work around eventual consistency in bucket creation
...
Deleting a bucket then testing its existence can give the wrong
result. Work around by keeping a flag as to whether we have deleted
the bucket.
2017-06-29 12:31:52 +01:00
Nick Craig-Wood
f3c7e1a9dd
Debug directory creation and removal - fixes #1192
2017-06-27 22:19:35 +01:00
Nick Craig-Wood
aa20486485
Add --stats-log-level so can see --stats without -v - fixes #1180
...
The most common use for this flag is likely to be showing the stats
without using -v by using `--stats-log-level NOTICE`.
2017-06-26 22:50:37 +01:00
Stefan Breunig
52b042971a
keep file permissions and try to keep user/group on supported systems ( fixes #1467 )
2017-06-25 09:05:24 +02:00
Stefan Breunig
2d2778eabf
don't delete remote if name does not change while renaming ( fixes #1495 )
2017-06-25 08:55:54 +02:00
Vasiliy Tolstov
6b89e6c381
add new http remote filesystem
...
Signed-off-by: Vasiliy Tolstov <v.tolstov@selfip.ru>
2017-06-20 21:27:14 +01:00
Nick Craig-Wood
b76cd4abd2
Fix Range header option
2017-06-20 21:27:14 +01:00
Nick Craig-Wood
4fd9570332
fs: Use an in place filter in ListDirSorted
2017-06-20 21:27:14 +01:00
Nick Craig-Wood
5250fcdf08
core: fix data race in walk
...
This was detected by the race detector when the client of Walk() sorted entries.
2017-06-15 16:46:32 +01:00
Nick Craig-Wood
64662bef8d
Deprecate --old-sync-method it is replaced with --fast-list
...
Remove old sync method code.
2017-06-14 16:49:40 +01:00
