Nick Craig-Wood
b1f131964e
Rename NewFsObject to NewObject
2016-06-28 08:51:57 +01:00
Nick Craig-Wood
1a87b69376
Get rid of LimitedFs - FIXME needs docs on copying single files
...
If remote:path points to a file make NewFs return a sentinel error
fs.ErrorIsFile and an Fs which points to the parent.
Use this to remove the LimitedFs and just add this file to the
--files-from list.
This means that server side operations can be used also.
Fixes #518
Fixes #545
2016-06-28 08:51:43 +01:00
Nick Craig-Wood
a67c7461ee
s3: skip SetModTime for objects > 5GB - fixes #534
2016-06-19 17:26:44 +01:00
Justin R. Wilson
6e35a3b3ce
Add AES256 server-side encryption for s3 - Fixes #491
...
Add a configuration key and support for AES256 server-side encryption.
2016-06-18 14:28:38 +01:00
Nick Craig-Wood
2cfb3834f2
Log errors with %v
2016-06-18 09:36:47 +01:00
Nick Craig-Wood
df1092ef33
Change Fs.Put so that it must cope with existing files
...
This should fix duplicate files on drive and 409 errors on
amazonclouddrive however it will slow down the upload slightly as
another roundtrip will be needed.
None of the other Fses needed adjusting.
Fixes #483
2016-06-13 19:29:10 +01:00
Nick Craig-Wood
4c5b2833b3
Convert to using github.com/pkg/errors everywhere
2016-06-13 17:43:03 +01:00
Nick Craig-Wood
c2d0e86431
Add more tests for List() and fix resulting problems
2016-05-07 14:50:35 +01:00
Nick Craig-Wood
68ec6a9f5b
Add a directory parameter to Fs.List()
2016-05-06 16:52:34 +01:00
Nick Craig-Wood
753b0717be
Refactor the List and ListDir interface
...
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.
Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.
This commit contains some code originally by Klaus Post.
Fixes #316
2016-05-06 16:52:34 +01:00
Nick Craig-Wood
414b35ea56
Change the interface of SetModTime to return an error - #348
2016-03-22 15:56:44 +00:00
Nick Craig-Wood
a1323eb204
s3: Fix uploading files bigger than 50GB - fixes #386
2016-03-10 16:48:55 +00:00
Nick Craig-Wood
88cca8a6eb
Simplify literals (after running gofmt -s over the code)
2016-02-29 16:57:23 +00:00
Jakub Gedeon
694d390710
s3: Check if directory exists during Mkdir
...
If you dont have privs to create a bucket in S3 but it exists, don't
fail with an auth error, but detect that the mkdir was not needed and
return successfully.
2016-02-27 13:24:46 +00:00
Nick Craig-Wood
e08e35984c
Add help to remote chooser in rclone config - fixes #43
2016-02-21 13:40:16 +00:00
klauspost
ef06371c93
Create separate interface for object information.
...
Take out read-only information about a Fs in a separate struct to limit access.
See discussion at #282 .
2016-02-19 13:31:09 +00:00
Nick Craig-Wood
ce4c1d4f35
s3: Fix empty checks in auth
2016-02-09 17:19:33 +00:00
Brian Stengaard
ce05ef7110
Add IAM role and Env credentials
...
This will make the s3 provider authentaction logic
- Configured credentials if both key and secret available
- Anonymous if key and secret missing and env_auth not set
- if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool )
- AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables
- IAM role credentials as fallback
2016-02-09 16:32:36 +00:00
Nick Craig-Wood
cae19df058
s3: URL escape CopySource
...
This fixes metadata update and copy for files with `+` in
Fixes #315
2016-01-27 17:39:33 +00:00
Nick Craig-Wood
ccba859812
Test all available hashes for each remote
2016-01-23 09:10:36 +00:00
klauspost
78c3a5ccfa
Add support for multiple hash types.
...
Add support for multiple hash types with negotiation of common hash types for comparison.
Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
Joseph Spurrier
fe0289f2f5
s3: Fix corrupting Content-Type on mod time update
...
This fixes an issue where updating the modification time resets the
content-type to the S3 default of binary/octet-stream which breaks
static websites that expect an html file to have a content-type of
text/html.
2016-01-02 11:47:52 +00:00
Nick Craig-Wood
a795d93bc3
swift, s3, googlecloudstorage: Don't delete the container/bucket if fs wasn't at root - fixes #172
2015-11-07 15:32:40 +00:00
Nick Craig-Wood
b257de4aba
Be more constistent with naming in remotes
...
* External objects are called Fs and Object
* Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
klauspost
345c98ed62
Update to AWS SDK 0.10.0
...
Tested with S3 and Dreamhost
Here is a link to the release notes:
http://aws.amazon.com/releasenotes/5476699172355228
2015-11-02 19:52:11 +00:00
Nick Craig-Wood
9f1daabb2c
s3: allow anonymous access to public repositories - fixes #154
...
When setting up the remote, leave both the access key and secret key
blank.
2015-10-03 16:04:21 +01:00
Nick Craig-Wood
e9c915e6fe
Fix golint warnings
2015-09-22 18:47:16 +01:00
Nick Craig-Wood
2ed158aba3
Fixes from go vet and errcheck
2015-09-22 07:31:12 +01:00
Nick Craig-Wood
e2717a031e
Implement Mover and DirMover interfaces fixes #115
...
* unit tests
* local
* drive
* dropbox
2015-09-01 21:49:13 +01:00
Nick Craig-Wood
92745aa950
Add Root() to Fs interface
2015-09-01 21:49:13 +01:00
Nick Craig-Wood
d298b578ab
s3: Fix after upstream API changes in aws-sdk-go/aws - fixes #114
2015-08-28 08:47:41 +01:00
Nick Craig-Wood
a96b522958
Implement server side copies if possible - fixes #99
...
Add optional fs.Copier interface
Implemented for
* swift
* s3
* drive
* dropbox
* google cloud storage
2015-08-23 21:18:38 +01:00
Nick Craig-Wood
fedf81c2b7
Add Name() to Fs interface to return name as passed to NewFs
2015-08-23 13:36:38 +01:00
Nick Craig-Wood
337110b7a0
s3: remove verbose debug about invalid md5sums for multipart upload
2015-08-16 18:14:22 +01:00
Nick Craig-Wood
6a8de87116
s3: make v2 signatures work for ceph
2015-08-16 15:51:38 +01:00
Nick Craig-Wood
0da6f24221
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2015-08-16 15:51:04 +01:00
Leonid Shalupov
3fcff32524
do not print stats in quiet mode - fixes #70
...
...unless had some errors or stats interval requested.
Add fs.ErrorLog to differentiate between Logs which should be
suppressed and errors which shouldn't.
2015-08-10 11:17:54 +01:00
Nick Craig-Wood
1d254a3674
Implement --timeout and --conntimeout - fixes #54
...
NB dropbox still to do
2015-05-10 11:29:55 +01:00
Nick Craig-Wood
f88d171afd
s3: ignore etags from multipart uploads which aren't md5sums - fixes #56
2015-05-10 11:29:55 +01:00
Nick Craig-Wood
46d39ebaf7
Factor Mime Type guessing into fs.MimeType()
2015-03-02 09:21:15 +00:00
Nick Craig-Wood
85f8552c4d
Tidy logging
2015-02-27 15:22:05 +00:00
Nick Craig-Wood
0faed16899
s3: list an unlimited number of items - fixes #22
2015-02-10 17:58:29 +00:00
Nick Craig-Wood
d3c16608e4
Test Listing the Root of each Fs
2014-07-31 23:20:39 +01:00
Nick Craig-Wood
3e27ff1b95
Add Root List test and fs.Limited tests for single files
2014-07-31 21:35:29 +01:00
Nick Craig-Wood
c389616657
all: make private functions / variables / constant which shouldn't be public
2014-07-29 17:50:07 +01:00
Nick Craig-Wood
0b51d6221a
s3: make reading metadata more reliable to work around eventual consistency problems
2014-07-29 17:32:06 +01:00
Nick Craig-Wood
457ef2c190
Automatically generate the tests files for each Fs
2014-07-29 17:18:22 +01:00
Nick Craig-Wood
125fc8f1f0
s3: strip trailing / from ListDir()
2014-07-24 23:13:33 +01:00
Nick Craig-Wood
a5b0d88608
Make tests for each Fs
...
Factor tests out of rclonetest
2014-07-24 23:13:32 +01:00
Nick Craig-Wood
282cba20a0
swift, s3, dropbox: fix metadata read on Update()
...
This was causing changed files to be marked as corrupted on upload
2014-07-20 11:23:05 +01:00