extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-11-23 00:43:49 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,907 Commits 177 Branches 114 Tags 315 MiB
7ee22fcdf9
Commit Graph

1973 Commits

Author SHA1 Message Date
Nick Craig-Wood
57a5b72d60 onedrive: implement quickXorHash algorithm #2262 2018-04-20 21:02:37 +01:00
Chris Redekop
a35e62e15c s3: Add an option to disable checksum uploading - fixes #2213 2018-04-20 20:51:12 +01:00
Michael G. Noll
d1ca8b8959 sftp: update docs to match code, fix typos and clarify disable_hashcheck prompt 2018-04-20 20:49:49 +01:00
Nick Craig-Wood
a0c65deca8 box: Parse file/directory size as a floating point number 
Very large directories can have their sizes returned as floating point
numbers, eg `1.0034576985781e+14` from the box API.

Before this change this would fail to parse as an int64.

This change parses the size as a float64 instead which will be
perfectly accurate for sizes up to 2**56 which is about 9 PB.

It is unknown whether box themselves use a float64 as an intermediate
representation in the API or not - it seems likely.

Fixes #2261
 2018-04-19 21:04:52 +01:00
Nick Craig-Wood
1f255a8567 Add a mega.nz remote #163 
Not supported yet:
  * Hash
  * ModTime
  * Server Side Copy

Otherwise fully functional and passing all the tests.
 2018-04-18 21:09:54 +01:00
Nick Craig-Wood
1ac6dacf0f about: complete other providers and re-work internals 
* Implement about for:
    * local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
  * Implement `--json` and `---full` flag for `rclone about`
  * change About interface to return a Usage structure
  * Remove operations.About as it is too thin an interface
  * Implement Integration test

Relates to #1138 and #1564
 2018-04-17 21:53:27 +01:00
a-roussos
94e277d759 about: add new command 'about' to get quota info from a remote 
Implemented for drive only.

Relates to #1138 and #1564.
 2018-04-17 21:50:14 +01:00
Nick Craig-Wood
b83814082b backend/http: if HEAD didn't return Content-Length use -1 as size 
This means that the files will be treated as an unknown length and
will download properly.

Fixes #2247
 2018-04-16 19:40:02 +01:00
Nick Craig-Wood
3d5106e52b drive: fix DirMove leaving a hardlinked directory behind #2245 
This bug was introduced by the v3 API conversion in 07f20dd1fd.

The problem was that dircache.FindPath doesn't work for the root directory.

This adds an internal error for dircache.FindPath being called with
the root directory.  This makes a failing test, which the fix to the
drive backend fixes.

This also improves the DirCache integration test.
 2018-04-15 10:12:21 +01:00
Nick Craig-Wood
dc247d21ff s3: add in config for all the supported S3 providers #2140 
These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other.

This configures endpoints where known and makes sure config doesn't
appear where it isn't valid where possible.
 2018-04-13 16:33:26 +01:00
Giri Badanahatti
acd5d4377e config,s3: hierarchical configuration support #2140 
This introduces a method of making provider specific configuration
within a remote.  This is useful particularly in s3.

This commit does the basic configuration in S3 for IBM COS.
 2018-04-13 16:05:35 +01:00
Craig Rachel
0e6faa2313 s3: add One Zone Infrequent Access storage class - fixes #2240 2018-04-13 13:36:25 +01:00
Peter Baumgartner
1db68571fd s3,swift: Add --use-server-modtime 
`--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.
 2018-04-13 13:32:17 +01:00
Nick Craig-Wood
ef412c1985 drive: fix misplaced log in dedupe MergeDirs 2018-04-13 13:23:55 +01:00
Nick Craig-Wood
e82452ce9a drive: check Open calls for google error messages 
This should also enable Open calls to retry properly
 2018-04-11 20:55:58 +01:00
Nick Craig-Wood
d5b2ec32f1 local: add --local-no-check-updated to disable update checks #2206 
This disables the `can't copy - source file is being updated` checks.
 2018-04-09 15:27:58 +01:00
Nick Craig-Wood
05e32cfcf9 dropbox: Fix crypt+obfuscate on dropbox - fixes #2191 
Before this change we lowercased the dropbox root directory.  This was
likely a leftover from when we used to build a dictionary to translate
the cases of dropbox files.  Now with the v2 API we can rely on
dropbox to do that for us, so we no longer need to lowercase the root.

This fixes issues using crypt with name obfuscation on dropbox.
 2018-04-09 11:53:41 +01:00
Nick Craig-Wood
0fa700b3cf Make integration tests use go1.7+ nested tests #2154 
* Removed generated code and code generator
  * Updated docs on how to write integration tests
  * Tidied up the actual integration tests
 2018-04-08 15:04:14 +01:00
Nick Craig-Wood
42f0963bf9 local: retry remove on Windows sharing violation error #2202 
Before this change asynchronous closes in cmount could cause sharing
violations under Windows on Remove which manifest themselves
frequently as test failures.

This change lets the Remove be retried on a sharing violation under
Windows.
 2018-04-07 17:36:26 +01:00
Nick Craig-Wood
be54fd8f70 Remove builds conditional on go1.7 since that is now guaranteed #2154 
Old fallback code was deleted and the go1.7 style code inlined where
appropriate.
 2018-04-07 11:42:55 +01:00
Nick Craig-Wood
e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood
80588a5a6b Replace "golang.org/x/net/context" with "context" for go1.7+ #2154 2018-04-07 11:42:08 +01:00
Nick Craig-Wood
c749cf8d99 dropbox: fix repeatedly uploading the same files - fixes #2218 
In #2134 and dfd0f4c5a4 some testing
changes got committed by accident which caused this regression.

This patch reverts it to how it was before.
 2018-04-06 15:34:56 +01:00
Nick Craig-Wood
0cb5c4aa73 gcs: detect bucket presence by listing it - fixes #2193 
Doing it like this enables the use of a service account that only has
the "Storage Object Admin" role.
 2018-04-06 12:45:15 +01:00
Nick Craig-Wood
92c5aa3786 s3: add --s3-chunk-size option - fixes #2203 2018-04-05 15:40:08 +01:00
Nick Craig-Wood
fbe1c7f1ea dropbox: remove unused code 2018-04-05 15:23:23 +01:00
Nick Craig-Wood
c4531daa43 local: work on spurious "can't copy - source file is being updated" errors #2206 
Update all the time comparisons to use time.Time.Equal instead of ==

Improve the logging for that error so we can see exactly what has changed
 2018-04-05 14:57:30 +01:00
remusb
6e11a25df5 cache: flush the memory cache after close 2018-04-04 23:25:53 +03:00
remusb
1dea99ab20 cache: purge file data on notification 2018-04-03 23:24:45 +03:00
remusb
e42cee5e02 cache: always forget parent dir for notifications - for #2117 2018-03-31 12:44:09 +03:00
Stefan
a8267d1628
link: allow creating public link to files and folders - closes #1562 2018-03-29 09:10:19 +02:00
Nick Craig-Wood
9df266a6b4 onedrive: Fix socket leak in multipart session upload 
This had gone unnoticed until recently when we changed to uploading
all files with a multipart session.
 2018-03-28 21:03:19 +01:00
Stefan Breunig
4d553ef701 drive: when initialized with a filepath, optional features used incorrect root path – see #2182 2018-03-28 20:33:39 +02:00
remusb
b52e34ef5e cache: add info log on notification - for #2150 2018-03-23 22:41:01 +02:00
remusb
0ed0d9a7bc cache: integrate with Plex websocket 2018-03-22 21:21:03 +02:00
seuffert
d9c13bff83 add rc cache/stats 2018-03-22 21:16:16 +02:00
Nick Craig-Wood
5ba5be9b37 gcs: ignore zero length directory markers at the root too 2018-03-21 20:10:00 +00:00
Nick Craig-Wood
e9a2cbec37 s3: ignore zero length directory markers at the root too 2018-03-21 20:09:37 +00:00
Nick Craig-Wood
f6020f1308 gcs: ignore zero length directory markers 2018-03-19 17:42:27 +00:00
Nick Craig-Wood
a46f2a9eb7 s3: ignore zero length directory markers - fixes #1621 2018-03-19 17:41:46 +00:00
Nick Craig-Wood
911a78ce6d sftp: require go1.8+ after github.com/pkg/sftp update 2018-03-19 16:37:40 +00:00
remusb
4924ac2f17 cache: reduce log level for plex api - for #2102 2018-03-17 11:57:36 +02:00
Nick Craig-Wood
d4cca8d9f9 onedrive: fix upload of zero length files #1716 
Unfortunately multi part upload can't upload zero length files so
bring back the single part upload for zero length files only.

This was broken when we made all uploads multipart uploads.
 2018-03-17 09:48:28 +00:00
Oliver Heyme
645cf5ec0f onedrive: fix wrong upload endpoint and createDate #1716 
This fixes the problem introduced by 7f744033d8
 2018-03-16 19:18:51 +00:00
Nick Craig-Wood
d1bb8efb88 sftp: follow symlinks correctly - fixes #2145 
The sftp library delivers the attributes of the symlink rather than
the object pointed to in directory listings, however when we use Stat
from the library it points to the objects.

Previous to this fix this caused items pointed to by symlinks to be
unusable.

After the fix both symlinked files and directories work as expected.
 2018-03-16 15:36:47 +00:00
Leo R. Lundgren
04e055fc06 sftp: Add --sftp-ask-password flag to prompt for password when needed - #2137 2018-03-15 20:24:30 +00:00
Giri Badanahatti
aba43cd3a4 Documention for IBM COS (S3) configuration. 2018-03-15 20:20:43 +00:00
Oliver Heyme
7f744033d8 onedrive: Removed upload cutoff and always do session uploads 
Set modtime on copy


Added versioning issue to OneDrive documentation
 2018-03-15 20:18:11 +00:00
remusb
078d705dbe cache: notify vfs and support crypt in rpc - #2111 2018-03-15 11:39:16 +02:00
Nick Craig-Wood
5981f9fab5 acd: disable integration tests 
We no longer have any working keys for Amazon Cloud Drive so disable
the integration tests.
 2018-03-14 22:44:46 +00:00
remusb
677971643c cache: add support for rc 2018-03-14 22:58:20 +02:00
Remus Bunduc
d76da1f5fd cache: fix dir cache issue - #2117 2018-03-14 11:08:30 +02:00
Nick Craig-Wood
89748feaa5 s3: update docs to discourage use of v2 auth - fixes #2120 
From testing it appears that CEPH no longer works properly with v2
auth and neither does Dreamhost, so update the docs anc configuration
to recommend v4 auth.
 2018-03-13 20:47:29 +00:00
Nick Craig-Wood
dfd0f4c5a4 sync: when using --backup-dir don't delete files if we can't set their modtime 
This is a problem when syncing a file which just needed its modtime
set with dropbox which can't set the mod time of a file without
re-uploading it.

Before this change we would delete the file, then the server side move
would fail moving the file to the backup-dir because it no longer
existed.

After this change the destination file is moved to the backup-dir
instead of being deleted and the new file is uploaded.

Fixes #2134
 2018-03-13 16:05:06 +00:00
Nick Craig-Wood
a7dbf32c53 cache: Implement --cache-db-wait-time flag 
This can be used to make the cache wait for other running cache
backends to finish rather than erroring after 1 second.
 2018-03-08 21:14:09 +00:00
Nick Craig-Wood
6025bb6ad1 local: fix race conditions updating the hashes 
This was causing occasional test failures for the -race test of mount
and cmount.
 2018-03-08 21:08:41 +00:00
Remus Bunduc
70f07fd3ac
fs: add ChangeNotify and backend support for it (#2094) 
* fs: rename DirChangeNotify to ChangeNotify

* cache: switch to ChangeNotify

* ChangeNotify: keep order of notifications
 2018-03-08 22:03:34 +02:00
Nick Craig-Wood
fe8eeec5b5 cache: improve efficiency with RangeOption and RangeSeek #1825 
* All remotes now support RangeOption so remove SeekOption
  * Correct off by one error as RangeOption arguments are inclusive.
  * Use RangeSeek in preference to Seek if available
 2018-03-02 17:10:56 +00:00
Nick Craig-Wood
7d4da1c66a local: fix crash on Stat error while reading a file 2018-03-01 13:17:40 +00:00
Nick Craig-Wood
f3e982d3bf azureblob,b2,gcs,qingstor,s3,swift: Don't check for bucket/container presense if listing was OK 
In a typical rclone copy to a bucket/container based remote, before
this change we were doing a list, followed by a HEAD of the bucket to
check it existed before doing the copy.  The fact the list succeeded
means the bucket exists so mark it OK at that point.

Issue #1421
 2018-03-01 12:11:34 +00:00
Nick Craig-Wood
fdebf9da31 local: Downgrade "invalid cross-device link: trying copy" to debug - Fixes #1875 2018-02-28 21:27:34 +00:00
Fabian Möller
167a4396c7 drive: remove debug binary 2018-02-27 09:59:06 +01:00
Nick Craig-Wood
724120d2f3 local: make DirMove return fs.ErrorCantDirMove to allow fallback 
Before this change `rclone move localdir /mnt/different-fs` would
error.  Now it falls back to moving individual files, which in turn
falls back to copying individual files across the filesystem boundary.
 2018-02-26 12:55:05 +00:00
Nick Craig-Wood
25bbc5d22b drive: make --drive-auth-owner-only look in all directories 
Previously it was ignoring directories which weren't owned by the user
which meant it was ignoring files owned by the user in those
directories.
 2018-02-26 12:30:59 +00:00
Fabian Möller
00adf40f9f
cryptdecode: use Cipher instead of NewFs (#2087) 
* crypt: extract NewCipher out of NewFs
* cryptdecode: make use of crypt.NewCipher

Fixes #2075
 2018-02-25 12:57:14 +01:00
Fabian Möller
aeefa34f62
fstests: add TestInternal (#2085) 
TestInternal allows to perform a custom test on the backend using the
optional InternalTester interface.
 2018-02-25 10:58:06 +01:00
Nick Craig-Wood
1383df4f58 b2: add more logging on multipart upload errors to debug #2036 2018-02-21 09:05:59 +00:00
Nick Craig-Wood
c99412d11e cryptcheck: make reading of nonce more efficient with RangeOption #1825 
...also only calculate the required hash which will speed things up slightly.
 2018-02-19 18:00:39 +00:00
Nick Craig-Wood
c929de9dc4 crypt: Implement RangeOption #1825 2018-02-19 15:45:24 +00:00
Victor
2f63a9f81c onedrive: Overwrite object size value with real size when reading file. 
Because of a bug in the Onedrive API it will sometime report the wrong
size. If the size is wrong other remotes that depend on the size might
fail. To fix this we overwrite the objects size with the real size
from ContentLength header.
 2018-02-16 13:21:26 +00:00
Nick Craig-Wood
b1945d0094 swift: fix refresh of authentication token 
Before this fix we were doing the token refresh but ignoring the new
tokens.

This bug was introduced in v1.39 by 4c0e2f9b3b

Fixes #2018
Fixes #2031
 2018-02-15 19:22:45 +00:00
remusb
9a34fd984c cache: fix dirmove with temp fs enabled 2018-02-14 23:47:45 +02:00
Nick Craig-Wood
644313a4b9 http: Fix handling of directories with & in 
This was caused by inconsistent escaping of the URL in the prefix
check, so check the URL links back to the correct host and scheme
instead of the prefix check.

The decoded path check will catch any URLs which are outside of the
root.
 2018-02-14 11:26:37 +00:00
remusb
54724a1362 cache: notify vfs when using temp fs - fixes #2051 2018-02-11 22:30:58 +02:00
remusb
b33e3f779c
cache: add support for polling 2018-02-10 22:01:05 +02:00
Nick Craig-Wood
8a25ca786c drive: add --drive-impersonate for service accounts #1491 2018-02-09 16:58:35 +00:00
Fabian Möller
b183bd7f00 alias: add new backend to create aliases for remote names #1049 
The alias backend is a wrapper for an existing remote.
It allows you to name a "remote:path" as an "alias:".
 2018-02-06 18:23:47 +00:00
Nick Craig-Wood
5055b340da swift: Fix extra HEAD transaction when uploading a new file - fixes #2053 
Also don't keep the swift.Headers as a pointer to a map, just use the map
 2018-02-06 14:43:21 +00:00
Nick Craig-Wood
82418c3021 box: improve accounting for chunked uploads 2018-02-02 15:14:41 +00:00
Nick Craig-Wood
bf6101cb6c azureblob: improve accounting for chunked uploads 2018-02-02 15:14:41 +00:00
Nick Craig-Wood
5723d2dbff pcloud: remove unused chunked upload flag and code 2018-02-02 15:14:41 +00:00
Nick Craig-Wood
d0d6b83a7a fs/accounting: rework to enable accounting to work with crypt and b2 
This removes the old system of part accounting and replaces it with a
system of popping off the accounting reader and wrapping up new ones
as necessary.

This makes it much easier to carry the context down the chain of
wrapped readers and get the limiting as near as possible to the
output.  This makes the accounting more accurate and the bandwidth
limiting smoother.

Fixes #2029 and Fixes #1443
 2018-02-02 15:14:41 +00:00
nbuchanan
9aa8815990 drive: add --drive-use-created-date to use created date as modified date 2018-02-02 14:20:11 +00:00
ishuah
4c1ffc7f54 copy/move: detect file size change during copy/move - fixes #1250 2018-02-02 13:49:11 +00:00
Nick Craig-Wood
a6227f34e2 drive: request the export formats only when required #320 
If the listing has no google docs in or the user uses
`--drive-skip-gdocs` then we don't fetch the export formats which
saves a transaction to drive.
 2018-02-01 12:05:00 +00:00
Nick Craig-Wood
2eb5cfb7ad fs: Formalize the ObjectUnWrapper interface 2018-01-31 16:21:41 +00:00
remusb
b3d8b7e22e cache: use atexit for cleanup 2018-01-30 22:35:53 +02:00
Nick Craig-Wood
11fe3fdc16 drive: update docs to clarify access to "Computers" tab #1773 2018-01-30 16:28:55 +00:00
Fabian Möller
cf6d522d2f
drive: fix upload to existing file (#2032) 
This fixes uploads to existing files for Google Drive introduced by #2007.
Instead of updating the old file a new "Untitled" file would be created
in the root folder.
 2018-01-30 14:37:06 +01:00
Fabian Möller
29d428040c
cache: clean root path (#2023) 
Trim "/" from the root path to fix "slice bounds out of range" panic
in cache.go:1272.

Fixes #1945
 2018-01-30 14:35:40 +01:00
Fabian Möller
1aa482c333
drive: fix chunked upload (#2030) 2018-01-29 23:36:39 +01:00
remusb
40af98b0b3
cache: offline uploading 2018-01-30 00:05:04 +02:00
Nick Craig-Wood
1852a0e0c9 dropbox: Fix custom oauth client parameters - fixes #2028 2018-01-29 20:04:41 +00:00
Nick Craig-Wood
a4fe2455ed drive: add scope configuration and root folder selection 
This allows:

  * appdata access - Fixes #1799
  * access to backup and sync folders - Fixes #1773
  * drives.file access - Fixes #2000
  * read only access - Fixes #337
 2018-01-29 14:40:10 +00:00
Fabian Möller
f622017539 drive: use contains for name matching in list 
Use contains for name matching in list to work around #1675.
 2018-01-29 14:18:49 +00:00
Fabian Möller
07f20dd1fd drive: migrate to api v3 2018-01-29 12:00:02 +00:00
Nick Craig-Wood
fe52502f19 fs: Adjust RangeOption.Decode to return -1 for read to end 
A Range request can never request 0 bytes however this change was made
to make a clearer signal that the limit means read to the end.

Add test and more documentation and fixup uses
 2018-01-27 14:31:29 +00:00
Fabian Möller
29286cc8b3 drive: fix single Drive Document as FS root 
Allow using Drive Documents as FS root by doing a direcoty list during NewFS.

Fixes #1772
 2018-01-26 09:59:36 +00:00
Fabian Möller
17dabf7a99 ftp: fix RangeOption support in Open #1825 2018-01-25 10:21:00 +00:00
Fabian Möller
9520992a54 sftp: fix RangeOption support in Open #1825 2018-01-25 10:20:43 +00:00
Fabian Möller
a3dd2c691e amazonclouddrive: remove unnecessary notifies from DirChangeNotify 
It is unnecessary to notify the node.Parents, because a cahnge event is
generated for all involved files and folders in a move from d1/f1 to
d2/f1. There will be a event for d1, d2 and f1.

Additionally a duplicate notification is resolved when them empty string
is in pathsToClear.

Related to #2006
 2018-01-25 10:19:06 +00:00
Nick Craig-Wood
38f829842a s3: fix server side copy and set modtime on files with + in - fixes #2001 
This was broken in 64ea94c1a4 when
putting a work-around for Digital Ocean.  PathEscape has now been
adjusted so it works with both providers.
 2018-01-23 10:50:50 +00:00
Nick Craig-Wood
88e0770f2d cache: Implement RangeOption #1825 2018-01-22 19:44:55 +00:00
Nick Craig-Wood
a6833b68ca local: factor RangeOption code to Decode() method and readers.LimitedReadCloser #1825 2018-01-22 19:44:00 +00:00
Nick Craig-Wood
e44dc2b14d box: fix RangeOption support in Open #1825 2018-01-22 17:05:47 +00:00
Nick Craig-Wood
d876392d15 onedrive: Factor code into fs.FixRangeOption 2018-01-22 17:05:00 +00:00
Fabian Möller
186f78d44f local: fix RangeOption support in Open #1825 2018-01-21 19:50:26 +00:00
Nick Craig-Wood
c963c74fbe onedrive: fix RangeOption support in Open #1825 2018-01-21 17:11:37 +00:00
Nick Craig-Wood
9c45125271 azureblob: fix RangeOption support in Open #1825 2018-01-21 17:11:32 +00:00
Nick Craig-Wood
8653944a6d Make RangeOption manadatory for Open - #1825 
Add an integration test to make sure all backends implement
RangeOption correctly.
 2018-01-21 17:09:12 +00:00
Jon Fautley
71bc108ce6 sftp: performance: don't consult config file outside of Fs setup 2018-01-21 09:37:22 +00:00
Fabian Möller
115d24e1f7 amazonclouddrive: implement DirChangeNotify 
Use the Changes API to invalidate cache entries.
The latest retrieved checkpoint is stored in the config file to allow
fast resumption after restart.
 2018-01-20 18:48:52 +00:00
Nick Craig-Wood
25a41e1945 drive: fix missing error handler 2018-01-20 18:04:23 +00:00
Nick Craig-Wood
97c414f025 config/hash: rename more symbols after factoring into own package 2018-01-18 20:27:52 +00:00
Nick Craig-Wood
71722b5b95 config: factor Obscure and Reveal into its own package 2018-01-18 20:19:55 +00:00
Nick Craig-Wood
59a8108fc3 webdav: add a new time format #1952 2018-01-18 16:54:13 +00:00
Iakov Davydov
acd55a8f65 local, fs: --exclude-if-present ignores directories which it doesn't have permission for - fixes #1959 2018-01-16 20:00:16 +00:00
Nick Craig-Wood
4b0c5f79b5 qingstor: Only support on go1.7+ 2018-01-16 17:05:26 +00:00
Nick Craig-Wood
1848e26183 dropbox: Only support on go1.7+ 
See https://github.com/dropbox/dropbox-sdk-go-unofficial/pull/40
 2018-01-16 17:05:02 +00:00
Nick Craig-Wood
11da2a6c9b Break the fs package up into smaller parts. 
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
 2018-01-15 17:51:14 +00:00
Nick Craig-Wood
92624bbbf1 Move all backends into backend directory 2018-01-12 20:27:08 +00:00
Nick Craig-Wood
60afda007b Move dircache, oauthutil, rest and pacer modules into lib 2018-01-12 17:07:38 +00:00
Nick Craig-Wood
b8b620f5c2 Move all backends into backend directory 2018-01-12 17:07:38 +00:00