extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-11-30 04:15:26 +01:00
Code Issues Packages Projects Releases Wiki Activity
2,222 Commits 177 Branches 114 Tags 318 MiB
96a62d55a2
Commit Graph

583 Commits

Author SHA1 Message Date
Nick Craig-Wood
a6833b68ca local: factor RangeOption code to Decode() method and readers.LimitedReadCloser #1825 2018-01-22 19:44:00 +00:00
Nick Craig-Wood
d876392d15 onedrive: Factor code into fs.FixRangeOption 2018-01-22 17:05:00 +00:00
Nick Craig-Wood
84bc4dc142 Clarify RangeOption semantics 2018-01-21 09:51:28 +00:00
Nick Craig-Wood
ee66419a27 fs/fserrors: Add test for error from #1964 2018-01-19 17:07:40 +00:00
Nick Craig-Wood
5ad226ab54 fs: Add dir option to fs.Purge #1891 
Purge optional interface needs fixing too.
 2018-01-19 11:45:50 +00:00
Nick Craig-Wood
97c414f025 config/hash: rename more symbols after factoring into own package 2018-01-18 20:27:52 +00:00
Nick Craig-Wood
71722b5b95 config: factor Obscure and Reveal into its own package 2018-01-18 20:19:55 +00:00
Ernest Borowski
5cce74d630 flags: remove --no-traverse flag because it is obsolete - fixes #1813 
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
 2018-01-18 11:00:25 +00:00
Iakov Davydov
acd55a8f65 local, fs: --exclude-if-present ignores directories which it doesn't have permission for - fixes #1959 2018-01-16 20:00:16 +00:00
Nick Craig-Wood
11da2a6c9b Break the fs package up into smaller parts. 
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
 2018-01-15 17:51:14 +00:00
Nick Craig-Wood
b8b620f5c2 Move all backends into backend directory 2018-01-12 17:07:38 +00:00
Nick Craig-Wood
712e6a8085 lsf: fix integration tests 2018-01-11 13:52:15 +00:00
Will Gunn
5c7e8d5a2b fs: Add --stats-file-name-length to specify the printed file name length for stats 
Fixes #1206
 2018-01-10 20:32:36 +00:00
Nick Craig-Wood
c74c3b37da lsf: add option to print hashes 2018-01-06 17:53:37 +00:00
Nick Craig-Wood
7c71ee1a5b fs: fix TestListFormat on remotes which return 0 as dir size not -1 2018-01-06 17:47:42 +00:00
Jakub Tasiemski
0d041602cf cmd: new command lsf 2018-01-06 14:39:31 +00:00
Nick Craig-Wood
b412c745a1 Start v1.39-DEV development 2017-12-23 13:40:28 +00:00
Nick Craig-Wood
f34a9116d4 Version v1.39 2017-12-23 13:07:45 +00:00
Nick Craig-Wood
7e585cda96 fs: fix TestRmdirsLeaveRoot test 2017-12-14 08:57:28 +00:00
lewapm
9c242edc10 rmdirs: add --leave-root flag - fixes #1874 2017-12-13 10:23:54 +00:00
ishuah
b05e472d2e stats: condensed transfer output to fit 80x25 terminals 2017-12-09 10:48:36 +03:00
remusb
25b073c767
fs: add Wrap feature for FS to identify their parent FS (#1884) 2017-12-06 17:14:34 +02:00
Nick Craig-Wood
fa4a25a73b fs: only test one level of cache 
Can't test multiple caches at once as can only have 1 DB open at once
 2017-12-04 15:50:59 +00:00
Ernest Borowski
934df67aef filter: warn the user if he use --include and --exclude together fixes #1764 
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
 2017-12-04 14:20:01 +00:00
Nick Craig-Wood
4b185355df fs: rcat - use in memory object and Copy for more reliable transfers 2017-12-03 12:14:15 +00:00
Nick Craig-Wood
7d15c33e42 fs: make Copy and Move return the destination object if possible 2017-12-03 12:14:15 +00:00
Nick Craig-Wood
11332a19a0 fs: make an in memory object for short transfers 2017-12-03 12:14:15 +00:00
Nick Craig-Wood
639e812789 fs: integration tests: add SUMMARY heading for log scraping 2017-11-29 15:55:37 +00:00
Nick Craig-Wood
1c6cad2252 fs: integration tests: add 30 minute timeout per test 2017-11-29 13:51:17 +00:00
ishuah
aab8051f50 move: add --delete-empty-src-dirs flag - fixes #1854 2017-11-28 11:38:19 +03:00
Nick Craig-Wood
fdb01437d8 fs: Allow the http Transport to have an optional filter request function 2017-11-24 09:07:56 +00:00
Nick Craig-Wood
02ffd43572 fs: Save the config before asking for a token - fixes #1220 
Before this if the client_id/client_secret was edited it would
disappear when asking for the new token.

This means the post config is done after the user has confirmed the
config is OK which can't be helped.
 2017-11-23 14:01:32 +00:00
Nick Craig-Wood
e53892f53b fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use 
RepeatableReaderSized has a pre-allocated buffer which should help
with memory usage - before it grew the buffer.  Since we know the size
of the chunks, pre-allocating it should be much more efficient.

RepeatableReaderBuffer uses the buffer passed in.

RepeatableLimit* are convenience funcitions for wrapping a reader in
an io.LimitReader and then a RepeatableReader with the same buffer
size.
 2017-11-23 13:53:46 +00:00
ishuah
6c62fced60 move: fixed root source directories getting deleted after move - fixes #1849 2017-11-23 12:01:35 +03:00
Nick Craig-Wood
159fce0106 fs: fix --cache-dir to have some effect 2017-11-22 17:05:02 +00:00
Nick Craig-Wood
cbe5d7ce64 fs: Remove X-Auth-Token: from headers when dumping for swift 2017-11-21 17:32:07 +00:00
Nick Craig-Wood
1a65a4e769 fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters 
Now --dump-flag is written as --dump flag. This is a comma separated list which can contain

  * headers - HTTP headers as before
  * bodies  - HTTP bodies as before
  * requests - HTTP request bodies
  * responses - HTTP response bodies
  * auth - HTTP auth
  * filters - Filter rexeps

Leave --dump-headers and --dump-bodies for the time being but remove
the other --dump-* flags as they aren't used very often.
 2017-11-21 17:32:07 +00:00
ishuah
b4aa920a3d stats: show the amount of data transferred in kb/mb - fixes #1167 2017-11-21 12:40:02 +03:00
Nick Craig-Wood
8951875c21 vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
e946a8eab0 fs: Add CacheDir config variable 2017-11-20 12:00:32 +00:00
Nick Craig-Wood
46947b3b9b rcat: fix goroutine leak 
This was leaking goroutines in the short file case beause it wasn't
calling Close() on the Account object.  This became apparent when
testing with mount.
 2017-11-20 12:00:32 +00:00
Iakov Davydov
7d3abdc463 tests for --exclude-if-present 2017-11-17 21:51:11 +00:00
Iakov Davydov
538246f6c3 support exclude file in --fast-list mode 2017-11-17 21:51:11 +00:00
Iakov Davydov
557dd8f031 ListDirSorted check for excludefile 2017-11-17 21:51:11 +00:00
Iakov Davydov
37aaa19f3a new option: --exclude-if-present 2017-11-17 21:51:11 +00:00
Iakov Davydov
cef2e3bf83 path -> startPath in walkRDirTree (we need the path package) 2017-11-17 21:51:11 +00:00
Iakov Davydov
a3a436ce16 WalkRDirTree: return error if unknown item type 2017-11-17 21:51:11 +00:00
Iakov Davydov
5d05df3124 ListContainsExcludeFile: checks for exclude file in the list 2017-11-17 21:51:11 +00:00
Iakov Davydov
421ba84e12 DirTree.Prune: deletes several directories 2017-11-17 21:51:11 +00:00
Iakov Davydov
7ae7080824 FileExists check if a file exists 2017-11-17 21:51:11 +00:00
Remus Bunduc
016abf825e cache: first version 2017-11-15 15:23:21 +00:00
ishuah
aec2265be0 rclone: implement exit codes - #1136 2017-11-15 17:48:37 +03:00
Substantiel
2423fa40e2 config: add password sub command for setting obscured passwords 2017-11-15 14:44:45 +00:00
ishuah
c65734ee69 move: delete source directory after successful move - fixes #1642 2017-11-07 22:21:38 +00:00
Substantiel
7749157596 Add --auto-confirm flag 2017-11-05 21:56:50 +00:00
Nick Craig-Wood
cc7ed13b9b fs: factor test running code into fstest/run.go 2017-11-03 12:59:58 +00:00
ishuah
10f27e2ff2 allow trailing+leading whitespace for passwords - #1717 
warn users when they enter passwords with leading/trailing whitespaces

Updated config_test.go, removing deprecated test case and updated TestReveal
 2017-10-20 15:46:17 +01:00
Nick Craig-Wood
115053930e Make error messages less crypting when revealing an unobscured password - fixes #1743 2017-10-16 22:03:06 +01:00
Nick Craig-Wood
edfab09eb9 config: add sub commands for full config file management 
Previously config sub commands were manually parsed rather than using
cobra.

Make config command have the following sub commands:

 * create    Create a new remote with name, type and options.
 * delete    Delete an existing remote <name>.
 * dump      Dump the config file as JSON.
 * edit      Enter an interactive configuration session.
 * file      Show path of configuration file in use.
 * providers List in JSON format all the providers and options.
 * show      Print (decrypted) config file, or the config for a single remote.
 * update    Update options in an existing remote.

The following changes were made to existing commands

 * listproviders was renamed to providers
 * listoptions was removed in favour of providing the output in providers
 * jsonconfig was renamed to create
 * an optional parameter was added to the show command
 2017-10-14 11:50:41 +01:00
thierry
0575623dff Add config listproviders, listoptions, jsonconfig for automated config 
Addition of a method listing the providers, a method listing the
options of a provider and method of manual configuration.
 2017-10-13 17:17:36 +01:00
Nick Craig-Wood
fc8b13c993 moveto/copyto: Fix to allow copying to the same name - fixes #1736 2017-10-12 20:45:36 +01:00
Nick Craig-Wood
43ced30f11 fs: Add more errors to retry - fixes #1733 2017-10-10 19:51:02 +01:00
Nick Craig-Wood
3eb7f52e39 fs: Add "unexpected EOF reading trailer" as a retriable error - fixes #1730 2017-10-09 17:29:16 +01:00
Nick Craig-Wood
7f3dc9b5c4 Implement WebDAV remote #580 
This has special knowledge of Owncloud and Nextcloud to enable more
functionality such as mod times.
 2017-10-09 16:19:37 +01:00
Nick Craig-Wood
00fe6d95da fs: fix duplicate files causing spurious copies 
Before this fix duplicate files (on Google Drive) caused the next file
to be spuriously copied.  `rclone dedupe` worked around the problem.
 2017-10-02 16:52:53 +01:00
Nick Craig-Wood
9e9297838f Implement pcloud remote - #418 2017-10-01 11:37:35 +01:00
Nick Craig-Wood
1c10497b68 Start v1.38-DEV development 2017-09-30 15:16:09 +01:00
Nick Craig-Wood
d96e45ba5b Version v1.38 2017-09-30 14:20:43 +01:00
Nick Craig-Wood
657b3a674d fs: fix test_all -clean to run just one cleaning thread per remote 2017-09-30 11:07:09 +01:00
Nick Craig-Wood
2857ed5c35 fs: fix --immutable tests on remotes which don't have modtime 2017-09-28 08:56:30 +01:00
Jacob McNamee
2d8e75cab4 Implement --immutable option 2017-09-24 20:00:00 +01:00
Stefan
85877f3adc config: add show/file subcommands which print the config/its path (fixes #1086) 2017-09-19 17:59:19 +02:00
Nick Craig-Wood
87335de8a8 fs: fix filename normalization issues in the tests when running on OS X 2017-09-17 15:31:22 +01:00
Stefan Breunig
12405f9f41 fuse: re-use rcat to support uploads for all remotes (fixes #1672) 2017-09-16 22:49:08 +02:00
Stefan
234bfae0d5 b2: implement streaming upload of files with unknown length (see #1614) (closes #1686) 2017-09-16 22:43:48 +02:00
Nick Craig-Wood
4ac9a65049 fs: stop normalizing file names but do a normalized compare in the sync 
This works by using a transform function to transform file names when
doing a compare when matching file names in a directory.  rclone now
UTF-8 normalizes the file names and does a case insensitive compare if
the destination remote is case insensitive.

This deprecates the --local-no-unicode-normalization flag.

Fixes #1477
 2017-09-16 19:49:31 +01:00
Nick Craig-Wood
a8e41f081c fs: re-implement check and cryptcheck using the same traversal as sync 
This makes them 100% consistent with sync and also make them use less
memory as they no longer build the whole tree in memory first.

Fixes #1657
 2017-09-16 19:49:31 +01:00
Nick Craig-Wood
261c7ad9e4 fs: make syncCopyMove use context for go routine cancellation 2017-09-16 19:49:31 +01:00
Nick Craig-Wood
fe96d5cf0a fs: factor multiple directory traverse out of sync 2017-09-16 19:49:31 +01:00
Nick Craig-Wood
6df12b3f00 fs: improve retriable error detection 2017-09-16 19:48:49 +01:00
Stefan Breunig
9d3f11b493 amazonclouddrive, rcat: ensure rcat integration test passes even with AmazonCloudDrive (fixes: #1680) 2017-09-15 18:09:04 +02:00
Nick Craig-Wood
798502b204 fs: add more errors to be considered temporary errors 
This makes a framework for adding temporary errors identified by
syscall number or by error string.

Fixes #1660
 2017-09-14 18:01:43 +01:00
Stefan Breunig
56dedc49e3 rcat: properly report if the upload fails 2017-09-13 20:21:52 +02:00
Girish Ramakrishnan
4c45cbea18 copy: error out if dst could not be listed 2017-09-12 09:29:44 +01:00
Stefan Breunig
a91448c83a rcat: honor --dry-run even for small files 2017-09-11 22:28:16 +02:00
Stefan Breunig
80b1f2a494 rcat: configurable small files cutoff and implement proper upload verification 2017-09-11 08:26:53 +02:00
Stefan Breunig
57817397a0 rcat: directly upload small files without streaming them 2017-09-11 08:25:34 +02:00
Nick Craig-Wood
cbea06026a Make check obey --ignore-size - fixes #1643 2017-09-01 11:20:41 +01:00
Nick Craig-Wood
2c8d6e86cc fs: fix gofmt 2017-08-31 16:01:19 +01:00
cbruegg
bb6300b032 Fix bwlimit toggle in conjunction with schedules (Fixes #1607) 2017-08-31 15:33:29 +01:00
Stefan Breunig
a122b9fa7a yandex: implement streaming uploads (see #1614) 2017-08-19 14:07:23 +02:00
Alex McGrath Kraak
2cc9071791 http: add --user-agent option. close #1557 2017-08-17 06:49:27 +02:00
Nick Craig-Wood
265fb8a5e2 fs: Manage empty directories - fixes #100 
During the sync we collect a list of directories which should be empty
and attempt to rmdir them at the end of the sync.  If the directories
are not empty then the rmdir will fail, logging a message but not
erroring the sync.
 2017-08-09 21:07:00 +01:00
Nick Craig-Wood
8a1a900733 fstest: use Feature.CanHaveEmptyDirectories to sharpen tests 
Now we actually test whether the directories are present or not,
filtering out empty directories in the test using the
CanHaveEmptyDirectories flag.
 2017-08-09 20:55:08 +01:00
Nick Craig-Wood
20ae7d562b fs: Add CanHaveEmptyDirectories and BucketBased feature flags to all remotes 2017-08-09 20:55:08 +01:00
Nick Craig-Wood
ec2ea37ad2 fs: Add --disable flag to disable optional features - fixes #1551 
Eg to disable server side copy use `--disable copy`, to see a list of
what you can disable, `--disable help`.
 2017-08-07 21:34:45 +01:00
Nick Craig-Wood
bfe812ea6b dedupe: implement merging of duplicate directories - fixes #1243 2017-08-07 15:36:41 +01:00
Nick Craig-Wood
db1995e63a Add MergeDirs optional interface and implement it for drive 2017-08-07 15:32:47 +01:00
Nick Craig-Wood
81a2ab599f fs: add optional ID to fs.Directory and set it in the remotes which care 2017-08-07 15:31:22 +01:00
Stefan Breunig
40f24e0ea3 config: use absolute ConfigPath to ensure newly written config is on the same mount - fixes #1569 2017-08-05 12:13:25 +02:00