diff --git a/.circleci/config.yml b/.circleci/config.yml index 15760322e..d95207738 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -6,7 +6,7 @@ jobs: build: machine: true - working_directory: ~/.go_workspace/src/github.com/ncw/rclone + working_directory: ~/.go_workspace/src/github.com/rclone/rclone steps: - checkout diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md index 98cf1780d..f99fde4eb 100644 --- a/.github/ISSUE_TEMPLATE.md +++ b/.github/ISSUE_TEMPLATE.md @@ -10,7 +10,7 @@ instead of filing an issue for a quick response. If you are reporting a bug or asking for a new feature then please use one of the templates here: - https://github.com/ncw/rclone/issues/new + https://github.com/rclone/rclone/issues/new otherwise fill in the form below. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 7281582f2..38613fee9 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -22,8 +22,8 @@ Link issues and relevant forum posts here. #### Checklist -- [ ] I have read the [contribution guidelines](https://github.com/ncw/rclone/blob/master/CONTRIBUTING.md#submitting-a-pull-request). +- [ ] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-pull-request). - [ ] I have added tests for all changes in this PR if appropriate. - [ ] I have added documentation for the changes if appropriate. -- [ ] All commit messages are in [house style](https://github.com/ncw/rclone/blob/master/CONTRIBUTING.md#commit-messages). +- [ ] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages). - [ ] I'm done, this Pull Request is ready for review :-) diff --git a/.travis.yml b/.travis.yml index 7e3aa798c..67d1d6ed2 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,7 +4,7 @@ sudo: required dist: xenial os: - linux -go_import_path: github.com/ncw/rclone +go_import_path: github.com/rclone/rclone before_install: - git fetch --unshallow --tags - | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 551537a87..3b266b7d1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,12 +29,12 @@ You'll need a Go environment set up with GOPATH set. See [the Go getting started docs](https://golang.org/doc/install) for more info. First in your web browser press the fork button on [rclone's GitHub -page](https://github.com/ncw/rclone). +page](https://github.com/rclone/rclone). Now in your terminal - go get -u github.com/ncw/rclone - cd $GOPATH/src/github.com/ncw/rclone + go get -u github.com/rclone/rclone + cd $GOPATH/src/github.com/rclone/rclone git remote rename origin upstream git remote add origin git@github.com:YOURUSER/rclone.git @@ -127,7 +127,7 @@ If you want to use the integration test framework to run these tests all together with an HTML report and test retries then from the project root: - go install github.com/ncw/rclone/fstest/test_all + go install github.com/rclone/rclone/fstest/test_all test_all -backend drive If you want to run all the integration tests against all the remotes, diff --git a/MAINTAINERS.md b/MAINTAINERS.md index ebc97f163..a5345cd9a 100644 --- a/MAINTAINERS.md +++ b/MAINTAINERS.md @@ -51,7 +51,7 @@ The milestones have these meanings: * Help wanted - blue sky stuff that might get moved up, or someone could help with * Known bugs - bugs waiting on external factors or we aren't going to fix for the moment -Tickets [with no milestone](https://github.com/ncw/rclone/issues?utf8=✓&q=is%3Aissue%20is%3Aopen%20no%3Amile) are good candidates for ones that have slipped between the gaps and need following up. +Tickets [with no milestone](https://github.com/rclone/rclone/issues?utf8=✓&q=is%3Aissue%20is%3Aopen%20no%3Amile) are good candidates for ones that have slipped between the gaps and need following up. ## Closing Tickets ## diff --git a/MANUAL.html b/MANUAL.html index b1f0b5e99..0cbdfbda5 100644 --- a/MANUAL.html +++ b/MANUAL.html @@ -82,7 +82,7 @@

Links

@@ -132,13 +132,13 @@ sudo mv rclone /usr/local/bin/ 
rclone config

Install from source

Make sure you have at least Go 1.7 installed. Download go if necessary. The latest release is recommended. Then

-
git clone https://github.com/ncw/rclone.git
+
git clone https://github.com/rclone/rclone.git
 cd rclone
 go build
 ./rclone version

 
You can also build and install rclone in the GOPATH (which defaults to ~/go) with:

-
go get -u -v github.com/ncw/rclone

-
and this will build the binary in $GOPATH/bin (~/go/bin/rclone by default) after downloading the source to $GOPATH/src/github.com/ncw/rclone (~/go/src/github.com/ncw/rclone by default).

+
go get -u -v github.com/rclone/rclone

+
and this will build the binary in $GOPATH/bin (~/go/bin/rclone by default) after downloading the source to $GOPATH/src/github.com/rclone/rclone (~/go/src/github.com/rclone/rclone by default).

 
Installation with Ansible

 
This can be done with Stefan Weichinger’s ansible role.

 
Instructions

@@ -1189,7 +1189,7 @@ umount /path/to/local/mount

Attribute caching

You can use the flag –attr-timeout to set the time the kernel caches the attributes (size, modification time etc) for directory entries.

The default is “1s” which caches files just long enough to avoid too many callbacks to rclone from the kernel.

-

In theory 0s should be the correct value for filesystems which can change outside the control of the kernel. However this causes quite a few problems such as rclone using too much memory, rclone not serving files to samba and excessive time listing directories.

+

In theory 0s should be the correct value for filesystems which can change outside the control of the kernel. However this causes quite a few problems such as rclone using too much memory, rclone not serving files to samba and excessive time listing directories.

The kernel can cache the info about a file for the time given by “–attr-timeout”. You may see corruption if the remote file changes length during this window. It will show up as either a truncated file or a file with garbage on the end. With “–attr-timeout 1s” this is very unlikely but not impossible. The higher you set “–attr-timeout” the more likely it is. The default setting of “1s” is the lowest setting which mitigates the problems above.

If you set it higher (‘10s’ or ‘1m’ say) then the kernel will call back to rclone less often making it more efficient, however there is more chance of the corruption issue above.

If files don’t change on the remote outside of the control of rclone then there is no chance of corruption.

@@ -2183,7 +2183,7 @@ htpasswd -B htpasswd anotherUser

Windows

If your names have spaces in you need to put them in ", eg

rclone copy "E:\folder name\folder name\folder name" remote:backup
-

If you are using the root directory on its own then don’t quote it (see #464 for why), eg

+

If you are using the root directory on its own then don’t quote it (see #464 for why), eg

rclone copy E:\ remote:backup

Copying files or directories with : in the names

rclone uses : to mark a remote name. This is, however, a valid filename component in non-Windows OSes. The remote name parser will only search for a : up to the first / so if you need to act on a file or directory like this then use the full path starting with a /, or use ./ as a current directory prefix.

@@ -2285,7 +2285,7 @@ rclone sync /path/to/files remote:current-backup

–ignore-size

Normally rclone will look at modification time and size of files to see if they are equal. If you set this flag then rclone will check only the modification time. If --checksum is set then it only checks the checksum.

It will also cause rclone to skip verifying the sizes are the same after transfer.

-

This can be useful for transferring files to and from OneDrive which occasionally misreports the size of image files (see #399 for more info).

+

This can be useful for transferring files to and from OneDrive which occasionally misreports the size of image files (see #399 for more info).

-I, –ignore-times

Using this option will cause rclone to unconditionally upload all files regardless of the state of files on the destination.

Normally rclone would skip any files that have the same modification time and are the same size (or have the same checksum if using --checksum).

@@ -3507,14 +3507,14 @@ $ echo $? 
$ go tool pprof -text http://localhost:5572/debug/pprof/heap
 Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
       flat  flat%   sum%        cum   cum%
- 1024.03kB 66.62% 66.62%  1024.03kB 66.62%  github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode
+ 1024.03kB 66.62% 66.62%  1024.03kB 66.62%  github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode
      513kB 33.38%   100%      513kB 33.38%  net/http.newBufioWriterSize
-         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/cmd/all.init
-         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/cmd/serve.init
-         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/cmd/serve/restic.init
-         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/vendor/golang.org/x/net/http2.init
-         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init
-         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init.0
+         0     0%   100%  1024.03kB 66.62%  github.com/rclone/rclone/cmd/all.init
+         0     0%   100%  1024.03kB 66.62%  github.com/rclone/rclone/cmd/serve.init
+         0     0%   100%  1024.03kB 66.62%  github.com/rclone/rclone/cmd/serve/restic.init
+         0     0%   100%  1024.03kB 66.62%  github.com/rclone/rclone/vendor/golang.org/x/net/http2.init
+         0     0%   100%  1024.03kB 66.62%  github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init
+         0     0%   100%  1024.03kB 66.62%  github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init.0
          0     0%   100%  1024.03kB 66.62%  main.init
          0     0%   100%      513kB 33.38%  net/http.(*conn).readRequest
          0     0%   100%      513kB 33.38%  net/http.(*conn).serve
@@ -3790,10 +3790,10 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 No
 Yes
 Yes
-No #575
+No #575
 No
 No
-No #2178
+No #2178
 No
 
 
@@ -3805,7 +3805,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 No
 Yes
 Yes
-No #2178
+No #2178
 No
 
 
@@ -3817,7 +3817,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 Yes
 Yes
 Yes
-No #2178
+No #2178
 No
 
 
@@ -3826,7 +3826,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 Yes
 Yes
 Yes
-No #575
+No #575
 No
 Yes
 Yes
@@ -3838,7 +3838,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 Yes
 Yes
 Yes
-No #575
+No #575
 No
 Yes
 Yes
@@ -3853,7 +3853,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 No
 No
 Yes
-No #2178
+No #2178
 No
 
 
@@ -3865,7 +3865,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 No
 Yes
 Yes
-No #2178
+No #2178
 No
 
 
@@ -3889,7 +3889,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 No
 No
 No
-No #2178
+No #2178
 No
 
 
@@ -3901,7 +3901,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 No
 Yes
 Yes
-No #2178
+No #2178
 Yes
 
 
@@ -3925,7 +3925,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 Yes
 No
 No
-No #2178
+No #2178
 Yes
 
 
@@ -3937,7 +3937,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 No
 Yes
 No
-No #2178
+No #2178
 No
 
 
@@ -3946,7 +3946,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 Yes
 Yes
 Yes
-No #575
+No #575
 No
 No
 Yes
@@ -3973,7 +3973,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 No
 Yes
 Yes
-No #2178
+No #2178
 Yes
 
 
@@ -3985,7 +3985,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 Yes
 No
 No
-No #2178
+No #2178
 Yes
 
 
@@ -3997,7 +3997,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 No
 Yes
 No
-No #2178
+No #2178
 No
 
 
@@ -4009,7 +4009,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 No
 No
 Yes
-No #2178
+No #2178
 Yes
 
 
@@ -4021,7 +4021,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 No
 No
 Yes ‡
-No #2178
+No #2178
 Yes
 
 
@@ -4666,7 +4666,7 @@ In the case the object is larger than 5Gb or is in Glacier or Glacier Deep Archi
 
For reference, here’s an Ansible script that will generate one or more buckets that will work with rclone sync.

 
Key Management System (KMS)

 
If you are using server side encryption with KMS then you will find you can’t transfer small objects. As a work-around you can use the --ignore-checksum flag.

-
A proper fix is being worked on in issue #1824.

+
A proper fix is being worked on in issue #1824.

 
Glacier and Glacier Deep Archive

 
You can upload objects using the glacier storage class or transition them to glacier using a lifecycle policy. The bucket can still be synced or copied into normally, but if rclone tries to access data from the glacier storage class you will see an error like below.

 
2017/09/11 19:07:43 Failed to sync: failed to open source object: Object in GLACIER, restore first: path/to/file

@@ -6233,7 +6233,7 @@ y/e/d> y

The SHA1 checksums of the files are checked on upload and download and will be used in the syncing process.

Large files (bigger than the limit in --b2-upload-cutoff) which are uploaded in chunks will store their SHA1 on the object as X-Bz-Info-large_file_sha1 as recommended by Backblaze.

For a large file to be uploaded with an SHA1 checksum, the source needs to support SHA1 checksums. The local disk supports SHA1 checksums so large file transfers from local disk will have an SHA1. See the overview for exactly which remotes support SHA1.

-

Sources which don’t support SHA1, in particular crypt will upload large files without SHA1 checksums. This may be fixed in the future (see #1767).

+

Sources which don’t support SHA1, in particular crypt will upload large files without SHA1 checksums. This may be fixed in the future (see #1767).

Files sizes below --b2-upload-cutoff will always have an SHA1 regardless of the source.

Transfers

Backblaze recommends that you do lots of transfers simultaneously for maximum speed. In tests from my SSD equipped laptop the optimum setting is about --transfers 32 though higher numbers may be used for a slight speed improvement. The optimum number for you may vary depending on your hardware, how big the files are, how much you want to load your computer, etc. The default of --transfers 4 is definitely too low for Backblaze B2 though.

@@ -6241,7 +6241,7 @@ y/e/d> y

Versions

When rclone uploads a new version of a file it creates a new version of it. Likewise when you delete a file, the old version will be marked hidden and still be available. Conversely, you may opt in to a “hard delete” of files with the --b2-hard-delete flag which would permanently remove the file instead of hiding it.

Old versions of files, where available, are visible using the --b2-versions flag.

-

NB Note that --b2-versions does not work with crypt at the moment #1627. Using –backup-dir with rclone is the recommended way of working around this.

+

NB Note that --b2-versions does not work with crypt at the moment #1627. Using –backup-dir with rclone is the recommended way of working around this.

If you wish to remove all the old versions then you can use the rclone cleanup remote:bucket command which will delete all the old versions of files, leaving the current ones intact. You can also supply a path and only old versions under that path will be deleted, eg rclone cleanup remote:bucket/path/to/stuff.

Note that cleanup will remove partially uploaded files from the bucket if they are more than a day old.

When you purge a bucket, the current and the old versions will be deleted then the bucket will be deleted.

@@ -6276,7 +6276,7 @@ $ rclone -q --b2-versions ls b2:cleanup-test /b2api/v1/b2_create_bucket /b2api/v1/b2_list_buckets /b2api/v1/b2_list_file_names -

The b2_list_file_names request will be sent once for every 1k files in the remote path, providing the checksum and modification time of the listed files. As of version 1.33 issue #818 causes extra requests to be sent when using B2 with Crypt. When a copy operation does not require any files to be uploaded, no more requests will be sent.

+

The b2_list_file_names request will be sent once for every 1k files in the remote path, providing the checksum and modification time of the listed files. As of version 1.33 issue #818 causes extra requests to be sent when using B2 with Crypt. When a copy operation does not require any files to be uploaded, no more requests will be sent.

Uploading files that do not require chunking, will send 2 requests per file upload:

/b2api/v1/b2_get_upload_url
 /b2api/v1/b2_upload_file/
@@ -6731,9 +6731,9 @@ chunk_total_size = 10G

Most of the issues seem to be related to the difference between filesystems on Linux flavors and Windows as cache is heavily dependant on them.

Any reports or feedback on how cache behaves on this OS is greatly appreciated.

Risk of throttling

Future iterations of the cache backend will make use of the pooling functionality of the cloud provider to synchronize and at the same time make writing through it more tolerant to failures.

@@ -6741,8 +6741,8 @@ chunk_total_size = 10G

Some recommendations: - don’t use a very small interval for entry informations (--cache-info-age) - while writes aren’t yet optimised, you can still write through cache which gives you the advantage of adding the file in the cache at the same time if configured to do so.

Future enhancements:

cache and crypt

One common scenario is to keep your data encrypted in the cloud provider using the crypt remote. crypt uses a similar technique to wrap around an existing remote and handles this translation in a seamless way.

@@ -8661,7 +8661,7 @@ trashed=false and 'c' in parents

–drive-alternate-export

Use alternate export URLs for google documents export.,

If this option is set this instructs rclone to use an alternate set of export URLs for drive documents. Users have reported that the official export URLs can’t export large documents, whereas these unofficial ones can.

-

See rclone issue #2243 for background, this google drive issue and this helpful post.

+

See rclone issue #2243 for background, this google drive issue and this helpful post.