diff --git a/MAINTAINERS.md b/MAINTAINERS.md new file mode 100644 index 000000000..2ce3e289c --- /dev/null +++ b/MAINTAINERS.md @@ -0,0 +1,58 @@ +# Maintainers guide for rclone # + +Current active maintainers of rclone are + + * Nick Craig-Wood + * Stefan Breunig + +**This is a work in progress Draft** + +This is a guide for how to be an rclone maintainer. + +## Triaging Tickets ## + +***FIXME*** this section needs some work! + +When a ticket comes in it should be triaged. This means it should be classified into a bug or an enhancement or a request for support. + +Quite a lot of tickets need a bit of back an forth to determine whether it is a valid ticket. + +If it turns out to be a bug or an enhancement it should be tagged as such, with the appropriate other tags. Don't forget the "quickie" tag to give new contributors something easy to do to get going. + +When a ticket is tagged it should be added to a milestone, either the next release, the one after, Soon or Unplanned. Bugs can be added to the "Known Bugs" milestone if they aren't planned to be fixed or need to wait for something (eg the next go release). + +***FIXME*** I don't think I've quite got the milestone thing sorted yet. I was wondering about classifying them into priority, or what? + +Tickets [with no milestone](https://github.com/ncw/rclone/issues?utf8=✓&q=is%3Aissue%20is%3Aopen%20no%3Amile) are good candidates for ones that have slipped between the gaps and need following up. + +## Closing Tickets ## + +Close tickets as soon as you can - make sure they are tagged with a release. Post a link to a beta in the ticket with the fix in, asking for feedback. + +## Pull requests ## + +Try to process pull requests promptly! + +Merging pull requests on Github itself works quite well now-a-days so you can squash and rebase or rebase pull requests. rclone doesn't use merge commits. + +Sometimes pull requests need to be left open for a while - this especially true of contributions of new backends which take a long time to get right. + +## Merges ## + +If you are merging a branch locally then do `git merge --ff-only branch-name` to avoid a merge commit. You'll need to rebase the branch if it doesn't merge cleanly. + +## Release cycle ## + +Rclone aims for a 6-8 week release cycle. Sometimes release cycles take longer if there is something big to merge that didn't stabilize properly or for personal reasons. + +High impact regressions should be fixed before the next release. + +Near the start of the release cycle the dependencies should be updated with `make update` to give time for bugs to surface. + +Towards the end of the release cycle try not to merge anything too big so let things settle down. + +Follow the instructions in RELEASE.md for making the release. Note that the testing part is the most time consuming often needing several rounds of test and fix depending on exactly how many new features rclone has gained. + +## TODO ## + +I should probably make a mailing list for maintainers or at least an rclone-dev list, and I should probably make a dev@rclone.org to register with cloud providers. diff --git a/MANUAL.html b/MANUAL.html index f07d0007b..fd1b9026e 100644 --- a/MANUAL.html +++ b/MANUAL.html @@ -12,7 +12,7 @@

Rclone

Logo

@@ -30,6 +30,7 @@
  • Yandex Disk
  • SFTP
  • FTP
    • +
  • HTTP
  • The local filesystem

    • Features

    @@ -135,12 +136,12 @@ sudo zypper install snapd

    OpenWrt

    Enable the snap-openwrt feed.

    Configure

    -

    First you'll need to configure rclone. As the object storage systems have quite complicated authentication these are kept in a config file. (See the --config entry for how to find the config file and choose its location.)

    +

    First, you'll need to configure rclone. As the object storage systems have quite complicated authentication these are kept in a config file. (See the --config entry for how to find the config file and choose its location.)

    The easiest way to make the config is to run rclone with the config option:

    rclone config

    See the following for detailed instructions for

    Usage

    @@ -387,7 +389,7 @@ if src is directory

    This doesn't transfer unchanged files, testing by size and modification time or MD5SUM. It doesn't delete files from the destination.

    rclone copyto source:path dest:path

    rclone cryptcheck

    -

    Cryptcheck checks the integritity of a crypted remote.

    +

    Cryptcheck checks the integrity of a crypted remote.

    Synopsis

    rclone cryptcheck checks a remote against a crypted remote. This is the equivalent of running rclone check, but able to check the checksums of the crypted remote.

    For it to work the underlying remote of the cryptedremote must support some kind of checksum.

    @@ -446,11 +448,13 @@ if src is directory

    rclone mount

    Mount the remote as a mountpoint. EXPERIMENTAL

    Synopsis

    -

    rclone mount allows Linux, FreeBSD and macOS to mount any of Rclone's cloud storage systems as a file system with FUSE.

    +

    rclone mount allows Linux, FreeBSD, macOS and Windows to mount any of Rclone's cloud storage systems as a file system with FUSE.

    This is EXPERIMENTAL - use with care.

    First set up your remote using rclone config. Check it works with rclone ls etc.

    Start the mount like this

    rclone mount remote:path/to/files /path/to/local/mount
    +

    Or on Windows like this where X: is an unused drive letter

    +
    rclone mount remote:path/to/files X:

    When the program ends, either via Ctrl+C or receiving a SIGINT or SIGTERM signal, the mount is automatically stopped.

    The umount operation can fail, for example when the mountpoint is busy. When that happens, it is the user's responsibility to stop the mount manually with

    # Linux
@@ -460,7 +464,7 @@ umount /path/to/local/mount

    Limitations

    This can only write files seqentially, it can only seek when reading. This means that many applications won't work with their files on an rclone mount.

    The bucket based remotes (eg Swift, S3, Google Compute Storage, B2, Hubic) won't work from the root - you will need to specify a bucket, or a path within the bucket. So swift: won't work whereas swift:bucket will as will swift:bucket/path. None of these support the concept of directories, so empty directories will have a tendency to disappear once they fall out of the directory cache.

    -

    Only supported on Linux, FreeBSD and OS X at the moment.

    +

    Only supported on Linux, FreeBSD, OS X and Windows at the moment.

    rclone mount vs rclone sync/copy

    File systems expect things to be 100% reliable, whereas cloud storage systems are a long way from 100% reliable. The rclone sync/copy commands cope with this with lots of retries. However rclone mount can't use retries in the same way without making local copies of the uploads. This might happen in the future, but for the moment rclone mount won't do that, so will be less reliable than the rclone command.

    Filters

    @@ -486,15 +490,17 @@ umount /path/to/local/mount --debug-fuse Debug the FUSE internals - needs -v. --default-permissions Makes kernel enforce access control based on the file mode. --dir-cache-time duration Time to cache directory entries for. (default 5m0s) + --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp. Repeat if required. --gid uint32 Override the gid field set by the filesystem. (default 502) --max-read-ahead int The number of bytes that can be prefetched for sequential reads. (default 128k) --no-checksum Don't compare checksums on up/download. --no-modtime Don't read/write the modification time (can speed things up). --no-seek Don't allow seeking in files. + -o, --option stringArray Option for libfuse/WinFsp. Repeat if required. --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) --read-only Mount read-only. --uid uint32 Override the uid field set by the filesystem. (default 502) - --umask int Override the permission bits set by the filesystem. (default 2) + --umask int Override the permission bits set by the filesystem. --write-back-cache Makes kernel buffer writes before sending them to rclone. Without this, writethrough caching is used.

    rclone moveto

    Move file or directory from source to dest.

    @@ -535,13 +541,13 @@ if src is directory

    Obscure password for use in the rclone.conf

    rclone obscure password

    rclone rmdirs

    -

    Remove any empty directoryies under the path.

    +

    Remove empty directories under the path.

    Synopsis

    This removes any empty directories (or directories that only contain empty directories) under the path that it finds, including the path if it has nothing in.

    This is useful for tidying up remotes that rclone has left a lot of empty directories in.

    rclone rmdirs remote:path

    Copying single files

    -

    rclone normally syncs or copies directories. However if the source remote points to a file, rclone will just copy that file. The destination remote must point to a directory - rclone will give the error Failed to create file system for "remote:file": is a file not a directory if it isn't.

    +

    rclone normally syncs or copies directories. However, if the source remote points to a file, rclone will just copy that file. The destination remote must point to a directory - rclone will give the error Failed to create file system for "remote:file": is a file not a directory if it isn't.

    For example, suppose you have a remote with a file in called test.jpg, then you could copy just that file like this

    rclone copy remote:test.jpg /tmp/download

    The file test.jpg will be placed inside /tmp/download.

    @@ -549,7 +555,7 @@ if src is directory 
    rclone copy --no-traverse --files-from /tmp/files remote: /tmp/download

    Where /tmp/files contains the single line

    test.jpg
    -

    It is recommended to use copy when copying single files not sync. They have pretty much the same effect but copy will use a lot less memory.

    +

    It is recommended to use copy when copying individual files, not sync. They have pretty much the same effect but copy will use a lot less memory.

    Quoting and the shell

    When you are typing commands to your computer you are using something called the command line shell. This interprets various characters in an OS specific way.

    Here are some gotchas which may help users unfamiliar with the shell rules

    @@ -564,6 +570,12 @@ if src is directory 
    rclone copy "E:\folder name\folder name\folder name" remote:backup

    If you are using the root directory on its own then don't quote it (see #464 for why), eg

    rclone copy E:\ remote:backup
    +

    Copying files or directories with : in the names

    +

    rclone uses : to mark a remote name. This is, however, a valid filename component in non-Windows OSes. The remote name parser will only search for a : up to the first / so if you need to act on a file or directory like this then use the full path starting with a /, or use ./ as a current directory prefix.

    +

    So to sync a directory called sync:me to a remote called remote: use

    +
    rclone sync ./sync:me remote:path
    +

    or

    +
    rclone sync /full/path/to/sync:me remote:path

    Server Side Copy

    Most remotes (but not all - see the overview) support server side copy.

    This means if you want to copy one folder to another then rclone won't download all the files and re-upload them; it will instruct the server to copy them in place.

    @@ -571,7 +583,7 @@ if src is directory 
    rclone copy s3:oldbucket s3:newbucket

    Will copy the contents of oldbucket to newbucket without downloading and re-uploading.

    Remotes which don't support server side copy will download and re-upload in this case.

    -

    Server side copies are used with sync and copy and will be identified in the log when using the -v flag. The may also be used with move if the remote doesn't support server side move.

    +

    Server side copies are used with sync and copy and will be identified in the log when using the -v flag. The move command may also use them if remote doesn't support server side move directly. This is done by issuing a server side copy then a delete which is much quicker than a download and re-upload.

    Server side copies will only be attempted if the remote names are the same.

    This can be used when scripting to make aged backups efficiently, eg

    rclone sync remote:current-backup remote:previous-backup
@@ -579,7 +591,7 @@ rclone sync /path/to/files remote:current-backup

    Options

    Rclone has a number of options to control its behaviour.

    Options which use TIME use the go time parser. A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

    -

    Options which use SIZE use kByte by default. However a suffix of b for bytes, k for kBytes, M for MBytes and G for GBytes may be used. These are the binary units, eg 1, 2**10, 2**20, 2**30 respectively.

    +

    Options which use SIZE use kByte by default. However, a suffix of b for bytes, k for kBytes, M for MBytes and G for GBytes may be used. These are the binary units, eg 1, 2**10, 2**20, 2**30 respectively.

    --backup-dir=DIR

    When using sync, copy or move any files which would have been overwritten or deleted are moved in their original hierarchy into this directory.

    If --suffix is set, then the moved files will have the suffix added to them. If there is a file with the same path (after the suffix has been added) in DIR, then it will be overwritten.

    @@ -591,18 +603,20 @@ rclone sync /path/to/files remote:current-backup

    --bwlimit=BANDWIDTH_SPEC

    This option controls the bandwidth limit. Limits can be specified in two ways: As a single limit, or as a timetable.

    Single limits last for the duration of the session. To use a single limit, specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. The default is 0 which means to not limit bandwidth.

    -

    For example to limit bandwidth usage to 10 MBytes/s use --bwlimit 10M

    -

    It is also possible to specify a "timetable" of limits, which will cause certain limits to be applied at certain times. To specify a timetable, format your entries as "HH:MM,BANDWIDTH HH:MM,BANDWITH...".

    +

    For example, to limit bandwidth usage to 10 MBytes/s use --bwlimit 10M

    +

    It is also possible to specify a "timetable" of limits, which will cause certain limits to be applied at certain times. To specify a timetable, format your entries as "HH:MM,BANDWIDTH HH:MM,BANDWIDTH...".

    An example of a typical timetable to avoid link saturation during daytime working hours could be:

    --bwlimit "08:00,512 12:00,10M 13:00,512 18:00,30M 23:00,off"

    In this example, the transfer bandwidth will be set to 512kBytes/sec at 8am. At noon, it will raise to 10Mbytes/s, and drop back to 512kBytes/sec at 1pm. At 6pm, the bandwidth limit will be set to 30MBytes/s, and at 11pm it will be completely disabled (full speed). Anything between 11pm and 8am will remain unlimited.

    Bandwidth limits only apply to the data transfer. They don't apply to the bandwidth of the directory listings etc.

    -

    Note that the units are Bytes/s not Bits/s. Typically connections are measured in Bits/s - to convert divide by 8. For example let's say you have a 10 Mbit/s connection and you wish rclone to use half of it - 5 Mbit/s. This is 5/8 = 0.625MByte/s so you would use a --bwlimit 0.625M parameter for rclone.

    +

    Note that the units are Bytes/s, not Bits/s. Typically connections are measured in Bits/s - to convert divide by 8. For example, let's say you have a 10 Mbit/s connection and you wish rclone to use half of it - 5 Mbit/s. This is 5/8 = 0.625MByte/s so you would use a --bwlimit 0.625M parameter for rclone.

    +

    On Unix systems (Linux, MacOS, …) the bandwidth limiter can be toggled by sending a SIGUSR2 signal to rclone. This allows to remove the limitations of a long running rclone transfer and to restore it back to the value specified with --bwlimit quickly when needed. Assuming there is only one rclone instance running, you can toggle the limiter like this:

    +
    kill -SIGUSR2 $(pidof rclone)

    --buffer-size=SIZE

    Use this sized buffer to speed up file transfers. Each --transfer will use this much memory for buffering.

    -

    Set to 0 to disable the buffering for the minimum memory use.

    +

    Set to 0 to disable the buffering for the minimum memory usage.

    --checkers=N

    -

    The number of checkers to run in parallel. Checkers do the equality checking of files during a sync. For some storage systems (eg s3, swift, dropbox) this can take a significant amount of time so they are run in parallel.

    +

    The number of checkers to run in parallel. Checkers do the equality checking of files during a sync. For some storage systems (eg S3, Swift, Dropbox) this can take a significant amount of time so they are run in parallel.

    The default is to run 8 checkers in parallel.

    -c, --checksum

    Normally rclone will look at modification time and size of files to see if they are equal. If you set this flag then rclone will check the file hash and size to determine if files are equal.

    @@ -631,22 +645,22 @@ rclone sync /path/to/files remote:current-backup

    --ignore-size

    Normally rclone will look at modification time and size of files to see if they are equal. If you set this flag then rclone will check only the modification time. If --checksum is set then it only checks the checksum.

    It will also cause rclone to skip verifying the sizes are the same after transfer.

    -

    This can be useful for transferring files to and from onedrive which occasionally misreports the size of image files (see #399 for more info).

    +

    This can be useful for transferring files to and from OneDrive which occasionally misreports the size of image files (see #399 for more info).

    -I, --ignore-times

    Using this option will cause rclone to unconditionally upload all files regardless of the state of files on the destination.

    Normally rclone would skip any files that have the same modification time and are the same size (or have the same checksum if using --checksum).

    --log-file=FILE

    -

    Log all of rclone's output to FILE. This is not active by default. This can be useful for tracking down problems with syncs in combination with the -v flag. See the Logging section for more info.

    +

    Log all of rclone's output to FILE. This is not active by default. This can be useful for tracking down problems with syncs in combination with the -v flag. See the Logging section for more info.

    --log-level LEVEL

    This sets the log level for rclone. The default log level is INFO.

    DEBUG is equivalent to -vv. It outputs lots of debug info - useful for bug reports and really finding out what rclone is doing.

    INFO is equivalent to -v. It outputs information about each transfer and prints stats once a minute by default.

    NOTICE is the default log level if no logging flags are supplied. It outputs very little when things are working normally. It outputs warnings and significant events.

    -

    ERROR is equivalent to -q. It only output error messages.

    +

    ERROR is equivalent to -q. It only outputs error messages.

    --low-level-retries NUMBER

    This controls the number of low level retries rclone does.

    A low level retry is used to retry a failing operation - typically one HTTP request. This might be uploading a chunk of a big file for example. You will see low level retries in the log with the -v flag.

    -

    This shouldn't need to be changed from the default in normal operations, however if you get a lot of low level retries you may wish to reduce the value so rclone moves on to a high level retry (see the --retries flag) quicker.

    +

    This shouldn't need to be changed from the default in normal operations. However, if you get a lot of low level retries you may wish to reduce the value so rclone moves on to a high level retry (see the --retries flag) quicker.

    Disable low level retries with --low-level-retries 1.

    --max-depth=N

    This modifies the recursion depth for all the commands except purge.

    @@ -668,18 +682,21 @@ rclone sync /path/to/files remote:current-backup

    Normally rclone outputs stats and a completion message. If you set this flag it will make as little output as possible.

    --retries int

    Retry the entire sync if it fails this many times it fails (default 3).

    -

    Some remotes can be unreliable and a few retries helps pick up the files which didn't get transferred because of errors.

    +

    Some remotes can be unreliable and a few retries help pick up the files which didn't get transferred because of errors.

    Disable retries with --retries 1.

    --size-only

    Normally rclone will look at modification time and size of files to see if they are equal. If you set this flag then rclone will check only the size.

    -

    This can be useful transferring files from dropbox which have been modified by the desktop sync client which doesn't set checksums of modification times in the same way as rclone.

    +

    This can be useful transferring files from Dropbox which have been modified by the desktop sync client which doesn't set checksums of modification times in the same way as rclone.

    --stats=TIME

    Commands which transfer data (sync, copy, copyto, move, moveto) will print data transfer stats at regular intervals to show their progress.

    This sets the interval.

    The default is 1m. Use 0 to disable.

    -

    If you set the stats interval then all command can show stats. This can be useful when running other commands, check or mount for example.

    +

    If you set the stats interval then all commands can show stats. This can be useful when running other commands, check or mount for example.

    +

    Stats are logged at INFO level by default which means they won't show at default log level NOTICE. Use --stats-log-level NOTICE or -v to make them show. See the Logging section for more info on log levels.

    +

    --stats-log-level string

    +

    Log level to show --stats output at. This can be DEBUG, INFO, NOTICE, or ERROR. The default is INFO. This means at the default level of logging which is NOTICE the stats won't show - if you want them to then use -stats-log-level NOTICE. See the Logging section for more info on log levels.

    --stats-unit=bits|bytes

    -

    By default data transfer rates will be printed in bytes/second.

    +

    By default, data transfer rates will be printed in bytes/second.

    This option allows the data rate to be printed in bits/second.

    Data transfer volume will still be reported in bytes.

    The rate is reported as a binary unit, not SI unit. So 1 Mbit/s equals 1,048,576 bits/s and not 1,000,000 bits/s.

    @@ -689,11 +706,22 @@ rclone sync /path/to/files remote:current-backup

    See --backup-dir for more info.

    --syslog

    On capable OSes (not Windows or Plan9) send all log output to syslog.

    -

    This can be useful for running rclone in script or rclone mount.

    +

    This can be useful for running rclone in a script or rclone mount.

    --syslog-facility string

    If using --syslog this sets the syslog facility (eg KERN, USER). See man syslog for a list of possible facilities. The default facility is DAEMON.

    +

    --tpslimit float

    +

    Limit HTTP transactions per second to this. Default is 0 which is used to mean unlimited transactions per second.

    +

    For example to limit rclone to 10 HTTP transactions per second use --tpslimit 10, or to 1 transaction every 2 seconds use --tpslimit 0.5.

    +

    Use this when the number of transactions per second from rclone is causing a problem with the cloud storage provider (eg getting you banned or rate limited).

    +

    This can be very useful for rclone mount to control the behaviour of applications using it.

    +

    See also --tpslimit-burst.

    +

    --tpslimit-burst int

    +

    Max burst of transactions for --tpslimit. (default 1)

    +

    Normally --tpslimit will do exactly the number of transaction per second specified. However if you supply --tps-burst then rclone can save up some transactions from when it was idle giving a burst of up to the parameter supplied.

    +

    For example if you provide --tpslimit-burst 10 then if rclone has been idle for more than 10*--tpslimit then it can do 10 transactions very quickly before they are limited again.

    +

    This may be used to increase performance of --tpslimit without changing the long term average number of transactions per second.

    --track-renames

    -

    By default rclone doesn't not keep track of renamed files, so if you rename a file locally then sync it to a remote, rclone will delete the old file on the remote and upload a new copy.

    +

    By default, rclone doesn't keep track of renamed files, so if you rename a file locally then sync it to a remote, rclone will delete the old file on the remote and upload a new copy.

    If you use this flag, and the remote supports server side copy or server side move, and the source and destination have a compatible hash, then this will track renames during sync, copy, and move operations and perform renaming server-side.

    Files will be matched by size and hash - if both match then a rename will be considered.

    If the destination does not support server-side copy or move, rclone will fall back to the default behaviour and log an error level message to the console.

    @@ -706,7 +734,7 @@ rclone sync /path/to/files remote:current-backup

    Specifying --delete-after (the default value) will delay deletion of files until all new/updated files have been successfully transferred. The files to be deleted are collected in the copy pass then deleted after the copy pass has completed successfully. The files to be deleted are held in memory so this mode may use more memory. This is the safest mode as it will only delete files if there have been no errors subsequent to that. If there have been errors before the deletions start then you will get the message not deleting files as there were IO errors.

    --fast-list

    When doing anything which involves a directory listing (eg sync, copy, ls - in fact nearly every command), rclone normally lists a directory and processes it before using more directory lists to process any subdirectories. This can be parallelised and works very quickly using the least amount of memory.

    -

    However some remotes have a way of listing all files beneath a directory in one (or a small number) of transactions. These tend to be the bucket based remotes (eg s3, b2, gcs, swift, hubic).

    +

    However, some remotes have a way of listing all files beneath a directory in one (or a small number) of transactions. These tend to be the bucket based remotes (eg S3, B2, GCS, Swift, Hubic).

    If you use the --fast-list flag then rclone will use this method for listing directories. This will have the following consequences for the listing: