Commit Graph

3200 Commits

Author SHA1 Message Date
Nick Craig-Wood
5a941cdcdc local: fix preallocate warning on Linux with ZFS
Under Linux, rclone attempts to preallocate files for efficiency.

Before this change, pre-allocation would fail on ZFS with the error

    Failed to pre-allocate: operation not supported

After this change rclone tries a different flag combination for ZFS
then disables pre-allocate if that doesn't work.

Fixes #3066
2019-06-03 18:02:26 +01:00
Nick Craig-Wood
62681e45fb Add Philip Harvey to contributors 2019-06-03 17:34:44 +01:00
Philip Harvey
1a2fb52266 s3: make SetModTime work for GLACIER while syncing - Fixes #3224
Before this change rclone would fail with

    Failed to set modification time: InvalidObjectState: Operation is not valid for the source object's storage class

when attempting to set the modification time of an object in GLACIER.

After this change rclone will re-upload the object as part of a sync if it needs to change the modification time.

See: https://forum.rclone.org/t/suspected-bug-in-s3-or-compatible-sync-logic-to-glacier/10187
2019-06-03 15:28:19 +01:00
Nick Craig-Wood
ec4e7316f2 docs: update advice for avoiding mega logins 2019-05-29 17:37:53 +01:00
buengese
11264c4fb8 jottacloud: update docs regarding devices and mountpoints 2019-05-29 17:16:42 +01:00
buengese
25f7f2b60a jottacloud: Add support for selecting device and mountpoint. fixes #3069 2019-05-29 17:16:42 +01:00
Nick Craig-Wood
e7c20e0bce operations: make move and copy individual files obey --backup-dir
Before this change, when using rclone copy or move with --backup-dir
and the source was a single file, rclone would fail to use the backup
directory.

This change looks up the backup directory in the Fs cache and uses it
as appropriate.

This affects any commands which call operations.MoveFile or
operations.CopyFile which includes rclone move/moveto/copy/copyto
where the source is a single file.

Fixes #3219
2019-05-27 16:14:55 +01:00
Nick Craig-Wood
8ee6034b23 Look for Fs in the cache rather than calling NewFs directly
This will save operations when rclone is used in remote control mode
or with the same remote multiple times in the command line.
2019-05-27 16:14:55 +01:00
Nick Craig-Wood
206e1caa99 fs/cache: factor Fs caching from fs/rc into its own package 2019-05-27 16:14:55 +01:00
Dan Walters
f0e439de0d dlna: improve logging and error handling
Mostly trying to get logging to happen through rclone's log methods.
Added request logging, and a trace parameter that will dump the
entire request/response for debugging when dealing with poorly
written clients.

Also added a flag to specify the device's "Friendly Name" explicitly,
and made an attempt at allowing mime types in addition to video.
2019-05-27 14:42:33 +01:00
Dan Walters
e5464a2a35 dlna: add some additional metadata, headers, and samsung extensions
Again, mostly just copying what I see in other implementations.  This
does seem to have done the trick so that I can now pause, fast forward,
rewind, etc., on my Samsung F series.
2019-05-27 14:42:33 +01:00
Dan Walters
78d38dda56 dlna: icons and compatibility improvements
Brings in icons for devices to display.  Based on what some
other open implementations have done, it's worth having a simple
stub implmentation of ConnectionManagerService.  Advertise
X_MS_MediaReceiverRegistrar as well, which sounds like it
is necessary for certain MSFT devices (like the X-Box.)
2019-05-27 14:42:33 +01:00
Dan Walters
60bb01b22c dlna: refactor the serve mux
Trying to make it a little easier to understand and work on all the
available routes, etc.
2019-05-27 14:42:33 +01:00
Dan Walters
95a74e02c7 dlna: use a template to render the root service descriptor
For various reasons, it seems to make sense to move away from generating
the XML with objects.  Namespace support is minimal in go, the objects we
have are in an upstream project, and some subtitlties seem likely to
cause problems with poorly written clients.

This removes the empty <iconList></iconList>, but is otherwise the
same output.
2019-05-27 14:42:33 +01:00
Dan Walters
d014aef011 dlna: reformat descriptors with tabs
Reduces size of embedded assets.
2019-05-27 14:42:33 +01:00
Dan Walters
be8c23f0b4 dlna: use vfsgen for static assets
As more assets are added, using vfsgen makes things a bit easier.
2019-05-27 14:42:33 +01:00
Nick Craig-Wood
da3b685cd8 vendor: update github.com/pkg/sftp to fix sftp client issues
See: https://forum.rclone.org/t/failed-to-copy-sftp-folder-not-found-c-ftpsites-ssh-fx-failure/9778
See: https://github.com/pkg/sftp/issues/288
2019-05-24 15:03:23 +01:00
Nick Craig-Wood
9aac2d6965 drive: add notes that cleanup works in the background on drive 2019-05-22 20:48:59 +01:00
Gary Kim
81fad0f0e3 docs: fix typo in cache documentation 2019-05-20 19:20:59 +01:00
Nick Craig-Wood
cff85f0b95 docs: add note on failure to login to mega docs 2019-05-16 10:10:58 +01:00
Nick Craig-Wood
9c0dac4ccd Add Robert Marko to contributors 2019-05-15 13:36:15 +01:00
Robert Marko
5ccc2dcb8f s3: add config info for Wasabi's EU Central endpoint
Wasabi has a EU Central endpoint for a couple months now, so add it to the list.

Signed-off-by: Robert Marko <robimarko@gmail.com>
2019-05-15 13:35:55 +01:00
Gary Kim
8c5503631a docs: add sftp about documentation 2019-05-14 15:22:43 +01:00
Nick Craig-Wood
2f3d794ec6 Add id01 to contributors 2019-05-14 07:55:38 +01:00
id01
abeb12c6df lib/env: Make env_test.go support Windows 2019-05-14 07:55:08 +01:00
Nick Craig-Wood
9c6f3ae82c local: log errors when listing instead of returning an error
Before this change, rclone would return an error from the listing if
there was an unreadable directory, or if there was a problem stat-ing
a directory entry.  This was frustrating because the command
completely aborts at that point when there is work it could do.

After this change rclone lists the directories and reports ERRORs for
unreadable directories or problems stat-ing files, but does return an
error from the listing.  It does set the error flag which means the
command will fail (and objects won't be deleted with `rclone sync`).

This brings rclone's behaviour exactly in to line with rsync's
behaviour.  It does as much as possible, but doesn't let the errors
pass silently.

Fixes #3179
2019-05-13 18:30:33 +01:00
Nick Craig-Wood
870b15313e cmd: log an ERROR for all commands which exit with non-zero status
Before this change, rclone didn't report errors for commands which
didn't return an error directly.  For example `rclone ls` could
encounter an error and rclone would log nothing, even though the exit
code was non zero.

After this change we always log a message if we are exiting with a
non-zero exit code.
2019-05-13 18:28:21 +01:00
Nick Craig-Wood
62a7e44e86 Add didil to contributors 2019-05-13 17:37:46 +01:00
didil
296e4936a0 install: linux skip man pages if no mandb - fixes #3175 2019-05-13 17:37:30 +01:00
Nick Craig-Wood
a0b9d4a239 docs: fix doc generators home directory in backend docs
Thanks @ctlaltdefeat for spotting this
2019-05-13 17:28:36 +01:00
Nick Craig-Wood
99bc013c0a crypt: remove stray debug in ChangeNotify 2019-05-12 16:50:03 +01:00
Nick Craig-Wood
d9cad9d10b mega: cleanup: add logs for -v and -vv 2019-05-12 10:46:21 +01:00
Nick Craig-Wood
0e23c4542f sync: fix integrations tests
2eb31a4f1d broke the integration tests for remotes which use
Copy+Delete as server side Move.
2019-05-12 09:50:20 +01:00
Nick Craig-Wood
f544234e26 gendocs: remove global flags from command help pages 2019-05-11 23:39:50 +01:00
Nick Craig-Wood
dbf9800cbc docs: add serve to README, main page and menu 2019-05-11 23:39:50 +01:00
Nick Craig-Wood
1f19b63264 serve sftp: serve an rclone remote over SFTP 2019-05-11 23:39:04 +01:00
Nick Craig-Wood
5c0e5b85f7 Factor ShellExpand from sftp backend to lib/env 2019-05-11 23:39:04 +01:00
Nick Craig-Wood
edda6d91cd Use go-homedir to read the home directory more reliably 2019-05-11 23:39:04 +01:00
Nick Craig-Wood
1fefa6adfd vendor: add github.com/mitchellh/go-homedir 2019-05-11 23:39:04 +01:00
Nick Craig-Wood
af030f74f5 vfs: make WriteAt for non cached files work with non-sequential writes
This makes WriteAt for non cached files wait a short time if it gets
an out of order write (which would normally cause an error) to see if
the gap will be filled with an in order write.

The makes the SFTP backend work fine with --vfs-cache-mode off
2019-05-11 23:39:04 +01:00
Nick Craig-Wood
ada8c22a97 sftp: send custom client version and debug server version 2019-05-11 23:39:04 +01:00
Nick Craig-Wood
610466c18c sftp: fix about parsing of df results so it can cope with -ve results
This is useful when interacting with "serve sftp" which returns -ve
results when the corresponding value is unknown.
2019-05-11 23:39:04 +01:00
Nick Craig-Wood
9950bb9b7c about: fix crash if backend returns a nil usage 2019-05-11 23:39:04 +01:00
Nick Craig-Wood
7d70e92664 operations: enable multi threaded downloads - Fixes #2252
This implements the --multi-thread-cutoff and --multi-thread-streams
flags to control multi thread downloading to the local backend.
2019-05-11 23:35:19 +01:00
Nick Craig-Wood
687cbf3ded operations: if --ignore-checksum is in effect, don't calculate checksum
Before this change we calculated the checksum which is potentially
time consuming and then ignored the result.  After the change we don't
calculate the checksum if we are about to ignore it.
2019-05-11 23:35:19 +01:00
Nick Craig-Wood
c3af0a1eca local: only calculate the required hashes for big speedup
Before this change we calculated all possible hashes for the file when
the `Hashes` method was called.

After we only calculate the Hash requested.

Almost all uses of `Hash` just need one checksum.  This will slow down
`rclone lsjson` with the `--hash` flag.  Perhaps lsjson should have a
`--hash-type` flag.

However it will speed up sync/copy/move/check/md5sum/sha1sum etc.

Before it took 12.4 seconds to md5sum a 1GB file, after it takes 3.1
seconds which is the same time the md5sum utility takes.
2019-05-11 23:35:19 +01:00
Nick Craig-Wood
822483aac5 accounting: enable accounting without passing through the stream #2252
This is in preparation for multithreaded downloads
2019-05-11 23:35:19 +01:00
Nick Craig-Wood
2eb31a4f1d sync: move Transferring into operations.Copy
This makes the code more consistent with the operations code setting
the transfer statistics up.
2019-05-11 23:35:19 +01:00
Nick Craig-Wood
0655738da6 operations: re-work reopen framework so it can take a RangeOption #2252
This is in preparation for multipart downloads.
2019-05-11 23:35:19 +01:00
Nick Craig-Wood
7c4fe3eb75 local: define OpenWriterAt interface and test and implement it #2252
This will enable multipart downloads in future commits
2019-05-11 23:35:19 +01:00