112 Commits

Author	SHA1	Message	Date
albertony	f0396070eb	sftp: fix issue with WS_FTP by working around failing RealPath	2022-07-20 18:07:50 +01:00
albertony	fdd2f8e6d2	Error strings should not be capitalized ... Reported by staticcheck 2022.1.2 (v0.3.2) See: staticcheck.io	2022-06-23 23:26:02 +02:00
albertony	dbf1234edf	docs: skip "Connection" suffix from FTP, SSH/SFTP and HTTP backend names	2022-06-21 23:43:00 +02:00
Nick Craig-Wood	60d87185e1	sftp: add --sftp-set-env option to set environment variables ... Fixes #6094	2022-06-16 22:13:50 +01:00
Nick Craig-Wood	78120d40d9	sftp: add --sftp-concurrency to improve high latency transfers ... See: https://forum.rclone.org/t/increasing-sftp-transfer-speed/29928	2022-06-16 22:13:50 +01:00
Nick Craig-Wood	95e0934755	sftp: add --sftp-chunk-size to control packets sizes for high latency links ... See: https://forum.rclone.org/t/increasing-sftp-transfer-speed/29928	2022-06-16 22:13:50 +01:00
albertony	5db9a2f831	sftp: use vendor-specific VFS statistics extension for about if available ... See #5763	2022-06-08 21:14:53 +02:00
albertony	b4091f282a	sftp: add support for about and hashsum on windows server ... Windows shells like cmd and powershell needs to use different quoting/escaping of strings and paths than the unix shell, and also absolute paths must be fixed by removing leading slash that the POSIX formatted paths have (e.g. /C:/Users does not work in shell, it must be converted to C:/Users). Tries to autodetect shell type (cmd, powershell, unix) on first use. Implemented default builtin powershell functions for hashsum and about when remote shell is powershell. See #5763 Fixes #5758	2022-06-08 21:14:53 +02:00
Eng Zer Jun	4f0ddb60e7	refactor: replace strings.Replace with strings.ReplaceAll ... strings.ReplaceAll(s, old, new) is a wrapper function for strings.Replace(s, old, new, -1). But strings.ReplaceAll is more readable and removes the hardcoded -1. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	2022-05-17 11:08:37 +01:00
albertony	8697f0bd26	about: improved error message	2022-05-13 12:08:10 +01:00
albertony	843b77aaaa	docs/ftp: improved default value description of port and username options ... See #5551	2022-01-16 14:57:38 +01:00
Nick Craig-Wood	a351484997	sftp: fix timeout on hashing large files by sending keepalives ... Before this fix the SFTP sessions could timeout when doing hashes if they took longer than the --timeout parameter. This patch sends keepalive packets every minute while a shell command is running to keep the connection open. See: https://forum.rclone.org/t/rclone-check-over-sftp-failure-to-calculate-md5-hash-for-large-files/27487	2021-11-22 15:26:29 +00:00
Nick Craig-Wood	099eff8891	sftp: refactor so we only have one way of running remote commands ... This also returns errors from running ssh Hash commands which we didn't do before.	2021-11-22 15:26:29 +00:00
thomae	f9e54f96c3	docs/sftp: fix typo	2021-11-11 19:20:15 +01:00
albertony	00aafc957e	sftp: add rclone to list of supported md5sum/sha1sum commands to look for ... See #5781	2021-11-11 15:16:45 +01:00
Nick Craig-Wood	e43b5ce5e5	Remove github.com/pkg/errors and replace with std library version ... This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.	2021-11-07 11:53:30 +00:00
albertony	f589dbc077	docs: don't show fictional example values for options as multiple choice items ... See #5538	2021-10-20 22:56:19 +02:00
albertony	e2f47ecdeb	docs: punctuation cleanup ... See #5538	2021-10-20 22:56:19 +02:00
Nick Craig-Wood	96e14bf456	sftp: fix initialization bug introduced by fs.ErrorIsDir return ... 3fbaa4c0b0 backends: make NewObject return fs.ErrorIsDir if possible	2021-10-16 10:38:24 +01:00
Nick Craig-Wood	93d85015af	sftp: fix timeout when doing MD5SUM of large file ... Before this change we were timing out MD5SUMs after 1 minute because rclone was closing the SSH session when there were sessions still aftive. This change counts sessions active for all SSH sessions now (Upload, Download, Hashes and running commands). See: https://forum.rclone.org/t/while-rclone-copying-large-files-md5sum-failed-with-exit-status/26845/	2021-10-15 15:19:22 +01:00
Nick Craig-Wood	3fbaa4c0b0	backends: make NewObject return fs.ErrorIsDir if possible ... This changes the interface to NewObject so that if NewObject is called on a directory then it should return fs.ErrorIsDir if possible without doing any extra work, otherwise fs.ErrorObjectNotFound. Tested on integration test server with: go run integration-test.go -tests backend -run TestIntegration/FsMkdir/FsPutFiles/FsNewObjectDir -branch fix-stat -maxtries 1	2021-10-14 17:15:50 +01:00
Ivan Andreev	24de896df2	build: apply gofmt from golang 1.17	2021-09-09 20:43:59 +03:00
Nick Craig-Wood	6545755758	sftp: remove spurious error message on --sftp-disable-concurrent-reads	2021-07-31 11:04:45 +01:00
albertony	ce83228cb2	sftp: expand tilde and environment variables in configured known_hosts_file (#5322) ... Fixes #5220	2021-05-11 19:58:26 +02:00
Nick Craig-Wood	0537791d14	sftp: Fix performance regression by re-enabling concurrent writes #5197 ... Betweeen rclone v1.54 and v1.55 there was an approx 3x performance regression when transferring to distant SFTP servers (in particular rsync.net). This turned out to be due to the library github.com/pkg/sftp rclone uses. Concurrent writes used to be enabled in this library by default (for v1.12.0 as used in rclone v1.54) but they are no longer enabled (for v1.13.0 as used in rclone v1.55) for safety reasons and it is necessary to enable them specifically. The safety concerns are due to the uncertainty as to whether writes come in order and whether a half completed file might have holes in it. This isn't a problem for rclone since a) it doesn't restart uploads and b) it has a post-transfer checksum test. This change introduces a new flag `--sftp-disable-concurrent-writes` to control the feature which defaults to false, meaning that concurrent writes are enabled as in v1.54. However this isn't quite enough to fix the problem as the sftp library needs to be able to sniff the size of the stream from the reader passed in, so this also adds a `Size` interface to the reader to enable this. This involved a patch to the library. The library was reverted to v1.12.0 for v1.55.1 - this patch installs v1.13.0+master to fix the Size interface problem. See: https://github.com/pkg/sftp/issues/426	2021-04-26 09:24:28 +01:00
Nick Craig-Wood	7dc7c021db	sftp: fix Update ReadFrom failed: failed to send packet: EOF errors ... In a3fcadddc8 sftp: close idle connections after --sftp-idle-timeout (1m by default) Idle SFTP connections were closed after 1 minute. However due to the way SSH multiplexes connections over a single SSH connection this meant that if uploads or downloads went on for more than one minute they failed with "EOF errors" as their underlying connection was closed. This fixes the problem by not clearing idle connections if there are any transfers in progress. Fixes #5197	2021-04-06 10:01:49 +01:00
Nick Craig-Wood	fe1aa13069	sftp: revert sftp library to v1.12.0 from v1.13.0 to fix performance regression #5197 ... This reverts the library update done in this commit. 713f8f357d sftp: fix "file not found" errors for read once servers Reverting this commit triples the performance to a far away sftp server. See: https://github.com/pkg/sftp/issues/426	2021-04-06 10:01:49 +01:00
Nick Craig-Wood	9ee3ad70e9	sftp: fix SetModTime stat failed: object not found with --sftp-set-modtime=false ... Some sftp servers don't allow the user to access the file after upload. In this case the error message indicates that using --sftp-set-modtime=false would fix the problem. However it doesn't because SetModTime does a stat call which can't be disabled. Update SetModTime failed: SetModTime stat failed: object not found After upload this patch checks for an `object not found` error if set_modtime == false and ignores it, returning the expected size of the object instead. It also makes SetModTime do nothing if set_modtime = false https://forum.rclone.org/t/sftp-update-setmodtime-failed/22873	2021-03-18 16:31:51 +00:00
Nick Craig-Wood	713f8f357d	sftp: fix "file not found" errors for read once servers - fixes #5077 ... It introduces a new flag --sftp-disable-concurrent-reads to stop the problematic behaviour in the SFTP library for read-once servers. This upgrades the sftp library to v1.13.0 which has the fix.	2021-03-13 15:38:38 +00:00
Nick Craig-Wood	a3fcadddc8	sftp: close idle connections after --sftp-idle-timeout (1m by default) ... This fixes a problem where sftp backends live on forever when using the rc and use more and more connections. Fixes #4883	2021-02-16 12:39:05 +00:00
Nick Craig-Wood	e6a9f005d6	sftp: implement --sftp-use-fstat ... See: https://forum.rclone.org/t/sftp-fails-to-sync-to-local-failed-to-copy-file-does-not-exist/21759	2021-01-28 16:07:26 +00:00
Nick Craig-Wood	e775328523	ftp,sftp: Make --tpslimit apply - fixes #4906	2021-01-08 10:29:57 +00:00
Nick Craig-Wood	ba51409c3c	sftp: implement keyboard interactive authentication - fixes #4177 ... Some ssh servers are set up with keyboard interactive authentication which previously the sftp backkend was ignoring.	2020-12-29 19:48:09 +00:00
Nick Craig-Wood	bcbe393af3	sftp: implement Shutdown method	2020-11-27 17:35:01 +00:00
Nick Craig-Wood	9d574c0d63	fshttp: read config from ctx not passed in ConfigInfo #4685	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	8b96933e58	fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Josh Soref	3e1cb8302a	docs: spelling: etc. ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	bbe7eb35f1	docs: spelling: server-side ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Stephen Harris	bbddadbd04	sftp: remember entered password in AskPass mode ... As reported in https://github.com/rclone/rclone/issues/4660#issuecomment-705502792 After switching to a password callback function, if the ssh connection aborts and needs to be reconnected then the user is-reprompted for their password. Instead we now remember the password they entered and just give that back. We do lose the ability for them to correct mistakes, but that's the situation from before switching to callbacks. We keep the benefits of not asking for passwords until the SSH connection succeeds (right known_hosts entry, for example). This required a small refactor of how `f := &Fs{}` was built, so we can store the saved password in the Fs object	2020-10-13 16:53:11 +01:00
Stephen Harris	9e925becb6	sftp: defer asking for user passwords until the SSH connection succeeds ... Issue: 4660 https://github.com/rclone/rclone/issues/4660 Unexpected side effect: a wrong password allows for the user to retry!	2020-10-07 12:01:17 +01:00
Stephen Harris	6dc28ef50a	sftp: Allow user to optionally check server hosts key to add security ... Based on Issue 4087 https://github.com/rclone/rclone/issues/4087 Current behaviour is insecure. If the user specifies this value then we switch to validating the server hostkey and so can detect server changes or MITM-type attacks.	2020-10-06 16:27:42 +01:00
Nick Craig-Wood	41ec712aa9	ftp,sftp: fix docs for usernames ... - factor env.CurrentUser out of backend/sftp - Use env.CurrentUser in ftp and sftp - fix docs to have correct username	2020-09-27 11:44:05 +01:00
Stephen Harris	17acae2b00	sftp: allow cert based auth via optional pubkey ... Discussion at https://forum.rclone.org/t/ssh-certificate-based-authentication-does-not-work/19222 Basically we allow the user to specify their own public key cert rather than letting the SSH client extract the pubkey from the private key. This allows certificate based authentication to work.	2020-09-27 11:10:13 +01:00
buengese	60cc2cba1f	sftp: always convert the checksum to lower case - fixes #4518	2020-09-22 03:15:09 +02:00
aus	b6d3cad70e	sftp: add options for subsystem and server_command - fixes #1801	2020-08-25 21:38:13 +01:00
Nick Craig-Wood	fefcbf60fa	sftp: use the absolute path instead of the relative path ... Before this change rclone used the relative path from the current working directory. It appears that WS FTP doesn't like this and the openssh sftp tool also uses absolute paths which is a good reason for switching to absolute paths. This change reads the current working directory at startup and bases all file requests from there. See: https://forum.rclone.org/t/sftp-ssh-fx-failure-directory-not-found/17436	2020-06-30 16:07:23 +01:00
Nick Craig-Wood	c820576329	fs: define SlowModTime and SlowHash features in the relevant backends	2020-06-30 12:01:36 +01:00