Commit Graph

139 Commits

Author SHA1 Message Date
albertony
5c6a958ad8 go mod tidy: github.com/pkg/xattr should be direct 2022-07-04 11:24:59 +02:00
albertony
1f9560e873 selfupdate: replace deprecated x/crypto/openpgp package with ProtonMail/go-crypto 2022-07-04 11:24:59 +02:00
Nick Craig-Wood
c556e98f49 local: add Metadata support #111 2022-06-29 14:29:36 +01:00
eNV25
36add0afbf
ncdu: replace termbox with tcell's termbox wrapper
The https://github.com/nsf/termbox-go library is no longer maintained
so this change replaces it with the maintained
github.com/gdamore/tcell library which has a termbox backwards
compatibility layer.

There are a few minor changes from the termbox library:

- Using Clear with fg bg ColorDefault resulted in a white background for some reason.
    - Clear with fg ColorWhite bg ColorBlack was used instead.
- tcell's termbox wrapper doesn't support ColorLightYellow.
    - ColorYellow + 8 was used instead.
2022-06-19 11:22:45 +01:00
Nick Craig-Wood
bb6edb3c39 build: update dependencies
Also:

- azureblob: fix compile after API change in upstream library
2022-06-08 18:29:42 +01:00
Nick Craig-Wood
26db80c270 ftp: revert to upstream github.com/jlaffaye/ftp from our fork
...now all of our patches have been merged #5810
2022-06-08 11:58:32 +01:00
Mateusz Puczyński
607172b6ec go: run go mod tidy and set version to minimum supported go - go1.16
This fixes `go mod tidy` needing manual intervention to build with
previous go versions.
2022-05-01 13:28:20 +01:00
Nick Craig-Wood
94757277bc build: fix go mod tidy removing golang.org/x/mobile dependency
Before this running `go mod tidy` caused the build to break because it
removed the dependency on golang.org/x/mobile and a command line tool
from this package is needed for the build.

This adds an explicit dependency which will mean the tool is always present.
2022-05-01 13:28:20 +01:00
Nick Craig-Wood
1b0128ecb2 cmount: update winfsp/cgofuse dependency for windows/arm64 build 2022-04-29 18:04:21 +01:00
Nick Craig-Wood
4c02f50ef5 build: update github.com/billziss-gh to github.com/winfsp 2022-04-13 10:18:26 +01:00
Nick Craig-Wood
285747b1d1 build: update to go1.18 and make go1.16 the minimum required version 2022-04-04 20:35:17 +01:00
Nick Craig-Wood
59c242bbf6 build: update dependencies
Also:

- dropbox: fix compile after API change in upstream library
2022-04-04 20:35:17 +01:00
KARBOWSKI Piotr
4f78226f8b sftp: Fix OpenSSH 8.8+ RSA keys incompatibility (#6076)
Updates golang.org/x/crypto to v0.0.0-20220331220935-ae2d96664a29.

Fixes the issues with connecting to OpenSSH 8.8+ remotes in case the
client uses RSA key pair due to OpenSSH dropping support for SHA1 based
ssh-rsa signature.

Bug: https://github.com/rclone/rclone/issues/6076
Bug: https://github.com/golang/go/issues/37278
Signed-off-by: KARBOWSKI Piotr <piotr.karbowski@gmail.com>
2022-04-01 12:49:39 +01:00
Tobias Klauser
54c9c3156c fs/config, lib/terminal: use golang.org/x/term
golang.org/x/crypto/ssh/terminal is deprecated in favor of
golang.org/x/term, see https://pkg.go.dev/golang.org/x/crypto/ssh/terminal

The latter also supports ReadPassword on solaris, so enable the
respective functionality in fs/config for solaris as well.
2022-04-01 12:48:18 +01:00
Nick Craig-Wood
f9354fff2f Version v1.58.0 2022-03-18 12:29:54 +00:00
Nick Craig-Wood
847868b4ba ftp: hard fork github.com/jlaffaye/ftp to fix go get
Having a replace directive in go.mod causes "go get
github.com/rclone/rclone" to fail as it discussed in this Go issue:
https://github.com/golang/go/issues/44840

This is apparently how the Go team want go.mod to work, so this commit
hard forks github.com/jlaffaye/ftp into github.com/rclone/ftp so we
can remove the `replace` directive from the go.mod file.

Fixes #5810
2022-03-07 09:55:49 +00:00
Nick Craig-Wood
251b84ff2c sftp: fix unecessary seeking when uploading and downloading files
This stops the SFTP library issuing out of order writes which fixes
the problems uploading to `serve sftp` from the `sftp` backend.

This was fixes upstream in this pull request: https://github.com/pkg/sftp/pull/482

Fixes #5806
2022-03-03 17:02:35 +00:00
Max Sum
c217145cae crypt: add base64 and base32768 filename encoding options #5801 2021-11-15 17:57:02 +00:00
Nick Craig-Wood
4198763c35 build: update all dependencies 2021-11-10 10:34:56 +00:00
Nick Craig-Wood
3de47b8ed4 build: upgrade go.mod file to go1.17 2021-11-10 10:34:56 +00:00
Nick Craig-Wood
e43b5ce5e5 Remove github.com/pkg/errors and replace with std library version
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.

This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
2021-11-07 11:53:30 +00:00
Ivan Andreev
6210e22ab5 bisync: implementation #5164
Fixes #118

Co-authored-by: Chris Nelson <stuff@cjnaz.com>
2021-11-01 21:00:27 +03:00
Ivan Andreev
1d50336615 ftp: replace jlaffaye/ftp by rclone/ftp in the build #5596 2021-10-21 14:18:23 +03:00
Rajat Goel
db6002952e dropbox: upgrade sdk version 2021-10-16 10:55:02 +01:00
Nick Craig-Wood
8b4a89d34b Update github.com/ncw/swift to v2.0.1 2021-10-07 12:02:09 +01:00
Ivan Andreev
8b8a943dd8 mountlib: correctly daemonize for compatibility with automount - #5593
This patch will:
- add --daemon-wait flag to control the time to wait for background mount
- remove dependency on sevlyar/go-daemon and implement backgrounding directly
- avoid setsid during backgrounding as it can result in race under Automount
- provide a fallback PATH to correctly run `fusermount` under systemd as it
  runs mount units without standard environment variables
- correctly handle ^C pressed while background process is being setting up
2021-10-02 23:45:16 +03:00
Ivan Andreev
8c10dee510 mountlib: use procfs to validate mount on linux - #5593
Current way of checking whether mountpoint has been already mounted (directory
list) can result in race if rclone runs under Automount (classic or systemd).

This patch adopts Linux ProcFS for the check. Note that mountpoint is considered
empty if it's tagged as "mounted" by autofs. Also ProcFS is used to check whether
rclone mount was successful (ie. tagged by a string containing "rclone").

On macOS/BSD where ProcFS is unavailable the old method is still used.

This patch also moves a few utility functions unchanged to utils.go:
CheckOverlap, CheckAllowings, SetVolumeName.
2021-10-02 23:45:16 +03:00
Ivan Andreev
aa0ceb6c5c cmd/version: add support for openbsd/386
After this patch the version command will be fully
supported on openbsd/amd64 and openbsd/386.
Remaining os/arch combinations stay as is.
2021-09-02 11:13:12 +03:00
Nick Craig-Wood
09fd258b5c build: update all dependencies 2021-08-20 22:03:38 +01:00
Nick Craig-Wood
8bd26c663a build: update golang.org/x/sys for go1.17 build 2021-08-20 19:04:29 +01:00
Ivan Andreev
e45c23ab79 cmd/version: add support for openbsd/amd64
After this patch the version command will be
- fully supported on openbsd/amd64
- stay stub on openbsd/i386 until we deprecate go 1.17
Remaining os/arch combinations stay as is.
2021-08-16 11:39:34 +03:00
Antoine GIRARD
daf449b5f2 cmd/serve: add serve docker command (#5415)
Fixes #4750

Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
2021-07-10 23:56:09 +03:00
Nolan Woods
95ee14bb2c feat: Add lib/http
lib/http provides an abstraction for a central http server that services can bind routes to
2021-04-28 22:54:15 +01:00
Nick Craig-Wood
0537791d14 sftp: Fix performance regression by re-enabling concurrent writes #5197
Betweeen rclone v1.54 and v1.55 there was an approx 3x performance
regression when transferring to distant SFTP servers (in particular
rsync.net).

This turned out to be due to the library github.com/pkg/sftp rclone
uses. Concurrent writes used to be enabled in this library by default
(for v1.12.0 as used in rclone v1.54) but they are no longer enabled
(for v1.13.0 as used in rclone v1.55) for safety reasons and it is
necessary to enable them specifically.

The safety concerns are due to the uncertainty as to whether writes
come in order and whether a half completed file might have holes in
it. This isn't a problem for rclone since a) it doesn't restart
uploads and b) it has a post-transfer checksum test.

This change introduces a new flag `--sftp-disable-concurrent-writes`
to control the feature which defaults to false, meaning that
concurrent writes are enabled as in v1.54.

However this isn't quite enough to fix the problem as the sftp library
needs to be able to sniff the size of the stream from the reader
passed in, so this also adds a `Size` interface to the reader to
enable this. This involved a patch to the library.

The library was reverted to v1.12.0 for v1.55.1 - this patch installs
v1.13.0+master to fix the Size interface problem.

See: https://github.com/pkg/sftp/issues/426
2021-04-26 09:24:28 +01:00
Nick Craig-Wood
ee502a757f ncdu: update termbox-go library to fix crash - fixes #5259 2021-04-24 15:17:14 +01:00
Nick Craig-Wood
7afb4487ef build: update all dependencies 2021-04-20 00:00:13 +01:00
Caleb Case
baa4c039a0 backend/tardigrade: Upgrade to uplink v1.4.6
Release notes: https://github.com/storj/uplink/releases/tag/v1.4.6

Follow up PRs will take advantage of the new bucket error and negative
offset support to remove roundtrips.
2021-04-19 16:14:56 +01:00
Nick Craig-Wood
fe1aa13069 sftp: revert sftp library to v1.12.0 from v1.13.0 to fix performance regression #5197
This reverts the library update done in this commit.

713f8f357d sftp: fix "file not found" errors for read once servers

Reverting this commit triples the performance to a far away sftp server.

See: https://github.com/pkg/sftp/issues/426
2021-04-06 10:01:49 +01:00
Ivan Andreev
3859fe2f52
cmd/version: print os/version, kernel and bitness (#5204)
Related to #5121

Note: OpenBSD is stub yet. This will be fixed after upstream PR gets resolved
https://github.com/shirou/gopsutil/pull/993
2021-04-05 21:53:09 +03:00
Nick Craig-Wood
f6dbb98a1d cmount: update cgofuse to the latest version to bring in macfuse 4 fix 2021-03-25 09:02:17 +00:00
nguyenhuuluan434
d6870473a1 swift: implement copying large objects 2021-03-24 08:56:39 +00:00
Nick Craig-Wood
713f8f357d sftp: fix "file not found" errors for read once servers - fixes #5077
It introduces a new flag --sftp-disable-concurrent-reads to stop the
problematic behaviour in the SFTP library for read-once servers.

This upgrades the sftp library to v1.13.0 which has the fix.
2021-03-13 15:38:38 +00:00
Ivan Andreev
75056dc9b2
ftp: update dependency jlaffaye/ftp (#5097) 2021-03-05 15:58:04 +03:00
Robert Thomas
e5aa92c922
dropbox: add polling support - fixes #2949
This implements polling support for the Dropbox backend. The Dropbox SDK dependency had to be updated due to an auth issue, which was fixed on Jan 12 2021. A secondary internal Dropbox service was created to handle unauthorized SDK requests, as is necessary when using the ListFolderLongpoll function/endpoint. The config variable was renamed to cfg to avoid potential conflicts with the imported config package.
2021-02-24 09:33:31 +00:00
Rauno Ots
9808a53416 webdav: add support for sharepoint with NTLM authentication (#2921)
Add new option option "sharepoint-ntlm" for the vendor setting.
Use it when your hosted Sharepoint is not tied to the OneDrive
accounts and uses NTLM authentication.
Also add documentation and integration test.

Fixes: #2171
2021-02-23 23:27:30 +00:00
Klaus Post
424aaac2e1 encoder/filename: Add SCSU as tables
Instead of only adding SCSU, add it as an existing table.

Allow direct SCSU and add a, perhaps, reasonable table as well.

Add byte interfaces that doesn't base64 encode the URL as well with `EncodeBytes` and `DecodeBytes`.

Fuzz tested and decode tests added.
2021-02-12 11:39:39 +00:00
Nick Craig-Wood
993146375e build: update all dependencies 2021-02-03 21:34:38 +00:00
Nick Craig-Wood
bbe791a886 swift: update github.com/ncw/swift to v2.0.0
The update to v2 of the swift library introduces a context parameter
to each function. This required a lot of mostly mechanical changes
adding context parameters.

See: https://github.com/ncw/swift/issues/159
See: https://github.com/ncw/swift/issues/161
2021-02-03 20:23:37 +00:00
buengese
d3b7f14b66 bump sgzip to v1.1.0 - fixes #4970 2021-02-01 18:34:42 +00:00
Nick Craig-Wood
189ef5f257 azureblob: fix memory usage by upgrading the SDK and implementing a TransferManager
In the Azure SDK there was a bug which caused excessive memory use
when doing repeated transfers:

https://github.com/Azure/azure-storage-blob-go/issues/233

This patch updates the SDK to v0.13.0 which allowed us to implement a
custom TransferManager which integrates with rclone's memory pool.

This fixes the excessive memory consumption.

See: https://forum.rclone.org/t/ask-for-settings-recommendation-for-azureblob/21505/
2021-01-30 17:26:59 +00:00