413 Commits

Author	SHA1	Message	Date
Yury Stankevich	b569dc11a0	hdfs: support kerberos authentication #42	2021-01-27 18:16:58 +00:00
Nick Craig-Wood	92b9dabf3c	fstests: only test with ASCII uppercase for case insensitive tests ... Upper/lower case for multilingual is provider dependent so we don't go there in the tests.	2021-01-27 14:28:17 +00:00
Nick Craig-Wood	c8ab4f1d02	fstests: add a test for finding a directory in a case insensitive way #4830	2021-01-26 14:48:06 +00:00
Nick Craig-Wood	e776a1b122	fstests: don't run encoding tests on -short	2021-01-26 14:46:23 +00:00
buengese	45b57822d5	compress: improve testing	2021-01-18 21:42:58 +01:00
Nick Craig-Wood	feaacfd226	hdfs: correct username parameter in integration tests	2021-01-08 09:05:25 +00:00
Yury Stankevich	71edc75ca6	HDFS (Hadoop Distributed File System) implementation - #42 ... This includes an HDFS docker image to use with the integration tests. Co-authored-by: Ivan Andreev <ivandeex@gmail.com> Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>	2021-01-07 09:48:51 +00:00
buengese	66c3f2f31f	new backend: zoho workdrive - fixes #4533	2020-12-30 17:56:08 +00:00
Nick Craig-Wood	b81b6da3fc	fstest: add test for reading object in a case insensitive way #4830	2020-12-07 17:38:22 +00:00
Nick Craig-Wood	56ad6aac4d	test_all: remove duplicated config for filefabric backend	2020-12-07 17:38:22 +00:00
Nick Craig-Wood	bcbe393af3	sftp: implement Shutdown method	2020-11-27 17:35:01 +00:00
Nick Craig-Wood	dfadd98969	azureblob,memory,pcloud: fix setting of mime types ... Before this change the backend was reading the mime type of the destination object instead of the source object when uploading. This changes fixes the problem and introduces an integration test for it. See: https://forum.rclone.org/t/is-there-a-way-to-get-rclone-copy-to-preserve-metadata/20682/2	2020-11-27 14:40:05 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	979bb07c86	filefabric: Implement the Enterprise File Fabric backend ... Missing features - M-Stream support - Oauth-like flow (soon being changed to oauth)	2020-11-25 21:11:29 +00:00
Nick Craig-Wood	7078311a84	compress: add integration tests	2020-11-23 18:02:22 +00:00
Nick Craig-Wood	45e8bea8d0	testserver: Make Test{FTP,SFTP,Webdav}Rclone run the current rclone ... Before this change the tests were run against the previous stable rclone/rclone docker image. This unfortunately masked errors in the integration test server. This change uses the currently installed rclone to run "rclone serve ftp" etc. This is installed out of the current code by the integration test server so will make a better test.	2020-11-10 18:01:15 +00:00
Nick Craig-Wood	1fb6ad700f	accounting: add context.Context #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	e3fe31f7cb	fs: add context.Context to fs.GetModifyWindow #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	8b96933e58	fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d69b96a94c	test: Add context to mockfs.NewFs #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Adam Plánský	cd2c06f2a7	testserver: speed up seafile integration test ... Before every seafile integration test we run Seafile server in docker environment. We don't have to sleep for 60 seconds to have everything ready before running integration test. We can assume everything is ready when Seafile webserver returns status code 200. Seafile Dockerfile (https://github.com/haiwen/seafile-docker/blob/master/image/seafile/Dockerfile) runs scripts/start.py where is defined that before init_seafile_server() we always wait for mysql wait_for_mysql() (https://github.com/haiwen/seafile-docker/blob/master/scripts/start.py)	2020-11-03 16:31:39 +00:00
Josh Soref	e4a87f772f	docs: spelling: e.g. ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	bbe7eb35f1	docs: spelling: server-side ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Nick Craig-Wood	bed83b0b64	test: add ListRetries config parameter to integration tests ... Occasionally the b2 tests fail because the integration tests don't retry hard enough with their new setting of -list-retries 3. Override this setting to 5 for the b2 tests only.	2020-10-25 18:10:50 +00:00
Nick Craig-Wood	85d35ef03c	test: remove TestS3Ceph: and TestSwiftCeph: from integration tests ... Unfortunately we don't have access to this server any more	2020-10-25 18:10:49 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Ivan Andreev	c797494d88	Merge pull request #4608 from ivandeex/pr-chunker-crypt ... chunker: fix upload over crypt (fixes #4570)	2020-09-18 17:58:44 +03:00
Ivan Andreev	e2a57182be	mailru: re-enable fixed chunker tests ... This reverts commit 9d3d397f50.	2020-09-18 17:56:34 +03:00
Nick Craig-Wood	5bf53fe3ac	test_all: only run the backend tests for 1fichier ... Running all the tests for 1fichier takes too long due to the directory reading rate limiter. The backend tests do complete in a reasonable time (21 mins).	2020-09-01 16:13:47 +01:00
Nick Craig-Wood	e2816629d0	fstest: fix upwrapping tests for bucket based remotes ... TestIntegration/FsRmdirNotFound was failing on crypt wrapping a bucket based remote. This was spotted by the integration tests.	2020-09-01 15:43:41 +01:00
Nick Craig-Wood	9d3d397f50	test_all: disable chunker + mailru tests while mailru is broken #4376	2020-08-20 12:50:20 +01:00
Nick Craig-Wood	38e8415e77	test_all: remove Digital Ocean s3 integration tests due to excessive rate limiting ... This is what I wrote to Digital Ocean support on July 10, 2020 - alas it didn't result in the rate limits dropping, so reluctantly I'm going to remove DO from the integration tests since they never pass and have no hope of ever passing while this rate limit is in effect. ---- Somewhere towards the end of June 2020 or the start of July 2020 my integration tests between rclone ( https://rclone.org ) and Digital Ocean started failing. I tried moving the tests to different regions (currently they are using AMS1 because I'm in Europe) with no improvement. Rclone seems to be hitting this rate limit as documented here: https://www.digitalocean.com/docs/spaces/#limits - 2 COPYs per 5 minutes on any individual object in a Space Rclone creates small objects about 100 bytes in size and renames them a few times - this involves using the COPY call as S3 does not have a rename API. The tests do this more than twice per object so hit the 5 minute timeout I think. Rclone does exponential backoff and fails after 10 retries not having reached 5 minutes delay after 10 retries. Having a 5 minute lockout on an S3 compatible API is surprising! Rclone integration tests with about 30 other providers, none of which have a rate limit like this. I understand the need for a COPY rate limit as server side copying large files can be resource intensive. However a 5 minute lockout for copying 100 byte files seems excessive! Might I humbly suggest that you reduce or eliminate this rate limit for small files? ---- This was the reply Unfortunately it is not possible to raise this limit or remove it currently on our platform. I do see how this would interfere with type of applications that need to copy many small files and will be happy to take the feedback to our engineering team to see how we can improve the spaces system in the future	2020-08-20 12:50:04 +01:00
Nick Craig-Wood	fb9edbe34e	test_all: export more internal variables to index.json for analysis	2020-08-20 12:23:33 +01:00
Nick Craig-Wood	63e4d2952b	fstests: Suggest that Purge on a nonexistent dir should return fs.ErrorDirNotFound	2020-08-19 18:03:42 +01:00
Nick Craig-Wood	a2afa9aadd	fs: Add directory to optional Purge interface - fixes #1891 ... - add a directory to the optional Purge interface - fix up all the backends - add an additional integration test to test for the feature - use the new feature in operations.Purge Many of the backends had been prepared in advance for this so the change was trivial for them.	2020-07-31 17:43:17 +01:00
Nick Craig-Wood	25662b9e05	fstest: add ability for mock objects and filesystems to have hashes	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	d8eea0e397	build: run gofmt -s to simplify the code: suggested by Go Report Card	2020-06-18 18:45:39 +01:00
Nick Craig-Wood	4b4ee72796	fstest: fix PublicLink tests to send non zero expiry and work with s3	2020-06-18 17:50:12 +01:00
Nick Craig-Wood	ba7fbfa8a7	testy: test utility functions	2020-06-15 21:34:37 +01:00
Roman Kredentser	55ad1354b6	link: Add --expire and --unlink flags ... This adds expire and unlink fields to the PublicLink interface. This fixes up the affected backends and removes unlink parameters where they are present.	2020-06-05 14:51:05 +01:00
Nick Craig-Wood	baaec5b126	testserver: don't allow starting of an empty remote	2020-05-25 15:26:06 +01:00
Martin Michlmayr	4aee962233	doc: fix typos throughout docs and code	2020-05-20 15:54:51 +01:00
Caleb Case	0ce662faad	Tardigrade Backend	2020-05-12 15:56:50 +00:00
Nick Craig-Wood	bd4b91bd57	test_all: revert running fichier tests one at a time ... This didn't achieve the objective of getting the tests to run clean and makes them take 16 hours! This reverts commit 97f6f8fe19.	2020-05-11 08:51:33 +01:00
Nick Craig-Wood	97f6f8fe19	test_all: run fichier tests one at a time ... This is in attempt to get the tests to run cleanly without hitting the rate limiter.	2020-05-07 20:39:54 +01:00
Nick Craig-Wood	94920d39ae	test_all: increase the test timeout to 60m from 30m ... Some tests are failing at 30m - 60m doesn't seem unreasonable	2020-05-07 20:38:40 +01:00
Nick Craig-Wood	9403bd2990	test_all: allow -list-retries to be overridden on the command line	2020-05-07 20:36:45 +01:00
Nick Craig-Wood	e098924e61	fstest: change -list-retries default to 3 from 6 ... The integration tests have got much better about retrying the tests, and we aren't testing ACD any more so we don't need it this high.	2020-05-07 20:28:41 +01:00
Fred	c754e89906	seafile: New backend for seafile server	2020-05-06 17:33:22 +00:00