297 Commits

Author	SHA1	Message	Date
Nick Craig-Wood	1fb6ad700f	accounting: add context.Context #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	e3fe31f7cb	fs: add context.Context to fs.GetModifyWindow #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	8b96933e58	fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d69b96a94c	test: Add context to mockfs.NewFs #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Adam Plánský	cd2c06f2a7	testserver: speed up seafile integration test ... Before every seafile integration test we run Seafile server in docker environment. We don't have to sleep for 60 seconds to have everything ready before running integration test. We can assume everything is ready when Seafile webserver returns status code 200. Seafile Dockerfile (https://github.com/haiwen/seafile-docker/blob/master/image/seafile/Dockerfile) runs scripts/start.py where is defined that before init_seafile_server() we always wait for mysql wait_for_mysql() (https://github.com/haiwen/seafile-docker/blob/master/scripts/start.py)	2020-11-03 16:31:39 +00:00
Josh Soref	e4a87f772f	docs: spelling: e.g. ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	bbe7eb35f1	docs: spelling: server-side ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Nick Craig-Wood	bed83b0b64	test: add ListRetries config parameter to integration tests ... Occasionally the b2 tests fail because the integration tests don't retry hard enough with their new setting of -list-retries 3. Override this setting to 5 for the b2 tests only.	2020-10-25 18:10:50 +00:00
Nick Craig-Wood	85d35ef03c	test: remove TestS3Ceph: and TestSwiftCeph: from integration tests ... Unfortunately we don't have access to this server any more	2020-10-25 18:10:49 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Ivan Andreev	c797494d88	Merge pull request #4608 from ivandeex/pr-chunker-crypt ... chunker: fix upload over crypt (fixes #4570)	2020-09-18 17:58:44 +03:00
Ivan Andreev	e2a57182be	mailru: re-enable fixed chunker tests ... This reverts commit 9d3d397f50.	2020-09-18 17:56:34 +03:00
Nick Craig-Wood	5bf53fe3ac	test_all: only run the backend tests for 1fichier ... Running all the tests for 1fichier takes too long due to the directory reading rate limiter. The backend tests do complete in a reasonable time (21 mins).	2020-09-01 16:13:47 +01:00
Nick Craig-Wood	e2816629d0	fstest: fix upwrapping tests for bucket based remotes ... TestIntegration/FsRmdirNotFound was failing on crypt wrapping a bucket based remote. This was spotted by the integration tests.	2020-09-01 15:43:41 +01:00
Nick Craig-Wood	9d3d397f50	test_all: disable chunker + mailru tests while mailru is broken #4376	2020-08-20 12:50:20 +01:00
Nick Craig-Wood	38e8415e77	test_all: remove Digital Ocean s3 integration tests due to excessive rate limiting ... This is what I wrote to Digital Ocean support on July 10, 2020 - alas it didn't result in the rate limits dropping, so reluctantly I'm going to remove DO from the integration tests since they never pass and have no hope of ever passing while this rate limit is in effect. ---- Somewhere towards the end of June 2020 or the start of July 2020 my integration tests between rclone ( https://rclone.org ) and Digital Ocean started failing. I tried moving the tests to different regions (currently they are using AMS1 because I'm in Europe) with no improvement. Rclone seems to be hitting this rate limit as documented here: https://www.digitalocean.com/docs/spaces/#limits - 2 COPYs per 5 minutes on any individual object in a Space Rclone creates small objects about 100 bytes in size and renames them a few times - this involves using the COPY call as S3 does not have a rename API. The tests do this more than twice per object so hit the 5 minute timeout I think. Rclone does exponential backoff and fails after 10 retries not having reached 5 minutes delay after 10 retries. Having a 5 minute lockout on an S3 compatible API is surprising! Rclone integration tests with about 30 other providers, none of which have a rate limit like this. I understand the need for a COPY rate limit as server side copying large files can be resource intensive. However a 5 minute lockout for copying 100 byte files seems excessive! Might I humbly suggest that you reduce or eliminate this rate limit for small files? ---- This was the reply Unfortunately it is not possible to raise this limit or remove it currently on our platform. I do see how this would interfere with type of applications that need to copy many small files and will be happy to take the feedback to our engineering team to see how we can improve the spaces system in the future	2020-08-20 12:50:04 +01:00
Nick Craig-Wood	fb9edbe34e	test_all: export more internal variables to index.json for analysis	2020-08-20 12:23:33 +01:00
Nick Craig-Wood	63e4d2952b	fstests: Suggest that Purge on a nonexistent dir should return fs.ErrorDirNotFound	2020-08-19 18:03:42 +01:00
Nick Craig-Wood	a2afa9aadd	fs: Add directory to optional Purge interface - fixes #1891 ... - add a directory to the optional Purge interface - fix up all the backends - add an additional integration test to test for the feature - use the new feature in operations.Purge Many of the backends had been prepared in advance for this so the change was trivial for them.	2020-07-31 17:43:17 +01:00
Nick Craig-Wood	25662b9e05	fstest: add ability for mock objects and filesystems to have hashes	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	d8eea0e397	build: run gofmt -s to simplify the code: suggested by Go Report Card	2020-06-18 18:45:39 +01:00
Nick Craig-Wood	4b4ee72796	fstest: fix PublicLink tests to send non zero expiry and work with s3	2020-06-18 17:50:12 +01:00
Nick Craig-Wood	ba7fbfa8a7	testy: test utility functions	2020-06-15 21:34:37 +01:00
Roman Kredentser	55ad1354b6	link: Add --expire and --unlink flags ... This adds expire and unlink fields to the PublicLink interface. This fixes up the affected backends and removes unlink parameters where they are present.	2020-06-05 14:51:05 +01:00
Nick Craig-Wood	baaec5b126	testserver: don't allow starting of an empty remote	2020-05-25 15:26:06 +01:00
Martin Michlmayr	4aee962233	doc: fix typos throughout docs and code	2020-05-20 15:54:51 +01:00
Caleb Case	0ce662faad	Tardigrade Backend	2020-05-12 15:56:50 +00:00
Nick Craig-Wood	bd4b91bd57	test_all: revert running fichier tests one at a time ... This didn't achieve the objective of getting the tests to run clean and makes them take 16 hours! This reverts commit 97f6f8fe19.	2020-05-11 08:51:33 +01:00
Nick Craig-Wood	97f6f8fe19	test_all: run fichier tests one at a time ... This is in attempt to get the tests to run cleanly without hitting the rate limiter.	2020-05-07 20:39:54 +01:00
Nick Craig-Wood	94920d39ae	test_all: increase the test timeout to 60m from 30m ... Some tests are failing at 30m - 60m doesn't seem unreasonable	2020-05-07 20:38:40 +01:00
Nick Craig-Wood	9403bd2990	test_all: allow -list-retries to be overridden on the command line	2020-05-07 20:36:45 +01:00
Nick Craig-Wood	e098924e61	fstest: change -list-retries default to 3 from 6 ... The integration tests have got much better about retrying the tests, and we aren't testing ACD any more so we don't need it this high.	2020-05-07 20:28:41 +01:00
Fred	c754e89906	seafile: New backend for seafile server	2020-05-06 17:33:22 +00:00
Nick Craig-Wood	1aa1a2c174	backend: add new backend command for backend specific commands ... These commands are for implementing backend specific functionality. They have documentation which is placed automatically into the backend doc. There is a simple test for the feature in the backend tests.	2020-04-29 10:10:57 +01:00
Nick Craig-Wood	bbb6f94377	fstest: create AssertTimeEqualWithPrecision from CheckTimeEqualWithPrecision	2020-04-22 11:33:00 +01:00
Nick Craig-Wood	cd3c699f28	lib/readers: factor ErrorReader from multiple sources	2020-04-19 15:18:49 +01:00
Nick Craig-Wood	9a5178be7a	test_all: optionally run Cleanup before cleaning a remote ... Add this to the QingStor remote to stop it running out of buckets	2020-03-31 15:56:58 +01:00
Nick Craig-Wood	3855c003ce	build: update to use go1.14 for the build	2020-02-26 21:26:47 +00:00
Nick Craig-Wood	bd9fd629bc	test: add TestS3MinioEdge to test leading edge minio too #3934	2020-02-13 11:01:06 +00:00
Nick Craig-Wood	3b19f48929	testserver: add provider to TestS3Minio #3934 ... This is necessary to pass the TestIntegration/FsMkdir/FsEncoding listing tests.	2020-02-13 11:01:06 +00:00
Nick Craig-Wood	24ef00a258	build: implement a framework for starting test servers during tests ... Test servers are implemented by docker containers and run real servers for rclone to test against.	2020-01-18 16:47:37 +00:00
Nick Craig-Wood	c789436580	The memory backend ... This is a bucket based remote	2020-01-18 10:41:08 +00:00
Nick Craig-Wood	bedeaf23af	sugarsync: new backend - fixes #622	2020-01-17 17:39:34 +00:00
Nick Craig-Wood	b81601baff	test_all: ignore TestIntegration/FsMkdir/FsPutFiles/FsPutStream/0 on wasabi ... This has been reported to Wasabi and they've confirmed as a known issue that multipart uploads can't be 0 sized even though that is incompatible with AWS S3.	2020-01-13 09:47:11 +00:00
Nick Craig-Wood	ad2bb86d8c	fstests: add test for 0 sized streaming upload	2020-01-05 12:32:55 +00:00
Ivan Andreev	77b42aa33a	chunker: fix integration tests and hashsum issues	2019-10-13 10:43:46 +01:00
Ivan Andreev	9049bb62ca	chunker: prevent chunk corruption, survive meta-like input	2019-10-13 10:43:46 +01:00
Nick Craig-Wood	6abaa9e22c	fstests: allow skipping of the broken UTF-8 test for the cache backend	2019-10-10 10:36:18 +01:00
Nick Craig-Wood	e2914c0097	test_all: ignore some encoding tests with nextcloud integration test	2019-10-02 11:34:08 +01:00