Commit Graph

56 Commits

Author SHA1 Message Date
Nick Craig-Wood
414b35ea56 Change the interface of SetModTime to return an error - #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood
8ef551bf9c Make dedupe remove identical copies without asking and add non interactive mode - fixes #338
* Now removes identical copies without asking
  * Now obeys `--dry-run`
  * Implement `--dedupe-mode` for non interactive running
    * `--dedupe-mode interactive` - interactive the default.
    * `--dedupe-mode skip` - removes identical files then skips anything left.
    * `--dedupe-mode first` - removes identical files then keeps the first one.
    * `--dedupe-mode newest` - removes identical files then keeps the newest one.
    * `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
    * `--dedupe-mode rename` - removes identical files then renames the rest to be different.
  * Add tests which will only run on Google Drive.
2016-03-06 18:15:01 +00:00
Nick Craig-Wood
280ac26464 Implement -u/--update so creation times can be used on all remotes - #226 2016-02-29 17:46:40 +00:00
Nick Craig-Wood
3343c1afa4 Don't make directories if --dry-run set - fixes #342 2016-02-28 19:56:50 +00:00
Nick Craig-Wood
867a26fe4f Implement --low-level-retries flag - fixes #266 2016-02-25 22:58:21 +00:00
Nick Craig-Wood
d2219a800a Fix and document the move command - fixes #334
* Don't attempt to use server side Move unless they are on the same Fs
  * Fix move in the presense of filters
2016-02-25 20:05:34 +00:00
klauspost
ef06371c93 Create separate interface for object information.
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
2016-02-19 13:31:09 +00:00
Nick Craig-Wood
a6320bbad3 Fix delete command to wait until all finished - fixes missing deletes.
This also could affect deletes at the end of the sync command.
2016-02-15 16:43:59 +00:00
Nick Craig-Wood
0f73129ab7 dedupe command to deduplicate a remote. Useful with google drive - fixes #41 2016-01-31 16:09:42 +00:00
Nick Craig-Wood
1373efaa39 Delete command which does obey the filters - fixes #327 2016-01-31 16:06:04 +00:00
Nick Craig-Wood
5c37b777fc Make the --dry-run warnings into logs so they appear without the -v flag 2016-01-31 16:06:04 +00:00
Nick Craig-Wood
8c211fc8df Warn the user about files with same name but different case
Relates to #107 & #119.
2016-01-26 16:57:09 +00:00
Nick Craig-Wood
12b51c5eb8 Remove duplicate check for filter IncludeObject 2016-01-26 16:57:09 +00:00
klauspost
14069fd8e6 Implement --delete-before, --delete-during, --delete-after - fixes #252. 2016-01-26 16:57:09 +00:00
Nick Craig-Wood
cd62f41606 Reduce number of logs and show hash type where appropriate 2016-01-24 18:06:57 +00:00
Nick Craig-Wood
eda4130703 Fix integration tests so they can be run independently and out of order - fixes #291
* Make all integration tests start with an empty remote
  * Add an -individual flag so this can be a different bucket/container/directory
  * Fix up tests after changing the hashers
  * Add sha1sum test
  * Make directory checking in tests sleep more to fix acd inconsistencies
  * Factor integration tests to make more maintainable
  * Ensure remote writes have a fstest.CheckItems() before use
    * this fixes eventual consistency on the directory listings later
  * Call fs.Stats.ResetCounters() before every fs.Sync()

Note that the tests shouldn't be run concurrently as fs.Config is global state.
2016-01-23 17:02:09 +00:00
Nick Craig-Wood
27f67edb1a Fix formatting problem in sha1sum 2016-01-17 13:56:42 +00:00
klauspost
78c3a5ccfa Add support for multiple hash types.
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
klauspost
25f22ec561 Add "--ignore-existing" flag.
Add option to completely ignore existing files and not consider them for transfer.

Fixes #274
2016-01-08 08:20:04 +00:00
Nick Craig-Wood
36f1bc4a8a Make ls/lsl/md5sum/size/check obey includes and excludes - fixes #169
* run check directory listings concurrently
2015-12-17 13:30:58 +00:00
Adriano Aurélio Meirelles
3cbd57d9ad Add support to filter files based on their age 2015-12-17 09:52:38 -02:00
Nick Craig-Wood
2c2cb84ca7 Make it so optional interface Purge can fail so it can be wrapped 2015-11-14 08:08:51 +00:00
Nick Craig-Wood
80ffbade22 Fix deletion of some excluded files without --delete-excluded #205
This only happened if the destination file was present but the source
file was missing.
2015-11-12 11:46:04 +00:00
Klaus Post
6344c3051c Add async readahead buffer
This adds an async read buffer of 4x4MB when copying files >10MB.

This fixes #164 and reduces the number of IO operations for copy/move.
2015-10-12 08:30:27 +01:00
Nick Craig-Wood
177dbbc29a Implement rclone size for measuring remotes - fixes #153 2015-10-03 16:04:21 +01:00
Nick Craig-Wood
a91bcaaeb0 Implement rsync like include and exclude - fixes #27
* Implement include/exclude
  * Implement rsync compatible file globbing
  * Implement command line filtering flags
    * --delete-excluded - Delete files on dest excluded from sync
    * --filter - Add a file-filtering rule
    * --filter-from - Read filtering patterns from a file
    * --exclude - Exclude files matching pattern
    * --exclude-from - Read exclude patterns from file
    * --include - Include files matching pattern
    * --include-from - Read include patterns from file
    * --files-from - Read list of source-file nam
    * --min-size - Don't transfer any file smaller than this in k or suffix k|M|G
    * --max-size - Don't transfer any file larger than this in k or suffix k|M|G
  * Document
2015-09-28 19:18:21 +01:00
Nick Craig-Wood
5f20ae707d Make lsl output times in localtime and fix tests - fixes #141 2015-09-22 19:04:12 +01:00
Nick Craig-Wood
e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
Nick Craig-Wood
2ed158aba3 Fixes from go vet and errcheck 2015-09-22 07:31:12 +01:00
klauspost
0a5870208e Display individual transfer progress
Improve progress printing by displaying individual file progress, as well
as a moving average speed with ETA. Example output:

2015/09/15 16:38:21
Transferred:    183599104 Bytes (4646.49 kByte/s)
Errors:                 0
Checks:                 1
Transferred:            0
Elapsed time:       38.5s
Transferring:
 * 01_06_14.mp3: 33% done. avg: 1280.5, cur: 1288.8 kByte/s. ETA: 1m12s
 * 01_12_15.mp3: 33% done. avg: 1002.2, cur:  943.4 kByte/s. ETA: 1m17s
 * 01_13_14.mp3: 48% done. avg: 1456.8, cur: 1425.2 kByte/s. ETA: 39s
 * 01_19_15.mp3: 28% done. avg: 1226.9, cur: 1114.4 kByte/s. ETA: 1m37s
2015-09-16 19:42:46 +01:00
Nick Craig-Wood
8d33ce0154 Check for source and dest being the same in sync/copy/move 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
59ba8f28c8 Implement move command - fixes #35
* Define Mover interface to move a single object
  * Define DirMover interface to move a directory
  * Implement DirMove operation
  * Add `rclone move` command
  * Tests for Dir Move

To Do
  * Implement Move, DirMover in local, drive, dropbox
  * unit test for Mover
  * unit test for DirMover
2015-08-28 08:49:16 +01:00
Nick Craig-Wood
a96b522958 Implement server side copies if possible - fixes #99
Add optional fs.Copier interface

Implemented for
  * swift
  * s3
  * drive
  * dropbox
  * google cloud storage
2015-08-23 21:18:38 +01:00
Nick Craig-Wood
dfe771fb0c Correct log messages for remotes which don't support modtime/md5sum 2015-08-20 20:48:58 +01:00
Nick Craig-Wood
1d9e76bb0f dropbox: remove datastore - Fixes #55 #84
This means that dropbox no longer stores MD5SUMs and modified times.

Fixup the tests so that blank MD5SUMs are ignored, and that if
Precision is set to a fs.ModTimeNotSupported, ModTimes can be ignored too.

This opens the door for other FSs which don't support metadata easily.
2015-08-17 17:26:36 +01:00
Leonid Shalupov
3fcff32524 do not print stats in quiet mode - fixes #70
...unless had some errors or stats interval requested.

Add fs.ErrorLog to differentiate between Logs which should be
suppressed and errors which shouldn't.
2015-08-10 11:17:54 +01:00
Nick Craig-Wood
97dced6a0b Don't check md5sum after download with --size-only - fixes #75 2015-06-09 13:18:40 +01:00
Nick Craig-Wood
e22b445cff Implement --size-only flag to sync on size not checksum & modtime - fixes #75 2015-06-06 08:49:01 +01:00
Alex Couper
7af1282375 Add --checksum flag to only discard transfers by MD5SUM - fixes #61
Useful for copying between backends where checksum fetching is fast,
ie any of s3, swift, drive or googlecloudstorage.
2015-06-05 19:46:03 +01:00
Nick Craig-Wood
5cc5429f99 drive: switch to insert from update when a failed copy deletes the upload 2015-03-15 15:27:55 +00:00
Nick Craig-Wood
c643e4585e core: Log duplicate files if they are detected 2015-03-15 15:27:55 +00:00
Nick Craig-Wood
46d39ebaf7 Factor Mime Type guessing into fs.MimeType() 2015-03-02 09:21:15 +00:00
Nick Craig-Wood
2360bf907a Add synchronization to list output to stop corruptions - fixes #29 2015-02-28 15:30:40 +00:00
Nick Craig-Wood
5fb6f94579 drive: fix rate limit exceeded errors - fixes #20
This is done by pacing the requests to drive and backing them off
using an exponential increase.  Put and Modify operations can now be
retried also.
2015-02-07 22:32:51 +00:00
Nick Craig-Wood
18439cf2d7 Move rclonetest into go tests for fs module 2014-08-03 11:18:25 +01:00
Nick Craig-Wood
17ffb0855f Fixes after running errcheck 2014-07-25 18:19:49 +01:00
Nick Craig-Wood
e09ef62d5b core: Fix race detected by go race detector 2014-07-22 23:03:14 +01:00
Nick Craig-Wood
e559194fb2 fs: Verify sizes are the same after transfer in Copy() 2014-07-19 13:05:07 +01:00
Nick Craig-Wood
8a76568ea8 core: Verify MD5 sums after each transfer 2014-07-15 19:27:05 +01:00
Nick Craig-Wood
eb2b4ea8aa rclone: Don't purge if --dry-run set 2014-07-13 10:54:30 +01:00