From 16a3d3eb627c70796ebcc6295855c7ddfeb977de Mon Sep 17 00:00:00 2001 From: lc63 <123966558+coffreo-lcabello@users.noreply.github.com> Date: Fri, 21 Jul 2023 17:12:21 +0200 Subject: [PATCH] Updated Big syncs with millions of files (markdown) --- Big-syncs-with-millions-of-files.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Big-syncs-with-millions-of-files.md b/Big-syncs-with-millions-of-files.md index 374cc4e..46458e1 100644 --- a/Big-syncs-with-millions-of-files.md +++ b/Big-syncs-with-millions-of-files.md @@ -9,9 +9,9 @@ Transferred: 0 B / 0 B, -, 0 B/s, ETA - Elapsed time: 1m0.0s ``` -... although HTTP REQUEST requests are made, with HTTP RESPONSE 200 in response (--dump-headers option), no copy is made. +... although HTTP REQUEST requests are made, with HTTP RESPONSE 200 in response (`--dump-headers` option), no copy is made. -This problem exists until at least version rclone v1.64.0-beta.7132.f1a842081. +This problem exists until at least version `rclone v1.64.0-beta.7132.f1a842081`. # Workaround @@ -33,7 +33,7 @@ comm -13 src dst > need-to-delete You now have a list of files you need to transfer from src to dst and another list of files in dst that aren't in src so should likely be deleted. -Then break the need-to-transfer file up into chunks of (say) 10,000 lines with something like split -l 10000 need-to-transfer and run this on each chunk to transfer 10,000 files at a time. The --files-from and the --no-traverse means that this won't list the source or the destination so will avoid using too much memory. +Then break the need-to-transfer file up into chunks of (say) 10,000 lines with something like `split -l 10000 need-to-transfer` and run this on each chunk to transfer 10,000 files at a time. The `--files-from` and the `--no-traverse` means that this won't list the source or the destination ``` rclone copy src:bucket dst:bucket --files-from need-to-transfer-aa --no-traverse