extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-12-11 09:41:47 +01:00
Code Issues Packages Projects Releases Wiki Activity
58fd6d7b94
rclone/cmd/bisync/testdata/test_normalization/golden/test.log

143 lines
11 KiB
Plaintext
Raw Normal View History

bisync: handle unicode and case normalization consistently - mostly-fixes #7270 Before this change, Bisync sometimes normalized NFD to NFC and sometimes did not, causing errors in some scenarios (particularly for users of macOS). It was similarly inconsistent in its handling of case-insensitivity. There were three main places where Bisync should have normalized, but didn't: 1. When building the list of files that need to be transferred during --resync 2. When building the list of deltas during a non-resync 3. When comparing Path1 to Path2 during --check-sync After this change, 1 and 3 are resolved, and bisync supports --no-unicode-normalization and --ignore-case-sync in the same way as sync. 2 will be addressed in a future update.
2023-10-09 05:16:23 +02:00
(01) : test normalization
(02) : touch-copy 2001-01-02 {datadir/}file1.txt {path2/}
(03) : test initial bisync
(04) : bisync resync
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying unique Path2 files to Path1
bisync: optimize --resync performance -- partially addresses #5681 Before this change, --resync was handled in three steps, and needed to do a lot of unnecessary work to implement its own --ignore-existing logic, which also caused problems with unicode normalization, in addition to being pretty slow. After this change, it is refactored to produce the same result much more efficiently, by reducing the three steps to two and letting ci.IgnoreExisting do the work instead of reinventing the wheel. The behavior and sync order remain unchanged for now -- just faster (but see the ongoing lively discussions about potential future changes in #5681!)
2023-11-06 16:34:47 +01:00
INFO : - Path2 Resync is copying UNIQUE files to - Path1
INFO : - Path1 Resync is copying UNIQUE OR DIFFERING files to - Path2
bisync: handle unicode and case normalization consistently - mostly-fixes #7270 Before this change, Bisync sometimes normalized NFD to NFC and sometimes did not, causing errors in some scenarios (particularly for users of macOS). It was similarly inconsistent in its handling of case-insensitivity. There were three main places where Bisync should have normalized, but didn't: 1. When building the list of files that need to be transferred during --resync 2. When building the list of deltas during a non-resync 3. When comparing Path1 to Path2 during --check-sync After this change, 1 and 3 are resolved, and bisync supports --no-unicode-normalization and --ignore-case-sync in the same way as sync. 2 will be addressed in a future update.
2023-10-09 05:16:23 +02:00
INFO : Resync updating listings
INFO : Bisync successful
(05) : copy-as-NFC {datadir/}file1.txt {path1/}測試_Русский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éééö 測試_Русский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éééö.txt
(06) : copy-as-NFC {datadir/}file1.txt {path1/}folder éééö.txt
(07) : copy-as-NFC {datadir/}file1.txt {path1/}folder HeLlO,wOrLd!.txt
(08) : touch-copy 2001-01-03 {datadir/}file1.txt {path2/}
(09) : copy-as-NFD {datadir/}file1.txt {path2/}folder éééö.txt
(10) : copy-as-NFD {datadir/}file1.txt {path2/}folder hello,WORLD!.txt
(11) : test bisync run with fix-case
(12) : bisync fix-case
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File is new - folder/HeLlO,wOrLd!.txt
INFO : - Path1 File is new - folder/éééö.txt
INFO : - Path1 File is new - "測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
INFO : Path1: 3 changes: 3 new, 0 newer, 0 older, 0 deleted
INFO : Path2 checking for diffs
INFO : - Path2 File is newer - file1.txt
INFO : - Path2 File is new - folder/éééö.txt
INFO : - Path2 File is new - folder/hello,WORLD!.txt
INFO : Path2: 3 changes: 2 new, 1 newer, 0 older, 0 deleted
INFO : Applying changes
INFO : - Path1 Queue copy to Path2 - {path2/}folder/HeLlO,wOrLd!.txt
INFO : - Path1 Queue copy to Path2 - {path2/}folder/éééö.txt
INFO : - Path1 Queue copy to Path2 - "{path2/}測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
INFO : - Path2 Queue copy to Path1 - {path1/}file1.txt
INFO : - Path2 Queue copy to Path1 - {path1/}folder/éééö.txt
INFO : - Path2 Queue copy to Path1 - {path1/}folder/hello,WORLD!.txt
INFO : - Path2 Do queued copies to - Path1
INFO : folder/HeLlO,wOrLd!.txt: Fixed case by renaming to: folder/hello,WORLD!.txt
INFO : folder/éééö.txt: Fixed case by renaming to: folder/éééö.txt
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
(13) : purge-children {path1/}
(14) : purge-children {path2/}
(15) : touch-copy 2001-01-02 {datadir/}file1.txt {path2/}
(16) : bisync resync
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying unique Path2 files to Path1
bisync: optimize --resync performance -- partially addresses #5681 Before this change, --resync was handled in three steps, and needed to do a lot of unnecessary work to implement its own --ignore-existing logic, which also caused problems with unicode normalization, in addition to being pretty slow. After this change, it is refactored to produce the same result much more efficiently, by reducing the three steps to two and letting ci.IgnoreExisting do the work instead of reinventing the wheel. The behavior and sync order remain unchanged for now -- just faster (but see the ongoing lively discussions about potential future changes in #5681!)
2023-11-06 16:34:47 +01:00
INFO : - Path2 Resync is copying UNIQUE files to - Path1
INFO : - Path1 Resync is copying UNIQUE OR DIFFERING files to - Path2
bisync: handle unicode and case normalization consistently - mostly-fixes #7270 Before this change, Bisync sometimes normalized NFD to NFC and sometimes did not, causing errors in some scenarios (particularly for users of macOS). It was similarly inconsistent in its handling of case-insensitivity. There were three main places where Bisync should have normalized, but didn't: 1. When building the list of files that need to be transferred during --resync 2. When building the list of deltas during a non-resync 3. When comparing Path1 to Path2 during --check-sync After this change, 1 and 3 are resolved, and bisync supports --no-unicode-normalization and --ignore-case-sync in the same way as sync. 2 will be addressed in a future update.
2023-10-09 05:16:23 +02:00
INFO : Resync updating listings
INFO : Bisync successful
(17) : copy-as-NFC {datadir/}file1.txt {path1/}測試_Русский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éééö 測試_Русский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éééö.txt
(18) : copy-as-NFC {datadir/}file1.txt {path1/}folder éééö.txt
(19) : copy-as-NFC {datadir/}file1.txt {path1/}folder HeLlO,wOrLd!.txt
(20) : touch-copy 2001-01-03 {datadir/}file1.txt {path2/}
(21) : copy-as-NFD {datadir/}file1.txt {path2/}folder éééö.txt
(22) : copy-as-NFD {datadir/}file1.txt {path2/}folder hello,WORLD!.txt
(23) : test bisync run with normalization
(24) : bisync norm force
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File is new - folder/HeLlO,wOrLd!.txt
INFO : - Path1 File is new - folder/éééö.txt
INFO : - Path1 File is new - "測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
INFO : Path1: 3 changes: 3 new, 0 newer, 0 older, 0 deleted
INFO : Path2 checking for diffs
INFO : - Path2 File is newer - file1.txt
INFO : - Path2 File is new - folder/éééö.txt
INFO : - Path2 File is new - folder/hello,WORLD!.txt
INFO : Path2: 3 changes: 2 new, 1 newer, 0 older, 0 deleted
INFO : Applying changes
INFO : - Path1 Queue copy to Path2 - {path2/}folder/HeLlO,wOrLd!.txt
INFO : - Path1 Queue copy to Path2 - {path2/}folder/éééö.txt
INFO : - Path1 Queue copy to Path2 - "{path2/}測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
INFO : - Path2 Queue copy to Path1 - {path1/}file1.txt
INFO : - Path2 Queue copy to Path1 - {path1/}folder/éééö.txt
INFO : - Path2 Queue copy to Path1 - {path1/}folder/hello,WORLD!.txt
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
(25) : test resync
(26) : bisync resync norm
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying unique Path2 files to Path1
bisync: optimize --resync performance -- partially addresses #5681 Before this change, --resync was handled in three steps, and needed to do a lot of unnecessary work to implement its own --ignore-existing logic, which also caused problems with unicode normalization, in addition to being pretty slow. After this change, it is refactored to produce the same result much more efficiently, by reducing the three steps to two and letting ci.IgnoreExisting do the work instead of reinventing the wheel. The behavior and sync order remain unchanged for now -- just faster (but see the ongoing lively discussions about potential future changes in #5681!)
2023-11-06 16:34:47 +01:00
INFO : - Path2 Resync is copying UNIQUE files to - Path1
INFO : - Path1 Resync is copying UNIQUE OR DIFFERING files to - Path2
bisync: handle unicode and case normalization consistently - mostly-fixes #7270 Before this change, Bisync sometimes normalized NFD to NFC and sometimes did not, causing errors in some scenarios (particularly for users of macOS). It was similarly inconsistent in its handling of case-insensitivity. There were three main places where Bisync should have normalized, but didn't: 1. When building the list of files that need to be transferred during --resync 2. When building the list of deltas during a non-resync 3. When comparing Path1 to Path2 during --check-sync After this change, 1 and 3 are resolved, and bisync supports --no-unicode-normalization and --ignore-case-sync in the same way as sync. 2 will be addressed in a future update.
2023-10-09 05:16:23 +02:00
INFO : Resync updating listings
INFO : Bisync successful
(27) : test changed on both paths
(28) : touch-copy 2001-01-05 {datadir/}file1.txt {path2/}
(29) : copy-as-NFC {datadir/}file1.txt {path1/}測試_Русский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éééö 測試_Русский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éééö.txt
(30) : copy-as-NFC {datadir/}file1.txt {path1/}folder éééö.txt
(31) : copy-as-NFC {datadir/}file1.txt {path1/}folder HeLlO,wOrLd!.txt
(32) : copy-as-NFD {datadir/}file1.txt {path2/}folder éééö.txt
(33) : copy-as-NFD {datadir/}file1.txt {path2/}folder hello,WORLD!.txt
(34) : bisync norm
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File is newer - folder/HeLlO,wOrLd!.txt
INFO : - Path1 File is newer - folder/éééö.txt
INFO : - Path1 File is newer - "測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
INFO : Path1: 3 changes: 0 new, 3 newer, 0 older, 0 deleted
INFO : Path2 checking for diffs
INFO : - Path2 File is newer - file1.txt
INFO : - Path2 File is newer - folder/éééö.txt
INFO : - Path2 File is newer - folder/hello,WORLD!.txt
INFO : Path2: 3 changes: 0 new, 3 newer, 0 older, 0 deleted
INFO : Applying changes
INFO : - Path1 Queue copy to Path2 - {path2/}folder/HeLlO,wOrLd!.txt
INFO : - Path1 Queue copy to Path2 - {path2/}folder/éééö.txt
INFO : - Path1 Queue copy to Path2 - "{path2/}測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö/測試_Русский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éééö.txt"
INFO : - Path2 Queue copy to Path1 - {path1/}file1.txt
INFO : - Path2 Queue copy to Path1 - {path1/}folder/éééö.txt
INFO : - Path2 Queue copy to Path1 - {path1/}folder/hello,WORLD!.txt
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Reference in New Issue Copy Permalink