rclone/cmd/bisync/testdata/test_normalization/scenario.txt
nielash f7f4651828 bisync: handle unicode and case normalization consistently - mostly-fixes #7270
Before this change, Bisync sometimes normalized NFD to NFC and sometimes
did not, causing errors in some scenarios (particularly for users of macOS).
It was similarly inconsistent in its handling of case-insensitivity.

There were three main places where Bisync should have normalized, but didn't:

1. When building the list of files that need to be transferred during --resync
2. When building the list of deltas during a non-resync
3. When comparing Path1 to Path2 during --check-sync

After this change, 1 and 3 are resolved, and bisync supports
--no-unicode-normalization and --ignore-case-sync in the same way as sync.
2 will be addressed in a future update.
2024-01-20 14:50:08 -05:00

53 lines
2.6 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

test normalization
# Tests support for --no-unicode-normalization and --ignore-case-sync
# note: this test is written carefully to be runnable regardless of case/unicode sensitivity
# i.e. the results should be the same on linux and macOS
# force specific modification time since file time is lost through git
touch-copy 2001-01-02 {datadir/}file1.txt {path2/}
test initial bisync
bisync resync
# copy NFC version to Path1
copy-as-NFC {datadir/}file1.txt {path1/}測試_Русский___ě_áñ👸🏼🧝🏾💆🏿🐨🤙🏼🤮🧑🏻🔧🧑🔬éééö 測試_Русский___ě_áñ👸🏼🧝🏾💆🏿🐨🤙🏼🤮🧑🏻🔧🧑🔬éééö.txt
copy-as-NFC {datadir/}file1.txt {path1/}folder éééö.txt
copy-as-NFC {datadir/}file1.txt {path1/}folder HeLlO,wOrLd!.txt
# place newer NFD version on Path2
touch-copy 2001-01-03 {datadir/}file1.txt {path2/}
copy-as-NFD {datadir/}file1.txt {path2/}folder éééö.txt
copy-as-NFD {datadir/}file1.txt {path2/}folder hello,WORLD!.txt
test bisync run with fix-case
bisync fix-case
# purge and reset
purge-children {path1/}
purge-children {path2/}
touch-copy 2001-01-02 {datadir/}file1.txt {path2/}
bisync resync
# copy NFC version to Path1
copy-as-NFC {datadir/}file1.txt {path1/}測試_Русский___ě_áñ👸🏼🧝🏾💆🏿🐨🤙🏼🤮🧑🏻🔧🧑🔬éééö 測試_Русский___ě_áñ👸🏼🧝🏾💆🏿🐨🤙🏼🤮🧑🏻🔧🧑🔬éééö.txt
copy-as-NFC {datadir/}file1.txt {path1/}folder éééö.txt
copy-as-NFC {datadir/}file1.txt {path1/}folder HeLlO,wOrLd!.txt
# place newer NFD version on Path2
touch-copy 2001-01-03 {datadir/}file1.txt {path2/}
copy-as-NFD {datadir/}file1.txt {path2/}folder éééö.txt
copy-as-NFD {datadir/}file1.txt {path2/}folder hello,WORLD!.txt
test bisync run with normalization
bisync norm force
test resync
bisync resync norm
test changed on both paths
touch-copy 2001-01-05 {datadir/}file1.txt {path2/}
copy-as-NFC {datadir/}file1.txt {path1/}測試_Русский___ě_áñ👸🏼🧝🏾💆🏿🐨🤙🏼🤮🧑🏻🔧🧑🔬éééö 測試_Русский___ě_áñ👸🏼🧝🏾💆🏿🐨🤙🏼🤮🧑🏻🔧🧑🔬éééö.txt
copy-as-NFC {datadir/}file1.txt {path1/}folder éééö.txt
copy-as-NFC {datadir/}file1.txt {path1/}folder HeLlO,wOrLd!.txt
copy-as-NFD {datadir/}file1.txt {path2/}folder éééö.txt
copy-as-NFD {datadir/}file1.txt {path2/}folder hello,WORLD!.txt
bisync norm