rclone/fs
nielash fd8faeb0e6 vfs: fix unicode normalization on macOS - fixes #7072
Before this change, the VFS layer did not properly handle unicode normalization,
which caused problems particularly for users of macOS. While attempts were made
to handle it with various `-o modules=iconv` combinations, this was an imperfect
solution, as no one combination allowed both NFC and NFD content to
simultaneously be both visible and editable via Finder.

After this change, the VFS supports `--no-unicode-normalization` (default `false`)
via the existing `--vfs-case-insensitive` logic, which is extended to apply to both
case insensitivity and unicode normalization form.

This change also adds an additional flag, `--vfs-block-norm-dupes`, to address a
probably rare but potentially possible scenario where a directory contains
multiple duplicate filenames after applying case and unicode normalization
settings. In such a scenario, this flag (disabled by default) hides the
duplicates. This comes with a performance tradeoff, as rclone will have to scan
the entire directory for duplicates when listing a directory. For this reason,
it is recommended to leave this disabled if not needed. However, macOS users may
wish to consider using it, as otherwise, if a remote directory contains both NFC
and NFD versions of the same filename, an odd situation will occur: both
versions of the file will be visible in the mount, and both will appear to be
editable, however, editing either version will actually result in only the NFD
version getting edited under the hood. `--vfs-block-norm-dupes` prevents this
confusion by detecting this scenario, hiding the duplicates, and logging an
error, similar to how this is handled in `rclone sync`.
2024-03-06 16:12:13 +00:00
..
accounting rc: add srcFs and dstFs to core/stats and core/transferred stats 2024-02-02 11:43:10 +00:00
asyncreader Replace deprecated ioutil 2022-11-07 11:41:47 +00:00
cache fs/cache: add PutErr to add an fs.Fs with an fs.ErrorIsFile error to the cache 2024-02-29 16:21:41 +00:00
chunkedreader build: add package comments to silence revive linter 2022-08-28 13:43:51 +02:00
chunksize azureblob,b2,s3: fix chunksize calculations producing too many parts 2022-08-09 12:57:38 +01:00
config cmd: refactor --retries and --retries-sleep to global config 2024-02-12 13:24:54 -05:00
dirtree dirtree: fix performance with large directories of directories and --fast-list 2023-07-03 14:09:21 +01:00
driveletter build: add package comments to silence revive linter 2022-08-28 13:43:51 +02:00
filter fs: add more detailed logging for file includes/excludes 2024-01-22 16:46:26 +00:00
fserrors Fix lint issues reported by staticcheck 2022-06-13 21:13:50 +02:00
fshttp fs/fshttp: fix --contimeout being ignored 2023-11-24 17:53:33 +00:00
fspath fspath: allow the symbols at and plus in remote names - fixes #6710 2023-01-25 13:37:24 +01:00
hash hash: allow runtime configuration of supported hashes for testing 2023-11-24 11:19:58 +00:00
list fs: allow Metadata calls to be called with Directory or Object 2024-02-27 10:56:19 +00:00
log systemd: Fix detection and switch to the coreos package everywhere 2023-12-02 14:17:15 +00:00
logger sync: report list of synced paths to file -- see #7282 2024-01-20 14:50:08 -05:00
march march: Fix excessive parallelism when using --no-traverse 2023-11-20 17:36:31 +00:00
object docs: fix typos found by codespell in docs and code comments 2023-09-23 12:20:01 +01:00
operations vfs: fix unicode normalization on macOS - fixes #7072 2024-03-06 16:12:13 +00:00
rc rc: Add GetFsNamedFileOK to get an fs which could also be a file 2024-02-29 16:21:42 +00:00
sync sync: fix TestMoveEmptyDirectories so they work on backends which don't support DirModTimes 2024-03-01 10:56:48 +00:00
walk fs: add more detailed logging for file includes/excludes 2024-01-22 16:46:26 +00:00
backend_config_test.go fs: add --all to rclone config create/update to ask all the config questions #3455 2021-05-14 14:07:44 +01:00
backend_config.go fs: fix typos found by codespell 2023-03-25 12:51:04 +01:00
bits_test.go fs: add IsSet convenience method to Bits 2023-10-23 15:43:42 +01:00
bits.go fs: fix docs for Bits 2023-10-23 15:43:55 +01:00
bwtimetable_test.go fs: fix option types printing incorrectly for backend flags 2023-10-03 11:23:58 +01:00
bwtimetable.go build: reformat comments to pass go1.19 vet 2022-08-05 16:35:41 +01:00
config_list_test.go fs: fix tests for *SepList 2018-08-21 10:58:59 +01:00
config_list.go Make flags show up with their proper names, eg SizeSuffix rather than int 2019-02-07 11:57:26 +00:00
config_test.go build: add lint ignore comment required for golangci-staticcheck in addition to stand-alone staticcheck 2023-03-26 14:28:15 +02:00
config.go cmd: refactor --retries and --retries-sleep to global config 2024-02-12 13:24:54 -05:00
configmap.go config: fix in memory config not saving on the fly backend config 2021-07-20 12:09:38 +01:00
countsuffix_test.go fs: fix option types printing incorrectly for backend flags 2023-10-03 11:23:58 +01:00
countsuffix.go fs: fix option types printing incorrectly for backend flags 2023-10-03 11:23:58 +01:00
cutoffmode_test.go fs: re-implement CutoffMode, LogLevel, TerminalColorMode with Enum 2023-10-03 15:14:24 +01:00
cutoffmode.go fs: re-implement CutoffMode, LogLevel, TerminalColorMode with Enum 2023-10-03 15:14:24 +01:00
daemon_other.go mountlib: correctly daemonize for compatibility with automount - #5593 2021-10-02 23:45:16 +03:00
daemon_unix.go mountlib: correctly daemonize for compatibility with automount - #5593 2021-10-02 23:45:16 +03:00
deletemode.go Break the fs package up into smaller parts. 2018-01-15 17:51:14 +00:00
dir_wrapper.go fs: add DirWrapper for wrapping Directory-s with optional methods 2024-02-27 10:56:19 +00:00
dir.go fs: allow Metadata calls to be called with Directory or Object 2024-02-27 10:56:19 +00:00
direntries_test.go build: fix up package paths after repo move 2019-07-28 18:47:38 +01:00
direntries.go fs: Allow sync of a file and a directory with the same name 2019-06-09 15:57:05 +01:00
dump_test.go fs: re-implement DumpMode with Bits 2023-10-03 15:24:09 +01:00
dump.go fs: implement --metadata-mapper to transform metatadata with a user supplied program 2023-11-18 17:49:35 +00:00
enum_test.go fs: re-implement CutoffMode, LogLevel, TerminalColorMode with Enum 2023-10-03 15:14:24 +01:00
enum.go fs: re-implement CutoffMode, LogLevel, TerminalColorMode with Enum 2023-10-03 15:14:24 +01:00
features.go fs: add Directory Metadata flags for backends and interfaces 2024-02-27 10:56:19 +00:00
fingerprint_test.go mockfs: make it so it can be registered as an Fs 2023-04-28 14:31:05 +01:00
fingerprint.go fs: add Fingerprint to detect changes in an object 2020-06-30 12:01:36 +01:00
fs_test.go rclone config redacted: implement support mechanism for showing redacted config 2023-07-07 16:25:14 +01:00
fs.go sync: update docs and error messages to reflect fixes to overlap checks 2022-07-13 16:04:53 +01:00
log_test.go fs: fix a typo in a comment 2023-11-16 17:15:00 +00:00
log.go fs: re-implement CutoffMode, LogLevel, TerminalColorMode with Enum 2023-10-03 15:14:24 +01:00
metadata_mapper_code.go fs: implement --metadata-mapper to transform metatadata with a user supplied program 2023-11-18 17:49:35 +00:00
metadata_test.go fs: implement --metadata-mapper to transform metatadata with a user supplied program 2023-11-18 17:49:35 +00:00
metadata.go fs: allow Metadata calls to be called with Directory or Object 2024-02-27 10:56:19 +00:00
mimetype.go fs: allow Metadata calls to be called with Directory or Object 2024-02-27 10:56:19 +00:00
mount_helper_test.go rclone.mount: ignore _netdev mount argument - FIxes #5808 2022-05-12 20:27:13 +01:00
mount_helper.go build: reformat comments to pass go1.19 vet 2022-08-05 16:35:41 +01:00
newfs_test.go mount: fix mount failure on macOS with on the fly remote 2023-06-23 14:12:03 +01:00
newfs.go mount: fix mount failure on macOS with on the fly remote 2023-06-23 14:12:03 +01:00
open_options_test.go fs: add --metadata-set flag to specify metadata for uploads 2022-06-29 14:29:36 +01:00
open_options.go build: fix lint errors when re-enabling revive exported & package-comments 2023-08-29 13:03:13 +01:00
override_dir_test.go fs: add OverrideDirectory for overriding path of directory 2023-05-14 11:22:57 +01:00
override_dir.go fs: add OverrideDirectory for overriding path of directory 2023-05-14 11:22:57 +01:00
override_test.go fs: move operations.NewOverrideRemote to fs.NewOverrideRemote 2022-11-21 08:02:09 +00:00
override.go fs: when creating new fs.OverrideRemotes don't layer overrides if not needed 2023-05-09 16:28:10 +01:00
pacer.go pacer: fix b2 deadlock by defaulting max connections to unlimited 2023-10-01 12:30:26 +01:00
parseduration_test.go fs: fix option types printing incorrectly for backend flags 2023-10-03 11:23:58 +01:00
parseduration.go accounting: limit length of ETA string 2023-02-04 17:29:08 +01:00
parsetime_test.go fs: fix option types printing incorrectly for backend flags 2023-10-03 11:23:58 +01:00
parsetime.go rc: fix output of Time values in options/get 2023-05-04 15:04:11 +01:00
registry.go backend: add description field for all backends 2024-01-03 10:57:59 +00:00
sizesuffix_test.go fs: fix option types printing incorrectly for backend flags 2023-10-03 11:23:58 +01:00
sizesuffix.go fs: fix option types printing incorrectly for backend flags 2023-10-03 11:23:58 +01:00
terminalcolormode_test.go fs: re-implement CutoffMode, LogLevel, TerminalColorMode with Enum 2023-10-03 15:14:24 +01:00
terminalcolormode.go fs: re-implement CutoffMode, LogLevel, TerminalColorMode with Enum 2023-10-03 15:14:24 +01:00
tristate_test.go fs: fix option types printing incorrectly for backend flags 2023-10-03 11:23:58 +01:00
tristate.go fs: fix tristate conversion to JSON 2023-03-27 16:17:49 +01:00
types.go fs: allow Metadata calls to be called with Directory or Object 2024-02-27 10:56:19 +00:00
version.go Configurable version suffix independent of version number 2022-05-17 12:10:01 +01:00
versioncheck.go build: update to go1.22rc1 and make go1.20 the minimum required version 2024-01-15 16:22:07 +00:00
versionsuffix.go Configurable version suffix independent of version number 2022-05-17 12:10:01 +01:00
versiontag.go Start v1.66.0-DEV development 2023-11-26 17:14:38 +00:00