splits m.key into separate functions for src and dst to prepare for
lib/transform which will want to do transforms on the src side only.
Co-Authored-By: Nick Craig-Wood <nick@craig-wood.com>
As part of the out of memory syncing code, in this commit
0148bd4668 march: Implement callback based syncing
we changed the syncing method to use a sorted stream of directory
entries.
Unfortunately as part of this change the sort order of files and
directories became undefined.
This meant that if there existed both a file `foo` and a directory
`foo` in the same directory (as is common on object storage systems)
then these could be matched up incorrectly.
They could be matched up correctly like this
- `foo` (directory) - `foo` (directory)
- `foo` (file) - `foo` (file)
Or incorrectly like this (one of many possibilities)
- no match - `foo` (file)
- `foo` (directory) - `foo` (directory)
- `foo` (file) - no match
Just depending on how the input listings were ordered.
This in turn made container based syncing with a duplicated file and
directory name erratic, deleting files when it shouldn't.
This patch ensures that directories always sync before files by adding
a suffix to the sort key depending on whether the entry was a file or
directory.
This changes the syncing method to take callbacks for directory
listings rather than being passed the entire directory listing at
once.
This will enable out of memory syncing.
This is done by making fs.Config private and attaching it to the
context instead.
The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
If your filenames contain two near-identical Unicode characters,
rclone will normalize these, making them identical. This flag
gives you the ability to keep them unique. This might
create unintended side effects, such as duplicating files that
contain certain Unicode characters, when downloading them from
certain cloud providers to a macOS filesystem.
Fixes#4228
When sorting fs.DirEntries we sort by DirEntry type and
when synchronizing files let the directories be before objects,
so when the destintation fs doesn't support duplicate names,
we will only lose duplicated object instead of whole directory.
The enables synchronisation to work with a file and a directory of the same name
which is reasonably common on bucket based remotes.
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.
The new code layout is documented in CONTRIBUTING.
