Before this change we released the ssh connection back to the pool
before the upload was finished.
This meant that uploads were re-using the same ssh connection which
reduces throughput.
This releases the ssh connection back to the pool only after the
upload has finished, or on error state.
See: https://forum.rclone.org/t/sftp-backend-opens-less-connection-than-expected/40245
This changes hasVirtual to an atomic struct variable that's updated on
add or delete from the virtual map.
This keeps it up to date and avoids deadlocks.
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
This empties the directory cache after twice the directory cache
period to release memory.
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
smb2.File implements the WriterAtCloser interface defined in
fs/types.go. Expose it via a OpenWriterAt method on
the fs struct to support multi-threaded writes.
As suddenly many people move to Box - another "unlimited" cloud story migration saga there are frequent questions about crypt files encoding to be used.
Box is base32768 friendly.
It has been tested with:
https://pub.rclone.org/base32768.zip
and:
rclone test info --check-length boxremote:
maxFileLength = 255 // for 1 byte unicode characters
maxFileLength = 255 // for 2 byte unicode characters
maxFileLength = 255 // for 3 byte unicode characters
maxFileLength = -1 // for 4 byte unicode characters
Before this change, the overlapping check could erroneously give this
error on case insensitive file systems:
Failed to sync: destination and parameter to --backup-dir mustn't overlap
The code was fixed and re-worked to be simpler and more reliable.
See: https://forum.rclone.org/t/backup-dir-cannot-be-in-root-even-when-excluded/39844/
The error is:
Error: failed to configure token with jwt authentication: jwtutil: failed making auth request: 400 Bad Request
With the following additional debug information:
jwtutil: Response Body: {"error":"invalid_grant","error_description":"Please check the 'aud' claim. Should be a string"}
Problem is that in jwt-go the RegisteredClaims type has Audience field (aud claim) that
is a list, while box apparantly expects it to be a singular string. In jwt-go v4 we
currently use there is an alternative type StandardClaims which matches what box wants.
Unfortunately StandardClaims is marked as deprecated, and is removed in the
newer v5 version, so we this is a short term fix only.
Fixes#7114
Before this change the new partial downloads code was causing symlinks
to be copied as regular files.
This was because the partial isn't named .rclonelink so the local
backend saves it as a normal file and renaming it to .rclonelink
doesn't cause it to become a symlink.
This fixes the problem by not copying .rclonelink files using the
partials mechanism but reverting to the previous --inplace behaviour.
This could potentially be fixed better in the future by changing the
local backend Move to change files to and from symlinks depending on
their name. However this was deemed too complicated for a point
release.
This also adds a test in the local backend. This test should ideally
be in operations but it isn't easy to put it there as operations knows
nothing of symlinks.
Fixes#7101
See: https://forum.rclone.org/t/reggression-in-v1-63-0-links-drops-the-rclonelink-extension/39483
Before this change if a directory entry could be listed but not
lstat-ed then rclone would give an error and abort the directory
listing with the error
failed to read directory entry: failed to read directory "XXX": lstat XXX
This change makes sure that the directory listing carries on even
after this kind of error.
The sync will be failed but it will carry on.
This problem was caused by a programming error setting the err
variable in an outer scope when it should have been using a local err
variable.
See: https://forum.rclone.org/t/sync-aborts-if-even-one-single-unreadable-folder-is-encountered/39653
Before this change, if you mounted the root of the smb then it would
give an error on rclone about and periodically in the mount logs:
Statfs failed: bucket or container name is needed in remote
This fix makes the smb backend return empty usage in this case which
will stop the errors and show the default 1P of free space.
See: https://forum.rclone.org/t/error-statfs-failed-bucket-or-container-name-is-needed-in-remote/39631
This sentence was written at the time when backend used access token, nowadays, users need to generate and use application password instead, see #6398.
This introduces a new fs.Option flag, Sensitive and uses this along
with IsPassword to redact the info in the config file for support
purposes.
It adds this flag into backends where appropriate. It was necessary to
add oauthutil.SharedOptions to some backends as they were missing
them.
Fixes#5209
Fix https://github.com/rclone/rclone/issues/7103
Before this change the RegExp validating the endpoint URL was a bit
too strict allowing only /dav/files/USER due to chunking limitations.
This patch adds back support for /dav/files/USER/dir/subdir etc.
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
The --progress flag overrides operations.SyncPrintf in order to do its
magic on stdout without interfering with other output.
Before this change the syncFprintf routine in operations (which is
used to print all output to stdout) was taking the
operations.StdoutMutex and the printProgress function in the
--progress routine was also attempting to take the same mutex causing
a deadlock.
This patch fixes the problem by moving the locking from the
syncFprintf function to SyncPrintf. It is then up to the function
overriding this to lock the StdoutMutex. This ensures the StdoutMutex
can never cause a deadlock.
Before this change if using --fast-list on a directory with more than
a few thousand directories in it DirTree.CheckParents became very slow
taking up to 24 hours for a directory with 1,000,000 directories in
it.
This is because it becomes an O(N²) operation as DirTree.Find has to
search each directory in a linear fashion as it is stored as a slice.
This patch fixes the problem by scanning the DirTree for directories
before starting the CheckParents process so it never has to call
DirTree.Find.
After the fix calling DirTree.CheckParents on a directory with
1,000,000 directories in it will take about 1 second.
Anything which calls DirTree.Find can potentially have bad performance
so in the future we should redesign the DirTree to use a different
underlying datastructure or have an index.
https://forum.rclone.org/t/almost-24-hours-cpu-compute-time-during-sync-between-two-large-s3-buckets/39375/
