Users have noticed that backends created via the rc have been failing
to refresh their tokens with this error:
Token refresh failed try 1/5: context canceled
This is because the rc server cancels the context used to make the
backend when the request has finished. This same context is used to
refresh the token and the oauth library checks to see if the context
has been cancelled.
This patch creates a new context for the cached backends and copies
the global and filter config into the new context.
See: https://forum.rclone.org/t/google-drive-token-refresh-failed/22283
Before this we just failed if the ftp connection or login failed.
This change adds a pacer just for the ftp connect and retries if the
connection failed to Dial or the login returns a 421 error.
When using --vfs-cache-mode writes or full if a file was opened for
write intent, the modtime was set and the file was closed without
being modified the modtime would never be written back to storage.
The sequence of events
- app opens file with write intent
- app does set modtime
- rclone sets the modtime on the cache file, but not the remote file
because it is open for write and can't be set yet
- app closes the file without changing it
- rclone doesn't upload the file because the file wasn't changed so
the modtime doesn't get updated
This fixes the problem by making sure any unapplied modtime changes
are applied even if the file is not modified when being closed.
Fixes#4795
Before this change, rclone would return the modification times of the
cache file or the pending modtime which would be more accurate than
the modtime that the backend was capable of.
This meant that the modtime would be change slightly when the item was
actually uploaded.
For example modification times on Google Drive would be rounded to the
nearest millisecond.
This fixes the VFS layer to always return modtimes directly from an
object stored on the remote, or rounded to the precision that the
remote is capable of.
Before this change, rclone would set the modification time of an
object after it had been uploaded. However with --vfs-cache-mode
writes and above, the modification time of the object is already
correct as the cache backing file gets set with the correct
modification time before upload.
Setting the modification time causes another version to be created on
backends such as S3 so it should be avoided if possible.
This change checks to see if the modification time needs changing and
only sets it if necessary.
See: https://forum.rclone.org/t/produce-2-versions-when-overwrite-an-object-in-min-io/19634
Backends for which additional config is detected (in the config string
or on the command line or as environment variables) will gain a suffix
`{XXXXX}` where `XXXX` is a base64 encoded md5hash of the config
string.
This fixes backend caching with config string remotes.
This much requested feature now works properly:
rclone copy -vv drive,shared_with_me:file.txt drive:
This adds AddOverrideGetter and GetOverride methods to config map and
uses them in fs.ConfigMap.
This enables us to tell which values have been set and which are just
read from the config file or at their defaults.
This also deletes the unused AddGetters method in configmap.
This is implemented as a state machine parser so it can emit sensible
error messages.
It does not use the connection strings elsewhere in rclone yet - see
subsequent commits.
An optional fuzzer is implemented for the Parse function.
Before this change if a config was altered via the rc then when a new
backend was created from that config, if there was a backend already
running from the old config in the cache then it would be used instead
of creating a new backend with the new config, thus leading to
confusion.
This change flushes the fs cache of any backends based off a config
when that config is changed is over the rc.
Before this change, when using an all create method with one of the
upstreams being read only, if there was an existing file on the read
only remote, it was impossible to update it.
This change detects that situation and creates the file on a
read/write upstream. This file will shadow the file on the read/only
upstream. If it is deleted the read only upstream file will be visible
again.
Fixes#4929
Before this fix using the epff policy could double close a channel.
The fix refactors the code to make that impossible and cancels any
running queries when the first query is found.
Before this change, sometimes preallocate failed with EINTR which
rclone ignored.
Retrying the syscall is the correct thing to do and seems to make
preallocate 100% reliable.
Before this we were building all architectures unnecessarily in the
compile_all step for the other_os build. There are built elsewhere so
we don't need to build them here too.
This fix adds the missing BUILD_FLAGS which excludes the other builds
and should speed up the workflow.
Before this change CTRL-C could come in to exit rclone which would
start the atexit actions running. The Fuse unmount then signals rclone
to exit which wasn't waiting for the already running atexit actions to
complete.
This change makes sure that if the atexit actions are started they
should be completed.
Before this change the config file needed to be explicitly reloaded.
This coupled the config file implementation with the backends
needlessly.
This change stats the config file to see if it needs to be reloaded on
every config file operation.
This allows us to remove calls to
- config.SaveConfig
- config.GetFresh
Which now makes the the only needed interface to the config file be
that provided by configmap.Map when rclone is not being configured.
This also adds tests for configfile
It introduces a new flag --sftp-disable-concurrent-reads to stop the
problematic behaviour in the SFTP library for read-once servers.
This upgrades the sftp library to v1.13.0 which has the fix.
This change checks the context whenever rclone might retry, and
doesn't retry if the current context has an error.
This fixes the pathological behaviour of `--max-duration` refusing to
exit because all the context deadline exceeded errors were being
retried.
This unfortunately meant changing the shouldRetry logic in every
backend and doing a lot of context propagation.
See: https://forum.rclone.org/t/add-flag-to-exit-immediately-when-max-duration-reached/22723
