extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2025-08-19 09:52:05 +02:00
Code Issues Packages Projects Releases Wiki Activity

Updated Using Volume Shadow Copy Service (VSS) with rclone (markdown)

albertony
2020-05-10 12:04:09 +02:00
parent 31a7ebbc38
commit 4f727011cf
1 changed files with 49 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
Using-Volume-Shadow-Copy-Service-(VSS)-with-rclone.md

@@ -10,7 +10,7 @@ we can use the description from [Microsoft TechNet Library](https://docs.microso
> This means that some of the data files might be open or they might be in an inconsistent state. > This means that some of the data files might be open or they might be in an inconsistent state.
> - If the data set is large, it can be difficult to ~~back up~~ *sync* all of it at one time. > - If the data set is large, it can be difficult to ~~back up~~ *sync* all of it at one time.
The [basic solution](#basic-use-of-vshadow-with-rclone) is quite simple, but it is disregarding error handling, which A [basic solution](#basic-use-of-vshadow-with-rclone) is quite simple, but it is disregarding error handling, which
in worst case would lead rclone to delete your files from the destination. I'm discussing some of the issues here, in worst case would lead rclone to delete your files from the destination. I'm discussing some of the issues here,
and describe improvements, so that the [complete solution](#the-resulting-version-of-execcmd-can-be-something-like-this) and describe improvements, so that the [complete solution](#the-resulting-version-of-execcmd-can-be-something-like-this)
should be quote safe to use as-is. should be quote safe to use as-is.
@@ -80,12 +80,12 @@ named according to `-script`. Now evaluating environment variable `%SHADOW_DEVIC
`-exec` script will return the device name of the temporary snapshot! So then we are close, but `-exec` script will return the device name of the temporary snapshot! So then we are close, but
the last challenge is how to actually access the shadow device. Most regular file commands and programs the last challenge is how to actually access the shadow device. Most regular file commands and programs
cannot access it by the device name directly! Some do actually: COPY can be used to copy cannot access it by the device name directly! Some do actually: COPY can be used to copy
a file directly using `copy %VSHADOW_DEVICE_1%\somefile.txt c:\somefile_bak.txt`, but DIR a file directly using `copy %VSHADOW_DEVICE_1%\somefile.txt C:\somefile_bak.txt`, but DIR
does not work etc so you will probably not get to do what you want. Luckily the built-in does not work etc so you will probably not get to do what you want. Luckily the built-in
`MKLINK` utility are able to create a directory symbolic link with the snapshot device as target, `MKLINK` utility are able to create a directory symbolic link with the snapshot device as target,
and then you can "mount" it to a regular directory path on your C: drive. and then you can "mount" it to a regular directory path on your C: drive.
One tip to test out your command step by step is to add `pause` statements in the batch One tip to test out your command step by step is to add `PAUSE` statements in the batch
script specified to the `-exec` option. Then you vshadow will suspend during execution script specified to the `-exec` option. Then you vshadow will suspend during execution
until you press a key, and you can e.g. use another Command Prompt instance to interact with until you press a key, and you can e.g. use another Command Prompt instance to interact with
temporary snapshot. temporary snapshot.
@@ -98,7 +98,7 @@ Command Prompt and access the snapshot. When you are done you just close the Not
and the snapshot is gone. and the snapshot is gone.
``` ```
vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\notepad.exe c: vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\notepad.exe C:
``` ```
Instead of notepad, you can also start a new instance of the "Command Interpreter". Instead of notepad, you can also start a new instance of the "Command Interpreter".
@@ -108,7 +108,7 @@ the command prompt. Now to resume vshadow, to let it clean up (remove the snapsh
you type `exit` to get out of the "inner" command prompt. you type `exit` to get out of the "inner" command prompt.
``` ```
vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\cmd.exe c: vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\cmd.exe C:
``` ```
Another alternative is to make the snapshot "persistent", by adding the `-p` option. Then Another alternative is to make the snapshot "persistent", by adding the `-p` option. Then
@@ -118,7 +118,7 @@ even persists across restarts, so you can play around with it as long as you wan
Create it: Create it:
``` ```
vshadow.exe -p -nw -script=setvars.cmd c: vshadow.exe -p -nw -script=setvars.cmd C:
``` ```
Load the generated script containing environment variables: Load the generated script containing environment variables:
@@ -141,22 +141,22 @@ vshadow -ds=%SHADOW_ID_1%
### Basic use of vshadow with rclone ### Basic use of vshadow with rclone
Let's say that we want to sync a directory `c:\data` to the cloud. Using rclone you would normally Let's say that we want to sync a directory `C:\Data\` to the cloud. Using rclone you would normally
execute `rclone sync c:\data remote:data`. To make this command read source files from a VSS snapshot, execute `rclone sync C:\Data\ remote:Data`. To make this command read source files from a VSS snapshot,
you can create a batch script with the following content: you can create a batch script with the following content:
``` ```
rem Load the variables from a temporary script generated by vshadow.exe rem Load the variables from a temporary script generated by vshadow.exe
call "%~dp0setvars.cmd" || exit /b 1 call "%~dp0setvars.cmd" || exit /b 1
rem Create the symbolic link to the snapshot (the backslash after shadow_device_1 is important!) rem Create the symbolic link to the snapshot (the backslash after SHADOW_DEVICE_1 is important!)
mklink /d c:\snapshot\ %shadow_device_1%\ || exit /b 1 mklink /d C:\Snapshot\ %SHADOW_DEVICE_1%\ || exit /b 1
rem Execute rclone with the source c:\snapshot\data containing a snapshot of c:\data rem Execute rclone with the source C:\Snapshot\Data containing a snapshot of C:\Data
rclone sync c:\snapshot\data\ remote:data rclone sync C:\Snapshot\Data\ remote:Data
rem Delete the symbolic link rem Delete the symbolic link
rmdir c:\snapshot\ rmdir C:\Snapshot\
rem Delete the temporary file created by vshadow.exe rem Delete the temporary file created by vshadow.exe
del "%~dp0setvars.cmd" del "%~dp0setvars.cmd"
@@ -166,7 +166,7 @@ If you save this script as `exec.cmd` you could now execute the following comman
directory: directory:
``` ```
vshadow.exe -nw -script=setvars.cmd -exec=exec.cmd c: vshadow.exe -nw -script=setvars.cmd -exec=exec.cmd C:
``` ```
Or, if you want a single-click solution; create a second batch script in the same directory, Or, if you want a single-click solution; create a second batch script in the same directory,
@@ -174,11 +174,11 @@ which executes this command - but with full paths of both file references to avo
changing working directory: changing working directory:
``` ```
vshadow.exe -nw -script="%~dp0setvars.cmd" -exec="%~dp0exec.cmd" c: vshadow.exe -nw -script="%~dp0setvars.cmd" -exec="%~dp0exec.cmd" C:
``` ```
When the vshadow.exe command is executed, possibly from your single-click wrapper script, it will: When the vshadow.exe command is executed, possibly from your single-click wrapper script, it will:
1. Create a read-only snapshot of your c: drive. 1. Create a read-only snapshot of your `C:` drive.
2. Generate a file named `setvars.cmd` containing variables identifying the snapshot created. 2. Generate a file named `setvars.cmd` containing variables identifying the snapshot created.
3. Execute `exec.cmd`, which will: 3. Execute `exec.cmd`, which will:
1. Load variables from `setvars.cmd`. 1. Load variables from `setvars.cmd`.
@@ -192,11 +192,11 @@ When the vshadow.exe command is executed, possibly from your single-click wrappe
Clarifications: Clarifications:
- Vshadow must be run with administrator privileges, so with UAC enabled you must start `vsshadow.exe` or the wrapper script using the "Run as administrator" option. - Vshadow must be run with administrator privileges, so with UAC enabled you must start `vsshadow.exe` or the wrapper script using the "Run as administrator" option.
- The path `c:\snapshot\` is a temporary symbolic link only accessible while exec.cmd is running, you will see - The path `C:\Snapshot\` is a temporary symbolic link only accessible while exec.cmd is running, you will see
it in Windows Explorer until the sync is completed. But the directory is read-only, because the it in Windows Explorer until the sync is completed. But the directory is read-only, because the
`vshadow.exe` command included the command line argument `-nw` ("no writers"). `vshadow.exe` command included the command line argument `-nw` ("no writers").
- The contents of `c:\snapshot\` that rclone sees, is a mirror image of `c:\`, meaning there will be a - The contents of `C:\Snapshot\` that rclone sees, is a mirror image of `C:\`, meaning there will be a
directory `c:\snapshot\Program Files` mirroring `c:\Program Files` etc. If this is confusing, read directory `C:\Snapshot\Program Files` mirroring `C:\Program Files` etc. If this is confusing, read
about improvements below. about improvements below.
- Upon successful return vshadow will print message "Snapshot creation done.". This just means everything - Upon successful return vshadow will print message "Snapshot creation done.". This just means everything
went well: It created the snapshot AND executed our script AND the snapshot was automatically deleted. went well: It created the snapshot AND executed our script AND the snapshot was automatically deleted.
@@ -212,32 +212,32 @@ About error handling and robustness:
assuming they are in the same directory as the script they are referenced from, and surrounding with double assuming they are in the same directory as the script they are referenced from, and surrounding with double
quotes in case there are spaces in the path. This will prevent any surprises when running from different quotes in case there are spaces in the path. This will prevent any surprises when running from different
working directories, e.g. "Run as administrator" is notorious for always setting working directory to `C:\Windows\System32`. working directories, e.g. "Run as administrator" is notorious for always setting working directory to `C:\Windows\System32`.
- The mklink command returns error when the link path (`c:\snapshot\`) already exists, and this is important to - The mklink command returns error when the link path (`C:\Snapshot\`) already exists, and this is important to
handle. If we let the script just continue, rclone will try to sync a subfolder named data (`c:\snapshot\data`). handle. If we let the script just continue, rclone will try to sync a subfolder named data (`C:\Snapshot\Data`).
If this does not exist then rclone will just abort with error, so that is ok. Worse if this path does exists, If this does not exist then rclone will just abort with error, so that is ok. Worse if this path does exists,
then rclone will actually sync it. What is normally expected in `c:\snapshot\data` is a snapshot of `c:\data`. then rclone will actually sync it. What is normally expected in `C:\Snapshot\Data` is a snapshot of `C:\Data`.
In worst case this has previously been successfully synced to `dest:data`, and after the last sync this new In worst case this has previously been successfully synced to `remote:Data`, and after the last sync this new
folder `c:\snapshot\data` has been created with content that is not at all similar to what is in `c:\data`. folder `C:\Snapshot\Data` has been created with content that is not at all similar to what is in `C:\Data`.
The result is that the current sync will end up deleting everything from `dest:data` and upload whatever is in The result is that the current sync will end up deleting everything from `remote:Data` and upload whatever is in
`c:\snapshot\data`. A lot of "ifs" here, perhaps a bit paranoid to expect it, but unless the link path is `C:\Snapshot\Data`. A lot of "ifs" here, perhaps a bit paranoid to expect it, but unless the link path is
very carefully chosen it could happen. very carefully chosen it could happen.
- The mklink command does not verify the link target, so even if the variable %shadow_device_1% contains an invalid - The mklink command does not verify the link target, so even if the variable `%SHADOW_DEVICE_1%` contains an invalid
value or for some reason is empty, the mklink command will succeed. The following rclone sync command using the value or for some reason is empty, the mklink command will succeed. The following rclone sync command using the
link as source will then simply fail (`Failed to sync: directory not found`), so this so ok. link as source will then simply fail (`Failed to sync: directory not found`), so this so ok.
### Improvements ### Improvements
It may be confusing that `c:\snapshot\` is a mirror image of `c:\`, and when referring `c:\snapshot\data` it is It may be confusing that `C:\Snapshot\` is a mirror image of `C:\`, and when referring `C:\Snapshot\Data` it is
a mirror of `c:\data`. When writing more complex scripts this confusion may easily lead to bugs. You may find a mirror of `C:\Data`. When writing more complex scripts this confusion may easily lead to bugs. You may find
it less confusing if you create an alias `t:` that you can use when referrring to the snapshot. This can easily it less confusing if you create an alias `T:` that you can use when referrring to the snapshot. This can easily
be done using the built-in `SUBST`command: Add `subst t: c:\` (error handling discussed later) before the be done using the built-in `SUBST`command: Add `subst T: C:\` (error handling discussed later) before the
`rclone sync` command and `subst t: /d` after, and change the sync command to use `t:` instead `rclone sync` command and `subst T: /d` after, and change the sync command to use `T:` instead
of `c:`: `rclone sync t:\snapshot\data\ dest:data`. of `C:`: `rclone sync T:\Snapshot\Data\ remote:Data`.
Instead of just making `t:` an alias to the entire `c:` drive, you can make it an alias directly into the Instead of just making `T:` an alias to the entire `C:` drive, you can make it an alias directly into the
snapshot image at `c:\snapshot\`. Then the paths on `c:` you would normally use with rclone directly without vss, snapshot image at `C:\Snapshot\`. Then the paths on `C:` you would normally use with rclone directly without vss,
now becomes identical when used on the snapshot `t:`. This may be even less confusing: `t:` is now the image of now becomes identical when used on the snapshot `T:`. This may be even less confusing: `T:` is now the image of
`c:`. This has the additional benefit of not increasing the path lengths of the source paths that rclone gets `C:`. This has the additional benefit of not increasing the path lengths of the source paths that rclone gets
to work with, in case that could hit some limit. to work with, in case that could hit some limit.
Proper error handling is still recommended, early abort when any of the commands fail. In addition to the original Proper error handling is still recommended, early abort when any of the commands fail. In addition to the original
@@ -257,38 +257,38 @@ the top level script (if it is a bit more complex than our `vs.cmd`).
``` ```
setlocal enabledelayedexpansion setlocal enabledelayedexpansion
call "%~dp0setvars.cmd" || set exit_code=!errorlevel!&&goto end call "%~dp0setvars.cmd" || set exit_code=!errorlevel!&&goto end
mklink /d c:\snapshot\ %shadow_device_1%\ || set exit_code=!errorlevel!&&goto end mklink /d C:\Snapshot\ %SHADOW_DEVICE_1%\ || set exit_code=!errorlevel!&&goto end
subst t:\ c:\snapshot\ || set exit_code=!errorlevel!&&goto remove_link subst T:\ C:\Snapshot\ || set exit_code=!errorlevel!&&goto remove_link
rclone sync t:\data\ dest:data rclone sync T:\Data\ remote:Data
set exit_code=%errorlevel% set exit_code=%errorlevel%
subst t: /d subst T: /d
:remove_link :remove_link
rmdir c:\snapshot\ rmdir C:\Snapshot\
:end :end
del "%~dp0setvars.cmd" del "%~dp0setvars.cmd"
exit /b %exit_code% exit /b %exit_code%
``` ```
If everything goes fine until the rclone command, then the exit code from rclone is what will be returned. If everything goes fine until the rclone command, then the exit code from rclone is what will be returned.
If the removal of virtual drive (`subst t: /d`) and directory symbolic link (`rmdir c:\snapshot\`) fails it is If the removal of virtual drive (`subst T: /d`) and directory symbolic link (`rmdir C:\Snapshot\`) fails it is
just ignored. This will lead to the drive/directory being left accessible after the script has completed, and just ignored. This will lead to the drive/directory being left accessible after the script has completed, and
if you run the script again it will abort with error because the path/drive is already in use. You could add if you run the script again it will abort with error because the path/drive is already in use. You could add
a check of the result from these two commands too. For example just write an additional warning a check of the result from these two commands too. For example just write an additional warning
(append `||ECHO WARNING: Manual cleanup required`), or also set the exit code if any of these fails (append `||echo WARNING: Manual cleanup required`), or also set the exit code if any of these fails
(append ` || set exit_code=!errorlevel!&&ECHO WARNING: Manual cleanup required`), depending if you see this (append `||set exit_code=!errorlevel!&&echo WARNING: Manual cleanup required`), depending if you see this
as something that should be reported as an error or since the rclone command passed you consider it more of as something that should be reported as an error or since the rclone command passed you consider it more of
a success. a success.
### Alternative variants ### Alternative variants
An alternative to using the SUBST command to create an alias, is to create a network share that you access An alternative to using the SUBST command to create an alias, is to create a network share that you access
via localhost. This can be done by replacing the `rclone sync c:\snapshot\data\ dest:data` line with something via localhost. This can be done by replacing the `rclone sync C:\Snapshot\Data\ remote:Data` line with something
like this (could be extended with similar error handling as above): like this (could be extended with similar error handling as above):
``` ```
net share snapshot=c:\snapshot net share Snapshot=C:\Snapshot
rclone sync \\localhost\snapshot\data\ dest:data rclone sync \\localhost\Snapshot\Data\ remote:Data
net share snapshot /delete net share Snapshot /delete
``` ```
A rather different approach could be to create the snapshot as a persistent one, using the `-p` option, as A rather different approach could be to create the snapshot as a persistent one, using the `-p` option, as