mirror of
https://github.com/rclone/rclone.git
synced 2025-08-19 09:52:05 +02:00
Updated Using Volume Shadow Copy Service (VSS) with rclone (markdown)
albertony
@@ -10,7 +10,7 @@ we can use the description from [Microsoft TechNet Library](https://docs.microso
|
||||
> This means that some of the data files might be open or they might be in an inconsistent state.
|
||||
> - If the data set is large, it can be difficult to ~~back up~~ *sync* all of it at one time.
|
||||
|
||||
The [basic solution](#basic-use-of-vshadow-with-rclone) is quite simple, but it is disregarding error handling, which
|
||||
A [basic solution](#basic-use-of-vshadow-with-rclone) is quite simple, but it is disregarding error handling, which
|
||||
in worst case would lead rclone to delete your files from the destination. I'm discussing some of the issues here,
|
||||
and describe improvements, so that the [complete solution](#the-resulting-version-of-execcmd-can-be-something-like-this)
|
||||
should be quote safe to use as-is.
|
||||
@@ -80,12 +80,12 @@ named according to `-script`. Now evaluating environment variable `%SHADOW_DEVIC
|
||||
`-exec` script will return the device name of the temporary snapshot! So then we are close, but
|
||||
the last challenge is how to actually access the shadow device. Most regular file commands and programs
|
||||
cannot access it by the device name directly! Some do actually: COPY can be used to copy
|
||||
a file directly using `copy %VSHADOW_DEVICE_1%\somefile.txt c:\somefile_bak.txt`, but DIR
|
||||
a file directly using `copy %VSHADOW_DEVICE_1%\somefile.txt C:\somefile_bak.txt`, but DIR
|
||||
does not work etc so you will probably not get to do what you want. Luckily the built-in
|
||||
`MKLINK` utility are able to create a directory symbolic link with the snapshot device as target,
|
||||
and then you can "mount" it to a regular directory path on your C: drive.
|
||||
|
||||
One tip to test out your command step by step is to add `pause` statements in the batch
|
||||
One tip to test out your command step by step is to add `PAUSE` statements in the batch
|
||||
script specified to the `-exec` option. Then you vshadow will suspend during execution
|
||||
until you press a key, and you can e.g. use another Command Prompt instance to interact with
|
||||
temporary snapshot.
|
||||
@@ -98,7 +98,7 @@ Command Prompt and access the snapshot. When you are done you just close the Not
|
||||
and the snapshot is gone.
|
||||
|
||||
```
|
||||
vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\notepad.exe c:
|
||||
vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\notepad.exe C:
|
||||
```
|
||||
|
||||
Instead of notepad, you can also start a new instance of the "Command Interpreter".
|
||||
@@ -108,7 +108,7 @@ the command prompt. Now to resume vshadow, to let it clean up (remove the snapsh
|
||||
you type `exit` to get out of the "inner" command prompt.
|
||||
|
||||
```
|
||||
vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\cmd.exe c:
|
||||
vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\cmd.exe C:
|
||||
```
|
||||
|
||||
Another alternative is to make the snapshot "persistent", by adding the `-p` option. Then
|
||||
@@ -118,7 +118,7 @@ even persists across restarts, so you can play around with it as long as you wan
|
||||
Create it:
|
||||
|
||||
```
|
||||
vshadow.exe -p -nw -script=setvars.cmd c:
|
||||
vshadow.exe -p -nw -script=setvars.cmd C:
|
||||
```
|
||||
|
||||
Load the generated script containing environment variables:
|
||||
@@ -141,22 +141,22 @@ vshadow -ds=%SHADOW_ID_1%
|
||||
|
||||
### Basic use of vshadow with rclone
|
||||
|
||||
Let's say that we want to sync a directory `c:\data` to the cloud. Using rclone you would normally
|
||||
execute `rclone sync c:\data remote:data`. To make this command read source files from a VSS snapshot,
|
||||
Let's say that we want to sync a directory `C:\Data\` to the cloud. Using rclone you would normally
|
||||
execute `rclone sync C:\Data\ remote:Data`. To make this command read source files from a VSS snapshot,
|
||||
you can create a batch script with the following content:
|
||||
|
||||
```
|
||||
rem Load the variables from a temporary script generated by vshadow.exe
|
||||
call "%~dp0setvars.cmd" || exit /b 1
|
||||
|
||||
rem Create the symbolic link to the snapshot (the backslash after shadow_device_1 is important!)
|
||||
mklink /d c:\snapshot\ %shadow_device_1%\ || exit /b 1
|
||||
rem Create the symbolic link to the snapshot (the backslash after SHADOW_DEVICE_1 is important!)
|
||||
mklink /d C:\Snapshot\ %SHADOW_DEVICE_1%\ || exit /b 1
|
||||
|
||||
rem Execute rclone with the source c:\snapshot\data containing a snapshot of c:\data
|
||||
rclone sync c:\snapshot\data\ remote:data
|
||||
rem Execute rclone with the source C:\Snapshot\Data containing a snapshot of C:\Data
|
||||
rclone sync C:\Snapshot\Data\ remote:Data
|
||||
|
||||
rem Delete the symbolic link
|
||||
rmdir c:\snapshot\
|
||||
rmdir C:\Snapshot\
|
||||
|
||||
rem Delete the temporary file created by vshadow.exe
|
||||
del "%~dp0setvars.cmd"
|
||||
@@ -166,7 +166,7 @@ If you save this script as `exec.cmd` you could now execute the following comman
|
||||
directory:
|
||||
|
||||
```
|
||||
vshadow.exe -nw -script=setvars.cmd -exec=exec.cmd c:
|
||||
vshadow.exe -nw -script=setvars.cmd -exec=exec.cmd C:
|
||||
```
|
||||
|
||||
Or, if you want a single-click solution; create a second batch script in the same directory,
|
||||
@@ -174,11 +174,11 @@ which executes this command - but with full paths of both file references to avo
|
||||
changing working directory:
|
||||
|
||||
```
|
||||
vshadow.exe -nw -script="%~dp0setvars.cmd" -exec="%~dp0exec.cmd" c:
|
||||
vshadow.exe -nw -script="%~dp0setvars.cmd" -exec="%~dp0exec.cmd" C:
|
||||
```
|
||||
|
||||
When the vshadow.exe command is executed, possibly from your single-click wrapper script, it will:
|
||||
1. Create a read-only snapshot of your c: drive.
|
||||
1. Create a read-only snapshot of your `C:` drive.
|
||||
2. Generate a file named `setvars.cmd` containing variables identifying the snapshot created.
|
||||
3. Execute `exec.cmd`, which will:
|
||||
1. Load variables from `setvars.cmd`.
|
||||
@@ -192,11 +192,11 @@ When the vshadow.exe command is executed, possibly from your single-click wrappe
|
||||
|
||||
Clarifications:
|
||||
- Vshadow must be run with administrator privileges, so with UAC enabled you must start `vsshadow.exe` or the wrapper script using the "Run as administrator" option.
|
||||
- The path `c:\snapshot\` is a temporary symbolic link only accessible while exec.cmd is running, you will see
|
||||
- The path `C:\Snapshot\` is a temporary symbolic link only accessible while exec.cmd is running, you will see
|
||||
it in Windows Explorer until the sync is completed. But the directory is read-only, because the
|
||||
`vshadow.exe` command included the command line argument `-nw` ("no writers").
|
||||
- The contents of `c:\snapshot\` that rclone sees, is a mirror image of `c:\`, meaning there will be a
|
||||
directory `c:\snapshot\Program Files` mirroring `c:\Program Files` etc. If this is confusing, read
|
||||
- The contents of `C:\Snapshot\` that rclone sees, is a mirror image of `C:\`, meaning there will be a
|
||||
directory `C:\Snapshot\Program Files` mirroring `C:\Program Files` etc. If this is confusing, read
|
||||
about improvements below.
|
||||
- Upon successful return vshadow will print message "Snapshot creation done.". This just means everything
|
||||
went well: It created the snapshot AND executed our script AND the snapshot was automatically deleted.
|
||||
@@ -212,32 +212,32 @@ About error handling and robustness:
|
||||
assuming they are in the same directory as the script they are referenced from, and surrounding with double
|
||||
quotes in case there are spaces in the path. This will prevent any surprises when running from different
|
||||
working directories, e.g. "Run as administrator" is notorious for always setting working directory to `C:\Windows\System32`.
|
||||
- The mklink command returns error when the link path (`c:\snapshot\`) already exists, and this is important to
|
||||
handle. If we let the script just continue, rclone will try to sync a subfolder named data (`c:\snapshot\data`).
|
||||
- The mklink command returns error when the link path (`C:\Snapshot\`) already exists, and this is important to
|
||||
handle. If we let the script just continue, rclone will try to sync a subfolder named data (`C:\Snapshot\Data`).
|
||||
If this does not exist then rclone will just abort with error, so that is ok. Worse if this path does exists,
|
||||
then rclone will actually sync it. What is normally expected in `c:\snapshot\data` is a snapshot of `c:\data`.
|
||||
In worst case this has previously been successfully synced to `dest:data`, and after the last sync this new
|
||||
folder `c:\snapshot\data` has been created with content that is not at all similar to what is in `c:\data`.
|
||||
The result is that the current sync will end up deleting everything from `dest:data` and upload whatever is in
|
||||
`c:\snapshot\data`. A lot of "ifs" here, perhaps a bit paranoid to expect it, but unless the link path is
|
||||
then rclone will actually sync it. What is normally expected in `C:\Snapshot\Data` is a snapshot of `C:\Data`.
|
||||
In worst case this has previously been successfully synced to `remote:Data`, and after the last sync this new
|
||||
folder `C:\Snapshot\Data` has been created with content that is not at all similar to what is in `C:\Data`.
|
||||
The result is that the current sync will end up deleting everything from `remote:Data` and upload whatever is in
|
||||
`C:\Snapshot\Data`. A lot of "ifs" here, perhaps a bit paranoid to expect it, but unless the link path is
|
||||
very carefully chosen it could happen.
|
||||
- The mklink command does not verify the link target, so even if the variable %shadow_device_1% contains an invalid
|
||||
- The mklink command does not verify the link target, so even if the variable `%SHADOW_DEVICE_1%` contains an invalid
|
||||
value or for some reason is empty, the mklink command will succeed. The following rclone sync command using the
|
||||
link as source will then simply fail (`Failed to sync: directory not found`), so this so ok.
|
||||
|
||||
### Improvements
|
||||
|
||||
It may be confusing that `c:\snapshot\` is a mirror image of `c:\`, and when referring `c:\snapshot\data` it is
|
||||
a mirror of `c:\data`. When writing more complex scripts this confusion may easily lead to bugs. You may find
|
||||
it less confusing if you create an alias `t:` that you can use when referrring to the snapshot. This can easily
|
||||
be done using the built-in `SUBST`command: Add `subst t: c:\` (error handling discussed later) before the
|
||||
`rclone sync` command and `subst t: /d` after, and change the sync command to use `t:` instead
|
||||
of `c:`: `rclone sync t:\snapshot\data\ dest:data`.
|
||||
It may be confusing that `C:\Snapshot\` is a mirror image of `C:\`, and when referring `C:\Snapshot\Data` it is
|
||||
a mirror of `C:\Data`. When writing more complex scripts this confusion may easily lead to bugs. You may find
|
||||
it less confusing if you create an alias `T:` that you can use when referrring to the snapshot. This can easily
|
||||
be done using the built-in `SUBST`command: Add `subst T: C:\` (error handling discussed later) before the
|
||||
`rclone sync` command and `subst T: /d` after, and change the sync command to use `T:` instead
|
||||
of `C:`: `rclone sync T:\Snapshot\Data\ remote:Data`.
|
||||
|
||||
Instead of just making `t:` an alias to the entire `c:` drive, you can make it an alias directly into the
|
||||
snapshot image at `c:\snapshot\`. Then the paths on `c:` you would normally use with rclone directly without vss,
|
||||
now becomes identical when used on the snapshot `t:`. This may be even less confusing: `t:` is now the image of
|
||||
`c:`. This has the additional benefit of not increasing the path lengths of the source paths that rclone gets
|
||||
Instead of just making `T:` an alias to the entire `C:` drive, you can make it an alias directly into the
|
||||
snapshot image at `C:\Snapshot\`. Then the paths on `C:` you would normally use with rclone directly without vss,
|
||||
now becomes identical when used on the snapshot `T:`. This may be even less confusing: `T:` is now the image of
|
||||
`C:`. This has the additional benefit of not increasing the path lengths of the source paths that rclone gets
|
||||
to work with, in case that could hit some limit.
|
||||
|
||||
Proper error handling is still recommended, early abort when any of the commands fail. In addition to the original
|
||||
@@ -257,38 +257,38 @@ the top level script (if it is a bit more complex than our `vs.cmd`).
|
||||
```
|
||||
setlocal enabledelayedexpansion
|
||||
call "%~dp0setvars.cmd" || set exit_code=!errorlevel!&&goto end
|
||||
mklink /d c:\snapshot\ %shadow_device_1%\ || set exit_code=!errorlevel!&&goto end
|
||||
subst t:\ c:\snapshot\ || set exit_code=!errorlevel!&&goto remove_link
|
||||
rclone sync t:\data\ dest:data
|
||||
mklink /d C:\Snapshot\ %SHADOW_DEVICE_1%\ || set exit_code=!errorlevel!&&goto end
|
||||
subst T:\ C:\Snapshot\ || set exit_code=!errorlevel!&&goto remove_link
|
||||
rclone sync T:\Data\ remote:Data
|
||||
set exit_code=%errorlevel%
|
||||
subst t: /d
|
||||
subst T: /d
|
||||
:remove_link
|
||||
rmdir c:\snapshot\
|
||||
rmdir C:\Snapshot\
|
||||
:end
|
||||
del "%~dp0setvars.cmd"
|
||||
exit /b %exit_code%
|
||||
```
|
||||
|
||||
If everything goes fine until the rclone command, then the exit code from rclone is what will be returned.
|
||||
If the removal of virtual drive (`subst t: /d`) and directory symbolic link (`rmdir c:\snapshot\`) fails it is
|
||||
If the removal of virtual drive (`subst T: /d`) and directory symbolic link (`rmdir C:\Snapshot\`) fails it is
|
||||
just ignored. This will lead to the drive/directory being left accessible after the script has completed, and
|
||||
if you run the script again it will abort with error because the path/drive is already in use. You could add
|
||||
a check of the result from these two commands too. For example just write an additional warning
|
||||
(append `||ECHO WARNING: Manual cleanup required`), or also set the exit code if any of these fails
|
||||
(append ` || set exit_code=!errorlevel!&&ECHO WARNING: Manual cleanup required`), depending if you see this
|
||||
(append `||echo WARNING: Manual cleanup required`), or also set the exit code if any of these fails
|
||||
(append `||set exit_code=!errorlevel!&&echo WARNING: Manual cleanup required`), depending if you see this
|
||||
as something that should be reported as an error or since the rclone command passed you consider it more of
|
||||
a success.
|
||||
|
||||
### Alternative variants
|
||||
|
||||
An alternative to using the SUBST command to create an alias, is to create a network share that you access
|
||||
via localhost. This can be done by replacing the `rclone sync c:\snapshot\data\ dest:data` line with something
|
||||
via localhost. This can be done by replacing the `rclone sync C:\Snapshot\Data\ remote:Data` line with something
|
||||
like this (could be extended with similar error handling as above):
|
||||
|
||||
```
|
||||
net share snapshot=c:\snapshot
|
||||
rclone sync \\localhost\snapshot\data\ dest:data
|
||||
net share snapshot /delete
|
||||
net share Snapshot=C:\Snapshot
|
||||
rclone sync \\localhost\Snapshot\Data\ remote:Data
|
||||
net share Snapshot /delete
|
||||
```
|
||||
|
||||
A rather different approach could be to create the snapshot as a persistent one, using the `-p` option, as
|
||||
|
||||
Reference in New Issue
Block a user