From 4f727011cff030cbb27264f8909b4555e92d377b Mon Sep 17 00:00:00 2001 From: albertony <12441419+albertony@users.noreply.github.com> Date: Sun, 10 May 2020 12:04:09 +0200 Subject: [PATCH] Updated Using Volume Shadow Copy Service (VSS) with rclone (markdown) --- ...e-Shadow-Copy-Service-(VSS)-with-rclone.md | 98 +++++++++---------- 1 file changed, 49 insertions(+), 49 deletions(-) diff --git a/Using-Volume-Shadow-Copy-Service-(VSS)-with-rclone.md b/Using-Volume-Shadow-Copy-Service-(VSS)-with-rclone.md index 23360a7..57992e8 100644 --- a/Using-Volume-Shadow-Copy-Service-(VSS)-with-rclone.md +++ b/Using-Volume-Shadow-Copy-Service-(VSS)-with-rclone.md @@ -10,7 +10,7 @@ we can use the description from [Microsoft TechNet Library](https://docs.microso > This means that some of the data files might be open or they might be in an inconsistent state. > - If the data set is large, it can be difficult to ~~back up~~ *sync* all of it at one time. -The [basic solution](#basic-use-of-vshadow-with-rclone) is quite simple, but it is disregarding error handling, which +A [basic solution](#basic-use-of-vshadow-with-rclone) is quite simple, but it is disregarding error handling, which in worst case would lead rclone to delete your files from the destination. I'm discussing some of the issues here, and describe improvements, so that the [complete solution](#the-resulting-version-of-execcmd-can-be-something-like-this) should be quote safe to use as-is. @@ -80,12 +80,12 @@ named according to `-script`. Now evaluating environment variable `%SHADOW_DEVIC `-exec` script will return the device name of the temporary snapshot! So then we are close, but the last challenge is how to actually access the shadow device. Most regular file commands and programs cannot access it by the device name directly! Some do actually: COPY can be used to copy -a file directly using `copy %VSHADOW_DEVICE_1%\somefile.txt c:\somefile_bak.txt`, but DIR +a file directly using `copy %VSHADOW_DEVICE_1%\somefile.txt C:\somefile_bak.txt`, but DIR does not work etc so you will probably not get to do what you want. Luckily the built-in `MKLINK` utility are able to create a directory symbolic link with the snapshot device as target, and then you can "mount" it to a regular directory path on your C: drive. -One tip to test out your command step by step is to add `pause` statements in the batch +One tip to test out your command step by step is to add `PAUSE` statements in the batch script specified to the `-exec` option. Then you vshadow will suspend during execution until you press a key, and you can e.g. use another Command Prompt instance to interact with temporary snapshot. @@ -98,7 +98,7 @@ Command Prompt and access the snapshot. When you are done you just close the Not and the snapshot is gone. ``` -vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\notepad.exe c: +vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\notepad.exe C: ``` Instead of notepad, you can also start a new instance of the "Command Interpreter". @@ -108,7 +108,7 @@ the command prompt. Now to resume vshadow, to let it clean up (remove the snapsh you type `exit` to get out of the "inner" command prompt. ``` -vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\cmd.exe c: +vshadow.exe -nw -script=setvars.cmd -exec=C:\Windows\System32\cmd.exe C: ``` Another alternative is to make the snapshot "persistent", by adding the `-p` option. Then @@ -118,7 +118,7 @@ even persists across restarts, so you can play around with it as long as you wan Create it: ``` -vshadow.exe -p -nw -script=setvars.cmd c: +vshadow.exe -p -nw -script=setvars.cmd C: ``` Load the generated script containing environment variables: @@ -141,22 +141,22 @@ vshadow -ds=%SHADOW_ID_1% ### Basic use of vshadow with rclone -Let's say that we want to sync a directory `c:\data` to the cloud. Using rclone you would normally -execute `rclone sync c:\data remote:data`. To make this command read source files from a VSS snapshot, +Let's say that we want to sync a directory `C:\Data\` to the cloud. Using rclone you would normally +execute `rclone sync C:\Data\ remote:Data`. To make this command read source files from a VSS snapshot, you can create a batch script with the following content: ``` rem Load the variables from a temporary script generated by vshadow.exe call "%~dp0setvars.cmd" || exit /b 1 -rem Create the symbolic link to the snapshot (the backslash after shadow_device_1 is important!) -mklink /d c:\snapshot\ %shadow_device_1%\ || exit /b 1 +rem Create the symbolic link to the snapshot (the backslash after SHADOW_DEVICE_1 is important!) +mklink /d C:\Snapshot\ %SHADOW_DEVICE_1%\ || exit /b 1 -rem Execute rclone with the source c:\snapshot\data containing a snapshot of c:\data -rclone sync c:\snapshot\data\ remote:data +rem Execute rclone with the source C:\Snapshot\Data containing a snapshot of C:\Data +rclone sync C:\Snapshot\Data\ remote:Data rem Delete the symbolic link -rmdir c:\snapshot\ +rmdir C:\Snapshot\ rem Delete the temporary file created by vshadow.exe del "%~dp0setvars.cmd" @@ -166,7 +166,7 @@ If you save this script as `exec.cmd` you could now execute the following comman directory: ``` -vshadow.exe -nw -script=setvars.cmd -exec=exec.cmd c: +vshadow.exe -nw -script=setvars.cmd -exec=exec.cmd C: ``` Or, if you want a single-click solution; create a second batch script in the same directory, @@ -174,11 +174,11 @@ which executes this command - but with full paths of both file references to avo changing working directory: ``` -vshadow.exe -nw -script="%~dp0setvars.cmd" -exec="%~dp0exec.cmd" c: +vshadow.exe -nw -script="%~dp0setvars.cmd" -exec="%~dp0exec.cmd" C: ``` When the vshadow.exe command is executed, possibly from your single-click wrapper script, it will: -1. Create a read-only snapshot of your c: drive. +1. Create a read-only snapshot of your `C:` drive. 2. Generate a file named `setvars.cmd` containing variables identifying the snapshot created. 3. Execute `exec.cmd`, which will: 1. Load variables from `setvars.cmd`. @@ -192,11 +192,11 @@ When the vshadow.exe command is executed, possibly from your single-click wrappe Clarifications: - Vshadow must be run with administrator privileges, so with UAC enabled you must start `vsshadow.exe` or the wrapper script using the "Run as administrator" option. -- The path `c:\snapshot\` is a temporary symbolic link only accessible while exec.cmd is running, you will see +- The path `C:\Snapshot\` is a temporary symbolic link only accessible while exec.cmd is running, you will see it in Windows Explorer until the sync is completed. But the directory is read-only, because the `vshadow.exe` command included the command line argument `-nw` ("no writers"). -- The contents of `c:\snapshot\` that rclone sees, is a mirror image of `c:\`, meaning there will be a -directory `c:\snapshot\Program Files` mirroring `c:\Program Files` etc. If this is confusing, read +- The contents of `C:\Snapshot\` that rclone sees, is a mirror image of `C:\`, meaning there will be a +directory `C:\Snapshot\Program Files` mirroring `C:\Program Files` etc. If this is confusing, read about improvements below. - Upon successful return vshadow will print message "Snapshot creation done.". This just means everything went well: It created the snapshot AND executed our script AND the snapshot was automatically deleted. @@ -212,32 +212,32 @@ About error handling and robustness: assuming they are in the same directory as the script they are referenced from, and surrounding with double quotes in case there are spaces in the path. This will prevent any surprises when running from different working directories, e.g. "Run as administrator" is notorious for always setting working directory to `C:\Windows\System32`. -- The mklink command returns error when the link path (`c:\snapshot\`) already exists, and this is important to -handle. If we let the script just continue, rclone will try to sync a subfolder named data (`c:\snapshot\data`). +- The mklink command returns error when the link path (`C:\Snapshot\`) already exists, and this is important to +handle. If we let the script just continue, rclone will try to sync a subfolder named data (`C:\Snapshot\Data`). If this does not exist then rclone will just abort with error, so that is ok. Worse if this path does exists, -then rclone will actually sync it. What is normally expected in `c:\snapshot\data` is a snapshot of `c:\data`. -In worst case this has previously been successfully synced to `dest:data`, and after the last sync this new -folder `c:\snapshot\data` has been created with content that is not at all similar to what is in `c:\data`. -The result is that the current sync will end up deleting everything from `dest:data` and upload whatever is in -`c:\snapshot\data`. A lot of "ifs" here, perhaps a bit paranoid to expect it, but unless the link path is +then rclone will actually sync it. What is normally expected in `C:\Snapshot\Data` is a snapshot of `C:\Data`. +In worst case this has previously been successfully synced to `remote:Data`, and after the last sync this new +folder `C:\Snapshot\Data` has been created with content that is not at all similar to what is in `C:\Data`. +The result is that the current sync will end up deleting everything from `remote:Data` and upload whatever is in +`C:\Snapshot\Data`. A lot of "ifs" here, perhaps a bit paranoid to expect it, but unless the link path is very carefully chosen it could happen. -- The mklink command does not verify the link target, so even if the variable %shadow_device_1% contains an invalid +- The mklink command does not verify the link target, so even if the variable `%SHADOW_DEVICE_1%` contains an invalid value or for some reason is empty, the mklink command will succeed. The following rclone sync command using the link as source will then simply fail (`Failed to sync: directory not found`), so this so ok. ### Improvements -It may be confusing that `c:\snapshot\` is a mirror image of `c:\`, and when referring `c:\snapshot\data` it is -a mirror of `c:\data`. When writing more complex scripts this confusion may easily lead to bugs. You may find -it less confusing if you create an alias `t:` that you can use when referrring to the snapshot. This can easily -be done using the built-in `SUBST`command: Add `subst t: c:\` (error handling discussed later) before the -`rclone sync` command and `subst t: /d` after, and change the sync command to use `t:` instead -of `c:`: `rclone sync t:\snapshot\data\ dest:data`. +It may be confusing that `C:\Snapshot\` is a mirror image of `C:\`, and when referring `C:\Snapshot\Data` it is +a mirror of `C:\Data`. When writing more complex scripts this confusion may easily lead to bugs. You may find +it less confusing if you create an alias `T:` that you can use when referrring to the snapshot. This can easily +be done using the built-in `SUBST`command: Add `subst T: C:\` (error handling discussed later) before the +`rclone sync` command and `subst T: /d` after, and change the sync command to use `T:` instead +of `C:`: `rclone sync T:\Snapshot\Data\ remote:Data`. -Instead of just making `t:` an alias to the entire `c:` drive, you can make it an alias directly into the -snapshot image at `c:\snapshot\`. Then the paths on `c:` you would normally use with rclone directly without vss, -now becomes identical when used on the snapshot `t:`. This may be even less confusing: `t:` is now the image of -`c:`. This has the additional benefit of not increasing the path lengths of the source paths that rclone gets +Instead of just making `T:` an alias to the entire `C:` drive, you can make it an alias directly into the +snapshot image at `C:\Snapshot\`. Then the paths on `C:` you would normally use with rclone directly without vss, +now becomes identical when used on the snapshot `T:`. This may be even less confusing: `T:` is now the image of +`C:`. This has the additional benefit of not increasing the path lengths of the source paths that rclone gets to work with, in case that could hit some limit. Proper error handling is still recommended, early abort when any of the commands fail. In addition to the original @@ -257,38 +257,38 @@ the top level script (if it is a bit more complex than our `vs.cmd`). ``` setlocal enabledelayedexpansion call "%~dp0setvars.cmd" || set exit_code=!errorlevel!&&goto end -mklink /d c:\snapshot\ %shadow_device_1%\ || set exit_code=!errorlevel!&&goto end -subst t:\ c:\snapshot\ || set exit_code=!errorlevel!&&goto remove_link -rclone sync t:\data\ dest:data +mklink /d C:\Snapshot\ %SHADOW_DEVICE_1%\ || set exit_code=!errorlevel!&&goto end +subst T:\ C:\Snapshot\ || set exit_code=!errorlevel!&&goto remove_link +rclone sync T:\Data\ remote:Data set exit_code=%errorlevel% -subst t: /d +subst T: /d :remove_link -rmdir c:\snapshot\ +rmdir C:\Snapshot\ :end del "%~dp0setvars.cmd" exit /b %exit_code% ``` If everything goes fine until the rclone command, then the exit code from rclone is what will be returned. -If the removal of virtual drive (`subst t: /d`) and directory symbolic link (`rmdir c:\snapshot\`) fails it is +If the removal of virtual drive (`subst T: /d`) and directory symbolic link (`rmdir C:\Snapshot\`) fails it is just ignored. This will lead to the drive/directory being left accessible after the script has completed, and if you run the script again it will abort with error because the path/drive is already in use. You could add a check of the result from these two commands too. For example just write an additional warning -(append `||ECHO WARNING: Manual cleanup required`), or also set the exit code if any of these fails -(append ` || set exit_code=!errorlevel!&&ECHO WARNING: Manual cleanup required`), depending if you see this +(append `||echo WARNING: Manual cleanup required`), or also set the exit code if any of these fails +(append `||set exit_code=!errorlevel!&&echo WARNING: Manual cleanup required`), depending if you see this as something that should be reported as an error or since the rclone command passed you consider it more of a success. ### Alternative variants An alternative to using the SUBST command to create an alias, is to create a network share that you access -via localhost. This can be done by replacing the `rclone sync c:\snapshot\data\ dest:data` line with something +via localhost. This can be done by replacing the `rclone sync C:\Snapshot\Data\ remote:Data` line with something like this (could be extended with similar error handling as above): ``` -net share snapshot=c:\snapshot -rclone sync \\localhost\snapshot\data\ dest:data -net share snapshot /delete +net share Snapshot=C:\Snapshot +rclone sync \\localhost\Snapshot\Data\ remote:Data +net share Snapshot /delete ``` A rather different approach could be to create the snapshot as a persistent one, using the `-p` option, as