AzCopy Sync as a Backup Solution
As a photographer I needed an inexpensive way to protect my image files. AzCopy now has a sync option that works well. Here is how to set it up. This is the “nuts and bolts” approach to get you up and running one time. Later, I’ll show you how I schedule the synchronization.
There are 4 steps. I assume that you already have an Azure tenant and know how to create an Azure Storage Account for your Blobs. Once you have that all set up you need to do the following:
In order for AzCopy to access Azure Blob Storage, the account performing the action must be in the Storage Blob Data Contributor (Preview) role. You can do this at different levels, for this post I am using the Resource Group level so that I can access several Storage Accounts.
Once you download and unzip the executable, put it somewhere easy to find. In my case I add it to a “tools” folder. Then to make it easy in PowerShell, I create an alias to it.
New-Alias -Name azcopy -Value "D:\Data\Azure Tools\azcopy.exe"
Then you can test that it worked with the --help
command.
azcopy --help
# or to get sync help
azcopy sync --help
The next step will create a local OAuth login file for your PowerShell session. I tried using only azcopy -login
but, for me, it only works if I specify my tenant-id
.
azcopy login --tenant-id=7c7c7b9e-f745-47e0-96a4-caa015491ec1
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code FOO12345 to authenticate.
Once you login with the same account you granted access to in the Configure Security section. You should see the prompt success in PowerShell.
Login succeeded.
I can check my access to the destination with the list
command:
azcopy list "https://myprodstorage.blob.core.windows.net/backups"
List is using OAuth token for authentication.
pictures/2011/2011-10-07/Agilty Practice-10.CR2
pictures/2011/2011-10-07/Agilty Practice-10.xmp
pictures/2011/2011-10-07/Agilty Practice-11.CR2
pictures/2011/2011-10-07/Agilty Practice-11.xmp
All that is left is to decide what to sync. In my case I keep my images and sidecar files on an external drive. Each year I automatically create a new folder and have subfolders by shoot date in YYYY-MM-DD
format. If I want to sync 2013 I use the following command.
#--log-level takes Log Levels: DEBUG, INFO, WARNING, ERROR, PANIC, and FATAL
azcopy sync "P:\Data\Pictures\2013\" "https://myprodstorage.blob.core.windows.net/backups/pictures/2013" --recursive=true --log-level=INFO
Using OAuth token for authentication.
Job 39425003-8a96-7a46-5293-80dcbfb44f40 has started
39425003-8a96-7a46-5293-80dcbfb44f40.log file created in C:\Users\Matthew/.azcopy
493 Done, 0 Failed, 3361 Pending, 0 Skipped, 3854 Total, 2-sec Throughput (Mb/s): 589.0365
AzCopy creates a sync job that you can interrogate with PowerShell (or just watch with rapt attention like I do). As you can see in this initial sync, AzCopy found 3854 files to sync. Once the job completes:
Job 39425003-8a96-7a46-5293-80dcbfb44f40 summary
Elapsed Time (Minutes): 21.0068
Total Number Of Transfers: 3854
Number of Transfers Completed: 3854
Number of Transfers Failed: 0
Number of Transfers Skipped: 0
TotalBytesTransferred: 60050731232
Final Job Status: Completed
If I make changes to the files and then run the command again.
azcopy sync "P:\Data\Pictures\2013\" "https://myprodstorage.blob.core.windows.net/backups/pictures/2013" --recursive=true --log-level=INFO
Using OAuth token for authentication.
Job f9193419-be1f-884b-5b92-ba77ba0c0b06 has started
f9193419-be1f-884b-5b92-ba77ba0c0b06.log file created in C:\Users\Matthew/.azcopy
0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 0
Job f9193419-be1f-884b-5b92-ba77ba0c0b06 summary
Elapsed Time (Minutes): 0.0333
Total Number Of Transfers: 1
Number of Transfers Completed: 1
Number of Transfers Failed: 0
Number of Transfers Skipped: 0
TotalBytesTransferred: 9
Final Job Status: Completed
I am cheap. It pains me to see the prices photo sharing sites charge for storage. Most won’t store my RAW and sidecar files either. If I cannot protect the original RAW file then even free storage is not worth the effort. Now, I can automate the sync job and use the inexpensive storage provided by Azure to ensure my precious photos are safe. Plus AzCopy is a community project with AMAZING support. They worked through the authentication issues I had and got me up and running.
Ready to start your next project with us? That’s great! Give us a call or send us an email and we will get back to you as soon as possible!
+1.512.539.0322