AzCopy Sync as a Backup Solution

AzCopy Introduces File Synchronization

As a photographer I needed an inexpensive way to protect my image files. AzCopy now has a sync option that works well. Here is how to set it up. This is the “nuts and bolts” approach to get you up and running one time. Later, I’ll show you how I schedule the synchronization.

There are 4 steps. I assume that you already have an Azure tenant and know how to create an Azure Storage Account for your Blobs. Once you have that all set up you need to do the following:

  1. Configure Security - I know, you are a Global Administrator! It doesn’t matter, to use AzCopy you need to configure security.
  2. Download AzCopy version 10.0.2 or better from GitHub
  3. Login with AzCopy (or use an alternate methods like a generated SAS key).
  4. Decide what to sync and issue the command.

Configure Security

In order for AzCopy to access Azure Blob Storage, the account performing the action must be in the Storage Blob Data Contributor (Preview) role. You can do this at different levels, for this post I am using the Resource Group level so that I can access several Storage Accounts.

  1. Locate the Resource Group for your Storage Account.
    Storage Account Resource Group
  2. For a Resource Group choose Access Control (IAM) | Add in the blade locate the role Storage Blob Data Contributor (Preview) and assign access to the Users, Groups or Roles as meets your needs.
    Access Control
  3. In the Add permissions blade, click Save to save your changes.
    Save the role assignment

Download AzCopy from GitHub

Once you download and unzip the executable, put it somewhere easy to find. In my case I add it to a “tools” folder. Then to make it easy in PowerShell, I create an alias to it.

New-Alias -Name azcopy -Value "D:\Data\Azure Tools\azcopy.exe"

Then you can test that it worked with the --help command.

azcopy --help
# or to get sync help
azcopy sync --help

Login to Azure with AzCopy

The next step will create a local OAuth login file for your PowerShell session. I tried using only azcopy -login but, for me, it only works if I specify my tenant-id.

azcopy login --tenant-id=7c7c7b9e-f745-47e0-96a4-caa015491ec1

To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code FOO12345 to authenticate.
Grant Device Access

Once you login with the same account you granted access to in the Configure Security section. You should see the prompt success in PowerShell.

Login succeeded.

I can check my access to the destination with the list command:

azcopy list "https://myprodstorage.blob.core.windows.net/backups"

List is using OAuth token for authentication.
pictures/2011/2011-10-07/Agilty Practice-10.CR2
pictures/2011/2011-10-07/Agilty Practice-10.xmp
pictures/2011/2011-10-07/Agilty Practice-11.CR2
pictures/2011/2011-10-07/Agilty Practice-11.xmp

Sync Your Files

All that is left is to decide what to sync. In my case I keep my images and sidecar files on an external drive. Each year I automatically create a new folder and have subfolders by shoot date in YYYY-MM-DD format. If I want to sync 2013 I use the following command.

#--log-level takes Log Levels: DEBUG, INFO, WARNING, ERROR, PANIC, and FATAL
azcopy sync "P:\Data\Pictures\2013\" "https://myprodstorage.blob.core.windows.net/backups/pictures/2013" --recursive=true --log-level=INFO

Using OAuth token for authentication.

Job 39425003-8a96-7a46-5293-80dcbfb44f40 has started

39425003-8a96-7a46-5293-80dcbfb44f40.log file created in C:\Users\Matthew/.azcopy
493 Done, 0 Failed, 3361 Pending, 0 Skipped, 3854 Total, 2-sec Throughput (Mb/s): 589.0365

AzCopy creates a sync job that you can interrogate with PowerShell (or just watch with rapt attention like I do). As you can see in this initial sync, AzCopy found 3854 files to sync. Once the job completes:

Job 39425003-8a96-7a46-5293-80dcbfb44f40 summary
Elapsed Time (Minutes): 21.0068
Total Number Of Transfers: 3854
Number of Transfers Completed: 3854
Number of Transfers Failed: 0
Number of Transfers Skipped: 0
TotalBytesTransferred: 60050731232
Final Job Status: Completed

If I make changes to the files and then run the command again.

azcopy sync "P:\Data\Pictures\2013\" "https://myprodstorage.blob.core.windows.net/backups/pictures/2013" --recursive=true --log-level=INFO

Using OAuth token for authentication.

Job f9193419-be1f-884b-5b92-ba77ba0c0b06 has started

f9193419-be1f-884b-5b92-ba77ba0c0b06.log file created in C:\Users\Matthew/.azcopy
0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 0

Job f9193419-be1f-884b-5b92-ba77ba0c0b06 summary
Elapsed Time (Minutes): 0.0333
Total Number Of Transfers: 1
Number of Transfers Completed: 1
Number of Transfers Failed: 0
Number of Transfers Skipped: 0
TotalBytesTransferred: 9
Final Job Status: Completed

Parting Thoughts

I am cheap. It pains me to see the prices photo sharing sites charge for storage. Most won’t store my RAW and sidecar files either. If I cannot protect the original RAW file then even free storage is not worth the effort. Now, I can automate the sync job and use the inexpensive storage provided by Azure to ensure my precious photos are safe. Plus AzCopy is a community project with AMAZING support. They worked through the authentication issues I had and got me up and running.

|| Azure || AzCopy || Azure Storage || Backup

comments powered by Disqus

Let's Get In Touch!


Ready to start your next project with us? That’s great! Give us a call or send us an email and we will get back to you as soon as possible!

+1.512.539.0322