Make Mine Cloud: Backups: To the Cloud!

Amazon Web Services is adding new functionality on a weekly basis, and a growing list of 3rd party applications has emerged to leverage these new capabilities. Many of these applications are forgettable, but one 3rd party utility has remained constant around the 2nd Watch offices: Cloudberry Explorer. Now anyone with an internet connection and a laptop can achieve data durability which eluded most enterprises a decade ago. Let's exploit this newfound power by creating an automated backup script!

There are three main components to this workflow:

Timer (CRON equivalent). For this we will use windows task scheduler.
Scripting shell. Since we are using windows, it is all PowerShell.
S3 interface. Cloudberry and its awesome PowerShell snap-in.

First, we will install Cloudberry here. The PowerShell snap-in will install automatically on Windows 2008 or later.

Next, we need to create a PS script which will execute our desired action. We will start by changing PowerShell’s default execution policy to allow scripts. Open up PowerShell and enter the command:

Set-ExecutionPolicy RemoteSigned

Next, create a new file in c:\scripts called sync.ps1, right click on it and choose edit. Now we will add the magic:

      $key = "YouAccessKeyIDGoesHere"
      $secret = "YourSecretAccessKeyGoesHere"
      $s3bucket = "YourS3Bucket"
      $localFolder = "c:\test\"

      Add-PSSnapin CloudBerryLab.Explorer.PSSnapIn

      $s3 = Get-CloudS3Connection -Key $key -Secret $secret
     $destination = $s3 | Select-CloudFolder -path $s3bucket
     $src = Get-CloudFilesystemConnection | Select-CloudFolder -path $localFolder
     $src | Copy-CloudSyncFolders $destination –IncludeSubfolders

As you can see in the code above, starting from the top, we have defined variables for our two AWS keys. Next we’ve defined variables for the S3 bucket and the local folder with which we want to synchronize. The middle line (Add-PSSnapin CloudBerryLab.Explorer.PSSnapIn ) just informs PowerShell that we’re going to be invoking Cloudberry specific commands. Next you can see those Cloudberry specific commands in action. Line 8 establishes the connection with our AWS account. Line 9 sets the S3 bucket as the destination. Line 10 sets the folder “c:\test” as the source. Line 11 synchronizes the two folders, in this case we’re telling cloudberry to compare the S3 bucket with our local folder, and if there are changes in the local folder, copy them up to the S3 bucket. We can reverse this behavior by inverting $src and $destination on Line 11 ($src | Copy-CloudSyncFolders $destination –IncludeSubfolders) Note that there are numerous flags which impact the way Copy-ClouSyncFolders is run, including 2 way sync, MD5 hashing (paid version only), etc. Check out the Cloudberry docs for a full list here.

Create a batch script called BatchProcess.bat and place it in your c:\scripts folder. Add this single line to the batch file:

powershell.exe C:\scripts\sync.ps1

Now that we have a sync script and a batch file to call it, we need a timer to automate everything. Go to the Control Panel -> Administrative Tools -> Task Scheduler. Choose “Create task”

Make sure you choose “Run whether user is logged on or not” and check “Run with highest privileges”. Also be sure to select the proper “Configure For” dropdown at the bottom, it doesn’t default to the OS you’re using. Next select the “Triggers” tab and create a new trigger. Change it to run every 5 minutes. Note that you can also change the “Begin on task” to run on start-up, useful for auto-scaling groups. Finally, go to the “Actions” tab and create a new action. Select your BatchProcess.bat file which you created above.

Now we have a sync script which will automatically compare our local folder (c:\test) with a remote S3 bucket (YourS3Bucket) and copy any deltas up to S3. This is a great way to automate backups.

Here are a number of helpful links:

http://www.cloudberrylab.com/?id=14

http://blog.pointbeyond.com/2010/04/23/run-powershell-script-using-windows-server-2008-task-scheduler/

http://wpstorm.net/2010/11/cloudberry-explorer-syncback-amazon-s3/

http://www.techrepublic.com/blog/10things/10-fundamental-concepts-for-powershell-scripting/2146