2nd Watch Named Amazon Web Services Advanced Consulting Partner

2nd Watch has been recognized by Amazon Web Services™ (AWS) as an AWS Advanced Consulting Partner. AWS reserves the Advanced Consulting Partner designation for companies like 2nd Watch that demonstrate a commitment to expert cloud solutions. We are honored to be one of the first AWS partners ever to be publicly recognized with the designation!

Backups: To the Cloud!

Amazon Web Services is adding new functionality on a weekly basis, and a growing list of 3rd party applications has emerged to leverage these new capabilities.  Many of these applications are forgettable, but one 3rd party utility has remained constant around the 2nd Watch offices: Cloudberry Explorer.  Now anyone with an internet connection and a laptop can achieve data durability which eluded most enterprises a decade ago.  Let's exploit this newfound power by creating an automated backup script!

There are three main components to this workflow:

• Timer (CRON equivalent).  For this we will use windows task scheduler.
• Scripting shell.  Since we are using windows, it is all PowerShell.
• S3 interface.  Cloudberry and its awesome PowerShell snap-in.

First, we will install Cloudberry here.  The PowerShell snap-in will install automatically on Windows 2008 or later. 

Next, we need to create a PS script which will execute our desired action.  We will start by changing PowerShell’s default execution policy to allow scripts.  Open up PowerShell and enter the command:

        Set-ExecutionPolicy RemoteSigned

Next, create a new file in c:\scripts called sync.ps1, right click on it and choose edit.  Now we will add the magic:

      $key = "YouAccessKeyIDGoesHere"
      $secret = "YourSecretAccessKeyGoesHere"
      $s3bucket = "YourS3Bucket"
      $localFolder = "c:\test\"

      Add-PSSnapin CloudBerryLab.Explorer.PSSnapIn

      $s3 = Get-CloudS3Connection -Key $key -Secret $secret
     $destination = $s3 | Select-CloudFolder -path $s3bucket
     $src = Get-CloudFilesystemConnection | Select-CloudFolder -path $localFolder
     $src | Copy-CloudSyncFolders $destination –IncludeSubfolders

As you can see in the code above, starting from the top, we have defined variables for our two AWS keys.  Next we’ve defined variables for the S3 bucket and the local folder with which we want to synchronize.  The middle line (Add-PSSnapin CloudBerryLab.Explorer.PSSnapIn ) just informs PowerShell that we’re going to be invoking Cloudberry specific commands.  Next you can see those Cloudberry specific commands in action. Line 8 establishes the connection with our AWS account.  Line 9 sets the S3 bucket as the destination.  Line 10 sets the folder “c:\test” as the source.  Line 11 synchronizes the two folders, in this case we’re telling cloudberry to compare the S3 bucket with our local folder, and if there are changes in the local folder, copy them up to the S3 bucket.  We can reverse this behavior by inverting $src and $destination on Line 11 ($src | Copy-CloudSyncFolders $destination –IncludeSubfolders)  Note that there are numerous flags which impact the way Copy-ClouSyncFolders is run, including 2 way sync, MD5 hashing (paid version only), etc.  Check out the Cloudberry docs for a full list here.

Create a batch script called BatchProcess.bat and place it in your c:\scripts folder.  Add this single line to the batch file:

      powershell.exe C:\scripts\sync.ps1

Now that we have a sync script and a batch file to call it, we need a timer to automate everything.   Go to the Control Panel -> Administrative Tools -> Task Scheduler.  Choose “Create task”

Make sure you choose “Run whether user is logged on or not” and check “Run with highest privileges”.  Also be sure to select the proper “Configure For” dropdown at the bottom, it doesn’t default to the OS you’re using.  Next select the “Triggers” tab and create a new trigger.  Change it to run every 5 minutes.  Note that you can also change the “Begin on task” to run on start-up, useful for auto-scaling groups.  Finally, go to the “Actions” tab and create a new action.  Select your BatchProcess.bat file which you created above.

Now we have a sync script which will automatically compare our local folder (c:\test) with a remote S3 bucket (YourS3Bucket) and copy any deltas up to S3.  This is a great way to automate backups.

Here are a number of helpful links:

http://www.cloudberrylab.com/?id=14

http://blog.pointbeyond.com/2010/04/23/run-powershell-script-using-windows-server-2008-task-scheduler/

http://wpstorm.net/2010/11/cloudberry-explorer-syncback-amazon-s3/

http://www.techrepublic.com/blog/10things/10-fundamental-concepts-for-powershell-scripting/2146

AWS Interviews 2nd Watch President, Jeff Aden

AWS recently interviewed Jeff Aden about how 2nd Watch is utilizing AWS to help businesses save money and be competitive on the cloud. Learn about 2nd Watch services and products, like 2W Insight, that could help your business.

AWS Tip of the Month - Free Cloud Watch Monitoring

AWS has a great monitoring service called Cloud Watch.  Cloud Watch includes basic monitoring on your infrastructure at no additional charge.  Below you can see a graph of CPU utilization across a few of our running instances:

If you add name tags to your EC2 instances Cloud Watch can be a great tool to quickly understand what your servers are doing and if you need to address any load issues or application issues.

Here is a graph of our disk IO on our mounted server storage (EBS):

The other benefits to Cloud Watch is that you can set alarms at specific thresholds (these do cost money) to trigger a notification or event based on something happening to your AWS infrastructure (e.g. CPU spike at 100% for over 2 mins).

We find that most of our clients have an existing monitoring system in place that they want to continue to use, and we integrate these into AWS.  Cloud Watch is helpful for the proof of concept, but it is also the only monitoring solution for AWS specific infrastructure (ELB, Simple DB, etc.) and can be a helpful tool in ensuring your application is running as desired.

-KB

How Cloud Computing is transforming scientific R&D!

Every once and awhile you run across something to do with Cloud Computing that makes you stop and say "Wow that is cool!".

Last week we had the privalege of being at the AWS Summit in New York to launch our beta of 2W Insight, our new billing application for AWS.  While at the summit we had a chance to listen to how innovative companies are using AWS to solve real business problems.

The most interesting of these to me was a company called Cycle Computing.  Jason Stowe from Cycle Computing was on stage during the keynote to describe the AWESOME work he and his team are doing around building supercomputers on AWS.  I've posted lots of blogs on how to run a server or utility for <$1 month on AWS and while these are neat tricks they pale in comparison to the social importance of Jason and his team's work.

Cycle Computing built a supercomputer on AWS with over 50,000 cores to do cancer research.  I will explain it terribly so please go over to their blog - blog.cyclecomputing.com and read about it yourself.  It is impressive to say the least.

Jason - keep it up!!

-Kris

2nd Watch Unveils Beta Billing Application Today

Kris and Jeff are at a summit in New York today to unveil the beta version of 2nd Watch's new cloud billing application, 2W Insight. 2W Insight will allow AWS customers to manage billing and consolidated billing environments in an easy to understand, streamlined manner. It will give customers easy visibility into their AWS bills - no more confusion in deciphering your bills!

2W Insight updates billing costs on a regular schedule, allowing accountants the most accurate billing information of any tool on the market, not just estimates. 2W Insight provides an organization's management with simple, clean and easy to understand summaries of AWS use. Services are organized into compute, storage and network categories with charges summarized by region, size and VPC. 2W Insight also enables analysts to create more detailed drilldowns of AWS charges, giving them control over how they view and organize thier bill. Users can sort and filter detailed cost and usage information, view instance level details and multiple department costs. The printing feature also allows users to print a PDF summary of bills, delivering easy reporting capabilities.

Are you an AWS Solution Provider? The application will also be available as a platform to manage customer billing. It will generate accurate bills across the consolidated billing set up, enabling solution providers to apply AWS pricing to end consumers in a consumer-friendly format.

We are very excited about the new application and plan to have a production version available in early June.

AWS Tip of the Month: Backup Server for less than $1 a month

AWS has a very neat way to backup your server with attached storage called EBS Snapshots.  This technology starts with a full snapshot of the entire volume and continues to do incremental snapshots based on the difference between the last snapshots.

Snap shotting is not a new technology, and many people use this in their on-premise environment to take snapshots of a virtual machine or specific disk for backup or copy purposes.

At AWS you cannot schedule these snapshots. You either have to call these via API or use the management console to do this.

Below is the high-level process to automate these snapshot backups at a particular time each day:

1. Build a t1.micro that has a script to run that calls the snapshot API for EBS for the volumes you want to backup.  Make sure this script runs on machine start up.
2. Setup an Auto-scale group to launch a server based on a schedule (we start ours at 3am and shut it off at 3:50 am).  This ensures time to kick off all the snapshot jobs - note: I do not have to run the server while they complete, only to kick off the snapshot and any pre-activities I need to accomplish before I can take my backup.
3. Check your snapshots to ensure they are occurring regularly and test them on a monthly or quarterly basis to ensure you can recover sufficiently in the case of an outage.

By doing this daily we have incurred the costs of our storage for snapshots and the $.02 an hour charge for our T1.micro - effectively $0.62 per month for a backup server.

-Kris

AWS Tip of the Month - Super fast Micro instance installs for less than a buck

Here is an easy but effective tip for managing micro instances in AWS.

Micro instances can be a challenge to install software or do application build upgrades due to the limited amount of CPU given to this instance size. Micro instances can make fantastic web servers, but when you need to do work on the server it can take more time than the typical IT pro has patience for.

Here is a quick tip: Change the instance size for patching and installs.

1) Stop the micro instance for your maintenance or install work.
2) Change the instance size to Large (or X-Large if you need a huge box for the install).
3) Start the instance
4) Run your install, upgrade, etc. I've found that I can usually do most of my maintenance or installs in less than an hour on a Large instance = $0.52
5) Stop the instance
6) Change the instance size back to Micro
7) Start the instance again.

The starting and stopping take seconds, and the upgrade to Large is instantaneous. Following these steps will lead to pain free patching, installs and upgrades for less than the price of a latte.

-Kris

Amazon Web Service

2nd Watch is now a solution provider for Amazon Web Service (AWS). We are excited for our customers as this means a greater level of support when leveraging AWS to expand revenues and cut IT costs. You can find us under Solution Providers at http://aws.amazon.com/solutions/solution-providers/.