Welcome New AWS Community Heroes

Earlier this year we welcomed the first AWS Community Heroes. As I wrote at the time, these hard-working folks were selected for the program because they routinely deliver high-quality, impactful, developer-focused activities to the AWS Community.

More Heroes
Today I would like to welcome eight new heroes!

Java developer Satoshi Yokota is the Founder & CEO of Classmethod, a writer for Developers.IO, and a founding member of JAWS-UG. In 2010 he participated in the kickoff meeting of JAWS-UG and has help to expand it to more than 40 branches. JAWS-UG currently features more than 100 leaders, over 100 meetup events, over 1000 members.

---

Brazilian developer Heitor Vital has been working with mobile games/applications and web development for over 10 years. He blogs on developer-oriented AWS topics and recently earned an Executive MBA. As CTO of Site Blindado SA, his work focuses on security, cloud computing, and infrastructure.

---

Based in Berlin, Chad Fowler is CTO of Wunderlist. As a leading voice on the topic of cloud-based deployment techniques, Chad coined the term "immutable infrastructure." Chad has been host and organizer of many technology conferences including the International Ruby Conference & Railsconf.

---

Norm Driskell is a public sector digital leader. He focuses on digital transformation, transparency, and open source. From his base in London, Norm founded the AWS UK User Group in 2012, with events every two months and sponsorship from leading tech and media companies.

---

Victor Oliveira is a founding partner of Concrete Solutions. As Director of Engineering for this Brazilian company, he runs Cloud Computing and Agile practices. Concrete Solutions was the first AWS Partner in Latin America, due in large part to Victor's efforts. Today he helps clients turn infrastructure in to code while also keeping a watchful eye out for optimizations that can reduce costs without sacrificing quality.

---

After leading technical operations at Pinterest, Ryan Park became Principal Engineer for Runscope. There, he leads the design and development effort for their AWS-powered debugging and testing service. Ryan speaks on the topic of application design and is a mentor for companies in the 500 Startups incubator.

---

As a founding member of the engineering team at Bizo (recently acquired by LinkedIn), Larry Ogrodnek began using AWS way back in 2008! He adopted additional AWS services as they came along, and enjoys shutting down homegrown alternatives in the process. Larry is the co-founder and organizer of the Advanced AWS Meetup Group in San Francisco.

---

Lynn Langit consults on big data and cloud architecture from her base in Southern California. She has designed production-quality AWS solutions and also delivers technical content (also related to AWS) at developer conferences all over the world. Lynn creates technical AWS screencasts and is the primary courseware author for Teaching Kids Programming.

Welcome!
Please join me in welcoming these new AWS Community Heroes to the roster!

-- Jeff;

Rapid Auto Scaling with Amazon SQS

Earlier this month an AWS user named Andy emailed the following question to me:

We're interested in using the number of items in our sqs queue as criteria for autoscaling our ec2 workers. The 5 minute delay is really way too long for this application, it would have to keep track of the queue in close to real time, 1 minute or less. Is this possible?

Well, Andy, that's a great question! In fact, the SQS team asked me to write about that very topic earlier this month. Thanks for providing me with the perfect introduction to this blog post.

Defining the Problem
Before jumping in, let's spend a few minute defining the problem that Andy faces. As you can read in the Scaling Based on Amazon SQS tutorial in the Auto Scaling documentation, you can use the number of messages stored in an SQS queue as an indicator of the amount of work that is waiting in line for eventual processing within an Auto Scaling Group comprised of a variable number of EC2 instances. Each SQS queue reports a number of metrics to CloudWatch at five minute intervals, including ApproximateNumberOfMessagesVisible. If your workload is spikey in nature, you may want to build an application that can respond more quickly to changes in the size of the queue.

You may also have an application that pulls work from multiple queues distributed across two or more AWS Regions. Since the metrics are collected on a per-queue, per-Region basis, additional work is needed in order to create a metric that accurately reflects the actual amount of pending work.

Solving the Problem
You can address either (or both) of these challenges by using a custom CloudWatch metric. The finished solution will use a pair of Auto Scaling Groups. The first Group (which I will call the Checker) will periodically check the queue depth and publish it as a custom metric. The second Group (the Worker) will do the actual processing of the messages in the queue, scaling up and down using the information provided by the custom metric.

The Checker Group exists in order to keep a single, modestly-sized EC2 instance up and running (a t2.micro will probably suffice), and to launch replacements if necessary (set the minimum and maximum number of instances to 1 for the Group). This instance periodically runs some simple code that is responsible for pulling the ApproximateNumberOfMessagesVisible metrics from one or more queues and publishing the results to a custom CloudWatch metric. Here are a couple of code snippets to get you started. The first step is to query SQS to retrieve the metric:

public static final String APPROXIMATE_NUMBER_OF_MESSAGES_ATT = "ApproximateNumberOfMessages";
public static final String NAMESPACE = "QueueDepthNamespace";

Map<String, String> attributes = sqs.getQueueAttributes(queueUrl, attributeList).getAttributes();
double approximateNumOfMsg = Double.parseDouble(attributes.get(APPROXIMATE_NUMBER_OF_MESSAGES_ATT));

And the second step is to publish the metric:

MetricDatum md = new MetricDatum()
  .withMetricName(queueName + "-OneMinute-" + APPROXIMATE_NUMBER_OF_MESSAGES_ATT)
  .withUnit(StandardUnit.Count)
  .withValue(approximateNumOfMsg);
 
cloudWatch.putMetricData(new PutMetricDataRequest()
  .withNamespace(NAMESPACE)
  .withMetricData(md));

Although the queue's metric is updated at one minute intervals, you should plan on collecting and publishing several samples per minute. This will provide enough data points to allow Auto Scaling to respond to changes more quickly and more accurately. For example, if you sample and publish every 10 seconds, each one-minute metric will reflect the average of six samples. Before choosing a sampling rate, be sure to do some calculations in order to make sure that you understand the per-call costs for the SQS and CloudWatch API functions. The metrics will be visible in the CloudWatch console:

The Worker Group will vary in size in response to changes in the custom metric published by the EC2 instance running in the Checker Group. To do this, you must associate CloudWatch alarms with the custom metrics. For example, you could use three alarms, configured as follows (you'll need to fine-tune these to the needs of your application):

1. MyQueue1-QueueDepthPassLowThreshold - When MyQueue1 accumulates a message backlog of 1000 messages, increase the size of the Worker Group by one instance.
2. MyQueue1-QueueDepthPassHighThreshold - When MyQueue1 accumulates a message backlog of 1500 messages, increase the size of the Worker Group by two instances.
3. MyQueue1-QueueDepthBelowSafeThreshold - When the number of messages in MyQueue1 is below 100, reduce the size of the Work Group by one instance.

Here's how these alarms would look in the console:

Learning More
To learn more, read my blog post Additional CloudWatch Metrics for Amazon SQS and Amazon SNS and the documentation on Monitoring Amazon SQS With CloudWatch.

-- Jeff;

AWS Week in Review - December 23, 2014

Let's take a quick look at what happened in AWS-land last week:

Monday, December 22	We announced that Amazon CloudSearch now supports Lean and Flexible Configuration with Dynamic Fields. A post on the N2WS Blog reviewed EC2 Virtualization Options and the Move from PV to HVM. Weekend posts on AWS Advent talked about Exploring AWS Lambda and Using Terraform to Build Infrastructure on AWS.
Tuesday, December 23	João Parreira wrote a nice article on Using Atomic Counters to Resolve Concurrency Conflicts in Android Game. We announced that Amazon Zocalo now Support Shared Folder Sync. We launched Version 0.0.17 of cfncluster (a framework that deploys and maintains HPC clusters on AWS). A post on the Big Door Blog discussed A Gift from AWS: Improved API Latency at Lower Costs. Our friends at edX announced that you can Get $1000 of Amazon Web Services free when you earn a Verified Certificate in Entrepreneurship 101 or Entrepreneurship 102. A post on the AWS Security Blog showed you How to Use External ID When Granting Access to Your AWS Resources. A post on AWS Advent discussed some CloudFormation woes and advised readers to Keep Calm and use Ansible.
Wednesday, December 24	A post on the AdRoll Tech Blog described Hologram, Taking EC2 Instance Roles Everywhere (related Hacker News discussion). We updated CloudFormation with Expanded Support for Amazon RDS and Elastic Load Balancing. A post on the CloudAcademy Blog reviewed AWS Cloud Design Patterns. A post on AWS Advent showed you how to build Repeatable Infrastructure with CloudFormation and YAML.
Friday, December 26	Developer Ted Timmons created an RSS Feed for AWS Service Release Notes.

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

-- Jeff;

Billions Saved with Prime?

Amazon put out a press release yesterday with the headline: Free Shipping Saved Amazon Customers Billions This Holiday Season While I have to admit that I personally paid a lot less for shipping than I would have without my Prime account, that savings had to come from somewhere, and I’d bet a lot … Continue reading Billions Saved with Prime? →

Another Record for Amazon

Once again, Amazon has achieved its goal of record-breaking growth at the expense of making very little (on-the-books) profit. Amazon uses several accounting tricks to bury the exact amount of profit, such as massive R&D expenses, so it’s a bit hard to tell just how much money they made. I suspect that someday … Continue reading Another Record for Amazon →

Amazon Zocalo Update - Shared Folder Syncing

Amazon Zocalo is a fully managed file storage and sharing service. It provides secure, enterprise-ready storage and strong administrative controls, along with built-in feedback capabilities. Zocalo organizes content into folders and allows individual items and entire folders to be shared with other users. The Zocalo sync client runs on your Windows or Mac desktop and synchronizes your local Zocalo My Documents folder with the copies in the cloud.

New Shared Folder Syncing
Today we are enhancing the Zocalo sync client with the ability to synchronize content that has been shared with you via a shared folder. You can set up per-project folders and share them with members of the project team on an as-needed basis. The project team can view and edit the documents on their chosen device; new versions are automatically saved to Zocalo. For example, I use a Zocalo shared folder to distribute draft blog posts to my colleagues for review.

Shaerd folder syncing is an optional feature and you must explicitly enable it for existing installations. If you are installing a new client, you can choose to enable syncing of shared folders as part of the installation and registration process.

Enabling Syncing for an Existing Client
You can enable syncing of shared folders for an existing installation of the Zocalo sync client by opening up the Zocalo Preferences:

Click on Enable Shared Folder Sync:

The folders that have been shared with me will then show up in Zocalo's Shared With Me folder on my desktop:

Enabling Syncing During Installation
You can also enable syncing of shared folders when you install the Zocalo sync client. Simply check Sync all files and folders from Zocalo in the Setup screen at the appropriate point in the registration process:

You can pause syncing at any time. Your Shared With Me folder will remain intact and will be brought up to date when you once again turn on shared folder syncing.

Available Now
This new feature is available now and you can start using it today!

-- Jeff;

AWS Week in Review - December 15, 2014

Let's take a quick look at what happened in AWS-land last week:

Monday, December 15	We announced that AWS OpsWorks Supports Custom Shutdown Timeouts and Elastic Load Balancing Connection Draining. A cool post on the 2lemetry Technical Blog talked about Machine Learning With Amazon Lambda. A post on the Cloudability Blog introduced New Cloudability tools for Enterprise Management. We announced that AWS Storage Gateway API Calls are now Available in AWS CloudTrail. We announced that VM Import for Amazon EC2 is now Available in the AWS GovCloud (US) Region. We announced that the AWS Management Console in the AWS GovCloud (US) Region now Supports Auto Scaling. We announced that AWS OpsWorks now Supports Custom Shutdown Timeouts and Elastic Load Balancing Connection Draining. A very detailed and informative post on the AWS Architecture Blog talked about Internet Routing and Traffic Engineering. A guest post on the AWS Big Data Blog talked about Building and Running a Recommendation Engine at Any Scale. A post on the AWS Partner Network Blog took a look at the re:Invent 2014 Launches from an APN Partner Perspective. Another post on the AWS Partner Network Blog looked at Four AWS Partners Coming Together for AWS Customers: Cloudnexa, AppDynamics, Sumo Logic, and Trend Micro. A post on AWS Advent talked about Finding AWS Resources Across Regions, Services, and Accounts with skew. A post on the Cloud Academy Blog talked about Architecting on AWS: Optimizing the Application Design.
Tuesday, December 16	We announced New Amazon CloudFront Reporting to help you to learn more about your visitors. A helpful Quora discussion addressed the question What can you do with Amazon Kinesis? We announced that Amazon CloudFront Now Allows a Directory Path as an Origin Name. A post on the AWS Windows and .NET Developer Blog announced the Preview Release of AWS Resource APIs for .NET. A post on AWS Advent talked about Managing EC2 Security Groups Using Puppet.
Wednesday, December 17	We announced that AWS CodeDeploy now Supports AWS CloudTrail. I showed the EC2 Container Service in Action. A post on the AWS Mobile Development Blog unveiled the AWS Mobile SDK for Unity Developer Preview. A second post on the blog talked about Using Amazon Cognito to Sync Data in Unity Games. We introduced Resource Groups and Tag Editing for AWS. We announced that Amazon Mobile Analytics can now Auto Export App Data to S3. We announced that you can now Assign Priority Levels to Tasks in Amazon SWF Workflows. A post on the 2lemetry Technical Blog talked about Sensor Data Loading for Force.com with AWS Lambda. A helpful post from Sylvain Perron stated that Scaling is Getting Easy (this is a paean to AWS Lambda). A post on AWS Advent took a Quick Look at AWS CodeDeploy.
Thursday, December 18	We announced that Amazon CloudWatch Logs are now Available in Three Asia Pacific Regions and in the Europe (Germany) Region. We launched the AWS Compute Blog with a post on Compute Content at re:Invent 2014. A post on the AWS Partner Network Blog took an advance look at the Microsoft Exchange Server 2013 Quick Start Reference Deployment (the reference deployment was made available to APN Partners one month ahead of general release). We announced that you can Easily Get Started with Auto Scaling from the AWS Management Console. A post on the AWS Application Management Blog showed you how to Automatically Deploy from GitHub Using AWS CodeDeploy. A post on the AWS Startup Collection showed you how to Simplify Code Deployments with AWS CodeDeploy. The AWS Windows and .NET Developer Blog unveiled the New AWS Elastic Beanstalk Deployment Wizard for the AWS Toolkit for Visual Studio. The AWS Windows and .NET Developer Blog offered Amazon EC2 ImageUtilities and Get-EC2ImageByName Updates. A post on the 2lemetry Technical Blog talked about Streaming MQTT to Redshift. A post on the 2nd Watch Blog asked (and answered) the question What’s Important to Enterprises Managing Workloads in the Cloud? A post on the Cloud Academy Blog showed you how to Deploy Application Code from S3 Using AWS CodeDeploy.
Friday, December 19	We launched an AWS Console Mobile App Update with Support for Elastic Beanstalk. A post on the AWS Java Blog revealed the AWS Resource APIs for SNS and SQS. A very cool post on the Blue Collar Bioinformatics Blog talked about Benchmarking Variation and RNA-seq Analyses on Amazon Web Services with Docker. A post on the CloudCheckr Blog introduced Advanced Alerting Ties to AWS Config and CloudTrail. A post from GigaOm told the story of How NASA Launched its Web Infrastructure into the Cloud. A post on the Cloud Academy Blog invited you to Learn AWS Lambda with our New Course.

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

-- Jeff;

AWS Console Mobile App Update - Support for Elastic Beanstalk

We have updated the AWS Console mobile app with support for AWS Elastic Beanstalk. I'll let the app's update notice tell you what's new:

Let's take a quick look at the new features! The main screen of the app includes a new Beanstalk Applications section:

I can see all of my Elastic Beanstalk applications:

From there I can zoom and see the Elastic Beanstalk environments for any desired application:

Diving even deeper, I can focus on a particular environment:

I can open up individual sections to see what's going on. Here's the Configuration section:

If a CloudWatch alarm fires, I can open up the CloudWatch Alarms section to see what's going on:

I can also take a detailed look at a particular CloudWatch metric:

I can also perform certain operations on the environment:

For example, I can deploy any desired version of the application to the environment:

Download & Install Now
The new version of the AWS Console mobile app is available now and you can start using it today. Here are the download links:

-- Jeff;