EC2 Container Service In Action

We announced the Amazon EC2 Container Service at AWS re:Invent and invited you to join the preview. Since that time, we've seen a lot of interest and a correspondingly high signup rate for the preview. With the year winding down, I thought it would be fun to spend a morning putting the service through its paces. We have already approved all existing requests to join the preview; new requests are currently being approved within 24 hours.

As I noted in my earlier post, this new service will help you to build, run, and scale Docker-based applications. You'll benefit from easy cluster management, high performance, flexible scheduling, extensibility, portability, and AWS integration while running in an AWS-powered environment that is secure and efficient.

Quick Container Review
Before I dive in, let's take a minute to review some of the terminology and core concepts implemented by the Container Service.

• Cluster - A logical grouping of Container Instances that is used to run Tasks.
• Container Instance - An EC2 instance that runs the ECS Container Agent and that has been registered into a Cluster. The set of instances running within a Cluster create a pool of resources that can be used to run Tasks.
• Task Definition - A description of a set of Containers. The information contained in a Task Description defines one or more Containers. All of the Containers defined in a particular Task Definition are run on the same Container Instance.
• Task - An instantiation of a Task Definition.
• Container - A Docker container that was created as part of a Task.

The ECS Container Agent runs on Container Instances. It is responsible for starting Containers on behalf of ECS. The agent itself runs within a Docker container (available on Docker Hub) and communicates with the Docker daemon running on the Instance.

When talking about a cluster or container service, "scheduling" refers to the process of assigning tasks to instances. The Container Service provides you with three scheduling options:

1. Automated - The RunTask function will start a Task (as specified by a Task Definition) on a Cluster using random placement.
2. Manual - The StartTaskfunction will start a Task (again, as specified by a Task Definition) on a specified Container Instance (or Instances).
3. Custom - You can use the ListContainerInstances and DescribeContainerInstances functions to gather information about available resources within a Cluster, implement the "brain" of the schedule (in other words, use the available information to choose a suitable Container Instance), and then call StartTask to start a task on the Instance. When you do this you are, in effect, creating your own implementation of RunTask.

EC2 Container Service in Action
In order to gain some first-hand experience with ECS, I registered for the preview and then downloaded, installed, and configured a preview version of the AWS CLI. Then I created an IAM Role and a VPC and set about to create my cluster (ECS is currently available in US East (Northern Virginia) with support for other Regions expected in time). I ran the following command:

$ aws ecs create-cluster --cluster-name MyCluster --profile jbarr-cli

The command returned information about my new cluster as a block of JSON:

{
    "cluster": {
        "clusterName": "MyCluster", 
        "status": "ACTIVE", 
        "clusterArn": "arn:aws:ecs:us-east-1:348414629041:cluster/MyCluster"
    }
}

Then I launched a couple of EC2 instances into my VPC using an ECS-enabled AMI that had been shared with me as part of the preview process (this is a very lightweight version of the Amazon Linux AMI, optimized and tuned for ECS). I chose my new IAM Role (ecs) as part of the launch process:

I also edited the instance's User Data to make the instance launch in to my cluster:

After the instances launched I was able to see that they were part of my cluster:

$ aws ecs list-container-instances --cluster MyCluster --profile jbarr-cli

{
    "containerInstanceArns": [
        "arn:aws:ecs:us-east-1:348414629041:container-instance/4cf62484-da62-49a5-ad32-2015286a6d39", 
        "arn:aws:ecs:us-east-1:348414629041:container-instance/be672053-0ff8-4478-b136-7fae9225e493"
    ]
}

I can choose an instance and query it to find out more about the registered and available CPU and memory resources:

$ aws ecs describe-container-instances --cluster MyCluster \
  --container-instances arn:aws:ecs:us-east-1:348414629041:container-instance/4cf62484-da62-49a5-ad32-2015286a6d39 \
  --profile jbarr-cli

Here's an excerpt from the returned data:

{
            "registeredResources": [
                {
                    "integerValue": 1024, 
                    "longValue": 0, 
                    "type": "INTEGER", 
                    "name": "CPU", 
                    "doubleValue": 0.0
                }, 
                {
                    "integerValue": 3768, 
                    "longValue": 0, 
                    "type": "INTEGER", 
                    "name": "MEMORY", 
                    "doubleValue": 0.0
                }
            ]
}

Following the directions in the Container Service Developer Guide, I created a simple task definition and registered it:

$ aws ecs register-task-definition --family sleep360 \
  --container-definitions file://$HOME/tmp/task.json \
  --profile jbarr-cli

Then I ran 10 copies of the task:

aws ecs run-task --cluster MyCluster --task-definition sleep360:1 --count 10 --profile jbarr-cli

And I listed the running tasks:

$ aws ecs list-tasks --cluster MyCluster --profile jbarr-cli

This is what I saw:

{
    "taskArns": [
        "arn:aws:ecs:us-east-1:348414629041:task/0c949733-862c-4979-b5bd-d4f8b474c58e", 
        "arn:aws:ecs:us-east-1:348414629041:task/3ababde9-08dc-4fc9-b005-be5723d1d495", 
        "arn:aws:ecs:us-east-1:348414629041:task/602e13d2-681e-4c87-a1d9-74c139f7335e", 
        "arn:aws:ecs:us-east-1:348414629041:task/6d072f42-75da-4a84-8b68-4841fdfe600d", 
        "arn:aws:ecs:us-east-1:348414629041:task/6da6c947-8071-4111-9d31-b87b8b93cc53", 
        "arn:aws:ecs:us-east-1:348414629041:task/6ec9828a-cbfb-4a39-b491-7b7705113ad2", 
        "arn:aws:ecs:us-east-1:348414629041:task/87e29ab2-34be-4495-988b-c93ac1f8b77c", 
        "arn:aws:ecs:us-east-1:348414629041:task/ad4fc3cc-7e80-4681-b858-68ff46716fe5", 
        "arn:aws:ecs:us-east-1:348414629041:task/cdd221ea-837c-4108-9577-2e4f53376c12", 
        "arn:aws:ecs:us-east-1:348414629041:task/eab79263-087f-43d3-ae4c-1a89678c7101"
    ]
}

I spent some time describing the tasks and wrapped up by shutting down the instances. After going through all of this (and making a mistake or two along the way due to being so eager to get a cluster up and running), I'll leave you with three simple reminders:

1. Make sure that your VPC has external connectivity enabled.
2. Make sure to use the proper, ECS-enabled AMI.
3. Make sure to launch the AMI with the requisite IAM Role.

ECS Quickstart Template
We have created an ECS Quickstart Template for CloudFormation to help you to get up and running even more quickly. The template creates an IAM Role and an Instance Profile for the Role. The Role supplies the permission that allows the ECS Agent to communicate with ECS. The template launches an instance using the Role and returns an SSH command that can be used to access the instance. You can launch the instance in to an existing cluster, or you can use the name "default" to create (if necessary) a default cluster. The instance is always launched within your Default VPC.

Contain Yourself
If you would like to get started with ECS, just register now and we'll get you up and running as soon as possible.

To learn more about ECS, spend 30 minutes watching this session from re:Invent (one caveat: the video is already a bit dated; for example, Task Definitions are no longer versioned):

You can also register for our upcoming (January 14^th, 2015) webinar, Amazon EC2 Container Service Deep Dive. In this webinar, my colleague Deepak Singh will will talk about why we built EC2 Container Service, explain some of the core concepts, and show you how to use the service for your applications.

CoreOS is a new Linux distribution designed to support the needs of modern infrastructure stacks. The CoreOS AMI now supports ECS; you can read the Amazon ECS on CoreOS documentation to learn more.

As always, we are interested in your feedback. With ECS still in preview mode, now is the perfect time for you to let us know more about your needs. You can post your feedback to the ECS Forum. you can also create AWS Support cases if you are in need of assistance.

-- Jeff;