When one gets the chance to go to New York, one takes it, as far as I’m concerned. So when PayByPhone offered to send me to the AWS NYC Summit, I totally jumped at the chance. In addition to getting to stand on top of two of the world’s tallest buildings, take a bike across the Brooklyn Bridge, and get some decent Times Square shots, I got to learn quite a bit about ops using AWS. Win-win!

The AWS NYC summit was short, but definitely one of the better conferences that I have been to. I did the Taking AWS Operations to the Next Level “boot camp” – a day-long training course – during the pre-day, and attended several of the breakouts the next day. All of them had great value and I took quite a few notes. I am going to try and abridge all of my takeaways on the products, “old” and new, that caught my eye below.

CloudFormation

CloudFormation was a product that was covered in my pre-day course and also during one of the breakouts that I attended. It’s probably the most robust ops product that is offered on AWS today, supporting, from my impressions, the most products versus any other of the automation platform services that are offered.

The idea with CloudFormation, of course, is that infrastructure is codified in a JSON-based template, and then create “stacks” – entities that group infrastructure and platform services up in ways that can be duplicated, destroyed, or even updated with a sort of intelligent convergence, adding, removing, or changing resources depending on what has been defined in the template. Naturally, this can be integrated with any sort of source control so that changes are tracked, and a CI and deployment pipeline to further automate things.

One of the neat features that was mentioned in the CloudFormation breakout was the ability for CloudFormation to use Lambda-backed resources to interface with AWS features that CloudFormation does not have native support for, or even non-AWS products. All of this makes CloudFormation definitely seem like the go-to product if one is planning on using native tools to deploy AWS infrastructure.

OpsWorks

OpsWorks is Amazon’s Chef product, although it’s a quite a bit more than that. It seems mainly like a hybrid of a deployment system like CloudFormation, with Chef being used to manage configuration through several points of the lifecycle. It uses chef-solo and chef-zero depending on the OS it is being employed for (Linux is chef-solo and Chef 11, and Windows is Chef 12 and chef-zero), and since it is all run locally, there is no Chef server.

In OpsWorks, an application stack is deployed using components called Layers. Layers exist for load balancing, application servers and databases, in addition to others such as caching and even custom ones that can utilize functionality that are created through Chef cookbooks. With support for even some basic monitoring, one can probably write an entire application in OpsWorks without even touching another AWS ops tool.

AWS API Gateway

A few new products were announced at the summit – but API Gateway was the one killer app that caught my eye. Ultimately this means that developers do not really need to mess around with frameworks any more to get an API, or even get a web application off the ground – just hook in the endpoints with API gateway, integrate them with Lambda, and it’s done. With the way that AWS’s platform portfolio is looking these days, I’m surprised that this one was so late the party!

CodeDeploy, CodePipeLine, and CodeCommit

These were presented to me in a breakout that gave a bit of a timeline on how Amazon internally developed their own deployment pipelines. Ultimately they segued into these three tools.

CodeDeploy is designed to deploy an application to not only AWS, but also to on-premise resources, and even other cloud providers. The configuration is YAML-based and pretty easy to read. As part of its deployment feature set, it does offer some orchestration facilities, so there is some overlap with some of its other tools. It also integrates with several kinds of source control platforms (ie: GitHub), other CI tools (ie: Jenkins), and configuration management systems. The killer features for me is its support for rolling updates, to automate the deployment of a new infrastructure while draining out the old one.

CodePipeline is a release modeling and workflow engine, and can be used to model a release process, working with CodeDeploy, to automate the deployment of an application from source, to testing, and then to production. Tests can be automated using services like RunScope or Ghost Inspector, to name just a couple. It definitely seems like these two – CodePipeline and CodeDeploy – are naturally coupled to give a very smooth software deployment pipeline – on AWS especially.

CodeCommit is AWS’s foray into source control, a la GitHub, Bitbucket, etc. Aside all the general things that one would hopefully expect from a hosted source control service (ie: at-rest encryption and high availability), expect a few extra AWS-ish perks, like the ability to use existing IAM scheme to assign ACLs to repositories. Unlimited storage per repository was mentioned, or at least implied, but there does appear to be some metering – see here for pricing.

EC2 Container Service (ECS)

The last breakout I checked out was one on EC2 Container Service (ECS). This is AWS’s Docker integration. The breakout itself spent a bit of time on an intro to containers themselves which is out of the scope of this article, but is a topic I may touch up on at a later time (Docker is on “the list” of things I want to evaluate and write on). Docker is a great concept. It rolls configuration management and containerization into one tool for deploying applications and gives a huge return on infrastructure in the form of speed and density. The one unanswered question has been clustering, but there has been several 3rd party solutions for that for a bit and Docker themselves has a solution that is still fairly new.

ECS does not appear to be Swarm, but its own mechanism. But in addition to clustering, ECS works with other AWS services, such as Elastic Load Balancing (ELB), Elastic Block Storage (EBS), and back office things like IAM and CloudTrail. Templates are rolled into entities called tasks, where one can also specify resource requirements, volumes to use, and more. Then, one can use this task to create a run task, which will run for a finite amount of time and then terminate, or a service, which will ensure that the task stays up and running indefinitely. Also, the ability exists to specify a specific instance count for a task, which is then spread out across the instance pool.

This is probably a good time to mention that ECS still does not provide abstraction across container infrastructure. There is a bit of automation to help with this, but ECS2 instances still need to be manually spun up and added to the ECS pool, from which it then derives the available amount of resources and how it distributes the containers. One would assume that there are plans to eliminate the visibility of the EC2 layer from the user – it seems like Amazon is aware of requests to do this, as was mentioned when I asked the question.

Ultimately, it looks like there is still some work to do to make ECS completely integrated. Auto-scaling is another feature that is still a manual process. For now, there are documented examples on how to do things like glue auto-scaling together using Lambda, for example.

Wrap-Up

You can see all – or at least most – of the presentations and breakouts that happened at the summit (and plenty of other conferences) – on the AWS YouTube page.

And as for me, expect to see, over the next several weeks, a product-by-product review of as much of AWS as I can eat. I will be starting with some of the products that were discussed above, with a mix of some of the old standbys in as well to ensure I cover things from the ground up. Watch this space!