AWS Batch

Posted on Fri 13 April 2018 in aws • Tagged with aws, boto, cloudformation, lambda, batch

A walk through of setting up AWS Batch. This simple POC demonstrates a fan in fan out workflow, AWS Batch managed environments, S3 triggers, Lambda and all the CloudFormation to hook it together. This is example of the Amazon Batch Service to deploy an example batch workflow. This example is meant to be a deployable artifact POC. This example will use CloudFormation, AWS Lambda, Docker, AWS Batch, and an S3 bucket trigger. The batch workflow created in this example is not a prescription for how batch processing should be done but merely an example. In this example all of the jobs for the workflow are scheduled at once. In doing so your scheduler could pass environment variables or change the command of each job to instruct the jobs where to store data, and where to expect data. Another approach would be to, have processing jobs schedule their dependent jobs, instructing those dependent jobs where to find the data that they produced. There are many ways batch processing can be utilized, this is simply an example.


Continue reading

cfn-boto-interface

Posted on Thu 12 April 2018 in aws • Tagged with aws, boto, cloudformation, lambda

Lambda Function that provides a direct interface to boto3 from CloudFormation as a structured object. I often find myself writing custom resources for things that either are not yet supported by CloudFormation, or for things that seem like they will never be supported, like a lookup. Often these custom resources are just a few boto3 calls that create, update, and delete a given resource. I has started to look at spot fleets for running ECS clusters after the pricing model changed in March 2018. The EC2 Launch Template was not supported at that time and neither was the ability to use EC2 Launch Templates with a Spot Fleet built by CloudFormation. With that challenge I figured I had a couple custom resources to build and I thought about how I could abstract that as much as possible, and I think I found it. I decided to build a direct interface to boto3 through CloudFormation.


Continue reading