AWS Batch

Posted on Fri 13 April 2018 in aws • Tagged with aws, boto, cloudformation, lambda, batch

A walk through of setting up AWS Batch. This simple POC demonstrates a fan in fan out workflow, AWS Batch managed environments, S3 triggers, Lambda and all the CloudFormation to hook it together. This is example of the Amazon Batch Service to deploy an example batch workflow. This example is meant to be a deployable artifact POC. This example will use CloudFormation, AWS Lambda, Docker, AWS Batch, and an S3 bucket trigger. The batch workflow created in this example is not a prescription for how batch processing should be done but merely an example. In this example all of the jobs for the workflow are scheduled at once. In doing so your scheduler could pass environment variables or change the command of each job to instruct the jobs where to store data, and where to expect data. Another approach would be to, have processing jobs schedule their dependent jobs, instructing those dependent jobs where to find the data that they produced. There are many ways batch processing can be utilized, this is simply an example.


Continue reading