News for hackers. Stuff that matters.
Re:AWS Fargate – Run Containers Without Managing Infrastructure by rm999 on Wednesday November 29, 2017 @01:16PM (#15809297)
Have you tried AWS Batch? My team moved a couple of our batch machine learning modeling jobs to it earlier this year and it's worked out great.

https://aws.amazon.com/batch/

Re:AWS Fargate – Run Containers Without Managing Infrastructure by bbgm on Wednesday November 29, 2017 @01:34PM (#15809464)
How would you use Batch + Fargate? Let’s assume Fargate is a supported compute environment in Batch.

(I run the containers org at AWS. I happen to run Batch as well)

Re:AWS Fargate – Run Containers Without Managing Infrastructure by kirillseva on Wednesday November 29, 2017 @02:32PM (#15810053)
My org is looking to move machine learning to batch as the underlying infra.

All I want is to be able to do this: 1) specify a DAG of tasks. Each task is a docker image, CMD string, CPU and memory limits 2) hit an API to run it for me. Each task runs on a new spot instance 3) be able to query this service about the state of the DAG and of each individual node

Sounds like if AWS provides an API to create a batch cluster (or whatever you call it) and lets the tasks be defined in terms of what docker image to run with what command you'll satisfy this desire

Re:AWS Fargate – Run Containers Without Managing Infrastructure by bbgm on Wednesday November 29, 2017 @03:21PM (#15810481)
That is in line with our vision for Batch; to be the engine for systems where you essentially describe a DAG and we run and hyperoptimize the execution for you. We do some of what you’re asking for but that’s great feedback around what you’d like to do.
Re:AWS Fargate – Run Containers Without Managing Infrastructure by rm999 on Thursday November 30, 2017 @11:25AM (#15816982)
Thanks for the response!

I'm curious, what would be the interaction between Batch and Fargate? Right now I use Batch to run a container and then exit out with as little thought about the underlying machine as possible. Is Fargate a push further towards serverless?

One concern I had with Fargate from the product description is around the configuration options. Our models require more than 100GB to build, but I'm seeing "Max. 30GB".

Re:AWS Fargate – Run Containers Without Managing Infrastructure by minhajuddin on Wednesday November 29, 2017 @01:58PM (#15809688)
I haven't really tried batch. But, from initial reading of documentation it didn't look like it supported running docker images. My use case requires running docker images of static site generators and that sort. Will take another look at it.
Re:AWS Fargate – Run Containers Without Managing Infrastructure by dastbe on Wednesday November 29, 2017 @03:06PM (#15810353)
See here for the details on how to define a job, specifically around running docker images on ECS

http://docs.aws.amazon.com/batch/latest/userguide/create-job...