The only caveat for those considering taking this path - AWS has an account-wide limit of 100 concurrent lambdas running at once. There's no way to know how many are currently running unless you keep track of it yourself. You'll only find out when you try to kick off a job and it get's throttled. I haven't contacted support yet to find out how hard it is for the limit to get raised or what they'll raise it to.
It's not technological - it's about ensuring they have a reasonable handle on capacity and what they happen to have plenty of.
It's not a problem as long as you're not asking for astronomical limits, but people need to be aware of it and request increases in advance of starting to hit limits rather than assume it'll be near immediate to get them lifted.
The guts to make all that work was 50 or so lines of config. I think my auto scale script was 20 lines or so of Python.
I guess the biggest downside was spinning up the new server took about 2 minutes, so for big load spikes it took a bit for it to level out... but with GCE per minute billing, all the napkin math I did says it is a fair bit cheaper per cpu unit to do it this way.
In conclusion, someone good with devops + a normal work queue system could do this years ago. I guess it's cool to lower the barrier to entry, but not it does not seem like a game changer. It totally IS cool to scale out backend work to a job farm. Just seems like not the only way to do it.
- Per-minute billing
- 0-to-cluster in under 90 seconds (aim for 30 seconds)
- Pre-emptiblem VMs
- Custom VMs
Now you start with a job, pay a 30 second penalty, and execute it on an entirely ephemeral cluster. The "get a cluster and fill it with jobs and round up to an hour" model is indeed outdated IMHO.
(work at Google Cloud)
Because those are the things that need to be done in the real world, 99% of the time.
Sadly, we don't all get to work on cool stuff :)
But I think as more people get access to the power and find uses cases, it will get used more.
The 'next' step for Lambda is serverless hosting for web-sites, something which Lambda is 'almost but not quite' ready for, the limiting factor being transactional latency, especially in some weird cases. Once they have that sorted out, I will never want to see a server again :)
Also a simple conversation with AWS increases the limit[0].
They obviously don't want people to write absurd runaway structures and 100 is an arbitrarily chosen value to guard against that.
And you've spent the cost of building out that 8 server infrastructure in one batch.
Assuming 70 hours at $0.000000834/100ms [1]
The whole job costs $2.10.
If you can peg the CPU and don't mind getting interrupted, spot instances are still a fair bit cheaper. But Lambda looks pretty attractive for any other use-case where the statelessness of Lambda doesn't bite you.
Meanwhile, on Lambda, you can actually run 1600 60s jobs (or 27 CPU-hours) in 3 minutes. This is inclusive of setup time, job submission, stragglers, etc. [1]
Of course, if you've got sustained load, it's cheaper to go with spot instances, but the "occasionally I need a buttload of compute," model is well-served by Lambda.
Joyent Manta: https://www.joyent.com/manta
Hyper: http://hyper.sh
Possibly Joyent Triton: https://www.joyent.com/triton
I personally often want to run a bunch of things for ~1-15 minutes, and have too much data or setup to fit neatly in a lambda function. However, I don't need 1000 things running simultaneously, although manta would help still there.
I'd love to see some more layers over the top of services like this, hopefully someday getting us back to picloud. I miss that service.
I also feel like your comment is implying that the outcomes of the lambda option vs the worker server are comparable. As mentioned, the worker machine will still run in hours, rather than less than a second.
http://i.imgur.com/BNAcSsn.png
I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invoking a lambda directly from your code.
Imagine a case where you had to parse a million documents with a relatively expensive computation, let's say 250MS per document. Maybe you have a solid machine with a few cores that's running your server, but even then you can't have the server cpu locked for so long, so naturally you'd need some sort of worker server set up. With a good machine and multiple cores, maybe you get 8 running at once. With a lambda, you can forego the worker server altogether. Just invoke a million lambdas directly from your application server, completely parallelized.
Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure.