News for hackers. Stuff that matters.
Re:Hyper.sh – Effortless Docker Hosting by jdc0589 on Monday November 07, 2016 @12:33PM (#12892922)
> serverless cron.

damn, thank you. Anywhere I can see implementation details? Are you rolling your own system from the ground up, or using something like dkron (http://dkron.io/) behind the scenes?

Re:Hyper.sh – Effortless Docker Hosting by Vekz on Monday November 07, 2016 @01:12PM (#12893347)
if you're interested in serverless cron you can use AWS lambda for the same experience
Re:Hyper.sh – Effortless Docker Hosting by djm_ on Monday November 07, 2016 @05:40PM (#12895898)
Agreed for some things but a massive limitation of Lambda with its timeout of 300 seconds is that it cannot be used for long running tasks.

If this feature lands it sounds like it will give the ability to run any container arbitrarily and pay by the second - this would be huge for things like web scraping and other tasks that don't occur all the time yet still come with all the pain of server maintenance and uptime fees.

Imagine the scenario where you have a web scraper that runs once a week for 3 hours. Ideally you only want the machine to be on for those 3 hours, and to only pay for those hours of usage; but you also don't want the hassle of writing all the scripts that go with creating and deleting a cloud machine, mostly because those would also need to be hosted somewhere. For that kind of use case there is little out there as far as I'm aware.

Re:Hyper.sh – Effortless Docker Hosting by 010a on Tuesday November 08, 2016 @09:38AM (#12900541)
There are ways to organize your lambdas to get around the 300s timeout. In classic AWS fashion... you just need to give them more money.

Look for ways to divide-and-conquer your lambdas into smaller parts. If you need to run some logic for every record in some table, give N records to a single lambda (where N is some small number which doesn't make the lambda take anywhere close to 300s).

You can orchestrate this workflow with several different AWS tools which exist all over the spectrum of cost and ease-of-use.

Easiest is definitely just having the master lambda directly invoke the other lambdas with InvocationType:Event.

SNS is another easy option. Lambda(master)->SNS(per N records)->Lambda(splinter). Downside is that you'll probably completely blast out your global AWS concurrent function execution limit pretty quickly because you have no control over how quickly SNS will trigger your functions.

Kinesis is a more powerful option. SQS also has potential, but you can't directly trigger a Lambda from SQS. One pattern I've seen used is to have a CWEvents cron trigger a lambda every M seconds to read N records from SQS. Depending on how consistent your workload is, this might make sense because it gives you really fine-grained control over that ratio between "how quickly will my jobs be processed" and "am I approaching my AWS global account limits". But if your jobs are really disparate you'd be invoking lambdas all day to do a whole bunch of nothing 90% of the time.

Re:Hyper.sh – Effortless Docker Hosting by jdc0589 on Monday November 07, 2016 @04:42PM (#12895450)
If I'm playing devil's advocate: yes and no. Lambda is still partially limited by the officially supported languages (I'm pretending that the hacks around this don't exist, they have issues). That said, Lambda is great, but I'm not really on-board with ECS, so it's nice to see an alternative here from someone who is also offering container based serverless infastrucutre.
Re:Hyper.sh – Effortless Docker Hosting by samstave on Monday November 07, 2016 @01:55PM (#12893759)
Is there a write up on howto use lambda for this? I've been wanting serverless crown for ages and I feel like an idiot for not thinking of lambda for this... this should be promoted (if it isn't) - but I'm an example learner... point me please?
Re:Hyper.sh – Effortless Docker Hosting by webo on Monday November 07, 2016 @04:01PM (#12895038)
Don't have an example, but I have set up a few "serverless" cron jobs:

1) Create a Lambda function

2) Trigger it using CloudWatch Events. You can set up cron like rules and AWS will trigger them for you.

Re:Hyper.sh – Effortless Docker Hosting by runako on Monday November 07, 2016 @02:04PM (#12893855)
Re:Hyper.sh – Effortless Docker Hosting by scprodigy on Tuesday November 08, 2016 @12:42AM (#12897962)
Re:Hyper.sh – Effortless Docker Hosting by mrmrcoleman on Tuesday November 15, 2016 @09:30AM (#12958232)
Hey jdc0589,

We have the first version of hyper cron ready for internal demo and would love to get your feedback.

If you're interested in being involved, please email us on [email protected] and we'll show you what we have so far.

Re:Hyper.sh – Effortless Docker Hosting by jdc0589 on Wednesday November 23, 2016 @10:05AM (#13023098)
man, sorry I missed seeing this. HN doesn't do comment reply alerts very well.

I did get the email about the beta this morning, and will definitely play around with it over the next week or so.

Re:Hyper.sh – Effortless Docker Hosting by mrmrcoleman on Wednesday November 23, 2016 @10:49AM (#13023389)
Cool!
Re:Hyper.sh – Effortless Docker Hosting by fredliu on Monday November 07, 2016 @04:42PM (#12895449)
Another "serverless cron" option from aws is their Data Pipeline service. It is intended for data processing as its name implies but does fits the bill for lots use cases as a generic "serverless cron". Note that Lambda has executing time limit that's pretty short, while data pipeline can be configured to have much longer timeout.
Re:Hyper.sh – Effortless Docker Hosting by 010a on Tuesday November 08, 2016 @09:32AM (#12900486)
Lambda is already limited enough when it comes to how you can write the functions. I'm not even sure how you'd write a generic JS/Python/Java function on DPL, unless its got some functionality which I'm not aware of (through a custom shell command bootstrapping your language environment maybe?).
Re:Hyper.sh – Effortless Docker Hosting by gnepzhao on Monday November 07, 2016 @12:46PM (#12893059)
We are going to release it very soon. Stay tuned :)
Re:Hyper.sh – Effortless Docker Hosting by saasthrowaway3 on Tuesday November 08, 2016 @03:23PM (#12903599)
Is serverless cron something you´d pay money for?
Re:Hyper.sh – Effortless Docker Hosting by jdc0589 on Tuesday November 08, 2016 @09:04PM (#12905571)
sure, if I was using the same vendor for something else already it would be a nice value ad.
Re:Hyper.sh – Effortless Docker Hosting by mrmrcoleman on Tuesday November 08, 2016 @03:55PM (#12903852)
More details coming up ASAP.