News for hackers. Stuff that matters.
Re:The Cost Savings Of Netflix's Internal Spot Market by mgummelt on Tuesday December 05, 2017 @06:49PM (#15857299)
If you'd like to learn about an open-source scheduler that does mimic some of the properties of an internal spot market, check out Cook: https://github.com/twosigma/Cook

It's unlikely that Netflix implemented anything this complex for video encoding, since Cook is only valuable when you're scheduling jobs from parties with conflicting incentives, rather than a singular purpose like "encode all Netflix video".

Re:The Cost Savings Of Netflix's Internal Spot Market by fragmede on Wednesday December 06, 2017 @12:58PM (#15863235)
Looking closer at the mission to "encode all Netflix video", a less idealized version involves people at Netflix and outside it, who have signed contracts with deadlines and monetary incentives and everyone wants to go first, but how many are willing to put their money where their mouth is, so to speak.

It's expensive to operate thousands of instances in AWS, so I presume there's also a relentless push internally to drive down costs.

Thanks for the link to Cook, that's fascinating. The fact that it plugs into Mesos seems quite useful!

Re:The Cost Savings Of Netflix's Internal Spot Market by bob_theslob646 on Tuesday December 05, 2017 @08:41PM (#15857828)
Holy Shit, two sigma, the hedge fund, developed that.(Cook)

>Cook is a powerful batch scheduler, specifically designed to provide a great user experience when there are more jobs to run than your cluster has capacity for.