If you're interested in building this type of infrastructure for yourself, I'd recommend taking a look at http://github.com/pachyderm/pachyderm. I'm one of the founders so this is a somewhat shameless plug, but Pachyderm is designed to get people most of the way to what AdRoll has built without having to build as much of the "plumbing" from scratch. AdRoll was some of our earliest inspiration for the design of Pachyderm.
Also, have you looked at Airflow?
Not comparing them, just curious.
* time-based and dependency-based scheduling do not mix well in our experience, if you try to do both at the same time you end up with a very complicated execution model. Luigi is much simpler, it is basically a makefile. To handle time one can just add dummy dependencies where "exists()" function returns true if now()>X, and/or just run the whole thing by cron.
* We heavily use dynamic dependencies in Luigi, where one job decides how many childs to spawn and what parameters to use at run time. Makes it easy to do e.g. mapreduce style jobs purely in Luigi.
We took a look at Airflow, which looks nice, but we haven't had a compelling reason not to use Luigi.
http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-d...
It is very convenient to be able to define any jobs as Docker containers, using the best language for the task, and define dependencies between their inputs and output explicitly, not being tied to any particular paradigm like MapReduce.
I am more than happy to let AWS handle plumbing for this instead of having to maintain and operate our custom solution.