Ask HN: What do you use to manage chains of long-running jobs?

Posted by polm23 on Friday April 28, 2017 @02:39AM
from the ask-hn dept.

For running chains of multi-hour jobs with dependencies I've always used Jenkins but always been unsatisfied with it. I've also heard of Luigi but haven't given it a shot yet. Is there anything else similar I should look at? I've had trouble finding a single name for programs like this to search with.

Luigi: https://github.com/spotify/luigi

Example jobs:

  - download a bunch of raw data from different sources
  - clean and transform raw data into a single format
  - push a lot of data into a DB
  - routine ML model updates
  - generate reports

These would have dependency relations, some would be short and some would be long, sometimes they fail (which may be important or not), etc...

( 5 comments | 4 points | on HN )

Re:Ask HN: What do you use to manage chains of long-running ... by jasonmotylinski on Thursday May 04, 2017 @09:05AM (#14264489)

Luigi, while great and simple, has been superseded by Apache Airflow: https://airflow.incubator.apache.org/.

Spotify built a scheduler system named Styx that handles orchestration (https://github.com/spotify/styx) but it's fairly beta for external use.

[ Parent | on HN ]

Re:Ask HN: What do you use to manage chains of long-running ... by ditemis on Friday April 28, 2017 @04:25AM (#14217794)

The search term "batch processing" might give you some alternative solutions.

Jberet, a Java Batch API (JSR 352) implementation, is a batch processing framework and can be run in a standalone Java process or within Wildfy/Jboss application server. It provides a robust batch execution runtime, XML configuration for jobs, REST API for batch management, a lightweight web frontend and various data readers/writers. https://github.com/jberet/jsr352

[ Parent | on HN ]

Re:Ask HN: What do you use to manage chains of long-running ... by eb0la on Friday April 28, 2017 @06:18AM (#14218209)

I use Talend Open Studio a lot for this kind of stuff.

It might not be the best tool for the ML part; but for moving data, and ssh'in automatically to servers and executing commands is really good.

I used it (a long time ago) to ssh to a set of Juniper routers, execute a command, parse the results (in XML), update some db tables, and execute another commands in other routers...

I really like the wait for file feature ( http://imgur.com/a/3kKLC )

[ Parent | on HN ]

Re:Ask HN: What do you use to manage chains of long-running ... by shoo on Friday April 28, 2017 @04:06AM (#14217702)

> For running chains of multi-hour jobs with dependencies I've always used Jenkins but always been unsatisfied with it

What in particular are you unsatisfied with?

What are you happy with that you wouldn't want to lose?

[ Parent | on HN ]

Re:Ask HN: What do you use to manage chains of long-running ... by weirc on Friday April 28, 2017 @03:51AM (#14217645)

Rundeck?

[ Parent | on HN ]