Re:Toil – A Python Workflow Engine
by batbomb on Thursday May 05, 2016 @12:11AM
(#11633969)
I work on a workflow engine (aka DAG processor) that dates to before 2008. We use it for many things in an astronomy experiment. It has executed over 900M batch jobs and 200M "scriptlets", and most of those workflows can be re-executed at any stage, even today, five+ years after they were originally ran, to recreate data products. The workflow engine also serves effectively as a provenance database for each data product to trace its origins back to raw data.
[ Parent
| on HN ]
There are no comments on this comment yet.