Re:Nextflow: Data-Driven Computational Pipelines
by esafak on Thursday August 10, 2023 @04:08PM
(#37081094)
In all fairness, they predate the competition (2013): https://github.com/nextflow-io/nextflow/releases?page=25
Re:Nextflow: Data-Driven Computational Pipelines
by geoffjentry on Thursday August 10, 2023 @04:26PM
(#37081297)
As GP referenced CWL, while NF had appeared first in terms of the bioinformatics world Nextflow, CWL, Snakelike, and WDL all erupted close enough to each other to be equal-ish. The people were aware of each other but they were all so nascent that it wasn't clear if it was worth joining in or not. At the end of the day these all came from groups trying to scratch particular itches, and not everyone agreed on the right way to scratch.
However all of them were rejections of prior models as well as the workflow solutions prominent in the business space.
Re:Nextflow: Data-Driven Computational Pipelines
by adolph on Thursday August 10, 2023 @04:56PM
(#37081664)
Yeah, the thing that I find disappointing is that there is a lot of science value locked into the different systems of describing a workflow, pipeline or DAG. Like you said, they all had different itches to scratch and even some barebones "standards" like csv have flavors/extensions/etc.
Re:Nextflow: Data-Driven Computational Pipelines
by bafe on Thursday August 10, 2023 @05:24PM
(#37081983)
They try to address similar solutions, but comparing snakemake and nextflow doesn't do either tool a favour. They use different computation models, nextflow is based on dataflow programming and therefore schedules processes dynamically as new data comes in, while snakemake is pull-based and schedules the processes based on the dag defined by the dependencies. Anyhow they are both great tools.
Re:Nextflow: Data-Driven Computational Pipelines
by geoffjentry on Thursday August 10, 2023 @07:26PM
(#37083039)
While true that’s a minor distinction when comparing the clusters of bioinformatics workflow systems vs workflow systems aimed at different domains
Re:Nextflow: Data-Driven Computational Pipelines
by bafe on Friday August 11, 2023 @01:34AM
(#37085431)
While these two are aimed at bioinformatics, they are general purpose enough that you can apply them to any computational workflow. I can say they saved my PhD
Re:Nextflow: Data-Driven Computational Pipelines
by mbreese on Thursday August 10, 2023 @06:38PM
(#37082622)
In fairness, this is an old problem with many other contenders. This issue is as old as batch schedulers. FWIW, I was at an ISMB conference in 2005 that had at least 2-3 workflow managers presented.
Re:Nextflow: Data-Driven Computational Pipelines
by bafe on Thursday August 10, 2023 @04:03PM
(#37081030)
If you refer to nextflow,the syntax is basically groovy
Re:Nextflow: Data-Driven Computational Pipelines
by notQuiteEither on Thursday August 10, 2023 @04:06PM
(#37081057)
Indeed, it's a fully functional scripting language that is an extension of groovy. So "looks programmery" is more than a little reductive.
Edit: functional as in, it's not half-hearted, not functional as in functional programming.
Re:Nextflow: Data-Driven Computational Pipelines
by adolph on Thursday August 10, 2023 @04:20PM
(#37081233)
Interesting. From the docs:
The Nextflow scripting language is an extension of the Groovy programming language. Groovy is a powerful programming language for the Java virtual machine. The Nextflow syntax has been specialized to ease the writing of computational pipelines in a declarative manner.
https://www.nextflow.io/docs/latest/script.html?highlight=gr...
Re:Nextflow: Data-Driven Computational Pipelines
by joysofpi on Friday August 11, 2023 @01:43PM
(#37091543)
Its really just a layer on top that uses the groovy DSL features
http://docs.groovy-lang.org/docs/latest/html/documentation/c...
https://github.com/common-workflow-language/common-workflow-...
mea culpa: The above was based on a first look at something titled "A DSL for parallel and and scalable computational pipelines"a as opposed to "Java workflow manager with Groovy language scripting." The presented screenshot still looks to me like an unholy union of yaml, js, py and sh. If that sounds groovy to you; have fun.