The interplay of nf and groovy (how I wish they hadn’t used groovy!) can be mind bending but if you’re writing your own thkng you have a different optimization model than nf-core that is trying to be one size fits all
Before you wonder why I don’t know that, I do top-down software development primarily and also maintain and upgrade my lab’s search engine, MetaMorpheus.
To answer your broader question about the general need for some structured pipeline or workflow orchestration.. That comes down to volume of data (we do screens as well as one-off studies) and a desire to reduce human involvement as much as possible. So the goal is to have raw files be immediately picked up, processed, and loaded into on internal application where it can be queried and interesting data can be highlighted. During my PhD, this was also a goal of mine (and I have at least two github repos where I got close) but it was definitely less of a priority since actually doing experiments and downstream analysis was the limiting factor.
PS: if you want to talk off-HN, I should be your latest stargazer
At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale, it works great. However, I don't have control over the resource utilization of individual steps, and in the past I've had issues with the pipeline blocking as a result of how I was chaining tasks together. I think something like Nextflow would remove these limitations, but one thing I think I would miss is the ability to debug individual pipeline steps locally with an interactive debugger. As far as I can tell, Nextflow has logging/tracing facilities but nothing quite like an interactive debugger. I'd be happy to be told I'm wrong, or even that I'm doing it wrong.
Other reasons I'd like to start using Nextflow:
- my homebrew pipeline would be easier to setup/share
- there some efforts in the proteomics community to develop Nextflow pipelines (eg. QuantMS²)
___
¹ https://github.com/coleifer/huey/
² https://docs.quantms.org/en/latest/