Snakemake – A framework for reproducible data analysis

Re:Snakemake – A framework for reproducible data analysis by teekert on Saturday July 15, 2023 @08:20AM (#36735919)

Snakemake is a beautiful project and evolves and improves so fast. Years ago I realized I needed to up my game from the usual bash based NGS data processing pipelines I was writing. Based on several recommendation I choose Snakemake. I have never regretted it, It worked perfectly on our PBS cluster then on our Slurm cluster. I made some steps to make it run on K8s, which is supports, and most recently, I'm still/again happy with my choice for Snakemake because it (together with Nextflow) seems to be the chosen framework for GA4GH's cloud work stream's "products" like WES and TES [0]. This seems to be the tech stack where Amazon Omics and Microsoft Genomics focus on [1]. It enables many cool things, like "data visiting": Just submit your Snakefile (the definition of a workflow, a DAG basically) to a WES API in the same data center where your data lives, and data analysis starts, near the data. Brilliant.

I owe a lot to Snakemake and Johannes Köster, I hope some day I can repay him and his project.

[0] https://www.ga4gh.org/work_stream/cloud/

[1] https://github.com/Microsoft/tes-azure

[ Parent | on HN ]

Re:Snakemake – A framework for reproducible data analysis by tetris11 on Saturday July 15, 2023 @08:24AM (#36735948)

Its fantastic but it doesn't scale laterelly particularly well, compared to just Make.

[ Parent | on HN ]

Re:Snakemake – A framework for reproducible data analysis by ta988 on Saturday July 15, 2023 @11:33AM (#36737723)

What dimension are you referring to?

[ Parent | on HN ]

Re:Snakemake – A framework for reproducible data analysis by tetris11 on Saturday July 15, 2023 @12:07PM (#36738122)

Large scale reproducibility was a problem a few years back for one. Conda and containers were a constant problem for us back then, especially if you had multiple NGS tools running in different environments. This has probably been solved by now, but we went with another workflow system

[ Parent | on HN ]

Re:Snakemake – A framework for reproducible data analysis by bafe on Saturday July 15, 2023 @03:19PM (#36740047)

Nextflow seems to scale very well

[ Parent | on HN ]

Re:Snakemake – A framework for reproducible data analysis by tetris11 on Sunday July 16, 2023 @05:48AM (#36745173)

We went with Nextflow and Galaxy

[ Parent | on HN ]

Re:Snakemake – A framework for reproducible data analysis by bafe on Monday July 17, 2023 @03:52PM (#36763141)

You can't go wrong with nextflow. I heard a lot of scientists complaining that it's too hard to understand, but honestly the DSl and the scheduling model (flow based) is just great

[ Parent | on HN ]

Re:Snakemake – A framework for reproducible data analysis by ta988 on Saturday July 15, 2023 @11:56PM (#36743462)

Agreed, Conda has always been a nightmare to maintain and redeploy, whatever you put it in.

[ Parent | on HN ]

Re:Snakemake – A framework for reproducible data analysis by matthew_stone on Saturday July 15, 2023 @12:53PM (#36738741)

100% agree, and it's wonderful to see Snakemake on the top of HN.

Snakemake is an invaluable tool in bioinformatics analysis. It's a testament to Johannes' talent and dedication that, even with the relatively limited resources of an academic developer, Snakemake has remained broadly useful and popular.

Super nice guy too, he's always been remarkably responsive and helpful. I saw him present on Snakemake back when he was a postdoc, and it really changed my approach to pipeline development.

[ Parent | on HN ]

Re:Snakemake – A framework for reproducible data analysis by bsmith89 on Saturday July 15, 2023 @06:59PM (#36741686)

I too owe a lot of my PhD and postdoc productivity to Snakemake. It's my bioinformatics super-power, allowing me to run a complex analysis, including downloading containers (Singularity/Apptainer) and other dependencies (conda), with one command.

Great for reproducibility. Great for development. Great for scaling analyses.

Snakemake is vital infrastructure for my work.

[ Parent | on HN ]