Re:Snakemake – A framework for reproducible data analysis
by jlduan on Saturday July 15, 2023 @08:54AM
(#36736213)
I am very fond of https://zenodo.org, especially for small datasets for scientific publications.
Re:Snakemake – A framework for reproducible data analysis
by samesense on Saturday July 15, 2023 @08:55AM
(#36736221)
I use snakemake along with dvc. It detects changes in files, and reruns steps to produce downstream files as needed.
Re:Snakemake – A framework for reproducible data analysis
by samuell on Saturday July 15, 2023 @09:38AM
(#36736629)
The way a workflow like Snakemake can help here is generally by letting the filenames pretty much describe how each particular output was created, meaning data outputs can act as immutable in a sense.
What I mean is that rather than create a new version of a file, if you run the same analysis with different sets of parameters, it should generate a new file with a different name rather than a new version of the old one. This also helps comparing differences between output from different parameters etc.
That said, there are workflow platforms which support data versioning, such as Pachyderm (https://pachyderm.com), but it is a bit more heavyweight as it runs on top of Kubernetes.
Re:Snakemake – A framework for reproducible data analysis
by bafe on Saturday July 15, 2023 @11:42AM
(#36737826)
The reliance on filenames to define (parametric) dependencies was among the reasons I later adopted nextflow. The model fit the type of computation dependencies better for my case. In the mean time snakemake grew and many DAG that were hard to describe back then are now expressed directly with snakemake primitives
Re:Snakemake – A framework for reproducible data analysis
by troymc on Saturday July 15, 2023 @10:24AM
(#36737055)
Pachyderm has data versioning / data version control built-in. I guess other tools do too.
Re:Snakemake – A framework for reproducible data analysis
by bafe on Saturday July 15, 2023 @11:04AM
(#36737447)
There is indeed an old issue asking whether they could return a provenance graph using the prov ontology in JSON format https://github.com/snakemake/snakemake/issues/2077
I think it's a good task to work on if you like to contribute
Or would you typically use this -in addition- to a tool like dvc? I've used dvc a bit and while it's quite good for data versioning, I find the workflow aspect is clunky.