News for hackers. Stuff that matters.
Re:Amazon Genomics CLI by epistasis on Saturday August 21, 2021 @10:47AM (#28257424)
I don't think that a single one of these was meant to be used by wetlab biologists, and every single one assumes that the user will be learning an entirely new text based language to express the workflow semantics. In contrast, I see useless GUIs as fairly common on non-bioinformatics workflow systems, and when I see that I know that it is unlikely to be able to scale what I need it to do. Manually drawing every box and line assumes simple workflows that are unlikely to scale very far.

Bioinformatics and Other workflow management systems just have different focuses, more than they have different end user skills. In the 2000s I tried to shoe-horn so many CS Systems type workflow onto using clusters, and every single one had abstractions that simply did not fit what I needed. Typically they seem to require knowledge of how many tasks a job must be split up into, for example, instead of dynamically adapting the workflow to the inputs as desired. Or they hard code the method does launching processes to a certain type of compute environment. So when somebody says "workflow," I assume they mean something entirely different then what I mean until proven otherwise.

For example, if I go to the Apache NiFi docs to see if it's suitable, the Getting Started page starts with a GUI, which is a huge red flag, I don't want to do workflow entry with a GUI because a pre-defined graph will only cover a fraction of my use cases. But worse then that, when it describes where to start creating a processor, it says you need to select from a list of processors. Now this is an even BIGGER red flag because I know absolutely none of the commands I need will be available, and I'll need to add tons of metadata for whatever I want, most likely in some sort of bad GUI, or even worse, XML or some other bad data standard that software engineering adopted due to cargo-cult design. The bigger danger is that the system assumes that processors must be written in some certain language, or assumes that data itself will be introspectable, or something that will require massive amounts of work.

Every one of these systems embodies a huuuuuuge number of assumptions about what it means to run a workflow, nobody ever states what those assumptions are, or probably even realizes how different their needs are from others' needs. At least with the workflow systems from other people in bioinformatics, I know that the assumptions they make will work with the tools and data sizes in the field, and won't have some deal-breaker I only find out about after three hours of making workflows.

Re:Amazon Genomics CLI by geoffjentry on Saturday August 21, 2021 @10:53AM (#28257479)
> I don't think that a single one of these was meant to be used by wetlab biologists

Speaking for myself, we didn't target wetlab biologists with WDL but we did target "non-programmers" as MontyCarloHall described. This was influenced by the groups from which the original developers originated prior to working on the DSL.

The degree to which we were successful can be debated, but hey that was the idea at least ;)

Re:Amazon Genomics CLI by epistasis on Saturday August 21, 2021 @12:31PM (#28258272)
Wetlands biologists tend to be pretty clever, and I have no more difficulty teaching them small bits of AWK to do some quick calculations on their laptop than I do new hires that are very skilled with Python.

If you look at a small WDL tutorial, such as this:

https://www.rc.virginia.edu/userinfo/howtos/rivanna/wdl-bioi...

I think that this is more complex than, say, defining Afro data types, as it involves calling tasks like functions, string substitutions, filling in variables from a config file, etc. There’s a certain base level of complexity that can be hard to abstract away, and I don’t think this has yet got to the simplicity of, say, pivot tables in Excel.

Re:Amazon Genomics CLI by geoffjentry on Saturday August 21, 2021 @01:28PM (#28258804)
> There’s a certain base level of complexity that can be hard to abstract away

100% and a conclusion I came to far too long into my tenure in the bioinformatics workflow domain. It is also why I soured a bit on some of the GA4GH efforts in the same space (WES & TES in particular).

I still very much agree w/ the goals & ideals (respectively), but in my experience the middle ground between "I don't care, just give me the defaults" and "please do not abstract away any of the complexity, I need that" is quite small. And that zone would be the sweet spot for these efforts.