News for hackers. Stuff that matters.
Re:Common Workflow Language by fowl2 on Thursday November 14, 2019 @07:25PM (#21541110)
"Workflow" is such a loaded/broad/bifurcated term.

In this case we seem to be talking about "Scientific Computing Workflow" or some sort of data pipeline. (As opposed to some sort of "Enterprise" workflow)

Re:Common Workflow Language by lewisjoe on Friday November 15, 2019 @08:13AM (#21544453)
Yes, "Workflow" has to be one of the most abused word in the industry.

People seem to use workflow for all of these things:

#1 Business process modeling and automating platforms - Camunda, BPMN, jBPM etc.

#2 Assembling together software functions (also called tasks, sometimes) for getting an output data - Airflow, Cadence, CWL is related too.

#3 Moving one entity (bug, document, deal) through different stages of a state machine, until the entity reaches an end state - Sharepoint workflow, Zoho Orchestly, Comala workflows

#4 Orchestrating several apps together for automating something, usually as reactions for events - Zapier

When developers talk about workflows, it's mostly about #2. When business people talk about workflows it's mostly about #1. When managers/leads of a team talk about workflows it's mostly about #3.

We should accept that Workflow as a word is too ambiguous, and come up with what kind of workflow category the tool belongs to, before branding it as an "one stop solution for all things workflow".

Re:Common Workflow Language by zekrioca on Friday November 15, 2019 @10:04AM (#21545338)
Couldn't agree more. I do work with workflows (mostly #2 and #4), but in the beginning of my journey I had problems understanding what researchers were actually trying to tell me, and I always got confused. I suspect that they themselves do not fully understand how ambiguous this word can be..
Re:Common Workflow Language by _8ljf on Friday November 15, 2019 @08:41AM (#21544636)
“we seem to be talking about "Scientific Computing Workflow" or some sort of data pipeline”

So, more along the lines of https://en.wikipedia.org/wiki/Dataflow_programming. A good problem space to bring declarative/functional/logic/concatenative concepts and tools, IMO, as opposed to the ubiquitous “hit it with a rock” of the usual C* suspects. Be great to see real progress here, but my bullshit detector is immediately dialed to MAX.

The great problem with “workflow systems” is that they very easily disappear up their own ass: “Potemkin solutions” that solve every problem except the customer’s own. (See: https://thedailywtf.com/articles/The_Inner-Platform_Effect). “Enterprise workflows” in particular I think are just a way for indolent code monkeys to shirk learning how the business works for themselves. Looks all very impressive without ever solving shit.

..

FWIW I do think Alan Kay was onto something with his Nile strategy (https://news.ycombinator.com/item?id=10535364). Pity there wasn’t followthrough (bit of a mayfly, Kay). Ditto Iverson and APL, although APL’s sidelining was more down to a nascent Programming Profession redefining Programming™ to be what they do for users rather than what users (mathematicians in this case) do for themselves.

Re:Common Workflow Language by tetron on Friday November 15, 2019 @02:08PM (#21547733)
CWL is a declarative/functional language for describing how to execute command line tools (staging input files, mapping arguments, collecting outputs) and how to connect the outputs of one tool to the inputs of the next. It is HPC and cloud agnostic. The same workflow description can run on a laptop or on 1000 cloud compute nodes. Lots of people are already using it to solve their problems, for some examples see https://github.com/search?q=extension%3Acwl+cwlVersion
Re:Common Workflow Language by _8ljf on Saturday November 16, 2019 @12:18PM (#21553692)
Quickly skimming the CWL User Guide, calling that a “language” is a huge stretch. Looks like writing a bunch of YAML config files which let you write more YAML files to construct a UNIX pipeline and run it.

In which case, why didn’t you just use bash?

Yeah, I know bash stinks. But this is not my fist rodeo* so I’m struggling here to see how CWL stinks less. See again: Inner-Platform Effect, Greenspun’s Tenth Rule.

--

* i.e. I already know how easily “workflow engines” go up their own arse because I’ve done it myself. And CWL fails the same sniff test. Not a good start.

Re:Common Workflow Language by hhas01 on Saturday November 16, 2019 @01:30PM (#21554138)
This is a serious question, BTW. “Lots of people are using it” is testament neither to sound architecture nor practical need. Lots of people have joined MLMs; does that mean MLMs are good and needed? Or does it just mean lots people are easily seduced by layers and layers of makework and grift?

The only bit that sounds at all novel or interesting is the dispatcher; and even that is really just an expression of `<load_balancer> | ssh`. At which point, Unix Philosophy tells us we should implement <load_balancer> as a small simple single-purpose Unix command which can easily pipe to other Unix tools. Itch scratched; everyone can now go get on with their actual work.

So if that is the case, the precisely what problem is all CWL’s Castles-in-the-sky YAML crap actually solving, other than bored developers’ need to keep entertained? Especially when [from what I can tell] the project doesn’t even provide you a dispatcher component but instead tells everyone to take a spec and write their own.

How many wheels need to be reimplemented before someone involved declares it a pig in a poke? And how many more before the rest can accept this?

Re:Common Workflow Language by tetron on Monday November 18, 2019 @11:03AM (#21565349)
I don't know if I can change your mind, or if anyone else is reading this thread, but CWL was designed to solve a particular set of problems, if you don't have those problems, you might not need it, but it doesn't mean those problems don't exist.

> the project doesn’t even provide you a dispatcher component but instead tells everyone to take a spec and write their own.

Close...

Software that supports CWL are SaaS vendors, FOSS projects, and various HPC schedulers that all have their own incompatible data management and dispatch/scheduling systems. If you want to write an analysis that runs on more than one of these platforms, you need some abstraction for it. CWL is one such an abstraction.

This matters because maybe you've developed a research pipeline that integrates a bunch of different tools written in different languages and want to run it on somebody else's data, and you need to run it on their infrastructure because copying 12 terabytes of HIPAA-restricted data from their LSF cluster to your Google cloud instance isn't an option.

"Just use bash" is what people who adopt CWL are trying to get away from. It is nearly impossible to write portable parallel / distributed analysis in bash, and the result is brittle scripts with more coordination code than code that actually does scientific work. Because CWL is declarative, the CWL engine handles all the coordination, scheduling and data staging for your particular infrastructure.

You may not have any of these needs, but suggesting that we're just bored developers creating castles in the sky is really unhelpful.

Re:Common Workflow Language by boohooimsad on Sunday November 17, 2019 @05:38PM (#21560485)
I'll play. CWL was designed for bioinformatics research first. We're using CWL for bioinformatics analysis, because as the "scientific workflow | data pipeline" grows (beyond 5-10 tools), bundling the execution and logical analysis together becomes difficult. If we can let researchers write just their analysis (which tools to run and what their dependencies are), and abstract the execution environment we can create more structured analysis that's portable and publishable, and also often quicker to run.

Bioinformatics software isn't perfectly written software, there are a number of weird behaviour that a simple unix pipe doesn't solve. There are engines that support CWL, and other existing engines have been adding CWL support.

I'm not saying that there aren't other frameworks out there for doing analysis, or that this is the best way but this is an option that IS working for researchers.

Edit: workflow -> scientific workflow | data pipeline

Re:Common Workflow Language by oeuviz on Friday November 15, 2019 @04:16AM (#21543466)
Fully agree. It seems nobody understands it in the sense of Business Processes/Workflow modelling anymore and it is arbitrary used for "something to do".
Re:Common Workflow Language by liotier on Friday November 15, 2019 @06:14AM (#21543966)
I did expect business process modeling... I was confused !