IMO, the reason the wiring model doesn't scale up is that most don't provide a way to create submodules (compounds?). I expect that decomposing a problem into a chain of smaller subproblems is a fine way to manage a dataflow.

Yahoo Pipes didn't really offer that, or any anyonymous stdin/stdout. So huge graphs were necessary for complex problems.

I agree that it is difficult to tackle bigger problems with this model, and everything that deviates from rss in -> manipulation -> rss out is maybe better served with textual programming. Though I did use pipes in a ruby program of mine basically as micro service for everything web related, shortly before it got turned off...

But Yahoo Pipes did have submodules. You could have one of your pipes as a block in another pipe (I'm planning on supporting that at a later stage). And it had userinput that could get assigned as additional input to blocks in a pipe (for that I don't have a mental model yet).

Checkout Rapidminer and Knime for great examples of box'n'arrows applied to machine learning pipelines. I've used Rapidminer a fair bit and they really have a great model including a dynamic number of inputs/outputs from a single box and ability to make boxes that encapsulate pipelines made from other boxes. I was able to do an order of magnitude more experimentation using this visual style than if I had tried to script it in code. I'm a software engineer who loves to program so the fact that I'd pick this tool for that job over programming I think says something.

For a more stable, and more business analyst friendly tool, check out Alteryx. It's not open source, but is powerful, stable, and has SDKs in multiple languages.

It only had submodules in the sense that you could embed a pipe, but IIRC, they were always sources -- you couldn't pipe data into them. It had a notion of the 'output' block but no input with which you could implement such decomposition.

I think I have the same opinion, but express it as: words (i.e. text) can represent or reference anything. They can reference modules. Trying to do this with lines becomes messy quickly.

OTOH unix-style pipes (also used in jq), shpuld be even more limited, being only a tree. They can get too complex, and then you use files as a way to implement modules (in a sense). So pipes have their place.

> I expect that decomposing a problem into a chain of smaller subproblems is a fine way to manage a dataflow.

It is, which is why basically all graphical modelling techniques used for system analysis use it. An executable diagram system that doesn't leverage the experience of system analysis diagrams seems somewhat poorly considered.

Quatz Composer allowed for composite patches. There is still something hard about using them though.