I'm with you. Any attempt at visual programming I've seen has been:

1. Limited in what you can do.

2. If you need to do anything out of the norm it's either impossible or very difficult to find which menu entry you have to set in which way.

3. Doesn't have a proper diff.

4. Needs a slow compile process to actual code to work, making TDD strategies impossible.

5. Attracts the wrong kind of developers (those who don't look into the generated code and make all kinds of mistakes, those that don't understand anything about how computers work, etc.).

6. Hard to debug, because there's little to no debugging support.

7. Impossible to use with the wealth of great tools available for text manipulation.

8. Impossible to search properly, because it's not just simple text.

9. Very prone to vendor lock-in.

10. Doesn't interact well with versioning systems.

I don't remember when I saw this presentation. But it hit the nail on the head. If you go into a Korean McDonald's you have a visual menu. It enables even a foreigner who doesn't speak a bit of Korean to order a bacon cheeseburger. However if you want anything special (like no tomatoes), all of a sudden you need the language interface.

Language and text has evolved because it's necessary to describe the kind of complexity we have in the real world. That's why text is amenable as a representation of programs. They eventually represent a similar level of complexity as the real world. Of course it's easier to teach somebody to point at the McDonald's menu, but they won't get anything complex done, and they need somebody to work the abstraction for them (i.e. a real programmer).

I don't get it.

>> 3. Doesn't have a proper diff.

What makes it not have a proper diff?

I mean, GraphViz can have a diff. You could even go as far as taking the graph of the program, generating a GraphViz graph out of it and then taking the diff and simply colouring the nodes that made it into the diff.

That's true, but I would say GraphViz is a textual interface for programming graphs. Not a GUI for modeling graphs.

I'm very much for using something like GraphViz for visualizations (for exmaple integrated in AsciiDoc) to have a very good diffable plain text documentation that's nice to look at.

Well, my question is essentially this: what difference is there between visual programming and it's representation as a GraphViz graph?

These are two ways of representing the same concept: you could save the visual representation as a text document.

So what makes Visual Programming not have a textual representation? What makes them inherently incompatible with each other?

Because as of now, what I am seeing are the comments from people who used primarily proprietary tools, and those proprietary tools don't allow you to edit textual representations of the graphs directly because it makes the vendor lock-in so much easier.

>So what makes Visual Programming not have a textual representation? What makes them inherently incompatible with each other?

Nothing really, but you would need a Visual programming environment that maps one to one to your language. So that you could switch between coding in code and looking at or modifying the visual representation when it's helpful.

This would probably limit what you can do in the GUI, just as GraphViz limits control of the graph in favor of automatic layouting.

The vendor lock-in is one of the main issues I mentioned above.

If you have a good idea how to visually represent and modify C# or some other language I wouldn't mind that as an additional tool. I would think it's even very helpful for getting an overview. However the code should still be the master. And open-source is probably the only way to have such a tool that's actually good.

And if you can map one language to the GUI you could probably port this to many languages.

I was mostly concerned with what we have right now (LabView, Simulink and others).

You're absolutely right. Visual programs are much harder to generalize because they've been pre-abstracted for you to a specific use case.

Building controls are a popular field for visual programming because the "wires" seem like they'll be intuitive to non-programming tradespeople.

If we have a little visual program that controls the temperature in a room with a thermostat block and a baseboard heater, that's fun. We can play all kinds of games hooking up limit blocks and schedule blocks and whatever. Intuitive.

Imagine we have seven thousand rooms and some of them have only a baseboard heater but some have an air conditioner, some have a lighting interlock, some have a heat recovery unit. 16 possible configurations of a room and you have rooms in each set.

Now... apply a visual program across all of those sets.

In traditional programming this is simple. A data structure representing room configurations and few conditional statements in your function or loop or whatever. In visual programming you're mostly looking at either making 16 versions of the program and applying each to its associated rooms manually or else passing a bunch of conditional variables around.

Those variables aren't visual anymore. So now you have a whole layer of abstraction that's no longer visual. And really, the parts that fit into the visual model are usually the easy bits. The hard parts still require abstract reasoning about stuff that isn't linked together. Pretty soon you're just using the visual language to draw the inside of your "functions" but your high-level datalinks or scripts are entirely non-intuitive and might as well be in text.

Why can't conditional variables be "visual"? This is not clear to me - data flow and control flow are both visual (albeit their flavors of "parallel" composition and the like are incompatible, so they must never be conflated when shown diagrammatically) and conditional variables can be accounted for in control flow.

Say I have 100 rooms. Each room has 10 characteristics that might vary. Those 10 characteristics are going to be shown or referred to by about 10 different blocks... some are making decisions about the temperature, some are showing it on graphics, some are logging data.

That's 10,000 links. You can't actually draw lines for each of those links to the room information. It's literally impossible to put them on a screen, let alone process them visually. So you have to refer to the variables by name and you still end up with 100 links to draw between variable names and function blocks.

Once you refer to variables by name the relationship between the source and destination is no longer visual. The visual part is just inside any given encapsulated function. So now you're programming with parameter passing but instead of just typing text you're also pulling away to move the mouse and try to line blocks up on a grid all the time.

This should be pretty obvious while coding in text. Imagine every time a you call a function there has to be a line somewhere that represents that function call. It's too much visual information. You can abstract away some of it by saying "oh, we have a block here that goes out and grabs all the room numbers", but now a lot of the logic starts to depend on the contents of those blocks and it's no longer clearly visual.

Visual diagramming is great for roughing out an idea of how something works but enforcing a rigorous correspondence between the diagram and the function of the program turns out to be a lot of trouble for not very much gain.

> Visual diagrams are also difficult to search, scan, or replace programmatically.

This. Whenever I investigate a new authoring tool, the first thing I want to know is whether there is a plain text (preferably json) serialization that I could import and export. A particular tool I otherwise love where this is sorely missing is Zapier.

Statebox is a bijection between code and visual diagrams (petri nets)

:rock:

exactly, every picture represents an "equivalence class" of expressions (or code).

I say equivalence class, because the picture represents many formulas: by topologically distorting the picture you get different code. however, they all behave in the same way. 2+3 = 1+4 = 5

I don't agree that text is the simplest way to represent a program. In fact, from experience in other areas of knowledge, the opposite is true. Math and physics has evolved from use of pure text to the use of diagrams and non-textual symbols. The problem with CS is that we don't have a shared, simple way to represent symbols and images. We feel that text files are simpler because it has become the universal way to represent computer code, and practically all tools we have are designed to work with textual representations.

I don't think that works. I think most of us here would have similar understandings of something like https://tex.stackexchange.com/questions/19941/example-of-sys..., similar to what physicists might have from a Feynman diagram. I assume physicists could illustrate a proof with one, but would actually run through the QED equations, also. Similarly, we could deeply understand a system from an architectural diagram, but we wouldn't use them as source code.

now you can. ok, well soon :) but the theory is in place and works. Also stochastic Petri nets are actually equivalent to Feynman diagrams in a precise sense (https://johncarlosbaez.wordpress.com/2012/12/20/petri-net-pr...), they could be the Feynman diagrams of functional programming :)

Well perhaps it's mathematics that could benefit from an update to its representation. Mathematical symbols have evolved over hundreds of years and aren't really suited to modern systems of representation. When it comes to computers is it really easier to look up the unicode symbol '∩' or its LaTeX representation when you're trying to write 'A ∩ B' -- or would it be better to begin noting mathematics online in a portable way such as: (intersection A B)

One of the reasons for infix notation in math is actually that it provides a 'visual' reminder of useful properties such as associativity, and possibly others e.g. commutativity or distributivity. If all we ever used was a strict LISP-like, function-based notation, such a reminder would be lost and understanding or manipulating non-trivial expressions would be quite a bit harder. The effort in OP is actually a way of generalizing this idea to broader settings, where one is dealing with something more complex than a single domain of number-like values, and a handful of operations on them. This is arguably how one should think of "graphical linear algebra" as well: the 'diagrams' one's dealing with there can be thought of as generalized expressions, so there's nothing overly strange in being able to manipulate those formally according to well-defined rules of some sort.

> 'visual' reminder of useful properties such as associativity, and possibly others e.g. commutativity or distributivity

These are semantic properties that most binary operators do not in fact have.

E.g. C syntax:

  A << B
  A = B
  A - B
  A / B
  A % B
  A < B

very well put!

this is exactly where the statebox project comes from, there is updated syntax for mathematics in the form of diagrams. of course not for all of it, but certainly very applicable to CS stuff

Lots of good ideas come back around every few years. That usually means they are in fact good ideas, but simply that the rest of the industry hasn't caught up yet -- either the hardware isn't fast enough, or operating systems can't support it, or the tooling and infrastructure isn't there yet.

For example, GC was conceived in the 1950's, and through at least the late 1990's it hadn't taken off "in a more mainstream way", and people were saying "this idea comes back around every few years" as evidence that it never would. It turns out that GC is actually a pretty good idea, but the average computer prior to 2000 or so wasn't so good at it, either in hardware or software. That didn't make it a bad idea. It was a good idea that we weren't yet great at implementing.

What I'm hearing about visual programming today sounds very similar. Everyone has a list of complaints, but they're very specific, and very fixable. No good editors -- agreed! The solution, then, is to write a good editor, not to throw out all of visual programming. (Remember what the machine language programmers said about compilers, back before we had decent text editors?) No good diff tools -- agreed! But once upon a time, text didn't have good diff tools, either, so we wrote them. And so on.

If you start with the premise that visual programming is bad, then you will see a list of 10 problems as evidence that it's insurmountable. If you start with the premise that visual programming is good, then a list of 10 problems is your TODO list for the next year.

The biggest problem for me with visual languages:

How do you talk about them? How do you write about them?

It may well be solvable - I'd like it to be, I spent a couple of years of my spare time toying with visual languages, and I still think as an idea there is a lot of potential. But you need a notation that can be read out in a way that makes semantic sense, the way mathematical notation (despite my very many reservations about mathematical notations) can. And mathematical notation is a good example of all the problems it brings in terms of tool support to even ensure you can reliably typeset it without having to include images of it.

I think it's more likely that we'll see improved interfaces to decorate or explore a textual code-base better, though, possibly with some languages starting to be designed with such tools and representations in mind if/when we start to see something closer to a standard emerge.

exactly this.

in general it is harder to build tools for graphical languages, I think this has been prohibitive. parser are hard, but diagrams require constraint solvers and what not, on top of the parsing.

but we came a long way since 1990, both on our actual abilities to build the stuff (JS and the browser can do powerful visual things) but we also understand functional programming and it's relation to mathematics much better.

anyway, thanks for your comments! Nice thread and you are 100% right with the TODO list; we actually build a tool (here is a basic pre-version https://github.com/wires/roadmap-viewer) to manage the intricate roadmap needed for this big project. Stay tuned! :-)

There's a benefit to both. There's a reason why humans talking about a problem so often get up and walk to the whiteboard to diagram a solution. There's a reason Visio has a process flow diagram template built in. There's a reason infographics are more popular than text articles.

It's easier to reason when you have a visual representation of the solution, even if text is a more concise way to represent complex relationships.

But when we diagram we rarely fill in all the details.

On the contrary, we often diagram to abstract away from details.

That is the challenge a visual programming tool needs to overcome: Most likely we'd need to diagram only some parts. Most likely the level of details and which details we want will depend greatly on who we're talking to and what we're trying to address.

This is exactly why we use petri nets and then we map them to a semantics. Drafting a net represents the "diagram part". You draw what your code is supposed to do as you would do on a whiteboard. But this time you have a lot of formal tools to do checks on the net (is it deadlock? Is it live? Does it have nice properties?). Once you are satisfied with the net, you populate it with meaning: Places of the net get mapped to datatypes and transitions to functions. This is the stage where you start typing stuff in. What you gain is that the translation between ideas (diagrams) and code is formal, so way less prone to error.

The most useful visual programming tools I've seen have either been hyper specific to their use cases or relatively thin abstractions over a workflow engine. IBM's BPM is basically the latter, and iirc it would let you drop in a node with a skeleton function that you'd implement yourself. Very useful for non-technical people, but maybe not what you'd use if you're a software shop.

An aspect that I feel is often ignored, or at least forgotten, is the law of leaky abstraction.

The whole point of visual programming seems to be to abstract more, but inevitably they always hit the exact same problem. The abstraction leaks, and now you have to implement an ugly hack.

The only case I see for visual programming is as a standing for a DSL. If you have a very specific domain where you need a nontechnical person to be able to rewire some things often and autonomously, you might need visual programming (although you might be better off with a 50 line text file)

I think you're right. Another angle is that complex programs have a structure that is difficult to draw on a 2-dimensional plane without a whole lot of cris-crossing. There's just too much interconnectedness. Even if you stick to purely functional code, which is typically fairly easy to represent as a DAG, you'll still get lines cris-crossing whenever you have two or more different values each getting used 2 or more times.

The nice thing about visual programs, is they have less requirements on long term memory, working memory, and use more recognition of items instead recall from memory - which is harder.

They are also more concrete and less abstract, which again, is easier, memory wise.

And maybe because of all of those, they also require less focused attention.

On the other hand, it's possible that it's incidental, that textual languages designers don't care deeply about that niche, or that textual languages(and their libraries) seem to evolve towards power/complexity, or that such requires deep work on an IDE, which is a big barrier-to-entry.

I believe that it's mostly because you need to abstract away from the visual form, be it a picture or text, and that, for complex programs, it's easier to abstract away from text than a picture (being more dense may help, but I suspect it's more than that - the apparent visual structure may be a distraction).

The phrase "CASE" covered a lot more things, but the term itself has vanished because nobody would dream of developing software without a computer unless they absolutely had to. Emacs is a CASE tool, for example.

When rustc is holding my hand and suggesting exactly what I should do it does feel like what the name CASE suggests

awesome comments, and we want to get that experience: a sound type-system and model checker that tells you when things go wrong (statebox' rustc)

but add (formalised) graphical scaffolding to glue parts together.

It is also interesting that another field went from visual diagrams to (almost) entirely text in the 1980s. RTL logic circuit design.

Your response and most others are biased because we have invested years and years of training in reading text.

Fundamentally though, icons and pictures are easier to understand. We can train ourselves to recognize other constructs that don't need to be represented with a fixed set of runes and left to right reading order.

Create a master of two languages one visual and another textual and he will have a real unbiased datapoint of which is better.

Additionally, any visual diagram can be compactly represented in textual form. It's called "good APIs".

Isn't this only true if you've got a formal system for converting the API to a diagram? I mentioned "pic" elsewhere, what were you thinking of?

Luckily we do. Our diagrams are sound and complete for free symmetric monoidal categories, meaning that every diagram can be converted to a morphism in a category. :)

Because anything programmed in a visual language that is successfull has to scale, and when you scale with these tools- the system.out.print.and.hang.to.the.fridge moment comes. Because its a rewrite by the guys who write REAL software.

True. We implement scalability by being able to glue nets together. This allows for a modular design where you can put together many sub-nets doing different things. This is again a purely formal process: It is graphical, yes, but what you are really doing under the hood is manipulating categories (precisely, gluing things together is done via cospans of monoidal categories, I suggest to check Brendan Fong's work about this if you want to know more. There's also a nice paper by John Baez and Jade Master about gluing Petri nets together using something akin to the cospan construction!)

>we have to ask why it hasn't taken off in a more mainstream way.

I also ask why has javascript taken off to be the most popular language in the world?

The answer is because javascript is the best programming language that was ever invented by mankind.

Same with visual programming. Why hasn't it taken off? It must be because it's bad.

Maybe whether something takes off or not has more to do with external factors than to do with how good something is.

Well, I think it's not really possible to divide a concept from the way it is realized. Many implementations of visual languages were badly done, for sure. This doesn't mean we can go from "this implementation of a visual language is bad" to "for all implementations of visual languages it is that they are bad". :)

To paraphrase Ansel Adams (who was speaking of cameras), there's no one best language, only the one best language for what you're doing. Javascript is the best language ever invented by mankind for the one thing that people do with it, which is write applications that can run anywhere. That's a pretty important use case, not a random external factor. This is a great example of the "worse is better" principle.

I agree if your goal is to destroy your brain use brainfuck. If your goal is to use the best language of all time that has ever existed use javascript.