News for hackers. Stuff that matters.
Re:Always bet on text by narag on Tuesday October 14, 2014 @06:34AM (#8452745)
OK, there are so many advantages of text. I wholeheartedly agree and prefer plain text over most "smart" formats.

BUT.

I have this pet idea about source code. Text isn't the optimal format because a program is not a lineal thing, but closer to a tree structure.

Why have we settled on programs written in text? Pretty much for all the reasons you wrote and the fact that we've had bad experiences with other kind of formats in the past. Being able to fall back to plain text when things go wrong is very nice.

But it has its own sort of usability problems. There's an impedance mismatch between text and programs and sometimes it shows. Actually we don't see it more often because we tend to think that text is the way it is done, always was done and always will be done.

Re:Always bet on text by gumby on Tuesday October 14, 2014 @04:21PM (#8455553)
>Text isn't the optimal format because a program is not a lineal thing, but closer to a tree structure.

I can tell you what it's like from experience. The Interlisp D environment used a structure editor rather than a text editor. I found it infuriating and clumsy. Admittedly I had come from the emacs-infused PDP-10/Maclisp & Lispm world, so I gave it several months, but in the end I adapted an Emacs someone else had started and did all my editing in that.

I figure if this would work for any language it would be Lisp, and it didn't work for me. It sounds like a great idea, since if the editor's "buffer structure" is the program structure it's easy to write lambda functions to, say, support refactoring your code. But it was rarely convenient.

The other thing that didn't work for me was that it was of course a mouse-driven interface (this was PARC after all) and I found shifting my hand off the keyboard all the time slowed me down a lot too.

Re:Always bet on text by narag on Wednesday October 15, 2014 @06:24AM (#8457799)
I found it infuriating and clumsy.

I've heard about programs that tried to do that before that had massive usability problems. Your comment seems to confirm that diagnostic.

I figure if this would work for any language it would be Lisp

I guess it would be more of a novelty for other languages.

I found shifting my hand off the keyboard...

That's a big no-no!

EDIT: By the way, I've been experiencing something similar recently, teaching Scratch and AppInventor to my son and a group of children at Coder-Dojo:

http://medialab-prado.es/article/coderdojo

Kids like the mouse interface, but I do find it very limiting and slightly infuriating.

Re:Always bet on text by habitue on Tuesday October 14, 2014 @07:23PM (#8456389)
Have you tried paredit mode for emacs?
Re:Always bet on text by ajuc on Tuesday October 14, 2014 @07:04AM (#8452812)
It's already done in most IDEs and you probably use it :)

Ctrl+click on method invocation - did it jumped to the declaration?

Click on some method name and choose "show all invocations" or sth like that. Here you have one of the possible tree views of code.

Another tree-view of code is visible when you have code folding feature enabled.

Another one - when you debug and show subfields of some variable.

Yes, it's nice to have these additional abstractions over plain text. But these abstractions are inherently leaky, and I much prefer to work with text files than with some binary format to fix the leaks.

Re:Always bet on text by narag on Tuesday October 14, 2014 @08:27AM (#8453077)
It's already done in most IDEs and you probably use it :)

That's a kludge. "True" form is text and the the IDEs add a layers on top. The right way is making the tree be the canonical form and leave text just as an interchange format.

But these abstractions are inherently leaky...

Inherently. That's bold and you give no justification. Anyway I like how you prove my last paragraph. It's been conventional wisdom for so long that you consider text representation as the fundamental form and using a format closer to the real structure a leaky abstraction.

Re:Always bet on text by ajuc on Tuesday October 14, 2014 @10:42AM (#8453601)
The tree would just give you code-folding for free. Callgraphs, searching for references etc would still need to be recalculated by the code indexer. The only advantage is - no parsing step.

Meanwhile you would need to rewrite all the universal text-based tools from scratch specifically for your particular binary format. And this almost never happens so people are left with no way to merge files (Oracle Forms I'm looking at you).

BTW - merging and diffing files is when the abstraction often leaks, too, or at least wants to leak.

For example - when you have 2 trees with every node the same, but root node changed from <interface> to <class>. I guess your tree-based tool shows the whole tree as a difference? What about when you wrap half of the tree in <namespace>?

Textual diff would be 2 lines in both examples.

There are many possible situations, and I admit that in some tree-based approach is better, but there are many situations where you need better granularity than is possible without leaking the lower level. With text format that lower level is human-readable, and you can automerge unsafely (but concisely) and leave fixing the result to human. With binary format if you couldn't detect concise description of the change - you just show "everything was deleted and that new file was added" which isn't particulary helpful to the person that merge changes.

Can you merge word documents or databases? There's certainly market for that.

Re:Always bet on text by narag on Tuesday October 14, 2014 @11:01AM (#8453701)
Callgraphs, searching for references etc would still need to be recalculated by the code indexer.

The what? Sorry, I really don't know what you're talking about. If you want to introduce a reference, logical thing to do is checking it in the same moment.

The only advantage is - no parsing step.

There are many more, see my other reply to icebraining.

About tools, you seem to give more importance to diff than to write programs with a more powerful environment. I can't agree. Also I don't think creating a suitable diff for trees is a hard problem. I just need to convince Linus of the idea and he will write one in a couple of days :-)

Re:Always bet on text by ajuc on Tuesday October 14, 2014 @11:47AM (#8453936)
Oh, my bad, so you write references into the sourcefiles - smart. Your code isn't a tree anymore, but a graph, but other than that it will work.

Of course you still need code indexers for reverse lookups. You can't write "I am called from a,b,c,..." to source files because printf would be several gigabytes and growing each second:) So you still need code indexer to look up the code graphs in your workspace and build the index where this is called from.

Regarding the diff/merge tooling - IMHO it's much more important than powerful editor. If my company forced me to write code in mcedit I could live with that. I would quit straight away if we had no version control, no matter the IDE we would use. I worked for a short time in oracle forms and I won't do that again.

Re: merging of trees - you keep references in the trees so it's not a tree anymore, but graph possibly with cycles - makes it even harder to merge them properly (and you can't do unsafe merges because people won't fix them if it's binary format).

Re:Always bet on text by narag on Wednesday October 15, 2014 @06:37AM (#8457822)
you keep references in the trees so it's not a tree anymore, but graph possibly with cycles

Cycles, why? Tree+links can be treated as a tree visually.

Edit: Oh, I see what you meant with indexer. That information would be in memory. In the database it's stored in just one direction. When loaded it expands bidirectionally.

Re:Always bet on text by ajuc on Wednesday October 15, 2014 @05:32PM (#8461913)
Visualy it's not a problem. But now you need to maintain the links at all times. So all the operations must be aware of the whole program calltree. I think this could cause problems, especially when linking across libraries boundary, doing dependency injection, reflection etc, but I may be wrong.

Would like to see it implemented out of curiosity. My mine problem with merging trees could be solved by @pshc idea so maybe it could work?

Re:Always bet on text by pshc on Tuesday October 14, 2014 @03:44PM (#8455318)
The key to merging trees is that your editor generates the diff on the fly as you write—your editor's undo/redo stack and your version control system become one and the same. Every time you add, move, copy, switch out, or delete a node, the editor notes the operation on your delta log. When you push your code externally, you're really just shipping the log.

With these semantic deltas, merging should be highly unambiguous and automatable, even in degenerate cases.

Re:Always bet on text by ajuc on Wednesday October 15, 2014 @05:23PM (#8461854)
Hm, the history will be dirty with abandoned experiments etc, but so what - great idea. I'd like to use that system for a while, but I'm still not sure if it will work.

I remember using jbpm graph language - it had nice graphical editor, but we still switched to xml view to very often, because it was much faster to work with text.

Maybe it's just a matter of proper tools, but I can't imagine how you allow graphic language to do this for example:

sed s/foo\("bar", ([a-Z0-9]+), "baz"\)/foo\(\1\); bar\(\1\); baz\(\1\);/g

Maybe it's problem with my imagination.

Re:Always bet on text by pshc on Saturday October 18, 2014 @05:08AM (#8474986)
When I look at that line, my brain sees its structure, the different layers... :) It is totally representable without restoring to plaintext. With good fundamentals (ADTs) and good UX (an editor that looks like a text editor but is so much more) we can make it work.
Re:Always bet on text by icebraining on Tuesday October 14, 2014 @08:53AM (#8453162)
That's a kludge. "True" form is text and the the IDEs add a layers on top.

Why is text the "true" form? What's the difference between an IDE that uses text as the "true" form and one which uses a tree as the "true" form but uses text as the UI and serialization format?

Re:Always bet on text by narag on Tuesday October 14, 2014 @10:52AM (#8453659)
Why is text the "true" form?

It shouldn't. It's what the current IDEs do.

The difference is that the program is currently written as text and passed to the compiler, that expects text. The IDEs make a parallel construction to offer some goodies to the programmer.

How I think it should be: the IDE would actually do the syntax checking and reference resolving work, so any program you have written is in fact pre-validated.

There would be performance enhancements, but that's not the only advantage. One example that comes quickly to mind is applying macros, writing GUI generation wizards based on DB schema and in general applying "DRY".

Re:Always bet on text by icebraining on Tuesday October 14, 2014 @04:30PM (#8455602)
How I think it should be: the IDE would actually do the syntax checking and reference resolving work, so any program you have written is in fact pre-validated.

But IDEs already do that. An example I'm familiar with is Java on Eclipse - it validates the code as you're writing it and marking compilation errors. Your others example have been done as well.

You're right that you need an AST to do the properly, but when there's a cheap and reliable way to convert a blob of text to an AST - parsing - the distinction is somewhat irrelevant.

Re:Always bet on text by narag on Wednesday October 15, 2014 @06:18AM (#8457784)
You are arguing that the fallout is manageable. I'd like to simply do it right from the start.
Re:Always bet on text by icebraining on Wednesday October 15, 2014 @07:20AM (#8457909)
No, I'm saying it's an irrelevant implementation detail.
Re:Always bet on text by wtbob on Tuesday October 14, 2014 @09:45AM (#8453340)
That's why my favourite language does represent source code as trees…

That is, in Lisp the source is lists of lists and atoms, which are trees of data. It's pretty cool.

Re:Always bet on text by jalanb on Tuesday October 14, 2014 @10:39PM (#8456936)
All languages actually represent source code as trees. It's just that the shavings are less obvious in other languages.
Re:Always bet on text by pjungwir on Tuesday October 14, 2014 @11:42AM (#8453905)
It would be nice to have an IDE that surfaced something like the AST. Of course lots do to some extent, but I bet there is room for improvement here. This also seems like a place where Lisp would have an advantage since the syntax is so transparent.

For readers of yesterday's article about whether all the "easy" stuff has already been accomplished, here is an easy-to-read survey of the state of the art in diffing trees:

http://useless-factor.blogspot.com/2008/01/matching-diffing-...

That sounds like a pretty interesting topic to research! And I note that his oldest citation is from 1997, and most are from the last ten years. He also briefly mentions "operational transformation," which I agree seems related and is another area of ongoing research. Both topics seem like they would have lots of practical applications, but right now the general-purpose tooling is weak or doesn't exist. So there is room not just for research but also for folks to implement that research.

Re:Always bet on text by kps on Tuesday October 14, 2014 @09:31AM (#8453297)
Our ‘plain text’ is unfortunately not up to the task of representing all human written text; I'm thinking specifically of traditional mathematical notation, which is also a tree structure represented in two dimensions, and ancestral to programming notation, in that we first squeezed mathematical notation down to one dimension¹ and then augmented it with notations for control flow.

¹with a few forgotten exceptions like the Klerer-May system.

Re:Always bet on text by mjcohen on Wednesday October 15, 2014 @01:26AM (#8457271)
TeX, runoff, others.
Re:Always bet on text by vorg on Tuesday October 14, 2014 @07:10AM (#8452838)
> prefer plain text over most "smart" formats

To be usable and easily read, plain text needs word spacing and line breaking, which is a form of "smart" formatting.

> Why have we settled on programs written in text?

Again, text with newlines, indenting, and fixed width fonts, without which it can't be read. So we're talking 2D text here.

Re:Always bet on text by pshc on Tuesday October 14, 2014 @03:37PM (#8455282)
Totally agree! Plaintext is an unnormalized form of program code, and working in it generates all sorts of artificial problems. I've started various pet projects to try to be able to edit the AST naturally, but haven't seen much success yet. The UX is really difficult.

I think the Light Table team is trying to do this now with Eve, although it sounds like they are turning it into something even more revolutionary but further from textual visualization.

Re:Always bet on text by narag on Wednesday October 15, 2014 @06:55AM (#8457859)
working in it generates all sorts of artificial problems.

Indeed. Escaping characters from comments or strings is one obvious example. Programmers serve the compiler instead of the other way around.

I've started various pet projects to try to be able to edit the AST naturally, but haven't seen much success yet. The UX is really difficult.

I've also been working in it some months ago, using SQLite and Lazarus. I hope I can recover the project now.

If you or someone else are interested, feel free to contact me by email, it's (for real) in the profile.

Re:Always bet on text by dennisgorelik on Saturday October 18, 2014 @05:20PM (#8476355)
The solution for graph representation in a program is "program graph over text". The same way as web pages are described as "HTML over text".