The trend as you move away from simple sequences, such as binary numbers, to state machines, toward trees and general graphs, is that problems become rapidly less tractable. The situation degrades strikingly fast: numbers have sophisticated algebras, regular languages are fairly comprehensible; but on the other end, graphs are replete with NP-complete (intractable) problems. Intractable problems aren't something you have to solve, obviously, because you won't get anywhere. But I think that not only are the hard problems in graphs intractable, the hardness of graphs carries over to the simple problems. Nothing is simple on graphs, actually, and trees are not far from graphs. So if unstructured text seems almost too simple, we should feel glad, because trees are already nettlesome even to write straightforward code for. I feel that hierarchies, as seductive as they seem, almost invariably bite; I think it's a good principle to rule out all flat solutions before considering hierarchies as solutions to any problem.

I feel the same way.

I had to write a library to transform JSON from one shape into another, carrying over the values but placing them in different nodes on a new tree. I found that thinking in terms of tree transformations was incredibly hard. The way I solved it was by flattening the JSON, transforming the flattened version, and when I'm done, inflating it back up.

In other words, working with the simpler, "2D" version of a JSON was much simpler than its "3D" tree structure, and writing flatten/unflatten independent from the transforming code was a nice modularity bonus.

It's really all about decomposing the problem and solving its parts.

If you think in terms of the operations that perform the tree transformations, of course you will have difficulty because, you're using a different tree: the syntax tree representing the code of those transformations! Not the syntax tree of the source and destination structure. So you are "one removed" from the real problem. Moreover, since both domains are trees, it's confusing.

It sounds like you could have benefited from a way to express pattern matching using JSON syntax, for destructuring a JSON object, and to represent the synthesis of new JSON using a "JSON quasiquote".

E.g. Lisp transformation with classic destructuring-bind and backquote:

    (destructuring-bind (a (b (c &rest d) e) f (g h)) some-obj
      `((,a ,b) ,c (,@d 1 2 3 ,e) (,f ,g, h)))

There are pattern matching libraries nowadays that do a lot more than destructuring-bind, but it illustrates the basic point.

The destructuring-bind macro writes the code to pull apart some-obj according to the tree picture with embedded variables. The backquote syntax generates the code whose evaluation synthesizes the new tree object according to a template, with the values of expressions indicated by , and ,@ substituted and spliced into the template.

I never thought of using destructuring-bind together with quasiquoting in such a direct way. It does seem that that's a simpler way than what I have done. I'll have to think about it some more. Though we have to consider I had to write this in Ruby, where I don't have access to CL niceties. :)

It did get me thinking, though, so I appreciate the advice. I'm wondering now how to port that idea to Ruby.

More commonly, this destructuring is done in macros, by their "macro lambda lists". destructuring-bind is just a binding gadget which performs almost the same destructuring as macro lambda lists.

(Why almost: because "destructuring lambda lists" don't support the environment parameter of macro lambda lists!)

The editor in http://en.wikipedia.org/wiki/Interlisp did work directly on list structure instead of on top of a character-string editor. (I never studied it carefully since at least at first it worked through a tty interface and that's pretty inherently a pain, like Unix ed.) I think I read somewhere that Deutsch's PDP-1 Lisp pioneered that editing model, though I may be completely misremembering.

When looking at pictures of PDP-1 http://de.wikipedia.org/wiki/PDP-1, one may have a hard time believing someone thought of structure based editing at all.

ps: I rarely hear about http://en.wikipedia.org/wiki/L_Peter_Deutsch but his work is impressive.

It seems to make sense: the interpreter already stores list structure to interpret, there's no space in RAM for a redundant second representation, and going to external storage for the text would practically force batch mode when the store is paper tape or punch cards. That's similar to 70s/80s home computers keeping BASIC code in RAM in tokenized form, except there you had to type a line over again to edit it.

(But on larger machines in the 70s I get the impression Interlisp stayed with structure-editing for its advantages, not just inertia. It was killed off by Common Lisp which owed more to the east-coast Lispers who gave us Emacs.)

Good point. Nowadays the constraint of communicating between processes makes text serializing valuable, when back in the days direct manipulation (I believe there was no processes and no isolation) was the norm ?

I thought emacs text buffer genes came from being implemented on and for *nix ..

I think the point is that there is only the tree structure in lisp. Even though you can embed things with the reader macro, most of the time this is frowned upon specifically because it hides the actual structure.

There are some difficulties in this, of course. Sometimes I don't think in terms of trees of information. This is relatively rare, though.