News for hackers. Stuff that matters.
Re:Literate programming: presenting code in human order by jarmitage on Thursday July 07, 2016 @08:18AM (#12048821)
Peter Norvig's comment hits the nail on the head for me:

" Peter Norvig 6 July 2016 at 11:47 I think the problem with Literate Programming is that assumes there is a single best order of presentation of the explanation. I agree that the order imposed by the compiler is not always best, but different readers have different purposes. You don’t read documentation like a novel, cover to cover. You read the parts that you need for the task(s) you want to do now. What would be ideal is a tool to help construct such paths for each reader, just-in-time; not a tool that makes the author choose a single pth for all readers."

Has anyone attempted something like this?

I've heard you can do transclusion in org-mode which might be a starting point

Edit: Some initial ideas:

- Code can be deconstructed into blocks, and where the code is not self-documenting, prose can be added. Or even visualisations etc if a block is conceptually tricky to grok. You could even have MOOC-style validations to verify reader understanding for each block.

- Some kind of topology of the blocks should be generated based on how they interact and how they are conceptually related

- The author creates a few 'starting points' for different audiences, e.g. 'if you've used X before, start at Y'

- From there next blocks to read are auto-suggested to the user. A map or network diagram of all blocks is also provided so the reader can chart their progress and see where the 'big ideas' lie.

Edit 2: This also really reminds me of Bret Victor's 'Humane Representation of Thought' lecture (https://vimeo.com/115154289, 45:40) where he says there is a conflict between code being an engineering specification and an authored work meant to be read by humans.

Re:Literate programming: presenting code in human order by EdwardCoffin on Thursday July 07, 2016 @10:57AM (#12049706)
I don't understand why there not being a single best order of presentation for explanation is grounds to dismiss the whole endeavour. I bet that there are many pretty good orders for humans that are an improvement over the order the compiler wants the stuff in. Besides, the compiler doesn't demand a single best order, most programs could be re-ordered in a number of ways acceptable to the compiler (my point being that this criticism seems to hold literate programming to a standard that the incumbent of compiler-required ordering is not held to.)

Edit: added the parenthetical comment

Re:Literate programming: presenting code in human order by jarmitage on Thursday July 07, 2016 @11:46AM (#12050041)
Yeah I agree. What I would imagine a tool like this would do is help to find and navigate the pretty good order(s) whether they have been hand crafted at first by the author or not.

A comparison would be how desire paths form and are often turned into real paths.

I like your parenthetical observation too!

Re:Literate programming: presenting code in human order by EdwardCoffin on Thursday July 07, 2016 @12:17PM (#12050261)
There was an interview with Guy Steele in the book Coders at Work, and in one bit he had something to say about actually referring to the (literate) source code for TeX [1]:

Sometimes I've got a specific goal because I'm trying to solve a problem. There have been exactly two times, I think, that I was not able to fix a bug in my TeX macros by reading The TeXbook and it was necessary to go ahead and read TeX: The Program to find out exactly how a feature worked. In each case I was able to find my answer in 15 minutes because TeX: The Program is so well documented and cross-referenced. That, in itself, is an eye-opener--the fact that a program can be so organized and so documented, so indexed, that you can find something quickly.

The other thing I learned from it is how a master programmer organizes data structures, how he organizes the code so as to make it easier to read. Knuth carefully laid out TeX: The Program so you could almost read it as a novel or something. You could read it in a linear pass. You'd probably want to do some jumping back and forth as you encountered various things. Of course, it was an enormous amount of work on his part, which is why very few programs have been done that way.

[1] link to the page (337) in Google Books https://books.google.ca/books?id=2kMIqdfyT8kC&pg=PA337

Re:Literate programming: presenting code in human order by jarmitage on Thursday July 07, 2016 @01:10PM (#12050683)
That's a great anecdote and makes we want to read TeX: The Program.

I really think there's something in the 'jumping back and forth' of the reader that could be leveraged, like if there was a way of recording, distributing, aggregating, analysing it etc.

Re:Literate programming: presenting code in human order by EdwardCoffin on Thursday July 07, 2016 @01:24PM (#12050765)
It doesn't add that much to the above, but the now defunct Bookpool had a column where they asked various authors for lists of their favourite 10 computer books, and they asked Guy Steele [1]. His first entry was:

Computers & Typesetting, Volumes A-E Boxed Set by Donald E. Knuth -- I'll read anything that Don Knuth writes, and he has written quite a bit, including Surreal Numbers and 3:16 Bible Texts Illuminated as well as the famous Art of Computer Programming series. But this five-volume set is my favorite. The two volumes titled (and sold separately) as The TeXbook and The METAFONT book are well-known, but what I really recommend to you are TeX: The Program and METAFONT: The Program, because these are simply the best-written, best-documented, best-debugged programs of their size ever published. They reward careful study.

I've tried to read the source for TeX: The Program (I actually have the boxed set he mentions), but was impeded by my lack of knowledge of TeX. I keep meaning to get back to it.

[1] Guy's list in archive.org: http://web.archive.org/web/20080422183814/http://www.bookpoo...

Re:Literate programming: presenting code in human order by akkartik on Thursday July 07, 2016 @02:07PM (#12051069)
I use a layer-based approach described at http://akkartik.name/post/wart-layers in https://github.com/akkartik/mu. It's 20kLoC of literate code, and I was super concerned about optimizing it for non-linear reading, because that's how I need to browse the code for myself. It's not a system to generate multiple views like you want; instead, it's a single view that I find easy to skim and easy to reorganize at will:

a) Each layer puts more important stuff up top. So you can start skimming from the first layer, but you don't have to read all the way down each layer, just get a sense of what it provides and why.

b) There's not much emphasis on lengthy comments (which would get linear). Instead just the order in which code is presented does a lot of heavy lifting.

c) The tangled code is intended to be readable (unlike Knuth's original tools), so I often jump down into it when I feel the need. Error messages and debugger support do show lines in the original literate sources, however.

Re:Literate programming: presenting code in human order by jarmitage on Thursday July 07, 2016 @02:59PM (#12051409)
Fascinating, thanks for sharing.

Tangential, but reading your website ("I'm working on ways to better convey the global structure of programs"), I've been talking a lot with a visually impaired researcher who works in Auditory Display. We've been talking about designing interactive audio displays to offer overviews of things like webpages and code.

I'll definitely keep your work as a reference if that turns into a project!

Re:Literate programming: presenting code in human order by akkartik on Thursday July 07, 2016 @03:07PM (#12051453)
I'd love that! I couldn't find an email for you but if you want to ever get in touch my address is on my profile.
Re:Literate programming: presenting code in human order by anonymousguy on Thursday July 07, 2016 @01:12PM (#12050699)
> Has anyone attempted something like this?

Yes, I write all my personal projects in human order. To me human order is when the flow control of human consumption and computer execution are most closely aligned. I attempt to achieve this with depth and order.

1) Have a giant library that defines your library or application.

2) Inside the giant function nest child functions for the primary tasks in the order with which they will execute. In the case of the following example the options evaluation is first, followed by the lexer/parser is the second child task, followed by various code presentation tasks, and finally by an analysis task. http://prettydiff.com/lib/jspretty.js

3) Break the child tasks down, as necessary, into reusable components.

4) I also believe a reference should always be declared before it used, which can dictate the order of reference declaration.

My opinion is that when the flow control of the application is unclear you are wasting time during maintenance. You may not know where a problem is occurring, but if the flow of the application is immediately clear from reading the code you immediately know where to start, where to step to next, and where to stop. There is minimal guessing and it doesn't require breakpoints to figure it out.

Re:Literate programming: presenting code in human order by h8liu on Thursday July 07, 2016 @11:31AM (#12049951)
My hobby project, with dependency graph visualized:

https://e8vm.io/e8vm

Source code:

https://github.com/e8vm/e8vm

Re:Literate programming: presenting code in human order by jarmitage on Thursday July 07, 2016 @11:43AM (#12050016)
That is so cool, thanks for sharing.

Is it possible to use the vis tool on other codebases?

Re:Literate programming: presenting code in human order by h8liu on Thursday July 07, 2016 @12:06PM (#12050181)
The layout engine is in this package:

https://godoc.org/e8vm.io/e8vm/dagvis

it is possible to layout any DAG (directed graph with no circles).

The dependency graph of golang programs are generated by this package:

https://godoc.org/e8vm.io/tools/godep

I have not very well packaged the drawing part (sorry), but the essentials are here:

https://github.com/e8vm/shanhu/blob/master/web/js/dag.js

it is using d3.

Re:Literate programming: presenting code in human order by jarmitage on Thursday July 07, 2016 @01:11PM (#12050695)
Super stuff, look forward to diving in deeper!

Are you planning to extend this aspect of the project? What are your other plans for the project?