patterns

While I (think) understand the sentiment (ergonomics and practicality), it is still worth studying Lisp for a number of reasons. In particular, I would recommend to everyone to study Lisp 1.5 [1] to appreciate how an entire universe of programming languages can be bootstrapped with just a few primitives and equations.

It is true that, today, we have a vast array of impressive tools at our disposal, from parser combinators and generators, code generators to entire language workbenches with projectional editing capabilities. However, if one were to design a language for any reason, having a deep understanding of the expressiveness of computational models such as the lambda calculus would certainly be an "advantage" (especially for those who end up having to use the language): A Lisp/Scheme is as close to interactive lambda calculus as it gets. From there on, one can learn about implementing different evaluation strategies, scoping rules and continuations (William Byrd's "The Most Beautiful Program Ever Written" [2] also to mind).

Now, I am not sure whether homoiconicity tends to lead people astray. It is true that is it not strictly necessary to make a language extensible (e.g., Smalltalk has no macros and is very extensible due to its powerful meta-object protocol and an elegant syntax for closures), but it's still worth studying the concept. For example, writing a metaintpreter in Prolog [3] is surprisingly easy because of its homoiconicity.

[1] https://softwarepreservation.computerhistory.org/LISP/book/L...

[2] https://www.youtube.com/watch?v=OyfBQmvr2Hc

[3] https://www.metalevel.at/acomip/

I picture clusters of information as they move through the computer. Sometimes they look like ships or motorcycles. :-)

Usually, I don't visualize anything in particular. I use tools for visualization, but I prefer to think algebraically. If I work on a complicated system, programming to me feels like like moving through a landscape, with points of interest that I memorize or lay out spatially in a tool, and going through the motions like patching things together, flicking switches, plugging things in and out etc.

I have been pondering about this issue for a while. Maybe it is inevitable that successful systems turn into big balls of mud eventually once the "inflection" point has been reached and (slow) deterioration begins.

It is somewhat of a clichè but I think that (interactive) documentation and tooling can make a difference, but it is very difficult to design the process and the tooling to be as frictionless as possible. Tudor Girba and his team at feenk have been doing a lot of interesting work in that area that's worth a look [1, 2].

The software in question might be an entangled mess, a large part of it might be even inherent due to its requirements or technical constraints, but if that messy web can be readily augmented with background information and sign-posts I think the situation could be significantly improved.

On a related note, there has been a project at Xerox PARC called PIE (Personal Information Environment) [3] which has put forward the idea of organizing software and its artifacts (source code, various kinds of documentation) as a network. Although that particular concept has never been adopted in any major programming system as far as I know, I think that it has huge potential, especially if the software, as a network, can be collaboratively navigated and extended with additional information, where and when needed -- online.

Now that all does not change the fact that we are still dealing with a ball (or a web) of mud, but at least it is accessible and we might have a better chance to understand its evolution and the reasons that made it complicated in the first place.

[1] https://feenk.com/

[2] https://gtoolkit.com/

[3] http://www.bitsavers.org/pdf/xerox/parc/techReports/CSL-81-3...

I think this is a fair assessment. I agree that Smalltalk is by far not a complete solution to the problem of building and maintaining complex system but "only" an attempt.

I found that message passing is an elegant approach to have interoperability on a very basic level. But when protocols and interactions between objects get more complex, it becomes more difficult retain control and comprehension of the evolving system, thus fundamentally better approaches and methods are needed than what is present in a typical Smalltalk system.

You might be interested in watching Alan Kay's seminar on object-oriented programming, in which he sketches some ideas on how to modularize an OO system, notably, using a kind of specification language to describe the functions/needs of components and letting the underlying system figure out how to hook them up and deliver messages automatically (as opposed to the direct message passing style in traditional Smalltalks). The relevant part can be found here [1], but I found the entire talk worth watching, since a whole set of issues with OOP and Smalltalk (difficulties in finding and reusing components, weak generality) is being touched upon.

Unfortunately, as far as I know, none of the critical ideas have been crystallized into a new kind of Smalltalk - which would be more focused on working on sets of components instead of individual objects/classes (or paraphrasing Alan Kay, making "tissues").

[1] https://www.youtube.com/watch?v=QjJaFG63Hlo&t=5775s

Related to this is I watched a fairly recent talk by David May who worked on the implemention of the transputer in the late 80s together with Tony Hoare:

https://m.youtube.com/watch?v=lXUWmHgLiyU

At the end of the talk, he mentions the staggering complexity of the design and manufacturing of chips with billions of transistors. We have long reached a point where nobody fully understands how the chips work. IIRC, only two companies can do the manufacturing.

That modern chips work at all almost seems like a miracle. Do we really need chips this complex?

>I picture three docs, A, B, and C which is a summary composite of things pulled from A & B - C will still need its own manual curation and updating if A and B are changed to remain legible, flowing, and meaningful, and I'd rather have a stale doc than a garbled one.

I think I see what you mean. Garbling, as you mention it, is actually what Xanadu is supposed to prevent. The problem is that it is not explicitely mentioned that a document/version (a collection of pointers to immutable content) is should also be an addressable entity (an part of the "grand address space") and must not change, once it has been published to the database. In particular, if a link, e.g., a comment, is made to a text appearing in a document/version, the link must, in addition to the content addresses, also contain the address of that document/version (In Fig. 13. [1] that is clearly not the case and I think that's a serious flaw).

This way, everything that document C refers to - and is presented - is at the time it was when the composition was made. How revisions are managed is an orthogonal (and important) problem, but with the scheme above we lose no information about the provenance of a composition and can use that information for more dilgent curation.

[1] https://xanadu.com.au/ted/XUsurvey/xuDation.html

I understand. I think your last objection is very valid and perhaps needs far more consideration.

Anyway, I have limited computer access at the moment, but maybe you find the following response I wrote in the meanwhile useful. Ill get back to you.

---

Some remarks that hopefully answer your question:

The Memex was specifically designed as a supplement to memory. As Bush explains in lengthy detail, it is designed to support associative thinking. I think it's best to compare the Memex not to a writing device, but more to a loom. A user would use a Memex to weave connections between records, recall findings by interactively following trails and present them to others. Others understand our conclusions by retracing the our thought patterns. In some sense, the Memex is a bit like a memory pantograph.

Mathematically, what Bush did is to construct a homomorphism to our brain. I think it is important to realize that, when we construct machines like the Memex or try to understand many research efforts behind hypertext. Somewhere in Linda Barnett's historical review, she mentions that hypertext is an attempt to capture the structure of thought.

What differentiates most word processing from a hypertext processor are the underlying structures and operations and ultimately, how they are reflected by the user-interface. The user interfaces and experiences may of course vary greatly in detail, but by large the augmented (and missing!) core capabilities will be the same. For example, one can use a word processor to emulate hypertext activities via cut and paste, but the supporting structure for "loom-like" workflows are missing (meaning bad user-experience), and there will be a loss of information, because there are no connections, no explicitely recorded structure, that a user can interactively and instantly retrace (speed and effort matter greatly here!), since everything has been collapsed to a single body of text. The same goes for annotations, or side trails that have no structure to be hanged onto and have to be put into margins, footnotes or various other implicit structural encodings.

Hypertext links, at least how Ted Nelson conceptualizes them, are applicative. They are not embedded markup elements like in HTML. In Xanadu, a document (also called version) is a collection of pointers and nothing more. The actual content (text, images) and the documents, i.e., the pointer collections, are stored in a database, called the docuverse. Each content is atomic and has a globally unique address. The solution to broken links is a bit radical: Nothing may be deleted from the global database. In other Hypertext systems, such as Hyper-G (later called Hyperwave), content may be deleted and link integrity is ensured by a centralized link server. (If I am not mistaken, the centralized nature of Hyper-G and the resulting scalability problem was its downfall when the Web came). Today, we have plenty of reliable database technologies and the tools for building reliable distributed systems are much, much better, so I think that a distributed, scalable version of Ted Nelson's docuverse scheme can be done, if there is enough interest.

How a document is composed and presented, is entirely up to the viewer. A document is only a collection of pointers and does not contain any instructions for layouting or presenting, though one can address this problem by linking to a layouting document. However, the important point is that processing of documents should be uniform. File formats such as PDF (and HTML!) are the exact opposite of this approach. I don't think that different formats for multimedia content can be entirely avoided, but when it comes to the composition of content there should be only one "format" (or at least very very simple ones).

I hope this answers some of your questions.

Yes, I agree a demo would be good.

I have to catch some sleep, but I will address your questions as good as I can later. In the meanwhile, you might want to take a look at how Xanadu addresses the problems of stale links, and maybe some of your other questions will be answered.

[1] https://xanadu.com.au/ted/XUsurvey/xuDation.html

Also, I highly recommend reading Nelson's 1965 ACM paper I mentioned to better understand the problems hypertext tries to solve and the limitations of classical word processing (which also expands to Google Docs).

The old guy in the video is Ted Nelson, the man who coined the term hypertext, made significant contributions to computer science, inspired two generations of researchers and continues to inspire as his works are being rediscovered.

There have been "big programs" but when the web came, fundamental hypertext research and development on other systems came to a grinding halt. Ted Nelson, and many other researchers, predicted many of the problems that we now face with the Web, notably broken links, copyright and payment as well as usability/user interface issues.

I don't know what an average user is, but what a user typically does or wants to do with a computer is somewhat (pre)determined by its design. Computer systems have, for better or worse, strong influence on what we consider as practical, what we think we need and even what we consider as possible. (Programming languages have a similar effect).

One of the key points of Ted Nelson's research is that much of the writing process is re-arranging, or recombining, individual pieces (text, images, ...) into a bigger whole. In some sense, hypertext provides support for fine-grained modularized writing. It provides mechanisms and structures for combination and recombination. But this requires a "common" hypertext structure that can be easily and conveniently viewed, manipulated and "shared" between applications. Because this form of editing is so fundamental, it should be part of an operating system and an easily accessible "affordance".

The Web is not designed for fine-grained editing and rearranging/recombining content and has started as a compromise to get work done at CERN. For example, following a link is very easy and almost instantaneous, but creating a link is a whole different story, let alone making a collection of related web pages tied to specific inquiries, or, even making a shorter version of a page with some details left out or augmented. Hypertext goes far deeper than this.

Although a bit dated, I recommend reading Ted Nelson's seminal ACM publication in which he touches many issues concerning writing, how we can manage different versions and combinations of a text body (or a series of documents), what the problems are and how they can be technically addressed.

[1] "Complex information processing: a file structure for the complex, the changing and the indeterminate" https://dl.acm.org/doi/pdf/10.1145/800197.806036

Since Smalltalk was mentioned, please consider following points:

1. Smalltalk has first class, live activation records (instances of the class Context). Smalltalk offers the special "variable" thisContext to obtain the current stack frame at any point during method execution.

If an exception raised, the current execution context is suspended and control is transferred to the nearest exception handler, but the entire stack frame remains intact and execution can be resumed at any point or even altered (continuations and prolog like backtracking facilities have been added to Smalltalk without changing the language or the VM).

2. The exception system is implemented in Smalltalk itself. There are no reserved keywords for handling or raising exceptions. The implementation can be studied in the live system and, with some precautions, changed while the entire system it is running.

3. The Smalltalk debugger is not only a tool for diagnosing and fixing errors, it also designed as tool for writing code (or put differently, revising conversational content without having to restart the entire conversation including its state). Few systems offer that workflow out of the box, which brings me to the last point.

4. I said earlier that Racket is different from Common Lisp. It's not only about language syntax, semantics, its implementation or other technicalities. It is also about the culture of a language, its history, its people, how they use a language and ultimately, how they approach and do computing. Even in the same language family tree you will find that there are vast differences, if you take said factors into account, so it might be worthwhile to study Common Lisp with an open mind and how it actually feels in use.

But Racket and Clojure are very different from Lisps such as Common Lisp that embrace the idea of a lively, malleable and explorable environment, which is arguably the biggest idea.

and Smalltalk ;-)!

If done right, a macro system allows you to make your language modular and experiment with new language features in a safe manner without having to change the core language and the compiler. With the macro approach, languages become libraries.

The Racket people took this concept very far. The kernel of the language is very small and well defined. All Racket programs (or more precisely, expressions) are eventually reduced to a handful of syntactic core forms (see [1]). For example, thanks to forms such as (#%variable-reference id) you can specify rules for variable bindings and variable access, for example, w.r.t. to scoping or life-time. With tools such as the Macro Stepper you can fully step through the transformations of any expression in your program, from the highest to the lowest level.

This has numerous benefits. Extensions or modifications of the language can be rolled out (and used!) as libraries. This makes collaboration and testing far easier. Also, if a language feature turns out to be a bad idea, you deprecate the library. You do not have to change the compiler. This allows you to shrink your language and explore different directions without the burden of an ever growing language spec and implementation.

Is it a perfect solution? No. Changing a widely used language always has big impact, but the impact can be compartmentalized and users of the language are given a graceful migration path by updating their libraries, at their choice and pace.

Is Racket perfect? No, not by a long shot. But, frankly, language authors should at least take a look at the possibilities and consider the technological options for controlling the evolution of a language.

https://docs.racket-lang.org/reference/syntax-model.html#%28...

That's interesting. Thanks for the paper. I skimmed through it, but I did not get every detail, since it's been a long time I worked with languages like ML or Haskell and my formal background is a bit rusty at the moment.

I do like the premise that, if I understood correctly, (1) you want to specify your computations with the most suitable representation(s) and you want them to be inter-operable. Also, (2) I find it interesting how the tension between pattern matching on explicit representations and abstract data types with information hiding can be reconciled.

On a less formal side, the Personal Information Environment done at Xerox PARC, extended Smalltalk with perspectives, which addresses point (1) from a different angle [1], but the multiple perspectives system is bolted on and requires tooling. Also the scope is a bit broader, since PIE also addresses the integration of non-functional artifacts (documentation and so forth).

I don't know if that can be done more cleanly without rethinking the entire object and interaction model. Taken (1) a bit farther, you end up with a configuration language (that Alan Kay discussed in a Seminar) and the system is responsible for delivering appropriated objects to the methods (or processes).

For larger systems I consider that as a must, because without that kind of modularity your components accumulate all sorts of different contexts, which makes it difficult to understand the evolution of the system or take requirements out of the system. Extension methods seem more of a work-around than a real solution. Traits are fine but the composition always ends up to be a global fact in your system, even if it is irrelevant for most contexts.

So what we need is a system that can dynamically adapt itself to different situations, in such a way, that the programmer (or user) can adapt the system locally without having to know the entire global configuration -- which does not work if you have conflicting behaviors but you want to keep them for, say, analysis purposes. In the end, we want the system to get us into a state where we can do experiments as fast as possible instead of doing 99% of (volatile!) preparation work all the time.

But this is of course an extremely complicated problem (theoretically and practically even more so with the growth of software) that needs adequate methods and languages. And this is where syndicate might close some gaps, especially since you are applying it to the messy domain of operating systems configuration and the insights we can use from that.

[1] http://www.bitsavers.org/pdf/xerox/parc/techReports/CSL-81-3...

[2] https://www.youtube.com/watch?v=QjJaFG63Hlo&t=4816s

Edit: To make my long winded post a bit clearer, many of it boils down to the illusive context problem. But rethinking the mentioned issues from a conversational point of view seems very promising to me and could address many issues, such as composition and discovery of components (or better yet, behavior), pragmatical concerns such as optimizations, keeping track of requirements -- all put under the umbrella of multi-faceted conversations within the system and its users.

Nice to see Red getting attention. I have been playing with Rebol a while ago and I really liked how easy it is to do program manipulation and writing small DSLs.

However, I found debugging to be very difficult, once I wrote a more complex application. Are there good debugging tools for Red?

Interesting. If I may ask: What made you switch "sides"? And what do you miss in Smalltalks coming from Scheme?

I see. Thanks for the clarification!

My pleasure! I found it to be a refreshing read. Thank you very much for your response. I will study your examples and read up the parts in your dissertation.

Another question, Smalltalk related.

What I noticed in various Smalltalk systems is that there is a lot of code duplication for handling UI interactions (such as dragging or selecting objects in various kinds of views). Garnet, for instance, addresses that problem with Interactors and constraints [1].

Also, in Smalltalk, you will find countless instances of re-implemented tree data-structures e.g., package trees, class trees, UI component trees, which increases maintenance effort and makes the system less uniform than it perhaps could be.

Could Syndicate address these problems?

[1] https://dl.acm.org/doi/abs/10.1145/67449.67512

In your dissertation you mention that "program" internals such as Exceptions should not be asserted as facts and pollute the data-space; only relevant facts/knowledge about the domain should be shared.

I understand this point, but what if you want to express knowledge on how to adapt the technical parts of a system and its infrastructure, not just the "problem" domain? I imagine that Syndicate and its abstractions could be very helpful in this regard.

What are your thoughts?

Edit: For example, you may want to delegate the diagnosis of critical Exceptions to engineers, perhaps suggest resolutions, patches. Would you use a dedicated syndicate network for this? What would be your approach?

"Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation). A prompting service which supplies such information is not a satisfactory solution.

Activities of users at terminals and most application programs should remain unaffected when the internal representation of data is changed and even when some aspects of the external representation are changed.

Changes in data representation will often be needed as a result of changes in query, update, and report traffic and natural growth in the types of stored information."

I agree.

One gripe that I have with functions like map is that it returns a generator, so you have to be careful when reusing results. I fell into this trap a few times.

I'd also like a simpler syntax for closures, it would make writing embedded DSLs less cumbersome.

Oh, this is great. I especially like the fire torch analogy and the animation you came up with (https://divan.dev/images/torch.gif).

That's exactly how I feel when reading code. Recently, I explained the problem to a dear colleague when working on a code base (with about 600K lines of code). I compared the situation to working in a large office building with all lights turned off and you have to skim through thousands of documents with a flashlight.

It's comforting to know that other people feel the same way.

This reminds of Robert Pike's "Systems Research is Irrelevant" speech [1]. Now, 20 years after his speech, we are still stuck with the same notions (such as everything being a string). It's not that there are not plenty of alternatives around, however, expectations are so high that it's almost impossible to make a new computer system economically viable. On the other hand, the hacker and maker scene is very active, some of them building operating systems and hardware such as tiny Lisp-based machines [2] and OSes [3]. (My only gripe is that most of the new "avant-garde" systems are still text/file-based.)

I'd love to see a next wave in personal computing, starting with a clean slate, building on the research, insights and learning from the mistakes that have been made. I have no doubt that it will happen, the question is only when.

As for interoperability: Even on the same platform there are countless problems getting software to talk to each other, so I don't think that a new system will make the situation any worse.

[1] http://www.herpolhode.com/rob/utah2000.pdf

[2] https://www.tindie.com/products/lutherjohnson/makerlisp-ez80...

[3] https://mezzanos.herokuapp.com/

If I remember correctly, Adele Goldberg once remarked that "in Smalltalk everything happens somewhere else". Although there has been ongoing work to collapse the many layers of indirection you find in message-passing systems, I am not convinced that throwing more tools at the problem will address the deeper underlying issues. At some level, you need more concise descriptions of your system and operators to shape its structure and organization; perhaps even compute different organizations. This is very difficult to accomplish in purely object-oriented systems/languages that have no direct and convenient "algebraic" correspondence for composing complex communication structures and specifying relationships such as inheritance. For example, I was very taken by the idea of class definitions as expressions, having seen it in Racket first many years ago.

On the other hand, I believe that when it comes to low-level "machine" work, objects are a good abstraction to model components such as activation records such that they can be uniformly explored and modified (on the fly). But this is perhaps a moot point.

Over the years, I have studied many object/component-oriented systems and come up with a sizable catalogue of message-exchange patterns, plumbing and machinery. My hope is that at some point, this can be crystallized into a language or calculus for specifying systems; and Scheme/Racket is certainly a good language to think about these issues from another perspective (which is worthwhile to preserve).

So I guess, I understand your point. Thanks for raising it, much appreciated.

I wonder whether the migration to Chez Scheme will improve the performance of Dr. Racket which consumes a lot of memory and feels very sluggish.

I have been experimenting with Racket for quite a while and I appreciate the effort that went into making the language extensible. That being said, I wish that the community would have embraced "object-oriented" techniques for building the VM - Ian Piumarta's COLA (Combined Object-Lambda Architecture) comes to mind [1]. I think this would have saved them a lot of performance troubles (but this is mere speculation) and would make the system far more flexible and pleasant to use.

While it's comforting to have a very powerful macro system at your fingertips to change the semantics of the language when needed, many of the macros in the base language and libraries could be eliminated with a more convenient default notation for closures and message-passing (to objects simulating control structures like in Smalltalk).

Racket has one of the best module systems I have worked with but modules cannot be (easily) parameterized. Research on systems such as Self and more recently Newspeak [2] gives ample of demonstration of the benefits (conceptual integrity, security) of having modules as objects.

What Racket also lacks is a good meta-object system with highly reflective capabilities (of say a CLOS or Smalltalk system). This makes it difficult to build tooling such as inspectors or browsers.

I hope that in the future these issues will be addressed.

[1] https://www.piumarta.com/papers/colas-whitepaper.pdf

[2] https://newspeaklanguage.org/

Yes, I had the same experience and I use it in the same way. It's a great book to have on your bookshelf as it covers a lot of topics and the chapters are, iirc, fairly self-contained; but I never read it from cover to cover.

I recommend having a look at "Practical Electronics for Inventors" by Paul Scherz and Simon Monk. The book offers a very good introduction about the basics of electricity with many helpful illustrations, written in a down-to-earth style. In case you are interested in electronics, you will find that the book covers many intermediate/advanced topics such as operational amplifiers with lots of practical examples.