News for hackers. Stuff that matters.
Re:Csexp: S-Expressions over the Network by JonChesterfield on Sunday June 11, 2023 @01:53PM (#36283721)
Doesn't say anything about cycles or what datatypes can be serialised. In general lisps _can't_ print arbitrary objects in a form that can be read back in. E.g.

    Welcome to Racket v7.9 [bc].
    > (lambda (x) x)
    #<procedure>
7.9 doesn't have csexp, was curious what it would do with a function object but not quite curious enough to install a racket from outside the package manager. It seems relatively likely that racket would have some serialisation feature that can round trip arbitrary objects through a bytestring though initial guesses haven't worked out for me

    > (require racket/serialize)
    > (serializable? (lambda (x) x))
    #f
Still, being able to take some arbitrary closure and kick it over the network to another racket instance would be great. It's not immediately obvious that csexp can do that.
Re:Csexp: S-Expressions over the Network by qawwads on Sunday June 11, 2023 @02:26PM (#36284150)
>Still, being able to take some arbitrary closure and kick it over the network to another racket instance would be great. It's not immediately obvious that csexp can do that.

I'm stupid but once your list is reveived by the remote host, isn't executing it only an (eval) away?

Re:Csexp: S-Expressions over the Network by jdkardia on Sunday June 11, 2023 @02:47PM (#36284358)
I'm no lisper, but I'm pretty sure it'd take more than an eval because of the context a closure carries with it.

if you were sending a pure, anonymous function, that would work fine. But if you're making a closure that makes use of any surrounding variables, libraries, or state, that's a non trivial set of information to transmit across the wire.

Re:Csexp: S-Expressions over the Network by alexisread on Sunday June 11, 2023 @03:39PM (#36284843)
I think this is where Delimited Dynamic binding would come in to play: https://okmij.org/ftp/papers/DDBinding.pdf

I think that you would be able to freeze certain variables in the closure, and have others open eg. API_ENDPOINT variable might be configured to change between environments.

Something like https://github.com/GiacomoCau/wat-js implements delimited continuations, ddbinding and algebraic effects (registered handlers for I/O etc) - pushing continuations over the network should be easier with these facilities.

Re:Csexp: S-Expressions over the Network by EddieJLSH on Sunday June 11, 2023 @02:47PM (#36284359)
Yeah that was my understanding, I've considered using this before (sending objects over the wire and eval'ing them) but never implemented it. Lisp seems very good for this sort of stuff.
Re:Csexp: S-Expressions over the Network by nerdponx on Sunday June 11, 2023 @02:02PM (#36283848)
Common Lisp has a reader macro for defining self-referential data structures: https://stackoverflow.com/a/605729

Maybe it wouldn't be so hard to extend this format to include a notation like that.

Re:Csexp: S-Expressions over the Network by convolvatron on Sunday June 11, 2023 @02:33PM (#36284233)
I used to do this on lispms...just `(lambda (x) ... ,z ...) assuming z is small

what you would really do is make a lookup table and keep the closure locally and send a reference and forth. idk why we don't see this more, its a pretty straightforward way to lash together some programs

Re:Csexp: S-Expressions over the Network by correnos on Sunday June 11, 2023 @03:44PM (#36284880)
The docs could do a better job of spelling it out, but csexp's really do only represent "trees of bytestrings". They're not a particularly rich format. If you're looking for "transfer this lambda over the network" semantics that's obviously a lot more complicated, but there's at least some research-project Schemes that've done it. The one I know of is Termite Scheme, which builds an Erlang-style distributed process model.
Re:Csexp: S-Expressions over the Network by nerdponx on Sunday June 11, 2023 @06:19PM (#36286313)
If that is indeed the case, then is it any better than an existing format like MessagePack?
Re:Csexp: S-Expressions over the Network by JonChesterfield on Sunday June 11, 2023 @08:45PM (#36287271)
I expect going digging would reveal that csexp is decades older than msgpack. As a format I agree it doesn't look great though.

May 1997 for csexp. https://web.archive.org/web/20230219084819/http://people.csa...

Msgpack less clear, maybe 2008. JSON is 2000ish.

Re:Csexp: S-Expressions over the Network by aportnoy on Sunday June 11, 2023 @04:03PM (#36285051)
> In general lisps _can't_ print arbitrary objects in a form that can be read back in.

Hmm, why not?

Re:Csexp: S-Expressions over the Network by JonChesterfield on Sunday June 11, 2023 @05:20PM (#36285815)
I haven't worked on any of the classic lisp implementations but can speculate. I suppose the ratio of usefulness to difficulty of implementation doesn't work out.

Mutable state with serialise/deserialise either breaks the aliasing or needs some way to keep the referenced variables alive (nasty interaction with garbage collection) and to re-establish aliasing on deserialise.

Serialising a DAG to a tree tends to duplicate the previously shared nodes. There's then the question of whether deserialising the tree should re-establish the original sharing or some other sharing. Serialising a graph to a (finite) tree needs some notation to represent the cycles.

It looks to me like mutable state is the blocker but I'd be interested in other opinions.

Re:Csexp: S-Expressions over the Network by aportnoy on Sunday June 11, 2023 @05:45PM (#36286054)
Yeah, something like "a" in the below code wouldn't have a "natural" serialization I don't think:

  (define a (list "a"))
  (define b (list "b"))
  (set-cdr! a b)
  (set-cdr! b a)
Re:Csexp: S-Expressions over the Network by cstrahan on Sunday June 11, 2023 @05:43PM (#36286036)
How would one print a function with a free variable? Consider this (but imagine there is surrounding code which defines y):

    (lambda () y)
How can we print this in such a way that preserves semantics?

You can’t just print the original source definition, as y would then be undefined (or I suppose it could be defined to be whatever y is in scope at the deserialization call site or something thereabouts, but that wouldn’t guarantee that the deserialized function is equivalent to the other with respect to how it behaves, despite the source being identical).

You can’t just inline the serialization of y, because the original function returns a particular reference, and there’s no guarantee that the rest of the dependent code will not expect a stable, unique (and possibly mutable/state-full) instance of this value.

I suppose you could, on the side, also serialize the entire program state (or rather, at least the stack and heap), but then things get tricky when any object refers to something like an opaque pointer (even if you have an extent you can safely copy, you’d have to handle issues like different virtual addresses being mapped) or file descriptors, etc.

But maybe all those challenges can be overcome — so I’ll ask: how would you go about serializing an arbitrary object in such a way that one could deserialize an equivalent object?

Re:Csexp: S-Expressions over the Network by endgame on Sunday June 11, 2023 @06:32PM (#36286395)
> How can we print this in such a way that preserves semantics?

Maybe try something like normalisation-by-evaluation, but if you go to look up a free variable, replace that lookup with a little bit of syntax tree instead?

Re:Csexp: S-Expressions over the Network by aportnoy on Sunday June 11, 2023 @05:59PM (#36286162)
Everything is an object that is either a cons cell or an atom. Associate the address of each object with a unique index.

Serialize each object as a pair (index, obj), where 'index' maps back to the original address where the object was stored, and 'obj' is a pair of indices if the object is a cons cell, or the appropriate representation (string literal, integer, etc.) if the object is an atom.

Then to deserialize allocate a memory location for every index and load the objects as is. Then replace the indices with the corresponding (new) addresses.

Re:Csexp: S-Expressions over the Network by aportnoy on Sunday June 11, 2023 @06:11PM (#36286253)
This definitely isn't "printing" in the original sense, just a serialization algorithm.
Re:Csexp: S-Expressions over the Network by aportnoy on Sunday June 11, 2023 @06:09PM (#36286240)
In the original LISP with dynamic scoping I think it would be fair to serialize that expression as is: "just use whatever value is bound to 'y' at the time of evaluation, if any".
Re:Csexp: S-Expressions over the Network by kazinator on Sunday June 11, 2023 @11:58PM (#36288290)
In fact, the Racket dialect being discussed has a facility for this:

https://docs.racket-lang.org/web-server-internal/closure.htm...

It is correct that usually Lisps do not have a print-read consistency for every object, just objects that someone has decided to care about. Most Common Lisps are like this. However, note that many of them have image saving. Image saving traverses the heap and serializes every object. That includes lexical closures. Their code vector, environment vector(s): everything is traversed. It's just not a printed notation!

If some object isn't printable and someone wants it badly to be, it can be arranged somehow.

It boils down to pragmatics.

Foreign objects (FFI handles holding foreign pointers) are going to be tricky; you would need some hooking mechanism to try to do something meaningful.

Image saving already cannot do some things, like saving open network connections such that they come up connected in a restarted image.

Re:Csexp: S-Expressions over the Network by zelphirkalt on Monday June 12, 2023 @02:39AM (#36289256)
The Racket webserver had serializable-lambda, but iirc it requires you to also make everything you use in it serializable (?). So it is kind of infactious like async in JS. You need to write your code with it in mind. Racketeers please correct me with an example of ad-hoc making a lambda serializable, if I am wrong about this.