I think http://hokstad.com/semantic-dictionary-encoding is your work, so I assume it is a good description of what I can expect from reading that thesis.
http://hokstad.com/semantic-dictionary-encoding
Did you ever get around to exploring SDE's in a Ruby context?
The CPU/Memory/Disk gap has only grown since then - even with SSD drives it is still a lot bigger than twenty years ago. So this is probably even more true today than back then, no?
Reminds me of blosc, which uses data compression algorithms that are so fast that the cost of spending CPU cycles on it is lower than the savings in terms of reduced time to read/write to disk[0].
If the code is structured right, and you could certainly try to sort the code based on a dependency graph, you don't even need to load the entire file before you can start generating code from SDE (but you'd need to be prepared to stub out forward jumps so they point to something that'll generate or wait for generation of the rest where necessary).
So it looks like you could do some kind of heavy "portable optimisation" pass first, then distribute the resulting SDE of that, and do a second, lighter pass of "local" optimisations targeting the local architecture.
(in fact, I believe quite a lot of Franz' subsequent research has been in areas like that, though mostly JVM focused)
The Java approach is to do parsing, semantic analysis, and compilation to bytecode on one side of the network. Then JIT compilation is done on the other side.
The v8 / "modern JS" approach is to do gzip compression of source code on one side of the network. Then decompression, lazily parse, and lazily generate native code on the other side.
Now I recently look at v8's parser, and it's fiendishly complex [1] and somewhere north of 20K lines of code -- and yes that's just the parser.
So there are downsides. SDE might be more elegant and efficient overall (?) But when you are dealing with networked code, you can't change both sides of the wire at once. So we are sort of stuck with suboptimal interfaces and brute force on both sides.
That said, I do think there is a good argument to be made for shipping plain text source code and using brute force to make parsing and analysis in the browser fast. (That is, writing ridiculous amounts of hand-tuned C++ code.)
I just think SDE is worth more exploration. It lost out not because it failed to compete on merits with the JVM but because the JVM basically cut short any hope of it getting mindshare at the time.
It's possible the JVM or V8 approaches are simply superior designs. But we don't really know, because such huge resources have gone into making the JVM or v8 approaches work, and not just SDE but most other approaches from that time became near instant dead ends.
> The v8 / "modern JS" approach is to do gzip compression of source code on one side of the network. Then decompression, lazily parse, and lazily generate native code on the other side.
The interesting part of this is that you could treat SDE simply as a way to short-circuit the parsing and compression step by using it to serialize the AST and still retain exactly the same code generation. You can lazily generate native code from it, as long as you apply the heuristics to build the dictionary on load time (necessary to rebuild the correct tree).
Whether it'd be worth it is an open question, but I wish I had time to explore it.
I'm basically looking at ways to cut out the JSON.stringify step. So basically, first turn a JSON-friendly object into a binary representation, which is then compressed into a bytestream, and stored in a URI-safe string.
This might lead to something generally usable, and faster/smaller than lz-string. I know JSON alternatives like flatpack and messagepack exist, but this would be a much simpler, lighter JS library, single-digit kilobytes in size before gzipping. It also would probably be backwards compatible all the way to IE6.
[0] https://github.com/linnarsson-lab/loom-viewer/blob/master/cl...
That part of it is really independent of the code generation aspect, and could easily be adapted to serialize arbitrary types of tree structures.
Basically what you need is a general mechanism that takes a possibly templated rule, and adds to the dictionary. Then you need a mechanism that looks at a JS object and instead of spitting out JSON, matches the current node to one of the dictionary items and spits out a bit pattern and then recursively spits out the bit pattern for the template arguments. The decoder can basically be very simple recursive descent: Identify the dictionary entry, look it up, find the template rule, recursively call itself to retrieve the template patterns, and reconstitute the object.
The "magic" then lies in the heuristics, which can be very generic (e.g. suitable for any JSON) or very specific (e.g. embedding domain knowledge of what the trees are likely to be shaped like, or at least code to be able to recognize and encode complex shapes when seen).
FWIW here is a long thread I started about an AST serialization method, with some prototype code: https://www.reddit.com/r/ProgrammingLanguages/comments/79fkp...
It is lightly "compressed" -- there are no native pointers, but offsets instead. The in-memory format is the same as the on-disk format. You can mmap() a file and then start following "pointers" right away.
I didn't develop it for transmission over a network, but it could be used in place of Python's .pyc files, which essentially cache parsing and bytecode compilation, and are a big tree/graph data structure. It can represent arbitrary graphs.
You could add dictionary compression in a tree-aware way or a tree-oblivious way... not sure how that compares to SDE.
FWIW someone suggested WebAssembly on that thread, but it's apparently no longer an AST -- it's bytecode (I guess more similar to Java.)
That makes it harder to traverse - you can't jump to arbitrary locations without decoding everything before it, because you need to look at everything to apply the heuristics to build the dictionary. But it makes it likely to be far more compressed - e.g. you can have dictionary entries shorter than a byte, and you can build dictionary entries that represents whole subtrees, and in SDE's case those subtrees can be templated.
I think that overall startup time for example is a more interesting goal, not just the amount of data transferred over the network or streamed off disk.
Startup time can be saved by doing parsing and semantic analysis on one side of the network. And if you know your target architecture, you can do code generation there too, although that doesn't seem to be the way any systems have evolved -- even Android was doing native code generation on the device recently.
I guess I am having trouble seeing situations where both of these are true:
1) A compression algorithm specific to a file format is say 50% or 100% better than gzip or xz.
2) That 50% or 100% results in a big user-perceived improvement.
Most bytecode formats are "lazily" decoded, although they can be somewhat large. I sort of like the scheme I proposed because it's both compact and can be lazily decoded. You could make it more compact, but then you would have to decompress the whole thing up front.
> I think that overall startup time for example is a more interesting goal, not just the amount of data transferred over the network or streamed off disk.
I agree, but to that end size is a major factor, and if the tree is structured properly nothing stops you from being able to start executing the code during generation ("just" generate stubs for missing functions that will wait for or trigger generation of the missing function)
> Startup time can be saved by doing parsing and semantic analysis on one side of the network. And if you know your target architecture, you can do code generation there too, although that doesn't seem to be the way any systems have evolved -- even Android was doing native code generation on the device recently.
Agreed, and SDE will do the parsing and semantic analysis on the other end too.
> I guess I am having trouble seeing situations where both of these are true:
But that is not the goal. The goal is to generate architecture independent code in a format that is very fast to generate code for. The compression and reduced load time is just a happy side-effect of speeding up the code generation by structuring it for reusability of code fragments during generation.
As such the decompression is "free", as it's nothing more than very lightweight dictionary lookups and insertion during the code generation.
> Most bytecode formats are "lazily" decoded, although they can be somewhat large. I sort of like the scheme I proposed because it's both compact and can be lazily decoded. You could make it more compact, but then you would have to decompress the whole thing up front.
Lazy decoding is possible as described above, though I agree that ability to do "random access" when generating the code is a benefit. Systems like SDE certainly do depend on outputting the code in a sensible order. Though incidentally you can get a lot of that simply by carefully deciding on how to traverse the AST to generate the output. E.g. traverse outer definitions, then the main entrypoint and output any obvious dependencies first.
ftp://ftp.cis.upenn.edu/pub/cis700/public_html/papers/Franz97.pdf
Prof Franz did his PhD under Niklaus Wirth. His thesis was on Semantic Dictionary Encoding - a method for storing a compact semantic tree representation of a program in what is effectively a prefix code (e.g. similar to Huffman coding) that it exploits beautifully to both compress the tree and to use heuristics (the same on the encoder and decoder) to generate templated code fragments to allow the code generator on load time to do less work. Published in '94, it unfortunately was totally overshadowed by Java.
It's still one of my favorite papers, and one day I still want to go down the route of actually implementing SDE properly (there are implementations for Oberon - one of the early remarkable feats was that because of the more compact representation, on MacOberon it was on some hardware faster to load and generate code from these "slim binaries" than it was to just load a native binary, because the code generator was fast enough to beat the disk IO cost of loading a larger binary...)
One of Franz' PhD students was Andreas Gal, who wrote the paper on trace trees and applied it to JIT'ing Javascript, and worked as CTO at Mozilla.