thesephist

Reminds me a lot of Infinite Craft (https://neal.fun/infinite-craft/).

The generative part of language models can make for really fun "single-player" games where you're really competing with the inventiveness of the language model, so there's some sense that you're playing a game with infinite hidden complexity.

Thanks for sharing Simon! I will note that by training an adapter layer between this autoencoder's embedding space and OpenAI's, it's possible to recover a significant amount of detail from text-embedding-ada-002's embeddings with this model too[0]. But as the paper author's reply in a different thread points out, their iterative refinement approach is able to recover much more detail in their research with a smaller model.

[0] https://twitter.com/thesephist/status/1698095739899974031

Definitely possible if supported by transformers.js. If I see enough folks wanting it I'll likely add it at some point.

This is definitely an interesting idea I've also pondered before. In my experience (just speaking from intuition) what's "easy" for LMs to predict often doesn't line up with our human expectations for what's "obvious". Often LLMs will learn seemingly "low information content" statistical correlations that just helps it lower its training loss.

Hi HN! I've found this visualization tool immensely helpful over the years for getting an intuition for how an LLM "sees" some piece of text, and with a bit of elbow grease decided to move all compute to client side so I could make it publicly available.

I've found it particularly useful for

- Understanding exactly how repetition and patterns affect a small LM's ability to predict correctly

- Understanding different tokenization patterns and how it affects model output

- Getting a general sense of how "hard" different prediction tasks are for GPT-style models

Known problems (that I probably won't fix, since this was a kind of one-off project)

- Doesn't work well with Unicode grapheme clusters that are multiple GPT-2 tokens (e.g. emoji, smart quotes)

- Support for other models (maybe later?)

Hey HN! Author here.

For the many Obsidian users here, wanted to share an Obsidian demo/plugin that I saw recently by Justin Smith[0] that I think faithfully carries over a lot of what I liked about this idea into the Obsidian land, complete with a semantic index w/ language models.

If you're an Obsidian user, do check out the demo. I can't take credit for any part of building it, but it's really cool to see the idea in action :)

[0] https://twitter.com/justindsmith/status/1679978286955532296