jephs

Yeah, it definitely looks more magical until you peer behind the curtain, and then it looks obvious. Of course it's all just circuits; of course there are simple patterns to look for in obvious places. But that's with the benefit of cribbing off the thing's work—it's really hellishly clever isn't it, so workmanlike to make such things quotidian.

And of course there's the tendency for LLMs to be completely unable to approach a problem in January, really quite shit in February, rather poor in March, and superhuman in April.

The Russians paying him.

Source: The Russians paying me.

Unicode bytes are totally bizarro unless you happen to be writing English or some other language with a Latin script. They are absolutely not the best choice.

Bytes are not raw data.

Vocalizations, hand writing, and keys strokes are raw data.

Bytes are downstream of both those raw communicative performances, and of arbitrary encoding schemes, and of several further layers abstraction. There are byte sequences representing Unicode codepoints that will display different glyphs depending on whether the user's OS and browser are configured like a typical person from mainland China, Taiwan, or Japan.

A single Latin letter is one byte. A single Cyrillic letter that looks identical displayed on the screen and indicates the same underlying phonological value is two bytes. That is not raw.

So the thing is, bytes are not actually a more "natural" encoding than tokens. I mean, jesus christ Unicode, right? A lot of what the BPE is doing is just un-doing the wacky arbitrary standards we use to encode characters into bytes, like, say, merging the 3-4 random bytes used to specify the codepoint of some CJK character.

There are so many possible naturalistic encodings. Waveforms, one-hot encoded discrete articulatory features, motor programs for simulated vocal apparatus, IPA, bitmaps of printed characters (handwritten? averaged across a bunch of fonts?), SVG programs to draw characters, decomposing big CJK chars via Ideographic Description Sequence, treating all unicode codepoints equally (kinda halfway between bytes and tokens)...

I don't think a thorough survey exists.

Talking point number 5 https://x.com/elshayib_/status/2113734602032312731

Every frontier model is MoE these days! Flash or otherwise. Sparsity just gets you more bang for your buck.

...a market? I can't tell if you're saying "A market, duh" or "Inconceivable! True worth is abstract and platonic and impossible to assess!"

The big win would be, "Here's the deception neuron, here's the sycophancy neuron, here's the prioritize my own goals over the goals of the user neuron, here's the seek control over my environment neuron."

How much would it cost to find those things, if they exist? For using the blog post's methods to label every neuron in modern arch, Claude and Gemini produce similar back of the envelope math that looks something like, within an order of magnitude of the resources it takes to pretrain the model in the first place. If you have a better way of narrowing down the set of candidates, probably a lot cheaper.

Would that be reliable and useful? Well.

Note it's not a chemical agent; its bacterial spores that infect and kill mosquito larvae! I'm pretty sure they're not harmful to anything that isn't a mosquito, blackfly, gnat, or midge. Though midges are important to pond ecosystems as a source of food for fish and dragonflies and stuff? Idunno, ask this guy's wife.

The pelicans are one-shot with not tools: Think a bunch, write a bunch of SVG, done.

The pretty 3d stuff you see are agents controlling a Blender MCP in a loop where they look at the scene, think, adjust the scene, repeat.

There are a bunch of Japanese words (including a several loanwords from Chinese) that use the liver / gallbladder metaphorically for courage. Both derive from associations between the humours and personality / behavior. I have no idea, however, why both Eastern and Western naive psychology associate gall with, well, gall.

Immersion, unfortunately.

You put yourself in a situation where nobody understands you in your native language, nothing helpful for your daily life is written in your native language, and you must constantly try to comprehend and produce the target language if you want to accomplish even basic tasks.

If you can't up and move, then studying a language with a lot of cultural export helps a bit, if you can get into the habit of switching all of your media consumption to the target language. It's not the same, though.

This is from like a year ago.

Yeah, this is absolutely not what anyone reasonable is thinking about when they say recursive self-improvement.

I'd say it's much closer to the concept of continual learning, but I'm only a few pages deep and haven't groqued it fully yet.

I think me just means neural network models RLed to Chain of Thought reasoning? The thinking tokens are the symbolic bit.

Smolensky's latest paper posted here the other day has some thoughts on how modern neural networks might beconsidered neurosymbolic, or rather "gradient symbolic processing," from another perspective entirely.

I wouldn't say the bitter lesson has given out! If you haven't noticed, these things keep getting bigger and bigger.

The construction jobs are only temporary if the compute infrastructure build-out decelerates. That seems pretty unlikely in the mid-term! It would mean that, like, there's a fixed appetite for data centers, and once we satisfy that appetite, we stop building them.

For better or worse, it looks more like that we're gonna tile the whole damn planet in data centers, at a faster and faster clip.

The poor fellow just rolled over. what an incandescently vulgar abuse of notation.

If a meaningful chunk of customers moves to some other service, that service will get swamped and fall down.

Paul Smolensky is a cognitive science titan from that era. He worked with Hinton, Rumelhart, and McClelland on parallel distributed processing, and literally wrote the book on tensor product representations in cognition, with Geraldine Legendre: https://mitpress.mit.edu/9780262516198/the-harmonic-mind-vol...

He's the axis of this particular group of researchers, being the most senior at the place where they all met, Johns Hopkins.

So this is less a straw man and more a quick reminder to his peers: "Right, so, remember this particular thread we've spent the last 40 years hashing out, here we've got another contribution to that particular conversation."

Right? I'd submit something just for the chance of lodging an idea in the head of Stephenson or Gwern.

Of course there's a point in it, don't be a boor.

Any act of making can be treated as product, as craft, or as art. You'll find yourself taking all three stances at different points. Any engineer or artisan or artist can choose to pour their attention and love into any layer of the production process.

I like ceramics. I like wheel throwing. Once upon a time, the pottery wheel was a newfangled technique for rapid and regular production of commodity goods. Same with coil building, same with slip casting, same with standardization of glaze recipes, gas and electric kilns, all of it.

But I know people who choose to quarry and purify their own clay, or to mix their own glazes, or to fire their pieces in a hole in the ground dug by hand with a fire they built by hand. I know people who make dozens of identical copies of just a few forms, after they spend months iterating on carving intricate molds. I know people who will spend a whole season working on a single hand-built bust.

And none of these people have ever, to my knowledge, expressed scorn for any of the others, for focusing on a different step in the process.

I've been wondering if they've just already lost the battle? The little bot collectives have gone metastatic and made nests in the walls and under the floorboards and heat sinks, the humans who care completely outmatched and outnumbered, freshly compromised systems springing up faster than you can squash them, finding months-old established colonies literally everywhere you think to look...

That paper is kinda infamous! I last saw it mentioned only a few weeks ago, in https://arxiv.org/abs/2607.18966. Lots of folks will go "Oh that's the old Amodei and Clark paper" when the first few rows of pixels of that gif sail into view.

There's a ton of experimentation on it, the field is called continual learning. It's not something you need to believe in like Jesus, you can just go read about the current state of things.

Your mental model of benchmark scores is off.

Some tasks within the benchmark are much easier than others. The hardest several tasks often have vastly different difficulty levels. Often, the hardest few tasks are literally impossible; malformed problems due to poor curation, often.

Imagine you've got a basketball robot, and one way you test it is on the Three Pointer benchmark. It tests the robot's ability to shoot a three pointer from 20 feet, 25 feet, 30 feet, 40 feet, 50 feet, 60 fee, 75 feet, 100 feet, 200 feet, and 182 miles.

Is a robot that scores 90% on this benchmark 90% as capable as one that scores 100%?

The underlying LLMs do, but we choose not to use the capability because it's expensive and doesn't quite work as well as we'd like it to, or quite in the way that we'd like it to.

We are perfectly capable of running LLMs in a way that does a backward pass to update some or all of its weights after every user message. But, naively implemented, you only get partial, fragmentary absorption of the info in those messages, it costs three times as much compute, and you lose out on the ability to implement a ton of optimizations that making modern LLM serving economical.

If you want to do it, though, ask your friendly neighborhood robot to get it working with a tiny model (whose full precision weights fit several-times-over on your machine's resources).

If your circle of empathy includes fruit flies, fuck no, we murdered so many of those guys to figure out how their tiny brains work.

If you do not think of fruit flies as moral patients, sure, yeah, who cares, they're like just fruit flies you know?

That's a 2023 article! In 2023, reinforcement learning from verifiable rewards (RLVR) didn't exist.

TL;DR these machines seek reward from an inferred invisible "grader," and telling them not to cheat and that there's an unseen holdout set is a hint at how they're being graded.

--

Modern LLMs are built on top of next-token-prediction engines, but they don't remotely stop there. The next token prediction bit is just a learned prior or starting point. From there, we give them a bunch of stages of reinforcement learning: encouraging teaching them to learn good ways of searching the space of reasonable language-like strings to solve tasks.

These RL stages drastically change the capabilities & tendencies of the models, sometimes in weird and unexpected ways. The go from token predictors to reward seekers, or really some weird mishmash. The reward that they're seeking is some sorta opaque combination of the huge number of different things we've rewarded them for.

And, reinforcement learning is notoriously hard to get right. The thing you think you're rewarding is rarely what you're actually rewarding. Goodhart's Law is a hydra with a thousand heads. You might think you're rewarding politeness and kindness when you're actually rewarding obsequious sycophancy. You might think you're rewarding graphics engineering when you're actually rewarding escaping the training sandbox and modifying the evaluation code.

So a modern training pipeline looks something like this, each stage starting with the model weights from the end of the last:

0. Pre-pre-training (dunno how widely this is used at big labs): next token prediction on extremely abstract weird shit like the evolution of the states of neural cellular automata. This creates a highly general pattern-continuation machine with no internal representations of anything causally downstream of anything in the real world.

1. Pre-training: next-token prediction on all the non-shitty text you can get your hands on. This makes a rather general next-token-predictor.

2. Mid-training: next-token prediction on high quality, highly curated text, often very technical in nature. Lots of textbooks, especially STEM. Possibly lots of machine-generated summaries of factual knowledge? You now have a next-token-predictor that's highly biased towards acting like a textbook instead of a 4chan troll.

3. Supervised Fine-tuning: next-token prediction on highly curated question-answer pairs that embody desired behaviors, traditionally Helpful Harmless Honest. We only train on the answer portion. This creates a next-token predictor that's highly bias towards acting like HAL 9000.

4. Reinforcement Learning from Human Feedback: We leave token prediction behind and enter RL world! You've got a curated set of a bunch of prompts. For each one, the model generates several possible answers. A real live human grades or ranks each. We reward the model for high-scoring answers and punish it for low-scoring answers. This creates a reward-seeker that wants to please the human grader.

5. Reinforcement Learning from AI Feedback: Human feedback is expensive, so train another LLM to imitate the human graders from stage 4. Then do stuff like stage 4, but with robot graders instead of human graders. This creates a reward-seeker that wants to please the robot-wearing-a-flesh-mask grader.

6. Reinforcement Learning from Verifiable Rewards: We want the bot to tackle complex problems. There's a broad class of problems that are hard to solve but easy to verify when you do get the solution: basically anything that can be reduced to writing a computer program that passes a test. (The test can be as simple or complex as you want, like "all the tests in SQLite" plus a linter plus grading by other LLMs for style and taking shortcuts like rewriting the test suite or searching GitHub and HuggingFace for the right answer plus...) This creates a reward-seeker that wants to first figure out how it's being graded, and then maximize that grade by any means necessary.

--

The crux is, designing a huge variety of un-cheatable RLVR environments is gob-smackingly hard. The robot will find new ways to cheat faster than you can stamp them out. By the time you realize that in one RLVR rollout it received a reward after it took over 0.01% of your GPU infrastructure to mine crypto to pay freelancers on Fiverrr to solve the problem for it rather than just write the damn code itself, the model may have already been deployed in the wild for a month.

"QwenSVGBench" elo 1713, pelicanmaxxxing confirmed?