versteegen

But what about the digital economy? It's interesting that the length of the supply chain for digital goods and knowledge work is nothing like physical goods. Google, Microsoft, and Apple are end to end, don't have to depend on any other software company if they don't want to (until LLMs, anyway, except Google).

> In Experiment 5, we ask Astra for networks that play three MinAtar games, and in Experiment 6, for a network that plays Go.

5 and 6 aren't surprising, merely impressive, a direct extrapolation of human abilityies. I've written transformer weights by hand to implement a simple shortest-path graph search. It takes hours, so an LLM that's >100x faster than me obviously can build super complex networks. It's just like creating any intricate system by hand such as writing machine code or designing computational complexity widgets.

On the other hand what is impressive is the remarkably few thinking tokens Astra needed to do things like analyse networks. That is well beyond human capability: experiments have shown that Astra can perform a huge number of arithmetic calculations per single output token, double Fable or any other. Its brain is wired for mathematics.

...and did it? What did it do?

This is actually not new, I believe GPT 5.2 already had remarkable similar capabilities. I wrote a very elaborate private benchmark involving hand-authoring highly optimised transformer circuitry, and I was shocked that GPT 5.2 could do it. Makes sense that it was specifically trained on building and debugging transformers.

And yet I often tried to edit questions or answers to fix broken formatting and was blocked because I changed too few characters! Only major edits allowed!

Thanks for pushing back, completely agree. GP's 99% trash is a massive exaggeration. It's actually super tough out there for indie game devs, and you have to put up with people lumping your game in with actual trash without a chance.

Getting offtopic... for my father growing up in the Netherlands, the early 50's was still as you describe: an outhouse with a bucket, no running water, an uninsulated house with a roof that let in snow; he caught every illness. They had electricity but no appliances (except a radio?). By the time he was 18, they had a heater, fridge, freezer, washing machine, he had a (rusty) Mercedes and they got a TV soon after. So progress has been very uneven and if you only go back 40 years goods are far more comparable.

Personally, a year or two ago writing like that did not really come across as particular bad or AI generated to me, although it was uncommon for humans to write phrases like that. Writing in an unusual way is not inherently bad, it can make your text interesting... but it is less direct and clear than alternatives. But the style becomes unpleasant if you keep seeing the same patterns over and over such as "X'ed against all of it"

It's kind of weird to say it's "taken off" recently. Yes what we have today is far better than half a year ago and solving new real use cases, but that was far better with more uses than we had half a year before that, etc, for years now. On their best usecases LLMs were already enormously useful years ago, and it's still very spiky.

That is not what Duverger's law says. (But it's a interesting hypothesis so thanks for mentioning it.)

You mean if you used a modern LLM of GPT-2-level quality. Vanilla transformers like GPT 2 are ridiculously inefficient in comparison.

You're asking for something unreasonable. The number of Google searches per day is enormous and they haven't even been able to roll out AI overviews to everyone yet (they're missing in a new Firefox profile I just created). I wouldn't be surprised if the free tier frontier models cost over 100x more to serve than the AI overviews.

It would be more accurate to say they can do math instantaneously without even thinking, at a level far beyond what humans can do. (I assume you're talking about doing arithmetic.)

  TLDR: Astra has 8.6x better odds of doing a reasoning task without CoT than the
  next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward
  pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1)

https://www.lesswrong.com/posts/eRmzz8J8Qkzqvzrgg/astra-can-...

Ugh, people are still saying the Codex limits are more generous. They're not, Claude's are over 2x higher, have been for months! [1] It's just that Claude uses far more tokens, 2-3x is common. Except sometimes GPT will use just as many or even go into a compact loop and then your quota is gone, little headroom for hard tasks.

[1] https://devforth.io/agents-for-code/?sortby=monthly-value And I can confirm the numbers, I subscribe to both and watch the numbers

> Ok the rear red turn signal lights were already a trend in the US before Tesla. But please reverse this too.

I'm surprised. But it shouldn't be up to Tesla; it's illegal and they never get on the road in countries with proper regulations.

I urge you to reconsider your beliefs. You are missing something important because you are thinking in terms of low-dimensional statistics. Deep learning doesn't just fit data, it finds features (abstractions) of the data.

IME using 5.6 Luna and DS V4 Flash, I notice that although they are excellent at programming, even Opus-like in the way they try to debug, the thing they are worst at is inferring user intent and making good decisions with little information. They are absolutely terrible at that, will misinterpret small wording ambiguities. I suspect that's an ability you can't add with RL training, that it requires the depth of understanding from vast pre-training.

I think this is the best and most useful way to measure model intelligence. In my experience it's what really sets apart the capable models from the best. A small model can be RL trained to be extremely good at programming or narrow problem solving for its size (eg 5.6 Luna, DS4 Flash, Qwen 3.6 27B), but even Luna is IME comparatively awful at understanding intent and making good decisions with limited guidance.

If doesn't correspond cleanly. I can see why you draw the link, because LZ compression will replace words with symbols but BPE is a non-contextual entropy encoding while LZ is contextual and adaptive and that makes it very different. I think BPE actually has more in common with Huffman encoding.

That conception of knowledge is interesting, but I think using the label 'knowledge' for it is very problematic, it's too far from common definitions. The fact that you have to carve out an exception for mathematics already shows there's a problem. Because if maths, shouldn't thought experiments also produce new knowledge? You're excluding special and general relativity. It seems to me that what the concept actually describes is "information about the world".

There is a distinction between a compressor for a fixed dataset and one for an unknown population from which we have a sample. The optimal compressor for the sample may be the single best guess for the population, but that's not what Solomonoff induction does. It begins with a prior that allows all possible programs, and it never assigns all probability to the single optimal compressor, so it has no problem with the all-zeroes example.

But the Hutter prize (of which I'm a big fan) is for ever-more-optimal compressors, and in fact many of the solutions don't generalise to other input data without stripping out various tricks.

> For me it’s the massive amount of resources it takes to produce and run one

It's amazing that LLM pretraining is both extremely data inefficient at learning concepts and cognitive functions from the training data compared to humans, while actually being quite efficient at learning facts, memorising things seen just a few times.

I used to likewise think that the resources required to run large transformers were absurd, but the architectures are far more efficient now than 3 years ago and I underestimated just massive the parallelisation advantage of transformers is, how many TFLOPS effective you can get. You can already run amazingly decent LLMs on PCs and phones.

I generally agree with you, but my view has shifted from "we need to augment or replace LLMs" to it there being far more efficient algorithms possible but it not actually being necessary for fulfilling most goals.

You misread. "Pain and suffering" not "death". Of all the pain and suffering in the world, a vast amount of it really is our own fault. Famines and wars shouldn't happen. And if you see a country border with poverty and one side and prosperity on the other you can't say that was the only possibility.

To save anyone else the trouble: discussion there is not really worth looking at (largely a flame war), except: the author of this disproof seems to be a crank, and the disproof's been refuted.

...but we're talking about compaction, and opencode's compaction is (or was) terrible. I've seen so many horrible problems that I keep it disabled (with an envvar flag, because even the config flag to turn it off was broken).

Wow, remarkable. Clearly Aum is a different league from the lone-wolf "Fort Detrick guy", treat the risks separately. But I'll take these questions as rhetorical. (See my reply to the sibling comment.) I can't answer them and I'm not defending the guardrails on Claude; in their current form I too find them pretty ridiculous.

I didn't argue "must be regulated". I'm arguing AI is powerful (at achieving things, and hence has dual-use dangers). Many people won't even admit that, which is the part that really annoys me: they don't even want to have a conversation about risks because somehow AI is not actually a powerful general-purpose tool. Of course everything you said is true, though many of those things aren't very powerful. But the internet and social media and smartphones certainly have been of great utility for terrorism and child exploitation (and probably also causing many would-be-terrorists to get picked up).

Thank you for taking this seriously enough to write this, and anyone else likewise.

But this is attacking a strawman, amateur bioterrorists. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.

Being at risk of killing yourself also just makes success X% less likely, but if X < 90 that doesn't mitigate much.

The old observation that people tend to define AI as whatever computers can't do yet is as true as ever. It's getting a bit absurd, moving from demanding "general intelligence" to replicating human cognitive phenomenology (the experience of cognitive activities). Yet LLMs can already somewhat (confabulation-prone) introspect their own "internal unspoken thoughts" in their residual streams, quite fascinating.

Isn't it ~$3000 per week? Extrapolating from the current limit on Pro plans.