achrono

Just reproduced the Pac-Man test with Kimi K3.1 Pro and it's able to do it too. I thought it might end up being a splurge but it came in at just $0.36!

Full output with reasoning trace:

https://pastebin.com/jVVMXHFk

Ignoring the checkbox is an utterly offensive move, but repeatedly manipulating it contrary to stated consumer intent is a whole other level. I didn't know we were supposed to take 'frontier' literally in every sense of the word.

I have now witnessed this myself after not believing this at first. Of course, screenshots etc. will hardly prove anything. This needs a proper third-party audit!

I've been suspecting over the last couple of years of the frontier companies using data for training anyway, regardless of training-use consent. "Using" the data doesn't have to mean they literally upload chat transcripts into pretraining datasets. My analogy has been money laundering -- if that can happen at massive scales, surely these companies can and will do the digital/data equivalent derivations/transformations. Even if one could have the access etc. to do so, how exactly would one prove that a given synthetic dataset that OAI/Anthropic uses is derived from particular user conversations that did not consent for the info to be used in training?

Consider, for instance that OpenAI's (consumer) terms say "If you do not want us to use your Content to train our models, you can opt out by following the instructions in this article ." but they also do say "We may use Content to provide, maintain, develop, and improve our Services". [1]

If you think that's quibbling, consider that OpenAI's business terms, in contrast, do state "OpenAI will not use Customer Content to develop or improve the Services, unless Customer explicitly agrees to such use.". [2]

[1] https://archive.is/EcwD8 [2] https://archive.is/yZdAF

My point is simply this: Chinese labs going to great lengths to distill Claude is evidence that distillation is useful but not that that is what makes them so good. That one doesn't even need them to admit it.

Nathan Lambert has made this distinction explicitly: he thinks Chinese labs likely innovate heavily on distillation, while saying he wouldn't call it a crucial factor in their post-training capabilities, in part because the RL work still has to be done by the lab itself -- just like what I said above.

https://www.interconnects.ai/p/how-much-does-distillation-re...

There was also this follow-up [1], pointing out that most of these 7 studies only had something like a 50–65% estimated chance of producing a significant result, yet all 7 did.

In other words, if a whole series of fairly noisy experiments all comes out squeaky clean, you have to wonder whether you're actually seeing the whole series.

[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC3382534

To whoever downvoted this, it would be helpful if you actually reply with something substantive. The post I'm responding to makes sweeping characterizations, and I challenge it with a relatively good heuristic and indirect evidence, and only get downvoted?

Yes, evidence is needed but especially for the claim that distillation is what makes these open models good. Serious citation needed.

Think about it: even if they distill the shit out of frontier models, the model still gotta learn, right?

If anything, as you can see from the K2 Horizon release, aggressive (self-proclaimed) reliance on distillation does not result in a model that has remotely any frontier capability. Try asking K2 Horizon to write iambic pentameter for instance, or even give it the car wash prompt. I tried both these on the Q8 quant for the 7B model and the results were depressing.

Certainly, real-life is the ultimate benchmark. But for various reasons that isn't always immediately possible to go evaluate a model on.

Maybe my Greek idea sounded too high falutin' or simply seemingly clever (I give an example below -- try it out!).

So here's something way simpler that Qwen3.8 27B does not get; only GLM-5.2 and K3 do.

**

Analyze the 2 structural (not semantic) patterns in this text:

1. Morning revient.

2. Birds saluent Morgenlicht.

3. We suivons Waldwege toward maison.

4. Rain tombe plötzlich; we cherchons Schutz beneath sapins.

5. Night vient langsam; we trouvons Wärme near le Feuer, sharing quelques Geschichten together.

**

The answer should get not just the obvious cyclic E-F-G pattern but also the word counts being Fibonacci. Surprisingly few models get this. The only way I got Qwen to do this was on the 2.4T model, with extensive prompt scaffolding. Claude (Opus & Fable), Sol, Grok etc. got it on the first attempt. (All models, all attempts max reasoning level.)

Sounds obvious but just try using the models for anything outside the evals. Take something arcane from Greek history, use it to create a masked linguistic puzzle, which you then ask the model to solve mathematically, all wrapped as an ask to generate ASCII art. Yes, all these elements exist in some form in the evals but the key is in how utterly unconventional the elements are that you pick and in how you combine them.

I have consistently noticed Opus 4.8 and GPT-5.6 far outshine the Chinese models. Gemini is sort of middle of the road, Grok is better than Gemini but not really close to Opus/GPT. OAI & Anthropic still remain unbeaten by a wide margin in my eyes.

At the risk of sounding like a conspiracy theorist, this sounds like a great opportunity to make a statement. US or China, but likelier to be the former. Maybe Clem's on a call with the US government right now?

Well, sorry but look at the methodology on this. The study the post leans on did not study learning. 36 adults went around copying familiar words, and their "typing" is key presses using one index finger (!).

I'm afraid this one's close to going down the replication crisis whirlpool. According to Sol:

"Direct replications found no reliable learning advantage, while a 2024 meta-analysis found only a *small* handwriting benefit but a far larger typing advantage in information captured."

Typed Versus Handwritten Lecture Notes and College Student Achievement: A Meta-Analysis” by Abraham E. Flanigan and colleagues, published in Educational Psychology Review on July 12, 2024

The Greeks themselves, and us in the modern age, really have not given Egypt credit enough for it being the fount from where Greece drew to formulate much of its technology.

There is very little direct textual attestation of course, and I'm not advocating for some Black Athena type of research, but in this context of robotics, simply take note of these two artefacts:

https://www.metmuseum.org/art/collection/search/566713 https://www.metmuseum.org/art/collection/search/544519

Someone well informed on the literature might say, "Homer was dreaming of robots already in the 8th century BC with Hephaestus' workshop", and that might indeed seem like the earliest possible imagination of robotic creation. But note the sheer antiquity of the artefacts above, and if you apply a layer of very reasonable heuristics, it will become apparent that the Greeks were uniquely brilliant innovators, but they did so by brilliantly synthesizing and building on top of various neighboring civilizations and of course their own IE heritage.

Nope, GLM 5.2 is only the latest and greatest in a long line of open-weights models. There are even fully open source models that are comparable to o1-mini (OLMo), or almost-fully-open ones that are comparable to o3 (Nemotron).

I'm super grateful to the open labs (who, importantly, do not have the word 'Open' in their name), all the more so to the likes of Ai2.

There is no magic moat indeed. It is math, engineering and of course copious amounts of data (and the political maneuvering required to secure it, e.g. how most everyone has trained on Anna's Archive by this point).

Beats Opus 4.5 on reasoning you say?

Prompt: If A goes to B who then goes to C, can A send something to C?

Response:

We need to interpret best. The phrase "If A goes to B who then goes to C, can A send something to C?" could be a puzzle about the concept of sending something (like passing a ball) and the relationships.

Scenario: A gives something to B, and B passes it on to C. Question: Can A also give the same thing to C? Answer: Only if A can obtain a second copy (e.g., the thing was duplicated). Otherwise, after handing it to B, A no longer holds it and cannot “send” it unless a copy exists.

[Lots of other unnecessary commentary and "scenarios" that make even lesser sense]

Take a step back and look at this article's diction and the rest of this entire website. Completely AI generated.

All those tokens have to go somewhere

After my own very exhaustive survey, I can just say '+1' and also good to note that OLMo has actually had one independent reproduction (albeit not open) done: https://www.amd.com/en/developer/resources/technical-article...

I often wonder why OLMo and Nemotron aren't more popular -- they are gold-standard / "frontier" of a year ago. If we had more support behind these, seeing a true open-source AI system that legitimately challenges OpenAI & Anthropic might not be far away!

How do we know that today's frontier models are merely scaled up versions of that? Genuine question, since the labs have narrowed what they share over the years to now almost nothing, in terms of how the model was trained and how it works under the hood.

Across 52 professional domains, current frontier models degrade 25% of document content after just 20 interactions!

Towards maximizing the sum of individual happiness, power, beauty and knowledge. Maybe a few other attributes in there, but these are the bare minimum that no civilization would deny for itself.

The question of course is 'how'. For the last few centuries, the answer has been technology.

From the article:

> According to the 115-page complaint, Baig discovered through

> internal security testing that WhatsApp engineers could “move

> or steal user data” including contact information, IP addresses

> and profile photos “without detection or audit trail”.

That isn't really the breach you're making it out to be. Profile photos, unless made private/contacts only, are already publicly visible, and so is "contact information".

Of course these are useful to intelligence services, but this doesn't mean that Baig found they don't have true end-to-end encryption.

I love even more how it's a .md file from well before Markdown even existed.

Really want to know what these "more interesting bits" are that GPT-5-thinking and other models of this calibre cannot do. Unless of course you choose to do them even though these models can in fact do them, in which case, please do share regardless.

Other than banks & ticketing, there is a whole host of things that do in fact need an app.

* Mobile payments

* Navigation

* All manner of IoT devices

* Wearables!

* Digital versions of ID (Mobile Passport Control)

etc.

So no, you can't just use the web.

I think this just further demonstrates the truth behind the truly small & scrappy teams culture at OpenAI that an ex-employee recently shared [1].

Even with the way the presenters talk, you can sort of see that OAI prioritizes speed above most other things, and a naive observer might think they are testing things a million different ways before releasing, but actually, they're not.

If we draw up a 2x2 for Danger (High/Low) versus Publicity (High/Low), it seems to me that OpenAI sure has a lot of hits in the Low-Danger High-Publicity quadrant, but probably also a good number in the High-Danger Low-Publicity quadrant -- extrapolating purely from the sheer capability of these models and the continuing ability of researchers like Pliny to crack through it still.

[1] https://calv.info/openai-reflections

Key highlights in addition to the model quality itself:

* real-time router that quickly decides which model to use based on conversation type, complexity, tool needs, and explicit intent (for example, if you say “think hard about this” in the prompt)

* router is continuously trained on real signals, including when users switch models, preference rates for responses, and measured correctness, improving over time.