stared

My take is slightly different.

- Brainstorming with AI is fine, but it takes skill to use it as food for thought, rather than as a source of answers.

- Do NOT structure your blog post with AI. Once the structure is there, it is almost impossible to humanize it.

- Do NOT use it for writing a draft. Same as above. A few times I did it, and ended up rewriting from scratch. Unsloppification is hard.

- AI for proofreading is wonderful. But I always prompt with "only fix grammar mistakes, do not change the style" and actually look at all changes. If not kept in check, it too eagerly homogenizes style.

In the line, I recommend „The Egg” by Andy Weir https://www.galactanet.com/oneoff/theegg_mod.html

If you like, you can run these tests yourself as well. It is ~$500 per a single combination, and assuming no failed runs.

Nope.

If you would like to do so, it is easy (and orders of magnitude cheaper) than running benchmarks.

Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration.

On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same.

So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.

A single result is binary. All we get from a run is which tasks were solved, which weren’t.

Nice! Sometimes the simplest approaches work the best.

I am curious what's the actual formula.

I mean, there so many headers and layers, it is tricky to make a choice that will resonate with our intuition . Is it some weighted average? Or maybe ablation test?

Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative".

It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between.

Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas.

That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).

I wrote this blog post myself, with AI for proofreading (typos and grammar, but not style). There were a few singular sentences for which I had a writer's block, but not much besides that.

So, if there are irrelevant remarks, these are mine. :)

Charts are vibe-coded - but it took quite a bit of hand-holding to get something decent. And the logarithmic scale for model size is my conscious choice (against Claude's initial ideas).

Nice! One thing I am missing is an easy „go up” a taxonomy group.

Is it a subtle effect that cannot be explained by Newtonian gravity (i.e. a different potential affecting, V(z) in the Hamiltonian)?

Don’t ask. Start from a few blog posts and see traction, read feedback.

You will also see hos long it takes - and what is thd difference between an idea and making it real.

An interesting catch!

Maybe it suffices to make this pre-title "the ai-design-slop fingerprint · defs 2026.09" lowercase.

In any case, from what I see what is AI aesthetics on the website: pre-title and horizontal lines.

With Internet, we live in an always-on culture.

It contrasts with times before, when we actually had to wait for a monthly magazine, and even if we wanted to watch a movie, it was aired (say) the next Tue 7PM.

While now we have all convenience, there is no anticipation - or rest.

While I like this index, calling in "Intelligence" might be confusing - it is a mix of coding and knowledge.

Compare and contrast with ARC-AGI, BabaIsBench (https://quesma.com/benchmarks/babaisbench/), or MazeBench (https://mazebench.com/blog?post=introducing-mazebench).

In particular, in one Baba Is Bench post (https://quesma.com/blog/baba-is-aug-2026/), while quoting a Pareto frontier chart from AA, I noted:

> Intelligence Index vs. Cost per Intelligence Index Task from Artificial Analysis. Note that it is based on score of benchmarks like Terminal-Bench v2.1, SciCode, Humanity’s Last Exam, GPQA Diamond - not necessarily fluid intelligence like in abstract puzzle games of ARC-AGI-3 or Baba is You.

Compare and contrast with tests on Qwen3.6 27B:

* how does quantization hurts pelican-drawing skills (https://quesma.com/blog/qwen-quantization-quality/)

* how does it hurt knowledge (https://quesma.com/blog/quantization-hurts-knowledge/)

Repo for reproducible science: https://github.com/stared/mushroom-hunting-llm-bench