tedsanders

Sounds like arena.ai: https://arena.ai/leaderboard

People vote on pairs of model responses and they aggregate everyone's votes into a leaderboard.

One thing to be aware of that the distribution of prompts on arena.ai can be pretty different than real-world usage. The people who spend time on that website casting votes tend to be more interested in entertainment than professional productivity. And turn-based voting can punish useful behavior like asking clarifying questions instead of guessing at assumptions.

To elaborate, I can certainly ask each model to write a doc about something I know well and tell you which output I like better.

But to assess whether my preference on my problem is going to generalize across other people, other programming languages, other documentation types/formats, etc., and then assess statistical significance is a much broader question that I personally don't have the data to answer. I'd rather make no recommendation than a poorly informed recommendation.

The error bars on eval graphs are commonly like ±5%. It's fairly common to get out-of-order points when the gap between them is only a few percentage points. If we ran it again, or ran it with minor alterations, I'd expect the ordering to be expected. In general, we don't massage any of our eval numbers and for good statistical hygiene we'll report the first run we do, even if later runs show slightly smoother curves.

(I work at OpenAI.)

Very fair. There's always a tradeoff in the number of products, and it's hard to make everyone happy. My suspicion is that the optimal number is probably a bit higher for developers on our API (who often build their own internal evals) than for individual users (who don't want to waste time re-optimizing every month). We're excited by Terra's uniquely fast computer use, but if it traction is poor, maybe we'll go back to 3. It all depends on the market signal - we only want to ship models people find uniquely useful.

These days, version number signifies a few things at once: the order of release, the relationship to other releases, and the magnitude of improvement. So something like 6.1 would indicate it was launched after 5.6, may share some similarities to other 6.1 models in terms of training data / personality (which is why it's not 5.7), and isn't a giant enough leap to merit the headlines and expectations that would come with a 7.

Back in the day we used numbers to also sometimes indicate size, which could make it extra confusing: e.g., we launched GPT-4.1 (an incremental update on GPT-4) after GPT-4.5 (a new model that was a big jump over GPT-4).

> Do you then recommend Sol for technical documentation?

I don't know, to be honest. Technical documentation is such a multidimensional skill, and I also don't know your budget. I'd imagine Terra might have the most readable prose, but Astra might be more likely to catch bugs or mistakes to highlight. If the documentation is about a more esoteric language or situation, Astra might have a greater edge than if it's a common language or situation. If you have a long list of documentation guidelines, that might also give more of an edge to Astra. I wouldn't want to make a recommendation without doing a lot of testing, and I haven't done a lot of testing.

I promise you this would become a giant mess. No one wants to pick GPT-good-at-writing-but-not-in-documents-and-decent-at-instruction-following-but-has-some-known-quirks-to-be-aware-of-p.s.-good-at-knock-knock-jokes-MAY-2027.

Even "writing" as a single concept is very multidimensional. There's fiction writing, nonfiction for experts, nonfiction for learners, deep research, quick lookup Q&A, copyediting, writing in documents / artifacts, instruction following about style, and default quirks (e.g., em dashes, negation, loving the word "tapestry"). Some models are great writers but less diverse, meaning people prefer the first story they read but disprefer the tenth. How does one label such a model?

What would you have like to seen instead?

The problem with small, medium, large naming is that they don't last. In a few years our names would have to advance to labels like giant, behemoth, gargantua. This is different than clothing, where we know that in five years humans will be most the same size as today.

As a point of comparison, GPT-3 was launched in a series of small, medium, large, and XL. Back then, XL meant a mere 1.3 billion parameters!

Same reason that Apple doesn't name its series of phones the iPhone cheap, iPhone regular, iPhone semi-expensive, and iPhone expensive. It's poor branding, isn't easily extended to future models, and gets confusing when comparing across model generations, where an iPhone 14 expensive might be cheaper than an iPhone 16 cheap.

I work at OpenAI and I can try to explain our logic.

In general, LLM performance is incredibly multidimensional. There's no guarantee of a fixed ordering where the model that is best at math is also the model that's best at jokes. Despite that complexity, much of the variation in performance is explained by just a single variable: model size. In general, models that are bigger will (a) be more performant / reliable, (b) cost more per token, and (c) have better recall of esoteric facts. As you note, our models are ordered:

Luna < Terra < Sol < Astra

Even today, I'd recommend this as a general starting point for picking models. If you care about quality only, pick Astra. If you're a bit price sensitive, pick Sol. If you're very price sensitive, pick Terra or Luna.

Of course, model size doesn't capture all variation. Models that are trained later might have new data or new techniques or new architectures that earlier models lack. You can see this with models from all providers - Sonnet 5.5 outperforms Fable 5.1 in ways, and Gemini 3.8 Flash outperforms 3.1 Pro in ways.

In the case of GPT-6.1 Terra, I'd say Terra is probably better than Astra and Sol at technical communication, similar on vision and computer use (but much faster!), and a bit worse on most other things. For API developers, it's a nice midrange option if you want something cheaper than Sol but a lot smarter than Luna (especially for vision / computer use / communication). For subscribers, it's not a bombshell release - if you're hitting usage limits, maybe it helps you go a bit further, and if you want clearer communication or faster computer use, it can be a nice option to select.

If you have suggestions on how to name our products better, I'm all ears. Our intention is to make things clear and simple, even if we miss the mark sometimes.

Unfortunately, a lot of the complexity is somewhat irreducible and due to to the actual complexity of the products. Scaling model size is different from scaling reasoning effort which is different from scaling multi-agent count. So if we just swept all of those under a single name of "more IQ", it would make the naming simpler but the experience much worse, as important details would be obfuscated from end users.

I can't edit it now, so let me disclose here: I work at OpenAI.

(1) OpenAI has never said they should be the only ones allowed to sell AI.

(2) Because gasoline is dangerous, refineries and fuel stations require government licenses and regular inspections in order to operate legally.

So I’m not sure how your comment contributes to our understanding of the situation.

I work at OpenAI. The obvious answer is that the models are not presently good enough to start companies.

You could ask the same question about any supplier. If your CPUs are so good, why not assemble PCs? If your lumber is so good, why not build houses? If your vaccine is so good, why not start a chain of pharmacies? Empirically there seems to be benefits of specialization and focus that outweigh the benefits of conglomeration.

Knowing what customers want is crucial, and people who buy our tokens often understand their customers better than we could hope to. The are also opportunity costs. For every bit of effort we expend chasing a new vertical, that’s a bit of effort and focus subtracted from our core competency.

Your theory is that this is self-interested PR. How does your theory explain why Anthropic is doing this, but so many other companies aren’t? If all company actions were self-interested, then wouldn’t you expect this to be a common move? Explaining why this is so uncommon requires either uncommon motive or uncommon circumstance. The theory that I find more explanatory is that the founders feel some degree of altruism and wish to exert their control over their company to effect their desired change in the world, especially before they go public and face greater scrutiny.

Sure, there will always be some trust and interpretation involved. I'm sure whatever they spend it on will never satisfy everyone. But I'd rather criticize actual dishonesty and theft, not the specter of its potential. If they spend the money on cars for execs, I assume they'd open themselves up to lawsuits, hurt their image, and damage employee morale. Voting for an exec comp increase would be far less risky than fraud. I feel confident this announcement is genuine, but of course none of us can read minds.

Super cool! 5% of $1.5T is $75B (and 7% of $2.25T is $156B). When added to the ~15% already pledged by founders and employees, that's roughly $300B going to charity/public benefit, which roughly matches the OpenAI Foundation. Of course you can ask questions like why this amount or why this way, but I for one think it's fantastic for rich developers of new technology to share their surplus with the world. I hope their stockholders approve the donation.

Some magnetic fun facts:

- Superconductors are perfect diamagnets and can levite on (or hang from) ferromagnets: https://www.youtube.com/watch?v=ZHT6NIebSfU)

- Superconductor quantum interference devices (SQUIDs) use the quantization of superconducting electron tunneling to measure tiny amounts of magnetic field, as little as a millionth of a flux quantum: https://en.wikipedia.org/wiki/SQUID

- Ferromagnetism is intrinsically a quantum phenomenon; spin is quantized and iron's magnetism is explained, in part, from electrons being identical particles that obey the exclusion principle: https://farside.ph.utexas.edu/teaching/sm1/Thermalhtml/node8...

- Despite the "super", superconductors are mostly not used in the world's strongest electromagnets, as they have limits on the current and magnetic fields they can take

- Magnetizing a magnet will actually cause it to spin a little, macroscopically: https://en.wikipedia.org/wiki/Einstein%E2%80%93de_Haas_effec...

- Charged particles are affected by magnetism even when traveling through space where electric and magnetic fields are zero: https://en.wikipedia.org/wiki/Aharonov%E2%80%93Bohm_effect

- Magnetic spin systems can technically have negative temperature: https://en.wikipedia.org/wiki/Negative_temperature

- Everything is magnetic, even frogs: https://www.youtube.com/watch?v=KlJsVqc0ywM

Yeah, and it's not even an accurate explanation either.

> The common one, the fridge magnet, is ferromagnetic — its atomic magnets all point the same way (up or down), adding their magnetic effects.

Ferromagnets typically have domains with magnetic moments that point in different directions. Ferromagnets rarely have every 'atomic magnet' pointing the same way.

https://en.wikipedia.org/wiki/Magnetic_domain

Refrigerator magnets in particular are usually magnetized as Halbach arrays, where the whole point is that the 'atomic magnets' are not pointing in the same direction. This is more energetically stable, which allows you to use cheaper materials.

https://en.wikipedia.org/wiki/Refrigerator_magnet

Lastly, I believe most refrigerator magnets are actually ferrimagnetic, not ferromagnetic. (The distinction doesn't matter much for users of magnets, but is important for the materials scientists studying and designing them.)

https://en.wikipedia.org/wiki/Ferrimagnetism

> We’re all used to two types of magnet. The common one, the fridge magnet, is ferromagnetic — its atomic magnets all point the same way (up or down), adding their magnetic effects. The less well known one, the antiferromagnet (AF), has neighbouring atomic magnets that point opposite ways and exactly cancel out magnetically.

This is a very bizarre introduction. People encounter diamagnets (e.g., copper) and paramagnets (e.g., aluminum) way more than they encounter antiferromagnets. I don't know why you'd ever cast magnetism as a false binary between ferromagnets and antiferromagnets, without acknowledging any other types of magnetic order.

(I did a PhD in magnetic materials)

Edit: I'll add that whether an antiferromagnet is useful, say, for exchange biasing a ferromagnetic thin film, depends on many factors. Just looking at antiferromagnetism alone you've got collinear vs non-collinear, G-type vs A-type vs C-type, commensurate vs incommensurate, and isotropic vs anisotropic; and all of that interacts with the interface structure, yada yada yada. It would be helpful if the authors elaborated on the expected properties of these materials. I personally don't know what people want room-temperature magnetic semiconductors for, but I'd be curious to learn what set of properties they think would be useful.

For me, 6.1 Sol nailed it immediately:

> A bare foot and ankle, pointing right, with three little toes.

I wonder how much of the wide variation in perceptions of LLM capabilities is driven by the gulf between free models and frontier models. Luna getting something wrong is not always great evidence for LLMs be unable to do that thing.

Edit: for the curious skeptics without access to 6.1 Sol, I tried 3 times and it got it right all 3 times. Convo share link: https://chatgpt.com/share/e/6abeb955-7614-832e-a5e1-b1bd134f...

GPT-6.1 Astra is what those headlines referred to. This is GPT-6.1 Sol.

Is it 90 independent tasks, or a single task with 90 pieces?

(it matters if they are independent or dependent)

API: https://developers.openai.com/api/docs/guides/your-data

ChatGPT: https://help.openai.com/en/articles/5722486-how-your-data-is...

ChatGPT data controls: https://help.openai.com/en/articles/7730893-data-controls-in...

If you have feedback on how to improve these, happy to consider it.

Looks like we phrase it as "your new conversations won’t be used to train OpenAI models" which is hopefully clearer than "OpenAI models will not be trained on your conversations", which could leave open the possibility of derived data or something.

Yeah, I'm not trying to trick anyone with wording that's technically true but actually misleading.

When I say no training, I mean no training. No gimmicks around data vs derived data, synthetic data, preference data, etc.

Places where I can imagine potential cracks in the literal interpretation of what I said are things like a financial analyst who does a statistical fit to predict revenue next quarter using a model based on last quarter's aggregate token consumption, which in some sense embodies your metadata (the length of your conversations) in a sea of other data. Or perhaps an infrastructure planner who makes a little model of internet bandwidth by time of day to help plan when we need a data center networking upgrade. Maybe things like these are technically training on your data in the most pedantic sense, but definitely not in the sense that most of us mean.

I promise you we're not doing any gimmicks where we transform your data and then pretend ah because it's transformed it's not your data.

My best guess is random variation and rising expectations. I once saw someone do in depth manual testing of 3 different models and write up a whole report on their perceived strengths and weaknesses… and then discover that all 3 models were identical. Very good reminder of why the scientific method is important.

Serial testing over time is much less reliable than side by side testing, and even when I do side by side testing, I try to look at multiple attempts per prompt. Seeing multiple per prompt helps me realize how much intrinsic variation there is. My brain always wants to see patterns even when there isn’t enough data to prove them.

API: By default, no training (opt in).

ChatGPT enterprise: By default, no training (opt in).

ChatGPT personal: By default, training (opt out).

By default, OpenAI does not train on API data. I promise you that Luna's low pricing is not a subsidy to get more training data. We've been lowering prices for years.

(I work at OpenAI.)

How many tasks were in this dataset?

If it's a single task where it dropped from 50 to 25, it could be random variation (not saying it is, but it could be). If it's the mean over hundreds of tasks, that suggests a problem with either the eval code/harness or our API.

Can you share more information on your methodology?

GPT-5.6 Sol's performance in the API should not change over time. If it has, that's a severe bug and we'll look into it.

We do sometimes tweak ChatGPT settings (e.g., tools, system prompts, efforts) over time, but we never play games to juice evals at launch times. You should always get what's advertised.

(I work at OpenAI.)