chriddyp

In my tests [1], GPT-6.1 Terra joins the Pareto Frontier in between GPT-6 Luna and GPT-6.1 Sol. So, exactly as advertised!

It's also a very fast model, of all 33 models we've tested so far, only Haiku 5.5 and Nemotron 3.5 Super VL have passed the exam faster but both scored lower. It also wrote the _least_ output in its responses.

[1] Benchmarks for data analytics tasks: https://plotly.com/blog/gpt-6-1-terra-plotly-data-analytics-...

Big improvement over Kimi K3 in our data analytics benchmark: it scored a 98% vs a 81% in July.

Quite slow in our exam. A lot of that was due to the number of tool calls that it's making. In one of our questions about finding data issues in a dataset, it made 73 queries. Other models like 6.1 Sol only made 18 for that question.

Full deep dive here: https://plotly.com/blog/kimi-k3-1-pro-plotly-data-analytics-...

In our data analytics benchmark, it got the best score of all of the Chinese models and tied Opus 5.5 & Argon (41/43 questions correct). It's also 1/3rd the price of both of those models. However, it seems to use A LOT of tokens and it's the slowest model that we've tested for the exam; it's slow because of the number of tokens it uses, not the inference speed.

We also tested Space Bunny (its stealth version) a few weeks ago and the behavior changed quite a bit. Went from 30s / question to 3 minutes / question and the accuracy improved quite a bit.[1]

This benchmark is short-horizon data analytics questions, so not fully representative of software engineering tasks. YMMV.

[1] https://plotly.com/blog/minimax-m3-1-flash-plotly-data-analy...

Awesome model. Tested it for data analytics and found it's in the class of Opus 5.5 in terms of performance and cost.

Compared to Muse Spark 1.3, found that it's more than 2x faster about 2x more expensive. I normally equate accuracy to size, and size to speed, so nice surprise to see a model get better _and_ faster.

Like Argon 4, models in this class are a bit over powered for short-horizon data analytics tasks. GPT-6.1 Sol, Haiku 5.5, GLM all perform a similar grade at 1/10th the cost.

Full results here: https://plotly.com/blog/muse-wonder-plotly-data-analytics-be...

Ran on our Data Analytics benchmark: https://plotly.com/blog/gemini-4-argon-plotly-data-analytics...

Excellent model. Same price and score as Opus 5.5 in our 23 question exam.

This size / class of model is now a bit overpowered for these short-horizon data analytics tasks. OpenAI 6.1-Sol, Haiku 5.5, GLM 5.3 Flash all achieve about the same results but for ~1/10th the cost.

In other words, we're getting to the stage where models are "good enough" for certain tasks like data analytics and competitiveness is going towards size & speed.

I've been testing the models against data analysis tasks, and found that the recent Luna & Haiku releases from the last month have really reset the narrative around price x quality.

DeepSeek 4.1 Flash is really is 10x cheaper than Opus, but it's in the same ballpark as Luna & Haiku in terms of quality, cost, and speed for data analysis. (Luna & Haiku actually perform marginally better, cheaper, and faster).

Here's a breakdown: https://plotly.com/blog/deepseek-v4-1-flash-data-analysis-be...

Note that this benchmark is for data analysis, not general software development. So the tasks are shorter and more constrained.

It's a valid concern. We're not releasing the dataset, the answers, or the full set of questions to help prevent this. At the same time, I like to share where it gets things wrong in a bit more detail, which involves sharing a bit of the exam. I expect that we'll create a new benchmark with a new dataset in 6 months.

Ran our DataAnalyticsBench benchmark on it: https://plotly.com/blog/claude-haiku-5-5-plotly-data-analyti...

9x cheaper than Haiku 4.5 and 2 letter grades better. It's also now the fastest model (using the default speeds, not trying any of the other models "Fast" mode) to complete the exam.

Similar ballpark to Luna in price, cost, and accuracy. These are very cheap models: $0.38 to answer 40 in-depth data analytics questions (compared to $15 for Opus 5.5 or $20 for Astra).

Overall very good at data analysis - handling all of the straightforward data analytics questions correctly. It fell short answering some of the questions that required some deeper statistical analysis like looking into other variables. In other words, it's not as persistent as other models in its analysis, which I think we'd expect from how they're positioning the model.

Compared to OpenAI: GPT-6 Luna did a bit better and was about 30% the cost of Haiku 5.5. GPT-6.1 Sol got all answers correct, but was 10x more expensive.

That’s correct. About the same in price, but Qwen does get more answers correct in this particular benchmark.

Just ran this through our data analytics benchmark (I work at Plotly).

It's 10x cheaper than Mistral Medium 3.5 from April and goes from 58% to 74% correct. Definitely a generational shift.

It's not on the Pareto curve yet, but it's good enough for data analytics, and at this rate I suspect it'll be excellent in another few months.

Full write up: https://plotly.com/blog/mistral-large-4-plotly-data-analytic...

This is a really cool idea. So much context is lost within agent conversations these days, and it requires far too much discipline to have the conversation between you and your agent in a shared place like GitHub (plus, thee conversation isn't always just a comment and response - it's... an actual conversation).

I'm very glad to see folks innovating in this space.

congrats! the paper that is published alongside this (Cordis) is super interesting. has anyone on the team given a talk or published a talk about this? would love to hear the authors break this down

While this is true, there is also a layer in the harness between the output of _any_ tool output (eg stdout or hand-rolled tools) and the LLM. A tool could read the file but then the agentic harness could redact the output before returning it back to the llm if any of the contents matched the file contents. We do something similar in Plotly Studio where we check the entropy of strings in the user input and flag & redact any high entropy strings to the user as “potential credentials” thay the user might have inadvertently copied and pasted into the prompt before sending to the llm.

There are ways around this - the llm can always be clever by invoking tools to read the file contents in a different way than the direct file contents - but this is all to say that the agentic harness layer _does_ allow for deterministic logic in between tool output and the LLM requests.

This is really cool. And timely! Check out the recent paper by Google et al re "Societies of Thought": https://arxiv.org/html/2601.10825v1. It goes into how different conversational behaviors (raising questions or just say "but wait..."), perspective shifts, conflict of perspectives, tension, tension release (jokes!), asking for opinions) and different personalities (planner, expert, verifier, pragmatist) is both a sign of and can result in much higher performance reasoning.

So I'd be curious to see if encouraging certain conversational behaviors might actually improve the reasoning and maybe even drive towards consensus.

The issue goes beyond access and into whether or not the data is "trusted" as the malicious prompts are embedded within the data. And for many situations its hard to completely trust or verify the input data. Think [Little Bobby Tables](https://xkcd.com/327/)

Curious to see Apple using real black as background on their home page but rgb(29, 29, 31) for their black text.