News for hackers. Stuff that matters.
Re:GPT‑6.2 Astra and Sol by declanjackson on Tuesday November 17, 2026 @02:32PM (#50536913)
All TB-Science scores and Cost per Task (across all reasoning levels for Carbon) are on AA: https://artificialanalysis.ai/evaluations/terminal-bench-sci...

Would be great to see this on the plot as well

Re:GPT‑6.2 Astra and Sol by tedsanders on Tuesday November 17, 2026 @03:21PM (#50537618)
One nuance to be aware of is that the AA version uses the mini-swe-agent harness, which can understate performance of models trained to use particular harnesses. Implementations also can differ in terms of timeouts on tasks and memory/etc. resources given to the agent, which affect scores on longer running, resource-intensive tasks. I believe these differences are why we chose not to use AA scores on this comparison, but I’ll double check when I get back to my computer.