News for hackers. Stuff that matters.
Re:GPT‑6.2 Astra and Sol by underdeserver on Tuesday November 17, 2026 @12:55PM (#50535388)
I wonder why they didn't show Gemini 4 Carbon's scores on the science benchmarks, especially since Google seems to be pushing it for that -

https://x.com/pushmeet/status/2120814064863461440?s=20

(FD: Google employee, I don't work on Gemini)

Re:GPT‑6.2 Astra and Sol by TiredOfLife on Tuesday November 17, 2026 @12:58PM (#50535438)
Two posibilities. 1. They couldn't figure out how to actually use those Google models.

2. They launched the benchmark days ago and Gemini is still rereading the file with tasks.

Re:GPT‑6.2 Astra and Sol by amluto on Tuesday November 17, 2026 @02:02PM (#50536472)
It's been most of year since I tried to use Google models, but I spent a couple of hours, without success, trying to pay for Gemini in a manner that didn't result in it training on my data. I did try asking Gemini, and it had no idea. Of course, I did not try using Gemini in an agent harness to pay for itself, partially because there was a bit of a chicken and egg problem [0]. :)

Admittedly it appears that one can get it through OpenRouter these days. Trying to do anything through Google's absurd proliferation of billing systems is miserable.

Maybe someone should add an agent benchmark specifically testing the agent's ability to figure out how to use Google's products :)

[0] I'm not being serious here. I am not about to give an agent unfettered permission to spend money on Google Cloud. But I also couldn't figure out how to give myself permission, as the admin, to do this, so it was kind of moot.

Re:GPT‑6.2 Astra and Sol by xyzzy_plugh on Tuesday November 17, 2026 @02:56PM (#50537213)
I don't know how relevant this is but I just received an email stating they're splitting retention settings so that history and allowing training are separate knobs. Seems like they're listening, at least.
Re:GPT‑6.2 Astra and Sol by knuppar on Tuesday November 17, 2026 @01:00PM (#50535449)
Yeah it feels shady how Carbon was selectively plotted (or not), unlike Opus/Fable
Re:GPT‑6.2 Astra and Sol by eknkc on Tuesday November 17, 2026 @01:21PM (#50535831)
Probably still waiting for it to finish reasoning on the benchmark. Should be ready in a week or two.
Re:GPT‑6.2 Astra and Sol by tedsanders on Tuesday November 17, 2026 @01:24PM (#50535868)
I work at OpenAI and I made the plots. Gemini 4 Carbon was omitted when we couldn’t find public data, usually cost. No intent to hide anything.
Re:GPT‑6.2 Astra and Sol by underdeserver on Tuesday November 17, 2026 @01:36PM (#50536071)
So all the plots are based off public data? You don't run the models internally yourselves?

(Again speaking only for myself)

Re:GPT‑6.2 Astra and Sol by BoorishBears on Tuesday November 17, 2026 @02:33PM (#50536928)
Non deterministic models = "oh look they put up a lower score than Google did for that bench!"
Re:GPT‑6.2 Astra and Sol by declanjackson on Tuesday November 17, 2026 @02:32PM (#50536913)
All TB-Science scores and Cost per Task (across all reasoning levels for Carbon) are on AA: https://artificialanalysis.ai/evaluations/terminal-bench-sci...

Would be great to see this on the plot as well

Re:GPT‑6.2 Astra and Sol by holzhaker1 on Tuesday November 17, 2026 @01:28PM (#50535937)
they are probably trying to authenticate their phone number to access agy