Re:Google, Nvidia, and OpenAI – Stratechery by Ben Thompson
by raw_anon_1111 on Monday December 01, 2025 @02:40PM
(#46112079)
You need a way to test model changes regardless as models in the same family change. Is it really a heavier lift to test different model families than it is to test going from GPT 3.5 to GPT 5 or even as you modify your prompts?
[ Parent
| on HN ]
Re:Google, Nvidia, and OpenAI – Stratechery by Ben Thompson
by spruce_tips on Monday December 01, 2025 @04:00PM
(#46113153)
no, i dont think it's a heavier lift to test different model families. my point was that swapping models, whether that's to different model families or to new versions in the same model family, isn't straightforward. i'm reluctant to both upgrade model versions AND to swap model families, and that in itself is a type of stickiness that multiple model providers have.
maybe another way of saying the same thing is that there is still a lot of work to make eval tooling a lot better!
Re:Google, Nvidia, and OpenAI – Stratechery by Ben Thompson
by DenisM on Monday December 01, 2025 @09:58PM
(#46116886)
Continuous eval is unavoidable even absent model changes. Agents are keeping memories, tools evolve over time, external data changes, new exploits are being deployed, partner agents do get upgraded.
Theres too much entropy in the system. Context babysitting is our future.