Re:Gemini 2.5 Pro Preview: even better coding performance
by jorvi on Wednesday May 07, 2025 @12:26PM
(#43917683)
I asked ChatGPT, Claude, Gemini and DeepSeek what the AE and OE mean in "Harman AE OE 2018 curve". All of them made up complete bullshit, even for the OE (Over Ear) term. AE is Around Ear. The OE term is absurdly easy to find even with the most basic of search skills, and is in fact the fourth result on Google.
The problem with LLMs isn't that they can't do great stuff: it's that you can't trust them to do it consistently. Which means you have to verify what they do, which means you need domain knowledge.
Until the next big evolution in LLMs or a revolution from something else, we'll be alright.
Re:Gemini 2.5 Pro Preview: even better coding performance
by KoolKat23 on Wednesday May 07, 2025 @12:48PM
(#43917950)
Both Gemini 2.5 Flash and Kagi's small built in model in their search got this right first try.
Re:Gemini 2.5 Pro Preview: even better coding performance
by jorvi on Wednesday May 07, 2025 @01:58PM
(#43918766)
That is my point though. Gemini got it wrong for me. Which means it is inconsistent.
Say you and I ask Gemini what the perfect internal temperature for a medium-rare steak is. It tells me 72c, and it tells you 55c.
Even if it tells 990 people 55c and 10 people 55c, with a tens to hundreds of million users that is still a gargantuan amount of ruined steaks.
Re:Gemini 2.5 Pro Preview: even better coding performance
by KoolKat23 on Wednesday May 07, 2025 @03:14PM
(#43919566)
I know what you're saying, I guess it depends on the use case and it depends on the context. Pretty much like asking someone off the street something random. Ask someone about an apple some may say a computer and others a fruit.
But you're right though.
Also, the reward functions that you mention don't necessarily lead to great code, only running code. The should be possible in the third bullet point does very heavy lifting.
At any rate, I can be convinced that LLMs will lead to substantially-reduced teams. There is a lot of junior-level code that I can let an LLM write and for non-junior level code, you can write/refactor things much faster than by hand, but you need a domain/API/design expert to supervise the LLM. I think in the end it makes programming much more interesting, because you can focus on the interesting problems, and less on the boilerplate, searching API docs, etc.
[1] https://ibb.co/pvm5DqPh