How do you know those large comprehensive test and validation suites you've never looked at actually reflect real user experiences at all?
Re:I will not be using LLMs in my coding
by Someone1234 on Sunday November 08, 2026 @09:37AM
(#50413736)
You must have replied to the wrong comment, since the one you did replied to never claimed that the test and validation suites were "never looked at." They are reviewed, and the test/data results are reviewed.
It said verbatim: "Don't trust. Verify."
Re:I will not be using LLMs in my coding
by verdverm on Sunday November 08, 2026 @08:37AM
(#50413243)
> Because we have test result data that PROVES it to be the case.
pgrust thought this too, but then another human discovered claude had hardcoded benchmark query plans.
Now pgrust has a "test" to make sure the Ai doesn't do this again (unless it decides this test is bad for the current goal)
https://github.com/malisper/pgrust/blob/38634b408dea373f3944...
Real verification seems beyond transformer capabilities.
Re:I will not be using LLMs in my coding
by mzhaase on Sunday November 08, 2026 @08:45AM
(#50413326)
Yeah but you don't test for stuff you don't know you have to test for. All tests can pass and internally its still done in a braindead way that will explode at some point.
Re:I will not be using LLMs in my coding
by Someone1234 on Sunday November 08, 2026 @09:35AM
(#50413712)
Absolutely, and this is exactly why humans are weaker than LLMs. We consider what is likely, they consider what the data/logic shows is possible. So you get edge cases tests/flagged, that wouldn't even be something a real person would know to look for.
Re:I will not be using LLMs in my coding
by springtimesun on Sunday November 08, 2026 @09:24AM
(#50413630)
This is my observation as well. I thought I was good at testing (I wanted to be good at testing), but the models are just better at it. Testing in particular rewards a tireless cataloging of the tiny and the tedious and in this respect LLMs are unmatched.
Single data point, my Sentry notifications across more than a half dozen client projects when from 2-4 a month to basically 0. That's while shipping 20x _more_ code.
In essence, it is writing the code faster (great), but then also writing the code that was often previously never written due to budget/time limitations (comprehensive testing/validation).
So, how do you know the code is the highest quality ever? Because we have test result data that PROVES it to be the case. It isn't "trust but verify" it is "Don't trust. Verify."