News for hackers. Stuff that matters.
Re:GLM 5.2 vs. Opus by post-it on Monday June 22, 2026 @09:04AM (#48629625)
The streetlight effect:

> A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is"

All of your suggestions are better but they're hard, so someone casually evaluating an AI isn't going to do them.

Re:GLM 5.2 vs. Opus by sanderjd on Monday June 22, 2026 @09:35AM (#48629927)
Sure, for casual evaluation, I agree. But are there serious analyses that are evaluating this kind of thing? I mean, these are the kinds of things I evaluate in my own work when a new model comes out, or when I'm evaluating a harness. But this is all very ad hoc and intuitional. I'd love to start bringing rigor to it, but I haven't found much prior art on this. In another thread someone said that's because it's probably impossible to do this rigorously because too much of it is subjective. And that does match my intuition. But I continue to suspect that intuition is wrong.
Re:GLM 5.2 vs. Opus by jerf on Monday June 22, 2026 @09:59AM (#48630240)
It's hard to bring much rigor to it. I'm not saying impossible, but it's not like it's completely obvious how to do it and people are just too lazy. Intrinsically, if I'm going to test a back-and-forth with a model I have a human in the loop making frequent decisions. Did the model fail or succeed at whatever rate it did that because of the model or the human? Did the testing protocols capture the actual problem, e.g., maybe if the model was given some particular bit of information that a normal human would have given it it would have done much better or worse, but the testing protocol in the interests of "rigor" excluded the human in the loop from doing it. Is the human going to be willing to sit down and do the same task 25 times, refreshing the model from scratch each time for a "valid" test? Can you get the same human to analyze every model in the test? Is their 10th pass of the problem an invalid test because you can't as easily erase the human's knowledge of the previous 9 tests? What do you do with a model that succeeds wildly 75% of the time and spins off into a loop the other 25%? Is that loop real or, again, did your "rigorous" testing protocol prevent the human from saving the model from the loop like any developer would?

And so on and so forth. Again, I'm not saying this is impossible but I am saying that if you tried to do it, and you got the money, and you built the test, and got the human subjects clearance, and you ignored that during the process of all that at least one more frontier model would come out, you can count on HN anklebiting your "rigorous" study even so, and probably being correct about a lot of the issues it could have because it would take several iterations of this to build a reasonable protocol... at which point it would quite possibly also be obsoleted by progress again.

Re:GLM 5.2 vs. Opus by sanderjd on Monday June 22, 2026 @08:43PM (#48638627)
Yes, well put.
Re:GLM 5.2 vs. Opus by abhgh on Monday June 22, 2026 @10:15AM (#48630444)
You usually see this kind of analyses in conference papers, esp. if they have a datasets track. The NeurIPS Datasets & Benchmarks (D&B) track is a good example. But you will have to monitor the proceedings yourself closely - there is little chance of being accidentally exposed to them, because most blogs, announcements and popular media only mention a handful of the popular ones, e.g., Tau^2. For ex., across the years 2022, 2023 and 2024, 900+ papers were accepted in the D&B track [1] - of course, not all of them are LLM-related. I find them interesting because they often focus on specific system behaviors, and like you said, study them scientifically, so you can draw authoritative conclusions (or at least know specifically what part of a model's behavior you now know about, and what parts you don't).

[1] https://blog.neurips.cc/2025/09/30/reflecting-on-the-2025-re...

Re:GLM 5.2 vs. Opus by rileyphone on Monday June 22, 2026 @12:22PM (#48632290)
DeepSWE is closer to that

https://deepswe.datacurve.ai/

Re:GLM 5.2 vs. Opus by nileshtrivedi on Monday June 22, 2026 @04:38PM (#48635825)
You might like SWE-WebDevBench which tries to do this comprehensive evals for webapp development. https://webdevbench.com/
Re:GLM 5.2 vs. Opus by echelon on Monday June 22, 2026 @09:49AM (#48630123)
The minute an open model breaks through and beats Claude Opus/Fable, it's over.

There are far more opportunities that can be served when the world's intellectuals have the raw weights and can fine tune, splice, distill, and reapply.

Imagine having raw unfettered access to Fable. It can be refit to structural biology. It can be fine tuned on the repo for smaller context requirements. It can be run cheaper and air gapped.

The world wants this.

Re:GLM 5.2 vs. Opus by barrenko on Monday June 22, 2026 @10:02AM (#48630278)
As crazy as this sounds, and as much I don't want to believe it myself, I think we're still underestimating LLMs, and we're gonna get to that point pretty soon.
Re:GLM 5.2 vs. Opus by jupr on Monday June 22, 2026 @10:14AM (#48630428)
The world does want this. Opus capabilities, in a box, securely tunneled to my family and I utilizing the resources I already have available to me which is, energy + network.
Re:GLM 5.2 vs. Opus by digitaltrees on Monday June 22, 2026 @12:55PM (#48632749)
I don’t think we need them. I think the models we have are good enough. It’s the orchestration layer that makes the biggest difference at this point. The open source models we have are capable of calling tools and the work is getting them to be capable enough to know which tools to call and what to do in response.

I think we are leaving the main frame era of AI and entering the PC era already. If there wasn’t a RAM shortage and we all had 2TB of ram and GPUs we would all have large local models or personal APIs serving our teams.

That’s why all the labs are moving to the App layer and moving away from being the API for intelligence like they were originally.

Re:GLM 5.2 vs. Opus by wahnfrieden on Monday June 22, 2026 @01:21PM (#48633053)
They are absolutely not good enough
Re:GLM 5.2 vs. Opus by digitaltrees on Monday June 22, 2026 @03:06PM (#48634584)
I disagree because I am getting real work done. But I have three Mac studios with 1.5 tb of ram and built my own harness around large models in my own ide (propelcode.app). It’s not perfect. And opus ChatGPT 5.5 are better but they’re good enough
Re:GLM 5.2 vs. Opus by noonething on Monday June 22, 2026 @05:18PM (#48636346)
if you had the information to do the task perfectly then those open models should be fine no? if so, your just paying for knowledge at that point.
Re:GLM 5.2 vs. Opus by digitaltrees on Monday June 22, 2026 @07:33PM (#48637983)
Yah that's my point.
Re:GLM 5.2 vs. Opus by wahnfrieden on Monday June 22, 2026 @07:39PM (#48638057)
Maybe write “I don’t think I need them. The models we have are good enough for me.” instead then
Re:GLM 5.2 vs. Opus by digitaltrees on Tuesday June 23, 2026 @03:14PM (#48649906)
What do you think I am arguing?
Re:GLM 5.2 vs. Opus by wahnfrieden on Tuesday June 23, 2026 @03:49PM (#48650367)
That we don't need better models because they're good enough for our work
Re:GLM 5.2 vs. Opus by digitaltrees on Tuesday June 23, 2026 @10:58PM (#48654551)
I understand why you drew that conclusion from my post, but that's not what I am arguing in the context of this thread.

The OP said "as soon as an open source model of the capability of fable or gpt5.5 are released it's over because the world will build on top of them" which I took to mean some amazing new level of performance is only able to be unlocked with those models.

I am arguing that the level of performance the OP seems to imply doesn't require fablw level models for the work the OP envisions because the harnesses can actually drive the same performance increase.

There are a few papers that have demonstrated it. And my own use of fable made me feel it was not that big of a jump from opus 4.8 compared to better systems around it.

Its a subtle but important distinction.

Re:GLM 5.2 vs. Opus by newaccountman2 on Monday June 22, 2026 @09:21AM (#48629775)
Feels a rather outdated little parable, since nowadays one would expect the police officer to either arrest or shoot the person.
Re:GLM 5.2 vs. Opus by post-it on Monday June 22, 2026 @10:42AM (#48630862)
It could be a taxi driver if you like. Or an anarchist passing by on xir way to a protest.
Re:GLM 5.2 vs. Opus by layer8 on Monday June 22, 2026 @11:32AM (#48631645)
…in the US.
Re:GLM 5.2 vs. Opus by blanched on Monday June 22, 2026 @10:18AM (#48630484)
This kind of hamfisted snark tends to make people take the actual and justified criticism of police less seriously.
Re:GLM 5.2 vs. Opus by newaccountman2 on Monday June 22, 2026 @10:27AM (#48630607)
If people were willing to take it seriously in the first place, then they wouldn't view it as "hamfisted snark"
Re:GLM 5.2 vs. Opus by blanched on Monday June 22, 2026 @11:21AM (#48631452)
I consider myself someone who takes it seriously, and have spent time and resources fighting for change. But it’s wholly unrelated to this particular thread, phenomenon, and story. So having a little “ha ha” moment accomplishes nothing towards the actual cause. It makes people uncomfortable, but not the useful kind of uncomfortable.

That said, maybe we just disagree on how to drive change, and that’s fine. I’ll leave it.