News for hackers. Stuff that matters.
Re:Qwen3: Think deeper, act faster by vunderba on Tuesday April 29, 2025 @12:02PM (#43834464)
The only thing I don't like about this test is that I prefer test questions that don't have binary responses (e.g. heads or tails) - you can see from the responses that you got from the thread that the LLMs success rates are all over the map.
Re:Qwen3: Think deeper, act faster by furyofantares on Tuesday April 29, 2025 @12:07PM (#43834540)
Yeah, same.

I had a more complicated prompt that failed much more reliably - instead of a mirror I had another person looking from below. But it had some issues where Claude would often want to refuse on ethical grounds, like I'm working out how to scam people or something, and many reasoning models would yammer on about whether or not the other person was lying to me. So I simplified to this.

I'd love another simple spatial reasoning problem that's very easy for humans but LLMs struggle with, which does NOT have a binary output.