News for hackers. Stuff that matters.
Re:All Souls exam questions and the limits of machine reasoning by lumost on Thursday August 14, 2025 @08:30PM (#44907346)
Human's are able to quickly converge on a pattern. While I doubt that I could immediately catch all LLMs, I can certainly catch a good portion by having simply worked with them for a time. On an infinite horizon Turing test, where I have the option to state that Chair A is a machine at any time - I would certainly expect to detect LLMs simply by virtue of their limited conversational range.
Re:All Souls exam questions and the limits of machine reasoning by tough on Thursday August 14, 2025 @10:24PM (#44907956)
if anything i would do differently, i'd try things only machines can reliably do.

unless the llm and the design for it is necessarily adversarial, not even going into red teaming or jailbreaks.

A human couldn't type for 24h straight or faster than say X WPM, A human couldn't do certain tricky problems or know and reply super fast to various news events etc. Search/training date seems important factor too to tie in.

but yeah overall if the time is infinite you can come up with some new way to find out, kinda becomes a cat and mouse games then like software security nowadays