Re:All Souls exam questions and the limits of machine reasoning
by tough on Thursday August 14, 2025 @10:24PM
(#44907956)
if anything i would do differently, i'd try things only machines can reliably do.
unless the llm and the design for it is necessarily adversarial, not even going into red teaming or jailbreaks.
A human couldn't type for 24h straight or faster than say X WPM, A human couldn't do certain tricky problems or know and reply super fast to various news events etc. Search/training date seems important factor too to tie in.
but yeah overall if the time is infinite you can come up with some new way to find out, kinda becomes a cat and mouse games then like software security nowadays
unless the llm and the design for it is necessarily adversarial, not even going into red teaming or jailbreaks.
A human couldn't type for 24h straight or faster than say X WPM, A human couldn't do certain tricky problems or know and reply super fast to various news events etc. Search/training date seems important factor too to tie in.
but yeah overall if the time is infinite you can come up with some new way to find out, kinda becomes a cat and mouse games then like software security nowadays