Re:AI knows things no one told it
by lucubratory on Thursday May 25, 2023 @11:42PM
(#36079558)
If you have put the answer into the program then by definition you had the test available to you when you finalised the program, which means it is definitionally not a novel test.
The test is novel to the program, just not its programmer. So are we testing the program or are we actually testing its programmer? If we're testing the program, then the programmer's foreknowledge is irrelevant.
Re:AI knows things no one told it
by lucubratory on Monday May 29, 2023 @03:37AM
(#36111654)
>The test is novel to the program
That's funny, I thought you said the test's answer was embedded into the program, making it definitionally not novel to the program.
Anyway, this is boring. You've had five or more opportunities to understand what the word "novel" means in an ML testing context and are choosing wilful obtuseness instead.
> The whole point of good tests is you invent or withhold new ones it has never seen before and test it on them
Then the Library of Babel must be really good at these tests, because it's never seen anything and yet has every answer.
> You said "Those tasks could be completed a [sic] traditional static program.", and no, they can't. You're incorrect.
They can, a static program as I described would indeed answer that one question correctly, resulting in a positive ToM score, without seeing any training data whatsoever. Did the programmer see it? Maybe, but it would pass the test regardless.