The author mentions that Chollet's ARC tasks contain 200 unreleased "hidden" problems. This set off a light bulb for me: hidden problems are going to be increasingly important for benchmarking AI meaningfully. With RLHF, ChatGPT is memorizing the answer to every problem thrown at it. Eventually it may become impossible to test with any public question bank.

We are going to go through a period of fooling ourselves, as we always do, then we're going to hit a wall where models are preforming perfectly on benchmarks while still clearly lacking in genuine intelligence. And then we are going to need to test with a concern for security as serious as that used for the gaokao or SAT. Benchmarking will be a paid service performed under highly controlled conditions.