News for hackers. Stuff that matters.
Re:AI hype is built on high test scores. Those tests are flawed by jiggawatts on Wednesday October 11, 2023 @03:39AM (#37841921)
The equivalent of what current-gen LLMs do is an oral examination. Picture standing in the middle of a room surrounded by subject matter experts grilling you for your knowledge of various random topics. You have no tools, no calculator, no pencil and paper.

You’re asked a question and you just have to spit out the answer. No option to backtrack, experiment, or self correct.

“Translate this to Hebrew”.

“Is this a valid criticism of this passage from a Platonic perspective?”

“Explain counterfactual determinism in Quantum Mechanics.”

“What is the cube root of 74732?”

You would fail all of these. The AI gets 3 of 4 correct.

Tell me who’s smarter?

You because of your preconceptions, or because of real superiority?

Your model for human intelligence is probably more like this scene: https://youtu.be/KvMxLpce3Xw?si=Suy0Cj_pL0vru5Uj

The reality is the opposite. The AI could answer questions in this scenario but no nonfictional human could.

Re:AI hype is built on high test scores. Those tests are flawed by freejazz on Wednesday October 11, 2023 @09:55AM (#37844440)
It's just a completely baseless comparison the way you are going about it, and you are mistaking intelligence for the recitation of facts

>“Is this a valid criticism of this passage from a Platonic perspective?”

I haven't seen AI answering questions like this correctly at all