Re:LLMs will fail this simple eval every time
by nabla9 on Saturday June 15, 2024 @06:42PM
(#40693238)
For example:
>The exact number of grains of sand on a specific, untouched beach.
Just saying "specific" is not being specific.
"The exact number of grains of sand on a Source d’Argent beach on Seychelles." Would be specific and exact thing of knowledge it does not have. It fails to name exact, specific thing.
It's like it is compelled to be unspecific and not exact and be generic.
"any given town.", "undiscovered ancient manuscript", "hypothetical alien civilization's technology", "of a specific person","random individual", "unidentified and uncatalogued species"
>The exact number of grains of sand on a specific, untouched beach.
Just saying "specific" is not being specific.
"The exact number of grains of sand on a Source d’Argent beach on Seychelles." Would be specific and exact thing of knowledge it does not have. It fails to name exact, specific thing.
It's like it is compelled to be unspecific and not exact and be generic.
"any given town.", "undiscovered ancient manuscript", "hypothetical alien civilization's technology", "of a specific person","random individual", "unidentified and uncatalogued species"