This is a fun one because it illustrates that you failed the test just like I did.

Reread carefully: the question compares "a kilogram of feathers or a kilogram of feathers".

I've never heard that one before but my brain pattern matched right over to "a kilogram of feathers or a kilogram of steel" and moved on. I had to read it 3 times before I got the subtlety.

I wouldn’t even say it’s failing. Recognizing and correcting obvious errors in speech is a feature not a bug, both in humans and LLMs. Is it more likely that they meant to say feather twice or they mistyped?

True, but would it not make more sense to ask rather than assume and provide a long winded explanation?

And even if it was a typo and I meant to say rocks, the model will often assume a variety of metals and answer with those.

How it proceeds in the case of human error is going to be almost totally removed from its "intelligence," and almost totally related to its training.

I’m not claiming otherwise here. I’m claiming that it’s not a good test because a human would (almost certainly correctly) also skip past the double feather and answer the likely spirit of the question.

A human would probably say “did you mean a kg of feathers and a kg of lead?”

It's a trick trick question.

It's meant to appear as the often used trick question comparing different densities of materials but OP says feathers twice.

The trick is that an LLM will add in a dense material to the answer because it's only ever seen the question asked that way.

not 'super intelligent' humans who are familiar with this old dad joke :)