Yes. Llama2 7B:

  >>> Which weighs more: a kilogram of gold or a kilogram of feathers?

  A kilogram of gold weighs more than a kilogram of feathers.

  One kilogram is equal to 1000 grams, so:

  * A kilogram of gold = 1000 grams x 1 kg = 1000 kg
  * A kilogram of feathers = 1000 grams x 1 kg = 1000 kg

  As you can see, a kilogram of gold weighs more than a kilogram of feathers.

Even the units on the math is wrong. But this is the 7B Llama 2 model. ChatGPT gets all of it correct.

Pretty much what you'd expect from glorified autocomplete.

> >>> Which weighs more: a kilogram of gold or a kilogram of feathers? > A kilogram of gold weighs more than a kilogram of feathers, which is 1 kg. Gold is denser and has a higher mass than feathers, so it would be heavier.

It is the right answer with the wrong explanation. Mass and weight are two different physical quantities. Weight is the resulting downforce acting on a body on Earth. It consists of downwards force of gravity and upwards force due to displacement of air. The gravity force at sea level acting on 1 kilogram of gold of gold and 1kg of feather is the same 9.8N but 1kg of feather displaces much more air than 1kg of gold because of lesser density and thus larger volume. At the end of the day, a precision scales will show that 1kg of feather is lighter than 1kg of gold if measured on Earth. Of course, LLM misses all of this phycics :-)

EDIT: On the Moon with no atmosphere 1kg of gold and feather will weigh the same.

As long as we're being pedantic it's worth noting that there's essentially nowhere without ambient gas or plasma -- that is to say, with no atmosphere.

It just varies in density over more orders of magnitude than we're casually comfortable with.

The moon has an atmosphere which is locally denser than the surrounding interplanetary medium.

So the displacement argument should remain technically true (the best kind) even on the moon.