I bet when you said a 12 litre jug and a 6 litre jug it wrongly assumed that you required it to actually make use of both the jugs in some way (not merely that they were available for possible use), leading to the pointless step.
Seems right! If you make it more of an inventory list of tools, it answers correctly.
> I have two jugs: a 12 liter jug and a 6 liter jug. I need 6 liters of water. How do I do it?
> GPT-4: If you just need 6 liters of water and you have a 6-liter jug, you simply fill the 6-liter jug to the top with water. You'll have exactly 6 liters! No need to use the 12-liter jug in this case.
I would bet a high percentage of humans would do the same thing if prompted as such.
This video covers the concept pretty well: https://www.youtube.com/watch?v=IJEaMtNN_dM
It is pretty normal to try to incorporate the extranneous details into the reply.
I've noticed that the LLMs are all tuned to emit corporate speak.
Everyone I've encountered that adds lots of obfuscating and tangential details to their day-to-day speech (and tries to establish that particular tone of faux-inclusivity and faux-authority) has turned out to be a sociopath and/or compulsive liar. I find it interesting that LLMs have the same symptom and underlying problem.
Me:
ChatGPT: Me: ChatGPT: I wish I understood why prompting it a second time gets a better answer.