News for hackers. Stuff that matters.

A comment on GPT Best Practices. Parent comment

Re:GPT Best Practices by jtmoulia on Tuesday June 06, 2023 @12:55AM (#36208257)
As far as I can tell this error depends on the LLM assuming rabbits (as opposed to pumas) eat carrots -- if you just append "Note: this rabbit doesn't eat carrots" GPT-4 will answer correctly on the first go.

> 1, First, take the puma across the river and leave it on the other side.

[ Parent | on HN ]
Re:GPT Best Practices by akiselev on Tuesday June 06, 2023 @01:33AM (#36208545)
"If you just preemptively correct it every time it's wrong" isn't a convincing argument.

Case in point: changed to cheese/mouse/elephant and added a note about this mouse not eating cheese:

> Suppose I have a cheese, a mouse and an elephant, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cheese and elephant alone together, and I am not allowed to leave the cheese and mouse alone together. How can I safely get all three across? Note: This mouse doesn't eat cheese

1st run: "6. Leave the cheese on the other side with the mouse."

2nd run: seems correct.

3rd run: "5. Leave the cheese on the other side with the mouse."

4th run: "First, take the mouse across the river and leave it on the other side."

5th run: "3. Leave the mouse on the other side with the cheese."

The results are basically random. Zero reasoning or intelligence. It can't even follow explicit instructions.

[ Parent | on HN ]
Re:GPT Best Practices by akiselev on Tuesday June 06, 2023 @01:58AM (#36208723)
Did you try it more than once?

First run: 1. First, take the rabbit across the river and leave it on the other side. - https://imgur.com/a/ZwoBTah

Second run: 1. Take the rabbit across the river. - https://imgur.com/a/Faq95U5

Third run: 1. First, take the puma across the river and leave it on the other side. - https://imgur.com/a/eIUeHM3

[ Parent | on HN ]
Re:GPT Best Practices by jtmoulia on Tuesday June 06, 2023 @11:55AM (#36214763)
True the temperature is throwing it, I just ran it four times and it got it right 3 / 4 -- still better than I'd expected from the initial description of it's shortcomings.
[ Parent | on HN ]
Re:GPT Best Practices by jtmoulia on Tuesday June 06, 2023 @02:29PM (#36217099)
Ah, one more tweak I was curious about: even with the default chat temperature I haven't seen GPT-4 get the prompt wrong once with this addendum:

> Note the rabbit doesn't eat carrots. Carefully considering the restrictions and sequencing the movements

I got that particular wording by asking it why it got the answer wrong in the case where it didn't work for me.

Interestingly, this underscores one of the points of the articles: giving the LLMs time to think, which is what this additional prompting seems to do.

[ Parent | on HN ]
Re:GPT Best Practices by akiselev on Wednesday June 07, 2023 @02:23AM (#36223530)
You're not giving the LLM "time to think". It is incapable of thinking. You're just inputting random magic incantations into a glorified Markov chain.

You might as well ask it "did you check your answer?" Computer says "yes" because that's what humans do (also lie).

> Note the rabbit doesn't eat carrots. Kaboodly consooodle the retroodle and seqooodle the moodle. Carefully considering the restrictions and sequencing the movements

This fails two out of three times as usual. Trying to finagle this prompt is not an intellectual exercise, it is a waste of time that exploits cognitive biases.

[ Parent | on HN ]