it's like hiring someone to come pick up your trash from your house and put it on the curb.

it's fine if you're disabled

Bazooka guarantees the hit

I like LLMs, but guarantees in LLMs are... you know... not guaranteed ;)

I think that was the point

You're missing the point, it's only a testing excersize for the new model.

No, the point is that you can set up the testing exercise without using an LLM to do a simple find and replace.

... I'm not sure if you're trolling or if you missed the point again. The point is to test the contextual ability and correctness of the LLMs ability's to perform actions that would be hopefully guaranteed to not be in the training data.

It has nothing to do about the performance of the string replacement.

Its a test. Like all tests, its more or less synthetic and focused on specific expected behavior. I am pretty far from llms now but this seems like a very good test to see how geniune this behavior actually is (or repeat it 10x with some scramble for going deeper).

This thread is about the find-and-replace, not the evaluation. Gambling on whether the first AI replaces the right spells just so the second one can try finding them is unnecessary when find-and-replace is faster, easier and works 100%.

do you know all the spells you're looking for from memory?

You could just, you know, Google the list.

and then the first thing you see will be at least one of ITS AI responses, whether you liked it or not

If all you have is a hammer.. ;)