From my experiments so far, I am unsure if chatgpt is effective at optimizing prompts, even though it thinks it is.

The workflow you get with this is that it helps you to think about what you really want answered with the prompt.

For example, latest try had this initial prompt:

> How can I create collisions in Bevy and Rust?

After 4-5 messages back and forth, I ended up with:

> What is the most efficient method to implement 2D collision detection for a large number of constant-radius circles randomly distributed in Bevy and Rust without using third-party tools, focusing on detection only, without any unique properties or attributes affecting the detection process?

Which is much more explicit and targeted to what I initially wanted to do, but didn't write. It ended up helping me implement a Quadtree solution which I don't have any experience with before, but overall it went smoothly.

How do you know that the prompt you got to is at least a local maximum? Would chancing "method" to "function" provide better results? Did you do any benchmarking?

The aim is not any local maximum, is improving the prompt by getting help providing more context.

That's a pretty bad optimizer then.

I've seen it help me once so far in a way I hadn't thought of: basically, the chat completion engine was reading a bug bounty report on my behalf, and believing generic, vague claims. ChatGPT-3.5 didn't "fall" for this, but 4 did consistently.

After failing to improve performance manually, I eventually asked 4 to improve my prompt after explaining the problem and it basically added "Don't assume the existence of any examples".

This makes perfect sense in hindsight, but I had been approaching things from the other direction - directing it to only consider explicit examples in the report, and for some reason that didn't work.

Thank you, I am collecting promtps creators.

Or just provide it with sample contexts and expected responses, and ask for a prompt to produce those responses from the given context.

If you prompt it with "sample contexts and expected responses," that is, in fact, a prompt that would produce those results. It's not a bad technique for crafting prompts, either.

That’s best when possible, but if each context and response very large, a good prompt is the best way to distill the information without fine tuning.

I suppose you could try asking for shorter examples to provide for a few-shot prompt, could work well in some cases

But ChatGPT has no special introspective powers. It's guessing based on the data it was trained on, which was primarily written before GPT by people who didn't know how GPT would work.

So it will give you suggestions that sound plausibility like how we tend to think an AI might work, but there's no reason to expect it to be correct.

The prompt it generates looks good and makes sense, but I doubt it's always correct or optimal. But then I doubt a human assistant could give a perfect prompt either (enemy of the good, etc.).

Gpt is really bad at optimizing prompts this way because there is no way it has the ability to simulate the effects, way too complex. Tools like this need to log and a/b test.

gpt can be layered and made into an agent etc. To do the AB testing or to make prompts longer by adding more end cases as time goes by. But the effects of one single word change are far too complex for gpt base output to understand anything about.

I'm sure it could be improved, including telling it to do what you suggest. Have you tried it as is though?

Yes I used it. The optimized prompt was not better for my use case. The playground was useful though. I believe prompt optimization is really only optimized by running it through many scenarios and understanding how changing a single word affects things down the line. And then a bunch of hardcoded conditions to change the system/assistant messages on demand as an output of the tool.

> The prompt it generates looks good and makes sense

What I'm trying to say is that that's exactly what it's optimized for. They're predicting what sounds plausible based on all the pre-gpt writing about AI.

But GPT was revolutionary! A lot of the pre-gpt blogspam and reddit comments and fiction and so on was wrong about how AI works in exactly the way you've been socialized to find plausible.

In general plausibility is the wrong metric to evaluate GPT on, and it's wronger than it seems like it should be.

you'd be better off with two chats, one is your prompt creator, you pick the prompt, use another chat to create answer (zero shot chat cleared every time), and you copy the answer back into the prompt creator, with feedback on what the answer get wrong.

that's because from the point of view of gpt, gpt is always providing is most accurate anwer already, and the only improvement you'd get is in context clarity as defined by gpt understanding of language, which is not bad by itself, but it's not as effective as having gpt fight itself.

Yes, the way the prompt I put is that it's a session, so you'll open a new session per prompt, for each prompt you want to work on.

Then when you want to use that prompt, you'll open yet another session. Sorry if that was unclear.

Computer science has become so weird ...

If you think creating and using prompts are computer science then yeah, it has gotten weird :) Fortunately, I think that's a minority view.