News for hackers. Stuff that matters.
Re:Show HN: Automatic Prompt Optimizer for LLMs by pigeons on Friday April 21, 2023 @08:15PM (#35661947)
From my experiments so far, I am unsure if chatgpt is effective at optimizing prompts, even though it thinks it is.
Re:Show HN: Automatic Prompt Optimizer for LLMs by capableweb on Friday April 21, 2023 @08:17PM (#35661965)
The workflow you get with this is that it helps you to think about what you really want answered with the prompt.

For example, latest try had this initial prompt:

> How can I create collisions in Bevy and Rust?

After 4-5 messages back and forth, I ended up with:

> What is the most efficient method to implement 2D collision detection for a large number of constant-radius circles randomly distributed in Bevy and Rust without using third-party tools, focusing on detection only, without any unique properties or attributes affecting the detection process?

Which is much more explicit and targeted to what I initially wanted to do, but didn't write. It ended up helping me implement a Quadtree solution which I don't have any experience with before, but overall it went smoothly.

Re:Show HN: Automatic Prompt Optimizer for LLMs by charcircuit on Friday April 21, 2023 @11:57PM (#35663216)
How do you know that the prompt you got to is at least a local maximum? Would chancing "method" to "function" provide better results? Did you do any benchmarking?
Re:Show HN: Automatic Prompt Optimizer for LLMs by capableweb on Saturday April 22, 2023 @05:22AM (#35664741)
The aim is not any local maximum, is improving the prompt by getting help providing more context.
Re:Show HN: Automatic Prompt Optimizer for LLMs by charcircuit on Saturday April 22, 2023 @05:05PM (#35670929)
That's a pretty bad optimizer then.
Re:Show HN: Automatic Prompt Optimizer for LLMs by ryanjshaw on Saturday April 22, 2023 @06:00AM (#35664930)
I've seen it help me once so far in a way I hadn't thought of: basically, the chat completion engine was reading a bug bounty report on my behalf, and believing generic, vague claims. ChatGPT-3.5 didn't "fall" for this, but 4 did consistently.

After failing to improve performance manually, I eventually asked 4 to improve my prompt after explaining the problem and it basically added "Don't assume the existence of any examples".

This makes perfect sense in hindsight, but I had been approaching things from the other direction - directing it to only consider explicit examples in the report, and for some reason that didn't work.