Ask HN: Prompt manager for developers / services?

Posted by theogravity on Thursday August 24, 2023 @08:06PM
from the ask-hn dept.

I'm working on a system that utilizes OpenAI / ChatGPT and hard-code a lot of prompt or chunks of prompts that would be combined together in variables.

Is there a product out there that allows me or a product person to:

  - Store a list of prompts in a key/value like fashion
  - Has an API where I can request a list of keys and get the prompts corresponding to them
  - Has version history
  - Can assign a version of a prompt to a particular environment (eg dev, prod)
  - Lets others edit prompts
  - (Optional) Can test prompts against an LLM
  - (Optional) The API / SDK can listen for updates to prompts like how Launch Darkly can push flag changes in real-time.

I've searched for "prompt manager", "ai prompt manager", "prompt management tool", and variants. The tools that do come up seem to be geared for non-developers or has no API for me to just fetch plain text prompts for use in our own LLM in our own infrastructure.

The core of it is probably a hosted versioned K/V system with ACL, and an API to access that K/V wrapped around a GUI for management.

( 6 comments | 2 points | on HN )

Re:Ask HN: Prompt manager for developers / services? by colesouth on Thursday August 24, 2023 @08:24PM (#37256294)

I'm building a consumer facing prompt manager (https://promptfolder.com/ - no API yet), but I've looked at most of the other tools... unfortunately I haven't seen anything that is a solid fit for what you're looking for.

You might want to contact the devs behind PromptPal (https://github.com/PromptPal/PromptPal), they seem to be some of the more technically advanced and fast moving people building in this area... maybe they can hack something together for you.

AIPRM is one of the more popular prompt managers, but as of May they don't have a public API (https://forum.aiprm.com/t/use-aiprm-prompt-templates-api/382...).

[ Parent | on HN ]

Re:Ask HN: Prompt manager for developers / services? by theogravity on Thursday August 24, 2023 @08:29PM (#37256327)

I actually wrote an issue on the PromptPal GH around 30 minutes before my HN post. They use Metamask / Ethereum addresses for auth / account management, which sounds crazy to me for a product that's completely divorced from web3 / cryptocurrency.

Promptfolder isn't what I'm looking for unfortunately based on the template variable assignment the example has. I really just want to store text and get it back via an API. The text just happens to be a prompt.

[ Parent | on HN ]

Re:Ask HN: Prompt manager for developers / services? by colesouth on Thursday August 24, 2023 @08:32PM (#37256351)

Pivoting from crypto to AI... many such cases :)

Sounds like there is maybe some code debt involved!

[ Parent | on HN ]

Re:Ask HN: Prompt manager for developers / services? by breckenedge on Thursday August 24, 2023 @08:14PM (#37256226)

Like promptfoo? https://github.com/promptfoo/promptfoo

[ Parent | on HN ]

Re:Ask HN: Prompt manager for developers / services? by theogravity on Thursday August 24, 2023 @08:17PM (#37256253)

Promptfoo looks like it's mainly built for testing prompts and the API is framed around testing.

The readme states: "The goal: test-driven prompt engineering, rather than trial-and-error."

Which is an optional part of my requirements.

[ Parent | on HN ]

Re:Ask HN: Prompt manager for developers / services? by juntao on Friday August 25, 2023 @05:43AM (#37259817)

Maybe you can try https://flows.network/, which supports environment variables when creating an app. You can also manage the environment variables later with their UI. Click on https://flows.network/flow/createByTemplate/Telegram-ChatGPT, you can see you could set up system_prompt.

[ Parent | on HN ]