I'm exploring a dev tool idea that helps you A/B test your prompts, but I'm not sure if there's a need for it. You'd be able to write and version your prompts in a web UI, then A/B test them and see results with metrics you define.So for example, with a bot that writes cold outbound emails, you can verify whether v1 or v2 of your system prompt results in a better reply rate.Does anybody...
I built a platform that allows you to more easily experiment with different system prompts in production. You can record your own metrics and it will automatically tie this information to whatever experiment treatment the user is in. For example, being able to test two system prompts and see which one actually improves user success rates or engagement. This might be useful in something like a...
I noticed that there are a lot of of LLMOps platforms focused on offline evals, but I couldn’t find anything that manages A/B tests in production and ties different prompts to quantifiable user metrics. For example, being able to test two system prompts and see which one actually improves user success rates or engagement. This might be useful in something like a sales or customer support...