Re:Why is promptfoo so popular?
by haldunanil on Saturday July 27, 2024 @01:31PM
(#41088065)
I can't speak to why it's popular, but I will say that the reason I chose it for one of my projects was because I pretty much had to as there were no better alternatives.
Most of the other tools like RAGAS and DeepEval are:
- Written in Python (vs. JavaScript like promptfoo), whereas our codebase is in Rails
- Get too into the weeds of AI, whereas I just want to write simple tests for my webapp not stare at a bunch of math equations
- Are hosted SaaS solutions (vs. open source like promptfoo) and I don't want to deal with the 6 month enterprise software sales cycle just to add a couple tests
So we basically ended up with promptfoo. That said, it felt really awkward to use promptfoo and I don't really love it. I'd love a super simple way to add some integration tests for an LLM so I can just keep iterating
Most of the other tools like RAGAS and DeepEval are:
- Written in Python (vs. JavaScript like promptfoo), whereas our codebase is in Rails
- Get too into the weeds of AI, whereas I just want to write simple tests for my webapp not stare at a bunch of math equations
- Are hosted SaaS solutions (vs. open source like promptfoo) and I don't want to deal with the 6 month enterprise software sales cycle just to add a couple tests
So we basically ended up with promptfoo. That said, it felt really awkward to use promptfoo and I don't really love it. I'd love a super simple way to add some integration tests for an LLM so I can just keep iterating