jipster

R2R uses deepeval for their evaluation :) https://github.com/confident-ai/deepeval

Hey, cofounder here. As far as I know, LangSmith focuses on observability (token usage, chain of thought insights, LLM cost), is vendor locked into LangChain, and uses LLMs to evaluate themselves.

Here are the problems with what LangSmith offers: 1. Focusing on observability brings insights to engineering teams in terms of cost, latency, but these conclusions aren't really actionable

2. Not all teams use LangChain, plus a lot of teams who originally used LangChain to prototype is moving away as they productionize and look to improve what they've built

3. Not every LLM implementation involves a chain of thought, lLamaIndex is a good example. Apart from vendor locking, there are other use cases such as evaluating fine tuned models that are overlooked in a closed source software like LangSmith

4. Using LLMs to evaluate themselves are fine, but a lot more can be done. We offer other metrics (metrics that can be quantified from 0 - 1) such as factual consistency trained on NLI models that are much more deterministic. You can also use our package to pick and choose metrics that you care about (bias, relevancy, toxicity, etc)

Hope that answers your questions.

Hey HN! We've built this platform that allows you to evaluate how well your LLM implementation is performing, despite that be using open source tools such as LangChain, lLamaIndex, or even your own internal framework.

The idea is you would use our open source package (https://github.com/confident-ai/deepeval) to evaluate LLM outputs using criteria such as factual consistency, relevancy, bias, etc, and be able to have a bird eye's view on the overall performance on app.confident-ai.com as our package allows you to run analysis on our servers, all for free!

Would appreciate it if you could give it a try, thanks :)