Show HN: Ragas – the de facto open-source standard for evaluating RAG pipelines

Re:Show HN: Ragas – the de facto open-source standard for ev... by jfisher4024 on Thursday March 21, 2024 @03:55PM (#39783599)

Congratulations on the launch! I was unable to use this library: I was trying to evaluate different non-OpenAI models and it consistently failed due to malformed JSONs coming from the model.

Any thoughts about using different models? Is this just a langchain limitation?

[ Parent | on HN ]

Re:Show HN: Ragas – the de facto open-source standard for ev... by shahules on Thursday March 21, 2024 @04:13PM (#39783805)

Thanks for your feedback. We have tested Ragas on alternatives like Claude, Mixtral, Gemini, etc.

Although we support all LLMs supported by Langchain, sadly many of the OSS models out of the box aren't capable of generating JSON-type output which is important for us to ensure reproducibility.

[ Parent | on HN ]

Re:Show HN: Ragas – the de facto open-source standard for ev... by jfisher4024 on Friday March 22, 2024 @12:05AM (#39787375)

Any tips for Mixtral? That’s what we tried

[ Parent | on HN ]

Re:Show HN: Ragas – the de facto open-source standard for ev... by shahules on Friday March 22, 2024 @12:43PM (#39792426)

Hey, I would recommend checking out our PRs. There would be PRs that have modified some of the prompts to better suit Mixtral.

[ Parent | on HN ]

Re:Show HN: Ragas – the de facto open-source standard for ev... by 3d27 on Friday March 22, 2024 @03:15AM (#39788104)

Checkout this instead: https://github.com/confident-ai/deepeval

Also has native ragas implementation but supports all models.

[ Parent | on HN ]