News for hackers. Stuff that matters.

joshreini

Hey @sirspacey - I'm a TruLens/TruEra dev. If we're not in contact already, feel free to shoot me an email or join our slack community if you need assistance or want to talk through how to best leverage TruEra for this.

email: [email protected] slack: https://communityinviter.com/apps/aiqualityforum/josh

Bias up front: I'm a TruLens developer

------

We've built all of these considerations:

1) Support for standard metrics like BLEU, ROUGE, BERT similarity, and model-graded evals; we serialize the entire LLM app call so you can test anything that happens in it (context chunks, tool calls/inputs, etc.)

2) Bc we serialize the whole record, you can use this tracing to debug failing evals. We also have chain-of-thought reasoning evals that can explain why an eval failed. Last - there's a streamlit UI you can launch (tru.run_dashboard) that'll run locally

3) Hallucination is probably the biggest problem we solve for. To do evals for hallucination, we typically see our users use a combination of groundedness (does the context support the LLM response) and context relevance (is the retrieved context relevant to the query). There's also a bunch more for the evaluations you mentioned (moderation models, sentiment, usefulness, etc.) and it's pretty easy to add custom evals.

Also - my hot take is that gpt-3.5 is good enough for evals (sometimes better) than gpt-4 if you give the LLM enough instructions on how to do the eval.

website: https://www.trulens.org/ github: https://github.com/truera/trulens

Re:Stop Evaluating LLMs on Vibes on Wednesday June 07, 2023 @10:55AM (#36227577)
Developer on the trulens team here. We just put out a pretty cool new release (0.1.2). Integration with huggingface pipelines in langchain and asynchronous feedback management are just some of the useful updates in this release.

Real evaluations (maybe evaluated by LLMs themselves) instead of just vibe checks is the next big step we need to take as an industry.