News for hackers. Stuff that matters.

A comment on Show HN: I implemented evals metrics for LLMs that runs locally on your machine.

Re:Show HN: I implemented evals metrics for LLMs that runs l... by srbhr on Tuesday December 12, 2023 @01:36AM (#38610292)
The idea around a testing framework for LLMs is nice. Can you provide some examples where this can be used?
[ Parent | on HN ]
Re:Show HN: I implemented evals metrics for LLMs that runs l... by 3d27 on Tuesday December 12, 2023 @02:13AM (#38610566)
The package I built is like a provider for 10+ different evaluation metrics that run both locally on your machine using models from hugging-face but also on the cloud IF you want more functionality.

If you want to evaluate a fine-tuned model, we have integrations with LM Harness and Stanford HELM coming out. If you want to evaluate a RAG application, we have 7+ metrics available for that.

You can also create your custom metrics using our interface!

[ Parent | on HN ]