News for hackers. Stuff that matters.

A comment on Web LLM – WebGPU Powered Inference of Large Language Models. Parent comment

Re:Web LLM – WebGPU Powered Inference of Large Language Models by amelius on Saturday April 15, 2023 @08:26PM (#35586026)
> This is unfortunately non-trivial to quantitatively evaluate the performance against ChatGPT :-(

Compare using the loss function?

[ Parent | on HN ]
Re:Web LLM – WebGPU Powered Inference of Large Language Models by junrushao1994 on Saturday April 15, 2023 @09:26PM (#35586404)
In LLM world, loss or perplexity may not be the best indicator of model performance :-( Perhaps HELM (https://crfm.stanford.edu/helm/latest/) but we didn't take deeper look as we are not the developers of this model
[ Parent | on HN ]