News for hackers. Stuff that matters.

A comment on Web LLM – WebGPU Powered Inference of Large Language Models. Parent comment

Re:Web LLM – WebGPU Powered Inference of Large Language Models by junrushao1994 on Saturday April 15, 2023 @05:59PM (#35585008)
This is unfortunately non-trivial to quantitatively evaluate the performance against ChatGPT :-(

We didn't do much evaluation because there isn't much innovation on model side, but instead we are demoing the possibility of running an end-to-end model on ordinary client GPUs via WebGPU without server resources.

[ Parent | on HN ]
Re:Web LLM – WebGPU Powered Inference of Large Language Models by amelius on Saturday April 15, 2023 @08:26PM (#35586026)
> This is unfortunately non-trivial to quantitatively evaluate the performance against ChatGPT :-(

Compare using the loss function?

[ Parent | on HN ]
Re:Web LLM – WebGPU Powered Inference of Large Language Models by junrushao1994 on Saturday April 15, 2023 @09:26PM (#35586404)
In LLM world, loss or perplexity may not be the best indicator of model performance :-( Perhaps HELM (https://crfm.stanford.edu/helm/latest/) but we didn't take deeper look as we are not the developers of this model
[ Parent | on HN ]
Re:Web LLM – WebGPU Powered Inference of Large Language Models by imranq on Sunday April 16, 2023 @01:11AM (#35587457)
There are plenty of LLM benchmarks that are used to test performance, some of them are: * Winogrande

* BoolQ

* PIQA

* SIQA

* HellaSwag

etc...

[ Parent | on HN ]
Re:Web LLM – WebGPU Powered Inference of Large Language Models by junrushao1994 on Sunday April 16, 2023 @03:13AM (#35587952)
Would be nice if anyone could help us benchmark! Our primary focus though is not model performance, but to demonstrate the capability that TVM Unity generates code targeting WebGPU and allows them to run with client GPUs :-)
[ Parent | on HN ]