WebGPU GPT Model Demo

Re:WebGPU GPT Model Demo by MuffinFlavored on Thursday April 20, 2023 @11:35PM (#35649573)

> At the moment, WebGPT averages ~300ms per token on GPT-2 124M running on a 2020 M1 Mac with Chrome Canary.

How do ChatGPT on GPT-3.5 / GPT-4 compare?

[ Parent | on HN ]

Re:WebGPU GPT Model Demo by zamadatix on Thursday April 20, 2023 @11:47PM (#35649638)

I don't have exact numbers but trying all 3 out: on a local 4090 this seems mildly slower than GPT-4 and ridiculously slower than GPT 3.5 (both via cloud GPUs of course). That said all are within the realm of usability in terms of speed though GPT 3.5 is really in a whole different class of being able to be used nearly interactively without delay.

[ Parent | on HN ]

Re:WebGPU GPT Model Demo by lxe on Friday April 21, 2023 @01:08AM (#35650000)

Interesting, I'm getting 100ms/token on plain old wasm with 4 threads via ggml, using a 1.7B quantized cerebras model.

[ Parent | on HN ]

Re:WebGPU GPT Model Demo by brucethemoose2 on Friday April 21, 2023 @01:42AM (#35650138)

I can do 60 tokens in 24s on my 2020 ASUS G14, Thorium 111 + Windows 10. Nvidia-smi says my RTX 2060 is 24% loaded and has ~1.1G eaten up.

Thats slower than Vicuna 7B (aka LLaMa 7000M, GPT 3ish? model) on linux on the same machine, where I get about 3.5 tokens/sec and 97% usage. So... yeah, performance is not so great yet.

[ Parent | on HN ]