News for hackers. Stuff that matters.

A comment on Transformers.js: State-of-the-art Machine Learning for the web. Parent comment

Re:Transformers.js: State-of-the-art Machine Learning for th... by janalsncm on Thursday April 11, 2024 @01:17PM (#40004536)
Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time.

For reference I have a 31mb vision transformer I run in my browser. Building the inputs, running inference, and parsing the response takes less than half a second.

[ Parent | on HN ]
Re:Transformers.js: State-of-the-art Machine Learning for th... by bilekas on Thursday April 11, 2024 @05:28PM (#40007013)
> Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time.

I can understand that but where time is not a factor and solely a question of data, can a model be streamed?

[ Parent | on HN ]
Re:Transformers.js: State-of-the-art Machine Learning for th... by pests on Thursday April 11, 2024 @10:04PM (#40008754)
LLMs like ChatGPT only generate one token at a time. To generate more you run inference repeatedly until you reach a stop token or some other predetermined limit.

I don't see streaming helping anything besides maybe Time-To-First-Inference, but regardless, you're still not getting any output until the entire weights are downloaded.

[ Parent | on HN ]