News for hackers. Stuff that matters.

A comment on Show HN: LLaMA 3 tokenizer runs in the browser.

Re:Show HN: LLaMA 3 tokenizer runs in the browser by mrbishalsaha on Monday April 22, 2024 @12:48PM (#40116129)
Really good. I am actually using js-tiktokken and wish there was a package to handle all the other LLMs also but still something I can work with.
[ Parent | on HN ]
Re:Show HN: LLaMA 3 tokenizer runs in the browser by belladoreai on Monday April 22, 2024 @02:54PM (#40117535)
If you need to work with multiple LLMs, you probably want to use transformers.js
[ Parent | on HN ]
Re:Show HN: LLaMA 3 tokenizer runs in the browser by mrbishalsaha on Monday April 22, 2024 @06:47PM (#40123478)
Isn't it to much for just calculating the number of token?
[ Parent | on HN ]
Re:Show HN: LLaMA 3 tokenizer runs in the browser by belladoreai on Tuesday April 23, 2024 @12:28PM (#40133821)
It's the best option you have if you need to work with multiple LLMs in the browser.
[ Parent | on HN ]