News for hackers. Stuff that matters.

A comment on Show HN: LLaMA tokenizer that runs in browser. Parent comment

Re:Show HN: LLaMA tokenizer that runs in browser by WilliamBerglund on Wednesday June 14, 2023 @04:08AM (#36322937)
Yes, tiktoken, here's a client side visualizer for it

https://github.com/functorism/gpt4-tokenizer-visualizer

[ Parent | on HN ]

There are no comments on this comment yet.