News for hackers. Stuff that matters.

A comment on Show HN: LLaMA tokenizer that runs in browser.

Re:Show HN: LLaMA tokenizer that runs in browser by adroitboss on Tuesday June 13, 2023 @07:20PM (#36319042)
Does anyone know of a chatgpt/gpt-4 tokenizer that can run client-side?
[ Parent | on HN ]
Re:Show HN: LLaMA tokenizer that runs in browser by sp332 on Tuesday June 13, 2023 @07:31PM (#36319142)
https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken
[ Parent | on HN ]
Re:Show HN: LLaMA tokenizer that runs in browser by oofsa on Wednesday June 14, 2023 @12:49AM (#36321577)
https://platform.openai.com/tokenizer is not for GPT-4 but GPT-3. https://tiktokenizer.vercel.app/ supports GPT-4.
[ Parent | on HN ]
Re:Show HN: LLaMA tokenizer that runs in browser by adroitboss on Wednesday June 14, 2023 @07:12AM (#36324121)
Thanks for the recommendations. I found the one I needed because of this comment thread.
[ Parent | on HN ]
Re:Show HN: LLaMA tokenizer that runs in browser by YetAnotherNick on Tuesday June 13, 2023 @07:26PM (#36319096)
https://github.com/cogentapps/chat-with-gpt/blob/main/app/sr...
[ Parent | on HN ]
Re:Show HN: LLaMA tokenizer that runs in browser by WilliamBerglund on Wednesday June 14, 2023 @04:08AM (#36322937)
Yes, tiktoken, here's a client side visualizer for it

https://github.com/functorism/gpt4-tokenizer-visualizer

[ Parent | on HN ]
Re:Show HN: LLaMA tokenizer that runs in browser by adroitboss on Wednesday June 14, 2023 @07:13AM (#36324125)
Thanks for the recommendations everyone.
[ Parent | on HN ]