News for hackers. Stuff that matters.

A comment on Understanding GPT Tokenizers. Parent comment

Re:Understanding GPT Tokenizers by hammeiam on Friday June 09, 2023 @09:40AM (#36257661)
Would you be willing to share a GitHub link? This seems like a fun project to read through.
[ Parent | on HN ]
Re:Understanding GPT Tokenizers by belladoreai on Friday June 09, 2023 @11:05AM (#36258921)
Sure. Check back to this comment after the weekend. I will post it here.
[ Parent | on HN ]
Re:Understanding GPT Tokenizers by belladoreai on Sunday June 11, 2023 @06:27PM (#36286368)
Published now: https://github.com/belladoreai/llama-tokenizer-js
[ Parent | on HN ]