Show HN: How LLMs Work – Interactive visual guide based on Karpathy's lecture

Re:Show HN: How LLMs Work – Interactive visual guide based o... by thesz on Friday April 24, 2026 @10:47AM (#47891050)

The page does very poor job tokenizing phrase "Noinceolik fiyulnabmed fyvaproldge" into "Noinceolik fiyulnabm ed fyvaproldge", factoring only "ed" suffix. As if made up words such as "noinceolik" are so common they are part of 100K token vocabulary.

The actual application of GPT-5 tokenizer at [1] to my made up phrase results in 14 tokens, only two of them are four characters long and there are tokens containing spaces.

[1] https://gpt-tokenizer.dev/

I will read along, though.

[ Parent | on HN ]

Re:Show HN: How LLMs Work – Interactive visual guide based o... by ynarwal__ on Friday April 24, 2026 @10:54AM (#47891139)

I appreciate the feedback, I did notice that as well and I had this thought perhaps this is not worth fixing since I have a link to tiktokenizer. I decided to remove it and just added a more prominent link to tiktokenizer.

[ Parent | on HN ]

Re:Show HN: How LLMs Work – Interactive visual guide based o... by thesz on Saturday April 25, 2026 @06:06AM (#47900177)

BPE that is used in tokenization is very simple: https://en.wikipedia.org/wiki/Byte-pair_encoding

[ Parent | on HN ]