Understanding GPT Tokenizers

Re:Understanding GPT Tokenizers by s-macke on Friday June 09, 2023 @02:55AM (#36254268)

Here is the list of the 100k GPT-4 tokens as text file.

Yes, a lot of tokens are just for code.

Edit: Here as raw link for the poor mobile devices:

[ Parent | on HN ]

Re:Understanding GPT Tokenizers by sillysaurusx on Friday June 09, 2023 @04:03AM (#36254760)

Thanks for this! That was very nice and thoughtful.

There’s something poetic about ULL being a token, but NULL not being one.

[ Parent | on HN ]

Re:Understanding GPT Tokenizers by thyselius on Friday June 09, 2023 @05:49AM (#36255550)

Saw many words missing their first letter. Realized it’s probably because it’s sometimes Null and sometimes null

[ Parent | on HN ]

Re:Understanding GPT Tokenizers by hammeiam on Friday June 09, 2023 @09:38AM (#36257650)

ULL is also used for designating Unsigned Long Long 64bit ints in c++, so it’s not just a part of the null symbol.

[ Parent | on HN ]

Re:Understanding GPT Tokenizers by josecyc on Thursday June 22, 2023 @10:30PM (#36442110)

Where are all the tokens for other languages though? How are these getting tokenized?

[ Parent | on HN ]