Understanding GPT Tokenizers

Re:Understanding GPT Tokenizers by sillysaurusx on Friday June 09, 2023 @04:03AM (#36254760)

Thanks for this! That was very nice and thoughtful.

There’s something poetic about ULL being a token, but NULL not being one.

[ Parent | on HN ]

Re:Understanding GPT Tokenizers by thyselius on Friday June 09, 2023 @05:49AM (#36255550)

Saw many words missing their first letter. Realized it’s probably because it’s sometimes Null and sometimes null

[ Parent | on HN ]

Re:Understanding GPT Tokenizers by hammeiam on Friday June 09, 2023 @09:38AM (#36257650)

ULL is also used for designating Unsigned Long Long 64bit ints in c++, so it’s not just a part of the null symbol.

[ Parent | on HN ]