News for hackers. Stuff that matters.
Re:tiktoken: OpenAI’s Tokenizer by moss2 on Friday December 16, 2022 @03:51AM (#34012340)
What's a tokenizer?
Re:tiktoken: OpenAI’s Tokenizer by sbergot on Friday December 16, 2022 @03:53AM (#34012353)
It is a processing step in natural language processing: https://www.analyticsvidhya.com/blog/2020/05/what-is-tokeniz...
Re:tiktoken: OpenAI’s Tokenizer by teruakohatu on Friday December 16, 2022 @04:17AM (#34012517)
Example: Split an input into tokens.

Tokens: [split, an, input, into, tokens, .]

Re:tiktoken: OpenAI’s Tokenizer by deelly on Friday December 16, 2022 @06:28AM (#34013367)
Example: umbrella

Tokens: [umb, rella] ?

Re:tiktoken: OpenAI’s Tokenizer by artifabrian on Friday December 16, 2022 @09:53AM (#34015141)
Could be as well, given there's many varieties of tokenisers, each with different pros and cons.

This particular tokenizer is very interesting given that it tries to be best of both worlds (word-level tokenizer and character-level).