You Should Probably Pay Attention to Tokenizers

Re:You Should Probably Pay Attention to Tokenizers by bunderbunder on Wednesday October 23, 2024 @04:08PM (#41928842)

I suspect that the holy grail here is figuring out how to break the input into a sequence of morphemes and non-morpheme lexical units.

[ Parent | on HN ]

Re:You Should Probably Pay Attention to Tokenizers by thaumasiotes on Wednesday October 23, 2024 @08:59PM (#41930854)

What do you mean by non-morpheme lexical units? Syntactic particles, units too small to be morphemes? Lexical items that contain multiple morphemes?

In either case, isn't this something we already do well?

[ Parent | on HN ]

Re:You Should Probably Pay Attention to Tokenizers by bunderbunder on Thursday October 24, 2024 @10:37AM (#41935939)

Punctuation, for example.

And no, at least for the languages with which I'm familiar SOTA tokenizers tend to only capture the easy cases.

For example, the GPT-4 tokenizer breaks the first sentence of your post like so:

  What/ do/ you/ mean/ by/ non/-m/orp/heme/ lexical/ units/?

Notice how "morpheme" gets broken into three tokens, and none of them matches "morpheme"'s two morphemes. "Lexical" and "units" are each a single token, when they have three and two morphemes respectively.

Or in French, the word "cafetière" gets chopped willy-nilly into "c/afet/ière". The canonical breakdown is "cafe/t/ière".

[ Parent | on HN ]