News for hackers. Stuff that matters.
Re:Tokenisation Is NP-Complete by bnjmn on Monday December 23, 2024 @08:09AM (#42494118)
Thanks for these detailed replies! Now I really want to read your paper.
Re:Tokenisation Is NP-Complete by mcyc on Monday December 23, 2024 @08:20AM (#42494191)
Thanks!

Our paper [1] is kind of a goofy adversarial thing where we thought "here's this cool metric, how can we break it?". The tokenizers we propose are definitely not tokenizers you should use in practice.

The original paper that proposes the metric is, imo, much more interesting theoretically [2].

[1]: https://aclanthology.org/2024.lrec-main.1469/

[2]: https://aclanthology.org/2023.acl-long.284/