News for hackers. Stuff that matters.
Re:Ask HN: Anyone working on something better than LLMs? by quadrature on Friday May 10, 2024 @05:32PM (#40323975)
>Yes, there are compounded words like strawberry (straw + berry) and you can capture meaning at a higher-resolution, but most words are not compounded

What's really cool about tokenization is that it breaks down words based on how often parts of the word are used. This helps a lot with understanding different forms of words, like when you add "-ing" to a verb, make words plural, or change tenses. It's like seeing language as a bunch of building blocks.

Re:Ask HN: Anyone working on something better than LLMs? by xucian on Saturday May 11, 2024 @02:15PM (#40329720)
agree, I think this is the best argument for autoregressive models, and why they are so 'unreasonably' good