News for hackers. Stuff that matters.
Re:LLMs are not greater than the sum of their parts: Stanfor... by mlyle on Tuesday May 09, 2023 @04:20PM (#35879880)
I dunno. Sometimes a group of numbers has a non-mathematical semantic meaning that's a good mapping to digits-- like an area code or '777'. A lot of the rest of the time it's pretty random. A tokenizer's job is to lower the size of the input vector for a given amount of input meaning without obscuring the real underlying relationships too much, and here it feels like it doesn't meet that goal.

My phone number is 6 tokens instead of 12 symbols... so this is only going to make a moderate difference on things like big lists of phone numbers.

There are no comments on this comment yet.