Re:You Should Probably Pay Attention to Tokenizers
by danielmarkbruce on Thursday October 24, 2024 @12:43PM
(#41937222)
I wouldn't consider that a thing that's going to work generally. That word may tokenize to one per char and have seen relevant data, or it may be relatively close to some other word and it's seen data which gives the answer.
[ Parent
| on HN ]
There are no comments on this comment yet.