Re:Weird GPT-4 behavior for the specific string “ davidjl”
by pphysch on Thursday June 08, 2023 @02:34PM
(#36246520)
That might work fine for English, but I would be very surprised if it performed better for non-ASCII logograms e.g. hanzi/kanji. Or would the network just learn those characters?
[ Parent
| on HN ]
Re:Weird GPT-4 behavior for the specific string “ davidjl”
by glandium on Friday June 09, 2023 @01:40AM
(#36253728)
Interestingly enough, for Japanese, the average is more than 1 token per Japanese character. It's not at the byte level, but it's clearly at a sub-utf-8 level. And yet, considering this, it works amazingly well.