Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on OpenAI Tokenizer. Parent comment
> a single user-perceived character might span into multiple tokens
Is this the way it works as designed or is this a bug?
There are no comments on this comment yet.
> a single user-perceived character might span into multiple tokens
Is this the way it works as designed or is this a bug?