News for hackers. Stuff that matters.

A comment on OpenAI Tokenizer. Parent comment

Re:OpenAI Tokenizer by swyx on Wednesday April 05, 2023 @10:32AM (#35454914)
hmm. so this is evidence that openai scraped twitch chat of all places? (notoriously ephemeral)

also opens a question as to how tokenizers are trained. should you discard or break up super niche words like this?

[ Parent | on HN ]
Re:OpenAI Tokenizer by squeaky-clean on Wednesday April 05, 2023 @04:19PM (#35459833)
It doesn't necessarily mean it scraped twitch chat. That is the name of a moderator. They also moderate the subreddit and probably some other places. And being a moderator for such a popular event they probably had their name mentioned in other places as well. Every time they comment on Reddit their username would also appear.

https://www.reddit.com/r/twitchplayspokemon/comments/2cxkpp/...

[ Parent | on HN ]