Re:OpenAI Tokenizer
by kevingadd on Wednesday April 05, 2023 @10:26AM
(#35454796)
Some of the magic tokens are related to Twitch Plays Pokemon.
https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...
[ Parent
| on HN ]
hmm. so this is evidence that openai scraped twitch chat of all places? (notoriously ephemeral)
also opens a question as to how tokenizers are trained. should you discard or break up super niche words like this?
Re:OpenAI Tokenizer
by squeaky-clean on Wednesday April 05, 2023 @04:19PM
(#35459833)
It doesn't necessarily mean it scraped twitch chat. That is the name of a moderator. They also moderate the subreddit and probably some other places. And being a moderator for such a popular event they probably had their name mentioned in other places as well. Every time they comment on Reddit their username would also appear.
https://www.reddit.com/r/twitchplayspokemon/comments/2cxkpp/...