News for hackers. Stuff that matters.

A comment on Introducing ChatGPT and Whisper APIs. Parent comment

Re:Introducing ChatGPT and Whisper APIs by dmw_ng on Wednesday March 01, 2023 @03:07PM (#34987560)
https://en.wikipedia.org/wiki/Byte_pair_encoding

In short, common words cost one token, rarer words cost a few tokens

[ Parent | on HN ]

There are no comments on this comment yet.