News for hackers. Stuff that matters.
Re:Xkcd 1425 (Tasks) turns ten years old today by conesus on Thursday September 26, 2024 @01:15PM (#41660850)
GPT-4o uses BPE (byte pair encoding). They released `tiktoken` which allows you to count tokens in strings in python.

    pip install tiktoken
    >>> import tiktoken
    >>> encoding = tiktoken.encoding_for_model("gpt-4o")
    >>> print(encoding.encode("hello marcellus"))
    [24912, 2674, 10936, 385]

There are no comments on this comment yet.