Tiktoken-rs: Batteries-included Rust library for working with GPT-4's tokenizer

Posted by zurawiki on Saturday March 18, 2023 @12:22AM
from the github.com dept.

( 1 comment | on HN )

Re:Tiktoken-rs: Batteries-included Rust library for working ... by zurawiki on Saturday March 18, 2023 @12:22AM (#35206062)

In case others find this useful. Just updated for GPT-4, tiktoken-rs is a Rust crate you can use to tokenize text sent to and from OpenAI's LLM models

[ Parent | on HN ]