DeepDive in everything of Llama3: revealing detailed insights and implementation

Re:DeepDive in everything of Llama3: revealing detailed insi... by simonw on Friday February 21, 2025 @01:10PM (#43130862)

I hadn't realized OpenAI's tiktoken Python library could work with other models outside of the OpenAI family, that's really useful: https://github.com/therealoliver/Deepdive-llama3-from-scratc...

[ Parent | on HN ]

Re:DeepDive in everything of Llama3: revealing detailed insi... by therealoliver on Friday February 21, 2025 @09:11PM (#43135402)

I'm glad to have helped you :)

[ Parent | on HN ]

Re:DeepDive in everything of Llama3: revealing detailed insi... by moffkalast on Saturday February 22, 2025 @07:28AM (#43138401)

It's more than just that, practically every notable open model released in the past year or so uses tiktoken as the tokenizer.

[ Parent | on HN ]