News for hackers. Stuff that matters.

A comment on Mistral: Our first AI endpoints are available in early access. Parent comment

Re:Mistral: Our first AI endpoints are available in early ac... by raverbashing on Monday December 11, 2023 @05:47AM (#38599394)
Do they all use the same tokenizer? (I mean, Mistral vs GPT)
[ Parent | on HN ]
Re:Mistral: Our first AI endpoints are available in early ac... by superkuh on Monday December 11, 2023 @11:14AM (#38601852)
No. Mistral uses sentencepiece and the GPT use tiktoken.
[ Parent | on HN ]