I was curious what would go wrong, this just worked for me:
py -3 -m pip install tokenizers
...
Building wheel for tokenizers (pyproject.toml) ... done
Created wheel for tokenizers: filename=tokenizers-0.13.2-cp311-cp311-win_amd64.whl
Re:Why I'm still using Python
by heavyset_go on Saturday December 31, 2022 @06:36PM
(#34201874)
Looks like if you try to install it via source you might need a compiler/linker toolchain installed for its Rust bindings.
A practical example: on Windows, try 'pip install tokenizers'.