Re:Prediction: AI will make formal verification go mainstream
by Uehreka on Tuesday December 16, 2025 @05:32PM
(#46295543)
No, it’s an example that shows that LLMs still use a tokenizer, which is not an impediment for almost any task (even many where you would expect it to be, like searching a codebase for variants of a variable name in different cases).
[ Parent
| on HN ]
Re:Prediction: AI will make formal verification go mainstream
by 8note on Tuesday December 16, 2025 @05:55PM
(#46295840)
the question remains: is the tokenizer going to be a fundamental limit to my task? how do i know ahead of time?
Re:Prediction: AI will make formal verification go mainstream
by worldsayshi on Tuesday December 16, 2025 @06:09PM
(#46295996)
Would it limit a person getting your instructions in Chinese? Tokenisation pretty much means that the LLM is reading symbols instead of phonemes.
This makes me wonder if LLMs works better in Chinese.