Re:The Myth of AI Omniscience: AI's Epistemological Limits
by jstanley on Saturday August 05, 2023 @12:12PM
(#37013447)
It depends which tokenisation you use. Here's the token list for GPT-4: https://incoherency.co.uk/interest/gpt4-token-list.txt
Some of them are words, some of them are not. It uses the same list even if you have read the word2vec paper. It probably doesn't even know whether you've read it.
This is plainly wrong if you read the word2vec paper.