News for hackers. Stuff that matters.
Re:The Myth of AI Omniscience: AI's Epistemological Limits by jstanley on Saturday August 05, 2023 @12:12PM (#37013447)
It depends which tokenisation you use. Here's the token list for GPT-4: https://incoherency.co.uk/interest/gpt4-token-list.txt

Some of them are words, some of them are not. It uses the same list even if you have read the word2vec paper. It probably doesn't even know whether you've read it.

There are no comments on this comment yet.