News for hackers. Stuff that matters.
Re:The Myth of AI Omniscience: AI's Epistemological Limits by dvt on Saturday August 05, 2023 @12:08PM (#37013405)
> It’s not a mapping of words. It’s a mapping of character sequences.

This is plainly wrong if you read the word2vec paper.

Re:The Myth of AI Omniscience: AI's Epistemological Limits by jstanley on Saturday August 05, 2023 @12:12PM (#37013447)
It depends which tokenisation you use. Here's the token list for GPT-4: https://incoherency.co.uk/interest/gpt4-token-list.txt

Some of them are words, some of them are not. It uses the same list even if you have read the word2vec paper. It probably doesn't even know whether you've read it.