News for hackers. Stuff that matters.
Re:Sutskever: OpenAI board doing its mission to build AGI th... by stevenhuang on Saturday November 18, 2023 @12:08AM (#38315544)
Ah the classic "LLMs are limited bc they can't do character level manipulations".

Buddy these are deliberate optimization trade-offs in the tokenizer, LLMs don't even see individual characters. Talk about a red herring.

I highly suggest a more refined understanding of LLMs before emphatically stating what they are or aren't. It's frankly embarrassing.

Re:Sutskever: OpenAI board doing its mission to build AGI th... by oivey on Saturday November 18, 2023 @01:41AM (#38316237)
It’s a performance critical optimization that has downsides. It’s a completely fair criticism.

It’s like someone claimed a hammer is a universal construction tool, and then someone else pointed out hammers aren’t great for screwing in screws. Your position is that a universal construction tool doesn’t need to work with screws because hammers were never designed to do so.

Re:Sutskever: OpenAI board doing its mission to build AGI th... by stevenhuang on Saturday November 18, 2023 @04:20AM (#38317312)
A hammer is a finished product.

LLMs as a technology are in their infancy and see groundbreaking developments every month.

There are models trained on character level tokens that handle precisely these issues.

https://arxiv.org/abs/2311.06158

https://www.reddit.com/r/LocalLLaMA/comments/17xj8wl/trainin...

The fact LLMs can't currently handle character-level changes tell us nothing about whether or not they truly understand meaning.

The parent commentator is not probing the right things to answer his question.