Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by janalsncm on Monday May 12, 2025 @02:43PM
(#43966245)
I think DeepSeek (v3 and r1) showed us that there’s still a ton of meat on the bone for fundamental research and optimization.
[ Parent
| on HN ]
Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by Lerc on Monday May 12, 2025 @04:38PM
(#43967214)
Absolutely, I have seen so many good ideas that have not yet made it into notable trained models.
A lot of that is because you need to have a lot more faith than "seems like a good idea" before you spend a few million in training that depends upon it.
Some of it is because when the models released now began training, a lot of those ideas hasn't been published yet.
Time will resolve most of that, cheaper and more performant hardware will allow a lot of those ideas to be tested without the massive commitment required to build the leading edge models.
Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by Workaccount2 on Monday May 12, 2025 @04:48PM
(#43967309)
The big guys are almost certainly incinerating millions a day on training "maybe it could show some promise" techniques. With the way things are right now, they are probably green lighting everything to find an edge.