Re:Transformers struggle with generalizing tasks beyond pre-...
by og_kalu on Tuesday November 07, 2023 @12:49PM
(#38180193)
Damn I linked the wrong paper lol.
Meant to link this. More pertinent to my point on Tokenization https://arxiv.org/abs/2310.02989
If this is true, I suppose based on what we know of the curse of dimensionality it makes sense. Very thought-provoking work.