News for hackers. Stuff that matters.
Re:Baba Is Eval by ninjha on Saturday July 05, 2025 @08:14AM (#44472249)
I think I remember one of the original ViT papers saying something about 2D embeddings on image patches not actually increasing performance on image recognition or segmentation, so it’s kind of interesting that it helps with text!
Re:Baba Is Eval by yorwba on Saturday July 05, 2025 @09:45AM (#44472775)
They seem to have used a fixed input resolution for each model, so the learnable 1D position embeddings are equivalent to learnable 2D position embeddings where every grid position gets its own embedding. It's when different images may have a different number of tokens per row that the correspondence between 1D index and 2D position gets broken and a 2D-aware position embedding can be expected to produce different results.