News for hackers. Stuff that matters.
Re:The Transformer Family by sebzim4500 on Sunday January 29, 2023 @06:20PM (#34573863)
>My point is that "more data" here simply represents the mental effort that has already been exerted in pre-AI/DL era, which we're now capitalizing on while we can

Not really. It's not simply that modern architectures are not adding additional inductive biases, they are actively throwing away the inductive bias that used to be used by everyone. For example, it was taken for granted that you should use CNNs to give you translation invariance, but apparently now visual transformers can match that performance with the same amount of compute.

Re:The Transformer Family by 0xBABAD00C on Sunday January 29, 2023 @07:56PM (#34574556)
I don't think what you're saying contradicts what I'm saying. My baseline / reference point wasn't CNNs.
Re:The Transformer Family by jerpint on Monday January 30, 2023 @06:50AM (#34578536)
There are bit caveats here:

Vision transformers outperform CNNs in HUGE data regimes. On small datasets, CNNs still shine.

Also, if you take a CNN with modern tricks, they can be on par with vision transformers, e.g convnext

Transformers really dominate when you scale the amount of data to infinity

Re:The Transformer Family by alexmolas on Tuesday January 31, 2023 @08:44AM (#34594893)
Yes, that's what I was saying: scale > inductive bias. Apparently, scaling is all you need ;)