Re:Reverse Engineering Self-Supervised Learning
by mjburgess on Monday June 05, 2023 @12:01PM
(#36198282)
Sure, I mean naive-wrt-semantics; not naive-wrt-statistics.
The naive statistical approach fails, the regularisation approach succeeds.
I'd say this does need explanation, but the paper really omits exactly what needs explaining... it isnt so much the introduction of regularisation...
Rather, it's why-it-should-be that our data generating processes encode metadata which can be capture by such regularisation
Phrased like this, i htink the answer is somewhat obviuos
They seem to be investigating the intermediate representations of each layer in deep networks, and the degree to which those representations can linearly separate classes. Not "naive" statistical compressions, but a technique called UMAP is used to generate 2d scatter plots of the high dimensional representations.
(As I understood it).