Re:TopoNets: High performing vision and language models with...
by mayukhdeb on Friday January 31, 2025 @09:43AM
(#42888146)
It is indeed brain-like in a functional way. Topographic structure is what enables the brain to have low dimensionality and metabolic efficiency. We find that inducing such structure in neural nets made them have significantly lower dimensionality and also more parameter efficient (After training, we could take advantage of the structure to remove ~80% of the weights in topographic layers without sacrificing performance)
Re:TopoNets: High performing vision and language models with...
by devmor on Saturday February 01, 2025 @02:16AM
(#42896484)
>After training, we could take advantage of the structure to remove ~80% of the weights in topographic layers without sacrificing performance
This is really interesting to me. Is it that the structure clustered the neurons in such a way that they didn't need to be weighted because their function were grouped by similar black box properties?
Re:TopoNets: High performing vision and language models with...
by mayukhdeb on Saturday February 01, 2025 @08:54AM
(#42898282)
> Is it that the structure clustered the neurons in such a way that they didn't need to be weighted
Yep. Because of the structure, we did not have to compute the output of each weight column and simply copied the outputs of nearby weight columns whose outputs were computed.
I know quite a bit about machine learning, but very little to nothing about neuroscience and human cognition, so I am curious how an expert (that didn't work on the paper) would describe it.
(Forgive me for the pre-emptive negativity but I am so utterly exhausted by dishonest comparisons to sapient thought in the field of artificial intelligence that it has nearly drained me of the incredible amount of enthusiasm I used to carry for it.)