Re:Unlimiformer: Long-Range Transformers with Unlimited Leng...
by vardhanw on Saturday May 06, 2023 @08:43AM
(#35840815)
Could you explain in simple terms (if possible) what is the similarity? For context, I worked for a brief time with ART before Y2K in BU CNS, and took a few courses there, but had to leave it for 'reasons'.
[ Parent
| on HN ]
Re:Unlimiformer: Long-Range Transformers with Unlimited Leng...
by intalentive on Saturday May 06, 2023 @02:35PM
(#35844072)
Some ART concepts
1) Normalize input (batch norm, 2015)
2) Competitive dynamics / lateral inhibition (softmax in attention layers, 2017)
3) Cluster best matching activation vectors (top-k keys, 2023)