Re:Ask HN: Can someone ELI5 Transformers and the “Attention ...
by taylorius on Wednesday May 17, 2023 @02:53PM
(#35979610)
Differentiable in this context, means that it can be learned by gradient descent, which uses derivatives to adjust a neural network model's parameters to minimise some error measure. As for how that applies to a hash function, I think the lookup gives some sort of weighting for each possible output, for each input, with the largest weightings corresponding to a "match". But tbh I'm not certain on that last part...
[ Parent
| on HN ]
Re:Ask HN: Can someone ELI5 Transformers and the “Attention ...
by throwawaymaths on Wednesday May 17, 2023 @03:01PM
(#35979699)
> As for how that applies to a hash function, I think the lookup gives some sort of weighting for each possible output, for each input, with the largest weightings corresponding to a "match"
Note that in general one "matches" more than one thing in which case (IIRC) the result will be a weighted linear combination of the query results.
I do think that this collapses to a datastructure equivalent in function to the "normal hash table" (but not in performance, lol) in the degenerate case where every input yields strictly a single result with zero activation in all other results, so it's not invalid to call it a differentiable lookup table.