Re:CoLT5: Faster Long-Range Transformers With Conditional Co...
by f_devd on Monday March 20, 2023 @08:33PM
(#35240930)
This still isn't technically dynamic allocation since it always takes a top-k (constant k) tokens from the sequence, so more like dynamic routing, which was explored in Mixture-of-Expert models but only in Feed-Forward blocks and with a different routing scheme.
[ Parent
| on HN ]
There are no comments on this comment yet.