Re:DeepNet: Scaling Transformers to 1k Layers
by oofbey on Thursday March 03, 2022 @02:28PM
(#30545550)
Oh yeah, I think you're right. If it's all fully-connected (and parts of a transformer are) then more thinner layers use fewer FLOPs than fewer thicker layers, for the same number of parameters. As long as the layers are wide enough to keep the GPU busy it'll run faster.
[ Parent
| on HN ]
There are no comments on this comment yet.