News for hackers. Stuff that matters.
Re:Ask HN: What's the best paper you've read in 2020? by mrfox321 on Tuesday December 08, 2020 @04:16PM (#25351443)
I would argue that input scaling is not fundamental to Transformers.

Recurrent neural network size is also independent of input sequence length.

The successful removal of inductive bias is really what differentiates this from previous sequence-to-sequence neural networks.

Re:Ask HN: What's the best paper you've read in 2020? by keithyjohnson on Tuesday December 08, 2020 @08:03PM (#25353973)
Which inductive bias?
Re:Ask HN: What's the best paper you've read in 2020? by nmfisher on Tuesday December 08, 2020 @09:09PM (#25354508)
Presumably that the output at step (n) is conditioned only the output of step (n-1).