News for hackers. Stuff that matters.
Re:Ask HN: What's the best paper you've read in 2020? by tehsauce on Tuesday December 08, 2020 @04:26PM (#25351577)
“it can detect patterns among an O(n) input size without requiring an O(n^2) size neural net”

This might be misleading, the amount of computation for processing a sequence size N with a vanilla transformer is still N^2. There has been recent work however which has tried to make them scale better.

Re:Ask HN: What's the best paper you've read in 2020? by m3at on Tuesday December 08, 2020 @08:16PM (#25354082)
You raise an important point. The proposed solutions are too many to enumerate, but if I had to pick just one currently I would go for "Rethinking Attention with Performers" [1]. The research into making transformer better for higher dimensional inputs is also moving fast and is worth following.

[1] https://arxiv.org/abs/2009.14794

Re:Ask HN: What's the best paper you've read in 2020? by thomasahle on Tuesday December 08, 2020 @06:21PM (#25352962)
n^2 time yes, but not n^2 weights in the net.