News for hackers. Stuff that matters.

A comment on Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences. Parent comment

Re:Biological Function Emerges from Unsupervised Learning on... by return1 on Tuesday April 30, 2019 @05:38PM (#19792227)
and transformers
[ Parent | on HN ]
Re:Biological Function Emerges from Unsupervised Learning on... by nl on Wednesday May 01, 2019 @06:27AM (#19795867)
The Attention Is All You Need paper is where Transforms were introduced:

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

[ Parent | on HN ]
Re:Biological Function Emerges from Unsupervised Learning on... by return1 on Thursday May 02, 2019 @12:06PM (#19809388)
Yup, and the state of the art BERT and gpt-2 are both based on transformers.
[ Parent | on HN ]