Biological Function Emerges from Unsupervised Learning on 250M Protein Sequences

Re:Biological Function Emerges from Unsupervised Learning on... by lucidrains on Tuesday April 30, 2019 @01:40PM (#19790122)

Language, music, and now amino acid sequences. Attention is all you need.

[ Parent | on HN ]

Re:Biological Function Emerges from Unsupervised Learning on... by mfatica on Tuesday April 30, 2019 @02:25PM (#19790573)

I would say you also need a fair bit of data too...

[ Parent | on HN ]

Re:Biological Function Emerges from Unsupervised Learning on... by bearmcbearsly on Tuesday April 30, 2019 @02:46PM (#19790729)

Well, yes. But I think lucidrains was referring to:

https://arxiv.org/abs/1706.03762

[ Parent | on HN ]

Re:Biological Function Emerges from Unsupervised Learning on... by return1 on Tuesday April 30, 2019 @05:38PM (#19792227)

and transformers

[ Parent | on HN ]

Re:Biological Function Emerges from Unsupervised Learning on... by nl on Wednesday May 01, 2019 @06:27AM (#19795867)

The Attention Is All You Need paper is where Transforms were introduced:

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

[ Parent | on HN ]

Re:Biological Function Emerges from Unsupervised Learning on... by return1 on Thursday May 02, 2019 @12:06PM (#19809388)

Yup, and the state of the art BERT and gpt-2 are both based on transformers.

[ Parent | on HN ]