News for hackers. Stuff that matters.
Re:Generative Language Modeling for Automated Theorem Proving by visarga on Wednesday September 09, 2020 @12:01PM (#24422410)
This shows transformers are capable of generating more than plain text.

Transformers are probably the most general architecture to date, they can handle plain text, images, math and speech, but the penalty is that they need lots of training data and compute.

Re:Generative Language Modeling for Automated Theorem Proving by grandmczeb on Wednesday September 09, 2020 @04:38PM (#24425655)
Most transformer models require that the data is euclidian (e.g. a sequence or image). Non-Euclidean deep learning methods like graph neural networks (GNNs) are much more general and can accept almost arbitrarily structured data (social networks, point clouds, trees, etc). You can actually model transformers as a special case of GNNs[1].

[1] https://towardsdatascience.com/transformers-are-graph-neural...

Re:Generative Language Modeling for Automated Theorem Proving by bollu on Wednesday September 09, 2020 @07:18PM (#24427090)
All smooth manifolds can be realised as submanifolds of euclidean space. This is the famous Nash embedding theorem: https://en.wikipedia.org/wiki/Nash_embedding_theorem

Hence, it's unclear if the power gained by GNNs is actually 'useful'.

Re:Generative Language Modeling for Automated Theorem Proving by grandmczeb on Wednesday September 09, 2020 @08:31PM (#24427628)
What do you mean by useful? GNNs get better performance than other methods on lots of real tasks. It's the same as CNNs being useful in practice despite not having more "theoretical" power than MLPs.
Re:Generative Language Modeling for Automated Theorem Proving by lostmsu on Friday September 11, 2020 @01:24PM (#24445080)
Which tasks where GNNs get better performance have standardized benchmarks?
Re:Generative Language Modeling for Automated Theorem Proving by riku_iki on Wednesday September 09, 2020 @12:34PM (#24422767)
> but the penalty is that they need lots of training data and compute.

I think it depends on how noisy and unstructured data is (e.g. human language), transformer will likely approximate few well correlated datapoints just fine.

Re:Generative Language Modeling for Automated Theorem Proving by nestorD on Wednesday September 09, 2020 @02:44PM (#24424290)
Here the train set was only 1000 proofs (with an average of 90 steps per proof) so it is decidely a small dataset (but highly structured and noiseless).
Re:Generative Language Modeling for Automated Theorem Proving by spolu on Wednesday September 09, 2020 @02:46PM (#24424306)
(30k proofs to be precise, valid and test are each 1k)