News for hackers. Stuff that matters.
Re:'Attention is all you need' coauthor says he's 'absolutel... by eli_gottlieb on Friday October 24, 2025 @03:58PM (#45698495)
> probabilistic graphical models- of which transformers is an example

Having done my PhD in probabilistic programming... what?

Re:'Attention is all you need' coauthor says he's 'absolutel... by pishpash on Friday October 24, 2025 @06:45PM (#45699860)
It's got nothing to do with PGM's. However, there is the flavor of describing graph structure by soft edge weights vs. hard/pruned edge connections. It's not that surprising that one does better than the other, and it's a very obvious and classical idea. For a time there were people working on NN structure learning and this is a natural step. I don't think there is any breakthrough here, other than that computation power caught up to make it feasible.
Re:'Attention is all you need' coauthor says he's 'absolutel... by dekhn on Friday October 24, 2025 @07:35PM (#45700164)
I was talking about things inspired by (for example) hidden markov models. See https://en.wikipedia.org/wiki/Graphical_model

In biology, PGMs were one of the first successful forms of "machine learning"- given a large set of examples, train a graphical model using probabilities using EM, and then pass many more examples through the model for classification. The HMM for proteins is pretty straightforward, basically just a probabilistic extension of using dynamic programming to do string alignment.

My perspective- which is a massive simplification- is that sequence models are a form of graphical model, although the graphs tend to be fairly "linear" and the predictions generate sequences (lists) rather than trees or graphs.