Re:'Attention is all you need' coauthor says he's 'absolutel...
by cauliflower2718 on Friday October 24, 2025 @03:41PM
(#45698353)
+1, I am also big user of PGMs, and also a big user of transformers, and I don't know what the parent comment talking about, beyond that for e.g. LLMs, sampling the next token can be thought of as sampling from a conditional distribution (of the next token, given previous tokens). However, this connection of using transformers to sample from conditional distributions is about autoregressive generation and training using next-token prediction loss, not about the transformer architecture itself, which mostly seems to be good because it is expressive and scalable (i.e. can be hardware-optimized).
Source: I am a PhD student, this is kinda my wheelhouse
Re:'Attention is all you need' coauthor says he's 'absolutel...
by nickpsecurity on Saturday October 25, 2025 @09:04AM
(#45703584)
Don't give up on older stuff just because deep learning went in a different direction. It's a perfect time to recombine the new with the old. I started DuckDuckGoing and found combinations of ("deep learning" or "neural networks") with ("gaussian," "clustering," "support vector machines," "markov," "probabilistic graphical models").
I haven't actually read these to see if they achieved anything. I'm just sharing the results from a quick search in your sub-field in case it helps you PGM folks.
https://arxiv.org/abs/2104.12053
https://pmc.ncbi.nlm.nih.gov/articles/PMC7831091/
And here's an intro for those wondering what PGM is:
As somebody who was a biiiiig user of probabilistic graphical models, and felt kind of left behind in this brave new world of stacked nets, I would love for my prior knowledge and experience to become valuable for a broader set of problem domains. However, I don't see it yet. Hope you are right!