News for hackers. Stuff that matters.
Re:Diffusion Without Tears by 3vidence on Friday February 14, 2025 @04:14PM (#43053136)
I have an undergraduate degree in Math and generally find math quite enjoyable, however, even I had a pretty hard time grasping most of what this blog was talking about. I think it pretty quickly went from extremely high level (add noise to image, then remove noise) to extremely low level specific.

Also I had a hard time figuring out which part of the equation differentiate the equations for different data points, is that what the meaning of "theta" is in all the equations? To guide the initial noise towards one type of image instead of another is that what theta is responsible for? Is the innovation in GenAI images to use text embeddings to create the theta?

I definitley feel some tears coming on.

Re:Diffusion Without Tears by Sharlin on Friday February 14, 2025 @04:41PM (#43053375)
Yeah, the article could definitely use some clarification in many parts (the author may be suffering from the curse of knowledge a bit). Plus there's the fact that even if you know your ODEs, SDEs are a different beast bringing in probability (certainly one may not be accustomed to seeing `p(x|y)` in the middle of a differential equation…)
Re:Diffusion Without Tears by sl8r on Saturday February 15, 2025 @03:40AM (#43056967)
Hey, I’m the post author — thanks for reading.

Theta represents all the model params — all the weights in the neural network. The convention is to write theta for the “learned” score function and omit theta for the “true” score function.

Re:Diffusion Without Tears by huqedato on Friday February 14, 2025 @04:10PM (#43053098)
Without tears, right ? Seeing those mountains of cryptographic math I've got red eyes.
Re:Diffusion Without Tears by russdill on Friday February 14, 2025 @08:24PM (#43054933)
There's a teaching method called "handwriting without tears". The title is almost certainly a play on that.
Re:Diffusion Without Tears by sl8r on Saturday February 15, 2025 @03:47AM (#43057001)
The production of tears is left as an exercise to the reader. /s

Thanks for reading. The 2D simulation section might be more interesting on a first read — it makes the math less mysterious, I hope!

Re:Diffusion Without Tears by gnatolf on Friday February 14, 2025 @04:32PM (#43053288)
As a material scientist, the roller coaster to figure out what 'diffusion' we're talking about here was surprisingly funny.

Means: maybe edit that title a bit :)

Re:Diffusion Without Tears by derbOac on Friday February 14, 2025 @05:56PM (#43053980)
Yeah there's lots of "diffusions" out there and I was similarly curious which one they were going to be writing about.

I liked the post though.

Re:Diffusion Without Tears by CamperBob2 on Friday February 14, 2025 @04:37PM (#43053340)
TL,DR: "Now time-reverse the stochastic differential equation to infer the rest of the fucking owl."

Thanks, guys, super helpful.

Re:Diffusion Without Tears by onecommentman on Friday February 14, 2025 @09:44PM (#43055402)
At least the article warns you upfront of the sort of mathematical sophistication required to get some of the explanations. The Author is a financial engineering sort, so their big thing is SDEs and assumes (for some of the explanations) that you bring that sort of intuition with you. If the Author was a signal processing type, they might use Kalman filter analogies, or pure statisticians would cite autocorrelation.

Don’t try to catch all the mathematical Pikachus in the paper, just choose the insights that resonate with you. Thankfully, there isn’t a pop quiz lurking at the end.

In honor of the Bay Area roots of HN, “believe it if you need it, if you don't, just pass it on”. I liked the paper even when skipping the SDE material.

Re:Diffusion Without Tears by sl8r on Saturday February 15, 2025 @03:54AM (#43057032)
I left the TENET references on the cutting room floor.

I too found it really surprising that the reverse-time equation has a simple closed form. Like, surely breaking a glass is easier than unbreaking it? That’s part of what got me interested in this stuff in the first place!

If you haven’t seen it yet, highly recommend the blogs of Sander Dieleman & Yang Song (who co-invented the SDE interpretation).

Re:Diffusion Without Tears by getnormality on Friday February 14, 2025 @05:28PM (#43053807)
I've noticed that when finance guys learn that there's a really useful AI thing called diffusion, they get all excited and start writing stochastic differential equations and drawing 1-D Brownian motion plots all over the place. It's not yet clear to me whether this helps anyone understand AI diffusion models, but they seem to enjoy it a lot.
Re:Diffusion Without Tears by markisus on Friday February 14, 2025 @05:39PM (#43053861)
If by helping anyone you mean helping the math challenged (which is technically almost everyone) then I would be inclined to agree.

But to the quant crowd I’m guessing that couching diffusion in the language of stochastic calculus is helpful.

Re:Diffusion Without Tears by patrick451 on Friday February 14, 2025 @09:01PM (#43055136)
I'm a controls guy, not a quant guy but I've found the SDE perspective and this blog post to be incredible in helping me understand how a diffusion model works.
Re:Diffusion Without Tears by currymj on Friday February 14, 2025 @09:32PM (#43055323)
it makes intuitive sense, but at the same time, you can get good performance replacing the Gaussian noise with deterministic transformations.

so in the end diffusion can’t explain that much about how diffusion models work. disappointingly, so many modern ML things are like this.

Re:Diffusion Without Tears by sl8r on Saturday February 15, 2025 @03:37AM (#43056955)
Hey! I’m the post author. Highly recommend Sander Dieleman’s blog for alternative interpretations https://sander.ai/2023/07/20/perspectives.html

I personally find the SDEs the most intuitive, and the deterministic ODE / consistency models / rectified flow stuff as ideas that are easier to understand after the SDEs. But not everyone agrees!

Re:Diffusion Without Tears by currymj on Saturday February 15, 2025 @10:53AM (#43059385)
Thanks for sharing this! I tend to agree that it’s easiest to understand this way.

I just find it a frustrating fact about modern machine learning that in fact, the nice SDE interpretation is somehow not the “truth”, it’s just a tool for understanding.

Re:Diffusion Without Tears by NBJack on Friday February 14, 2025 @06:18PM (#43054125)
Oof. I honestly can't tell if the author is joking at times. Gave up on reading it.
Re:Diffusion Without Tears by crystal_revenge on Friday February 14, 2025 @06:21PM (#43054157)
A much simpler exploration of this topic I always liked is the "Linear Diffusion" [0] example which implements a basic Diffusion model using only linear components. Given it's simplicity it gets surprisingly good results at generating digits.

0. https://www.countbayesie.com/blog/2023/4/21/linear-diffusion

Re:Diffusion Without Tears by dandanua on Saturday February 15, 2025 @05:02AM (#43057352)
Awesome tutorial, thanks!