News for hackers. Stuff that matters.
Re:Mercury: Ultra-Fast Language Models Based on Diffusion by thelastbender12 on Monday July 07, 2025 @09:53AM (#44490388)
The speed here is super impressive! I am curious - are there any qualitative ways in which modeling text using diffusion differs from that using autoregressive models? The kind of problems it works better on, creativity, and similar.
Re:Mercury: Ultra-Fast Language Models Based on Diffusion by orbital-decay on Monday July 07, 2025 @10:54AM (#44491026)
One works in the coarse-to-fine direction, another works start-to-end. Which means different directionality biases, at least. Difference in speed, generalization, etc. is less clear and needs to be proven in practice, as fundamentally they are closer than it seems. Diffusion models have some well-studied shortcuts to trade speed for quality, but nothing stops you from implementing the same for the other type.
Re:Mercury: Ultra-Fast Language Models Based on Diffusion by ekunazanu on Monday July 07, 2025 @04:54PM (#44494575)
I once read that diffusion is essentially just autoregression in the frequency domain. Honestly, that comparison didn’t seem too far off.