News for hackers. Stuff that matters.

nathan-barry

Re:Can gzip be a language model? on Tuesday June 16, 2026 @12:30PM (#48557841)
LLMs are very good at lossless compression via arithmetic coding. But I didn't know that it was possible to go the reverse direction (do language modeling via a compressor). It's not super great quality, but I'm surprised it worked! Other compression algorithms (like PPMd) use variable n-grams under the hood, and should be much better (although less interesting due to already containing basic language models internally).

Show HN: Tiny Diffusion – Minimal diffusion LM in 364 lines

https://github.com/nathan-barry/tiny-diffusion

I previously posted this a month and a half ago: https://news.ycombinator.com/item?id=45876742I have rewritten it to be much more simple, from 951 lines of code to just 364, with no loss in core functionality or generation quality.I also added a tiny GPT implementation as a comparison (inspired by Andrej Karpathy's code). The two model implementations are ~80% identical, with the core...

( Read More... | no comments | on HN )
Actually NVIDIA made one earlier this year, check out their Fast-dLLM paper
Hey, I’ll add the MIT licenses later today!
Yes, this is the case. During training, the model will get a sequence of text (ex, 512 tokens long) with a percentage of them masked out (with a special <MASK> token). It learns how to unmask those tokens to construct the original text.

In the case that you mentioned, if we had 4 <MASK> tokens in a row, all we are doing for decoding is predicting what those 4 tokens should be.

Generally, this does not seem to be a significant problem, as there are usually multiple ways to express an idea in varying lengths. Also, with confidence-aware parallel decoding, it can usually avoid the scenario you mentioned, as focusing on decoding the highest confident tokens will generally avoid such scenarios with a well trained model.

This is a character-level language diffusion model for text generation. The model is a modified version of Nanochat's GPT implementation and is trained on Tiny Shakespeare! It is only 10.7 million parameters, so you can try it out locally.

Older »