Re:Show HN: Tiny Diffusion – A character-level text diffusio...
by rand0mwalk on Friday November 14, 2025 @03:15PM
(#45931649)
Tokens start as a special [MASK] token. Then as the diffusion process runs they are "unmasked" i.e. sampled.
So yes, you define a sequence of [MASK] tokens with some length ahead of time.
In practice, if a model wants to write a shorter sequence, it'll just fill the remaining tokens with empty content. If it wants to write a longer sequence, you'll have to identify this and extend the sequence with more [MASK] tokens. This is typically obvious since there's no "end of sequence" token present if the model wants to generate more.
So yes, you define a sequence of [MASK] tokens with some length ahead of time.
In practice, if a model wants to write a shorter sequence, it'll just fill the remaining tokens with empty content. If it wants to write a longer sequence, you'll have to identify this and extend the sequence with more [MASK] tokens. This is typically obvious since there's no "end of sequence" token present if the model wants to generate more.