News for hackers. Stuff that matters.

rand0mwalk

Tokens start as a special [MASK] token. Then as the diffusion process runs they are "unmasked" i.e. sampled.

So yes, you define a sequence of [MASK] tokens with some length ahead of time.

In practice, if a model wants to write a shorter sequence, it'll just fill the remaining tokens with empty content. If it wants to write a longer sequence, you'll have to identify this and extend the sequence with more [MASK] tokens. This is typically obvious since there's no "end of sequence" token present if the model wants to generate more.

Re:Schedule-Free Learning – A New Way to Train on Saturday April 06, 2024 @01:43PM (#39954025)
Is there an accompanying paper out there?