nathan-barry
Can gzip be a language model?
Large Text Compression Benchmark
Language Modeling Without Neural Networks
Quines in Every Programming Language (Rosetta Code)
Mercury Edit 2: Fastest next-edit prediction with a diffusion LLM (221ms)
Noise schedules considered harmful (2024)
Show HN: Diffuji – a diffusion-powered instant camera
Generating Shakespeare Without Neural Networks
Recursion and Induction: A Self-Contained Course Using ACL2
Fidenae amphitheatre collapse (27 AD) – 20k dead
Show HN: Tiny Diffusion – Minimal diffusion LM in 364 lines
https://github.com/nathan-barry/tiny-diffusion
I previously posted this a month and a half ago: https://news.ycombinator.com/item?id=45876742I have rewritten it to be much more simple, from 951 lines of code to just 364, with no loss in core functionality or generation quality.I also added a tiny GPT implementation as a comparison (inspired by Andrej Karpathy's code). The two model implementations are ~80% identical, with the core...
Anchored Diffusion Language Model
Text Diffusion Models Are Faster at Writing Code
Inside ThunderKittens' Python Bindings
Kickstart.nvim: A minimal, single-file starting point for Neovim configuration
How we made AWS Trainium 17x faster (for conv1d)
In the case that you mentioned, if we had 4 <MASK> tokens in a row, all we are doing for decoding is predicting what those 4 tokens should be.
Generally, this does not seem to be a significant problem, as there are usually multiple ways to express an idea in varying lengths. Also, with confidence-aware parallel decoding, it can usually avoid the scenario you mentioned, as focusing on decoding the highest confident tokens will generally avoid such scenarios with a well trained model.