BarakWidawsky

You’re mostly right but conflating attention with autoregressive/causal which is the real issue that prevents you from using more compute

You can use diffusion with attention, and this model does in fact use attention

I think it's extremely important to distinguish being friendly (perhaps overly so), and agreeing with the user when they're wrong

The first case is just preference, the second case is materially damaging

From my experience, ChatGPT does push back more than it used to

Looks like the dataset is distilled from Gemini nano-banana

Definitely very useful, but I’m so curious how the original datasets from these image editing models were created. I’m guessing a lot of it is synthetic data to construct scenes programmatically with layers

I’m guessing this is so they optimize processor yields as manufacturing improves

Smaller chips means more of a wafer is usable when a defect exists

I wonder how much of this is due to Diffusion models having less capacity for memorization than auto regressive models

The auto regressive models consistently show better loss for the same number of training tokens

I find a lot of the conclusions compelling but I would’ve loved to see more epochs of training on the 1B model with a 10B dataset, as that model was showing epoch over epoch improvements

It’s interesting that it looks like they didn’t apply their own RL to the model, and instead fine tuned on reasoning traces from large datasets and generating reasoning traces from larger models

If this is a faster backend for LLVM, does it potentially obviate the niche Cranelift is optimizing for?

I have taken a Waymo in the rain before, if they have stopped supporting that as part of the service that’s new, buts it’s definitely within the systems capabilities. It worked great