News for hackers. Stuff that matters.
Re:NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute by ACCount37 on Thursday March 19, 2026 @05:37PM (#47446596)
There's "cheap" bulk data - simple synthetics, unfiltered scrapes. Used for pre-training, especially early pre-training. And then there's "expensive" data. Human domain expert solutions, made by people you hire for $100 an hour. Used for SFT.

For "expensive" data, it makes a lot of sense to use every trick in the book to squeeze that data for all its worth.

There are no comments on this comment yet.