selim-now

[edit] have you tried with a smaller model? I'm afraid a 3B model will brick my rusty laptop

sorry, missed that part, the accuracy drop-off isn't that bad!

where you considering fine-tuning the SLM as well?

let's connect, I can hook you up with a custom SLM -> dm on X

isn't that the premise of the Nvidia paper? https://arxiv.org/pdf/2506.02153

If the cost of getting the model is $200, then the cost of the trade-off seems to be quite clear.

You are right that the labor is a factor, unless you use a platform like https://www.distillabs.ai/ then the process is automated. (I'm affiliated)

interesting, I would argue that fine-tuning makes sense especially in cases where you want to narrow down a small model to a single task – in this case you can get the most bang-per-parameter in a way, using a small model that performs very well in a very narrow space.

well, fine-tuning is possible on consumer hardware, the problem is that it would be slow and that you're limited in the size of the dataset you can use in the process.

In case you would want to follow the approach in this paper and synthetically augment a dataset – using an LLM for that (instead of a smaller model) just makes sense and then the entire process cannot be easily run on your local machine.

That would definitely make the evaluation more robust. My fear is that with LLMs at hand people became allergic to preparing good human-labelled evaluation sets and would always to some degree use an LLM as a crutch.

yes! check out https://distillabs.ai/ – follows a similar approach except the evaluation set is held out before the synthetic data generation, which I would argue makes it more robust (I'm affiliated)