News for hackers. Stuff that matters.
Re:Model Distillation in the API by simonw on Thursday October 31, 2024 @02:54PM (#42010172)
They announced this at DevDay at the beginning of October.

It's effectively a layer of (well needed) sugar on top of their existing fine-tuning mechanism.

The challenge with fine-tuning is collecting a representative dataset to tune against. The tooling they added makes it easy for you to persist your prompts and responses within the OpenAI platform, and then later select those persisted pairs (that were created with e.g. GPT-4o) and use them to fine-tune a cheaper model (like GPT-4o mini) - such that the more expensive model is effectively "teaching" the cheaper model what to do.

You could do this before, but it was a LOT of work. The new "distillation" features make it easier.

Re:Model Distillation in the API by msp26 on Thursday October 31, 2024 @03:18PM (#42010445)
But doesn't this fully lock you into using OpenAI's offerings? If you store the finetuning dataset yourself, you are free to use it on whatever model/provider/self hosting. And I'm finding it fairly straightforward to switch between providers at will. Plus it's nice to do additional programmatic validation on LLM outputs before using them for finetuning (e.g. standardizing country names, calculations).

Am I missing something?

Re:Model Distillation in the API by simonw on Thursday October 31, 2024 @04:08PM (#42010962)
Yes, total lockin.

Remember though, OpenAI and pretty much all of the other providers have things in their terms that roughly say you're not allowed to use their models to train models from other providers - so fine-tuning a model against synthetic data created using OpenAI is likely a terms violation anyway.

Re:Model Distillation in the API by janalsncm on Friday November 01, 2024 @04:08PM (#42021015)
Terms violation or not, it’s a common use case and a huge contributor to their traffic. If they actually enforced it people would move because the marginal quality benefits don’t justify indefinite lockin.