Model Distillation in the API

Re:Model Distillation in the API by patelajay285 on Thursday October 31, 2024 @02:04PM (#42009589)

We've been working on a Python framework where one of the use cases is easy distillation from larger models to smaller open-source models and smaller-closed source models (where you don't have to still use / pay for the closed-source API service): https://datadreamer.dev/docs/latest/

Here's an (now slightly outdated) example of OpenAI GPT-4 => OpenAI GPT-3.5: https://datadreamer.dev/docs/latest/pages/get_started/quick_...

But you can also do GPT-4 to any model on HuggingFace. Or something like Llama-70B to Llama-1B.

For some tasks, this kind of distillation works extremely well given even a few hundred examples of the larger model performing the task.

[ Parent | on HN ]

Re:Model Distillation in the API by bangaladore on Thursday October 31, 2024 @02:43PM (#42010066)

> OpenAI GPT-4 => OpenAI GPT-3.5

I'm confused why you are mentioning 3.5 here. The weights aren't public, so you aren't actually running any derivative of GPT-3.5

Or am I mistaken. Can you clarify?

[ Parent | on HN ]

Re:Model Distillation in the API by Tiberium on Thursday October 31, 2024 @03:10PM (#42010352)

> distillation from larger models to smaller open-source models and smaller-closed source models

They don't limit it only to open-source models. And you can finetune 3.5 Turbo on OpenAI API.

[ Parent | on HN ]