News for hackers. Stuff that matters.

A comment on Running large language models like ChatGPT on a single GPU. Parent comment

Re:Running large language models like ChatGPT on a single GPU by moffkalast on Monday February 20, 2023 @02:48PM (#34872055)
This may be a flawed approach, but an interesting idea would be to use the current models as a preprocessor to generate a huge "labelled" dataset of inputs and outputs, and then using that more accurate and specific data to train a smaller one that would fit.

It likely wouldn't have nearly as much general knowledge since the data just wouldn't be there but the behaviour could be similar?

[ Parent | on HN ]
Re:Running large language models like ChatGPT on a single GPU by zeknife on Monday February 20, 2023 @03:29PM (#34872511)
Look up model distillation
[ Parent | on HN ]
Re:Running large language models like ChatGPT on a single GPU by moffkalast on Monday February 20, 2023 @04:14PM (#34872994)
Ah TIL, so it's actually a thing and with some cleverer approaches too. Then we do have something to look forward to :D
[ Parent | on HN ]