Fine Tuning Language Models

Re:Fine Tuning Language Models by brucethemoose2 on Monday July 03, 2023 @09:55PM (#36581223)

Most AI runners just ignore licensing and (illegally?) run LLaMA finetunes. There are tons of good 33B/65B instruct models you can run on any decent PC.

But if you want to avoid the non commercial LLaMA license, you have 3 good options for a base model.

- OpenLlama 13B

- MPT 30B

- Falcon 40B

Of these, Falcon 40B is very difficult to run (slow in 4 bit, basically requires a professional GPU, no good cpu offloading yet).

OpenLLaMA 13B only supports a context size of 2048 as of today... But that could change soon.

So you probably want MPT instruct 30B, specifically this one:

https://huggingface.co/TheBloke/mpt-30B-instruct-GGML

As the page says, you can try it out on a decent PC of your own with the OpenCL build of KoboldCPP. Change it to "instruct" mode, use the template on the page, offload as many layers as you can to your PC's dGPU, and run it in instruct mode. It may already work for your summarization needs.

If not, you can finetune it with MPT's code and summarization data.

https://github.com/mosaicml/llm-foundry

Or train OpenLLaMA 13B with SuperHOT + summarization data using QLORA.

[ Parent | on HN ]

Re:Fine Tuning Language Models by brucethemoose2 on Monday July 03, 2023 @10:07PM (#36581313)

*But if your input prompts are short, an OpenLLaMA finetune like this would work:

https://huggingface.co/TheBloke/open-llama-13b-open-instruct...

[ Parent | on HN ]