Yeah I remember in undergrad I was working on using transformation learning to train an object detector. Basically you only needed 100ish images to get the model to detect that new object really well.

I'm not sure what the analogous term is for a similar process on LLMs, but that will be huge when there is a service for it.

LLMs can do that without any examples (zero shot) or with one or a few demonstrations in the prompt, if you can describe the task in the limited context window.

If you want for example to train the model to learn to use a very large API, or access the knowledge in a whole book, it might need fine-tuning.

Could I just train a very small LLM with an English dictionary + Python + large API documentation + large Python code base?

Then do some chat fine tuning (like what HF did with StarCoder to get ChatCoder)

And get a lightweight LLM that knows the docs and code for the thing I need it for

After that, maybe incrementally fine tune the model as part of your CI/CD process

How similar were the object to other objects?

E.g., were you trying to distinguish an object vs nothing, a bicycle vs a fish, a bird vs a squirrel, or two different species of songbird at a feeder?

How much would the training requirements increase or decrease moving up or down that scale?

The PaLM 2 stuff released yesterday has fine tuning for their newest large models as a core feature.

Open-source LLM projects have largely solved this using Low-Rank Adaptation of Large Language Models (LoRA): https://arxiv.org/abs/2106.09685

Apparently an RTX 4090 running overnight is sufficient to produce a fine-tuned model that can spit out new Harry Potter stories, or whatever...

How much would they cost aprox per training?

What would the quality of the model be, compared to what Karpathy uses on his video “let’s build gpt from scratch”?

In that video, he builds a decoder-only transformer model, that learns Shakespeare from 1MB of data and trains in 15min

These just fine-tune existing models, so the cost is single digit dollars. Whatever it costs in electricity to run a desktop GPU for a day or a few days.

Just use a machine learning method other than an LLM. If you're going to go to the effort of fine tuning and training, pay people to collect and label data like we used to do before Feb 2023.

If your main concern is question answering or summarization or code completion there are plenty of ways to do that now. If you really require the advanced emergent properties of LLMs, you'll have to work with a company that can afford to train a transformer on the Entire Internet.