Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by joewferrara on Sunday September 24, 2023 @04:55PM
(#37637104)
They test two fine tuning tasks in the article - reliable output formatting and custom tone. These are two tasks (reliable output formatting in particular) that are advertised regularly as areas where fine tuning an LLM should work. The goal is not to change what the LLM knows, but to change how the LLM communicates what it knows. In theory the user wants to leverage the LLMs knowledge base and using the different output format is more useful to the user.
The hard question IMO is the question of when does it make sense to fine tune an LLM to update it's knowledge and how much data is needed in this case? I have not seen anyone show a real example of succeeded in this case and wonder if it's close to as difficult as training the LLM from scratch or if it's a feasible fine tuning use case.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by ozr on Sunday September 24, 2023 @05:24PM
(#37637345)
Fwiw, unpublished testing on LLaMA-1 13B showed that it was able to learn a new word and it's meaning via PEFT with <50 examples. Finetuning can unquestionably add new data to a model.
Jeremy Howard has written a bit about how quickly LLMs can pick up new concepts as well:
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by mikeagb on Sunday September 24, 2023 @06:09PM
(#37637652)
The question of how to fine-tune to teach LLMs facts/knowledge is definitely something we're interested in exploring more in future work. The common opinion seems to me at least to be that fine-tuning is more to teach the model how to use the knowledge it already has to complete a specific task rather than to instill new knowledge, and that RAG should be sued to provide more specific context. However, I personally believe there is potential in fine-tuning for "memorization" or learning, and am excited to see new developments in the field.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by dnnssl2 on Sunday September 24, 2023 @06:30PM
(#37637795)
Knowledge instillation is probably the holy grail of fine tuning. The hard part is:
1. Generalizing new facts. You can create a question answer pair of: “what is the population of the world in 2023?” “8 billion”, but it may not be able to pick up alternate phrasing or “does the world have 8 billion people on it?”
2. Catastrophic and behavioral forgetting. Continued fine tuning after RLHF and instruction fine tuning may result in the loss of the alignment and instruction following capabilities trained by OpenAI. At worst, it will start spewing random tokens like the example in the post.
I have not yet seen it successfully done, and I suspect that updating fractions (~.1%) of the original weights with PEFT methods won’t help.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by BoorishBears on Sunday September 24, 2023 @06:37PM
(#37637845)
Your answer is not really answering and is liable to confuse someone asking the question this person asked... the answer to their question is a simple: No.
Current fine tuning techniques can only contribute to knowledge indirectly (getting better queries for an external data source for example), you cannot directly embed new facts in the model is any generally efficient/effective manner.
There are toy examples of fine tuning in facts that are not of use outside of academic considerations at this point, and I sense it's contributing to the widespread confusion about fine-tuning's value proposition
I don’t believe that the answer is strictly no. There are still many questions around the fine tuning method and the scale of data, as well as expectations of task accuracy from the perspective of an end user.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by mikeagb on Sunday September 24, 2023 @07:07PM
(#37638047)
I agree that stating outright that the answer is no is a bit too strong of a statement. The general consensus has definitely been that fine-tuning (especially instruction fine-tuning) is primarily to pick up style over facts, but that doesn't mean it's not doable. Continuous pre-training is used to instill new knowledge, and the line where it becomes "fine-tuning" rather from "continuous pre-training" is not obvious to me.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by joewferrara on Monday September 25, 2023 @01:52PM
(#37647875)
This line between fine tuning and continuous pre-training is what I’m interested in. What is the investment difference between fine tuning, contoured-training and training from scratch? Do you (or anyone else) have any good sources or know of good examples where continuous pre-training is being done?
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by BoorishBears on Sunday September 24, 2023 @07:09PM
(#37638057)
> There are toy examples of fine tuning in facts that are not of use outside of academic considerations at this point, and I sense it's contributing to the widespread confusion about fine-tuning's value proposition
The answer for someone asking that question is a strict no. Many people asking this stuff only have access to SFT, so it's a super no for them.
Honestly I don't get this weird obsession right now with LLMs and throwing random roadblocks in any sort of common knowledge of the subject. If someone in CS 101 asked if they could write a game engine in CSS you wouldn't get people lining up to tell them the answer isn't "No." despite it technically being possible (https://github.com/brookjordan/css-game-engine) because we understand that sometimes to enable understanding of a subject you need to setup some solid ground for new entrants to stand on.
Fine-tuning is not for knowledge. If you get comfortable enough to start experimenting with that application, you'll understand that there's some nuance to that statement either way and get to research/tinker/push boundaries armed with enough knowledge to not accept the simple no.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by Phemist on Monday September 25, 2023 @03:32AM
(#37640599)
Funny, as a corollary to how impractical a game engine would be in CSS: my first pass over the text had me reading CSS as something like C Sharp Sharp. A non-existent language that, according to my brain, still seems like a more likely language to build the game engine in
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by ozr on Sunday September 24, 2023 @08:29PM
(#37638457)
This is simply wrong. Information that did not exist in the model can be added to a model by finetuning, both full and PEFT. It has been repeatedly demonstrated in practice and in multiple papers.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by BoorishBears on Sunday September 24, 2023 @08:40PM
(#37638509)
That is simply noise.
Surely you saw the sibling comment that tries to make the exact same point and did so hours ago, the reply to them is the same to you:
> There are toy examples of fine tuning in facts that are not of use outside of academic considerations at this point, and I sense it's contributing to the widespread confusion about fine-tuning's value proposition
The answer for someone asking that question is a strict no. Many people asking this stuff only have access to SFT, so it's a super no for them.
Honestly I don't get this weird obsession right now with LLMs and throwing random roadblocks in any sort of common knowledge of the subject. If someone in CS 101 asked if they could write a game engine in CSS you wouldn't get people lining up to tell them the answer isn't "No." despite it technically being possible (https://github.com/brookjordan/css-game-engine) because we understand that sometimes to enable understanding of a subject you need to setup some solid ground for new entrants to stand on.
Fine-tuning is not for knowledge. If you get comfortable enough to start experimenting with that application, you'll understand that there's some nuance to that statement either way and get to research/tinker/push boundaries armed with enough knowledge to not accept the simple no.
_
It's no different than teaching kids the Bohr model of the atom: we know it doesn't hold up to discoveries that you'll come across after it is established, but it doesn't matter because by the time you know enough to revisit the topic, you understand why the answer was a flat no then and can move past it on your own.
OP could have googled the topic but they asked human beings the question. They likely presumed they'd use their human sensibilities to understand the underlying intent of the question instead of parroting a list of toy experiments that would have zero benefit to them.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by zwaps on Monday September 25, 2023 @01:05AM
(#37639835)
Having literally done it in an enterprise setting (and participated in experiments for some of the largest companies in the world in their respective domain fields), I have to say: your lack of nuance and abundance of arrogance does not come across very well.
It is important to distinguish between something being impossible, infeasible and not well understood. Fine-tuning "for effect" is mostly the latter.
You say "current fine-tuning techniques can only contribute to knowledge indirectly" and then in the next post row back to "except in toy examples" because the former is - literally - not correct.
This is HN. We are not advising clients on how "to get their data into their AI best". We can discuss here the actual technical detail of a thing. An intellectually honest discussion begins with saying: "From a scientific standpoint, and even from a practical standpoint, we are not sure yet, however..."
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by BoorishBears on Monday September 25, 2023 @01:58AM
(#37640085)
"advising clients" is such an odd way of describing "making a complex topic approachable"
But you're correct, this is HN: so much pontificating without producing a single counterfactual implies you should speak for yourself and not the collective.
They said "LLM", but given the context it's an RLHF LLM, and presumably they want a generalized way to add factual information in a way that doesn't cripple the model's general performance (yes, I am being so arrogant as to draw obvious conclusions to give them a useful answer)
No paper on the subject has achieved this, the ones that come close (and by close I mean very far) fall back to BERT sized models which I already addressed below: so please petition your "enterprise" to share their secrets
(wrong crowd to get any gravitas out of the word enterprise btw, we understand it means "constrained usecase with minimal external validation")
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by redox99 on Monday September 25, 2023 @02:01AM
(#37640105)
> I have not yet seen it successfully done, and I suspect that updating fractions (~.1%) of the original weights with PEFT methods won’t help.
Nitpick, but although when training LoRAs you're only training 1% or less (depending on rank) of the number of parameters of the entire model, the adapters affect the entire model and after merging the LoRA all of the weights of the model are updated.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by jawerty on Monday September 25, 2023 @01:57AM
(#37640082)
A lot of it is based on what you’re aiming to generalize. Typically for data that is fairly “conceptual” I find a DPR type solution to be better for most people with a similarity search. But if you want to generalize on style/data extraction/basic pattern matching fine tuning is incredible. I talk through some of these concepts on a fine tuning llama 2 live stream I did a couple months ago https://www.youtube.com/live/TYgtG2Th6fI?si=7VVv0Qooe2o1wuzD
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by tomohelix on Sunday September 24, 2023 @06:27PM
(#37637768)
Is this something like short term vs long term memory? The context window for LLMs is its short term memory where you can tell it to do things or quickly define something and the LLM can learn very quickly even with just 1 example or a sentence. But it forgets immediately once the work is done. But for finetuning, it commits the knowledge into its weight network and have a "deeper" understanding? The cost is it takes more effort and energy to do so?
If so, let say in the future, we have an LLM with 100K token context windows but with a subsystem where it notices some knowledge keeps being repeated in the context and then store that knowledge for finetuning when the LLM is not doing inference. Basically a mirror of the way we human work? Is that possible? An LLM that constantly improved and can adapt to new knowledge?
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by BoorishBears on Sunday September 24, 2023 @06:33PM
(#37637815)
Fine tuning is mostly useless for direct addition of knowledge.
You can use it to improve knowledge in indirect ways:
- get the model better at crafting queries for an external data source
- get the model better a tool usage to do computation with an external system
- get more useful embeddings from BERT/SBERT
- tell the model what it cannot answer accurately
But in general, fine tuning is noise right now because 99% of the people chasing it actually don't need it.
If you want to change how the model presents text, use fine tuning. If you want to change what the model can present, fine tuning is a hopeless way of doing it.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by BoorishBears on Sunday September 24, 2023 @10:49PM
(#37639185)
Put the information in the context window of the model.
If you're answering questions on current events, hook it up to a news API and let it take in a user query, return a search to you, and give the result back to the LLM before it responds.
Right now LLMs perform best for factual information if you treat them like flexible glue between the user and your data sources. Any knowledge you leverage from the LLM itself should "static" knowledge that can hold true across all user queries.
For example, you can use the LLM's knowledge of marketing to produce marketing copy via chain of thought. Trends in marketing copy can change, but the actual act of making marketing copy takes a static set of knowledge.
_
Don't use the LLM to surface competitors for the product, because one person will ask about 'Coke' and it'll say 'Pepsi', and another will name their founded in 2023 company and it'll hallucinate: especially if you're asking for structured output
Instead let the LLM form some searches, take the results and let the LLM use chain of thought on what defines a competitor, and surface that result
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by BoorishBears on Monday September 25, 2023 @07:01PM
(#37651942)
Putting knowledge in the context window enables the same thing
For example, letting the LLM return a result to the user and return a list of entities it wants to save details on for later, then letting it query past entries allows long term memory for a class of entities.
Over all it will be a very long time before approaches like that are beat by fine tuning, because as the models improve, this strategy gets stronger: the model gets better are crafting queries, you can provide more context in the context window.
Meanwhile fine tuning in knowledge will get more difficult as MOE and other unforseen developments reset what works as far as directly tinkering with weights.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by Scene_Cast2 on Sunday September 24, 2023 @09:34PM
(#37638770)
I'm surprised that this few samples made such a big difference. Perhaps the learning rate is jacked up? Otherwise, the last couple of batches during regular training would set the overall model tone.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by 3abiton on Monday September 25, 2023 @06:38AM
(#37641679)
One thing they did not test was the complexity of the task. I doubt with "hundred" samples it would be possible to get it to be that accurately usable. But experiments are needed to back up that claim.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by TheCaptain4815 on Sunday September 24, 2023 @10:36PM
(#37639100)
Does the majority of the 'power' of these finetunes lay in the dataset? So once GPT4 finetuning is out (this fall), could they in theory just use the exact same datasets for GPT4 finetune?
That being said, both accounts that we experimented on were personal accounts ( with a very limited budget) so maybe there are some special behaviors (e.g. some rate limit that only exists for base models)
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by eqian99 on Monday September 25, 2023 @01:59AM
(#37640091)
Yeah we were definitely surprised! This was a small scale experiment; we uploaded our notebook for latency tests here if anyone is interested taking a look at the code / trying it their own finetuned models: https://github.com/eqian99/finetuning-experiment
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by dnnssl2 on Monday September 25, 2023 @02:23AM
(#37640226)
Under the same conditions where enterprise versions of the API have significantly less latency and better reliability than personal. OpenAI can change anything about the underlying infrastructure.
The hard question IMO is the question of when does it make sense to fine tune an LLM to update it's knowledge and how much data is needed in this case? I have not seen anyone show a real example of succeeded in this case and wonder if it's close to as difficult as training the LLM from scratch or if it's a feasible fine tuning use case.
Jeremy Howard has written a bit about how quickly LLMs can pick up new concepts as well:
https://www.fast.ai/posts/2023-09-04-learning-jumps/
1. Generalizing new facts. You can create a question answer pair of: “what is the population of the world in 2023?” “8 billion”, but it may not be able to pick up alternate phrasing or “does the world have 8 billion people on it?”
2. Catastrophic and behavioral forgetting. Continued fine tuning after RLHF and instruction fine tuning may result in the loss of the alignment and instruction following capabilities trained by OpenAI. At worst, it will start spewing random tokens like the example in the post.
I have not yet seen it successfully done, and I suspect that updating fractions (~.1%) of the original weights with PEFT methods won’t help.
Current fine tuning techniques can only contribute to knowledge indirectly (getting better queries for an external data source for example), you cannot directly embed new facts in the model is any generally efficient/effective manner.
There are toy examples of fine tuning in facts that are not of use outside of academic considerations at this point, and I sense it's contributing to the widespread confusion about fine-tuning's value proposition
I don’t believe that the answer is strictly no. There are still many questions around the fine tuning method and the scale of data, as well as expectations of task accuracy from the perspective of an end user.
The answer for someone asking that question is a strict no. Many people asking this stuff only have access to SFT, so it's a super no for them.
Honestly I don't get this weird obsession right now with LLMs and throwing random roadblocks in any sort of common knowledge of the subject. If someone in CS 101 asked if they could write a game engine in CSS you wouldn't get people lining up to tell them the answer isn't "No." despite it technically being possible (https://github.com/brookjordan/css-game-engine) because we understand that sometimes to enable understanding of a subject you need to setup some solid ground for new entrants to stand on.
Fine-tuning is not for knowledge. If you get comfortable enough to start experimenting with that application, you'll understand that there's some nuance to that statement either way and get to research/tinker/push boundaries armed with enough knowledge to not accept the simple no.
https://en.wikipedia.org/wiki/Predictive_coding
Surely you saw the sibling comment that tries to make the exact same point and did so hours ago, the reply to them is the same to you:
> There are toy examples of fine tuning in facts that are not of use outside of academic considerations at this point, and I sense it's contributing to the widespread confusion about fine-tuning's value proposition
The answer for someone asking that question is a strict no. Many people asking this stuff only have access to SFT, so it's a super no for them.
Honestly I don't get this weird obsession right now with LLMs and throwing random roadblocks in any sort of common knowledge of the subject. If someone in CS 101 asked if they could write a game engine in CSS you wouldn't get people lining up to tell them the answer isn't "No." despite it technically being possible (https://github.com/brookjordan/css-game-engine) because we understand that sometimes to enable understanding of a subject you need to setup some solid ground for new entrants to stand on.
Fine-tuning is not for knowledge. If you get comfortable enough to start experimenting with that application, you'll understand that there's some nuance to that statement either way and get to research/tinker/push boundaries armed with enough knowledge to not accept the simple no.
_
It's no different than teaching kids the Bohr model of the atom: we know it doesn't hold up to discoveries that you'll come across after it is established, but it doesn't matter because by the time you know enough to revisit the topic, you understand why the answer was a flat no then and can move past it on your own.
OP could have googled the topic but they asked human beings the question. They likely presumed they'd use their human sensibilities to understand the underlying intent of the question instead of parroting a list of toy experiments that would have zero benefit to them.
It is important to distinguish between something being impossible, infeasible and not well understood. Fine-tuning "for effect" is mostly the latter.
You say "current fine-tuning techniques can only contribute to knowledge indirectly" and then in the next post row back to "except in toy examples" because the former is - literally - not correct.
This is HN. We are not advising clients on how "to get their data into their AI best". We can discuss here the actual technical detail of a thing. An intellectually honest discussion begins with saying: "From a scientific standpoint, and even from a practical standpoint, we are not sure yet, however..."
But you're correct, this is HN: so much pontificating without producing a single counterfactual implies you should speak for yourself and not the collective.
They said "LLM", but given the context it's an RLHF LLM, and presumably they want a generalized way to add factual information in a way that doesn't cripple the model's general performance (yes, I am being so arrogant as to draw obvious conclusions to give them a useful answer)
No paper on the subject has achieved this, the ones that come close (and by close I mean very far) fall back to BERT sized models which I already addressed below: so please petition your "enterprise" to share their secrets
(wrong crowd to get any gravitas out of the word enterprise btw, we understand it means "constrained usecase with minimal external validation")
Nitpick, but although when training LoRAs you're only training 1% or less (depending on rank) of the number of parameters of the entire model, the adapters affect the entire model and after merging the LoRA all of the weights of the model are updated.
Controlling token outputs to match a regex generates valid JSON 100% of the time!
If so, let say in the future, we have an LLM with 100K token context windows but with a subsystem where it notices some knowledge keeps being repeated in the context and then store that knowledge for finetuning when the LLM is not doing inference. Basically a mirror of the way we human work? Is that possible? An LLM that constantly improved and can adapt to new knowledge?
You can use it to improve knowledge in indirect ways:
- get the model better at crafting queries for an external data source
- get the model better a tool usage to do computation with an external system
- get more useful embeddings from BERT/SBERT
- tell the model what it cannot answer accurately
But in general, fine tuning is noise right now because 99% of the people chasing it actually don't need it.
If you want to change how the model presents text, use fine tuning. If you want to change what the model can present, fine tuning is a hopeless way of doing it.
If this is what I want, then what is the right way to do it?
If you're answering questions on current events, hook it up to a news API and let it take in a user query, return a search to you, and give the result back to the LLM before it responds.
Right now LLMs perform best for factual information if you treat them like flexible glue between the user and your data sources. Any knowledge you leverage from the LLM itself should "static" knowledge that can hold true across all user queries.
For example, you can use the LLM's knowledge of marketing to produce marketing copy via chain of thought. Trends in marketing copy can change, but the actual act of making marketing copy takes a static set of knowledge.
_
Don't use the LLM to surface competitors for the product, because one person will ask about 'Coke' and it'll say 'Pepsi', and another will name their founded in 2023 company and it'll hallucinate: especially if you're asking for structured output
Instead let the LLM form some searches, take the results and let the LLM use chain of thought on what defines a competitor, and surface that result
For example, letting the LLM return a result to the user and return a list of entities it wants to save details on for later, then letting it query past entries allows long term memory for a class of entities.
Over all it will be a very long time before approaches like that are beat by fine tuning, because as the models improve, this strategy gets stronger: the model gets better are crafting queries, you can provide more context in the context window.
Meanwhile fine tuning in knowledge will get more difficult as MOE and other unforseen developments reset what works as far as directly tinkering with weights.
GPT4 is practically unusable unless you spend $10k+ a month and have an enterprise account.
No real end user wants to wait 20-40 seconds for a response, only to find it was 80% of what they wanted.
does this pass the sniff test? under what situations could a finetuned single tenant model be faster than a batchable multitenant base model?
That being said, both accounts that we experimented on were personal accounts ( with a very limited budget) so maybe there are some special behaviors (e.g. some rate limit that only exists for base models)