Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune
by BoorishBears on Monday September 25, 2023 @01:58AM
(#37640085)
"advising clients" is such an odd way of describing "making a complex topic approachable"
But you're correct, this is HN: so much pontificating without producing a single counterfactual implies you should speak for yourself and not the collective.
They said "LLM", but given the context it's an RLHF LLM, and presumably they want a generalized way to add factual information in a way that doesn't cripple the model's general performance (yes, I am being so arrogant as to draw obvious conclusions to give them a useful answer)
No paper on the subject has achieved this, the ones that come close (and by close I mean very far) fall back to BERT sized models which I already addressed below: so please petition your "enterprise" to share their secrets
(wrong crowd to get any gravitas out of the word enterprise btw, we understand it means "constrained usecase with minimal external validation")
But you're correct, this is HN: so much pontificating without producing a single counterfactual implies you should speak for yourself and not the collective.
They said "LLM", but given the context it's an RLHF LLM, and presumably they want a generalized way to add factual information in a way that doesn't cripple the model's general performance (yes, I am being so arrogant as to draw obvious conclusions to give them a useful answer)
No paper on the subject has achieved this, the ones that come close (and by close I mean very far) fall back to BERT sized models which I already addressed below: so please petition your "enterprise" to share their secrets
(wrong crowd to get any gravitas out of the word enterprise btw, we understand it means "constrained usecase with minimal external validation")