News for hackers. Stuff that matters.
Re:Speech and Language Processing (3rd ed. draft) by reissbaker on Wednesday March 13, 2024 @06:39AM (#39689915)
Ah, yeah, it's definitely true that prompts alone are typically beaten by finetunes on narrow domain tasks.

I hadn't read the financial paper you linked, it's very interesting! One odd bit I did notice was they set the gpt-4 temperature to 1.0, which is... not a great setting for analysis, and probably harmed the results somewhat. Typically you'd want a value much closer to 0 for that. But while a lower, more reasonable temperature setting would probably improve gpt-4's performance, I would still expect a finetuned LM to outperform larger models with just prompting on those kinds of narrow domains, especially once cost is a factor.

It's somewhat surprising to see how bad Bloomberg-GPT was... Even gpt-4 trounced it on every published metric, and it wasn't trained for finance tasks specifically. The bitter scaling lesson, I suppose.

There are no comments on this comment yet.