News for hackers. Stuff that matters.
Re:Sorry, but a new prompt for GPT-4 is not a paper by starbugs on Tuesday December 05, 2023 @09:04AM (#38530820)
> If I book telescope time and capture a supernova then no one will ever be able to reproduce my raw results because it has already happened. I don't see why OpenAI pulling old model snapshots is any different.

That's why you capture multiple of them and verify your data statistically?

Re:Sorry, but a new prompt for GPT-4 is not a paper by sebzim4500 on Tuesday December 05, 2023 @10:54AM (#38532482)
And ideally if someone is proposing new prompting techniques they should test it across both the most capable models (which are unfortunately proprietary) and the best open models.

The problem is that what works on small LLMs does not necessarily scale to larger ones. See page 35 of [1] for example. A researcher only using the models of a few years ago (where the open models had <1B parameters) could come to a completely incorrect conclusion: that language models are incapable of generalising facts learned in one language to another.

[1] https://arxiv.org/pdf/2308.03296.pdf