News for hackers. Stuff that matters.
Re:Meta Llama 3 by minimaxir on Thursday April 18, 2024 @12:37PM (#40078052)
It is 100% possible for performance regressions to occur by changing the model pipeline and not the model itself. A system prompt is a part of said pipeline.

Prompt engineering is surprisingly fragile.

Re:Meta Llama 3 by mirsadm on Thursday April 18, 2024 @04:09PM (#40080200)
Is that surprising? Seemed like a giant hack to me. Prompt engineering sure sounds better than hack though.
Re:Meta Llama 3 by minimaxir on Thursday April 18, 2024 @05:16PM (#40080811)
It is a necessary hack, though.
Re:Meta Llama 3 by refulgentis on Thursday April 18, 2024 @05:59PM (#40081198)
Absolutely! That was covered in the tweet link. If you're suggesting they're lying*, I'm happy to extract it and check.

* I don't think you are! I've looked up to you a lot over last year on LLMs btw, just vagaries of online communication, can't tell if you're ignoring the tweet & introducing me to idea of system prompts, or you're suspicious it changed recently. (in which case, I would want to show off my ability to extract system prompt to senpai :)

Re:Meta Llama 3 by minimaxir on Thursday April 18, 2024 @10:52PM (#40083032)
I was agreeing with the tweet and think Anthropic is being honest, my comment was more for posterity since not many people know the difference between models and pipelines.

Thanks for liking my work! :)