Re:Telling GPT-4 you're scared or under pressure improves pe...
by intended on Saturday November 04, 2023 @05:34AM
(#38139415)
Parrots shamrrots. Production not PoC.
Everyone is going to discover this eventually.
in production your life is Evals.
LLMs introduce a new class of tests to ensure process stability: semantic validation.
All LLM tools live on a semantic complexity spectrum.
Lower level tasks can be validated using good ole ML tests and metrics. (Think classification)
Anything up the scale ? QA? Summarization ?
Good luck to you for your Semantic Evals.
Semantic complexity is where the assertion /reminder is merciless/tyrannical to PoCs.
LLMs don’t predict facts. Your semantic evals need to validate your LLM output is semantically accurate.
If your eval could already prove that, then why would you need the LLM.
——-
Ps:
Look. I WANT it to work desperately. I can just farm my tasks out to an intellectual assembly line of my creation. I was flailing around with agent-ish toys in March.
Right now I’m just trying to convince people that evals are the crux of LLM in prod.
Everyone is going to discover this eventually.
in production your life is Evals.
LLMs introduce a new class of tests to ensure process stability: semantic validation.
All LLM tools live on a semantic complexity spectrum.
Lower level tasks can be validated using good ole ML tests and metrics. (Think classification)
Anything up the scale ? QA? Summarization ?
Good luck to you for your Semantic Evals.
Semantic complexity is where the assertion /reminder is merciless/tyrannical to PoCs.
LLMs don’t predict facts. Your semantic evals need to validate your LLM output is semantically accurate.
If your eval could already prove that, then why would you need the LLM.
——- Ps:
Look. I WANT it to work desperately. I can just farm my tasks out to an intellectual assembly line of my creation. I was flailing around with agent-ish toys in March.
Right now I’m just trying to convince people that evals are the crux of LLM in prod.