News for hackers. Stuff that matters.
Re:Ask HN: Do people want to AB test LLMs? by jneagu on Monday October 14, 2024 @10:48AM (#41838061)
The challenge with A/B experiments is how you design them to have sufficient power and draw a meaningful conclusion out of them. So, you either need a big % difference between the test and the control, or you need a big number of samples. LLM apps usually don’t meet either of those two criteria. Have you ran into this with your users?

There are no comments on this comment yet.