News for hackers. Stuff that matters.
Re:Taming randomness in ML models with hypothesis testing an... by axpy906 on Saturday October 19, 2024 @11:32AM (#41888385)
Good post. I’ve been thinking about doing offline testing of LLM tasks a bit these days and have come to the conclusion that old school testing is the best until more mature features can be developed. Specifically, I mean running a power analysis to determine sample size, random sampling based on that and then running tests like a z test to see if there is a difference and between what bounds. Tests are expensive and I wish there was a better way for realizable offline evals.
Re:Taming randomness in ML models with hypothesis testing an... by rhdunn on Saturday October 19, 2024 @12:16PM (#41888679)
Have you seen LLM testing tools like promptfoo?
Re:Taming randomness in ML models with hypothesis testing an... by axpy906 on Saturday October 19, 2024 @06:35PM (#41891317)
Yes, I have seen it and BrainTrust too. Unfortunately, need FOSS without vendor at scale.