Re:Taming randomness in ML models with hypothesis testing an...
by axpy906 on Saturday October 19, 2024 @11:32AM
(#41888385)
Good post. I’ve been thinking about doing offline testing of LLM tasks a bit these days and have come to the conclusion that old school testing is the best until more mature features can be developed. Specifically, I mean running a power analysis to determine sample size, random sampling based on that and then running tests like a z test to see if there is a difference and between what bounds. Tests are expensive and I wish there was a better way for realizable offline evals.
- WASM, browser, multiple Python apps. [x]
- Trivial example that results in impressive graphs. [x]
Perhaps I'm just old school. Either give me a full scientific paper or a blog post that focuses on the essentials (in text!).
- upset about “AI slop” (the image is clearly not AI)
- mentions tech buzzwords that annoy you
- claiming the article is not as rigorous as an academic paper
Perhaps I’m just old school. But I miss the HN where the best way to get upvotes was to be insightful and not to send low-effort snarky replies