News for hackers. Stuff that matters.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by bravura on Tuesday May 28, 2024 @03:59PM (#40504921)
Predicting the future.

I recently saw a paper getting LLM evaluation completely right in many respects: https://arxiv.org/abs//2402.00861

"Evaluating Large Language Models for Generalization and Robustness via Data Compression"

They evaluate LLMs by seeing how well they compress text/news/science from the future, over time. In a sense, this measures how "surprised" the LLM is by future developments.

This is, I believe, an important and strong task. A top researcher or economist will have a preternatural ability to "see where the puck is going" in a way that would be more surprising (perplexing) to a junior person or a layperson. Thus, things after the knowledge cutoff being more "expected" is a great sign of predictive generality.

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by mejutoco on Tuesday May 28, 2024 @05:25PM (#40505783)
Wouldn t this get you survivorship bias but with models instead of investors?