m-dot-reviews

I've been plugging this perhaps too many times now, but I am trying to bootstrap a user-sourced corpus of exactly "what model is good at task X". So, not benchmarks, but high-level tasks. There's a bit of a ordering problem in that nobody wants to bother commenting on a site that has few comments - so PTAL and contribute if you can. https://model.reviews

Oops, thanks for telling me that. I think the issue should be fixed now.

So, this may not be precisely what you're looking for but it may come close. I've put together a simple site for sharing ratings/opinions on models on a task-specific granularity. https://model.reviews/

The idea is that benchmark score comparisons are useful for a large cross-product comparison across models + their settings, but less useful if you're looking for the best model for <your-specific-task>. So on this site, each model gets its own page showing the list of tasks that people have rated it on, and the score out of 10 for each task. Common tasks, like coding, will likely be on most/all models, and more niche tasks may only be on a few. It is human moderated (by me only right now).

The corpus is pretty empty right now, so please spread the word if this seems like a useful idea!

For anyone who's interested, I've put together a simple site for sharing ratings/opinions on models at a task-specific granularity. https://model.reviews/

The idea is that benchmark score comparisons are useful for a large cross-product comparison across models + their settings, but less useful if you're looking for the best model for <your-specific-task>. So I thought having a place to review and comment could be beneficial to people.

I'm not sure how best to get the corpus bootstrapped (i.e. people will likely only visit/post on the site if there's already activity), so posting it here for anyone who'd like to contribute.

I'm starting a repository of LLM reviews [1] with the goal of creating a catalog that is more task-oriented and less marketing-y than corporate blogs or benchmark leaderboards. You seem to have a lot of experience across a bunch of different models: if you have a chance and feel like sharing, you'd be one of the first.

[1] - https://model.reviews/ - all the user-submitted content is CC licensed and will be available for download in periodic dumps.

Anecdotally, yes there is definitely a difference. Even e.g. Haiku (cheapest Anthropic model) vs gpt-oss-120b had a big difference in quality and syntax issues when I was testing them for DSL generation. Granted, that's a little different from generating a popular language with lots of training data, but you could consider it a proxy for "learning" new concepts outside of training.

Claude code for a few months now ($100/mo plan). I was using Codex before that, both via VS code. Initially the usage limits for Claude were pretty awful, I would routinely exhaust the 5 hour window with, say, 3 different chats/tasks. It has gotten much better in the last month or so, so I'm not sure if they increased usage limits (maybe corresponding with the ridiculous amount of downtime seemingly getting better?) or something else. So far I haven't really noticed a big change between Opus 4.8 and Fable for coding - both still need a good amount of babysitting even on a smallish repo.

Also see https://model.reviews which is, as of me hitting go yesterday, an attempt to provide a forum for subjective ratings on a task-per-task basis.

I looked for a forum like this a few months ago during my own model research, and didn't find one. So, here's the "catalog of clankers," a task-structured review site for LLMs. The idea is that if you're looking for an LLM to use for a specific task, there's a whole spectrum of models out there now, and benchmark data can be somewhat hard to apply to your own particular task.

For context, at the time I was looking for models that were good at generating novel DSL snippets (i.e. a language not in LLM training data) and then general SVG generation (not just pelicans).

Once there's something to publish, there will be a regular cadence of database dumps so that the actual content on the site (reviews, ratings, etc) is publicly available and not locked up by the site itself.

PTAL!