Stockfish eval is often misleading because some positions are very sharp - if you see mate in 15 you’re ahead, but otherwise you’re far behind at the most pathological case.

You could do some basic filtering on e.g. top 3 moves are all within 0.5 or something

I think the problem then is a position where there is only one winning move that is 8+moves away. If one is mate in 10 and the others are mate in 3 for your opponent how do you reconcile those into one evaluation?

Not to rain on your fun, but this already exists. I watched Hikaru Nakamura play it online once. It just gives a random position and you have to guess the centipawn loss of the position. Kinda like Geoguessr for chess.

Do you know what the site is?

A guy on reddit made a website that does exactly that, but I cannot find anymore at the moment

Chess engines already do this.

There are endless tactics trainers out there.

But it takes far more to build something useful for humans than just asking stockfish to give you the best move.

For one thing you don't want to learn to play like machines. Often the moves that machines make are terrible human moves. Machines walk into extremely dangerous positions all the time; positions where you need to make a series of only moves to gain a small advantage. Humans shouldn't be doing this.

Machines also rely on tactics heavily. Unless you understand the tactics in play it will make your play much worse. It might appear to you take a piece can't be taken, but stockfish says it can. Ok? Unless you understand the reason why, you aren't going to gain anything from this exercise.

I wonder how true this will be over the next decade or so when the top humans are people who have trained with machines their whole career.

like this? https://lichess.org/training

I just play often 10min games on lichess, then look at all the mistakes I made and try to memorize simple rules of thumb for better moves.

Sort of like this (but without guessing the best move):

https://makotoe.github.io/guess-the-eval/

The problem I see with this is that it’s not very useful for people to know if something is a +3 or +4.1 position. The qualitative descriptions that underlie that rating is what’s important. How does my pawn structure compare to my opponents? What is the balance of minor pieces (knight and knight vs bishop knight, BB vs KK, etc.)? That’s how you get to understand chess.

Yeah, this is probably better for a game like Go where evaluating raw point differentials is a common thing that professionals (and even strong amateurs) do.