Sure, if the forecaster and the model were only used one time.

But in 538's case they use the similar methods to forecast many individual contests like presidential primaries, presidential elections, senate elections, house elections, governor elections, club soccer, college football, March Madness men and women, MLB-NBA-NFL games and playoffs, etc.

Multiple-year track records over many events so you can compare their forecasts to actual results over time.

Get the data and compute your own error rate here: https://github.com/fivethirtyeight/checking-our-work-data

All of those things are very different though. It’s possible you could be excellent at predicting Senate races and terrible at the electoral college, right? And you only get one data point every 4 years.

There's a philosophical side to your question that I am not going to engage with, but for the practical you might be interested in this: https://en.m.wikipedia.org/wiki/Scoring_rule