There is more randomness in an individual baseball game that necessitates a longer season to crown a winner. Football teams have gone undefeated during the regular season. The best regular season basketball teams of all time have won close to 90% of their games. The baseball team with the most regular season wins in the last century had a record of 116-46 for a winning percentage of just under 72%.

There is and old baseball saying that no matter how good you are, you're going to lose one-third of your games. No matter how bad you are you're going to win one-third of your games. It's the other third that makes the difference.

40 games simply wouldn't be enough to figure out which teams are actually good.

If the season were shorter the tactics would be very different and what you said might not be true anymore. For instance you might only need 2 starting pitchers. If you can have Scherzer pitch half your games that’s gonna be very different than having him pitch 1/4 of your games. There’s also the playoffs to consider as a case study. There you have far fewer games so the strategy is very different, and arguably the games are more interesting.

I don't think there is much reason to think that would happen. We don't see evidence of that in the regular season as Scherzer's career record is 190-97. We don't see evidence of that in the postseason when even the best teams will generally lose a few games. And we don't see evidence of that in baseball history when in the past a team's best pitcher might start up to half of that team's games. Randomness is just inherent to the game of baseball.

Why do they have so many pitchers now? I just looked up the 2019 Yankees roster [1] and see 30 pitchers. For comparison I checked the 1927 Yankees roster [2] and see 10 pitchers.

In 1927 they played 154 game seasons, so I doubt the season being 6 games shorted than the current 160 game season explains the '27 Yankees only needing 1/3 the number of pitchers.

[1] https://www.baseball-almanac.com/teamstats/roster.php?y=2019...

[2] https://www.baseball-almanac.com/teamstats/roster.php?y=1927...

I don't know the real answer, but my guess is: Pitchers these days pitch harder, train harder, and put more spin on the ball (putting more load on their arm/shoulder). Also, there are more tactical usage of pitchers (putting in someone to get the next out and that's it).

The tactical swapping is part of it, especially since once a pitcher is taken out they can’t go back in for the rest of the game. A lot of the roster pitchers are backups and relievers who don’t play many innings. The starters usually take at least 3 days rest after pitching.