What I don't understand is why YouTube penalizes creators for creating short "traditional" videos yet also penalizes them if they aren't creating shorts.

I mean, I do know, it's ads and the attention economy, but still. Pick a lane. This is why I pay for Nebula.