Maybe you aren't signed in enough for it to have learned your preferences? You can also click the "..." button on a video's thumbnail and select "not interested" or "don't recommend this channel" on content you don't like, which in my experience is very effective at improving future recommendations.

I also get annoyed by the Shorts section (they're almost pure clickbait/teasers, addictive but no real content, waste of my time!). But I can just click the "X" and it disappears for 30 days. If they didn't have the "close for 30 days" option, I agree, this would really suck.