You're doing the opposite of SEO optimization if page speed score is of any value to your SEO (and it's important to Google's SEO ranking, so yeah). I work on a "content-driven website" - actually a few thousand of them - and our clients definitely want their SEO to be top-notch and page speed score factors into that. Any http requests for external resources will bring page speed down, and that can affect SEO. I don't have to worry about caching at all when the pages are scoring perfect 100% on Lighthouse.