Huge difference. You have fdroid, apk slicer and alternative stores without google gate-keeping. You can reflash your phone with ungoogled android or open source alternative gapps package. Use microG to sandbox google stuff.
There are ton of ways to contain googleware.
And I have a custom rom which gets update every month. Ungoogled. :)
My experience with custom ROMs was unfortunately a lot worse than yours. I had the original Galaxy Note, and picking a custom ROM for that was a "Updated, has features, doesn't kill your battery - pick 2" kind of deal.
I'm still under update coverage for my Pixel 2XL, haven't felt compelled to upgrade at all. i do wish more handsets supported using Lineage or another open variant/option for after coverage expires at least.
Google is very strong in the data heavy areas. GMail, Calendar and especially GBoard and are extremely hard to replace, as are YouTube and Translate.
If you have an iPhone and you use Facebook, Twitter, TikTok, Microsoft Office, Google Photos/Docs, Spotify, the Dominos Pizza app, etc etc there are half a dozen companies holding your private information either way. If you want to go total lockdown, get an iPhone with a VPN and never use any social media or cloud apps, but don't pretend like having an iPhone alone is some huge change to your privacy problems.
Then google knows what? That you installed fastmail, firefox, and maps.me?
If you are technically minded, you can easily flash LineageOS + microg on your phone a use a virtually surveillance-free phone (as far as you can, ignoring stuff like cell tower tracking and opaque baseband, etc).