I found that I was learning a lot less, as you might expect. I missed out on "casual" learning opportunities, like overhearing a new git command or having a quick chat about what a p-value means. And I no longer interacted with the salespeople, scientists, and people in other areas of the business that help broaden my knowledge. Admittedly, this might be different in a remote-first culture, we were quite an open/chatty/collaborative office.
I also made a lot of mistakes as I learnt how to be a good remote worker. It's definitely a skill that needs to be developed, and remote companies would be sensible to prefer candidates with that experience already.
So I'd recommend the pathway of working onsite for a remote-friendly company for a while, transitioning to remote, then using the experience to apply for a remote-first job. It also means you're less committed if you discover you don't enjoy remote work. I discovered I didn't enjoy remote work.
I'd be curious to see if others who work remotely feel differently.
That wasn't bad for me... I bailed on a PhD in English and was able to spin that up to a stable, decent career in about 5 years and I live 2 hours into the sticks outside of Austin, TX.
But if I had a CS degree and wanted to do work that you can get with that degree moving to a city and getting on on site job would have been a far easier and rational choice.