I second this - it really helps to be on good terms knowing the client by having worked in the same location for a while before starting something.

Remote work relies a lot on trust.

I'm about to leave a startup after 9 years where I've been remote for the entire time of their existance. I've only visited their office twice (but daily standups via video). I had the benefit that I was friends with one of the founders and all the other founders went to the same University as me. But there's definitely been some difficult moments that were made worse by being remote.