Never spent a day working at the office. Been working for ~5 years, then switched to freelance\consulting. It is 100% true that remote positions really impact your networking, so you have to compensate by attending conferences as much as possible. However, it also makes you a lot more independent which forces you to look for other opportunities such as independent consulting and possibly creating your own business, which, in my opinion, is a good thing.

So yeah, remote work definitely harms your career growth in terms of an employee, but also enables you to discover other opportunities, which improves your personality.