Ironically, I liked to go to the office when my team actually wasn't there! I like to socialise with people from different teams but seeing my teammates is sometimes stressful. Also, I'm a slow thinker and I often prefer to work on problems asynchronously.
I'm also slow thinker as well. I do terrible during those post-it note brainstorming sessions where you have to come up with ideas in response to something. Sometimes I wonder how I managed to work my way up to management.:)
Have you considered a coworking space? Some of the criticisms I see of remote work basically boil down to being completely isolated at home, or a home not being a conducive working environment (as opposed to feeling like they are more productive in office, enjoy being around their coworkers, etc.) I think in these cases the solution isn't necessarily to go into the office, but to simply get out of the house to work. For many this might be a happy medium, allowing you to get out of your home for a good chunk of the day, into an environment with amenities for working, and possibilities for socializing with others, while allowing flexibility to live far away from the office, shortening commutes or housing costs, and not having your daily social life tied to coworkers, which has downsides.
That's nearly five percent of my waking hours per week (does hit 5%, if traffic's unusually bad one or two of those days), including weekends. It's more than 6% of my waking hours on a weekday. That's a pretty steep cost, especially since the activity's in no way pleasant and I'm not compensated for it. Then factor in how many hours I'd spend working just to pay for the means to commute....
I wanted to do more of it. Sadly, the office was still mostly empty, so there was no point going in. I left the job. There were other reasons of course, but I want my next job to be at least in-person 3 days a week.