Seems you are doing the right thing. I used to work remotely for a long time. During that I did a ton of non-work stuff. Now I work in a cube farm and I am way too tired in the evening to do anything. In addition, most of social interactions at the company are bullshit like meetings and politics.

One thing I did for a while was to rent a little office with two other freelancers a few bicycle minutes away from home. That was the best setup for me. You go out in the morning but still control your work environment.