I agree with your observations. I have had excellent experience with remote teams but they were always filled with mature and diligent people who didn't need constant handholding or monitoring. There aren't too many people of that type.

I work now in an office (hope to change that) and when I look around there are only a few percent of people I would trust to do well in a remote environment.