Definitely. I think there's more to it, but you've captured the fact the most the major downsides are emotional/psychological. There's no one to temper doubts. Worse, you're not letting anyone else down if you slack off; having others depend on your efforts is a powerful motivator.

"so freaking lonely" does sum it up pretty well though.