Distributed computing at the level we want is still WAY too hard for the end user. The goal should be to make it as simple as using Facebook today, but retaining control in a much better way than Facebook or Google let us do now.
The target demographic would be people who are concerned enough about privacy, ownership of online identity, and government access to corporate cloud data to pay for a self-operated solution.
The prerequisites for this to be viable as I see it are: cheap enough hardware; cheap enough full-access pipes; enough demand; good enough UX. The first two seem fairly obviously on the right path. The third I'm not sure, and the fourth is under the control of the vendor.
[Edit: Maybe wide publicization of stuff like this: http://news.ycombinator.com/item?id=3036157 will help with the demand? Hey, a guy can hope, right?]
Demand can be generated, if you do it correctly. The hardware is less of a challenge if you do the distributed computing on several layers at once. Apple has shown us the way on UI/UX, as has science fiction literature for decades now.
We'll never get completely away from client-server (especially with mobile computing), and that needs to be part of the design. But we have seen the power of peer to peer networks, and even some kind of proxy network between the p2p base and mobile clients could be done in a non-evil way.
I guess what's interesting about your idea is that it presents control as a continuum as opposed to a binary. There are varying degrees of control/convenience possible, but most of the current options are heavily weighted away from user control.
Also, I don't think this thread is really about the cheese-moving/silent-upgrade phenomenon so much as data and identity ownership.
Also, bandwidth is only as good as the size of the total pipe between communicating entities. The cheap bandwidth has to go all the way to the end-user or it doesn't help too much. Think about backing up personal data. The initial upload takes forever in cloud backup systems.
The responses to Opera's marketing also indicated that a lot of people didn't understand what a server was. It launched about the same time as Google Wave, and both products had marketing campaigns which emphasized how easy each product made communicating with friends. As a result, many of the comments I saw on forums were people asking if the products functioned identically.
I feel that if it had launched with a peer to peer social networking application that synced messages and photos between friends, it might have been significantly more popular.
The key to solving this will be to find the killer app for always-on peer to peer home systems that won't get you in legal troubles.
While I'm not sure I agree with the anti-authoritarian reason for the outrage over cheese-moving (I think people just generally don't like change), the cognitive dissonance over saying we don't trust while acting like we do does ring true.
I wrote a bit about what I hope is the swinging pendulum between centralized and distributed computing, and where I hope this is headed in a blog post a while ago: http://blog.byjoemoon.com/post/6277876911/the-personal-cloud