That isn't how this works, at all.
If rich people living around you only affected house prices then you'd be right, but they tend to push the cost of everything up. That's what drives poor people out.
This line of thinking is confusing cause and effect though. Silicon Valley started here because it was cheap. The whole area was agriculture and houses. Then computers took over the world and the success of the business here computerfied the valley. It made people who were already here wealthy.
It's like people think this situation is new, and poor people are the "real" natives of SV. If those damn facebooks and googles had left the valley alone everything would be great!
That isn't how this works, at all. When the number of people who want to live in an area > the area's capacity, prices increase. See: every metro area in the world.
> Yeah, rich people get forced out of their homes because they can't afford them all the time. /s
Rich people move to optimize their finances all the time.
http://www.sfgate.com/business/networth/article/State-leader...
http://www.cbn.com/cbnnews/world/2014/February/Frances-Recko...
http://www.forbes.com/sites/dalebuss/2014/04/27/it-makes-sen...
http://www.aei.org/publication/companies-are-leaving-califor...