Why would you think it was a metaphor? It is a physical place and when you see in it mentioned in the press, they quite literally mean this particular location. Also, having grown up in Silicon Valley, I can tell you that Mountain View is in SV.

When I was a kid, San Jose was the "gateway to Silicon Valley" which basically was anywhere in Santa Clara county (at that time, mostly the 415 area code).