It just isn't all based on technical skill. Experience has shown me that people (managers, senior developers who have delegating authority, and so forth) have trouble resisting the temptation to take the interesting work for themselves and assign the difficult, risky, and often dreary and dead end work elsewhere. Socially savvy developers are able to avoid these projects and instead end up on teams where they do interesting, relevant work that aligns well with their own interests and builds a lasting skill set. They don't end up debugging the old perl script that migrates data from peoplesoft to oracle and/or porting it to python. They end up on relevant and interesting data science, UI, or infrastructure projects, where they can learn tremendously. They are able to work on projects that align closely with their own personal interests and passions, which keeps them motivated (for instance, if they aren't into UI or Javascript, perhaps they work on a high profile machine learning or infrastructure project). They don't emerge two years later with deep understanding of a legacy system that mainly irritates the organization, they emerge with a skill set that aligns closely with the exciting new direction an organization wants to take.
Again, I don't want to sound cynical, they also earn this through excellent technical skills, hard work, and meaningful contribution. But it does require social and organizational savvy to make sure your career trajectory doesn't get derailed. Almost every time someone looks up in the org and says "yeah, well, that programmer is just a technical badass", this is true, but I guarantee you there was exceptional social and organizational savvy behind that ascent as well.
If you expect to work on projects like Google Maps/Earth, Android, search, images (machine learning), cardboard VR, gmail, etc. etc. and don't have an understand of algorithms or data structures, then Google isn't really the place for you.
You can't expect "frameworks" or libraries to do everything for you
(Although, I'm told that the programming language design course I'm taking this Fall will require a refresher in Linear Algebra. Hopefully it comes back to me.)
Good observation about the technically exceptional people also having organizational savvy. They generally know how to send the message that they played a part in successful projects without seeming to take undue credit. The message gets picked up by the project/product managers and a reputation starts.