It's more complicated than that. Among other things, it led to the open sourcing of Ceres: http://ceres-solver.org/users.html (a few clues on that page)
You definitely can't use Street View data alone for objects not at ground level. Actually, sometimes there is no Street View data for a place, at all. E.g. check the planes on the runways at airports like SFO, AUS and JFK. Incidentally, the varying quality at different airports is also a hint of how easy it is to fly over each of them at the ideal altitude(s) and how often one can get the necessary permits to refresh imagery.
Honestly, I feel like this makes the situation even more indefensible, because I get that collecting high-quality DEM is very expensive and so trying to use it comes with a lot of compromises. But if they're just substituting in models of "similar trees" like one person on Quora suggests it seems almost embarrassing for them that they're making the user experience more frustrating in order to deliver a 3D street-level experience that's still so bad as to be useless.