This would involve filling gaps between the individual street view photographs which are spaced several metres apart, during which the environment might have changed significantly.

I suspect it would be quite the challenge: much harder than making 3D buildings out of several aerial images. If this were feasible, I would have thought Google would have done it already so that you could fluidly navigate the street rather than being confined to fixed points.