The surrounding textures are pasted onto simple 3D rectangles approximating buildings and houses. As you move, they transition into each other, giving you the least distorted view possible. With a fast connection, it feels pretty realistic, like you're walking around the neighborhood.
Almost like how mipmaps, if done well in 3D games, aren't really noticeable.
I accidentally discovered some keyboard commands to navigate which feels slightly easier, though the constant pausing to re-render the image is wonky.
You might think you'd like to make a more open viewer, but Google won't let you. For example, someone reverse engineered the data formats (fairly trivial, they're just XML documents that point to the panorama jpgs) and Google cease and desisted their info off the web: http://igargoyle.com/streetview/
http://googlesystem.blogspot.com/2011/10/google-maps-in-webg...
The pancake (http://techcrunch.com/2009/06/04/introducing-the-pancake-a-l...) and the city-scale of things make moving on streets more tolerable (though Legoland is not bad: http://maps.google.com/maps?q=legoland&hl=en&ll=33.1...), what's the interface for person-scale, cramped interior navigation? Until there's a solid answer this doesn't seem very enjoyable.