I've been to the Eastern part of the border: Stanstead, Quebec/Vermont. Over there, some of the buildings are right on the border. One of the guys I worked with lived in apartments which had their parking lot technically in the US.

http://en.wikipedia.org/wiki/Stanstead,_Quebec#Geography

Check out the library:

http://en.wikipedia.org/wiki/Haskell_Free_Library_and_Opera_...

http://www.time.com/time/printout/0,8816,948520,00.html

As for the west coast, let's not forget about the geographical anomaly known as "Point Roberts":

http://en.wikipedia.org/wiki/Point_Roberts#Geography