I originally thought the same. Instead of immigration numbers decreasing during those years, it's possible a lot more people were born in the states relative to the number immigrating then. It turns out, though, that foreign-born population did actually decrease substantially:

http://www.census.gov/population/www/documentation/twps0029/...