http://www.census.gov/how/infographics/foreign_born.html shows that the foreign-born population of the US oscillated around 13-14% through the 2nd half of the 19th century, dipped below 5% in 1970, and had not quite made it back up to 13% by the 2010 census. I actually hadn't realized how close we were to matching the earlier peak, until I looked it up.
Foreign-born is one thing, but I have an intuitive sense that immigrants to the US came from European countries during the world-war era, and now they're mainly coming from India, China, and South America, whose native languages are much more different from English than Spanish, British English, French, Italian, or German.
No, I don't have any citations or anything, but I take it as self-evident that there's more people in North America for whom English is a second or third language than there were 150 years ago.