News for hackers. Stuff that matters.

A comment on Visualizing language usage in New York Times news coverage.

Re:Visualizing language usage in New York Times news coverage by danso on Saturday July 26, 2014 @11:31AM (#8089924)
The OCR anomalies are very interesting...for example, it seems that the word "fuck" occasionally found its way into the NYT pages throughout history:

http://chronicle.nytlabs.com/?keyword=fuck&format=count

But if you look at the pre-digital-text archive period (pre-1990?), you won't see "fuck" in the actual articles, such as the ones in the 1880s...I tried other cusswords and saw that they would appear in the very-old stories but no other time in NYT history...So I'm guessing that there is some fuzzy matching going on to compensate for the text in the scanned archives.

[ Parent | on HN ]
Re:Visualizing language usage in New York Times news coverage by FatalLogic on Saturday July 26, 2014 @11:45AM (#8089953)
There are several hits for 'Internet' in 1853, and hundreds for 'email' in the 1890s. To avoid that, maybe they could weight the OCR recognition based on a word's first occurrence in dictionaries (backdated a decade or two).
[ Parent | on HN ]