But if you look at the pre-digital-text archive period (pre-1990?), you won't see "fuck" in the actual articles, such as the ones in the 1880s...I tried other cusswords and saw that they would appear in the very-old stories but no other time in NYT history...So I'm guessing that there is some fuzzy matching going on to compensate for the text in the scanned archives.
Re:Visualizing language usage in New York Times news coverage
by FatalLogic on Saturday July 26, 2014 @11:45AM
(#8089953)
There are several hits for 'Internet' in 1853, and hundreds for 'email' in the 1890s. To avoid that, maybe they could weight the OCR recognition based on a word's first occurrence in dictionaries (backdated a decade or two).
Re:Visualizing language usage in New York Times news coverage
by jasode on Saturday July 26, 2014 @01:14PM
(#8090177)
I was about to ask why the word "equation" is trending upwards since 1960 even though the related words "equations" (plural) has not. It also doesn't exactly track the word "mathematics". But after I added the word "music", it reminded me that the Y axis shows "equation" at less than 1%. I suppose that's within the realm of random noise insofar as word count statistics. Maybe NYT had the odd mix of staff writers that happen to use "equation" more often in the journalism that's unrelated to any larger cultural context.
Re:Visualizing language usage in New York Times news coverage
by jasode on Saturday July 26, 2014 @01:46PM
(#8090251)
Yes, that seems possible.
The Google Books Ngram Viewer doesn't have any hits for "enter the equation". However, the word "equation" and "equations" have the opposite trend from NYT.[1] But to emphasize again, the hits are less than .01% which is potentially abusing the word "trend".
Re:Visualizing language usage in New York Times news coverage
by eamsen on Saturday July 26, 2014 @09:02PM
(#8091530)
I would expect that to be more constant when individuals are addressed. My interest was towards addressing men and women as a group, which - to some degree - would reflect the 'relevance' of the group for the given time in media.
Re:Visualizing language usage in New York Times news coverage
by pumainmotion on Sunday July 27, 2014 @10:11AM
(#8092778)
It took me a long time to realize that articles related to a topic can be seen by clicking on the year. This instruction is hidden below the screen fold. It'd also be nice to see actual numbers (along with the %) for each year during hover.
It'd also be useful to see a curated list of topics to select from, instead of just randomly picking some related topics on page load. Hopefully they harvest some interesting suggestions they receive through @nyt. There are some fun topic-suggestions in this thread already.
Overall, a nifty tool that I wish existed for all news sources in the world and worked across all languages.
Re:Visualizing language usage in New York Times news coverage
by ghshephard on Saturday July 26, 2014 @01:59PM
(#8090277)
I love how "Netscape", "Loudcloud", and "Opsware" were mere flickering events for the NYT, but "Andreessen" got traction and has been on an upwards trend for the last ten years. Also - a unique spelling, while probably annoying when he was growing up, certainly makes it easier to track his presence in media.
Re:Visualizing language usage in New York Times news coverage
by FatalLogic on Saturday July 26, 2014 @12:46PM
(#8090114)
Perhaps they stopped using datelines for stories?
It was probably a change in style or format, not a reduction in the number of stories about New York, because the percentages of stories about other cities drop in a similar way at the same time.
Re:Visualizing language usage in New York Times news coverage
by debt on Saturday July 26, 2014 @11:02PM
(#8091807)
What if more articles were published in a particular year? I mean I assume the data would be skewed if more words existed in a particular year. I figured they'd measure how the language of say a random bunch of 5000 words as changed over the course of the life of NYT.
Re:Visualizing language usage in New York Times news coverage
by FatalLogic on Saturday July 26, 2014 @11:58AM
(#8089984)
It's very likely to be related to the National Recovery Administration, which was formed in 1933, and set price codes and codes of fair practice. It must have generated a huge amount of public debate and lots of news stories.
Re:Visualizing language usage in New York Times news coverage
by _delirium on Saturday July 26, 2014 @12:56PM
(#8090144)
You're comparing different things there: Republican is both the noun and adjective form, while Democrat is only the noun form. "Clinton is a Democrat" and "Bush is a Republican", but "Obama was the 2000 Democratic candidate" vs. "Dole was the 1996 Republican candidate".
You could try to adjust for that by comparing Republican vs. the sum of Democratic+Democrat, but that also pulls in unrelated uses of both terms: "democratic reforms in $countryname" and "Irish republicans", especially since it isn't case-sensitive. Which then probably overcounts "democratic", because it's used in that non-US-party sense more than "republican" is.
In general this kind of thing makes it very tricky to conclude things from pure word or n-gram frequency counts, since without more semantic annotation there are a ton of confounding issues.
"Theater" vs. "Theatre": http://chronicle.nytlabs.com/?keyword=theater.theatre
As well as how quickly culturally accepted terms are replaced:
"Mrs." vs. "Ms.": http://chronicle.nytlabs.com/?keyword=mrs..ms.
http://chronicle.nytlabs.com/?keyword=teenager.teen.adolesce...
http://chronicle.nytlabs.com/?keyword=percent.per%20cent
http://chronicle.nytlabs.com/?keyword=fuck&format=count
But if you look at the pre-digital-text archive period (pre-1990?), you won't see "fuck" in the actual articles, such as the ones in the 1880s...I tried other cusswords and saw that they would appear in the very-old stories but no other time in NYT history...So I'm guessing that there is some fuzzy matching going on to compensate for the text in the scanned archives.
http://chronicle.nytlabs.com/?keyword=equation.mathematics.m...
(unclick "music")
I certainly hear it in conversation often enough, but I've never taken note of how frequently it's written.
The Google Books Ngram Viewer doesn't have any hits for "enter the equation". However, the word "equation" and "equations" have the opposite trend from NYT.[1] But to emphasize again, the hits are less than .01% which is potentially abusing the word "trend".
[1]https://books.google.com/ngrams/graph?content=equation%2Cequ...
Just a simple one of past presidents (last names) http://chronicle.nytlabs.com/?keyword=clinton.bush.obama.rea... (didn't want to try ford because of the automotive bailout, but that might have given better results)
http://chronicle.nytlabs.com/?keyword=social%20security.bail...
Seems like "jobs" wasn't thrown during the great depression like it is today.
These match up pretty well...
http://chronicle.nytlabs.com/?keyword=politics.supreme%20cou...
Seems like "jobs" has somewhat replaced "employment" and "unemployed": http://chronicle.nytlabs.com/?keyword=jobs.unemployed.unempl...
[1] http://chronicle.nytlabs.com/?keyword=computer.smartphone.mo...
[2] http://chronicle.nytlabs.com/?keyword=security.freedom
[3] http://chronicle.nytlabs.com/?keyword=men.women
[4] http://chronicle.nytlabs.com/?keyword=homophobic.homosexual....
[5] http://chronicle.nytlabs.com/?keyword=love.money
http://chronicle.nytlabs.com/?keyword=his.her
Surprisingly, the difference seems to be be fairly constant.
It'd also be useful to see a curated list of topics to select from, instead of just randomly picking some related topics on page load. Hopefully they harvest some interesting suggestions they receive through @nyt. There are some fun topic-suggestions in this thread already.
Overall, a nifty tool that I wish existed for all news sources in the world and worked across all languages.
Thanks to the creator(s)!
http://chronicle.nytlabs.com/?keyword=vietnam.korea.iraq.rus...
Also, a nice way of visualizing just how far back this data reaches: http://chronicle.nytlabs.com/?keyword=reconstruction
http://chronicle.nytlabs.com/?keyword=beatles.bieber
http://chronicle.nytlabs.com/?keyword=new%20york&format=coun...
It was probably a change in style or format, not a reduction in the number of stories about New York, because the percentages of stories about other cities drop in a similar way at the same time.
http://chronicle.nytlabs.com/?keyword=new%20york.washington....
Bonus... the end of that chart, I think you can see when the NYT really did begin to expand its proportion non-local news.
Publishing schedule changes? http://chronicle.nytlabs.com/?keyword=saturday.sunday.monday...
Strong correlation: http://chronicle.nytlabs.com/?keyword=moon.mars
Each peak higher than the previous leader: http://chronicle.nytlabs.com/?keyword=microsoft.google.faceb...
Resilience: http://chronicle.nytlabs.com/?keyword=computer.phone
And then there was one: http://chronicle.nytlabs.com/?keyword=fax.iphone
Exponential decrease of area: http://chronicle.nytlabs.com/?keyword=telegram.fax.iphone
There's still hope: http://chronicle.nytlabs.com/?keyword=science.pop%20music
We meet again, after almost a century: http://chronicle.nytlabs.com/?keyword=america.russia
The baby boomers had it so good: http://chronicle.nytlabs.com/?keyword=marriage.divorce
Didn't last: http://chronicle.nytlabs.com/?keyword=blessings
Interesting: http://chronicle.nytlabs.com/?keyword=luck
Added:
Increasing aspirations or inflation? http://chronicle.nytlabs.com/?keyword=millionaire.billionair...
Obligatory: http://chronicle.nytlabs.com/?keyword=hacker
In Chronicle-golf, the most common word I can find is "new" at 74% of articles in 1978 (which beats out "who","what","when", "why", and "how"...).
I wonder how much of the change is terms becomes slurs and how much is NYT racism.
http://chronicle.nytlabs.com/?keyword=assault.attack
Ok, what happened in 1871?
http://en.wikipedia.org/wiki/National_Recovery_Administratio...
You could try to adjust for that by comparing Republican vs. the sum of Democratic+Democrat, but that also pulls in unrelated uses of both terms: "democratic reforms in $countryname" and "Irish republicans", especially since it isn't case-sensitive. Which then probably overcounts "democratic", because it's used in that non-US-party sense more than "republican" is.
In general this kind of thing makes it very tricky to conclude things from pure word or n-gram frequency counts, since without more semantic annotation there are a ton of confounding issues.