News for hackers. Stuff that matters.
Re:Exploration of NYT Crossword answers from 1994-2017 by rabidrat on Thursday September 07, 2017 @11:14AM (#15192584)
It sounds like they wanted to do the scraping themselves, but for anyone else who wants to play with this data, it's available for download at http://xd.saul.pw/data.
Re:Exploration of NYT Crossword answers from 1994-2017 by slig on Thursday September 07, 2017 @01:01PM (#15193705)
Also here: https://github.com/doshea/nyt_crosswords and a couple of other repos on GH.
Re:Exploration of NYT Crossword answers from 1994-2017 by rabidrat on Thursday September 07, 2017 @01:38PM (#15194055)
Thanks for posting this, but not all data is created equal. These crosswords only go back to 1976, are in .json format, and require nontrivial processing to get into a useful format for data analysis. By contrast, I very meticulously designed the formats available at xd.saul.pw to be easy to use. You can even extract some interesting results with just e.g. `grep` and `wc`.
Re:Exploration of NYT Crossword answers from 1994-2017 by slig on Thursday September 07, 2017 @01:51PM (#15194150)
Thank you, I did not realize at first that the link you posted is so complete! I PM'd you on reddit a little while ago asking for the dataset, but didn't have the time to play with it yet. Thanks again for your time and consideration.
Re:Exploration of NYT Crossword answers from 1994-2017 by pwaivers on Thursday September 07, 2017 @01:46PM (#15194117)
This was actually really fascinating to me. I loved to see all the words that are important:

- nsfw - This one is hilarious. Of all the internet terms (e.g. AFAIK, AMA, LOL), this is the most used.

- laiditontheline - Why is this important in 2016? Does "Laid it on the line" have a particular meaning?

- idriselba - This one is great too. Stringer Bell!

Re:Exploration of NYT Crossword answers from 1994-2017 by jordanpg on Thursday September 07, 2017 @11:52AM (#15192961)
It's interesting that word length emerges as such a clear proxy for difficulty. I often find it easier to think of a longer word or phrase (assuming no wordplay) than something short and obscure -- the parameter space is much smaller.

Also interesting to see the step increase between Thursday and Friday. This finding is consistent with my anecdotal experiences, but I doubt it's deliberate on the part of the editors. I think there is probably a technical explanation for this. For example, maybe there are fewer words available of medium to long length between the set of shorter words and multi-word answers, so no smooth transition between Thursday and Friday is possible.

Re:Exploration of NYT Crossword answers from 1994-2017 by finnh on Thursday September 07, 2017 @01:02PM (#15193723)
Sunday's word length slots in nicely between Thursday & Friday, which matches what I've read the NYT editors desire (for Sunday to be "roughly as difficult as thu/fri") ... and gainsays your hypothesis =)

Rather than the set of possible answers being U-shaped, maybe it's a size thing: it is hard to fit in N medium-sized words unless you have the extra space of the Sunday puzzle.

The top & bottom of Fri/Sat puzzles are often split into just two words (one long, one short), this might inform our developing hypothesis.

Re:Exploration of NYT Crossword answers from 1994-2017 by vinchuco on Thursday September 07, 2017 @03:53PM (#15195138)
> word length as a proxy for difficulty

Frequency of words by string length: https://www.researchgate.net/figure/229065838_fig1_Figure-1-...

Re:Exploration of NYT Crossword answers from 1994-2017 by ChristianGeek on Thursday September 07, 2017 @11:45AM (#15192894)
That third chart is a thing of beauty!
Re:Exploration of NYT Crossword answers from 1994-2017 by hackeraccount on Thursday September 07, 2017 @03:08PM (#15194820)
what was the clue for CCCCCCC?

14 across - sound made by someone choking