News for hackers. Stuff that matters.
Re:Things I Learned Building The Most Powerful Language Proc... by bane on Friday April 19, 2013 @09:29PM (#5579470)
"200,000 words. Ha! 400K words. I laugh. 3.2 Million words. I still know I am missing stuff. Single word nouns in just the singular form exceeds 150k. 40k verbs and conjugates. 37k adjectives. 10K adverbs. I know I am still missing things."

I was all set to hate this post, but I found I ended up largely agreeing with it, especially the quoted bit above. I remember working on a very focused NLP tool a few years ago and needed a comprehensive English Lexicon. No problem, I'll just scrape WordNet or similar, not even close. Then you start dealing with stemming and conjugations and such and realize that almost all of the algorithms for dealing with this kind of thing would barely even be hacks in software terms, yet there they sit, regurgitated in countless libraries, generating garbage stems all over the place. It ends up just being easier collecting all the stemmed forms as well and just building some smart in-memory indexes and data structures for searching millions of words.

Vector space models? Why do they work? Nobody really seems to know! Just jam all your words into some matrixes and run some simple calculations and voila you get something seems to kind of like it sorta works some of the time.

Sentence tokenization is stupid hard, but shouldn't be, parantheses have all kinds of different meanings, commas are a mess...English is stupid.

The worst bit though really is that most of the research->turned into software assumes astonishingly brittle models about the language which almost never seems to describe any actual usage of the language which always means very frustrating almost right results out of NLP systems. My previous sentence, for example, would cause most NLP software to blow a gasket.

Re:Things I Learned Building The Most Powerful Language Proc... by drakaal on Friday April 19, 2013 @09:47PM (#5579560)
I ran it through our stuff we do fine with segmentation of your comments. We still have trouble with certain poorly formatted numbers, or if you do something like for get a space after a sentence that ends in a number. "I'll take 2.In case I need one later."

Typos are really hard for rules systems.

PS Glad you didn't hate it.

Re:Things I Learned Building The Most Powerful Language Proc... by mchaver on Friday April 19, 2013 @11:10PM (#5579867)
Aren't you arguing against prescriptivism and not linguists (as in people with a background in Linguistics)? My background is Linguistics, Computer Science and NLP. From what I was taught, and what seems to be the norm, is linguists tend to do descriptive linguistics. You learn from your data and change the rules and methods accordingly, not the other way around. Anyway, interesting read.

I am curious about one thing. Are you just handling white spaced words, or do you handle lexical items as well such like polywords (inside out), phrasal verbs (put up with, beat up), idioms, etc.?

Re:Things I Learned Building The Most Powerful Language Proc... by drakaal on Friday April 19, 2013 @11:17PM (#5579894)
We handle polywords. We had to. Noun adjuncts being one of the worst in terms of how often they show up, and I'll be honest I don't even know what you call Gerunds as Adjectives like "Running shorts" but those are a pain too. But you can kind of find ways to get lists of those. Parts catalogs, shopping sites. People want to buy polyword nouns. There are fewer of the verbs but they are harder to find lists of.

Idioms we haven't addressed much. Fortunately in news they come up less than other things. Sarcasm we deal with in most cases, but not all. There are hints that things are sarcastic, but if they are written like "the Onion" and they don't change tone because they are really a parody rather than sarcasm we can't tell that.

I like to say that if your 5 year old would figure it out we will figure it out. If your 5 year old won't probably we won't either. But this is a huge leap forward since the competition is about the level where your dog would understand.

Re:Things I Learned Building The Most Powerful Language Proc... by arafalov on Friday April 19, 2013 @10:15PM (#5579639)
Hah, you thought that was hard!

Try one of these: http://www.undocs.org/A/Res/66/2 (start from finding the end of the first sentence).

Or, you can have a go at a whole corpus of these broken at paragraph level at http://www.uncorpora.org/