Zipf’s Law Arises Naturally When There Are Underlying, Unobserved Variables

Re:Zipf’s Law Arises Naturally When There Are Underlying, Un... by rahimnathwani on Friday December 30, 2016 @05:06AM (#13283610)

"though the paper concludes that the distribution is a good representation of common Chinese words"

I don't know why the word distribution doesn't follow a power law, but I can tell you that it isn't a good representation of common Chinese words. I downloaded the data set a while back, intending to use it for improving my vocabulary. I found some words (like the Chinese word for 'vampire') had too high a frequency.

The lists are based on American TV shows' Chinese fansubs. I guess vampires and zombies are common themes.

[ Parent | on HN ]

Re:Zipf’s Law Arises Naturally When There Are Underlying, Un... by wodenokoto on Friday December 30, 2016 @06:54AM (#13283919)

I completely agree that the lists looks odd in many regards, but they test it on humans, and find correlations with frequency in the list and how quickly native speakers can recognise the word.

I don't consider myself an expert in word-frequencies, but the article makes a convincing argument that these frequencies are representative of natural Chinese and a better or just as good measure as their baseline corpus.

[ Parent | on HN ]