Re:Tokenising the english text of 30TB common crawl
by monatron on Monday December 12, 2011 @08:16AM
(#3343430)
Very interesting. Surprising to see Lithuania as the second most popular language. Oddly enough, I was having a conversation with someone this weekend when they mentioned that Lithuania had the highest fiber-to-the-home penetration in Europe and one of the fastest average internet connections in the world. Does anyone know why this is?
[ Parent
| on HN ]
Re:Tokenising the english text of 30TB common crawl
by malkung on Monday December 12, 2011 @08:53AM
(#3343508)
I once tried to identify the language in (supposedly) English-language Facebook posts. The second-most frequent language was Estonian. Looking at actual posts classified as Estonian, it turned out they contained text like "Soooooo nice". Because the language identifier looks at character sequences, and Estonian tends to have unusually common double vowels, such posts were classified as Estonian. Dutch and Norwegian were other two languages for which English was mistaken - apparently some character sequencies there have frequency distributions that are similar to English.
Re:Tokenising the english text of 30TB common crawl
by mat_kelcey on Monday December 12, 2011 @11:18PM
(#3346260)
I think the Lithuania bit is a false alarm. The pipeline needs a lot more work and investigation of results. The main thing I was focusing on first was scaling it out.