Learning Word Vectors for 157 Languages

Re:Learning Word Vectors for 157 Languages by sp332 on Friday February 23, 2018 @02:20PM (#16448612)

I guess I can't argue with the results, but the data doesn't look very clean. I downloaded the Esperanto file because I expected it to be small, but it was a 1.1 GB download that expanded to 4.4 GB. A lot of the most popular "words" are punctuation marks, and the first two real words are "la" and "La". Perhaps naively, I expected those to be the same word.

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by exgrv on Friday February 23, 2018 @04:13PM (#16449674)

We decided to keep the casing, as it is useful for some applications such as named entity recognition.

Regarding the punctuation, as pointed out in another comment, these tokens might also be useful for some applications (and they are easy to filter out if you don't need them).

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by sp332 on Friday February 23, 2018 @04:19PM (#16449722)

In the Tagalog file, } is near the top but { is over 8,000 lines down. Is there a reason they have such different frequencies? ( and ) are right next to each other.

And yes I realize this is a really odd question :)

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by exgrv on Friday February 23, 2018 @04:29PM (#16449788)

This is probably due to our preprocessing of Wikipedia that did not get rid of all the '}' from the markup.

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by sp332 on Friday February 23, 2018 @04:32PM (#16449810)

Oh true. I tried to clean up Wiki markup for ML years ago and it was a huge pain. Next time I think I'll parse the HTML version and pull out the text from the tags explicitly.

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by mkl on Friday February 23, 2018 @11:40PM (#16451701)

This is a much better way to do it. It's easier, cleaner, and gets the text which is generated by templates, which there is a surprising amount of (you get weird artifacts from that otherwise).

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by jauntbox on Friday February 23, 2018 @04:25PM (#16449759)

Your comment has twice as many ) as it does (

My first guess would be emojis ;)

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by gojomo on Friday February 23, 2018 @03:50PM (#16449495)

Does a word-vector dataset that "looks clean" perform better on real applications, justifying the extra attention required for more preprocessing/cleanup?

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by alexott on Friday February 23, 2018 @02:31PM (#16448689)

In some tasks punctuation marks may play significant role, but I agree that for things like text classification, they may not be necessary...

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by posterboy on Friday February 23, 2018 @05:51PM (#16450326)

yes, me too. I like cooking dogs and my family by he way.

[ Parent | on HN ]