Learning Word Vectors for 157 Languages

Re:Learning Word Vectors for 157 Languages by sp332 on Friday February 23, 2018 @04:19PM (#16449722)

In the Tagalog file, } is near the top but { is over 8,000 lines down. Is there a reason they have such different frequencies? ( and ) are right next to each other.

And yes I realize this is a really odd question :)

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by exgrv on Friday February 23, 2018 @04:29PM (#16449788)

This is probably due to our preprocessing of Wikipedia that did not get rid of all the '}' from the markup.

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by sp332 on Friday February 23, 2018 @04:32PM (#16449810)

Oh true. I tried to clean up Wiki markup for ML years ago and it was a huge pain. Next time I think I'll parse the HTML version and pull out the text from the tags explicitly.

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by mkl on Friday February 23, 2018 @11:40PM (#16451701)

This is a much better way to do it. It's easier, cleaner, and gets the text which is generated by templates, which there is a surprising amount of (you get weird artifacts from that otherwise).

[ Parent | on HN ]

Re:Learning Word Vectors for 157 Languages by jauntbox on Friday February 23, 2018 @04:25PM (#16449759)

Your comment has twice as many ) as it does (

My first guess would be emojis ;)

[ Parent | on HN ]