Show HN: 3d Word Vectors with Neural Networks, T-SNE, and WebGL

Re:Show HN: 3d Word Vectors with Neural Networks, T-SNE, and... by sp332 on Thursday November 14, 2013 @11:32AM (#6733248)

How did you extract text data from Wikipedia? I've been trying to clean up an XML dump but I'm having trouble getting rid of the wikitext markup.

[ Parent | on HN ]

Re:Show HN: 3d Word Vectors with Neural Networks, T-SNE, and... by dave_sullivan on Thursday November 14, 2013 @05:34PM (#6736177)

Check out this page: http://mattmahoney.net/dc/textdata.html

[ Parent | on HN ]

Re:Show HN: 3d Word Vectors with Neural Networks, T-SNE, and... by sp332 on Thursday November 14, 2013 @06:17PM (#6736456)

Thanks, but I've tried that and it gives really messy results. For example, every accented character turns into a space, dividing the word into two meaningless pieces.

Edit: it also seems to put the entire article on a single line with no punctuation. But Google's word2vec examples run with each sentence on one line. Wouldn't this make a difference in training?

[ Parent | on HN ]