News for hackers. Stuff that matters.

Show HN: Graph of wikipedia articles semantic similarity (LSI, Python, d3.js)

http://similarityapi.appspot.com/graph/?title=cyborg

Small experiment of visualization of wikipedia articles as a graph using d3.js.<p>Articles with more traffic are bigger. I computed the semantic similarity using LSI with python (gensim) You have to scroll down/right a bit!<p>http://similarityapi.appspot.com/graph/?title=blade%20runner<p>There is also a JSON api: http://similarityapi.appspot.com/api/v1/?limit=100&title=blade%20runner<p>All feedback is appreciated:<p>@lucamartinetti [email protected]
( 13 comments | 54 points | on HN )
Re:Show HN: Graph of wikipedia articles semantic similarity ... by lucamartinetti on Wednesday March 21, 2012 @10:05AM (#3734427)
Small experiment of visualization of wikipedia articles as a graph using d3.js.

Articles with more traffic are bigger. I computed the semantic similarity using LSI with python (gensim) You have to scroll down/right a bit!

http://similarityapi.appspot.com/graph/?title=blade%20runner

There is also a JSON api: http://similarityapi.appspot.com/api/v1/?limit=100&title...

All feedback is appreciated:

@lucamartinetti [email protected]

Re:Show HN: Graph of wikipedia articles semantic similarity ... by 3pt14159 on Wednesday March 21, 2012 @03:19PM (#3736066)
I've had much, much better results with LDA than LSI. Give that a shot if you have a chance, you'll be blown away. Stop word ratios are important, and make the max number of tokens 500,000.
Re:Show HN: Graph of wikipedia articles semantic similarity ... by viscanti on Wednesday March 21, 2012 @11:08AM (#3734705)
The JSON api should degrade gracefully if results aren't found. I.E. There should be a JSON message explaining that that item doesn't exist.
Re:Show HN: Graph of wikipedia articles semantic similarity ... by lucamartinetti on Wednesday March 21, 2012 @11:29AM (#3734791)
Right! It could use some input checking / normalization too. It expects the title parameter to be lower case now.
Re:Show HN: Graph of wikipedia articles semantic similarity ... by rplnt on Wednesday March 21, 2012 @12:04PM (#3734968)
Option to select language version could be a good feature (defaulting to en as now).
Re:Show HN: Graph of wikipedia articles semantic similarity ... by Radim on Wednesday March 21, 2012 @03:29PM (#3736133)
how much data did you use for the semantic analysis?
Re:Show HN: Graph of wikipedia articles semantic similarity ... by lucamartinetti on Wednesday March 21, 2012 @04:42PM (#3736507)
The whole text of all articles from wikipedia english (then filtered those with more the 1k views last month)
Re:Show HN: Graph of wikipedia articles semantic similarity ... by Edootjuh on Wednesday March 21, 2012 @01:09PM (#3735296)
I've never liked these scrolling animations. You need too much precision to see a part of the page clearly, while with normal scrolling it wouldn't matter if the information you're reading is at the bottom or top of the screen.
Re:Show HN: Graph of wikipedia articles semantic similarity ... by stephengoodwin on Wednesday March 21, 2012 @12:29PM (#3735104)
Does the font size for a node represent it's similarity with the query page?
Re:Show HN: Graph of wikipedia articles semantic similarity ... by lucamartinetti on Wednesday March 21, 2012 @04:35PM (#3736471)
It represents the traffic of the article. Ten most related articles are displayed for each expanded node. Articles with more inbound links are darker
Re:Show HN: Graph of wikipedia articles semantic similarity ... by lucian1900 on Wednesday March 21, 2012 @12:30PM (#3735109)
Blank page in Chrome.
Re:Show HN: Graph of wikipedia articles semantic similarity ... by ssn on Wednesday March 21, 2012 @11:48AM (#3734895)
Down?
Re:Show HN: Graph of wikipedia articles semantic similarity ... by lucamartinetti on Wednesday March 21, 2012 @12:08PM (#3734990)
Not for me. You need a modern browser (chrome or firefox) and scroll a bit