Re:Json-wikipedia
by DenisM on Monday November 11, 2013 @11:05AM
(#6712222)
So, does it parse the markdown of the text as well? I could not quite get it from the documentation. It would be quite exciting if it could - the markdown is notoriously complex, and creates a lot of problem if all you want is te body of text (e.g. for training a machine learning algorithm).
[ Parent
| on HN ]
Re:Json-wikipedia
by TorKlingberg on Monday November 11, 2013 @12:36PM
(#6712768)
Wikipedia articles are not written in markdown but in wikicode. Wikicode is much more complex to parse, but allows more complex formatting. You may be looking for the word 'markup' which is a general term that included markdown, HTML, wikicode, bbcode etc.
I doubt this library parses the wikicode. What format would it parse it into? As matthewarkin said, there is a Wikipedia API to convert wikicode into HTML.
It does that indeed! It will process a field paragraph, containing a list of parsed paragraphs in the article, as simple UTF8 text. Lists and Tables are collected in separated fields, have a look to the Article object. For the wikicode parsing I relied on the JWPL Library http://www.ukp.tu-darmstadt.de/software/jwpl/
Re:Json-wikipedia
by matthewarkin on Monday November 11, 2013 @12:29PM
(#6712724)
I doubt it (though I haven't tested it). Wikipedia's API though lets you get an HTML version of a page.