News for hackers. Stuff that matters.
Re:Wikipedia's in Trouble (2019) by PaulHoule on Friday December 04, 2020 @10:29AM (#25303314)
If you are extracting information from Wikipedia and trying to parse the markup you're doing it wrong. It's the difference between "screwed around with handwritten parsers for years and it still doesn't work 100% right" and "look this CSS selector extracts what you want."

Freebase, DBpedia, and many others (me) have tried, but the reality is that the markup language is poorly defined and the only path that is really tested is the one that ends up rendering HTML.

If you feed HTML from Wikipedia into a web parser that supports the DOM (say Beautiful Soup) you can generally parse out what you want pretty effectively. Once I switched from the "markup rabbithole" to "parsing standard HTML" I was able to turn my MediaWiki extractor into a Flickr extractor in about 15 minutes.

Re:Wikipedia's in Trouble (2019) by indymike on Friday December 04, 2020 @10:53AM (#25303682)
This is exactly what I was thinking. Wikipedia articles carry semantic data (ld-json), so it should be pretty easy to just scrape away.
Re:Wikipedia's in Trouble (2019) by PaulHoule on Friday December 04, 2020 @12:06PM (#25304808)
The LD-JSON data is quite limited. I looked at this page

https://en.wikipedia.org/wiki/Thomas_Sowell

I went and used this

https://addons.mozilla.org/en-US/firefox/addon/openlink-stru...

and the JSON-LD data comprised of "wrapper metadata" for the document, a photograph of that person, and a link to a concept in Wikidata where you would find some of the facts that you'd be looking for if you mistook the wikipedia page for the person.

The page at Wikidata for that person

https://www.wikidata.org/wiki/Q553449

has the birth date, gender and other facts such as he taught at both Cornell and Stanford and also links to identifiers for that person in many bibliographic databases.

Re:Wikipedia's in Trouble (2019) by bawolff on Friday December 04, 2020 @03:32PM (#25307446)
There's also an alternate html output that has more semantic tags (and the html is consistent enough there are parsers to take this html and turn it back into wikitext): https://en.wikipedia.org/api/rest_v1/page/html/Albert_Einste...