News for hackers. Stuff that matters.
Re:Data-Mining Wikipedia for Fun and Profit by jancsika on Thursday August 19, 2021 @11:59AM (#28235493)
> The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps.

How is it possible that "give me all the infoboxes, please" is more than a single query, download, or even URL at this point?

Re:Data-Mining Wikipedia for Fun and Profit by ceejayoz on Thursday August 19, 2021 @12:13PM (#28235675)
The problem lies in parsing them.

Look at the template for a subway line infobox, for example. https://en.wikipedia.org/wiki/Template:Bakerloo_line_RDT

It's a whole little clever language (https://en.wikipedia.org/wiki/Wikipedia:Route_diagram_templa...) for making complex diagrams out of rather simple pictograms (https://commons.wikimedia.org/wiki/Template:Bsicon).

Re:Data-Mining Wikipedia for Fun and Profit by jancsika on Thursday August 19, 2021 @05:17PM (#28239492)
Oh wow.

But every other infobox I've seen has key/value pairs where the key was always a string.

So what's the spec for an info box? Is it simply to have a starting `<table class="hello_i_am_infobox">` and an ending `</table>`?

Re:Data-Mining Wikipedia for Fun and Profit by bawolff on Thursday August 19, 2021 @06:27PM (#28240152)
En wikipedia has some standards. Generally though they are user-created tables and its up to the users to make them consistent (if they so desire). En Wikipedia generally does, but its not exactly a hard garuntee.

If you want machine readable use wikidata (if you hate rdf you can still scrape the html preview of the data)

Re:Data-Mining Wikipedia for Fun and Profit by cldellow on Thursday August 19, 2021 @12:18PM (#28235736)
Even just being able to download a tarball of the HTML of the infoboxes would be really powerful, setting aside the difficulty of, say, translating them into a consistent JSON format.

That plus a few other key things (categories, opening paragraph, redirects, pageview data) enable a lot of powerful analysis.

That actually might be kind of a neat thing to publish. Hmmmm.

Re:Data-Mining Wikipedia for Fun and Profit by jancsika on Thursday August 19, 2021 @05:19PM (#28239503)
Better yet-- what is the set of wikipedia articles which have an info box that cannot be sensibly interpreted as key/value pairs where the key is a simple string?
Re:Data-Mining Wikipedia for Fun and Profit by squeaky-clean on Thursday August 19, 2021 @12:37PM (#28235980)
The infoboxes aren't standardized at all. The HTML they generate is.
Re:Data-Mining Wikipedia for Fun and Profit by jancsika on Thursday August 19, 2021 @05:23PM (#28239547)
Hehe-- I am going to rankly speculate nearly all of them follow an obvious standard of key/value pairs where the key is a string. And then there are like two or three subcultures on Wikipedia that put rando stuff in there and would troll to the death before being forced to change the their infobox class to "rando_box" or whatever negligible effort it would take them if a standard were to be enforced.

Am I anywhere close to being correct?

Re:Data-Mining Wikipedia for Fun and Profit by bawolff on Thursday August 19, 2021 @06:29PM (#28240174)
I think you'll have to more clearly define what you mean by "key-value" pairs.