Extracting data from Wikipedia using curl, grep, cut and other bash commands

Re:Extracting data from Wikipedia using curl, grep, cut and ... by lacksconfidence on Monday August 15, 2016 @04:34PM (#12293209)

Because i was randomly curious, can extract this data from the structured html with some dom selectors in a similarly haphazard way:

Start with: https://en.wikipedia.org/api/rest_v1/page/html/List_of_Olymp...

Run this js one-liner: [].slice.call(document.querySelectorAll('table[typeof="mw:Transclusion mw:ExpandedAttrs"] tr td:nth-child(n+2) > a:nth-child(1), table[typeof="mw:Transclusion mw:ExpandedAttrs"] tr:nth-child(3) td > a:nth-child(1)')).map(function(e) { return e.innerText; }).reduce(function(res,el) { res[el] = res[el] ? res[el] + 1 : 1; return res; }, {});

The result is an object with the medalists as keys, and the count as values. JS objects are unordered so sorting is left as an excercise for the reader.

[ Parent | on HN ]

Re:Extracting data from Wikipedia using curl, grep, cut and ... by austinjp on Tuesday August 16, 2016 @05:53AM (#12296407)

Nice. Nasty, but nice :)

I can't help but notice a small bug.... Driulis Gonzalez for example has medalled 4 times, but your script gives his count as only 3. Similarly Gévrise Émane isn't listed by your script. Something to do with split tables cells I suspect.

Still, it's inspiring. I've often used bash and perl to scrape data from web pages. I'll definitely consider JS in future.

[ Parent | on HN ]

Re:Extracting data from Wikipedia using curl, grep, cut and ... by merpnderp on Monday August 15, 2016 @05:01PM (#12293414)

Wish I could upvote this more. That is some serious queryselector fu.

[ Parent | on HN ]