News for hackers. Stuff that matters.
Re:Data-Mining Wikipedia for Fun and Profit by papercrane on Thursday August 19, 2021 @10:49AM (#28234529)
For this particular problem I wonder if wikidata would be better instead of scraping the HTML.
Re:Data-Mining Wikipedia for Fun and Profit by dalf on Thursday August 19, 2021 @01:33PM (#28236784)
Yes:

* http://www.entitree.com/en/family_tree/Elizabeth_II

* https://family.toolforge.org/ancestors.php?q=Q187114

Tools found on this page: https://www.wikidata.org/wiki/Wikidata:Tools/Visualize_data/...

---

Some SPARQL queries: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/...

---

Out of topic: I wish wikipedia would provide an API to get the infoboxes (made using Lua or wikidata).

Re:Data-Mining Wikipedia for Fun and Profit by zaik on Thursday August 19, 2021 @03:16PM (#28238072)
There is a tool to get infobox data from Wikipedia into Wikidata: https://pltools.toolforge.org/harvesttemplates/

The easily parsable Infobox data can probably already be found in Wikidata (assuming there is a property).

Re:Data-Mining Wikipedia for Fun and Profit by beardyw on Friday August 20, 2021 @09:16AM (#28245574)
So am I right in thinking Wikidata is sort of scraped from Wikipedia internally?

The other way round seems better, but obviously too late.

Re:Data-Mining Wikipedia for Fun and Profit by dalf on Friday August 20, 2021 @12:40PM (#28248088)
A lot of items don't have a wikipedia page, but all wikipedia, wikitravel, wiki* pages have a wikidata item (see "Wikidata item" link of the left side of the pages).

Some wikipedia infoxes are based on wikidata. I can't find an example, but here some links:

* https://commons.wikimedia.org/wiki/Commons:Wikidata_infobox_...

* https://commons.wikimedia.org/wiki/Template:Wikidata_Infobox

* https://en.wikipedia.org/wiki/Template:Infobox_person/Wikida...

There are lexeme too, and it is not based on wiktionary. Search the prefix "L:" (without quote).

Example:

* https://www.wikidata.org/w/index.php?search=L%3Acat&search=L...

* https://www.wikidata.org/wiki/Lexeme:L7

Also, there are a lot of tools on toolforge.org. One is reasonator which produce sentences from a wikidata item: https://reasonator.toolforge.org/?q=Q1339

Re:Data-Mining Wikipedia for Fun and Profit by billpg on Thursday August 19, 2021 @10:57AM (#28234642)
(Author here.)

Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley.

Also, it was only my reading your comment just now that told me wikidata was even a thing.

Re:Data-Mining Wikipedia for Fun and Profit by 3pt14159 on Thursday August 19, 2021 @11:04AM (#28234740)
Don't worry about the haters. You needed a paltry amount of data and you got it with the tools you had and knew.

When I was analyzing Wikipedia about 10 years ago for fun and, later, actual profit. I did the responsible thing and downloaded one of their megadumps because I needed every English page. That's what people here are concerned about, but it doesn't matter for your use case.

Re:Data-Mining Wikipedia for Fun and Profit by tcgv on Thursday August 19, 2021 @03:49PM (#28238477)
> Don't worry about the haters

To be fair, the original comment just made a valid observation in a casual way, he didn't criticize the approach of the OP, nor was he impolite.

But I know it's pretty common to see haters nitpicking things all around ;)

Re:Data-Mining Wikipedia for Fun and Profit by shock-value on Thursday August 19, 2021 @11:03AM (#28234731)
Generally Wikidata would definitely be the way to go here, though I just now tried to retrace your graph in Wikidata and it seems to be missing at least one relation (Ada of Holland has no children listed -- https://www.wikidata.org/wiki/Q16156475).
Re:Data-Mining Wikipedia for Fun and Profit by boxed on Friday August 20, 2021 @03:01PM (#28250034)
I am doubtful. I tried for a long time to use it to get data or for my taxonomic graph project (https://relatedhow.kodare.com/) and SPARCQL was just not usable at all. The biggest problem was the 60s time limit. Totally not workable for what I wanted. I also had issues with seemingly inconsistent results, but it was hard to tell.

I ended up loading the full nightly db dump and filtering it streaming from the zip instead. Faster and it actually worked.

The code to do that is at https://github.com/boxed/relatedhow

Re:Data-Mining Wikipedia for Fun and Profit by shock-value on Saturday August 21, 2021 @01:13AM (#28254405)
There's an alternate Wikidata query engine available here: https://qlever.cs.uni-freiburg.de/wikidata (from https://github.com/ad-freiburg/QLever)

Currently it doesn't support some SPARQL features, but I've found it to generally be quite a bit faster for most queries.

Re:Data-Mining Wikipedia for Fun and Profit by papercrane on Thursday August 19, 2021 @11:55AM (#28235427)
Hard to use it if you don't know about it!

I only thought of it myself because you mentioned the problem with deducing which parent is the mother and which is the father, and I remember in wikidata those are separate fields.

Re:Data-Mining Wikipedia for Fun and Profit by hutzlibu on Thursday August 19, 2021 @02:57PM (#28237809)
Yup, I had the same situation some months before, even though I knew wikidata was a thing.

I know javascript and had the pages at hand.

I looked at wikidata and some pages about, but still had no clear idea how to use it and no motivation to digg into it. Because js just worked with a small custom script, to retrieve some pages and data.

Re:Data-Mining Wikipedia for Fun and Profit by mistrial9 on Thursday August 19, 2021 @11:02AM (#28234709)
earth calling ivory tower, earth calling ivory tower

superior RDF triples are like martian language to millions of humans

over

Re:Data-Mining Wikipedia for Fun and Profit by udp on Thursday August 19, 2021 @11:03AM (#28234736)
The ivory tower is working on it: https://github.com/w3c/EasierRDF
Re:Data-Mining Wikipedia for Fun and Profit by tasogare on Thursday August 19, 2021 @05:16PM (#28239479)
RDF has to be the best and saddest example of sunk cost fallacy. Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers, its cultists are double downing on their abstruse technology stack, making it always more complicated while still not addressing any of its fundamental problems.
Re:Data-Mining Wikipedia for Fun and Profit by udp on Friday August 20, 2021 @11:12AM (#28246951)
I mean, imho RDF isn't the problem. RDF itself is very simple. As you correctly point out, the stack is overcomplicated.

> Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers

neo4j is basically this. You can also load RDF into neo4j using neosemantics and query it using Cypher instead of using a conventional triplestore with SPARQL, which is nice.

Re:Data-Mining Wikipedia for Fun and Profit by namedgraph on Friday August 20, 2021 @03:08AM (#28243452)
Let us see your proposal of the superior model?

RDF was designed primarily for data interchange and there's nothing that beats it at that.

Re:Data-Mining Wikipedia for Fun and Profit by tasogare on Friday August 20, 2021 @12:56PM (#28248332)
Nothing beats it for data exchange? You must be joking, because if it were remotely true RDF would be in wide use, which is totally not. Except a few niche domains like bioinformatics, it is not used. No killer application use it as a data format, no popular data format is based on it either. Actually I can think of a single data format based on RDF, and the only open-data I know which use it have been converted to it and was simplier to use in their original format.

And for the model: property graph. But yeah, enjoy your Stockholm syndrome with your model where reification is required to annotate an edge. Also even your nickname is an aknowledgment of RDF failure: named graphs (n-quads) were created because RDF triples aren't good enough for modeling data.

Re:Data-Mining Wikipedia for Fun and Profit by namedgraph on Friday August 20, 2021 @04:03PM (#28250754)
Yes, let us see how you do data interchange without global identifiers. Such as URIs, which RDF has built-in natively and property graphs do not.

You're right about bioinformatics, but lets do a quick check on http://sparql.club/ on who else is looking for RDF/SPARQL specialists. Oh look: automotive industry, finance, publishing, medical, research etc.

Re:Data-Mining Wikipedia for Fun and Profit by liminal on Thursday August 19, 2021 @05:41PM (#28239724)
I just looked at the linked repo. Have they made any progress? It looks like _very_ early days.
Re:Data-Mining Wikipedia for Fun and Profit by bawolff on Thursday August 19, 2021 @06:15PM (#28240032)
Are you saying parsing html is easier than parsing rdf triples? Because i dont know about that.

In real life you use tools for both.

Re:Data-Mining Wikipedia for Fun and Profit by namedgraph on Friday August 20, 2021 @03:08AM (#28243455)
Triples are already structured, machine-readable data. HTML is not.
Re:Data-Mining Wikipedia for Fun and Profit by brodo on Thursday August 19, 2021 @11:00AM (#28234687)
It’s so sad that almost nobody knows or uses SPARQL…
Re:Data-Mining Wikipedia for Fun and Profit by matkoniecz on Thursday August 19, 2021 @11:10AM (#28234830)
In my experience SPARQL is really hard to use, and Wikidata data quality is really low. To the point that one of my larger project is trying to filter data to make it usable for my usecase.

Yes, I made some improvements ( https://www.wikidata.org/wiki/Special:Contributions/Mateusz_... ).

But overall I would not encourage using it, if I would know how much work it takes to get usable data I would not bother with it.

Queries as simple as "is this entry describing event, bridge or neither" are requiring extreme effort to get right in a reliable way, including maintaining private list of patches and exemptions.

And bots creating millions of known duplicated entries and expecting people to resolve this manually is quite discouraging. Creating Wikidata entries for Cebuano Wikipedia 'articles' was accepted, despite that Cebuano botpedia is nearly completely bot-generated.

And that is without unclear legal status. Yes, they can legally import databases covered by database rights - but they should either make clear that Wikidata is a legal quagmire in EU or forbid such imports. But Wikidata community did neither.

Re:Data-Mining Wikipedia for Fun and Profit by namedgraph on Friday August 20, 2021 @03:10AM (#28243465)
Who knew that a global machine-readable knowledge base would involve some complexity?
Re:Data-Mining Wikipedia for Fun and Profit by matkoniecz on Saturday August 21, 2021 @02:47AM (#28254824)
"is this entry describing (a) bridge (b) event" should have some reasonable way to answer.

So far I have not found way to achieve this without laboriously maintaining my own database of errata, and new exceptions keep appearing.

Re:Data-Mining Wikipedia for Fun and Profit by devbas on Thursday August 19, 2021 @11:09AM (#28234812)
Because the syntax is relatively complex and it is difficult to judge which endpoints and definitions to use.
Re:Data-Mining Wikipedia for Fun and Profit by ZeroGravitas on Thursday August 19, 2021 @02:11PM (#28237249)
I learned SPARQL recently, and would agrre its complicated to get info out of Wikidata.

However, having read the article, they didnt have an easy time with scraping Wikipedia either.

So I'd probably still recommend people look into wikidata and SPARQL if they want to do this kind of thing.

Theres a few tools that generate queries for you, and some cli tools as well:

https://github.com/maxlath/wikibase-cli#readme

It makes Wikipedia better too, in a virtuous cycle, with some infoboxes like those that he scraped being converted to be automatically populated from wikidata.

Re:Data-Mining Wikipedia for Fun and Profit by guidovranken on Thursday August 19, 2021 @11:24AM (#28235034)
At least the last times I checked, the WikiData SPARQL server was extremely slow, frequently timing out.
Re:Data-Mining Wikipedia for Fun and Profit by epaulson on Thursday August 19, 2021 @06:04PM (#28239947)
There's some mix between "it's slow" and "it sets its timeout threshold too low" - a lot of queries would be OK if they just had a bit more time to run. And unfortunately, the time wasted on the badput of the killed queries just slows down everyone else. (They really need a batch queue)

The Wikidata folks are well aware of the limits on their SPARQL service. They just posted an update the other day:

https://lists.wikimedia.org/hyperkitty/list/[email protected]...

Re:Data-Mining Wikipedia for Fun and Profit by lacksconfidence on Thursday August 19, 2021 @12:24PM (#28235816)
seems to depend on the query. I can issue straight forward queries that visit a few hundred thousand triples easily. But when i write a query that visits tens of millions of triples it times out.
Re:Data-Mining Wikipedia for Fun and Profit by pphysch on Thursday August 19, 2021 @11:10AM (#28234837)
Why should more people know SPARQL?
Re:Data-Mining Wikipedia for Fun and Profit by namedgraph on Friday August 20, 2021 @03:04AM (#28243427)
E.g. to get a job in FAANG, finance or pharma where SPARQL is used extensively on enterprise Knowledge Graphs? Check the jobs here: http://sparql.club/
Re:Data-Mining Wikipedia for Fun and Profit by pphysch on Friday August 20, 2021 @12:56PM (#28248330)
The few openings I flipped through all mention SPARQL in an offhand manner, in the sense of "familiarity with query languages and data ontology".
Re:Data-Mining Wikipedia for Fun and Profit by powera on Thursday August 19, 2021 @10:57AM (#28234643)
Definitely. There's even a public query service ( https://query.wikidata.org/ ) which can do a lot of this (though SQL is not good with searching for chains).
Re:Data-Mining Wikipedia for Fun and Profit by bawolff on Thursday August 19, 2021 @07:48PM (#28240747)
Note: that's the sparql endpoint. Much better at searching chains than SQL (but easy to make slow queries that timeout).

The SQL endpoint is at https://quarry.wmflabs.org/ however it doesn't have the actual data so much as metadata (mostly) so its not super useful.

Re:Data-Mining Wikipedia for Fun and Profit by squeaky-clean on Thursday August 19, 2021 @12:38PM (#28236004)
Like people on the other comment have said, if you've actually tried getting data from wikidata/wikipedia you very quickly learn the HTML is much easier to parse than the results wikidata gives you.
Re:Data-Mining Wikipedia for Fun and Profit by mark_l_watson on Friday August 20, 2021 @08:38AM (#28245315)
+1 use WikiData’s SPARQL endpoint.

Still, it was a cool article and a good example of scraping information.