News for hackers. Stuff that matters.
Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @02:22AM (#27340824)
Then let those projects fight and advertise for themselves.

People donating to wikipedia only want wikipedia, they are unaware of the other stuff wikimedia does.

Those projects would crash and burn though, because frankly, no one cares about them. Stuff like wikidata and SPARQL endpoints are an academic curiosity with no real value.

The semantic web won't happen, articles do and will.

In addition to that, the whole deletion first and no primary information attitude makes sentences like "ecosystem of free knowledge" laughable.

Re:Wikipedia is swimming in money–why is it begging people t... by zozbot234 on Monday May 31, 2021 @02:30AM (#27340895)
> People donating to wikipedia only want wikipedia, they are unaware of the other stuff wikimedia does.

On the contrary, Wikipedia itself has a lot of reliance on the other projects. For example, Wikidata solves the problem of interlinking Wikipedia languages and other projects when they're hosting content about the same real-world entity. In fact, this was Wikidata's MVP - the reason it got funded in the first place. It then took off from there.

Wikimedia Commons is a similar story - in addition to the obvious languages issue, there are a lot of concerns about media (such as licensing. enhancement etc.) that really are best addressed in a dedicated venue, with its own committed contributors. This has been very beneficial to Wikipedia itself.

Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @02:46AM (#27340984)
Using RDF to store multiple languages of the same article is gross overengineering. Interlinking with other projects isn't nessecary when there are no other projects.
Re:Wikipedia is swimming in money–why is it begging people t... by bawolff on Monday May 31, 2021 @02:55AM (#27341044)
It isn't stored as RDF. RDF is provided as an export format. The language contents also aren't stored in wikidata, its just matching which title in language X matches which title in language Y.

Wikidata isn't just used for interlanguage linking. Its also used to keep infoboxes synced across different languages, and a way to easily query that data (Depends a bit on the language, english wikipedia isn't fully on board with the system, but its been a big boon to smaller languages).

As a result, people can update these "facts" independently of language, keeping them altogether more up to date, and it allows the data to be queried independently (via https://query.wikidata.org - try some of the example queries (button top left) if you haven't, the level of power SPARQL gives for querying this type of data set really is very cool).

Part of Wikimedia's mission is to spread knowledge. Allowing querying of factual data with a query language like sparql helps advance that goal by letting people use extract the knowledge for new purposes.

Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @03:03AM (#27341107)
> its just matching which title in language X matches which title in language Y

Like I said, gross overengineering.

"You can save cents and seconds of time updating knowledge boxes, by spending millions and hours developing federated SPARQL query capabilities!"

Re:Wikipedia is swimming in money–why is it begging people t... by bawolff on Monday May 31, 2021 @03:10AM (#27341149)
More like, we put the facts and knowledge in a central (mysql) db, so that instead of updating them in 300 places (Wikipedia has a lot of languages), you just update them in one place.

And since the data happened to be there anyways, we added an RDF export feature, and threw that into a BlazeGraph DB so that people could do SPARQL queries if they want. The (not really federated) SPARQL queries have basically nothing to do with the central-place-to-store-facts that are used in multiple places, feature, but it was pretty cheap to do once everything was in a central place.

Re:Wikipedia is swimming in money–why is it begging people t... by lifeisstillgood on Monday May 31, 2021 @03:50AM (#27341383)
Is there a entry point for "how to make FOSS contributions to wiki-projects". I at the very least would be interested in finding out how all the pieces fit together, and maybe browsing the big list to see if anything was in my wheelhouse.

Edit: And thank you for some insight into the back end of wikipedia - I did not know multiple languages was such a big push and it is such a sensible direction . Keep up the good work and keep pestering me for cash each year.

Re:Wikipedia is swimming in money–why is it begging people t... by bawolff on Monday May 31, 2021 @04:03AM (#27341466)
https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hac... is probably what you are looking for.

There's a very old architecture doc from 2012: https://www.mediawiki.org/wiki/Manual:MediaWiki_architecture its kind of outdated and missing a lot of the newer pieces.

p.s. I used to work for the Wikimedia foundation about 1.5 years ago, but don't anymore. Prior to working there I was a volunteer for the FOSS project, and in theory I still am but I honestly don't do much anymore. My opinions are my own and don't represent WMF or anyone else.

Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @04:08AM (#27341497)
As if thowing the messy complexities of MySQL queries into the serene simplicity of static hypertext is somehow better.

How often is numerical (which is the only knowledge you can really automate this way) knowledge updated? How often is said knowledge dependent on cultural idiosyncracies? How much effort is the code maintenance of the specialised editing system? How much effort is maintainance of the centralised infrastructure. How much effort is it really to change the data in multiple places, given that enough eyes every bug is shallow? What are the oppirtunity costs of not being able to properly distribute and share articles because they are suddenly tied to a very specific codebase? How much infrastructure cost do I have because I can't get volunteers to host my data?

Overengineering solving the wrong problems at it's finest, no wonder the software has stayed this bad for decades now.

Re:Wikipedia is swimming in money–why is it begging people t... by bawolff on Monday May 31, 2021 @04:26AM (#27341619)
> How often is numerical (which is the only knowledge you can really automate this way) knowledge updated?

Numerical knowledge is not the only knowledge that is stored there.

I'm going to go with update rate of often. It has a higher edit rate than english wikipedia does.

> How often is said knowledge dependent on cultural idiosyncracies?

Sometimes that is true, other times it isn't. The system is used where it makes sense, and not used where it doesn't.

> How much effort is the code maintenance of the specialised editing system? How much effort is maintainance of the centralised infrastructure.

There is certainly some. I wouldn't say its dominating, but it certainly exists. There is plenty of centralized infrastructure for other things too.

Prior to the introduction of wikidata, more complex fragile systems existed to try and keep things in sync in a more manual fashion. They also had a cost.

> How much effort is it really to change the data in multiple places, given that enough eyes every bug is shallow?

A lot. And we don't have that many eyes in lots of languages. There's a lot after all: https://en.wikipedia.org/wiki/Special:SiteMatrix

> What are the oppirtunity costs of not being able to properly distribute and share articles because they are suddenly tied to a very specific codebase

Not sure what this has to do with it. Articles are written in a custom markup language. That's already tying it to a specific code base quite heavily. The wikidata dependency is pretty trivial, and there are open apis to get info out of wikidata, and regularly made dumps of all the data in wikidata if you want to copy it. If that's not good enough, just copy the rendered version instead of the source.

> How much infrastructure cost do I have because I can't get volunteers to host my data

I'm not sure what you're talking about. Wikipedia was never hosted in some magical peer-to-peer fashion on volunteer web servers, nor is there any situation where it is likely to be hosted that way. The goals of (near) instant update time, as well as some central quality control and user blocking, makes distributed hosting pretty impossible.

Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @05:00AM (#27341876)
> I'm not sure what you're talking about. Wikipedia was never hosted in some magical peer-to-peer fashion on volunteer web servers, nor is there any situation where it is likely to be hosted that way. The goals of (near) instant update time, as well as some central quality control and user blocking, makes distributed hosting pretty impossible.

Everything wrong with wikipedia summed up in one paragraph, nice!

Re:Wikipedia is swimming in money–why is it begging people t... by bawolff on Monday May 31, 2021 @05:17AM (#27342011)
I mean, if you want to make a website with different goals and properties than wikipedia, by all means go make one. Wikipedia is just one way of doing things. Its hardly the last word on how to make an online encyclopedia, and quite frankly, some competition in this space would probably be good for everyone.

But disagreing with wikipedia's goals as a product is very different from a claim that the solution intended to meet those product goals is over-engineered.

Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @06:36AM (#27342549)
> Wikipedia's purpose is to benefit readers by acting as a free widely accessible encyclopedia;

How are centralization, real time editing, and user blocking properties toward the above goal?

You probably know full well that network effects are prohibiting any other collaborative knowledge base from from ever croping up again.

All big companies link against wikipedia because the network effect, all funding and donations go to wikipedia due to the network effect, all editors spend their time on wikipedia due to the network effect.

Wikipedia is eating it's own childen, and if we ever want to get something better, it either needs to grow up and start acting like a responsible adult (which we know it won't because at it's core is the failed web equivalent of the stanford prison expetiment), or die and make space for a new genration of tools.

Re:Wikipedia is swimming in money–why is it begging people t... by nickik on Monday May 31, 2021 @05:19AM (#27342021)
As somebody who has used these facilities I highly disagree. They are really awesome.

And if you have some magically technical solution that is much better and much easier, then please share it. And don't just complain that everybody else is doing everything wrong.

Re:Wikipedia is swimming in money–why is it begging people t... by kqr on Monday May 31, 2021 @02:30AM (#27340889)
Every long-term successful operation needs to sustain a few wild bets that look pointless today, with great probability will turn out to have been pointless tomorrow, but has just a tiny chance of becoming really important.

If you just optimise for today, you will die tomorrow.

Gradual improvements won't get you out of a local peak when that local peak is swallowed by rising tides.

If your organisation isn't doing things that look pointless today but in a principled way, I don't think it has great prospects of long-term survival.

Heck, even the wiki concept itself started out as this long-shot experiment nobody at the time expected to be successful.

Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @02:48AM (#27340990)
Wikipedia exploded over night.

The semantic web is "tomorrow technology" for 20 years now. At what point does tomorrow technology become yesterdays vaporware to you?

Re:Wikipedia is swimming in money–why is it begging people t... by zozbot234 on Monday May 31, 2021 @03:17AM (#27341192)
Wikidata gets more edits per second right now than the English wikipedia does. If that's not "exploding overnight", I don't know what is. It's just that fewer people are aware of it, because it powers sily things like search engine infoboxes and AI assistants - in addition to fostering unprecedented cooperation among knowledge producers in the Linked Data/Semantic Web ecosystem.
Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @03:58AM (#27341440)
Ah yes, and that's not totally skewed by the fact that an "edit" in a knowledge graph is at the granulatity of a triple while an edit in an article has the granulatity of a coherent piece of structurally meaningfull information.

Piling onto the glorified garbage dump (just follow the isA edges of any concept you'll see what I mean) is a strawman metric.

Re:Wikipedia is swimming in money–why is it begging people t... by zozbot234 on Monday May 31, 2021 @04:49AM (#27341804)
It might be a "garbage dump", but it's one that large pieces of the Web are increasingly relying on. The pre-existing alternatives (mainly DBpedia - incidentally, based on scraping Wikipedia articles) were pretty disgusting hacks compared to what we have now.
Re:Wikipedia is swimming in money–why is it begging people t... by NoInkling on Tuesday June 01, 2021 @01:39AM (#27351040)
I use Wikidata data (in a non-academic capacity) shrug

It's not that easy to work with, but there aren't any good alternatives in my case. Freebase died. DBpedia doesn't compare. Domain-specific databases don't have the data I'm looking for (or it's woefully incomplete and/or hard to link to external entities) because it straddles multiple domains, not to mention they're usually not free. I for one am very thankful for Wikidata existing and seeing good activity.

Re:Wikipedia is swimming in money–why is it begging people t... by metasj on Tuesday June 01, 2021 @11:33AM (#27356041)
Nice (& same!) How do you use it that you couldn't use dbpedia?