News for hackers. Stuff that matters.
Re:Wikipedia is swimming in money–why is it begging people t... by zozbot234 on Monday May 31, 2021 @02:30AM (#27340895)
> People donating to wikipedia only want wikipedia, they are unaware of the other stuff wikimedia does.

On the contrary, Wikipedia itself has a lot of reliance on the other projects. For example, Wikidata solves the problem of interlinking Wikipedia languages and other projects when they're hosting content about the same real-world entity. In fact, this was Wikidata's MVP - the reason it got funded in the first place. It then took off from there.

Wikimedia Commons is a similar story - in addition to the obvious languages issue, there are a lot of concerns about media (such as licensing. enhancement etc.) that really are best addressed in a dedicated venue, with its own committed contributors. This has been very beneficial to Wikipedia itself.

Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @02:46AM (#27340984)
Using RDF to store multiple languages of the same article is gross overengineering. Interlinking with other projects isn't nessecary when there are no other projects.
Re:Wikipedia is swimming in money–why is it begging people t... by bawolff on Monday May 31, 2021 @02:55AM (#27341044)
It isn't stored as RDF. RDF is provided as an export format. The language contents also aren't stored in wikidata, its just matching which title in language X matches which title in language Y.

Wikidata isn't just used for interlanguage linking. Its also used to keep infoboxes synced across different languages, and a way to easily query that data (Depends a bit on the language, english wikipedia isn't fully on board with the system, but its been a big boon to smaller languages).

As a result, people can update these "facts" independently of language, keeping them altogether more up to date, and it allows the data to be queried independently (via https://query.wikidata.org - try some of the example queries (button top left) if you haven't, the level of power SPARQL gives for querying this type of data set really is very cool).

Part of Wikimedia's mission is to spread knowledge. Allowing querying of factual data with a query language like sparql helps advance that goal by letting people use extract the knowledge for new purposes.

Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @03:03AM (#27341107)
> its just matching which title in language X matches which title in language Y

Like I said, gross overengineering.

"You can save cents and seconds of time updating knowledge boxes, by spending millions and hours developing federated SPARQL query capabilities!"

Re:Wikipedia is swimming in money–why is it begging people t... by bawolff on Monday May 31, 2021 @03:10AM (#27341149)
More like, we put the facts and knowledge in a central (mysql) db, so that instead of updating them in 300 places (Wikipedia has a lot of languages), you just update them in one place.

And since the data happened to be there anyways, we added an RDF export feature, and threw that into a BlazeGraph DB so that people could do SPARQL queries if they want. The (not really federated) SPARQL queries have basically nothing to do with the central-place-to-store-facts that are used in multiple places, feature, but it was pretty cheap to do once everything was in a central place.

Re:Wikipedia is swimming in money–why is it begging people t... by lifeisstillgood on Monday May 31, 2021 @03:50AM (#27341383)
Is there a entry point for "how to make FOSS contributions to wiki-projects". I at the very least would be interested in finding out how all the pieces fit together, and maybe browsing the big list to see if anything was in my wheelhouse.

Edit: And thank you for some insight into the back end of wikipedia - I did not know multiple languages was such a big push and it is such a sensible direction . Keep up the good work and keep pestering me for cash each year.

Re:Wikipedia is swimming in money–why is it begging people t... by bawolff on Monday May 31, 2021 @04:03AM (#27341466)
https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hac... is probably what you are looking for.

There's a very old architecture doc from 2012: https://www.mediawiki.org/wiki/Manual:MediaWiki_architecture its kind of outdated and missing a lot of the newer pieces.

p.s. I used to work for the Wikimedia foundation about 1.5 years ago, but don't anymore. Prior to working there I was a volunteer for the FOSS project, and in theory I still am but I honestly don't do much anymore. My opinions are my own and don't represent WMF or anyone else.

Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @04:08AM (#27341497)
As if thowing the messy complexities of MySQL queries into the serene simplicity of static hypertext is somehow better.

How often is numerical (which is the only knowledge you can really automate this way) knowledge updated? How often is said knowledge dependent on cultural idiosyncracies? How much effort is the code maintenance of the specialised editing system? How much effort is maintainance of the centralised infrastructure. How much effort is it really to change the data in multiple places, given that enough eyes every bug is shallow? What are the oppirtunity costs of not being able to properly distribute and share articles because they are suddenly tied to a very specific codebase? How much infrastructure cost do I have because I can't get volunteers to host my data?

Overengineering solving the wrong problems at it's finest, no wonder the software has stayed this bad for decades now.

Re:Wikipedia is swimming in money–why is it begging people t... by bawolff on Monday May 31, 2021 @04:26AM (#27341619)
> How often is numerical (which is the only knowledge you can really automate this way) knowledge updated?

Numerical knowledge is not the only knowledge that is stored there.

I'm going to go with update rate of often. It has a higher edit rate than english wikipedia does.

> How often is said knowledge dependent on cultural idiosyncracies?

Sometimes that is true, other times it isn't. The system is used where it makes sense, and not used where it doesn't.

> How much effort is the code maintenance of the specialised editing system? How much effort is maintainance of the centralised infrastructure.

There is certainly some. I wouldn't say its dominating, but it certainly exists. There is plenty of centralized infrastructure for other things too.

Prior to the introduction of wikidata, more complex fragile systems existed to try and keep things in sync in a more manual fashion. They also had a cost.

> How much effort is it really to change the data in multiple places, given that enough eyes every bug is shallow?

A lot. And we don't have that many eyes in lots of languages. There's a lot after all: https://en.wikipedia.org/wiki/Special:SiteMatrix

> What are the oppirtunity costs of not being able to properly distribute and share articles because they are suddenly tied to a very specific codebase

Not sure what this has to do with it. Articles are written in a custom markup language. That's already tying it to a specific code base quite heavily. The wikidata dependency is pretty trivial, and there are open apis to get info out of wikidata, and regularly made dumps of all the data in wikidata if you want to copy it. If that's not good enough, just copy the rendered version instead of the source.

> How much infrastructure cost do I have because I can't get volunteers to host my data

I'm not sure what you're talking about. Wikipedia was never hosted in some magical peer-to-peer fashion on volunteer web servers, nor is there any situation where it is likely to be hosted that way. The goals of (near) instant update time, as well as some central quality control and user blocking, makes distributed hosting pretty impossible.

Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @05:00AM (#27341876)
> I'm not sure what you're talking about. Wikipedia was never hosted in some magical peer-to-peer fashion on volunteer web servers, nor is there any situation where it is likely to be hosted that way. The goals of (near) instant update time, as well as some central quality control and user blocking, makes distributed hosting pretty impossible.

Everything wrong with wikipedia summed up in one paragraph, nice!

Re:Wikipedia is swimming in money–why is it begging people t... by bawolff on Monday May 31, 2021 @05:17AM (#27342011)
I mean, if you want to make a website with different goals and properties than wikipedia, by all means go make one. Wikipedia is just one way of doing things. Its hardly the last word on how to make an online encyclopedia, and quite frankly, some competition in this space would probably be good for everyone.

But disagreing with wikipedia's goals as a product is very different from a claim that the solution intended to meet those product goals is over-engineered.

Re:Wikipedia is swimming in money–why is it begging people t... by j-pb on Monday May 31, 2021 @06:36AM (#27342549)
> Wikipedia's purpose is to benefit readers by acting as a free widely accessible encyclopedia;

How are centralization, real time editing, and user blocking properties toward the above goal?

You probably know full well that network effects are prohibiting any other collaborative knowledge base from from ever croping up again.

All big companies link against wikipedia because the network effect, all funding and donations go to wikipedia due to the network effect, all editors spend their time on wikipedia due to the network effect.

Wikipedia is eating it's own childen, and if we ever want to get something better, it either needs to grow up and start acting like a responsible adult (which we know it won't because at it's core is the failed web equivalent of the stanford prison expetiment), or die and make space for a new genration of tools.

Re:Wikipedia is swimming in money–why is it begging people t... by nickik on Monday May 31, 2021 @05:19AM (#27342021)
As somebody who has used these facilities I highly disagree. They are really awesome.

And if you have some magically technical solution that is much better and much easier, then please share it. And don't just complain that everybody else is doing everything wrong.