It's worse. The MediaWiki PHP code doesn't implement a proper scanner and parser, it's a bunch of regexes around which the code has grown more or less organically. Silent compensation for mismatched starting and ending tokens abounds, and causes problems for all consumers of the markup, in the same way that lenient HTML parsers have. The difference is that Wikipedia, as the sole channel for editing markup, could have easily rejected syntax errors with helpful messages instead of silently compensating.

If it was anything else, I'd say "who cares," but this is "the world's knowledge" -- we absolutely should care about the format it's stored in. I'm glad to see people tackling this problem.

Interestingly markdown has the same problem.

Another example of the imperfect but working implementation winning.

Markdown...ugh. Let's just stick to DokuWiki or Mediawiki syntax for everything, please. If you need something more advanced than that, you should be using LaTeX. Actually, it'd be cool to build a working MediaWiki + Markdown => LaTeX converter....in something like Python.

there is markdown2pdf written Haskell, which seems to have XeTeX as an intermediary step.

Personally I'd be happy to see any markup language becoming the default, regardless which one it is. Having a proper grammar would be a bonus.

"Personally I'd be happy to see any markup language becoming the default."

I don't agree with this. All lightweight humane markup languages are not born equal, some are better others, and Mediawiki's is not in the list of the best ones. Now there seem to be a trend towards Markdown, but it should be improved and then, migrating Wikipedia to this Markdown2 could be a real good thing.

> it'd be cool to build a working MediaWiki + Markdown => LaTeX converter....in something like Python.

For parsing MediaWiki in Python, check out mwlib [1], which was part of cooperation between Wikimedia Foundation and PediaPress. It's neither very complete nor very fast, but you might be able to hack up some LaTeX conversion with it.

[1] http://code.pediapress.com/wiki/wiki/mwlib

There are actually a bunch of Markdown parsers that create a DOM, or at least a proper tree (e.g markdown2 in PyPI). Markdown is outrageously simple compared to MediaWiki.

I actually wrote a subset-of-MediaWiki parser in C#, which stayed at the "subset" stage because of the ridiculous complexity and corner-cases that crop up even very early.

Regarding the storage and access of Wikipedia's "world knowledge", DBpedia is a project that scrapes Wikipedia (InfoBoxes and categories) to create a structured, semantic database of knowledge.

https://secure.wikimedia.org/wikipedia/en/wiki/DBpedia