News for hackers. Stuff that matters.
Re:Wikipedia says traffic is falling due to AI search summar... by johnnyanmac on Monday October 20, 2025 @10:52PM (#45651951)
scraping Wikipedia feels like the stupidest possible move. You can in fact download the entire encyclopedia at any time and take all the time in the world parsing offline.

For such purposes, I'd naively just setup some weekly job to download Wikipedia and then run a "scrape" on that. Even weekly may be overkill; a monthly snapshot may do more than enough.

Re:Wikipedia says traffic is falling due to AI search summar... by yorwba on Tuesday October 21, 2025 @12:46AM (#45652556)
You can download twice-monthly database dumps, but they consist of the raw wikitext, so you need to do a bunch of extra work to render templates and stuff. Meanwhile, if you write a generic scraper, it can connect to Wikipedia like it connects to any other website and get the correctly-rendered HTML. People who aren't interested in Wikipedia specifically but want to download pretty much the entire internet unsurprisingly choose the latter option.
Re:Wikipedia says traffic is falling due to AI search summar... by parpfish on Tuesday October 21, 2025 @07:48PM (#45663239)
as somebody that has wrassled with the wikipedia dumps a number of times, i don't understand why wiki doesn't release some sort of sdk that gives you the 'official' parse
Re:Wikipedia says traffic is falling due to AI search summar... by skywal_l on Wednesday October 22, 2025 @03:10AM (#45665781)
This. I tried that a few years ago and fell off my chair when I started to realized how DYI the thing is. It's a bunch of unofficial scripts and half-assed out of date help pages.

At the time I though, well it's a bunch of hippies with a small budget, who can blame them? Now I learn that there is 600 of them with a budget in the hundreds of millions??

This is becoming another Mozilla foundation...

Re:Wikipedia says traffic is falling due to AI search summar... by philipkglass on Wednesday October 22, 2025 @11:29AM (#45670630)
I have wrestled with it too. I believe it's because wikitext is an ad-hoc format that evolved so that the only 100% correct parser/renderer is the MediaWiki implementation. It's like asking for an SDK that correctly parses Perl. Only Perl can do that.

There are a bunch of mainly-compatible third party parsers in various languages. The best one I've found so far is Sweble but even it mishandles a small percentage of rare cases.

Re:Wikipedia says traffic is falling due to AI search summar... by Gander5739 on Wednesday October 22, 2025 @03:07AM (#45665763)
There are also dumps of Wikipedia in html format.
Re:Wikipedia says traffic is falling due to AI search summar... by yorwba on Wednesday October 22, 2025 @03:47AM (#45666001)
Do you mean the discontinued Enterprise HTML dumps https://dumps.wikimedia.org/other/enterprise_html/ or the even older discontinued static HTML dumps https://dumps.wikimedia.org/other/static_html_dumps/current/... or is there another set of dumps I'm not aware of?