News for hackers. Stuff that matters.
Re:Download the Entire Wikimedia Database by dwheeler on Saturday March 06, 2021 @04:27PM (#26370667)
I'm so glad the download-entire-wikipedia function continues to exist. That will help counter the "lost the entire library problem" from the city of Alexandria. To be fair, Wikipedia only has summaries, not the detailed material, but it's still important.
Re:Download the Entire Wikimedia Database by LeoPanthera on Saturday March 06, 2021 @05:27PM (#26371090)
Kiwix offers downloadable material (including the full Wikipedia), in a format specifically designed for offline browsing.

https://wiki.kiwix.org/wiki/Content

Their Wikipedia bundle wasn't being updated for a while and had fallen out of date, but that seems to have been fixed now.

Re:Download the Entire Wikimedia Database by easton on Saturday March 06, 2021 @06:03PM (#26371349)
This is what makes the 256GB iPad worth it, you can carry the entire Wikipedia (with pictures) around with you. It’s fantastic when on a plane or if you need to entertain yourself without Wi-Fi.
Re:Download the Entire Wikimedia Database by m-p-3 on Saturday March 06, 2021 @06:15PM (#26371418)
I "upgraded" the internal storage of my Kobo Glo HD (which is in fact using a 4GB MicroSD card as its onboard storage) to a 128GB. I can store the "all maxi" 82GB ZIM file on it. The Kobo firmware includes (but doesn't advertise) ZIM support, so I can browse Wikipedia on an eInk display and great battery life without Internet connectivity.

That would be a great post-apocalyptic knowledge archive, and it would be easy enough to recharge using a USB solar panel.

Re:Download the Entire Wikimedia Database by ObsoleteNerd on Saturday March 06, 2021 @11:03PM (#26373040)
I've been wanting something like this for ages. I always thought the Kindle not having the full Wikipedia archive as an optional install was a massive missed opportunity, especially back when the Kindle had the keyboard at the bottom. It would have literally been the Hitchhikers Guide as explained in the books.
Re:Download the Entire Wikimedia Database by lxgr on Sunday March 07, 2021 @05:20AM (#26374713)
> The Kobo firmware includes (but doesn't advertise) ZIM support

This is one of the nicest hidden features/easter eggs I've ever come across. Thanks for sharing this!

Re:Download the Entire Wikimedia Database by SahAssar on Sunday March 07, 2021 @06:39AM (#26375070)
This comment convinced me to buy a kobo.

How's the search functionality in wikipedia when used like this?

Re:Download the Entire Wikimedia Database by m-p-3 on Monday March 08, 2021 @09:43AM (#26386263)
You open the ZIM file as if it was an eBook, then you can use the search feature within it.

Example: https://www.mobileread.com/forums/showthread.php?t=276219

Re:Download the Entire Wikimedia Database by ianbooker on Sunday March 07, 2021 @05:24AM (#26374730)
Time to update XKCD: https://xkcd.com/548/
Re:Download the Entire Wikimedia Database by herpderperator on Saturday March 06, 2021 @11:22PM (#26373111)
This sounds like an amazing idea. Thanks for the tip!
Re:Download the Entire Wikimedia Database by andrepd on Sunday March 07, 2021 @08:37AM (#26375724)
Or a Kobo with an external SD slot!
Re:Download the Entire Wikimedia Database by SomewhatLikely on Saturday March 06, 2021 @09:32PM (#26372594)
The link for this post has the full wikipedia, just not in a very readable form. I believe the grandparent comment was saying that wikipedia is a summary of original material such as books and scholarly articles.
Re:Download the Entire Wikimedia Database by porphyra on Saturday March 06, 2021 @04:39PM (#26370754)
It is pretty awesome that there are people like /r/datahoarder that are obsessed with backing up the collective knowledge of humanity.
Re:Download the Entire Wikimedia Database by aarchi on Saturday March 06, 2021 @05:57PM (#26371302)
There's also Archive Team, focused on preserving at-risk sites before being taken offline.

https://wiki.archiveteam.org/

Re:Download the Entire Wikimedia Database by capableweb on Saturday March 06, 2021 @04:46PM (#26370815)
I'm not familiar with r/datahoarder, but if the name bears any significance, it seems they are mostly centered on hoarding data, which means just digital I guess? If so, I much rather would want to promote efforts like Internet Archive that back up all kind of things, not just digital data.
Re:Download the Entire Wikimedia Database by KMnO4 on Saturday March 06, 2021 @05:04PM (#26370944)
What does the Internet Archive back up that isn’t represented by 1s and 0s?
Re:Download the Entire Wikimedia Database by cguess on Saturday March 06, 2021 @05:09PM (#26370985)
A ton of 35mm and 16mm film reels, vinyl, and even wax recording, physical books and a lot more. They make digital copies of them, but they also archive the physical versions as well. Here's a selection of the movies: https://archive.org/details/moviesandfilms?tab=about
Re:Download the Entire Wikimedia Database by LeoPanthera on Saturday March 06, 2021 @05:28PM (#26371101)
Re:Download the Entire Wikimedia Database by CompuHacker on Sunday March 07, 2021 @01:02PM (#26378054)
Promote both, rest easier.
Re:Download the Entire Wikimedia Database by monadic8 on Sunday March 07, 2021 @03:44AM (#26374279)
There's not much evidence that the loss of the Library of Alexandria constituted much loss to human knowledge. This idea seems to come from Carl Sagan's Cosmos. While there a lot of good things to say about him he's not much of a historian.
Re:Download the Entire Wikimedia Database by dwheeler on Thursday March 11, 2021 @09:21AM (#26424247)
I think it is almost certain that a vast amount of human knowledge was lost. The policy of the city of Alexandria was to copy every book that went through it, and give the copy back. As a result, they ended up with a spectacularly large collection of original materials.

The amount of ancient literature that is no longer available is vast. History, literature, you name it, so much has been lost. The odds are very good that at least a significant portion of that were in the Alexandrian library.

Re:Download the Entire Wikimedia Database by orblivion on Wednesday March 10, 2021 @08:24AM (#26410941)
I'm going to ask my parents about this to see if they learned this idea in school. They didn't grow up in an English speaking country, and I don't think they're big fans of Sagan.
Re:Download the Entire Wikimedia Database by mixmastamyk on Tuesday March 09, 2021 @11:55PM (#26407916)
Maybe that facts such as the rough circumference of the earth were lost for a thousand years had something to do with it? Or is that false?
Re:Download the Entire Wikimedia Database by 1vuio0pswjnm7 on Sunday March 07, 2021 @03:01AM (#26374133)
This is a also user privacy solution that could be applied in other areas. If one is only querying a local store of Wikipedia pages, then third party observation of those queries is perhaps impossible, and so the "surveillance capital" "business model" may become infeasible. (Now this is perhaps a poor example because Wikipedia generally does not appear to be engaged in "surveillance capitalism". Maybe their straightforward provision of bulk data coupled with this fact is not a coincidence.) But there are other databases besides Wikipedia that internet users query weekly, daily, hourly or even more frequently. Making incessant queries over the internet can a rich source of data for a third party observer, one that can form the foundation of "surveillance capitalism" revenue strategies. Whereas if users can download bulk data, and thereby avoid queries over the internet, in many cases they can avoid falling victim to "surveillance capitalism". Bulk data can be a viable privacy solution.
Re:Download the Entire Wikimedia Database by qwertywert_ on Saturday March 06, 2021 @05:41PM (#26371186)
Can it also download sources if available too, would be cool.
Re:Download the Entire Wikimedia Database by tablespoon on Saturday March 06, 2021 @05:26PM (#26371087)
> I'm so glad the download-entire-wikipedia function continues to exist. That will help counter the "lost the entire library problem" from the city of Alexandria. To be fair, Wikipedia only has summaries, not the detailed material, but it's still important.

Personally, I think Wikipedia's quality is too poor for that. Plus, it's digital, so when our civilization is at risk of "[losing] the entire library" it probably would have already lost the ability to maintain the computer systems to access Wikipedia dumps.

Re:Download the Entire Wikimedia Database by gerdesj on Saturday March 06, 2021 @06:01PM (#26371330)
"Personally, I think Wikipedia's quality is too poor for that."

You should see some of the crap in the books at Alexandria: The world is flat and there are four elements and other bollocks. Obviously I'm taking the piss. The content is sometimes just as important as the factual accuracy of the content. For a given value of factual accuracy.

WP is written by people and holds a vast amount of stuff. It is flawed in my opinion in many ways but that is the human experience.

I live in a town called Yeovil in Somerset, UK. https://en.wikipedia.org/wiki/Yeovil . Several years ago I noticed an incorrect old name for the place and I tried to correct it. I appealed to the Domesday Book which is considered quite authoritative hereabouts but I linked to the only site I could find which sold copies of it. My edit was thrown out by the local Somerset editor rather than being fixed. I own a coffee cup bought from the local museum that lists the >60 spellings of this tiny town over the last 1500 odd years. The editor wouldn't accept that either "original research" WTF! I didn't put the names on the mug - archaeologists, historians and a bunch of tax gatherers hired by a hoard of Normans back in the day did that. OK, no they didn't - they scrawled stuff and the museum gathered together the scrawls and made my mug. I did one for a tourist shop on the Plymouth Barbican wrt the Mayflower complement, about 30 years ago. To be fair, I simply copied the names off the board near the Mayflower steps!

My point is that WP is what it is and you need to see it for that. It is both a store of knowledge and also a store of knowledge and blatant lies and everything in between ... about knowledge. It contains its own metadata and also omits vast amounts of it.

WP is without question in my mind absolutely magnificent but you do need to learn how and when to interpret it to fit in with your idea of factual - whatever that is.

Re:Download the Entire Wikimedia Database by tablespoon on Saturday March 06, 2021 @06:10PM (#26371387)
> You should see some of the crap in the books at Alexandria: The world is flat and there are four elements and other bollocks. Obviously I'm taking the piss. The content is sometimes just as important as the factual accuracy of the content. For a given value of factual accuracy.

But that reflected the actual state of knowledge at the time, which is what you'd really want to study.

Let me put it this way: Wikipedia wouldn't even allow Wikipedia to be used as a source for one if it's articles, because it's too unreliable: https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources#Use...

Re:Download the Entire Wikimedia Database by ris on Saturday March 06, 2021 @06:32PM (#26371541)
> Let me put it this way: Wikipedia wouldn't even allow Wikipedia to be used as a source for one if it's articles, because it's too unreliable

Er.. I think you're missing the point of citing sources. If an otherwise unverified claim could cite as its source another unverified claim, it would make citing sources meaningless. If a wikipedia article wanted to cite a verified claim from another wikipedia article that did have a verified source, they may as well just use that original source as their source too.

I think your example actually shows wikipedia in a very good light.

Re:Download the Entire Wikimedia Database by tablespoon on Saturday March 06, 2021 @09:50PM (#26372700)
> Er.. I think you're missing the point of citing sources. If an otherwise unverified claim could cite as its source another unverified claim, it would make citing sources meaningless. If a wikipedia article wanted to cite a verified claim from another wikipedia article that did have a verified source, they may as well just use that original source as their source too.

1) The Wikipedia guideline doesn't mention that logic, 2) quite a lot of Wikipedia doesn't cite anything, and 3) it not unusual for a passage in Wikipedia to not actually be supported by the citation given (e.g. https://en.wikipedia.org/w/index.php?title=Special:WhatLinks...).

In any case, this is getting off topic. My point was that, if you're looking for an ark of cultural knowledge to survive some apocalypse, Wikipedia's a bad choice. IMHO, something like Project Gutenberg plus a newspaper archive would be about 1000% better, and take up far less space. If you got space to spare, throw in Libgen. Wikipedia's not a replacement for its sources, and I pity the future scholar that would have to rely on it without being able to check them.

But unless your data storage is orders of magnitude more reliable than any current technology and you package your archive with an equally reliable computer to read it, the concept of a digital ark fails. If you don't to that, your archive will be unreadable, and and unreadable archive is useless.

Re:Download the Entire Wikimedia Database by smoldesu on Saturday March 06, 2021 @05:32PM (#26371126)
The content on Wikipedia is really not that bad. Obviously a Wikipedia article will never be the final say on any specific subject, but it tends to do a pretty good of aggregating sources and condensing it into a reader-friendly synopsis. This data is super valuable, if not just for the sources alone.
Re:Download the Entire Wikimedia Database by buzzerbetrayed on Saturday March 06, 2021 @05:39PM (#26371170)
> so when our civilization is at risk of "[losing] the entire library" it probably would have already lost the ability to maintain the computer systems to access Wikipedia dumps

But as long as it continues to exist, some future civilization could figure out how to read the data again, eventually. Just like we eventually discovered how to read ancient languages that were once forgotten.

Re:Download the Entire Wikimedia Database by tablespoon on Saturday March 06, 2021 @05:56PM (#26371293)
> But as long as it continues to exist, some future civilization could figure out how to read the data again, eventually. Just like we eventually discovered how to read ancient languages that were once forgotten.

Eh, I think you're vastly underestimating how difficult that would be.

1. The media would have to last hundreds of years at least, when it's hoped modern archival media can last maybe fifty.

2. Even assuming the media did last, the new civilization would have to reverse engineer encoding on top of encoding on top of encoding (e.g. physical disk encoding, complex filesystems, file formats, character encodings). Our civilization already has trouble reading some old file formats, and our disks already have trouble reading their data (which is why they pack a ton of error correction information).

It took the Rosetta stone to figure out how to read encoding of Egyptian hieroglyphics, when that language was still alive in the form of Coptic.

3. Then you're dealing with the probability that the hard disks the future archeologists find will even have a Wikipedia dump on them. That probability will be very small, given very few people will download these dumps.

Re:Download the Entire Wikimedia Database by bawolff on Saturday March 06, 2021 @07:19PM (#26371844)
If people still understand english in some form (a good bet. We still understand latin. English has more reach than latin did at its peak) understanding charsets is pretty easy. Just assume its a shift cipher.

As far as media goes. That's true, but its a bit of a numbers game. After all you only need one unusually preserved specimen. The dead sea scrolls survived after all. Not to mention intentional preservation efforts. I know github has its artic vault thing. There's even a copy of wikipedia on the moon! https://meta.m.wikimedia.org/wiki/Wikipedia_to_the_Moon/Wrap...

Re:Download the Entire Wikimedia Database by tablespoon on Saturday March 06, 2021 @09:44PM (#26372659)
> If people still understand english in some form (a good bet. We still understand latin. English has more reach than latin did at its peak) understanding charsets is pretty easy. Just assume its a shift cipher.

IIRC, Coptic is directly descended from Ancient Egyptian, but the Rosetta Stone was still needed to decipher hieroglyphics.

It won't be as simple as you think. The problem will be more like: here's 10TB of partially corrupted binary data, find the text when you don't know the encoding (oh, and the text may be compressed with an algorithm you also don't know).