News for hackers. Stuff that matters.
Re:Technical Feasibility of Building Hitchhiker's Guide to t... by jerf on Tuesday April 01, 2014 @10:28AM (#7508406)
As a thought experiment, packing up the whole of a website is a good upper bound. If we were doing this for real, we could further aggressively trim, taking advantage of the fact that content utility tends to follow a power law rather than a uniform distribution. For instance, take stack overflow; plink everything with a negative point score. Plink everything with no replies and no votes. Plink everything that has a reply with more than, say, 5x the points (i.e., if there's a 100 point reply and a 4 point reply, plink the four-point reply). Plink entire sites/tags that are unlikely to be useful (no Python software on the mission? No Python interpreter shipped? Remove all Python questions). For Wikipedia, remove all information about localities that have either less than 5KB of text or 10,000 people. Spend some time hitting the "random" button on Wikipedia and you can get a sense of how much of Wikipedia does not need to be sent into space. Remove vast swathes of the catalog of each species on Earth... I doubt a Martian settler is going to have a compelling reason to suddenly look up a cricket particular to northern Utah. And so on.

You could probably cut all those estimates down by 90%, with very little loss observable to the users. It wouldn't be zero loss, but use the savings to stuff other useful stuff in there instead, such as redundant copies to ensure we don't lose anything.

Re:Technical Feasibility of Building Hitchhiker's Guide to t... by rasz_pl on Tuesday April 01, 2014 @03:25PM (#7511004)
Im making my own offline wiki (fun side project). EN wiki dump is ~10GB. 8-9GB if you preprocess XML dropping useless tags and wikimarkup

  falink = {b'Link FA|', b'Link GA|', b'link FA|', b'link GA|', b'Link FL|', b'link FL|'}
different language links/references, every article has a list of all the language version links (ex: [[af:April]] [[als:April]] ..)

  langcolon = {b'aa:', b'ab:', b'ace:', b'af:', b'ak:', b'aln:', b'als:', b'am:', ................-my:', b'zh-sg:', b'zh-tw:', b'zh-yue:', b'zu:'}
I have a feeling changing wikimarkup to something more size optimized would help even more. Swapping compression algorithm can squeeze additional 1-3GB of savings without sacrificing decompression speed.

All in all with a lot of tricks you can go from 10GB to 5GB. Doesnt feel worth it. Bigger problem is searching thru this data on a 2-4 core ~1GHz ARM processor with 1-2GB of available ram. Indexing titles is simple and fast, but quite useless. Indexing all useful words would eat at least couple of gigabytes. Searching for a simple combination of three words would take couple of seconds every time. Its not like you can haul Hadopi cluster in your pocket.