Re:Technical Feasibility of Building Hitchhiker's Guide to t...
by rasz_pl on Tuesday April 01, 2014 @03:25PM
(#7511004)
Im making my own offline wiki (fun side project). EN wiki dump is ~10GB. 8-9GB if you preprocess XML dropping useless tags and wikimarkup
falink = {b'Link FA|', b'Link GA|', b'link FA|', b'link GA|', b'Link FL|', b'link FL|'}
different language links/references, every article has a list of all the language version links (ex: [[af:April]] [[als:April]] ..) langcolon = {b'aa:', b'ab:', b'ace:', b'af:', b'ak:', b'aln:', b'als:', b'am:', ................-my:', b'zh-sg:', b'zh-tw:', b'zh-yue:', b'zu:'}
I have a feeling changing wikimarkup to something more size optimized would help even more. Swapping compression algorithm can squeeze additional 1-3GB of savings without sacrificing decompression speed.All in all with a lot of tricks you can go from 10GB to 5GB. Doesnt feel worth it. Bigger problem is searching thru this data on a 2-4 core ~1GHz ARM processor with 1-2GB of available ram. Indexing titles is simple and fast, but quite useless. Indexing all useful words would eat at least couple of gigabytes. Searching for a simple combination of three words would take couple of seconds every time. Its not like you can haul Hadopi cluster in your pocket.
You could probably cut all those estimates down by 90%, with very little loss observable to the users. It wouldn't be zero loss, but use the savings to stuff other useful stuff in there instead, such as redundant copies to ensure we don't lose anything.