News for hackers. Stuff that matters.
Re:Technical Feasibility of Building Hitchhiker's Guide to t... by rasz_pl on Tuesday April 01, 2014 @03:25PM (#7511004)
Im making my own offline wiki (fun side project). EN wiki dump is ~10GB. 8-9GB if you preprocess XML dropping useless tags and wikimarkup

  falink = {b'Link FA|', b'Link GA|', b'link FA|', b'link GA|', b'Link FL|', b'link FL|'}
different language links/references, every article has a list of all the language version links (ex: [[af:April]] [[als:April]] ..)

  langcolon = {b'aa:', b'ab:', b'ace:', b'af:', b'ak:', b'aln:', b'als:', b'am:', ................-my:', b'zh-sg:', b'zh-tw:', b'zh-yue:', b'zu:'}
I have a feeling changing wikimarkup to something more size optimized would help even more. Swapping compression algorithm can squeeze additional 1-3GB of savings without sacrificing decompression speed.

All in all with a lot of tricks you can go from 10GB to 5GB. Doesnt feel worth it. Bigger problem is searching thru this data on a 2-4 core ~1GHz ARM processor with 1-2GB of available ram. Indexing titles is simple and fast, but quite useless. Indexing all useful words would eat at least couple of gigabytes. Searching for a simple combination of three words would take couple of seconds every time. Its not like you can haul Hadopi cluster in your pocket.

There are no comments on this comment yet.