Re:Xee: A Modern XPath and XSLT Engine in Rust
by philipkglass on Friday March 28, 2025 @02:43PM
(#43508698)
In the case of Wikipedia dumps there is an easy work-around. The XML dump starts with a small header and "<siteinfo>" section. Then it's just millions of "page" documents for the Wiki pages.
You can read the document as a streaming text source and split it into chunks based on matching pairs of "<page>" and "</page>" with a simple state machine. Then you can stream those single-page documents to an XML parser without worrying about document size. This of course doesn't apply in the general case where you are processing arbitrary huge XML documents.
I have processed Wikipedia many times on a machine with only 8 GB of RAM.
Re:Xee: A Modern XPath and XSLT Engine in Rust
by hyhjtgh on Friday March 28, 2025 @02:10PM
(#43508394)
Xml and textual formats in general are ill suited to such large documents. Step 1 should really be to convert and/or split the file into smaller parts.
Re:Xee: A Modern XPath and XSLT Engine in Rust
by intrasight on Friday March 28, 2025 @09:32PM
(#43511918)
Or shred into a proper database designed for that
Re:Xee: A Modern XPath and XSLT Engine in Rust
by nicoburns on Friday March 28, 2025 @07:09PM
(#43510878)
Shouldn't parsed XML be smaller than the raw uncompressed text? (as you could deduplicate strings). I'd expect that to be a significant saving for something like wikipedia in XML
Re:Xee: A Modern XPath and XSLT Engine in Rust
by philipkglass on Friday March 28, 2025 @07:32PM
(#43511027)
For Wikipedia, the bulk of the data in the XML is inside a "<text>" block that contains wikitext: https://en.wikipedia.org/wiki/Help:Wikitext
For the English Wikipedia the wikitext accounts for about 80% of the bytes of the uncompressed XML.
In any case I don't have $1500 to blow on a new computer with 100GB of ram in the unsubstantiated hope that it happens to fit, just so I can play with the Wikipedia data dump. And I don't think that's a reasonable floor for every person that wants to mess with big xml files.