That was my first thought when I saw the headline "I bet it's lxml". I have lost count of the number of times I've had to do special nonsense to handle lxml's memory leaks. (I've even had to go so far as to launch parsing code in a subprocess just so the part that created an etree object would quit & return all its RAM.)

This sort of thing is why calling C from Python is a marginal idea. Debugging such C is tough, especially if it manipulates Python objects in complex ways. There are many invariants of the Python data structures which must be carefully maintained by hand in C code. Getting one wrong will result in obscure bugs like the parent.

"Pickle", Python's old seralizer, had a similar leak problem. It has a cache. You can reuse a Pickle stream, which is done in interprocess communication. But the cache of previously-send objects wasn't being cleared at the end of each Pickle block. I found that, did a workaround, and submitted a bug report. Not sure it was ever fixed properly.

I still have a bug report in on CPickle, a "faster" implementation of Pickle written in C.[1] In a complicated situation with multiple threads using CPickle, memory becomes corrupted and the program will crash. It doesn't happen using Python Pickle, so I just quit using CPickle. The bug report got the usual "reproduce in a simpler situation" reply to make it go away, and the bug report remains open. It may be the same bug as this one [2] from 2012, although I doubt it.

For parsing HTML, I use html4lib. It's slower, but it's all in Python.

[1] http://bugs.python.org/issue23655 [2] http://bugs.python.org/issue12680

I've had quite a few issues with cPickle myself, so I see what you mean.

Indeed, Python packages built over C extensions can be quite hard to debug, as seen with lxml. But what makes it even harder is the fact that lxml is partly built using Cython... so you deal with Python code, C code generated with Cython, and pure C code (libxml2).

While Cython - in my experience - doesn't have many bugs, it still has some, and can, at times, generate blaringly wrong C code (iirc one simple example: slice assigmnent to a uint8_t* tries to call a Python runtime function on the uint8_t* as a PyObject*).

Good to know! My only experience so far with Cython comes from lxml. It's a weird language that seems to have a lot of corner cases, though. Just like C++/CLI.

Yes. Yes it does. Also, because it works on two type systems it has many cases were you'll want to take a look at the generated C code to verify that the "cheap path" was taken and no intermediary Python objects are constructed or Py operators are used -- if performance matters, that is.

On the other hand it is a radically simpler way to write bindings that also contain logic, or to write rather fast code without straying to far from Python. Plus, it can cythonize almost all code, even very dynamic code with closures, which will still often improve performance on it's own (no interpreter, but still Python runtime for every op). And that is then a nice base to do further optimizations.

(So many typos above. "html4lib" should be "html5lib". "Seralizer" should be "serializer". "Previously-send" should be "previously-sent". Sorry.)

No kidding. I actually once had a freaky memory leak of my own that I was never able to fix. I was in fact using libxml2 there, I wonder if that was the cause.

There's more than one leak actually. It's quite a party :)

Exact same thing. Odd unsolved memory leak, using lxml for HTML parsing.

I think I'll stay away from it in the future. Unfortunate, since it's the fastest HTML parser in Python that I know of.

Have you compared it to the C implementation of ElementTree?

With stanzas like

    for event, n in ElementTree.iterparse(xmlfile, events=("end",)):
        if n.tag == "blar":
            # do stuff
            n.clear()

it doesn't even take much memory to parse files of arbitrary size.

In libxml2? That's a reasonable guess, but I would be open to other possibilities I were you, hehe :) Stay tuned!

The post's tags, displayed at the top, spoiled it for me.