Re:Python to get Google-backed speedup
by CUViper on Saturday June 13, 2009 @06:47PM
(#656589)
One perl nicety is that the VM will optimize to only compile the expression once. Python's re.sub() has to compile the expression every time, since it's just a string argument rather than a language construct. You can manually cache the result of re.compile() -- it's just not as sugary that way.
[ Parent
| on HN ]
That's actually untrue. If you look in the implementation of the `re` module you would see that it caches the compilation of any regular expression.
Aha -- I almost suggested that they could do some memoization, and indeed they are doing just that. The last 100 unique patterns are compiled and stored in a dictionary for future use. You could still thrash badly if your regex "working set" is more than 100 patterns, but for most people it's probably Good Enough. (Plus you could override re._MAXCACHE if you really want.)
Thanks for pointing out my error, and getting me to read the actual code instead of speculating. :)