News for hackers. Stuff that matters.
Re:Automatic Generation of Regular Expressions from Examples by merlincorey on Monday October 22, 2012 @12:32PM (#4683940)
You have apparently useless genes in your genetic code as well.

These regex's are generated with genetic programming techniques, so they are not guaranteed to be an optimal solution, just a good (by the metric of working) one.

Re:Automatic Generation of Regular Expressions from Examples by tlrobinson on Monday October 22, 2012 @01:06PM (#4684149)
You're right, I was just pointing out that with some basic optimizations you could get a much cleaner result.

Applying some fairly simple optimizations to my example result I get the following equivalent regular expression:

   \d{2}[\d\.]{3,}
which isn't too bad.

EDIT: Here are the optimizations:

The original:

    (?:\d\d[\d\d\.\d\.][\d\.](?:(?=([\d\.]+))\1))
1. Remove non-capturing groups and lookahead:

    \d\d[\d\d\.\d\.][\d\.]([\d\.]+)\1
2. Simplify set:

    \d\d[\d\.][\d\.]([\d\.]+)\1
3. Remove redundant backreference:

    \d\d[\d\.][\d\.][\d\.]+
4. Combine duplicates:

    \d{2}[\d\.]{3,}
Re:Automatic Generation of Regular Expressions from Examples by mdda on Monday October 22, 2012 @01:48PM (#4684321)
The optimizations are great to apply to an end result, but the Genetic Programming process probably won't benefit from the expressions being optimized 'along the way' because some of the 'junk' subexpressions allow for more robustness in the face of the genetic operators.

Putting it another way, your highly optimized version (4) is likely to be much more brittle to small 'defects' than (1).

Re:Automatic Generation of Regular Expressions from Examples by DannyBee on Monday October 22, 2012 @02:18PM (#4684533)
The last point can't be overstated. Optimization is highly likely to overfit the training data.

For example, given training data 000001 000002 000003 000004

The "optimized" version would be \d+[1-4]

If the goal was simply to find the smallest regex that matches all the training data, that's trivial to code, NP-hard to execute.

But the goal isn't to find the smallest regexp, it's to find a "good" regexp.

Re:Automatic Generation of Regular Expressions from Examples by khafra on Monday October 22, 2012 @02:20PM (#4684554)
Actually, the last point can be overstated, because the optimizations he applied created a functionally equivalent state machine to the non-optimized version. If it overfitted, the original GA-generated algorithm overfitted as well.
Re:Automatic Generation of Regular Expressions from Examples by tlrobinson on Monday October 22, 2012 @03:09PM (#4684882)
My "optimizations" were trivial, in that they don't change the behavior of the regex at all, they just simplify it. As khafra points out, they are equivalent.

I also wasn't trying to say those simplifications should be applied between iterations of the algorithm, only at the end.

Re:Automatic Generation of Regular Expressions from Examples by mdda on Monday October 22, 2012 @06:24PM (#4685867)
Understood - and I was just making the observation that the expressions produced during the process were junky in a way that's characteristic of the way they are evolving : Mixing together two honed expressions would produce much more extreme deviations in 'children' than the offspring of two of these hairy expressions.

Not only are the expressions tending to get more accurate with each generation, but their degree of 'evolvability' is also being selected for.