News for hackers. Stuff that matters.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by perlgeek on Wednesday June 04, 2014 @09:20AM (#7845352)
Basically the same advice, from the year 2000: http://www.perlmonks.org/?node_id=24640

There is one legitimate use case of .* though: advancing to the last match of something. If you want to find the last digit in a string, /.*(\d)/ will readily find it for you.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by sanderjd on Wednesday June 04, 2014 @09:56AM (#7845526)
Also, if you want to capture everything after some prefix. For instance, in a log: /(WARN|ERR) (.*)/
Re:Why Using .* in Regular Expressions Is Almost Never What ... by eCa on Wednesday June 04, 2014 @09:51AM (#7845500)
A somewhat clearer way (imho) is /(\d)\D*$/ since it anchors to the end of the string.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by claar on Wednesday June 04, 2014 @12:42PM (#7846602)
According to the debugger at http://regex101.com/:

  /.*(\d)/
- Searches right-to-left, backtracking until it finds the match.

  /(\d)\D*$/
- Searches left-to-right, going forward a step, backtrack, forward, backtrack, until it finds a match.

If you're looking for a match toward the end of a string, the .* version will be faster.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by hamburglar on Wednesday June 04, 2014 @01:32PM (#7846932)
Wow, regex101.com is very nice, but I never would have noticed that debugger pane if I hadn't gone looking for it after your comment. Incredibly useful tool.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by Rynant on Wednesday June 04, 2014 @10:03AM (#7845573)
This is not the same as finding the last match though. The parent's example will match '2' in '1 of 2 steps.'
Re:Why Using .* in Regular Expressions Is Almost Never What ... by ronaldx on Wednesday June 04, 2014 @10:27AM (#7845694)
On the contrary, it does gives the same result.

$ anchors to the end of the string, \D clears the non-digits from the end to allow \d to match the digit '2'.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by Rynant on Wednesday June 04, 2014 @11:00AM (#7845909)
Thanks, I see where I was wrong now.

In this case when finding the last match from the end, would the lazy quantifier reduce backtracking? e.g. /(\d)\D*?$/

Re:Why Using .* in Regular Expressions Is Almost Never What ... by icambron on Wednesday June 04, 2014 @01:10PM (#7846770)
No, that would work very similarly to the greedy version. The backtracking happens because the \d gets matched to the '1' and the whole thing has to be rolled back when the $ attempts match and instead finds '2' (this would happen again if there were more digits for \d to speculatively match on). So the backtracking is not caused by the laziness or greediness of the \D* ; we really do want to gobble up all of the non-digits.

On the two options generally:

    /(\d)\D*$/
is problematic if you have a lot of digits, while

    /.*(\d)/ 
is problematic if you have a lot of text after the last digit. Both could potentially be optimized by the engine to run right-to-left (the former because it's anchored to the end and the latter because it greedily matches to the beginning), and then both would do well. I'm not sure if that happens in practice.

Overall, I prefer the latter, both because I think it's clearer and because its perf characteristics hold up under a wider variety of inputs.

Edit: how do you make literal asterisks on HN without having a space after them?

Re:Why Using .* in Regular Expressions Is Almost Never What ... by nkozyra on Wednesday June 04, 2014 @01:38PM (#7846979)
In addition to the other explanation, the lazy qualifier is redundant here anyway since there should only be one $ in any given expression.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by chinpokomon on Wednesday June 04, 2014 @10:36AM (#7845756)
As others before me have said, this pattern works as expected. Putting it through regexper, you can visually see this.

http://www.regexper.com/#%2F(%5Cd)%5CD*%24%2F

Re:Why Using .* in Regular Expressions Is Almost Never What ... by ygra on Wednesday June 04, 2014 @10:46AM (#7845816)
Technically, while they both capture the same digit, the match itself it different, including either everything before that digit or everything after it. But I tend to liberally use lookaround to keep the actual match clean myself; maybe others go more often for a capturing group. (Well, and not being able to use arbitrary-length lookaround in most engines might be a reason too.)
Re:Why Using .* in Regular Expressions Is Almost Never What ... by abuzzooz on Wednesday June 04, 2014 @10:27AM (#7845692)
Actually, both will match the '2' in '1 of 2 steps.'
Re:Why Using .* in Regular Expressions Is Almost Never What ... by perlgeek on Wednesday June 04, 2014 @03:47PM (#7847718)
Yes, but that's harder to do for more complicated regexes, because you need to negate a regex (here \d => \D) for this trick.

If you have a complicated regex $r, you can only negate it with (?:(?!$r).), and in that case, .$r is much easier to read :-)

Re:Why Using .* in Regular Expressions Is Almost Never What ... by Mithaldu on Wednesday June 04, 2014 @09:34AM (#7845409)
Be aware though that \d will also find arabic or roman numerals. :)
Re:Why Using .* in Regular Expressions Is Almost Never What ... by mhaymo on Wednesday June 04, 2014 @09:57AM (#7845533)
Roman numerals? I have never encountered a regex implementation where '\d' matches 'X'. It's certainly not the case for Javascript: http://regexpal.com/?flags=g&regex=\d%2B&input=10%0Ax%0AX%0A...
Re:Why Using .* in Regular Expressions Is Almost Never What ... by mmastrac on Wednesday June 04, 2014 @10:06AM (#7845590)
There's a whole lot of these numeral characters in unicode, for example:

http://www.charbase.com/2169-unicode-roman-numeral-ten

(even more in http://www.charbase.com/block/number-forms)

I'm not sure if Javascript matches these in its \d pattern, however, but I think that most regexp engines default to the ascii [0-9] unless you are using \p{Number}.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by pyre on Wednesday June 04, 2014 @11:06AM (#7845953)
There's also the character classes in Unicode:

http://www.fileformat.info/info/unicode/category/index.htm

Re:Why Using .* in Regular Expressions Is Almost Never What ... by fennecfoxen on Wednesday June 04, 2014 @10:55AM (#7845862)
> /\d/.test("\u2169")

false

Re:Why Using .* in Regular Expressions Is Almost Never What ... by jaredmcateer on Wednesday June 04, 2014 @10:59AM (#7845896)
Ruby also does not match roman numerals with \d

http://rubular.com/r/bOCmLIfKdZ

Re:Why Using .* in Regular Expressions Is Almost Never What ... by UweSchmidt on Wednesday June 04, 2014 @10:33AM (#7845728)
\d finds arabic numerals? As in 0-9? Scandalous :)

http://en.wikipedia.org/wiki/Arabic_numerals

Re:Why Using .* in Regular Expressions Is Almost Never What ... by extra88 on Wednesday June 04, 2014 @11:56AM (#7846315)
Perhaps they mean Eastern Arabic numerals. http://en.wikipedia.org/wiki/Eastern_Arabic_numerals
Re:Why Using .* in Regular Expressions Is Almost Never What ... by acdha on Wednesday June 04, 2014 @02:51PM (#7847388)
Indeed - those are in use by millions of people. I wonder if any regexp implementation has started matching all of the other number symbols in Unicode:

http://en.wikipedia.org/wiki/Numerals_in_Unicode

Re:Why Using .* in Regular Expressions Is Almost Never What ... by greglindahl on Wednesday June 04, 2014 @03:52PM (#7847749)
Perl does, by default.

There's a modifier if you want to only match ASCII digits.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by icambron on Wednesday June 04, 2014 @12:05PM (#7846364)
About eight years ago I finally read Friedl's Mastering Regular Expressions [1]. I know, right? A 500-page book about regular expressions, a tool I already knew (or thought I did). But it's actually a great book-- easy to read and full of genuinely good information on the how and why of regex, and it totally changed my understanding of them. If absolutely anything in this article surprised you, I highly recommend you read the book.

[1] http://regex.info/book.html

Re:Why Using .* in Regular Expressions Is Almost Never What ... by Pxtl on Wednesday June 04, 2014 @10:05AM (#7845579)
I've gotten into the habit of using the "not" operation instead of .* a lot. If I'm looking for bracketed text, I use not-bracket to match the contents.

I tend to avoid the non-greedy operator just because it often fails in terrible half-assed regex implementations (eg. visual studio 2010)

Re:Why Using .* in Regular Expressions Is Almost Never What ... by bane on Wednesday June 04, 2014 @12:36PM (#7846567)
I wish the not operator allowed for sub-expressions instead of just character classes. It'll probably make it slower, but it would remove lots of unreadable convolutions people have to go through.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by ori_b on Wednesday June 04, 2014 @01:59PM (#7847086)
There are some regex implementations that allow it, but it's a very confusing feature. Remember that '' is not 'a'.

Arbitrary expressions can have arbitrary length, so excluding an expression simply will match it, fail the match, and backtrack to the next option.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by blueblob on Wednesday June 04, 2014 @12:40PM (#7846589)
Me too. I find that if you work in a bunch of different languages it seems more portable (and one less thing to remember). It also seems easier to debug.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by tjgq on Wednesday June 04, 2014 @09:36AM (#7845421)
Also read Russ Cox's writeup on implementing regular expressions [0]. Backtracking can be done efficiently; it's just that most regular expression engines have suboptimal implementations for it.

[0] http://swtch.com/~rsc/regexp/regexp1.html

Re:Why Using .* in Regular Expressions Is Almost Never What ... by BugBrother on Wednesday June 04, 2014 @09:40AM (#7845442)
I've cursed over Python's backtracking, at least a few years back. (Why can't they just use PCRE? :-( Any advantage at all?)
Re:Why Using .* in Regular Expressions Is Almost Never What ... by ori_b on Wednesday June 04, 2014 @10:35AM (#7845740)
PCRE has the same problems with backtracking.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by BugBrother on Wednesday June 04, 2014 @11:43AM (#7846212)
As bad? I might have had smarter coworkers at different times... :-)
Re:Why Using .* in Regular Expressions Is Almost Never What ... by maxerickson on Wednesday June 04, 2014 @10:46AM (#7845817)
Python beat PCRE to Unicode support by several years.

(So there at least was an advantage)

Re:Why Using .* in Regular Expressions Is Almost Never What ... by BugBrother on Wednesday June 04, 2014 @11:39AM (#7846189)
I thought the Unicode support was still spotty (< 3.X)? Or you mean the support is better than in PCRE?
Re:Why Using .* in Regular Expressions Is Almost Never What ... by acdha on Wednesday June 04, 2014 @02:58PM (#7847432)
It depends on what you mean by Unicode support - this turns out to be a surprisingly painful area if you need something like case folding:

strasse = straße

or treating combining characters the same as their single character equivalents:

ñ = ñ

(That's LATIN SMALL LETTER N WITH TILDE and LATIN SMALL LETTER N followed by COMBINING TILDE)

A surprising number of languages (mostly everything but Perl) won't handle advanced uses like this.

The good news is that the next version of the stdlib regex module is being developed independently:

https://pypi.python.org/pypi/regex

Simply "pip install regex" and:

    >>> regex.match(r"(?iV1)strasse", "stra\N{LATIN SMALL LETTER SHARP S}e").span()
    (0, 6)
    >>> regex.match(r"(?iV1)stra\N{LATIN SMALL LETTER SHARP S}e", "STRASSE").span()
    (0, 7)
Re:Why Using .* in Regular Expressions Is Almost Never What ... by maxerickson on Wednesday June 04, 2014 @11:50AM (#7846269)
In 2000, PCRE simply didn't support Unicode. Python 1.6 and 2.0 did (at least, based on some quick searching PCRE added support for Unicode in 2004).

"spotty" probably isn't the right word either, the change in 3.0 was to default to treating text as always being Unicode, the 'unicode' type in 2.x is reasonably complete (as these things go), just not the default treatment for text.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by BugBrother on Wednesday June 04, 2014 @04:16PM (#7847914)
As late as 2004? I didn't know that.

'acdha' (in the brother comment) wrote what I meant with "spotty" better and more pedagogical than I ever could. :-)

Re:Why Using .* in Regular Expressions Is Almost Never What ... by moron4hire on Wednesday June 04, 2014 @09:45AM (#7845465)
I think the advent of automatic regex match highlighting in text editors is changing the regex use-case for a lot of people. It certainly did for me. I no longer see regexs as just "something you use in code to test input". I now use them as general purpose text editing tools. In a way, it's like templated text output, with input specified in the same buffer.

I know this has been done forever, but usually only by extreme greybeards in Vi or Emacs world. The auto-highlighting now makes it possible for everyone to do it.

So that said, with the ability to restrict regexes to just a selection of text, it's more about regex golf--the fewest characters, the most productive--than it is about semantic correctness. If it works for my input, that's all that matters, because the regex is getting discarded thereafter.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by Pxtl on Wednesday June 04, 2014 @10:08AM (#7845597)
Yeah, I do all my data-imports from flat files using regex - easy to export from spreadsheet programs as flat files, then regex them into a bunch of insert/update statements.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by collyw on Wednesday June 04, 2014 @01:34PM (#7846951)
Yes, I do plenty of similar things.

Format a load of data using regexes first, then use it hard coded as string to do quick one off script to update the database. It beats trying to parse Excel directly, as you never know what data type a cell will return.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by chernevik on Wednesday June 04, 2014 @10:15AM (#7845633)
To be picky, it's always what I want, but with a lot of other stuff I don't.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by blt on Wednesday June 04, 2014 @10:34AM (#7845732)
Very interesting. I've had the greedy .* "overmatch", like probably almost everyone who's used regular expressions. Had no idea they are a performance drain even when giving the right answer though.

I like posts about details of software craftsmanship like this.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by prohor on Wednesday June 04, 2014 @09:30AM (#7845392)
A bit of a problem with lazy quantifiers is that they are not so widely supported out of the perl world. Therefore I often need to find some tricks to get similar behavior (eg. "[^,]*," - if coma is separator)
Re:Why Using .* in Regular Expressions Is Almost Never What ... by hnriot on Wednesday June 04, 2014 @09:40AM (#7845440)
Except they are! Java, python, JavaScript etc all support lazy quantifies. In fact I can't think of a single language that doesn't.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by mkehrt on Wednesday June 04, 2014 @06:35PM (#7848850)
Except bash, awk, sed and vim. So, everywhere I use regular expressions.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by fmoralesc on Thursday June 05, 2014 @04:18PM (#7854119)
vim has \- and variants.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by caio1982 on Wednesday June 04, 2014 @10:07AM (#7845591)
Sed doesn't AFAIK.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by BugBrother on Wednesday June 04, 2014 @09:38AM (#7845431)
Huh, PCRE is universal (almost, not in e.g. Emacs :-( ) and supports (almost) everything?

Edit: Cough, after rechecking... PCRE is not as universal as I thought. It seems I've been lucky. :-) http://en.wikipedia.org/wiki/Comparison_of_regular_expressio...

Edit 2: "Atomic groups" on that wikipedia link is when you can write a full grammar in a large regexp, right? Answer myself: No, it is the name for stopping backtracking. I've seen it as named "possessive" (perldoc perlre).

Re:Why Using .* in Regular Expressions Is Almost Never What ... by SiVal on Thursday June 05, 2014 @03:00AM (#7850592)
I thought it was becoming more universal, but now I'm not so sure.

grep on the Mac used to use PCRE regexes if you used the -P option (`grep -P ....`), but beginning with OS X 10.8 the -P option was removed, so an important place that used to offer PCRE (default grep on a default Mac) actually removed support for it. They didn't replace it with something better; they just took it away. Maybe a Unicode issue?

Not only is PCRE not universal, overall support might even be waning.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by collyw on Wednesday June 04, 2014 @01:30PM (#7846920)
MySQL doesn't have them, it has some POSIX version, which is way less intuitive. Plenty of people use MySQL.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by BugBrother on Wednesday June 04, 2014 @04:18PM (#7847931)
Isn't that part of an SQL standard (1999?).
Re:Why Using .* in Regular Expressions Is Almost Never What ... by guynamedloren on Wednesday June 04, 2014 @09:40AM (#7845439)
I've run into this problem so many times. Everywhere I think I want .* , I actually want .*? (non-greedy matching). Make a mental note of this. It'll save you lots of headaches.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by kstenerud on Wednesday June 04, 2014 @09:56AM (#7845527)
Actually, it only needs to be \[([^,]+),([^\]]+)\] because you're only going up to a comma in the first capture group and a square bracket in the second.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by poolunion on Wednesday June 04, 2014 @11:12AM (#7846000)
That would also match all of "[a] more [b,c]" though.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by kstenerud on Wednesday June 04, 2014 @11:56AM (#7846314)
And the other regex would match all of "[a more [b,c]". Your regex must be designed around your expected input.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by NanoWar on Wednesday June 04, 2014 @11:33AM (#7846142)
The regex fiddle is really useful: http://regex101.com/r/qQ2dE4
Re:Why Using .* in Regular Expressions Is Almost Never What ... by larubbio on Wednesday June 04, 2014 @11:27AM (#7846101)
Is the early example in the document correct?

Using an input string of abc123 he claims [a-z]+\d+ will match the entire string (which I agree with). He then says that [a-z]+?\d+? will only match abc1. Wouldn't it fail since the non-greedy match on [a-z] would just match 'a' causing the non-greedy match on \d to fail trying to match 'b'?

Re:Why Using .* in Regular Expressions Is Almost Never What ... by simcop2387 on Wednesday June 04, 2014 @11:32AM (#7846133)
no it'll still match but because both are non-greedy it could match on just c1 instead of abc123.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by prawks on Wednesday June 04, 2014 @01:21PM (#7846846)
It could match on c1, but I believe since most (all?) regex parsers parse left-to-right, it will match the a, look for another a-z character or a digit, find b, repeat, find c, then find 1 which completes the pattern.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by gatehouse on Wednesday June 04, 2014 @01:22PM (#7846855)
http://regex101.com/r/aR5xM2

I used this tester posted elsewhere in the thread, it seems like since the lazy components expand "as needed" to achieve a match, it will succeed on "abc1".

EDIT: I wrapped it in a group for clarity.

Re:Why Using .* in Regular Expressions Is Almost Never What ... by bthornbury on Wednesday June 04, 2014 @03:06PM (#7847473)
I once used .* in a crawler. Came back the next day to find much rogue html amongst the content of my site. I find something like [^{{delimiting character}}]* to be better
Re:Why Using .* in Regular Expressions Is Almost Never What ... by gesman on Wednesday June 04, 2014 @09:33AM (#7845406)
.*? Is the solution!
Re:Why Using .* in Regular Expressions Is Almost Never What ... by bthornbury on Wednesday June 04, 2014 @03:05PM (#7847470)
I once used .* in a crawler. Came back the next day to find much rogue html amongst the content of my site. I find something like [^
Re:Why Using .* in Regular Expressions Is Almost Never What ... by mschuster91 on Wednesday June 04, 2014 @10:05AM (#7845581)
For the "greedy" behaviour, PHP has the "U" flag... dunno about other implementations though.
Re:Why Using .* in Regular Expressions Is Almost Never What ... by spb on Wednesday June 04, 2014 @07:51PM (#7849226)
This is why I like Lua's '-' for non-greedy matching in its pattern facilities.