News for hackers. Stuff that matters.
Re:Rulex – A new, portable, regular expression language by SonOfLilit on Friday June 10, 2022 @06:23AM (#31692153)
Note that "minimal DFA" is not very relevant these days, with Google's `re2` being the only implementation I'm aware of that implements a DFA. All others sacrifice predictable performance for feature parity with Perl.
Re:Rulex – A new, portable, regular expression language by burntsushi on Friday June 10, 2022 @07:22AM (#31692527)
It's certainly not the only one. Rust's regex crate (of which I'm the author) and Go's regexp package both use finite automata.

It should be noted that none of them build out complete DFAs in the general case. Building a DFA is worst case exponential time and space, and they can in practice be expensive even when not hitting the worst case.

Instead, they use different techniques. RE2 and the regex crate, for example, use a hybrid NFA/DFA that does subset construction at search time. Outside of pathological cases, it provides the speed of a DFA without needing to build the whole thing up front.

Hyperscan is also built on finite automata. GNU grep uses finite automata in many cases too.

Re:Rulex – A new, portable, regular expression language by ankitpati on Friday June 10, 2022 @06:56AM (#31692358)
Not a single other implementation has feature-parity with Perl. Perl regexes are not only Turing-complete, but can also execute arbitrary code¹ as part of the regex evaluation.

PCRE is the pinnacle of deceptive advertising. It is emphatically not Perl-Compatible.

Source: I used to write Perl for a living, and I have battle scars from (¹) above.

Re:Rulex – A new, portable, regular expression language by macintux on Friday June 10, 2022 @07:52AM (#31692763)
When Sun released Java and went on a marketing road show, I drove to Chicago to hear the presentation. Since Perl was my favorite language, I had to ask about regular expression support; the Sun engineer said there wasn’t any, but I could build the first library.

To me, it was inconceivable that you could release a major language and not only not have regular expressions built into the language like Perl, but not even have it in the standard library. Crazy.

Re:Rulex – A new, portable, regular expression language by latchkey on Friday June 10, 2022 @01:36PM (#31697138)
When I found out that Jonathan Locke had even a minimal Java implementation of a regex engine, I jumped on it and messaged him to open source it to the Jakarta project. Crazy that that was 20 years ago...

https://jakarta.apache.org/regexp/changes.html

Re:Rulex – A new, portable, regular expression language by burntsushi on Friday June 10, 2022 @08:07AM (#31692890)
We did the same thing for Rust. Regexes are neither in the language nor in the standard library. Go doesn't have them in the language either (although they're in the standard library).
Re:Rulex – A new, portable, regular expression language by macintux on Friday June 10, 2022 @08:43AM (#31693265)
Rust wasn’t trying to become the standard web programming language.
Re:Rulex – A new, portable, regular expression language by burntsushi on Friday June 10, 2022 @08:52AM (#31693364)
I didn't realize that was a necessary precondition in your evaluation of what makes something "crazy." Sorry.
Re:Rulex – A new, portable, regular expression language by macintux on Friday June 10, 2022 @09:08AM (#31693561)
Sorry, early morning. I should have been more verbose.

Rust has a different target problem space, a different standard library philosophy, a package management system, and exists in a world where adding external dependencies is trivial.

Java didn’t have any of those excuses.

Re:Rulex – A new, portable, regular expression language by oblio on Friday June 10, 2022 @08:56AM (#31693410)
Well, Java turned out quite all right on that front :-)
Re:Rulex – A new, portable, regular expression language by kevincox on Friday June 10, 2022 @06:26AM (#31692165)
It is incredibly relevant if you want to process uset-provided expressions. Otherwise you expose yourself to a DoS attack.

The Rust regex crate is also DFA based and safe for untrusted inputs.

Re:Rulex – A new, portable, regular expression language by Someone on Friday June 10, 2022 @06:31AM (#31692204)
.NET added timeouts to handle that. It’s a per-regex setting with a global default (https://docs.microsoft.com/en-us/dotnet/api/system.text.regu...).
Re:Rulex – A new, portable, regular expression language by nsajko on Friday June 10, 2022 @06:45AM (#31692280)
I guess that adding timeouts can significantly decrease the run time efficiency?
Re:Rulex – A new, portable, regular expression language by Someone on Friday June 10, 2022 @07:10AM (#31692442)
It can, but this isn’t a hard timeout. From the description (“The MatchTimeout property defines the approximate maximum time interval for a Regex instance to execute”), it seems they have their regex engine periodically (say after handling every 10,000 instructions of the compiled regex) check whether they went over the limit.
Re:Rulex – A new, portable, regular expression language by orra on Friday June 10, 2022 @07:44AM (#31692707)
Plus, as a general point, .NET has pretty decent performance. It's no C++, but at the same time substantial effort is put into optimisation.

In fact for both .NET v5 and v7, they specifically did work to improve the performance of regexes. For example there is a new non-backtracking option. https://devblogs.microsoft.com/dotnet/regular-expression-imp...

Re:Rulex – A new, portable, regular expression language by SonOfLilit on Friday June 10, 2022 @06:30AM (#31692194)
Nice to know about Rust. From reading the docs the situation seems to be more complicated and more impressive: https://docs.rs/regex/latest/regex/#untrusted-input
Re:Rulex – A new, portable, regular expression language by masklinn on Friday June 10, 2022 @07:10AM (#31692445)
There’s also the postgres regex engine, I don’t know how it works under the hood but it’s very resistant to catastrophic backtracking despite features which normally translate to nfa (e.g. backrefs).
Re:Rulex – A new, portable, regular expression language by nsajko on Friday June 10, 2022 @07:14AM (#31692475)
> features which normally translate to nfa (e.g. backrefs)

You're very wrong, backreferences in general can't be translated to finite automata (NFA is short for nondeterministic finite automaton). See these comments:

https://news.ycombinator.com/item?id=18431610

https://news.ycombinator.com/item?id=27877743

Re:Rulex – A new, portable, regular expression language by masklinn on Friday June 10, 2022 @11:34AM (#31695596)
Maybe try to read my comment again because I don't think you understood what I wrote.
Re:Rulex – A new, portable, regular expression language by nsajko on Friday June 10, 2022 @04:04PM (#31698798)
You say:

> resistant to catastrophic backtracking despite features which normally translate to nfa

I interpret this as you equating NFA with backtracking engines, which would be incorrect. Is my interpretation wrong?

Re:Rulex – A new, portable, regular expression language by joe_guy on Friday June 10, 2022 @01:29PM (#31697067)
https://devblogs.microsoft.com/dotnet/regular-expression-imp...

Stephen has put incredible work into having an alternative implementation in .net 7.

For anyone interested in regex engine inner workings and optimizations, including SIMD, the blog post is fantastic.

Re:Rulex – A new, portable, regular expression language by btown on Friday June 10, 2022 @08:11AM (#31692943)
On a related note, if you have Python regex code that you want to make more stable/performant, https://pypi.org/project/pyre2/ is a drop-in replacement for `re` that (configurably) falls back to `re` if you use lookaheads, etc.

The design philosophy behind RE2 for those unfamiliar with the library: https://github.com/google/re2/wiki/WhyRE2

Re:Rulex – A new, portable, regular expression language by nsajko on Friday June 10, 2022 @06:33AM (#31692215)
Various regexp implementation possibilities do exist, whether in software or in hardware. I think my general point stands independently of which regexp implementation is used (although the comparisons get hairy if you include the irregular expression "regexps").