For years and years I ran a web service that scraped another site's HTML to extract data. There were other APIs doing the same thing. They used a proper HTML parser, and I just used the moral equivalent of String.IndexOf() to walk a cursor through the text to locate the start and end of strings I wanted and String.Substring() to extract them. Theirs were slow and sometimes broke when unrelated structural HTML changes were made. Mine was a straight linear scan over the text and didn't care at all about the HTML in between the parts I was scraping. It was even an arbitrarily recursive data structure I was parsing, too. I was able to tell at each step, by counting the end and start tags, how many levels up or down I had moved without building any tree structures in memory. Worked great, reliably, and I'd do it again.

Why regular expressions? Why not just substring matching?

This, much more deterministic!

Not sure if you are joking, but regexes are deterministic.

Oh, no I didn't mean to say regex or not, I meant regex over XML vs regex over a string. The first has the illusions everyone is bringing up that XML is not regular, but having clarity that it is ultimately a string is the correct set of assumptions.

It really is a central example of the bell curve meme, isn't it?

The reason we tell people not to parse HTML/XML/whatever with regular expressions isn't so much that you can't use regular (CS sense) patterns to extract information from regular (CS sense) strings that happen to be drawn from a language that can also express non-regular strings, but because when you let the median programmer try, he'll screw it up.

So we tell people you "can't" parse XML with regular expressions, even though the claim is nonsense if you think about it, so that the ones that aren't smart and independent-enough minded to see through the false impossibility claim don't create messes the rest of us have to clean up.

One of the most disappointing parts of becoming an adult is realizing the whole world is built this way: see https://en.wikipedia.org/wiki/Lie-to-children

This reminds me of cleaning a toaster with a dishwasher: https://news.ycombinator.com/item?id=41235662

It's always the edge cases that make this a pain.

The less like 'random' XML the document is the better the extraction will work. As soon as something oddball gets tossed in that drifts from the expected pattern things will break.

Of course. But the mathematical, computer-science level truth is that you can make a regular pattern that recognizes a string in any context-free language so long as you're willing to place a bound on the length (or equivalently, the nesting depth) of that string. Everything else is a lie-to-children (https://en.wikipedia.org/wiki/Lie-to-children).

You can, but you probably shouldn't since said regex is likely to be very hard to work with due to the amount of redundant states involved.

Our discourse does a terrible job of distinguishing impossible things from things merely ill-advised, and I think intellectual honestly requires us to be up front about the difference.

Yeah, I'd almost certainly reject a code review using, say, Python's re module to extract stuff from XML, but while doing so, I would give every reason except "you can't do that".

What is a “regular (CS sense) string”, especially if it’s apparently from a non-regular language? That’s not a thing. Maybe you mean you can parse regular subsets of non-regular languages, which is true but usually doesn’t lead to a robust solution when it comes to commonly-encountered non-regular data interchange languages like HTML and XML.*

> So we tell people you "can't" parse XML with regular expressions, even though the claim is nonsense if you think about it

It’s not nonsense. The actual meaning is ambiguous and a little context-sensitive (ha ha), but you can’t use a regular expression alone (CS sense) to do any of these things:

- determine whether an XML document is syntactically valid

- match an element, attribute, or text at a particular path under the assumption that the document is well-formed

And of course anything that isn’t possible for XML really isn’t possible for HTML. It is possible to extract the first/nth overall instance of an element that can’t contain itself under the assumption that the document is well-formed, though!

(The article is about non-robust solutions.)

* Edit: In another comment, you specified “so long as you're willing to place a bound on the length (or equivalently, the nesting depth) of that string”, which is obvious, yes, and leads to solutions that are not only not robust but also not even practical.

If I’m not mistaken, even JSON couldn’t be parsed by a regex due to the recursive nature of nested objects.

But in general we aren’t trying to parse arbitrary documents, we are trying to parse a document with a somewhat-known schema. In this sense, we can parse them so long as the input matches the schema we implicitly assumed.

> If I’m not mistaken, even JSON couldn’t be parsed by a regex due to the recursive nature of nested objects.

You can parse ANY context-free language with regex so long as you're willing to put a cap on the maximum nesting depth of constructs in that language. You can't parse "JSON" but you can, absolutely, parse "JSON with up to 1000 nested brackets" or "JSON shorter than 10GB". The lexical complexity is irrelevant. Mathematically, whether you have JSON, XML, sexps, or whatever is irrelevant: you can describe any bounded-nesting context-free language as a regular language and parse it with a state machine.

It is dangerous to tell the wrong people this, but it is true.

(Similarly, you can use a context-free parser to understand a context-sensitive language provided you bound that language in some way: one example is the famous C "lexer hack" that allows a simple LALR(1) parser to understand C, which, properly understood, is a context-sensitive language in the Chomsky sense.)

The best experience for the average programmer is describing their JSON declaratively in something like Zod and having their language runtime either build the appropriate state machine (or "regex") to match that schema or, if it truly is recursive, using something else to parse --- all transparently to the programmer.

What everyone forgets is that regexes as implemented in most programming languages are a strict superset of mathematical regular expressions. E.g., PCRE has "subroutine references" that can be used to match balanced brackets, and .NET has "balancing groups" that can similarly be used to do so. In general, most programming languages can recognize at least the context-free languages.

It's impossible to parse arbitrary XML with regex. But it's perfectly reasonable to parse a subset of XML with regex, which is a very important distinction.

Can regular expressions parse XML: No.

Can regular expressions parse the subset of XML that I need to pull something out of a document: Maybe.

We have enough library "ergonomics" now that it's not any more difficult to use a regex vs a full XML parser now in dynlangs. Back when this wasn't the case, it really did mean the differnce between a one or two line solution, and about 300 lines of SAX boiler-pate.