This way, the contents of the memory wouldn't be state at all; they would just be, as it were, the scratchpad where sentences in the language accepted by the state machine, or output by the state machine, are found.
That's what regular expressions do for us: they are a compact way of specifying a language. Just a few characters--which is to say, just a few states--can specify a language which contains arbitrarily long strings.
But--even though the input and output streams can be very large, they no longer part of the state, and thus do not contribute to the exponential blowup of states.
Here is a java lib that apply regex to stream of Objects that could be used to achieve this purpose.
This has a huge impact on trying to hook actions, call-backs, etc (your "arbitrary procedure") to NFA states as you're frequently making many overlapping entries to these states, many of which go nowhere. Trying to figure out which entries correspond to which other entries isn't easy.
There are very stylized automata that are used in parsing that do interesting things beyond the finite automata space (like pushing things onto a stack), but they don't correspond to regex per se - instead they are generated from a grammar.
> interesting things beyond the finite automata space
State machines certainly can be glorified into LR parsers, or even into Turing machines by adding various bells and whistles. So sure, this idea shouldn't be limited to just regular expressions--we've got good methods of specifying even very complicated state handling.
I hope you saw the excellent comment on this thread where somebody talked about how Ken Thompson implemented paxos using yacc for state management. I've been around the block with state management too, but using yacc for state management is something that never occurred to me, and frankly, opens up a whole new world of possibilities.
BTW, it's also an answer to your point about the pitfalls of nondeterminism: yacc is a great example of how to specify a state machine while detecting, reporting, and resolving nondeterminism.
"Lex helps write programs whose control flow is directed by instances of regular expressions in the input stream."
Exactly what I was talking about. There's no reason to suck at state management, guys....
I watched Ken Thompson write a Paxos implementation in yacc once.
Nope, not a joke. As you say, this is just the application of parsing technology to a tokenized stream of input events.
its super-useful in creating state machines to do parsing---and it can be super-useful to create state machines for other things as well.
> I watched Ken Thompson write a Paxos implementation in yacc once.
In real time? dude, you gotta post video to youtube or post a "Tell HN" story about it.
Ken discussed yacc briefly in Coders at Work, which I quoted at https://research.swtch.com/yyerror:
Seibel: And are there development tools that just make you happy to program?
Thompson: I love yacc. I just love yacc. It just does exactly what you want done. Its complement, lex, is horrible. It does nothing you want done.
Seibel: Do you use it anyway or do you write your lexers by hand?
Thompson: I write my lexers by hand. Much easier.
De Gustibus. Some of us are into that sort of thing. Thanks for the link.
> Thompson: I write my lexers by hand. Much easier.
LMAO. If you've been doing it since the 60's, I suppose you get the hang of it:
Ken Thompson, “Regular expression search algorithm,” Communications of the ACM 11(6) (June 1968), pp. 419–422
Imagine your program is a big state machine, and inputs and other events are making it transition from one state to another, and when in a state the apropos actions are performed. On this view, your program just is a regular expression parser--the tokenized stream of input events is what it is parsing.
And what is a great, high-level way of describing a parsing state machine? Yup--a regular expression. When compiled, a regular expression is translated into a state machine. However, you can't really specify that an arbitrary procedure is called when it enters a state--it can only do limited actions, e.g. extract a substring.
If we could let the compiled state machine call arbitrary procedures when it enters a state, well, we'd have a new program-control-flow statement. A super-duper if/then/else.
Instead of writing state charts--absolutely the lowest level you can program a state machine at--we could specify the state machine in a very high-level, easier to understand and maintain way.