Your classic 'finite state machine' (FSM) is pretty simple. Boxes with arrows coming out of them leading to other boxes.
You can't store additional state in a FSM. Your state is what box you're in. As a result, more complex processes get really complex fast in a FSM.
Hence statecharts. This is what most people these days actually think of as state machines. They have history, you can implement guard conditions, you can store state (hence removing the finitude), and so on.
So, actually, yes! Infinite state machines are very common. They're very, very useful. I love programming using statecharts.
[0]: https://www.sciencedirect.com/science/article/pii/0167642387...
What all of these terms really describe are formalisms over finite state - the linked paper is precise in this respect. "You can't store state in an FSM" misses the point: you can store as many concepts as you have enumerated states, and the act of storage itself is a change from one state to another state. Breaking with this concept and allowing unlabelled storage is indeed a useful quality because it helps with dealing with bulk permutations.
But every programming language is intricately tied to their models of finite state and how it's presented to the user. Most PLs, and likewise hardware instruction sets, hide away state bookkeeping with notions of "assignment" and "branching", so when the formalisms appear it's usually because you really need to explicitly enumerate more of your state. The formalized FSM goes hand-in-hand with concurrent programming because it clarifies how coordination occurs and when state changes are allowed. And real implementations of the FSM usually piggyback in some degree on the base programming environment, which has always been a source of muddiness: oftentimes a "goto" is exactly what you need to describe a transition, and yet in structured programs, we've eliminated goto.