I've been hesitant to use state machines for anything complex... I find they work great for small-ish things like an image uploading component in the front end, where the states are very well defined, there is a sort of flow, predictable actions need to be taken, etc.
I've wondered a lot about composing machines. It seems like it could be extremely useful and powerful. The actor model is the approach the XState team is taking and it seems promising, but some things holds me back. The API is certainly one reason, but it's the mounding complexity that scares me more. Like you say, it seems like it could get really messy.
I have a feeling with a good enough API this problem could be reduced, but that's not a trivial thing to figure out at all. I'd love to see it happen, though.
Point of fact, I just started a new job and am already being thrown into fixing their state machine code (not javascript / ts though). I can't say that what they're doing requires fewer lines, tests or has fewer bugs than if they'd just written it out without the whole state machine concept. I can say that it would certainly be a lot simpler and more flexible, though.