Take a look at BAP, it heavily uses OCaml features and implements a beautiful design:
https://github.com/BinaryAnalysisPlatform/bap
https://binaryanalysisplatform.github.io/bap/api/master/argo...
I'm not convinced that OCaml is unsuitable for such tasks.
That said, I don't know if this makes sense in a out-of-order, parallel world. I think as an organizational schema, imagining a machine that's doing things works better when that machine is relatively intuitive and simple.
It's not about maths, actually. You don't need to understand much math to comprehend monads, you need math only to reason about some properties and formal verification.
So think of monads as of codification of imperative effectful computations in a more general way.
I really like that example:
https://binaryanalysisplatform.github.io/bap/api/master/Mona...
What monad's bind does is just binding a step of computation to the rest of computation, so it is nothing new but a small step (operational) semantics written explicitly. That's why haskell has this do sugar that looks like as an imperative computation. Because it is a first class imperative computation.
Do you want first class features (functions, computations, effects) or features baked into the language is a subjective matter of course.