I think the big turnoff for me is its closed-source nature
Well, the issue of open-vs-closed is less clear with things like MATLAB. Sure you can't download it for free-as-in-beer, but if you own a copy, you are free to inspect its source code, as MATLAB is mainly written in MATLAB (same for Mathematica and other similar packages).
A subset of non-performance-critical functions, and things like argument validation layers, are written in m code that you can read. Most of the actual work happens in obfuscated p code or calls into compiled libraries. Many of those libraries are available independently of Matlab, but you can't see how they're calling them, or important details like how the Matlab interpreter or JIT compiler work.
As for Matlab, I think the big turnoff for me is its closed-source nature, which is maybe a more serious problem than you're giving it credit for. I also think that things like Python, that are more general-purpose, integrate much better than Matlab into other systems (e.g., web systems--you might not think it would matter if your estimation program doesn't integrate into a web app easily, but I didn't either and now I'm very sensitive to it). This is partially due to the open-source nature of things like Python, which lends itself to competition and experimentation, but also it being a more general-purpose language (this latter point is another sticking point with Julia, although wouldn't apply to OCaml).
This OCaml library is really beautiful, actually. I looked into OCaml several years ago for this sort of thing, and if it had been around then, I might have invested more in it at the time, and kept using it. I'll have to revisit it.