The direct $$$ cost of maintaining an engine is probably not as high as you think, as it really needs a small team to move fast enough and have the customizability advantage over COTS engines.

The value question can't be answered in general, it depends on what the engine is specialized to, how well it's executed compared to the alternative, what are the cultural factors surrounding the idea of using in-house tech, how well the support tooling supporting creatives is executed etc.

The standard curve applies, most in-house engines aren't very good, but few are.