I actually opensourced this code at https://github.com/blaze33/dem2mesh, just have to use dem2mesh.png2mesh which given a png data, computes the heightmap, builds a mesh, simplifies it and return the data to javascript. (can be tricky though to deal with wasm)
The RTIN processing itself is very straightforward / lo-rent ... that said, if you were optimising the DEM tiles on the fly you'd probably refactor / integrate the two processes to save some loops. Dropped into a service worker (or WASM) I'm guessing it would be reasonably performant. Though perhaps meshoptimizer is more flexible with mesh dimensions?? (rtin is limited to a 2^nish grid)
With that in mind had you considered zooming out for your dem only, so as to lower the heightmap resolution (but leave the env map as is)? Maybe you're already doing that?
https://github.com/mapbox/martini
Proof of context demo (garbage code!): https://lab.citizensgbr.org/census-map/index-martini.html