Futile wish. Even if you controlled the modem entirely the "other side" can do whatever they want with your packets.

what if you had both sides? I could see this being useful in a building/complex that already has coax, or as a way to cheaply do longer distances while keeping hardware out of a landfill

You can get CMTS equipment like Arris C3 and run your own DOCSIS headend if you want, but it's a lot of complexity to the point where it's not worth the effort unless it just happens to exactly fit your niche. Due to the way that cable plants work (upstream channels) the 'provider side' equipment must be complimentary to the 'customer side' equipment. That is there is no method for peer-to-peer traffic between cable modems in the DOCSIS standard even if the hardware is theoretically capable.

OTOH you can get a 2.5gbps MoCA adapter for like 50 bucks, and it can be installed, working, and reasonably secure in about 30 seconds. So I don't think DIY DOCSIS is a worthwhile pursuit outside the novelty.

I mean you can already do normal ethernet over coax if it's all you have in your building, thanks to MoCa.