I use a lot of custom Serializers (not tied to any Model), and lots of SerializerMethodFields to selectively override generated ModelSerializer fields; this can get you quite far.
I agree with your point that the further afield you go, the less you gain from the framework, though.
DRF doesn't like it when you do things that don't map to the models, and the documentation is pretty bad. I had to read the source in the end to figure out how to do what I wanted, and I got the sense that I was lucky it was doable.
If your database models map to your CRUD endpoints, though, everything is super easy.