I really like Django, but you're right that if you for whatever reason needs to make Django do something that's it's not exactly designed for, you're going to have a bad time.

What I found is that if you can manage stay within the confines of Django, you will be rewarded and can avoid writing a ton of code. There really isn't to many situation where you can't just to things "The Django Way". If you find one, then maybe rethink your solution. I had a ton of code where I tried to fight the Django Rest Framework, when I stopped and just accepted that Django and the Rest Framework is opinionated I was able to reduce complexity and line count.

Of cause there are problems where Django is simply the wrong solution.