I'm currently working on rebuilding a fairly large Django app in Flask. My situation is exactly what you describe, the use case is not one which Django is really built to accomplish. The result is lots of difficulty dealing with the data. I feel like Django is great if you're building content driven websites. Our website barely scratches the surface of our application, and the Django backend has become very.
This is why Django is great for a particular kind of sites, and not as great for other kinds. "Originally designed for the newsroom", as they note on their site; it still feels.
As it's typical with frameworks, Django helps you as long as you do things "the Django way", and does not help, or counteracts you, if your problem domain does not fit well into Django's.
I suspect it's the case with any reasonably specific tool. A hammer is damn efficient as long as you have to deal with nails, and a wrench is great for nuts and bolts — but neither is going to help you to assemble an electronic device. Choosing the right tool for the job is as important as always.
Django, on the other hand, is not a tool, but a toolset ("framework"). Replacing any tool in it is not impossible but loses some of the affordances of the original custom-adjusted tool. I used SQLAlchemy with Django, and it definitely can be done, but you lose e.g. automatic admin pages. I had to go to immense lengths to change the way settings initialization works in Django when I had to update some of them after `import settings`.
Flask is a more loosely coupled bunch of tools. E.g. SQLAlchemy is an entirely separate project, just adopted to work in Flask with a custom layer. From the point of view of long-term development, it's better, it's more flexible. From the POV of a rapid prorotyping of a typical CRUD app, Django is better because it's more integrated. It will bite later, but at the start, most people are ready to make technical debt in exchange for velocity.