I'm pretty sure that, once you populate a class instance with your data, the rest of Django is usable with very few changes. It's pretty loosely coupled, generally.
Yeah, but most of the libraries that make up Django (and Rails), such as auth, are based around the ORM so if you don't use the ORM, you have a framework mismatch.
Django (and Rails) both came from a time when the RDBMS ruled so they were built around an ORM and are very ORM centric. People are moving away from using a relational database and onto using Mongo or a graph database as the primary datastore.
Graphs are a much more elegant way of storing relational data, and so I prefer a framework that isn't so tied to the RDBMS. Now I use Flask as my Web framework and Bulbflow (http://bulbflow.com) to connect to the Neo4j graph database.