Dropping down to raw SQL is equivalent to rewriting the affected components. This might be trivial with simple CRUD apps but with large, complex systems that can be a big undertaking. Not to mention that if you end up spending a significant amount of time using straight SQL instead of the ORM that draws into question the value of using the ORM at all. Not to mention causing maintenance and development headaches down the road.

Also, the "you can always drop down to SQL" defense of ORMs is extremely weak. It's great that most ORMs are not so intensely broken as to prevent you from not using it, but in general the "you can just stop using it" argument is not a strong one. Imagine if C++ was defended based on the idea that you can just drop down to raw machine code if you run into problems with the language.

All code has bugs. In my experience ORMs don't save you from having to drill all the way down to native SQL to debug. Then you figure out what SQL changes you'd need to fix the bug and from there you work your way back up to how you'd need to change your ORM config or client code to effect the same thing. ORMs are typically some of the most leaky abstractions of any sort. If people spent as much time examining and tweaking machine code as most ORM users do with SQL code people would question the usefulness of their "high level" programming languages.