That depends on the scale of what you're building. Generally I would agree with you, but if you building some small, something that would only ever use a small subsection of whatever framework you pick, then you could end up having more issues than if you didn't use a framework.

Just because you aren't using parts of a framework, doesn't mean that security issues in the framework won't affect you.

Let's say your application only ever needs to do one or two queries to a database. It's going to be a whole lot easier for you to lock those two queries down, than securing an entire ORM.