A framework DOES include stuff that you're not gonna need, and also makes several architectural decisions for you. That's a given.
A custom build (with libraries) site does not imply it will be done in some bizzaro way ("cereal in the closet", etc). That's NOT a given.
So that particular accusation against frameworks is definitive, whereas what your particular accusation against projects not using frameworks is a maybe.
(Not to mention that some frameworks, e.g a lot of Java ones, do enforce themselves a "cereal in the closet" --or AbstractFactorySingletonFactoryProxy -- mentality).
On the other hand, if you store your cereal under a squeaky board in the kitchen, that's not helping anyone.
You could easily walk into my house and find the kitchen. The stove and sink are both there as expected. But despite how carefully I thought it through and built it, and despite the fact that it made perfect sense to me, I doubt you'd find my custom-built intranet app easy to navigate.
On the other hand, my last Rails app has a structure familiar to any Rails developer. The actual business logic is unique, but that's the part that has to be unique. "How do we update the database schema?" is a given.
And if it's unfamiliar, there's plenty of documentation and blog posts to help you. With custom-built code, there's no help on the web.
Yes, and that may be a fine decision. If it's your house, and you want ringing the doorbell to play a symphony, and you want to store your cereal in the closet, that's fine. Whatever make s you happy.
The issue is when you ask other people to live in the house with you. How hard will it be for them? Is the cost worth it?
Maybe yes, maybe no.