Frameworks and libraries are two different points along a continuous line (not even endpoints). Not black and white, just different shades of gray.
I haven't looked at Zend in a long time, but the difference between Zend 1.x and a pile of libraries was marginal, yet Zend was labelled an "enterprise grade" framework.
(Not dissing Zend, this was a deliberate design approach.)
We can fix this, though. The best part is, the skills learnt and code written while doing so are completely transferable and directly usable by the big frameworks too! So, it's a win-win :)
For what it's worth though, yeah I totally agree with you. It's not an either-or situation.
Its actually something we've been doing for an API at work. The main reason was so we can easily port legacy code [read hundreds of includes/ and that type of fluff] into a PSR compliant system pretty much on the fly without having to jump through all the hoops of a framework - all while still having access to everything in packagist.
I'm currently working on a series of articles (maybe a small ebook, unsure yet) on developing applications in PHP -- sans frameworks like Zend, Laravel, Symfony. All those force a way of thinking that is very powerful if you're a good software engineer and web developer, but allow for a novice to build things while not learning the "right" way to do it.
So, PHP 5.5, Composer, and Packagist, combined with the PSR4 mean that a complete novice, and a hardened web dev, can take small composable libraries and create powerful reusable applications and libraries themselves, with best practices and great code. It's the anathema to PHPs historical shittyness. It's how I've been building my current startup. It's the future of PHP web dev, side by side with those big frameworks for when you need the big guns and a large team, etc.
It's made developing in PHP awesome, and I never thought I'd say that. Now the issue is in educating others to give it a try :)