The different between libraries and frameworks is control. The framework tells you how to do it. But just using libraries, you tell the libraries how to do it.
> If you're building a web app for low/moderate traffic (i.e. not Youtube or Facebook) that talks to a database, authenticates users, has a load of assets that require preparing/precompiling, serves dynamic server-rendered templates, implements caching etc you may as well use a framework.
All of this can be done with libraries.
Or you could use Rails.
Using a framework is optimizing for the first few months of development and shooting yourself in the foot later on. That especially includes the time spent either finding devs who are very competent with the framework or training someone.
If you just use libraries properly, you can hire anyone (even someone who doesn't know the language at all) and just throw them into the code base. They can follow the code and actually understand it without googling every 2 minutes. There's no magic.
Have you tried both approaches long-term on multiple projects? If you haven't, how can you know that frameworkless is more work?
Not all from are as magic as you say, and if they were such a pain they wouldn't be in such demand. For most people they provide a helping hand.
You might be expert enough not to need one but most people, myself included, quite like having the foundations laid and just concentrating on functionality.
I wouldn't classify Sinatra in the same way as the other frameworks you mention; it's very lightweight and (intentionally) doesn't offer much out of the box. It's ideal for building APIs and other simple things but, if you need a fully-fledged site that does everything listed above, the case for just using Django or Rails is much stronger.
I now build APIs and services with Go. It's well suited for the task and makes deployment super-easy.