A framework gets you going real quick, but at some point, probably 3-6 months in, you'll need to do something that the framework doesn't like. At that point the framework suddenly becomes a hindrance rather than a help. You spend more and more time fighting the framework and writing more and more code that lives outside the framework. After a few years you've wasted way more time than you ever gained from using the framework.
Libraries are built to provide capabilities to your code. Until you've experienced a good standard library (like Golang's) then this makes no sense. It's common for newbies to Golang (like I did) to try and create supersets of the libraries and create their own framework. But every new project comes back to the simplicity and ease of using the standard library.
I like the OP's approach of having repos full of useful middleware and handlers. This makes much, much more sense for Golang development than a Django clone.
Try it, you'll like it.
The fact Django is batteries-included, and everything is well documented in a single place, developers aren't wasting time working out what they can or should use for URL routing or handling database connections.
So no, I stick by my comment. A good framework saves time by being easier to learn, creating a more homogeneous product that is easier for other <framework>-developers to pick up, not forcing developers to form opinions about architecture they just need to work.
It's great not to have to worry about frameworks, that often last just for a few years, than you have to rewrite or migrate to a newer partly incompatible version.
Yes, some platforms (e.g., WordPress) give you certain functionality off the shelf, but if your application isn't a CMS or blog, then the utility of that functionality is low. When building a custom REST or REST-like server for an application, a CMS platform isn't necessarily a reasonable starting point.
As with anything in technology, many variables need to be evaluated to make a decision. My point here is that performance is a consideration—more significant and valuable than many people understand—and it at least somewhat objective. Developer productivity is more difficult to objectively measure and varies by team on matters such as previous experience.
You shrug off the established players but all else is very much not equal. Until Go has its own mature Django or WordPress projects, you're spending a lot more time reimplementing the wheel.