I also write my templated queries myself and talk with the server directly. No need to go "FizzBuzz Enterprise Edition" for a simple task.
In your case, if the data is not fitting into neat boxes, it's not being forced into that in the first place. I select tools/features according to the problem at hand, not try to fit what I have into the problem at hand. If that requires a new language to learn, I'm down with that too. This is why I learnt Go, for example.
Sometimes the design stretches to its limits. If that happens, I stop, refactor and continue development. I call this outgrow/expand model.
It's not the fastest way to develop software, but it consistently gives me the simplest architecture to implement the requirements, with some wiggle room for the future.
For example, the latest tool I have written has broken the design I made in the beginning, because I overgrown what I designed (i.e. added more functionality than I anticipated). Now, I'm refactoring it, and will continue adding things after refactoring.
Every tool, every iteration brings learnt lessons, which are stored in a knowledge base.
What I am saying is that Objects are not rows. And we sometimes try to force the Object model onto a data-schema that ultimately is more rich than a chain of objects.
Which is not using the most appropriate tool for the job, per your example.
An ORM is _fine_ for stuff that has a fairly standard shape, like blob posts, user accounts, things like that. Lots of relation questions against persisted data end up not being those exact shapes and patterns yet folks generally reach for the ORM.
Does object handle everything? Of course not, but having it as a capability at hand allows some neat tricks and tidy code.
I believe every problem needs a different mix of capabilities/features, and that particular mix directs my choice about which tools to use. This mindset always served me well, but who knows, maybe I'm the dumbest one in the room.