Sure, Unity3d is very flexible and has given us some outstanding games. Like Kerbal Space Program, for example. KSP has quite a bit of modifications to make Unity3d work for them, they have had to do some black magic to make Unity3d work in space scale scenes on both, the physics and graphics departments.
When looking at dev talks given by the KSP team, it begs the question whether or not they would have been better off not using Unity at all and doing a specialized engine from scratch. There's no real answer to this question and Unity probably made their time to market shorter so they could actually ship and sell the game to keep the development alive. And then they went on later to rewrite a lot of the stuff they did early on.
But from a purely software engineering standpoint, they might have been better off not using an off the shelf 3d engine, because there are no space scale 3d engines available.
Again, for most people it makes sense to pick up an engine and go with it but there are use cases where it might pay off to invest the time and effort to build your own tech from scratch.
I'm doing a combination of scientific and practical computing, and I'm currently in the middle of the choice of whether to use DirectX or OpenGL. Portability counts, but so does simplicity. It's a very relevant question.
Of course, if you want to be (or are) an engine programmer, go for it and get your feet wet using OpenGL/DirectX, but for the broad mass of game developers, these aren't issues anymore.
Playing with it directly can be fun yes, but it can also be frustrating if all you want is so much easier with a game engine.