Of course, if you want to be (or are) an engine programmer, go for it and get your feet wet using OpenGL/DirectX, but for the broad mass of game developers, these aren't issues anymore.
Playing with it directly can be fun yes, but it can also be frustrating if all you want is so much easier with a game engine.
Sure, Unity3d is very flexible and has given us some outstanding games. Like Kerbal Space Program, for example. KSP has quite a bit of modifications to make Unity3d work for them, they have had to do some black magic to make Unity3d work in space scale scenes on both, the physics and graphics departments.
When looking at dev talks given by the KSP team, it begs the question whether or not they would have been better off not using Unity at all and doing a specialized engine from scratch. There's no real answer to this question and Unity probably made their time to market shorter so they could actually ship and sell the game to keep the development alive. And then they went on later to rewrite a lot of the stuff they did early on.
But from a purely software engineering standpoint, they might have been better off not using an off the shelf 3d engine, because there are no space scale 3d engines available.
Again, for most people it makes sense to pick up an engine and go with it but there are use cases where it might pay off to invest the time and effort to build your own tech from scratch.
I'm doing a combination of scientific and practical computing, and I'm currently in the middle of the choice of whether to use DirectX or OpenGL. Portability counts, but so does simplicity. It's a very relevant question.
Several times I tried expanding an existing modular 3D engine (OGRE) towards my goal of displaying a huge virtual universe of planets and stars. While I got it rendering 3D planets after some work (where I basically had to implement manual rendering code anyway, since the engine didn't support the odd use-case of spherical terrain), contorting it to support massive coordinate spaces, huge view ranges, dynamic loading/unloading etc. was just far too messy to even hope to attempt in that engine.
After several attempts to cleanly implement a space-scale engine as an extension of an existing engine, I started over from scratch using pure OpenGL (WebGL). And this time, I managed to keep the code reasonably tidy while supporting trillions of stars, planets etc. as I had originally planned [1].
Would I have been able to do the same thing in OGRE if I kept trying? Probably... but likely with far more ugly code, terrible hacks, bloated engine, and slower loading times.
To be fair, I'm not advocating always "rolling your own" 3D engine. It's usually a huge task, and using a pre-existing engine will in almost every case be the smarter choice. But with some people trying to argue "you can make it fit in [Unity/OGRE/etc]", I thought I'd provide at least one anecdotal counter-example.
1. Kosmos: A virtual 3D universe in your web browser (https://github.com/judnich/Kosmos)
Yes and no.
If you want to be writing a typical 3d game like an FPS, and RTS or 3rd person platformer, you can and probably should use an off the shelf 3d engine and stay away from the lower level APIs, OpenGL and d3d.
However, if you want to do something more exotic, say processing your webcam input stream with your GPU, or write a space scale render of the solar system or do anything that diverges from the use cases you typically use 3d engines for, you might still be better off using D3D or OpenGL directly.
Then there are aspiring future game engine programmers and hobbyists who want to use OpenGL directly. It's arguably more interesting than paying for a 3d engine but it's more work of course.
So most people do not need to touch D3D and OpenGL, in the same way that most people do not need to write C. But some people must and some people want to.