First of all, there's no "Linux API". The kernel does expose a stable interface but you're not going to write a game just by issuing system calls.
But the kernel is pretty much the only piece of this tech stack that exposes a stable interface. Pretty much everything else changes a lot. There are exceptions -- SDL2, for example, is great to write against. But most things aren't -- they change a lot, and they're undocumented. This tends to result in duplicate code that goes out of date and results in all sorts of strange quirks and bugs. phk has a good summary of this here: https://queue.acm.org/detail.cfm?id=2349257 .
As for drivers: there's one set of drivers that aren't baked in and they're critical for games, namely nvidia drivers. Anyone who's done graphics programming on Linux has their own horror stories about nvidia's stack. I won't bore you with my own, suffice to say that things are never, ever as easy as "you just write against the OpenGL API and you're set".
"Historically", I'm not aware of any major console that's built on top of a stripped Linux kernel. Sony has been using their own franken-BSD since at least the PS3. Don't know about Nintendo but given how much effort has gone into the occasional port for Nintendo hardware, I doubt they're using Linux.
Linux has been only recently adopted by some console devices, and those are basically embedded systems. You target a single hardware platform (with a single, known set of quirks), a stable kernel, a stable driver, a stable set of userland libs. It's not great -- backporting kernel fixes is notoriously finicky, for example. But it's a scheme that, at the very least, allows you to write code to implement whatever functionality you want to implement, instead of wasting time impedance-matching between userland APIs.
If Linux were just a niche platform, that would be fine. But supporting desktop Linux systems is disproportionately difficult -- it's a small niche, but it's a lot of effort. Userland APIs break all the time. The constant churn in the graphics and desktop space leads to a lot of instability and bug reports. Between Wayland being default on some distributions, unstable graphics drivers, and the umpteenth rewrite of God knows what other component in Gnome or KDE, you get to deal with the sort of bug reports and crash dumps that used to pop up fifteen years ago when a Longhorn beta was leaked and everyone tried to run their games on it for shits and giggles. Linux desktop is basically a neverending beta. People don't write games (or software) for it for the same reason why you don't write games for a leaked Windows beta.
It's no coincidence that Linux is such a big player in the embedded and server markets, but not in the desktop market. Writing desktop software for Linux in 2020 is about as enjoyable as chewing nails. It's an unstable and flimsy platform to target, and not a very profitable one, either. I can't blame any company for skipping it.
Edit: someone else posted this here and yep, it's spot on: https://twitter.com/bgolus/status/1080213166116597760 .