Definitely go for it. My very fondest years were spent staying up until 4 in the morning 2 nights in a row making shareware games in C++ in the early 90s before video cards, until my programming partner at the time and I passed out from exhaustion. There is nothing quite as intoxicating as the full control that comes with C-style bare-metal programming.

If you grasp CUDA, you'll be ahead of me as far as programming GPUs goes. I was never able to quite let my love of desktop CPU programming go to transition to shader and cloud programming. I'm more into higher-order methods like map/reduce/filter. But languages like Julia are working to unite the two paradigms under a common runtime.

There are many tools available to help you get started. I would highly recommend also learning Docker and git (if you haven't yet), so that you can rapidly iterate with full undo ability to remove fear of failure. There are many operations in git that I find easier than the built-in SCM in Unity, for example. And I know they can be used together to give you a local scratch sandbox, then commit your final changes when you're ready. I still want to do this, so you'd be ahead of me again.

And if you could go ahead and build J.A.R.V.I.S. for the rest of us, that'd be great!