Yep, that matches my experience. There was a lot of Feynman debugging as well. It's a good job for somebody who is eager to learn new stuff and get out of their comfort zone. Another great thing about it is that people tend to stick around for a long time, so you build good friendly relationships with your coworkers.
The biggest downside of driver development is that it becomes very repetitive after a while: you are the maintainer of this immense complex piece of clockwork (the hardware) that you must to keep ticking reliably at all times by working around all the bugs in it. It sometimes feels like your beautifully crafted software contains as much code to track and work around hardware bugs as it contains code fof the idealized hardware described in the origibal documentation.
The best firmware teams I've worked on have always had a mix of disciplines. Heck, I mentored a mechanical engineer for a while (he became a very good firmware dev). You have to have good programmers who manage to write modular, maintainable code even if the language and hardware limitations fight them the whole way. You need folks who understand Electrical Engineer-speak. Depending what you're doing, you might HDL people, and that stuff might as well be a completely separate discipline from regular software. You need people who are comfortable debugging with nothing more than log messages, and if you're lucky, a coredump. You need a manager who can wrangle the occasional "it's a hardware problem" vs "it's a firmware problem" debate.