Many highly competent individuals like to understand the fundamentals and first principles of how systems work and how different problems can be solved, and thus find work that provides no transferable knowledge/experience to be less appealing.

Of course, there is a balance here. Sometimes frustrating tasks must be done. In that case, ideally the end goal is sufficiently motivating.

Right, but to those people, whether you use Docker or Kubernetes or custom scripting or a unicorn whose horn magically creates new instances when you need them is mostly just an implementation detail. It's no more interesting than exactly which compiler or DB you use; these are just tools, means to an end. If the differences are significant for your use case, sure, you evaluate and choose accordingly, but then you use the tools to get on with the job. The deep, interesting stuff is almost always in what you can build once you're into that territory.

However, I do think it's likely there are some fundamental principles at play that make tool 1 better than tool 2 in specific contexts, and I think it is valuable and interesting to explore that.

Perhaps we are getting into the scientist vs. engineer mindset. I would guess, though, that the best engineers undoubtedly have some scientific curiousity helping them excel in their craft.

One extreme is that I see companies hold on to toxic employees because they're brilliant, even if they are bad for the company long term.