An example close to my heart. The company I'm currently at had a totally ossified Spark monolith and data eng team because they only wanted Spark developers. Drop one C++ developer in there to start spreading the good news about data-oriented design and suddenly there are a dozen previously-intractable "big data" problems we can rip out of the ball of mud and run outside of the cluster. Drop in another SRE who doesn't know shit about Java but can actually write shell scripts, and hey alerts from bad orchestration from weekly to twice yearly.
Yeah if you have a Spark job, you probably need a Spark dev. But if you have a team of Spark devs, suddenly everything that involves more than 1MB of input data looks like a Spark problem and you're in for a real bad time in a couple years.
C++ programmers know map/reduce/filter too.
Thanks for generalizing Spark engineers to an insulting level, though.
I have no clue how you got that from what I wrote. I even said a Spark project needs a Spark dev!
It really sounds to me like you're the one generalizing from "the one C++ dev we hired couldn't do data engineering right at all" to "senior programmers are dangerous to us unless they are Spark developers" (which imo, is more indicative of serious deficiencies in your architecture / design process), to "all senior programmers crossing subdomains are dangerous."
The lat thing I need about any of my colleagues is to learning all gotchas, at the same time as integrating into our team.
Some tech I'll argue against, other tech I will require.
Some tech is fundamental to a domain. Some tech is a nice to have. Some tech is just generic.
My current team will not hire anyone for a senior role, without Spark experience. No matter how good you are in C++...