I've had a conversation with a few bosses where at some point I stopped to say words to the effect of, "It's like you think I expect everyone a the project to be like me. I don't." <looks of surprise> A project with just me would have problems too. But every project needs at least a couple of people who think like me, which we aren't accomplishing.
This whole expert thing isn't tenable on a long term project. The expert will get bored and wander off, at which point all the mistakes they avoided will just be explored in their absence. Unless they built up people, documentation and processes to take over the work.
My read of Kernighan's Law, back-ported onto my own philosophy, is that most of the time you don't want the most capable person working on a solution. You want the 2nd or 3rd most capable person on it, have them reach consensus with the other two. Maybe 10% of your code should be as clever as you can manage. Most of the rest should be so boring that only the junior people find it interesting. The bottom rungs of an orchard ladder everyone is trying to climb.
And from then it’s a mentoring role. The expert spends more time PR-ing and adding to the framework than building end product features. They build stuff that touches every feature.
Force multiplier is the game.
Because how good is the expert? They just built a bunch of stuff to enforce one approach; they're invested in it even if they're wrong, or even if the project changes.
The ideal is an expert who will be involved with the project, offering suggestions, providing a sounding board to non-experts working through issues, etc. Mentor and guidance, yes, but without first trying to lock people into a specific trajectory. And if ever the non-experts decide to do something different, -that's their prerogative-. Because it's they who will be dealing with the fallout if they're wrong. And unfair to make them deal with the fallout if they were right, and the expert was wrong. It's incumbent on the expert to convince them, not dictate.
Now the guy is gone and we are too invested to be able to fix any fundamental issues without incurring pain, so it's not done.
It's fine if people want to do something different: just first become an expert.
The other problem is that there aren't enough experts and how can they become experts without strengthening expert muscles. They have to practice and learn.
Depending on the distribution of abilities, what I've found to be give the most leverage is to convey a few specific design ideas significant for a particular project to someone on the project who can understand or at least commit to using them. You can check in from time to time and provide some input and clarity. Regardless of good intent on all sides, results still vary. But if done smoothly, everyone gets better at working together on high-level tech things and there's no resentment that doesn't fade away quickly.
Ensuring the non-experts understand their purpose is the role of the experts. NOT making the decision as to when to deviate. That's my point.
An org that leverages an expert to make others experts is an org that ends up with a team full of experts. An org that leverages an expert to make decisions for the non-experts is an org that is going to have a lot of issues in production, and likely high turnover.
If any one of those things is missing, it doesn't work. And in my experience, the most likely ingredient to be present is it willingness of the experts to teach. If you're good at something, it's fun to explain it to others.
Without it, you start out with a multiplier but end with a divisor.
Lone points of breakage such as a single visionary or the mythic 10x-er are not optimal for the product if the unit taking care of the product is a team or several teams.
1.) Mentoring is such a difficult aspect as an expert. I likely would have built up my expertise from a combination of my studies, my past work experience and my previous, extremely skilled mentors. I can't expect people to get up to speed simply through my mentoring alone - especially if I'm not a capable enough communicator.
2.) Experts like to retain their expertise, stay unique. Regardless of what idealistic scenario you have in mind, everywhere I've seen, experts are always very cautious to giving away ground. I've been guilty of it, especially when I knew that my unique expertise was responsible for my sizeable bonus check in an IB.
3.) Experts will not do all the grunt work you mentioned. They come with the expectation that they've been hired for their expertise and not to be another cog in the wheel, but the engine itself (for a particular project).
Ideally now, what you would want would be a team of generalists, and a small crack team of experts (or a lone wolf expert). The generalists set up the project, plan how to build the guard rails, etc, then explain it to the experts, then when both are up to speed, generalists build it out and give the heavy lifting work to the experts to take over from.
Converting a generalist to an expert is a time intensive process that has to be structured and well-thought out. It should not be for just one project, but should continue for multiple projects.
In sports they call that a Ball Hog.
Is software a collaborative effort or an ego trip for three people?
As someone else mentioned, their expert was spread too thin. I’ve had this happen to me, too, and the only solution I know is to give away (delegate) almost everything you can. If some part really makes you happy, hold onto that even if you could give it away. Not for ego, but for burnout. You need some reasons to show up in the morning, just like everybody else.
In some sense, that's exactly how top level frameworks like react and Django get made- experts build them to make it easier for the rest of us to write the business-specific logic.
When it's done well, I think it's a fantastic approach. I'm in a situation right now where I'm kind of that "expert". Not because it was planned that way, but because I was the only person on the project at the start. Now I've got a couple of young EEs working under me, and they do most of the development work while I sit in meetings for a good chunk of the day. I come in and help with the hard problems, but most of the day-to-day work is on them. It's working out great because it seems I got the structure relatively right (they've changed things a little bit as they've added new things I didn't think of, and I've reviewed and supported those changes)
The flip side is ${JOB-2} for me. It was a similar situation; an "expert" laid the foundations and then got pulled off to do other projects. There were two main differences between my current situation and that one:
- The infrastructure he put together was not great and full of footguns (e.g. accidentally implicitly double-encoding HTTP requests, bizarre cookie handling, etc)
- The code was hard to change because of how it was integrated, and the team didn't feel particularly empowered to fix it
- He was barely around (being assigned to something in a different building).
- He pushed back hard when people suggested that there might be problems with it and fixes required. Instead of facilitating and reviewing, he'd pop by occasionally, look at things that changed in his absence, and shit on them.
As I'm writing this, I'm reflecting on whether or not I'm contributing enough to the long-term development of this codebase, and I'm feeling relatively ok with the answer. Ultimately, this product is still my baby and whether or not it's successful is on me. Maybe that's the difference? I'm quite invested in the foundations I laid, and more than happy to help my team work through issues they find in it?
I think you're right that having skin in the game is critical. It's important to have real accountability, so if there's painful design flaws, the expert feels it.
But it's easy to think it should have been done differently and better when you don't have the perspective of the designer. I still vividly recall being a junior engineer and thinking the design decisions made by the "expert" were ridiculous, and talking with the expert, and finding out I didn't know everything and I was wrong. Thank goodness he was patient.
I'm laughing pretty hard at this. The codebase in this current project is definitely a bit... quirkly. It's semi-embedded (Linux on an 8-core ARM unit) and doing high-performance image processing (C++ on Linux). The new guy last month has a fair bit more programming experience than "fresh grad" would lead one to expect, and the first week he had a lot of questions like that.
Him: "Why is it like this? Couldn't we just..."
Me: "Would that keep all 8 cores busy processing things as fast as possible?"
Him: "Oh. Yeah. I guess not!"
Him: "Wait... couldn't we just...hmmm...no... that would serialize everything..."
I suppose one other angle to it is that I don't have a lot of ego in it; if someone figures out a way to squeeze more juice out of the system we've got, they're going to get high-fives.
Google et all probably have lots of internal tooling and frameworks. Hell, react started as internal to FB didn't it? Does that count as a framework or as a product?
Lots of other companies have similar internal frameworks, probably with less resources devoted to them, but in the other hand with less use cases. Do those not similarly count as frameworks written by experts?
Personally I'll take a generalist any day since specialists suffer from the "if you only have a hammer everything looks like a nail" syndrome.
Having said that, my 1 career advice to anyone would be, become a specialist in 1-single thing, and then become a generalist in everything else. I'm not that, but I feel that that I would have been best served if I had been that.
And yet... as 'boring' as those things are, I constantly see basic errors crop up all the time because people aren't even moderately versed in the technologies.
For example, you don't need to "know every crevice of a technology", but if you're doing LOB reports for business systems, I would reasonably expect you to understand what indexes are, or what joins are and how to use them, in your SQL engine of choice,
The article isn't saying that specialists aren't valuable. In fact is says the opposite: "If you know exactly what you need, and you need it right now, choose depth." It is just saying that perhaps the short fat engineer would be the one who realizes they need an OpenGL specialist on the team in the first place because the graphics are hitting a bottleneck and slowing down everything else. The author is merely pointing out that in the nascent portion of projects their contributions are, at times, under valued.
to expand on this, I've found most of my happiness as a generalist in early stage startups. to me, a larger company looking for a generalist with no concrete role is a sign that they don't have their stuff together. obviously there are exceptions to this rule - some companies just prefer generalists - but usually once teams get big enough, people settle into specific roles, even as a generalist.
Or that their culture is such that they have entrenched, stovepiped personnel that don't venture out of their specific knowledge domains and they need somebody to bridge that gap.
An analogy that I've used far too often when describing software deployment in traditional orgs is one of the dev team and IT playing tennis with the grenade that is the software. With each volley back and forth they give each other nigh-useless information - the operations team will tell the dev team that it crashed without sending logs and that it's because of random setting X that they pulled out of a hat, then the dev team will tell the ops team that said configuration value is hardcoded but that can't be the problem because it runs perfectly on their dev machine, and so on and so forth. Having somebody to mediate this interaction so that it's actually productive is inherently invaluable to, y'know, getting actual work done.
Unfortunately it's not particularly valuable from a business perspective because if your org is operating this way then it means that they place greater value on the performance of "software engineering" than the actual value added by the software produced. So basically what you said.
Generalists could also be the people in the project who get things done: they may have sufficient domain knowledge to implement the recommendations of a specialist. In poorly-managed or understaffed teams, a generalist might feel a bit frustrated because they're a go-to for any Job That Needs Doing Right Now, even if it falls outside of the field of their deepest experience.
My general take is that generalists would excel in small scrappy disorganized organic teams, while "specificists" would do well in teams with high level of organization and management structure. Unfortunately, large organizations tend to push out generalists, as structure increases, and their value in the eyes of management decreases.
I think a large organization should have both kinds of teams. This kind of alludes to Clayton Christensen's Innovator's Dilemma -- "well managed" big orgs should be internally disrupted by small-scale "startup" style teams, and those scrappy organic type of teams need generalists.
The point is there are less project slots for generalists to be in right now.
That why as another post commented they move into management or product at larger organizations. They get to move the needle and apply their generalist knowledge even to mature projects.
Generalists are highly valuable for many types of orgs and projects especially nascent projects. But sometimes you need the precise output of a specialist to achieve something.
If my business has a product that is highly dependent of let's say OpenGL, then an OpenGL specialist will generate more specific output that someone who knows a little of OpenGL and a lot of other things.
However, there's always the counterargument that generalists can compensate with their holistic view and understanding of problems throughout the whole stack. I understand this and agree with it, but I think there's a point in every innovation where the generalist contribution declines sharply. And I'm saying this as a generalist.
If you are a generalist and you feel undervalued, you're likely in the wrong project or too involved in phases of a project where your contribution can't move the needle significantly anymore.