The problem is a bit more insidious than that

As an engineer, you want to be working on cool new features too! Very few folks will be content sitting on their laurels just fixing the occasional bug or adding a touch more polish to a product that's already "done"

If you setup a team to work that way, very soon you'll find that most of your engineers have left. Heck, the manager might get bored and leave too.

"Okay, that's fine," you might think "The product is still doing alright even without an owner. Higher level leadership should be fine with that"

Until the day comes when the service crashes unexpectedly, and you realize that no one left on the engineering team has enough context to debug the issue properly

Hello two week long outage

Examples: Heroku - https://twitter.com/GergelyOrosz/status/1520770263977271296 Atlassian - https://twitter.com/GergelyOrosz/status/1513605414029516806

Part of this is due to companies shooting themselves in the foot over and over, recruiting developers looking for challenges rather than grunt developers okay doing largely maintenance for a solid income. If they advocate themselves as providing the challenges for the former and filter out the latter, yes, obviously your employees are going to leave after they have to move that one div by 5 pixels for the umpteenth time and get no mental stimulation for months.

Does anyone recruit grunt devs like that? That honestly sounds like what I'd prefer. I just want to come in, keep the lights on, and have enough mental energy for other stuff after work.

I don't know too much about Google, but I can talk about other companies. Everyone recruits grunt devs like that. That's what every job in Silicon Valley is. It's just that some companies and teams want you to spin it like you're some amazing lone wolf 100x genius while you're scraping dogshit off the bottom of the company's shoes.

A big part of it is that the executives decide what's making money for the company, and they'll focus on that. If you're scraping turds on the "rockstar" team in the "rockstar" org, you'll get showered with bonus money and RSUs. If you're scraping turds on a product that none of the VPs care about, you'll probably get screwed over. Some of the people in the non-"ninja"/"wizard"/"rockstar" orgs will do OK because they look like indispensable geniuses, and I think that's what a lot of this sentiment comes from.

> That's what every job in Silicon Valley is

I am curious - is this informed by your experience of having sampled every job in SV? Or perhaps a representative sample? Is it possible other people might be different than you and have a different perspective.

What a fucking joke of a comment.

Well, you have to maintain things after they're made. New things are usually made out of multiple old things. Figuring out how stuff works, then fixing and reusing, supporting, or replacing it is the bulk of every programming job I've done or heard of.

I can't even conceive of what programming job wouldn't be describable as "grunt work" from the point of view of some hypothetical super-genius. Writing "new" code that just does the same thing as something you're tossing out, but with a different set of mistakes? Cutting-edge science work is mostly glue code between hardware and spreadsheets or databases. Cell phone firmware is mostly repurposing and updating the old firmware. Anything involving a GUI is going to be a massive time-sink getting the GUI to work the way the non-technical people using it want it to work.

How much time do you spend in any given year figuring out how to operate debuggers, find the right log files, narrow down some crash, etc?

As you said, those are required components of shipping software. I don’t consider them grunt work, any more than I would consider being a janitor as a grunt.

Grunt implies a heirarchy and a job that anyone can do. It’s pejorative. There is work that I don’t enjoy doing but needs to be done well (cleaning a toilet). I appreciate those jobs even more than what I do!

My point is that you should try to shift to language of “work I don’t enjoy” rather than “work that is beneath such an exalted 10x programmer such as myself”. Because that’s how I interpreted your original comment.

All across corporate America, there are devs who make 80-120k a year and keep it 9-5 (but really like 11-2). Especially with the recent turmoil in the market, you can find a very easy yet well paying (for normal people) job. Now if only HR would advertise the jobs this way...

I could see a market for that guy, and he takes 10 jobs like that Reddit thread.

Experienced something similar, but it was not an outage. Had several knowledgeable people leave the company, and they all happened to be experts in a particular service. Positions weren’t backfilled even though the people gave advance notice. About two months later we ended up getting out butts kicked when nobody knew the details of the service implementation and the service was expected to be updated to support some new features.. couldn’t get it updated for about 3-4 weeks because we couldn’t afford an outage.

> couldn’t get it updated for about 3-4 weeks

That doesn't sound like a large issue (but the details can completely change it). Maybe it was the right decision.

Kind of experienced that when interviewing for one of the well-known music steaming companies years ago.

They had one critical service in Erlang and the developer who coded it had left years ago and nobody knew the language so they wanted to rewrite it.

I was starting using Elixir back then and said I could maybe took that code over but they chose the other candidate ;)

Arguably those folks who left didn't do quite as good a job as they perhaps should have when they were still there. A high quality developer leaving a service behind should have already written sufficient documentation so that another high quality developer (especially one at the same company) can ramp up more quickly than 3-4 weeks. I think this is just another symptom of many tech companies throwing up their hands and pretending like "legacy" services are inescapable technical debt, when really they just never bothered to emphasize to their employees that services should be built in a maintainable way from the outset.

This is true but only so long as they have been given time to do it. I've left a company, with notice, and they had me working on building stuff almost to my last day. Sure I did have a few "hand over" sessions in my schedule where I was expected to walk other devs through some stuff I was the owner for in a broad sense, but they never gave me time to produce solid docs for anything even knowing I was headed out the door

It can get worse: me and my teammate trained a new dev for half a year with the explicit intention for him to take over should one of us leave. When the day came, said new dev was moved to another team, the product was given to an outsourcer and I was told to train them how to develop in two 2h sessions before I left. (The product was a legacy mess we were in the (multi year) process of refactoring into something maintainable. Documenting the intentions midway or even the status before was beyond the time budget we were allotted. All of this while doing feature development, without having proper requirements or user stories or access to customers who understand their use case, of course.)

That sounds rough. It's very painful not having time to do things properly, and it's also painful when you don't have proper requirements so you can't even be totally sure what properly even is. I semi-recently left a job (the one I was talking about originally) after more than once being told "we just want a toy, proof of concept version and any time you spend on things like performance is a waste of time and we will scold you for it, even if there isn't any other work for you to do" before they decided to ship my "toy proof of concept" to a client requiring an ungodly rework of it while it was running in a production environment.

I honestly think people denigrate having at least some actual design work and requirements done upfront as being "too waterfall" and think doing it is a bad idea so we as an industry end up with not even the intentions of how a piece of code should work written down and no idea how stuff works once the original author moves on

That's exactly my point. If companies aren't expecting their developers to document their services - while the developers are still employed and building those services in the first place - that's a failure in my books. It's exactly the sort of corporate culture that props up this mindset of "promotion-driven development".

Ah my apologies, I thought you meant it was a failing of the developers themselves - which it definitely could be in some circumstances - and I felt the need to point out that a corporate culture can lead to docs not existing.

Yeah and this problem is exacerbated at Google by the relatively easy process of moving between teams. Other companies I was at made this hard. Google encourages it.

When friends complain to me about Google killing projects, they act as if it's upper management making a decision to kill. And that is sometimes the case. But often it's just: nobody wanted to work on it anymore. And Google isn't the kind of "command and control" culture where you crack the whip and tell all your engineers that they're doing X and assign them. At Google they'll just leave your org and move to some other team, and you won't have the power to stop them.

Not saying that's a bad thing, but it has some bad outcomes occasionally.

You can find maintenance experts too. When someone tells me that they have this important, dangerous, buggy system wthat has trouble handling its ever growing load, and they need someone to come in and fix it, I cannot be any happier. But I know the system has to be really important, and that the fire has to be so bad that upper management is willing to spend the money to let some specialists come in, few questions asked, knowing that they will be rewarded as if they were building a new shiny, high visibility doodad. What is difficult is to have management that will keep that level of attention, instead of realizing that the reason that nobody has heard a bad thing from the system in months, if not years, is because there's a lot of work being done trying to make the system invisible.

What usually happens is that after a year or two they forget, and the maintenance programmer jumps to a different fire, typically in a different company, because there is far less reward to keep improving the system than to stop it from being a raging fire.

There are plenty of people who like fixing the occasional bug and adding a touch more polish. The problem is that this is actively disincentivized on every level. On top of that, a lot of people have internalized this.

If the world weren't so obsessed with differentiating pay, you would have people who are enthusiastic about a wider variety of things.

I think you nailed the crux of the problem.

The challenge for engineering management is how to provide metrics to measure your bus factor reduction efforts and the strength of your insurance prior to the emergency.

It is highly possible though that the new support team members are actually coasting up until the disaster so you didn't really have the insurance you thought you were paying for.

There is that too, but you could solve that by having engineers/teams working in new, cool, and useful products. I feel my company doesn't have a good mechanism to maintain services that aren't actively developed.

We experience it first hand when one of our services got deprecated and we moved to a new org. The solution was literally to hire a new team in a low CoL country and hand over the service to them. Needless to say it was difficult to hire for those positions.

Isn't the 20% time to work on whatever cool shit you want enough ? (like how googlers created that garbage angular1 because they were border, have no clue about GUIs and had some fun scewing around ) ? I know people that are fine with getting paid to maintain shit so maybe the problem is Google only hires cool developers and the cool developers only want to work on cool stuff and in 2 years the newest cool stuff of-course.

It's a lie.

I think this is one of the key frustrations I have with modern software development, change for the sake of change. I feel as though many products degrade over time and as a user I’m generally quite hesitant to upgrade anything if I don’t have to.

It's a whole "growth" thing.

Like, it's not good enough to have a quality product that generates a sustainable revenue stream year after year. You have to "grow" because companies don't really do dividends anymore, they want a constant increase in stock prices, product be damned.

They have to show on investor calls the line went up.

We’re propping up the wealth of a generation that has no idea how anything works, but they got there first so of course they are now the de facto deciders of our agency.

True, it’s difficult to know how things work when they become needlessly complicated and one is unable to move on from a project that is essentially complete.

It’s presumed needlessly complicated is necessary to make the line go up; behind the scenes, they say, mathmagical forces exist that only the chosen few truly understand. You see everything is really a graph of infinite brand names, and the flow of value is determined by the taxonomy of brand ownership.

Many a study have been attempted to quantify what character qualities or technologies boost productivity. Their model becomes so nebulous no reasonable conclusions can be made.

But now of course computers learning helps us untangle that web and low and behold the same economic winners emerged! Wow!

> If you have a team, you need to "justify" their existence. Is not enough to keep the lights on or slowly polish the product, you need grand roadmaps to keep yourself busy the next year or two. Ideally you want to justify that you need extra headcount to keep the product expanding.

This. It generates the Bullshit Jobs that David Graeber talked about. As a middle manager or tech lead (Taskmaster) you hire people (Flunkies) to make yourself seem more important as well as for roles (Box Ticker) that you might not need but that any "important" project will retain. In the end, this generates duplicate effort and needless work that requires fixing (Duct-Tapers). The only one of the five Graeber categories not represented is the Goon, and that's because those get moved to MTV and fast tracked to the executive suite.

> and that's because those get moved to MTV and fast tracked to the executive suite.

"The Goon" is the analyst from an investment firm who tells your directors and CEO how many of you are going to be fired to make him happy.

This sounds like Eric Weinstein's Embedded Growth Obligation (EGO) concept. Notably also occurs in academia and economics.

Arguably one of the causes of this is high salaries and only hiring (supposedly, I have no personal experience) Very Smart Developers. You're going to pay Google salaries for someone just to keep the lights on?

It would be a smart business decision. A Google mid-tier employee or two just isn't a large cost to keep a project running. It's not cheap, sure, but to have a product running at scale?

The rationale for developing Go is that Googlers are fresh out of school and only barely smart enough to program Java, so I think they’ve eased off on the notion they only hire super-geniuses.

I think it's more that they're smart as hell but know very little or have poorly developed taste. So you have to give them a short leash otherwise they'll hang themselves with it through over-engineering and accidental complexity.

The literal quote from Rob Pike is "They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.".