News for hackers. Stuff that matters.
Re:Google Blew a Ten-Year Lead by abc_lisper on Wednesday June 24, 2020 @11:53AM (#23629405)
It is well known the incentives at Google are aligned at working on new features than maintaining them and keeping users happy. May be this is just a natural outcome of that. This is one of the most damaging things Google and the people it influenced did to programming profession. There is a generation of programmers now who can do very well on leetcode, but god help those who maintain that code after them - I know this from personal experience working on projects left behind by people who ended at Google
Re:Google Blew a Ten-Year Lead by EthanHeilman on Wednesday June 24, 2020 @12:01PM (#23629539)
All organizations that grow rapidly face an "Alexander bias" where the sort of incentivizes, perspectives and people that allowed them to rapidly capture territory cause them to value capturing new territory even when it is no longer possible or in their organizational interest to do so. Even when organizations identify the problem, evolving from the cult of the new to the cult of incremental improvement is always hard.
Re:Google Blew a Ten-Year Lead by abc_lisper on Wednesday June 24, 2020 @12:26PM (#23629958)
This may be true, but there have been articles in public since 2010s if not longer, that show general contempt for having clean code and code maintenance, like having clean code is what lesser engineers worry about. I’m saying that thinking seeped into universities where smart kids, who generally imitate the smart professionals in their profession first. Google may not be the only company guilty of it, there are places like Uber where backend code is held by so much duct tape , 3m shares rose up as a result.
Re:Google Blew a Ten-Year Lead by throwawaygler on Wednesday June 24, 2020 @12:35PM (#23630179)
That doesn't match my experience at all. If anything, Google suffers from the opposite problem: engineering teams spend enormous resources on migrations and rewrites that make their systems cleaner/simpler/more general, but have little or no business value. There are tons of teams that could be making incremental user-facing improvements, but are instead spending their cycles on projects that are mostly about internal development velocity.
Re:Google Blew a Ten-Year Lead by nostrademons on Wednesday June 24, 2020 @02:24PM (#23631862)
I think it's changed, dramatically. I was there from 09-14 and then just rejoined last week. In 09-10 there was a huge problem of "launch and run" - people would launch, they'd get promoted, then they'd be reassigned to other higher-priority projects to get them launched too, and their previous project would get shut down rather than maintained. Nowadays there seems to be a lot more emphasis on stability and long-term code-health, people are getting rewarded for internal cleanups rather than launches, and many people have multi-year tenures on the same team rather than lots of single-quarter launches.
Re:Google Blew a Ten-Year Lead by foobiekr on Wednesday June 24, 2020 @03:24PM (#23632661)
are they getting _promoted_ though. successful promo packet for sustaining work would be a very big change.
Re:Google Blew a Ten-Year Lead by iknowpromo on Wednesday June 24, 2020 @04:36PM (#23633535)
I've seen a bunch of successful promo packets for work on code health, system reliability, etc. - including promos to L6.

The "trick" to successful L5 and L6 promos (and above) for this sort of work is to have credible estimates of the actual impact of your work. Way too many engineers spend quarters or years refactoring or rewriting systems, and then go for promo with a case that's basically: "System X was kludgy, crufty, and engineers complained about it. I designed and lead implementation of a clean-up effort, and now people say it's nicer."

That's generally not going to cut it. You might get lucky, or you might get bailed out by a peer reviewer that provides solid data about the impact of your work, but you can't count on this, and you should be doing the legwork yourself.

You need impact estimates. That generally means you need some measurements, although the measurements don't have to be perfect. What metrics measure the pain that the existing system is causing? Some examples might be:

- Average # of hours required to push a release

- Average SWE-days/SWE-quarters/etc. required to develop a representative feature change

- Average monthly user-reported bugs

Spend some time actually measuring this stuff. Write some queries, run an internal survey of the developers on affected teams, etc. Take it seriously. Ideally, do all this before you've actually started work on the cleanup you want to do. Write a proposal doc, get it reviewed by others, and make sure they find your estimates credible. If the numbers are smaller than you expected, reconsider whether the clean-up is worth the time you're considering investing in it.

After you've completed your project, measure again. If your project is amenable to an experiment-style launch, that's ideal, but pre- and post- measurements are fine too. Share the stats - advertise your team's success! Package them up in a nice doc you can link to in your promo case.

"How am I supposed to find the time to do this while I'm doing my normal job?" This is your normal job. The opportunity cost of your time is $XXXk/quarter. The opportunity cost of your team's time is $X million-$XX million/quarter. The single most important thing you can do is make sure that time is being invested in a high value way. In your personal life, you would look for a lot of data before you invested that kind of money - you should be doing the same at work.

In general, as you move up you should expect to spend a higher percentage of your time figuring out what problems you're going to work on. As a result, the fraction of your time that you spend actually designing and implementing the solutions to those problems will decrease, but it's still a net win because you and your team will be focused on tackling the really important problems.

In summary, it's possible to be really successful and quickly move up the ladder by doing "sustaining work". The catch is that you have to be rigorous about choosing the specific "sustaining work" to spend your time on.

Re:Google Blew a Ten-Year Lead by dlubarov on Wednesday June 24, 2020 @06:52PM (#23634764)
Another ex-Googler here. This seems like a viable strategy, but it seems rather inefficient for the organization, don't you think? E.g.

> Write some queries, run an internal survey

> Write a proposal doc, get it reviewed by others

Even if one could convincingly claim a causal effect on those noisy metrics, that sounds like quite a bit of overhead. IMO the majority of cleanup or bug fixing work shouldn't require any sort of formal planning or justification. At other companies, we would just go ahead and do the work. Then if an IC's work shows a pattern of code quality improvements, the manager should take notice and make sure it's considered in any promotion decisions.

Moreover, the strategy you mention requires a concerted effort to improve the quality of a particular component in a short period of time. Ideally one should evaluate bugs and quality issues as they arise, and most of them should be fixed promptly. But if a Googler is optimizing for promotions, it seems better to let code rot for months or years, then fix many issues in a short sprint to create a nice dip in the metrics.

Re:Google Blew a Ten-Year Lead by amznthrowaway5 on Wednesday June 24, 2020 @08:32PM (#23635481)
These "objective" estimates are usually nonsense, or deceptively only show one side of the story. Not everything can be easily quantified.
Re:Google Blew a Ten-Year Lead by duncanawoods on Wednesday June 24, 2020 @01:52PM (#23631383)
Sounds like a different symptom of the same illness.

The team rushes at the problem, has a honeymoon period where things look nice and progress is fast but... it all then goes to shit. If your reaction is to blame technologies or the previous generation of staff, dump the code and repeat the cycle, you end up in the same place.

Re:Google Blew a Ten-Year Lead by mav3rick on Thursday June 25, 2020 @12:04PM (#23642217)
I love the arm chair analysis of every external person here. Code quality is extremely important at Google. That doesn't mean people don't write any features.
Re:Google Blew a Ten-Year Lead by foobiekr on Wednesday June 24, 2020 @03:23PM (#23632653)
each unit of additional reliability and supportability is probably a bigger impact for google than any given user feature.
Re:Google Blew a Ten-Year Lead by mav3rick on Wednesday June 24, 2020 @07:33PM (#23635072)
Google infra is far from duct tape.
Re:Google Blew a Ten-Year Lead by mav3rick on Wednesday June 24, 2020 @05:28PM (#23634079)
You don't know anything about Google's engineers or internals. If anything, people complain more about an anal attitude towards cleanliness. Talk to Xooglers at FB about code quality vs Google.
Re:Google Blew a Ten-Year Lead by hinkley on Wednesday June 24, 2020 @12:31PM (#23630069)
One of my employers was spun out twice, after having been acquired years ago. It was sort of a joke to tell people who the owners were. It made no goddamned sense. Sure, we had customers in common, but so do Xerox and people who lease out coffee machines. The toner-flavored coffee jokes practically write themselves.

They just got onto a growth tear and bought a bunch of things they shouldn’t, and eventually rounded up all of the vaguely similar ones and divested.

I wonder sometimes if Google would be better off having a system set up for spinning off ideas that are complementary to their business model but distracting, or not high volume enough. Not every idea has to be a billion dollar idea, and a one time cash infusion to five $200M ideas would be a pretty small opportunity cost for them.

The fact that such ideas are not a direct competitor also achieves another goal of overly large employee pools, which is to choke off the supply of talent to upstarts.

Re:Google Blew a Ten-Year Lead by munificent on Wednesday June 24, 2020 @01:48PM (#23631335)
> I wonder sometimes if Google would be better off having a system set up for spinning off ideas that are complementary to their business model but distracting, or not high volume enough.

Is that not what Alphabet is?

Re:Google Blew a Ten-Year Lead by hinkley on Wednesday June 24, 2020 @02:16PM (#23631747)
That's what I was hoping Alphabet would be, but it seems like this bit is still a sticking point:

> or not high volume enough

Kind of a let-down

Re:Google Blew a Ten-Year Lead by throwawaygler on Wednesday June 24, 2020 @12:40PM (#23630284)
Re:Google Blew a Ten-Year Lead by hinkley on Wednesday June 24, 2020 @12:44PM (#23630370)
Does “sign up for early access” just mean get sneak previews, or that this is a new thing?

If the latter, then good for them but another thing they should have done ten years ago.

Re:Google Blew a Ten-Year Lead by felixgallo on Wednesday June 24, 2020 @12:18PM (#23629824)
I like the "Alexander bias" term -- is that yours, or are you quoting someone else?
Re:Google Blew a Ten-Year Lead by cpeterso on Wednesday June 24, 2020 @02:10PM (#23631665)
What does "Alexander" refer to here?
Re:Google Blew a Ten-Year Lead by robotresearcher on Wednesday June 24, 2020 @03:23PM (#23632659)
'And Alexander wept, seeing as he had no more worlds to conquer.'

Attributed to Alexander the Great.

https://www.theparisreview.org/blog/2020/03/19/and-alexander...

Re:Google Blew a Ten-Year Lead by cercatrova on Wednesday June 24, 2020 @10:08PM (#23636133)
"It is reported that King Alexander the Great, hearing Anaxarchus the philosopher discoursing and maintaining this position: That there were worlds innumerable: fell a-weeping: and when his friends and familiars about him asked what he ailed. Have I not (quoth he) good cause to weep, that being as there are an infinite number of worlds, I am not yet the lord of one?"
Re:Google Blew a Ten-Year Lead by cercatrova on Wednesday June 24, 2020 @10:09PM (#23636140)
Alex's Alexander the Great, who was great at conquering but not at administrating.
Re:Google Blew a Ten-Year Lead by necheffa on Wednesday June 24, 2020 @02:26PM (#23631893)
Probably Alexander the Great.
Re:Google Blew a Ten-Year Lead by scarface74 on Wednesday June 24, 2020 @12:35PM (#23630177)
Yet Microsoft, Amazon and Apple continuously diversify. Google still gets most of its profit from ads.
Re:Google Blew a Ten-Year Lead by jethro_tell on Wednesday June 24, 2020 @12:23PM (#23629907)
How does this apply to all the chat apps? Google chat was great, wouldn't redoing the tech and making apps for all platforms have captured more territory than changing every couple years? When you have network effect, it's just crazy to try to rebuild that network every couple years if the point is to grow.
Re:Google Blew a Ten-Year Lead by recursivecaveat on Wednesday June 24, 2020 @12:36PM (#23630186)
It's not about capturing marketshare for the company, it's about capturing bullet points for your resume. Working on new stuff is valued much more highly than fixing old stuff, and you naturally move much faster with more impressive sounding accomplishments in the early days of a product.
Re:Google Blew a Ten-Year Lead by bsmith89 on Wednesday June 24, 2020 @12:38PM (#23630245)
I think the premise is that the culture/behaviors/tactics that allowed Google to capture new territory early on are hanging on past their usefulness, not the actual capturing of territory itself. Growing existing products wasn't the important thing before, so _now_ the incentive structure at Google is mis-aligned with capturing territory in that way.
Re:Google Blew a Ten-Year Lead by scarface74 on Wednesday June 24, 2020 @12:34PM (#23630147)
I was listening to an interview from a manager at GCP on the “Screaming in the Cloud” podcast. He had written a great cost calculator that would show how much money that someone could save over AWS.

But because he couldn’t convince corporate to adopt it and when it became a feature it fell under the reliability guidelines, they canned it even though customers liked it.

Re:Google Blew a Ten-Year Lead by jariel on Wednesday June 24, 2020 @03:24PM (#23632672)
So there are at least a few things here:

1) A 'cost calculator' will be a hugely sensitive product, that is absolutely in the domain of sales, it has nothing to do with dev. So many sales issues around that - this will be the primary issue with such a product.

2) Just because some people like it doesn't mean it's worth maintaining. Maybe it is, maybe it's not. Paradoxically, because it was such a marketing/sales focused product, it's possible the Eng. driven culture didn't really understand it's value and thought it was maybe to simple of a product. Arguably, it's hardly a product, more of a marketing feature.

I suspect if sales was pushing hard for it it would have happened.

Re:Google Blew a Ten-Year Lead by scarface74 on Wednesday June 24, 2020 @05:47PM (#23634246)
If you are trying to get enterprise customers, do you really want to be known as the company that drops features at the drop of dime?

AWS has a number of services that have been deprecated for years but it still supports like SimpleDB and running an EC2 instance outside of a VPC. Heck they still support transferring files from S3 over BitTorrent in older regions.

Re:Google Blew a Ten-Year Lead by wutbrodo on Wednesday June 24, 2020 @03:31PM (#23632773)
> 1) A 'cost calculator' will be a hugely sensitive product, that is absolutely in the domain of sales, it has nothing to do with dev. So many sales issues around that - this will be the primary issue with such a product.

I can't tell what you're alluding to with "hugely sensitive", and "many sales issues around it". Could you clarify a concrete example these fuzzy expressions describe?

Re:Google Blew a Ten-Year Lead by jariel on Wednesday June 24, 2020 @04:20PM (#23633368)
Very few products and services in this world are commodities to the point wherein you just 'add up the cost'.

In fact, I can hardly think of a serious product or service quite like this. Not even cars are like this. You have maintenance, resale, insurance and financing to consider.

This idea of an easy, objective way to 'calculate costs' is never really quite possible.

There are usually going to be some degree of Apples-to-Oranges comparisons, situations wherein small issues make a big difference. You want a sales person to be there to make sure that your potential customer is aware of the options.

In addition, there is of course the 'Total Cost of Ownership' - meaning that much of the 'cost' of something is not apparent in the calculation. Training, support, etc.. On the far end of that spectrum you have the strategic issues such as vendor lock-in etc..

So, even just trying to establish a fairly objective 'true cost' calculation will be hard.

Next, you have the fact that most enterprise sales are not 'off-the-shelf' - pricing is subject to negotiation, which can vary a lot. This makes it difficult to set price.

Then, you have communications issues around a sale - these are complicated products, the last thing you ever want is for customers inputting some information, and getting the wrong numbers back, or the wrong impression. If there is any money involved, it's worth the time to talk to a sales person to smooth over issues.

Finally, is of course the sales organisations ability to pitch, sell and spin. Of course this includes the shifty areas wherein the team will want to sell a service even when they know they are not the best option.

So a 'raw cost calculator' is possibly just a big, risky bit of possible misinformation and lost opportunity to make a sale.

The 'technical' aspect of making a 'cost calculator' is completely mundane to the point of being irrelevant. In fact, Google Engineers might be too overqualified to even do such a thing.

I see a 'cost calculator' has having two completely different roles:

1) As a 'sales lead' - something new customers can input information into, which is going to almost assuredly give them a good perspective, but wherein the true and only purpose of the calculator is to generate a lead, so that a sales person can chime in.

2) A customer-centric calculator, for established customers that is tuned to their account status and discounts and let's the IT manager or business lead do projections on the service.

Re:Google Blew a Ten-Year Lead by wutbrodo on Wednesday June 24, 2020 @10:47PM (#23636367)
Thank you for the thorough clarification! I don't know much about sales, so the previous comment was pretty opaque to me.
Re:Google Blew a Ten-Year Lead by celim307 on Wednesday June 24, 2020 @01:58PM (#23631469)
I figure they thought they weren’t going to be always cheaper than AWS
Re:Google Blew a Ten-Year Lead by milesward on Wednesday June 24, 2020 @03:11PM (#23632480)
Nope, we will.
Re:Google Blew a Ten-Year Lead by milesward on Wednesday June 24, 2020 @03:10PM (#23632475)
Me!
Re:Google Blew a Ten-Year Lead by wutbrodo on Wednesday June 24, 2020 @03:29PM (#23632734)
For an anecdotal counter-example, the habits of code maintenance I picked up at Google are 10x better than anywhere else I've worked at. I've worked at small and medium-sized companies (and my current company is research-focused enough that a lot of people don't have a strong engineering background), but this is also what I hear from my Xoogler friends at (eg) Uber and Airbnb.

I'd be surprised if this tendency had anything to do with Google exporting its culture.

Re:Google Blew a Ten-Year Lead by cmrdporcupine on Wednesday June 24, 2020 @04:43PM (#23633626)
I haven't moved beyond Google yet (I want to, though, who is hiring... :-) ), but yes, to be honest, better code cleanliness, testing, style conformity, and good code review ability are the major things I've learned in the 8 years I've been there. Most other skills one picks up at Google are not transferrable outside of Google.
Re:Google Blew a Ten-Year Lead by lowiqengineer on Wednesday June 24, 2020 @11:44PM (#23636682)
My company (Amazon) is hiring fast right now. Somehow I feel like you aren’t going to consider us because you think we are inherently inferior engineers though!
Re:Google Blew a Ten-Year Lead by wutbrodo on Thursday June 25, 2020 @01:57PM (#23643533)
Personally, I have no reason to believe Amazon engineers are inferior, but I've been hearing bad things about how they're treated for almost a decade now. That isn't to say I absolutely wouldn't consider Amazon without trying to look into it more, but I definitely know people who, for that reason, don't consider Amazon when planning a move to a bigco.
Re:Google Blew a Ten-Year Lead by cmrdporcupine on Thursday June 25, 2020 @02:27PM (#23643858)
Exactly. And some of the company's recent moves re: labour practices etc. have also been questionable, or up for debate at least.

What I also hear about Amazon is that the experience varies wildly from group to group. That's not a gamble I'd be willing to take. Google is not nearly so balkanized; there are fairly consistent expectations across the company in terms of what is acceptable manager behaviour.

I am not happy at Google. But if I were to jump ship it would be to a small or medium sized company that is more agile and internally cohesive and where things move quickly, not to another BigCo with all the excessive bureaucracy and politics that goes with that.

I miss the teams I worked on where there were less than 30 or so engineers in the company.

Re:Google Blew a Ten-Year Lead by scarface74 on Thursday June 25, 2020 @06:29PM (#23646501)
Exactly. And some of the company's recent moves re: labour practices etc. have also been questionable, or up for debate at least.

The only reason that you don’t hear as much about labor practices with the other tech companies is because they outsource all of the low skill employees that make their products to China. Google has a lot fewer physical products but the people who make their few hardware products are treated worse than any Amazon employee. Yes the same applies to Apple.

I’m not making a value judgement. Just calling a spade a spade.

Re:Google Blew a Ten-Year Lead by lowiqengineer on Thursday June 25, 2020 @04:07PM (#23645004)
You’re really just implying that if the experience is worse and we are treated badly (aka no company paid off sites to Hawaii or seasons working remotely in Tahoe) the only people at Amazon are there because they aren’t capable of getting better offers.
Re:Google Blew a Ten-Year Lead by cmrdporcupine on Thursday June 25, 2020 @04:31PM (#23645328)
That I doubt very much. They're there because they're motivated by different things. I wouldn't last at Amazon, from what I have read. But some people thrive with different motivations. The most satisfied Amazon engineers I've spoken to spoke with pride about the pace and intensity of development. Goes for Apple, too. Those people would not be happy at Google. Things move slow there, and most people have little control over tech stack or many design choices at all, TBH. I hear that's better at Amazon.

The era of Hawaii offsites at Google is long over.

Re:Google Blew a Ten-Year Lead by lowiqengineer on Thursday June 25, 2020 @04:09PM (#23645040)
Of course. It’s the same code words people use when they describe neighborhood schools while avoiding race.

“Wouldn’t consider Amazon when planning a move” seems to have the same tenor as “Wouldn’t consider moving to a less white neighborhood”

Re:Google Blew a Ten-Year Lead by cmrdporcupine on Thursday June 25, 2020 @08:45AM (#23640019)
Your engineers sound very impressive. Too bad about the management.
Re:Google Blew a Ten-Year Lead by mapgrep on Wednesday June 24, 2020 @12:54PM (#23630532)
>It is well known the incentives at Google are aligned at working on new features than maintaining them and keeping users happy

I'm not so sure that's the case. It seems like historically certain products are starved of feautures and resources while other products are made a priority, and it also seems like these priorities can shift pretty rapidly.

In the Google+ era for example there was the concept of "more wood behind fewer arrows" and a ton of money and resources went into plus to the detriment of other products (some of which were outright cancelled) and now obviously plus is no longer a priority.

Hangouts was developed (iirc) for plus to attract users for a social network then eventually became Meets which seems to be attached to Google Apps as a business offering to attract paying subscribers. Which on some level makes sense but from the vantage of the actual product leads to some wasted effort and changes that seem arbitrary to users. There doesn't seem to have been a clear incentive to put features into Hangouts/Meet; instead the incentive seems to have been to grow the distinct platforms the product was attached to as a feature of those platforms. (Hence Duo as a separate product, it is attached to yet another platform, Android). Meanwhile Zoom, by staying focused on just the actual product of videoconferencing software, left Google's offering in the dust.

Are there as many people working on features at say Blogger as there were 10 years ago? I doubt it. I'm not even saying that's wrong just that it contradicts the idea that Google can be said to incentivize features in the arbitrary case. I think it's clearly only for blessed products and that set of products that are blessed will change over time.

Re:Google Blew a Ten-Year Lead by jogjayr on Wednesday June 24, 2020 @03:26PM (#23632694)
I don't understand this "leetcoders produce bad code" meme. Is there something about being good at whiteboard interviews that makes you bad at writing clean code?

Outside Google, I've seen plenty of terrible code by people who would fail FizzBuzz.

Within Google, I've seen way more attention paid to code quality and system design than at any other place I've worked at.

Re:Google Blew a Ten-Year Lead by Drew_ on Wednesday June 24, 2020 @05:32PM (#23634104)
I think it just comes from the fact that solutions that come up in leetcode/HackerRank discussions are often unreadable gibberish.
Re:Google Blew a Ten-Year Lead by hodgesrm on Wednesday June 24, 2020 @03:36PM (#23632845)
The "new features" ethos is really harmful in business apps.

Business users want focused things like accurate budget vs. actuals reports where you can flag and fix deviations quickly. There's an overall design--of course--but great enterprise software is the result of attention to very granular features related to specific work practices.

In this case the devil really is in the details.