I think you would be surprised how many 100k engineers at Fortune 500 companies can't do things FAANG engineers would consider very basic. It's not that there's a 7x and a 1x engineer but that there's a 0.1x and a 1x engineer.

For example I did a stint at a F500 company where 1) a data engineer did not know how to use multiprocessing to parallelize a multi-terabyte S3 upload 2) an internal tool that fired off hundreds of network requests did not use concurrency and hence had 30+ second latency 3) a certain other team was dumping hundreds of terabytes of logs that nobody even knew the schema of into S3 automatically, and none of it was ever touched. Also seen: "data scientists" who did not know Python and executives that spent more time talking about Scrum than any concrete execution.

Exactly. Essentially, there are a lot of frauds in the industry.

To be fair, you'd also be shocked to learn how many top developers will rarely need any of those tricks. Even more shocking, to learn that FAANG developers are just as guilty of many of these "sins."

Take the uploading to s3 point. If you are saturating your network... going multithreaded will do nothing. If you are not, find out why.

The other examples are also lacking a lot of details. 30 seconds of latency can be bad, of course. But so can adding concurrency to a team that doesn't understand it well. (Really, this one sounds like it just needs more context. I'd normally push to remove extra calls moreso than jumping to concurrency. But, each has its place.)

Edit: And the logs... this is just a result of not putting cost as a tracked item for teams. Too many folks fall afoul of a hoarder mentality when it comes to logs.

FAANGs are full of terrible programmers. It's obvious. What they do have though is a higher number of good programmers, and they try to leverage them as best they can to make use of the bad ones.

I hesitate to call them terrible. Developing is a hilariously multi paradigmed skill that is tough to train. Such that a good programmer may make a terrible developer, without the appropriate supporting team.

Similarly, an otherwise great developer can be horrible in a team that doesn't know how to use them.

>The other examples are also lacking a lot of details. 30 seconds of latency can be bad, of course. But so can adding concurrency to a team that doesn't understand it well.

It might also be that it's not worth spending effort on speeding up some script like this. How often does it run?

Do these 30 seconds matter? Maybe they do now but they didn't in the beginning, but then that's ok, too, and you refactor.

I’ve worked in demanding startups where I question my own competence and lethargic enterprises where I’ve never felt more under challenged. It’s hard to find a good job that strikes a nice balance.

Fraud is a complete overreaction. A fraud might be someone posing as a doctor with no doctoral degree, or an electrician with no such training. Software is so broad and has no real, agreed upon training that it simply a gap between what an employer thinks someone can do and what they can actually do.

No, there is fraud. People who know they can’t do the role, but memorize or cheat on the interview because the lure of money is too much.

Fraud is a bit strong, but there are many people that have mastered the art of beating that particular system. I've met engineers that have become exceptionally good at pattern matching to the FAANG hazing ritual but which otherwise have unremarkable engineering skill.

Did we work together :) ?

I’m not sure if you’ve worked with a 100k engineer in the US but I’d almost always rather have one really good 300k engineer than 3 100k engineers. I mean 100k is pretty much entry level and not the cream of the crop for entry level either (which is closer to $150-250k).

When you get to 500+k you’re either talking about stock appreciation, or people in more of a leadership position/having specific technical expertise.

I’ll add that software is not necessarily a quantity thing. If you want things to have tons of features and have highly parallelizable tasks, it is. But if you want localized technical choices to be high quality you’d rather have one good person than three people who don’t know what they’re doing.

Problem I have seen many times is one engineer can only do so much work. Where in the world is $150k entry level?

> Where in the world is $150k entry level?

Finance, Management Consulting, Law, Medicine.

SFBA, apparently.

I was of similar mindset until I worked with some. Not to say a 200k dev couldn’t do it, but especially at big tech or rapidly growing large scale startups, the price difference is worth it from experience alone.

Not only are the technical challenges different at large scale, the product and human challenges are different. The 750 principle is basically leading a 500+ engineer org’s technical vision and steering the ship. Not only is it crazy difficult to get 500 people align on a vision, but the mechanics of organizing and leading that large number of people are unique too.

Plus product wise a 750k engineer can save the company multiples of that just by advocating on what not to do, and avoid expensive mistakes.

I’ve learned more in 2 years at big tech than I have in 7 in local companies or smaller startups.

This. A principle engineer probably wouldn’t be expected to write that much code in most cases. They’d be leading other senior/staff engineers. They’re spending a lot of time doing research, reviewing design docs, mentoring, and contributing their technical knowledge to high level planning meetings with directors and VPs.

I’d also agree with others that something like a “10x engineer” does exist, it just doesn’t look like the legend might have you believe. They’re not fast typing antisocial weirdos. They just understand the technical and business problems more deeply and have the breadth and depth of knowledge to come up with really good solutions. So in that regard, the value of their output is at least 10x that of their more junior peers.

> The 750 principle is basically leading a 500+ engineer org

So not that different from senior management pay grades in other industries.

I'm not sure about other companies, but in Amazon a $1M engineer tells you how to build a VM called firecracker, another $1M engineer single-handedly implemented a distributed log service and made it the backbone of multiple AWS services. Another $1M engineer convinced the company to build EC2 and SQS. In Snowflake, a $1M engineer spearheaded the Apache BEAM initiative and wrote the best book on stream processing when he was in Google. In FB, a $1M engineer lead the RxJava and Hystrix projects for years until functional reactive programing turned from a niche into something mainstream. The list can go on.

One our our engineers in a few of his sprints rewrote major pieces of Apache Tomcat, which is used in our very high TPS APIs. He fixed a bunch of the way the memory loading works, and drastically improved latency and stability of over 100 teams APIs. This guy had deep understanding of dependency injection, how java works under the hood, etc. This kind of thing might not be a huge win for a startup, but at a large corporation, you need people that can push the bounds on already well tested and well known open source software

Wouldn't it be easier to just use a higher performance application server instead of rewriting Tomcat?

I don't know the situation of the company, but very often the answer to "Wouldn't it be easier to just change platforms?" is just "no".

So, say you have 500 machines serving something. This rewrite allows you to bring it down to 100.

This code change saves you 400 instances x 24 hrs x 365 x aws cost per hour. And the next year, and the year after that, and maybe the one after that too.

Throwing more metal at the problem is the favored solution when the issue is transient, the company doesn’t scale, or the company is shitty.

I wasn't suggesting throwing more hardware at the problem. There are higher performance application servers available.

> Maybe a bigger question is: why are these big companies letting fungible labor take such a big piece of the pie?

there's a lot to dig into in this comment, but I think this is the biggest thing you are missing. the leadership of big tech companies generally does not look at revenue as a fixed-size pie to be divvied up between themselves and their subordinates. instead, they try to grow the pie as fast as possible so they can justify their own salaries and bonuses to the board. if this means paying way over market for labor, so be it. they'll keep doing it as long as they grow revenue by more than they spend on SDE salaries (modulo some organization dysfunction, of course).

the point of the $300k salaries is not to hire engineers that are 3x better; it's to hire armies of engineers very quickly. this is where the absurd algo interviews come in. a large company hiring at this pace does not have the time or the manager incentive alignment to make holistic hiring decisions. some terrible engineers that test well do slip through the cracks, but they can be PIPed out before they cost too much money.

“ the leadership of big tech companies generally does not look at revenue as a fixed-size pie to be divvied up between themselves and their subordinates. instead, they try to grow the pie as fast as possible so they can justify their own salaries and bonuses to the board.”

Right. And yet they collude to keep salaries low.

If they have more of a constrained supply, they win. Right. There aren't enough SWEs, so if you have them, your competition can't. Get enough SWEs together and you get some cool shit like K8s. That and probably keeping the good ones who are 2x in some areas, but make fewer mistakes than two 1x would.

Pay top dollar for software developers and then give K8s away to the world? Doesn’t sound like a wise use of stockholder assets.

How are your portfolio companies doing? Oh right, you don’t run a VC?

This is why others are running the ship. Your exact example, which you don’t understand, is a perfect example of why you SHOULD give k8s away.

If you're Google and you sell GCP, you give away the tool that helps more customers use more of your product.

Also it means you don't have as hard a time training new hires to think in Googly terms.

Typically, a $750k engineer designs and builds the software that makes the $100k engineers productive, or can solve Hard Problems that a $100k engineer is unlikely to solve and may not even recognize. They are paid as much as they are not just because of the impact on productivity, but also because the $100k engineers do not have the skills to produce what the $750k engineer produces. One cannot be replaced with the other. Even if you do not have an immediate use for these very expensive engineers, it can be extraordinarily valuable to have them around for when you do need them.

Having engineers that are very good at solving Hard Problems is a major asset. They have the potential to save you countless millions of dollars when Hard Problems occur. They may be doing low-value things part of the time, but if they can deliver on the Hard Problems when they occur then they are worth it.

Generally speaking, $750k class engineers, and I know many, are extremely capable and have skills far beyond the average engineer. That's why they are paid what they are paid. They have achieved a degree of mastery that few engineers achieve.

> How many 750k engineers are actually producing 7x a 100k engineer?

A huge part of the market is hilarious bad. It's insane. People graduate from school and can barely program, and even after years they don't get a whole lot better.

So yeah, absolutely there are people 7x better.

Yes, people often think of the baseline as good. This may be true in fields where people only do it for love, like programming back in the day.

The absolute dominance of computing has created such a draw that the average is in the gutter. Something as simple as pointers is beyond the skill of untold thousands.

We don’t have the gatekeepers that medicine and law do, so what to do? Pay more, make more interesting work, etc to filter out the cream, or rather, the previously decent.

This comment if I understood it correctly is basically saying that Google is probably over-paying Senior Engineers; Based on the OP's observation that "seems like very little of actual value is actually happening".

A Googler is likely best to answer here but my thesis is that since Google is just founded on the PageRank ideas of Larry Page and Sergey Brin, doesn't that already prove the value of highly skilled and highly intelligent people?

One highly intelligent person that can create a breakthrough like PageRank already creates immense value in the world so likely that is the reason that Google are willing to pay such high salaries for the best of the best.

I would maybe consider it the other way; these companies are wildly profitable with huge margins, why don't the employees capture _more_ of the value they create?

Facebook definitely needs the "fanciest" engineers. Simple problems become much more complex at scale and Facebook also solves plenty of really hard problems. Also when your whole business relies on tech the costs of bad engineering and the benefits of great engineering are both enormous. I think I'd always have an incredible engineer for 700k or however many k rather than a 5 mediocre engineers. More people rarely solve problems, better people do

Way more. Most of these <$300k engineers I’d pay to fire. They can’t identify the problem and fix it. I’ve got to hand hold them and then treat them like a robotic arm (I do all the hard work and then just use them as a glorified autocomplete).

The only engineers who produce value are a small set. Everyone else you try not to have because they’ll slow everyone down.

What amazing problems are you solving? Most of what is coming out of these companies is toxic and barely moving the needle — many times actually making software development worse with ever more bloated “pet project” platforms that get released to the dev world allowing for lesser skilled devs to churn out copycat glitchy web-based products.

HFT. Pet projects don’t happen here because we’re a prop trading shop. We can backtest, we can trial, we have immediate financial feedback. If your change doesn’t lead to improved PNL it’s not going to hold.

And, in any case, you’re not going to want it to hold. We all have failing hypotheses. The trick is to have lots of succeeding ones and for the failing ones to fail early.

Whether amazing or not isn’t the deal. The only question is whether it yields more money tomorrow. If it doesn’t, it’s not good.

isn't HFT very math heavy and the actual solutions are done by essentially galaxy brain physicists/mathematicians?

seems like a very different skill than big tech coders.

In my limited experience of HFT, the galaxy brain mathematicians write terrible code. And there isn't much time for heavy anything at HFT speeds. So there's a lot of clever software and hardware engineering that is implementing what are often quite simple strategies. Those simple strategies are backed up by lots of clever analysis of course.

Right, the analysis is the heavy part. and yes the math guys themselves don't write the code, the coders do that part.

even if the impl can be quite involved, it seems rather strange that someone could make money from very simple strategies? seems to contradict common sense.

You can make money on average with simple strategies because you react faster to signals than the competition. Your order goes in first. You just have to be very fast.

I worked at a hedge fund that did a bit of HFT. They mostly took pride though in the fact that they made money using intelligence rather than just being really fast.

HFT is all about speed, and at the minimal amounts of time to react all you can do is simple strategies.

It is not only a function of value being produced, but of supply and demand. If Google wants to hire a certain level of engineer in a certain location, they have to pay $750K, just because those engineers have other options that pay as much. Yes, at some point Google may think it’s not economically feasible to hire SWEs at that cost anymore, but given the huge profits of the tech industry it doesn’t seem like we are close to it yet.

There have been a lot of companies that tried the strategy you're suggesting; Friendster and Yahoo, for example. It turns out that "letting ... labor take such a big piece of the pie" is a much more profitable strategy.

This results from the underlying factor structure of the industry over the past quarter century. Classically, the factors of production are land, labor, and capital. Capital provides the means of production such as factories, bribed politicians, and lathes; land provides natural resources such as rainfall, sunlight, coal, and zinc ore; and labor applies one to the other to produce something which can be sold for revenue. This revenue is then divided up among the owners of land, labor, and capital.

The division doesn't reflect any kind of moral or objective valuation of the importance of the three factors; rather, it results from market processes in which the owners negotiate with one another.

Power in these negotiations results from owning a scarce resource. We're having this conversation with 100,000 other people on a web site hosted on a single server running on probably 200 watts. So the server costs maybe US$1000, the "land" costs probably US$1000/year, and it required labor that would have cost maybe US$100,000 to get it running in the first place, plus on the order of another US$100,000 a year to pay dang for moderation. The land and capital in this case are pretty fungible, and very little of them is needed. The labor is not.

A rank-and-file steelworker negotiating with the owner of a blast furnace is in a poor negotiating position because there aren't many blast furnaces around, he doesn't have one in his backyard, and there are lots of potential steelworkers with enough impulse control to avoid tripping and falling into the melt. The foreman who knows how to prevent breakouts from damaging the blast furnace and killing his coworkers is scarcer, but he's still probably limited to quitting to work for the other steel mill across town; he can't set up his own steel mill either.

By contrast, all the capital I need for my work is a US$300 laptop and a US$10/month VPS.

So the bizarre algorithm tests are not the cause of these salaries. The underlying factor structure is.

This also explains why hardware engineers are paid so much less than software engineers: an Intel chip designer can't quit and go into competition with Intel, the way the Intel founders and Chuck Peddle did, because even though you can buy foundry services on the open market, Intel's designs are festooned with patent barbed wire, and it commonly takes a million dollars and a year to get an ASIC fabbed, so you need significant startup capital. Modern patents are correctly analyzed as a way to improve the bargaining position of the owners of capital with respect to labor so that the revenues flow to investors instead of employees.

So why is it different in Europe and India? My guess is that you can't run a profitable business there without the right connections (not necessarily outright bribery), so people with the pull have the negotiating power, so they can take home the profits. And Europeans and Indians can't work in the US without visas, which are controlled by (capital-intensive) immigration lawyers and, of course, pull.

I think their reasoning is that if they have the 750k salary engineer, their competitor doesn't. But I'm just speculating.