DevOps / GitOps. Good lord, DevOps / GitOps. Do the things nobody wants to do:

* Build out systems from scratch

* Maintain those systems (using the most arduous-but-verifiable methods)

* Maintain security of those systems

* Scale those systems

It means dealing with the nuances that app developers don't love. The jobs that used to be relegated to the "IT Guys" in the organization running the datacenter. With infrastructure-as-code, K8s, etc. it's not unlikely (in fact arguably the best idea) to maintain the "deploy" logic alongside your app as much as the app code itself.

Most app developers don't like doing this. It means tweaking, tinkering, testing, re-testing, re-re-testing and edge cases that break main usage cases. It means checking your app works in Internet Explorer as much as it does in Chrome and that every layer of your SSL configurations are right. It means pouring through five generations of AWS documents that may or may not be valid any longer (I'm looking at you, Launch Configurations!).

But it also means ease. It means your app developers can get their code to production safely, securely and capable of scaling effortlessly. It means your business team doesn't have to "wait" because something needs to be deployed in the middle of the night because your work has built in health checks, canaries, etc. that prevent a shitty deployment.

A friend once said "DevOps means automate all-the-things" and I agree but it's just the start of that role (and the exact reason that role can get paid the way it does). A truly great DevOp will have a near-infinite level of understanding of "how" the systems work from top to bottom but ALSO they are willing to step into anyone ELSE's code in the organization to learn and help optimize it for the steps outlined above.

Cloud certs help but honestly I think most folks I've seen doing it are just people that got fed up with a broken deployment one too many times and took it upon themselves to find an alternative.

It's nice compared to many other industries because the interview process is pretty open and well-discussed and it's more rare to be blocked entirely just because you don't have the right names on your resume.

One part is pure numbers game: apply everywhere that pays well until you get some interviews. This may take a while.

The other part is practice: can you figure out the sort of questions that those companies ask? If not, can you do so if you practice enough?

Personally, I did the jump in two steps. This was ten years ago, but I've seen people do similar more recently. First I found a company with interesting work but still not a lot of prestige (after getting rejected, mostly silently, by dozens of others on LinkedIn). This gave me a lot of hands-on experience with the non-pure-algorithmic sorts of questions that I'd be asked in future interviews (like "what happens when you visit google.com" or say, around how internet networking and web service stacks work). This didn't increase my pay, though - but it was a place with a more conversational interview style plus a smaller pool of candidates, so my desire to learn came through and counted for more.

Second, I continued applying and bugging recruiters (thought not quite as frequently, since I was busier at work now!) until I got some interviews at higher paying places. Failed a couple, succeeded at one, got a 60% raise. It got easier from there since recruiters use company history as a big heuristic.

do you think your productivity or output or skill level increased 60% alongside the 60% raise or do you think your skill level increase / change in 'worth' was less?

Or in other words once you got in, did things get way more difficult / stressful?

My skill set and abilities increased the most rapidly in the job before the 60% raise. It then increased further in that job, but because it was a bigger, more established company, it took longer to get exposed to different things, and because my skill set was already larger, fewer of those things were as brand new.

Stress level was about the same as a result: on one hand, I had more to lose - my other jobs hadn't paid nearly as well. On the other hand, I found I had a broad enough base of skills to be useful immediately plus was becoming more confident in my ability to learn things.

I graduated from a very respectable but not elite university, with a respectable but not noteworthy GPA and internship. (I do feel I took to algorithms and data structures somewhat better than most of my classmates, which helped with interviewing years later.)

I worked some jobs in "low engineering" as described here, for several years. I got into this new Android thing, and published some relatively successful apps on the side, before convincing my employer to let me build their Android app, which was also reasonably successful. (I also broke into some light algorithmy stuff to improve ad serving/ad sales for that employer, though it was entirely trivial compared to what "elite" shops were doing.)

I believe that skill was the key to getting contacted by a Google sourcer - there just wasn't much of an external talent pool for it at the time. I failed one interview after insufficient prep, a delayed flight, and very short sleep. They called me again a year later and I prepped some, got in early, and went in fresh.

I imagine ML is maybe the closest thing to an analogous "door-opener skill" over the last few years, but I'm not confident about that. Research roles may really emphasize the sterling academic credentials, and for applied ML roles, Kaggles don't give experience crafting a data set and productionizing a model.

From the (admittedly small) pool of folks i've talked to. It seems like you leetcode until your eyes bleed and hope you nail the interview.

It's really not impossible. The main difference is in LC practice and system design experience. The latter of which is likely covered by any reasonably scaled startup.

The barrier to the former is practice and knowledge of fundamental algorithms. There are a lot of algorithms that come up in practice, and it can be intimidating to learn the first time.

As the parent mentioned part of the problem is that engineering productivity as measured by dollars/engineer is proportional to the profitability of the firm more so than the skills of the engineer. High end engineering firms are highly profitable and select the best engineers. There are diminishing returns to hiring more engineers so companies prefer to hire the number they need at the maximum competency they can pay for.

IIRC 2 decades ago engineering was more evenly distributed in salary expectations as there weren't as many mega cap companies, and the ones that did exist had the philosophy that they should hire more engineers rather than better engineers.

This was codified by Charles Simonyi in his dissertation on Metaprogramming, which he implemented at Microsoft.

The idea was there would be one software architect called the Metaprogrammer who would write the design and specification for the program. All the other programmers would implement this specification in as straightforward a way as possible without making any design decisions. If a design decision comes up, they refer it to the Metaprogrammer. This way you can hire large numbers of cheap low-level programmers, leveraging the skills and experience of a few experts. The idea was ultimately that the mundane code should be auto-generated.

It took Microsoft a long time to recover, if it ever really did.

We want software engineers who understand compilers and query optimizers and systems.

This requires good theoretical foundations with solid engineering skills.

It is extremely hard to get these. Most CS majors don’t know much CS - top grads from top US schools undergrads are clueless usually.

That’s why you get projects like Hadoop- a large scale systems project written in Java instead of systems languages - because the engineers are poor quality in top companies as well.

I don't know that your example is a fair one. I'm not a huge fan of java, but I have run it in anger (and specifically abused the crap out of hadoop).

I do not think that being written in C++ would have solved any of the problems we had with it. The biggest issue was the thread replication problem on writes (each write is replicated 3x, and each operation is done synchronously with a thread. Lots of parallel writes will mean tens of thousands of threads). Async was known and possible at the time, and I think even fixed upstream.

Java doesn't share machines well (at the time it never returned memory to the operating system, this is fixed now), but your hadoop machines shouldn't be shared. If you want to run a monolithic server, java will do just fine. The garbage collector will work fine unless you do dumb things (which I've also done!)

> We want software engineers who understand compilers and query optimizers and systems.

I worked in that space once. You get paid more doing web development.

Money isn't everything though. I've passed up many higher-paying cookie-cutter jobs for lower-paying but more-fulfilling and interesting problem-solving jobs.

Of course, web development can be interesting and problem-solving for some, depending on the job and the person. I just mean to say that money isn't the only criteria that some use to evaluate jobs.

True, but the top comment on this thread was about high end engineers being paid high salaries. Compilers is a good counterexample.

What is the work you are doing that requires this sort of fundamental knowledge? Just curious, as it seems like most of the business (read: monetary) value is in working much higher in the stack, leveraging the bedrock software already out there. Is there present moving and shaking happening at that level?

It's because schools teach to the market. How many schools teach COBOL? There are still companies that want to hire COBOL resources, but most schools aren't going to teaching because the market is too small. It's the same for query optimizers and compilers, only even more pronounced by their very nature - these are things that a very small number of people create because you want a standard or consistent experience with this level of tool. It's like my cryptography course - I'll never use it because the best practice is to use the industry standards.

Another reason you don't find many people with this experience is that there aren't many jobs for it. Going back to my cryptography class example, if I don't have the need to use the info after taking the class, then I'm not going to remember it.

That said, there should be plenty of candidates who are willing and capable to learn this stuff if companies would actually train people.

I have a follow up question along the same vane.

What defines a high performance engineer? If we are talking about 10x engineers, then I would argue that level of performance isn't likely to be reachable by most people and teaching plays less of a role.

I mean, I would love to be a 10x dev, but I don't think any amount of training will make that happen. I'm just a shitty mid-level and have to live with that.

I was listening to a podcast by Amjad Masad of Replit. Listen to him explain how Replit works. https://softwareengineeringdaily.com/2019/09/06/repl-it-brow...

Start at the 45:00 minute mark.

That's a 10x engineer to me.

I don't understand. It sounds like he's just explaining the architecture and process flow.

But maybe that's related to this:

"That's a 10x engineer to me."

The definition changes from person to person.

All my experience is with the job market in Seattle. A decade ago I took a job at 80k (via craigslist), against competing offers around 95k, because it involved NLP and Hadoop. A year later I got a 40% raise to a Series E startup that ultimately flamed out, working on ML-adjacent big data stuff. I slowly bumped along until, with an average job tenure of ~2 years, until in the past four years I went from ~160k to slightly over 300k. Technologies have been Hadoop/Spark/Kubernetes ecosystems. First two job hops were via LinkedIn, next two were personal referrals.

The two books I think had the most impact were Mining Massive Datasets by Ullman et al, and Data-Intensive Text Processing by Lin & Dyer, but neither is very cutting-edge now.

I've never worked at a FAANG, but I've had a ~40% interview success rate with them.

My advice would be:

- If you're early in your career or want to switch subspecialties for higher potential remuneration, be willing to take a slight pay cut or work for a slightly shady/low-prestige company for the right experience.

- Actively maintain a detailed profile on LinkedIn. It's a shitty site/company, but that's where the recruiters are. Another tip I've used to get interviews is to do a search for recruiters at a company of interest, and just visit all their profiles. They'll get the notification, and if your profile has the right keyword bait, they'll reach out. Probably more effective than cold submitting a resume through their website.

- Be really lucky. There seems to be a huge amount of variability in terms of how well interview questions match your experience/competencies, as well as being lucky in picking sub-specialties in technologies that end up booming. Also I'd say my personal productivity across all my jobs had probably a 3x variation, so be lucky in that wherever you end up working happens to match your personal strengths and weaknesses. Productivity (and promotions/experience to showcase in interviews) is not just a function of the developer, it's a function of the developer plus the working environment.

A final comment I'll make is that I have a STEM BS, but it's not in CS. Instead I gave myself a virtual CS degree by working through textbooks over 3-4 years, covering the entire course curriculum of a CS undergrad. I'm probably lucky in being temperamentally suited to completing that sort of project.

It is certainly not impossible. Every "high-end" engineer started out as a junior at some point in their career/life. And I don't think it requires working at a FAANG either.

I don't think there is a clear, unambiguous process but the gist of it is that to get there you need to continually take on larger and more complex tasks. In particular try and find your way onto greenfield projects where you can meaningfully contribute to system design (or at least observe "high-end" engineers engaging with these tasks so you can see how the process looks from "the inside"). If you can't find those opportunities in your day job, either try and find a new day job or try working on OSS projects.

> It is certainly not impossible. Every "high-end" engineer started out as a junior at some point in their career/life. And I don't think it requires working at a FAANG either.

There's a significant group of people who follow a path like: "get an internship at a company with a known brand, then get more and better-known full time interviews as a result, then get one of those well-known jobs and as a result get EVEN MORE interest from recruiters" which must be nice.

If you aren't at one of the schools those companies hire interns from, it's definitely a lot harder, but still far from impossible.

For sure, it's a lot easier for people who did undergrad at Standord/MIT/CM to get the internship (and the FTE role). But we shouldn't confuse "Stanford CS grads are overrepresented in high-end engineering roles" with "most high-end engineering roles are done by Stanford CS grads". Most people doing this kind of work went to a college you never heard of and maybe didn't even study CS.

Oh, I think the pool is a lot larger than that. I've seen pretty good industry hiring representation at a bunch of first-and-second-tier non-Ivies, includes UCSD, Washington, Waterloo, Purdue, Chicago, UTexas, UCLA, and a lot of others. (And similarly, the company base is a lot broader than just FAANG.)

I bet there's a set of 20 or so schools, though, that make up a disproportionate amount of those high-paying jobs, though certainly the long tail is very long.

> a bunch of first-and-second-tier non-Ivies, includes UCSD, Washington, Waterloo, Purdue, Chicago, UTexas, UCLA, and a lot of others.

You are putting schools of vastly different calibers in the same bucket there.

> I tend to agree here. There must be top performers in pool B willing to invest time and effort to move to pool A given how lucrative it is.

One thing to remember is that pay might be bi or multi-modal but that doesn't mean the work is actually likewise shared out by "performance". There are lots of reasons other than merit why people might end up in a lower or higher paying position. You'll also find people going up or down between these positions over their careers. A normal one from my experience is people taking big paycuts to work in Europe.

I posted something similar to you. I think the jump is not doable because it requires massive investment in a lottery ticket. Not only are FANG interviews leetcode level hard, I have heard friends at such orgs talk about the anti-loop. Even if you got to the point of solving most leetcode problems, the anti-loop can reject you. I know plenty of brilliant engineers not getting paid the value they create, because they have kids or work a tough job or volunteer or whatever, that makes it so they will never get a FANG offer.

The... anti-loop...?

See Steve Yegge's essay http://steve-yegge.blogspot.com/2008/03/get-that-job-at-goog...

If the set of people who are interviewing you by luck happen to prefer interview questions that match your strengths, you get the job; if they instead happen by luck to prefer interview questions that match your weaknesses, you don't get the job. The latter is an anti-loop.

It's a yegge-coined term for getting a set of ~five interviewers who won't pass anyone. That is for any candidate who passes interviewer X's idea of a good candidate, you won't pass Y.

I don't think you see it much anymore, but I wouldn't be surprised if it does still happen from time to time.

Yep, lots of talented leetcode level medium developers stuck in leetcode level low SWE roles.

I think there's a possibility for a SW-developer, but I wonder whether that's even possible for other branches. If you don't have the required background (sufficiently advanced university degree) then you just don't have the knowledge. I can imagine that if you really work on high-tech engineering, you need a solid background of physics, engineering, math etc. You can't just learn that on the job, or better, expect people to learn it on the job.

> Is it very hard or impossible to make a move from let's say a web developer to a high end engineering role?

Depends. What was your major? Where did you study?