News for hackers. Stuff that matters.
Re:A Senior Engineer's Guide to the System Design Interview by mgraczyk on Thursday March 02, 2023 @02:58PM (#35001083)
I am on the interviewer side of ~1 system design interview per week and have done probably around 100 of them at Google and other companies.

Most of the advice here is good, although it could be condensed quite a bit.

The biggest red flag for me as an interviewer is when candidates list off concepts or technologies they don't understand. DO NOT say things like "we can't do this because of the CAP theorem" or "we should add a cache to reduce latency" unless what you are saying is true and you can explain why this is true (the guide mentions this).

The biggest green flag is when the candidate obviously has thought deeply about related problems and made design decisions based on real world constraints and requirements. If you bring up a lot of related things you have worked on, explain why you made the decisions you did in those scenarios, if its obviously related to the question, and if you don't spend too much doing it, then you will definitely get a positive recommendation from me.

The purpose of the system design interview is to convince me that you could build, or lead a team to build, the thing that we are discussing at a company like the one where you are interviewing. I feel like I can easily see through interview practice or coaching, so I don't recommend spending more than an hour or so "preparing" for system design interviews. Unless you are interviewing somewhere with very inexperienced interviewers...

Re:A Senior Engineer's Guide to the System Design Interview by leonidasv on Thursday March 02, 2023 @03:01PM (#35001150)
>I feel like I can easily see through interview practice or coaching

How do you spot it? I've had candidates with stellar performance on system design interviews, and I've never had any suspect of them (and the resumé also seemed pretty strong in those cases). But now I'm curious if I missed something.

Re:A Senior Engineer's Guide to the System Design Interview by mgraczyk on Thursday March 02, 2023 @03:15PM (#35001331)
It's possible in those cases that the candidates were really good.

The main tells are a mismatch between apparent practical experience and apparent knowledge of the design space. I always dig into a few specific technical aspects of the design as far as possible to see how deep the candidate can go.

For example, if there's a queue I'll ask what would happen if in production the queue starts to fill, and how to mitigate. Most candidates reflexively suggest making the queue bigger, some suggest adding more downstream capacity to drain faster. People with real experience will have encountered this problem and know that you have to spend time investigating to find the root cause of the backpressure, otherwise you could add capacity to random things without reducing the queue occupancy. Are your queue consumers getting throttled by some downstream service? Are your workers CPU bound? The "right" answer here is basically "we should figure out why the queue is filling up before changing anything", but very few candidates answer that way.

Another kind of mismatch is the red flag I listed, where candidates will basically list buzzwords or ideas regardless of whether or not they are related to the question I asked. For example if I ask a question where the user-facing API is very simple (ie, write opaque data to a log) then they start talking about the tradeoffs between graphql vs REST, that's a bad sign.

Re:A Senior Engineer's Guide to the System Design Interview by jefQuery on Thursday March 02, 2023 @04:53PM (#35002614)
It feels cool to me that my nontraditional degree for software (ChemE) drilled this into us and now it comes up in my current job: Rate of input - rate of output = rate of accumulation

It’d be fine to just up the queue capacity if the input is just inconsistent/spikey but averages to the input, but under steady state (or steadily increasing growth) higher input, you must increase the output rate or you will outgrow whatever capacity the system has

Re:A Senior Engineer's Guide to the System Design Interview by yunwal on Thursday March 02, 2023 @05:30PM (#35003015)
> People with real experience will have encountered this problem and know that you have to spend time investigating to find the root cause of the backpressure, otherwise you could add capacity to random things without reducing the queue occupancy. Are your queue consumers getting throttled by some downstream service? Are your workers CPU bound? The "right" answer here is basically "we should figure out why the queue is filling up before changing anything", but very few candidates answer that way.

This is the opposite of what "people with real experience" do, at least in production-down situations. The first step is to mitigate, the second step is to root cause. If there's some no-brainer step that has a chance of alleviating the issue while you root-cause, and is unlikely to make things worse, you should take it.

Re:A Senior Engineer's Guide to the System Design Interview by apsurd on Thursday March 02, 2023 @05:57PM (#35003275)
Your point is good, but in making it, you sound like an adversarial interviewer.

You and OP are actually agreeing with "the correct answer is it depends and now let's discuss context". This echoes my experience as interviewer too. It's a red flag when the candidate responds with "the correct answer". That's what OP is calling out.

I'm replying here because I get the impression you're looking for "the right answer" as you see it: "the first step is to mitigate then do root cause". You're right! But it also could be too adversarial.

Re:A Senior Engineer's Guide to the System Design Interview by bradlys on Thursday March 02, 2023 @06:06PM (#35003362)
> Your point is good, but in making it, you sound like an adversarial interviewer.

Most interviewers are adversarial.

Let's be honest - if an interviewer wants you to pass a system design interview, they'll make it work. I see this with particular candidates all the time. If we want the person to make it through - we'll let them get through. If we don't want them to get through - no amount of correct and behaviorally appropriate answers are gonna make them get through.

Re:A Senior Engineer's Guide to the System Design Interview by siggen on Friday March 03, 2023 @05:52AM (#35008014)
Pretty much this. You can have experience and also perform well in a system design interview. However, passing some of these rounds involves chance. Some interviewers are looking for specific answers. For example, some interviewers are looking for you to be knowledgeable about recent papers specific to the heavy-hitters problem. Or the two ways to best support lynchpin objects. It’s not so much as dropping well-known building blocks to construct a system but rather the interviewer asking you to design a system requiring specific characteristics you might not have any specialization in building.
Re:A Senior Engineer's Guide to the System Design Interview by mgraczyk on Thursday March 02, 2023 @06:10PM (#35003402)
This might be true of SRE roles, but not really true for most engineers, particularly not where I currently work. I wasn't completely clear but what I'd be asking for here is how to predict and design for bottlenecks, not how to mitigate active disasters.

Also making a queue bigger is a bad reflexive response to queues being full. I was very involved in the SEV review (postmortem) process at Facebook and witnessed lots of cases where bad situations were made much worse by misguided hasty responses.

Re:A Senior Engineer's Guide to the System Design Interview by Wagthesam on Thursday March 02, 2023 @06:54PM (#35003786)
do you work at google? most engineering roles ive been in engineers absolutely own their systems and take on call rotations.

I agree though, hasty response is the mark of a junior eng

Re:A Senior Engineer's Guide to the System Design Interview by mgraczyk on Thursday March 02, 2023 @07:16PM (#35003975)
I do, but I don't interview SREs. "Production down" situations are hopefully rare, so the purpose of the interview is to understand if the engineer can plan ahead by adding monitoring, logging, and knobs, and to know how to design for scale. I don't really ask about what to do in a fire because many people don't have that experience.
Re:A Senior Engineer's Guide to the System Design Interview by yunwal on Friday March 03, 2023 @12:47PM (#35012807)
> Also making a queue bigger is a bad reflexive response to queues being full.

Depends on the function of the queue. If it's a dead-letter queue, there's literally no downside besides a likely-trivial amount of cost. If the problem is upstream of the queue, and the queue is actively feeding well-functioning consumers, yeah, it could make things way worse. This is where being an experienced engineer comes in. Also having 2-person approval like you would for any code changes. Point still stands that you should take low-risk actions to mitigate if they're available to you before root-causing.

Re:A Senior Engineer's Guide to the System Design Interview by nonethewiser on Thursday March 02, 2023 @06:21PM (#35003485)
What do you expect them to say instead?

I take your word for it that they’re trying to trick you or something - it’s not my story. But it kinda comes off like you’re denigrating people for not knowing something.

Re:A Senior Engineer's Guide to the System Design Interview by mgraczyk on Thursday March 02, 2023 @06:28PM (#35003560)
If they bring up graphql, I expect them to say something like "none of the advantages of graphql are applicable in this case, so it may make sense to use if it's already well supported. All else being equal it's probably not a good fit"

The key is that the response should reflect the actual design we're discussing, it should be more specific than something that could be said in any interview.

Re:A Senior Engineer's Guide to the System Design Interview by kevinmchugh on Thursday March 02, 2023 @10:28PM (#35005436)
In an interview it is a negative to not know something. It's a bigger negative to not know something and try to hide that, or to be ignorant of the ignorance. "I don't know" is a fine answer in an interview. "I don't know, here's where I would look" is better
Re:A Senior Engineer's Guide to the System Design Interview by moandcompany on Thursday March 02, 2023 @03:11PM (#35001274)
One of the "heavy practice/coaching/leetcode grinding" signals is when the interviewee immediately dives into proposing a solution and discusses facets of that solution in great detail without bothering to ask clarifying questions or discussing assumptions and the risks/consequences of those assumptions.
Re:A Senior Engineer's Guide to the System Design Interview by ditonal on Thursday March 02, 2023 @03:37PM (#35001675)
I think the opposite is true. An experienced engineer not as versed in interview prep might hear the question to design something, and go ahead and show how they would design it based on their experience doing it in the real world.

The leetcode grinder would almost certainly ask clarifying questions since it’s heavily emphasized as part of the rubric in pretty much all system design interview prep material.

Of course, the non leetcode grinder might ask those questions, but even the social cues that you’re allowed to ask clarifying questions might not be understood.

In my experience, I’ve seen a lot of people say that systems design test the seniority of a candidate, but then stick to a rubric that the leetcode grinder has memorized and the senior engineer might miss several important points on despite having the experience because they didn’t understand all the unwritten rules of the song and dance.

Re:A Senior Engineer's Guide to the System Design Interview by apsurd on Thursday March 02, 2023 @06:03PM (#35003330)
It's funny cuz the leetcode grinder also learns that "asking clarifying questions" is part of the right answer structure too. So the cat-&-mouse game becomes more nuanced, but I do agree that it usually does become apparent that less-experienced candidates are _blindly_ following a strategy they learned somewhere. Interestingly this is not _inherently_ wrong since if you think about it, don't experienced people do the same thing? playbooks and checklists etc.

But TLDR: the hollowness of each step of the approach does become apparent in my experience.

Re:A Senior Engineer's Guide to the System Design Interview by leonidasv on Thursday March 02, 2023 @11:02PM (#35005661)
The business goal of interviews is not to select the best one, but to narrow down candidates with an acceptable fit for the role.

An interview system with a high false-negative rate is not seem as problematic if the false-positive rate is also low. The cost of not hiring a very good candidate is way lower than the cost of hiring a bad one (even considering the opportunity cost in most cases).

Re:A Senior Engineer's Guide to the System Design Interview by coolspot on Thursday March 02, 2023 @03:49PM (#35001842)
> DO NOT say things like "we should add a cache to reduce latency”

We should add a cache to reduce latency tho…

Re:A Senior Engineer's Guide to the System Design Interview by gilbetron on Thursday March 02, 2023 @04:20PM (#35002241)
If you need to reduce average latency and the cache would actually do that, sure. Cache doesn't magically make all your network latencies drop, though.
Re:A Senior Engineer's Guide to the System Design Interview by jeron on Thursday March 02, 2023 @04:03PM (#35002044)
yeah, not sure what is too hard to understand about that. Perhaps it depends on the context but I highly doubt you can go wrong by saying a cache should be added to reduce latency.
Re:A Senior Engineer's Guide to the System Design Interview by lhorie on Thursday March 02, 2023 @04:39PM (#35002455)
The way it typically fall aparts for my candidates is I ask about optimizations, they say X could be cached, then I ask about cache invalidation and lo and behold, it didn't even occur to them that the cache needs to be invalidated at some point, so they blurt something along the lines of doing the entire expensive operation again and checking it against the cache, or something similarly nonsensical.

More common than I'd like to admit.

Re:A Senior Engineer's Guide to the System Design Interview by ThalesX on Thursday March 02, 2023 @05:25PM (#35002953)
> then I ask about cache invalidation

No doubt with a follow up on how the variable would be best named.

Re:A Senior Engineer's Guide to the System Design Interview by old_hat on Thursday March 02, 2023 @05:44PM (#35003131)
That's obvious. It's called "x".
Re:A Senior Engineer's Guide to the System Design Interview by Apocryphon on Thursday March 02, 2023 @06:05PM (#35003359)
You mean x1
Re:A Senior Engineer's Guide to the System Design Interview by nonethewiser on Thursday March 02, 2023 @06:09PM (#35003385)
Isn’t it better to know you can add a cache to reduce latency, even if you’re not sure how to invalidate it?
Re:A Senior Engineer's Guide to the System Design Interview by lhorie on Thursday March 02, 2023 @07:00PM (#35003846)
The problem isn't demonstrating understanding of what a cache does conceptually, in fact any SWE worth their salt ought to be able to explain caching. The problem is if the candidate's decision to add a cache causes the system behavior to become obviously incorrect upon an ounce of further inspection, because this demonstrates a lack of foresight about local maxima vs global maxima, and systems are all about trade-offs at the macro scale.

Caching is an easy thing to blurt out in an interview setting, but not all problem spaces benefit from caching and caching often isn't the only available solution, or the most ideal one.

Re:A Senior Engineer's Guide to the System Design Interview by Misdicorl on Thursday March 02, 2023 @11:00PM (#35005635)
I started typing no, but then....

I honestly can't decide which would be worse: a developer who literally doesn't know what a cache is or one who installs caches with bad invalidation policies.

Re:A Senior Engineer's Guide to the System Design Interview by N_A_T_E on Thursday March 02, 2023 @04:16PM (#35002186)
Only two hard things in computer science, blah blah.
Re:A Senior Engineer's Guide to the System Design Interview by mgraczyk on Thursday March 02, 2023 @06:15PM (#35003444)
In practice, "latency" is too vague to be what matters. In many systems it's the worst 1% or 0.1% of latencies that matter. Adding a cache will improve latency in those cases only in some situations.

Usually adding caches helps by reducing load on the service behind the cache.

Re:A Senior Engineer's Guide to the System Design Interview by mgraczyk on Thursday March 02, 2023 @06:18PM (#35003461)
What I am saying here is that it's not true in all cases, so you should avoid blurting out rehearsed sentences like that.

Caches usually reduce load on the thing behind the cache. They can sometimes reduce latency in ways that matter, but often don't. For example if the p99 requests all miss in cache, then the cache won't help.

Re:A Senior Engineer's Guide to the System Design Interview by Kranar on Thursday March 02, 2023 @10:49PM (#35005560)
But that's not true, the total latency including a cache is:

P(cache_hit) * cache_read_latency + (1 - P(cache_hit)) * cache_miss_latency

Assuming your scenario of 99% cache misses, that means you need a cache whose read latency is less than 1% of the cost of a cache miss. There are plenty of ways to design such a cache so that you still get a net performance benefit, even in your greatly exaggerated scenario.

One very simple example of an incredibly cheap cache that is almost certainly going to cost less than 1% of the cache miss latency is a bloom filter. Bloom filters can be tuned to be incredibly space efficient as well.

https://en.wikipedia.org/wiki/Bloom_filter

Please don't be so dismissive of people who answer questions by giving generally good and standard advice, just because you have some silly corner case that you want to play "gotcha" on.

Re:A Senior Engineer's Guide to the System Design Interview by mgraczyk on Thursday March 02, 2023 @11:02PM (#35005653)
You are talking about minimizing the expected or average latency, which is very rarely important. The actual goal will depend on the broader system but typically you want to minimize the latency of the worst 95%, 99%, 99.9% etc of requests. Whether or not the cache reduces latency depends on the distribution of the cache hits. In practice, items that miss a simple LRU cache often have the longest latency in other parts of the system, in which case the cache would not reduce p99 latency.

EDIT: I recommend reading the Google SRE book or the famous "tale at scale" article.

https://www.barroso.org/publications/TheTailAtScale.pdf

Re:A Senior Engineer's Guide to the System Design Interview by Kranar on Friday March 03, 2023 @12:19AM (#35006112)
Well folks, you heard it here. Average latency is rarely important and if someone presents a very standard and common way to reduce latency of a system, you should dismiss them because they should be aware that your specific system that they have no knowledge of would result in 99% of cache misses.

I used to work at Google long ago in the platforms division, in fact I worked on the BigTable cache (among other things, mostly related to performance). It would be very sad indeed if today's SRE book dismisses caching as a vital and standard optimization strategy and instead plays all kinds of gotchas with potential candidates as I believe you're doing.

With that said, you article you linked does not in any way support your claim about caching, on the contrary it hardly discusses caching at all. It's as if you just wanted to dump a document you thought I wouldn't read as a way to be dismissive.

Re:A Senior Engineer's Guide to the System Design Interview by mgraczyk on Friday March 03, 2023 @12:27AM (#35006166)
What's your ldap? I can look you up and see what you were working on, and whether it required making design decisions around reduced average system latency.

And yeah this is exactly my point. If you present a "standard and common" solution that isn't applicable to the question I actually asked, and if you blindly apply solutions without thinking about what problem is actually being solved, then that's bad in an interview.

Average latency is usually not the thing you want to reduce because it's not representative of what any actual user is experiencing.

Re:A Senior Engineer's Guide to the System Design Interview by Kranar on Friday March 03, 2023 @12:30AM (#35006177)
Why would you do that? Now you're being weird and kind of creepy.

With that said, to the best of my knowledge it is the same as my HackerNews handle, so you are welcome to find whatever info you'd like on that, but please understand your request is very creepy and akward.

Re:A Senior Engineer's Guide to the System Design Interview by bluedevilzn on Friday March 03, 2023 @09:54AM (#35010353)
I work on Ads latency at Google. If we remove our cache, our p95 will definitely suffer. We have experiments to confirm this.

In fact, one of the key projects that is being worked on is to improve caching to reduce latency both at median and at tail.

Re:A Senior Engineer's Guide to the System Design Interview by yodsanklai on Thursday March 02, 2023 @05:45PM (#35003140)
> I feel like I can easily see through interview practice or coaching, so I don't recommend spending more than an hour or so "preparing" for system design interviews.

I'm very surprised by this recommendation which doesn't match my experience at all. First, lots of candidates don't have experience building such systems, so they need to learn by reading books and articles. Second, the candidates need to be familiar with the structure of the interview and what's expected from them, and that takes practice too.

Re:A Senior Engineer's Guide to the System Design Interview by mgraczyk on Thursday March 02, 2023 @06:13PM (#35003419)
As an interviewer, my job is to differentiate between people who have experience vs people who have just read books and articles. I interview mostly L6+, so I will not recommend people who I don't believe have real experience building complex systems at scale.

I think it only takes an hour or so to become familiar with the structure. I can tell during the interview if that's the problem and I will be very patient in explaining what I'm looking for in that sense.

Re:A Senior Engineer's Guide to the System Design Interview by Kinrany on Saturday March 04, 2023 @10:00PM (#35026564)
What is the difference between real experience and books in your opinion? How similar does the experience need to be to the problem at hand for it to be relevant?
Re:A Senior Engineer's Guide to the System Design Interview by paxys on Thursday March 02, 2023 @11:42PM (#35005905)
I can't say about other companies but I have personally done many, many of these interviews over the years and can say with 100% confidence that someone who has spent days/weeks/months studying system design interview material but not had any actual experience building such systems in a production setting will get weeded out in the first few minutes.

Now some interview prep might actually help a lot of people, but it will help you better present your existing knowledge, not replace it.

Re:A Senior Engineer's Guide to the System Design Interview by yodsanklai on Friday March 03, 2023 @05:27AM (#35007866)
I may be a counter example, but I did very well at system design interviews at both Google and Meta (level L5), with no experience building such systems. As a matter of fact, my resume didn't mention any such things and everybody was aware I didn't have that experience - recruiters, interviewers. A lot of candidates don't have that background (there are lot of things to do in our field besides building Twitter newsfeed and Google search autocomplete), and companies don't want to exclude them.

Also, you can build knowledge during your preparation. It's not only about "faking" it.

Re:A Senior Engineer's Guide to the System Design Interview by nvarsj on Thursday March 02, 2023 @06:22PM (#35003496)
No offense meant, but your comment already shows your inherent bias.

You are looking for people who have solved specific system design problems. That means you are already being overlay narrow in your selection criteria.

I have a background in quant dev and HFT for example. When I was doing system design for big tech (at L6 level), I found that the kind of system design questions asked had very little overlap with my own experience. Because the fields are different and the problems are different.

I absolutely had to prepare by spending a lot of time familiarizing myself with big data/internet scale type stuff. Like spending months studying the DDIA book. Which is a lot different from low latency systems. And there's no way I would have passed those system design interviews without doing so.

The stark reality is most of the people passing these have prepared heavily and you probably don't pick up on that - because part of the practice (e.g. mock interviews) is to make you sound natural.

Now that I'm in the system, the other stark reality is that little of this system design matters. Most of the challenge is understanding the internal technologies and figuring out who to talk to to get stuff done (like pretty much every big company).

I think ultimately system design, just like LC interviews, are a proxy for "can you study hard and pass exams". Which is supposed to be a loose proxy for what makes you successful on the job. I have mixed feelings on whether it gets the right people for the job or not.

Re:A Senior Engineer's Guide to the System Design Interview by mgraczyk on Thursday March 02, 2023 @06:32PM (#35003594)
Well for my own startups I don't do system design interviews. I don't think they're perfect and there are better ways of measuring the same thing if you have more time per candidate.

Big companies like Google have to sometimes choose mediocre processes that are Lindy and agreed on. System design interviews are less bad than coding interviews IMO, and they are much more scalable for Google.

I think I'm better at spotting genuine experience vs preparation than most interviewers because I've done a lot of interviews. On the other hand I don't think I can tell 100% of the time, and it's not black and white to begin with.

Another point, your specific background may not matter for L6 roles. I want to know if you can build the sort of thing you will be building in the role you're interviewing for. I don't care if you're good at building something else. This is less true for L4 and L5.

Re:A Senior Engineer's Guide to the System Design Interview by ZephyrBlu on Thursday March 02, 2023 @09:47PM (#35005196)
> Another point, your specific background may not matter for L6 roles. I want to know if you can build the sort of thing you will be building in the role you're interviewing for. I don't care if you're good at building something else

Why would you narrow your pool so much, especially when specific experience is usually not what defines higher engineering levels but rather architecture, leadership and communication skills?

If someone is actually skilled in system design they would be able to come into a new domain, understand the constraints and design a system within those constraints.

People seem to superficially pattern match on experience that looks similar, but doesn't really have that much overlap a lot of the time.

My thinking is that most things are far more specific than we realize, so why not mostly forget about specific domain knowledge and focus on applicable high level skills? Domain knowledge is a nice bonus, but you're almost certainly going to have to learn things on the job regardless.

Re:A Senior Engineer's Guide to the System Design Interview by moandcompany on Thursday March 02, 2023 @08:10PM (#35004497)
I'm also a former interviewer from Google. I typically interview people for L6+ roles.

What's being called "bias" here may simply be "experience" and a fundamentally different understanding of the System Design interview's purpose.

Side note: Systems Design interviews are reserved for "senior" level candidates (L5+). It is a significant inflection point for expectations, as Senior-level employees are expected to navigate through and resolve ambiguity. These interviews are not about determining if the candidate can or has solved a particular problem. When a candidate has a particular solution in mind for the presented problem, they better be prepared to explain and justify why.

(Take everything I say with a grain of salt, as there's no guarantee that an arbitrary interviewer you encounter shares the following understandings)

While Google's interviewing process for tech ladders is deliberately designed to minimize any potential interviewer bias in the process (i.e. the interviewer's role is structured to ask, as a starting point, approved questions and take notes --sometimes verbatim-- on the candidate's response for the hiring committee to make a hiring decision, the Systems Design interview is the one type of interview that does and --should-- rely on the interviewer's judgment.

The same Systems Design interview question can be given to candidates across a range of target levels.

What may be frustrating for junior candidates in particular is that unlike leetcode or cracking the coding interview questions, these questions are not intended to be or can be "completely solved." There are no specific "correct" answers. There is no book of solutions to be memorized for such questions. This is deliberate, knowing that people try to memorize answers for interviews. This does not mean a candidate can not, nor should not practice how to show their experience.

The tenets of systems design questions in engineering interviews are to:

0) Foster collaboration. The interview is about understanding how the candidate goes about solving a problem and understanding their experience solving problems (with others).

1) Give the candidate an open-ended question that is not meant to be memorizable, nor exhaustively solvable within the allotted time. This allows an interviewer/hiring committee to observe and judge a candidate's problem solving approach and experience, versus memorization.

2) Give the candidate an opportunity to show their experience -- The question and approach should be sufficiently broad to allow the candidate to surface areas where they have particular depth from their past work experience, and the interviewer to probe/explore those depths.

The interviewer's evaluation of the candidate responses and performance during a systems design interview should include the interviewer's expectations of what a candidate would at least ask or address with the presented problem. Better yet, the interviewer should include what they would expect a candidate for a given target level to address, and further specify what additional things an L+1, L+2, etc would have addressed.

Ultimately systems design interviews are not about a candidate's answers for "What" or "How" to build ______, but surfacing a candidate's judgement skills and understanding of the "Whys" along the way.

Re:A Senior Engineer's Guide to the System Design Interview by ldjkfkdsjnv on Friday March 03, 2023 @12:57AM (#35006298)
It would be great if every interview was like this. But I see system design, even at top tier companies, to be similar to leetcode. An algorithmic problem with a standard solution, where the interviewer is looking for that standard solution. There is more degrees of freedom, but not really. Interviewers are looking for memorized answers from the same few prep materials.
Re:A Senior Engineer's Guide to the System Design Interview by moandcompany on Friday March 03, 2023 @01:09AM (#35006377)
Unfortunately, that type of experience is becoming more and more common, and that speaks to the interviewers and their lack of understanding of the whys of the systems design interview process itself...

I suppose this is one more sad byproduct of the title/level-inflation or skill-dilution that has been happening across industry, as well as the 'gamification' of the interview process on both sides.

Some interviewers can also end up being lazy as well.

Re:A Senior Engineer's Guide to the System Design Interview by dilyevsky on Thursday March 02, 2023 @09:06PM (#35004930)
Like other commenter pointed out you were interviewing for what basically amounts to “tech lead” role without relevant experience. I’m not sure other L4-5s on the team would be happy if someone less experienced is hired in a TL role
Re:A Senior Engineer's Guide to the System Design Interview by sockgrant on Friday March 03, 2023 @12:11AM (#35006060)
Was on a team this happened on and a lot of the l4 l5 Eng gossiped about how little our L7 knew
Re:A Senior Engineer's Guide to the System Design Interview by emodendroket on Friday March 03, 2023 @01:00AM (#35006322)
I don't think someone gossiping about how their work superiors are fools is really notably strange to be honest.
Re:A Senior Engineer's Guide to the System Design Interview by dilyevsky on Saturday March 04, 2023 @04:55PM (#35024830)
I dunno where you worked but this is def not normal at least within technical ladder ime
Re:A Senior Engineer's Guide to the System Design Interview by nvarsj on Sunday March 05, 2023 @01:38PM (#35032545)
* * *
Re:A Senior Engineer's Guide to the System Design Interview by thrashh on Thursday March 02, 2023 @09:32PM (#35005105)
Well I mean your experience isn’t any different than applying for a job that uses Python when you’ve only used C++.

Sure, they can train you to learn Python, but they’re specifically looking for a Python developer. They’re not looking for a generic “person that can learn stuff” when there are many Python developers out there that can pick up the job day one.

Re:A Senior Engineer's Guide to the System Design Interview by ZephyrBlu on Thursday March 02, 2023 @09:49PM (#35005205)
Is that the highest order bit when you're hiring someone though? For almost all roles, I don't think it is.
Re:A Senior Engineer's Guide to the System Design Interview by thrashh on Friday March 03, 2023 @02:10AM (#35006717)
No but for a popular job listing in a well known company with hundreds of applicants, there are going to be dozens of people who have similar experience as you, plus have the exact qualification that they are asking for, putting you at an automatic disadvantage.
Re:A Senior Engineer's Guide to the System Design Interview by emodendroket on Thursday March 02, 2023 @11:41PM (#35005902)
Weird example since letting you take an interview in a different language than they use is very common
Re:A Senior Engineer's Guide to the System Design Interview by sockgrant on Friday March 03, 2023 @12:10AM (#35006055)
I don’t agree because not all interviews work that way, especially for teams that write cpp and c — they don’t let candidates interview in JS for instance.

I had a high level eng join a domain specific team I was on and he didn’t do well because he didn’t know the domain very well. He had general skills around leadership and communication that someone of his level should have, and it kept him afloat, but for a high level technical IC you need to be at worst fluent in the domain and at best a technical leader in the group.

At lower levels of IC generalist is fine, the blast radius of your technical decisions is smaller so it’s less risky to learn on the job.

But yeah, not knowing the domain you’re going to join, whether it’s a type of work or a programming language, you won’t be as effective.

Re:A Senior Engineer's Guide to the System Design Interview by emodendroket on Friday March 03, 2023 @12:58AM (#35006301)
> I don’t agree because not all interviews work that way, especially for teams that write cpp and c — they don’t let candidates interview in JS for instance.

I think this is a bit of a special case since these languages are largely restricted to specialized domains these days. I don't think many people are deciding between building a Web app with Python or C.

Re:A Senior Engineer's Guide to the System Design Interview by Hermitian909 on Thursday March 02, 2023 @09:53PM (#35005236)
> Now that I'm in the system, the other stark reality is that little of this system design matters. Most of the challenge is understanding the internal technologies and figuring out who to talk to to get stuff done (like pretty much every big company).

Having been on the "support" side of an engineering (supporting teams building product features) the thing I've noticed is that if you don't maintain a certain "density" of people with these system design skills things start to go off the rails. People start showing up with designs resting on some very fundamental misunderstandings or designed with no eye towards simplicity and things turn dysfunctional quickly.

It's worth noting that there are also better and worse designed system design interviews. Good ones give space to demonstrate "taste" as well as technical chops.

Re:A Senior Engineer's Guide to the System Design Interview by emodendroket on Thursday March 02, 2023 @11:35PM (#35005855)
Even if you do have the exact kind of experience they're looking for, the interviews usually are built around an assumption like "this system needs to be built from day one to scale to massive traffic" which is not the way projects usually are approached in the real world.
Re:A Senior Engineer's Guide to the System Design Interview by fatnoah on Friday March 03, 2023 @12:57PM (#35012924)
> Now that I'm in the system, the other stark reality is that little of this system design matters. Most of the challenge is understanding the internal technologies and figuring out who to talk to to get stuff done (like pretty much every big company).

A few years ago, I did my round of "big tech" interviews as a manager and was somewhat naive in that I did zero prep work, but I did have almost 2 decades of actually building systems. I found system design to be the "easy" and fun part of the process (including at Google). However, if I didn't have experience with scaling and redundancy in web applications, it wouldn't have been nearly as fun.

That said, once I was in the door (different big tech co), there was absolutely zero need for any of that knowledge. Now that I'm back at a startup, it's all relevant again!

Re:A Senior Engineer's Guide to the System Design Interview by roguas on Thursday March 09, 2023 @09:51AM (#35081740)
> Most of the challenge is understanding the internal technologies and figuring out who to talk to to get stuff done.

This is something interesting that often is skipped due to "sterile" nature of system design questions(solve puzzle). 90% of system design problems are tightly connected with how your org works internally. Can you depend on particular team or is it better to provide requested functionality internally. This is very often neglected but this moves or breaks stuffs.

Imagine you have to build a complete traffic system architecture for the city that will start settling in a year? It will likely be ongoing thing for you, you will observe how street layout works and adjust, build new streets, make city highway from point A to point B etc. but over time. Doing this apriori is going to go horribly bad.

Re:A Senior Engineer's Guide to the System Design Interview by Plasmoid on Thursday March 02, 2023 @07:06PM (#35003902)
A cache is such a red flag when I'm doing a system design interview.

I ask people to design a system but the system itself is very write heavy. Like 50 writes per read imbalanced. People will often suggest a cache because it's in the standard interview prep but not actually think about why they need it. They just go right to "I need a cache somewhere".

Re:A Senior Engineer's Guide to the System Design Interview by dilyevsky on Thursday March 02, 2023 @09:07PM (#35004935)
Ok if not cache then it must be hashtable?
Re:A Senior Engineer's Guide to the System Design Interview by rippercushions on Thursday March 02, 2023 @10:14PM (#35005347)
To crack any IT interview, just breadth-first search the hashtable in a dictionary and you're golden.
Re:A Senior Engineer's Guide to the System Design Interview by sophonX on Friday March 03, 2023 @07:53AM (#35008911)
lol, i don't get it.
Re:A Senior Engineer's Guide to the System Design Interview by quadrifoliate on Thursday March 02, 2023 @11:59PM (#35006008)
> I ask people to design a system but the system itself is very write heavy. Like 50 writes per read imbalanced. People will often suggest a cache because it's in the standard interview prep but not actually think about why they need it.

I mean, it's a cool thought process to go "Ooh, maybe I can store up repeated things so I don't need to repeat an expensive access process often!" It's a byproduct of modern computing hardware that we frequently have the luxury of not thinking about caching very often.

I don't think you should punish people for considering a cache. Write caching is a thing, it might just not be applicable for the system you're considering.

Re:A Senior Engineer's Guide to the System Design Interview by paxys on Friday March 03, 2023 @12:23AM (#35006147)
Well yeah, if you can say all this in the interview then you will be fine. 90% of candidates will instead read "add cache to improve performance" in their interview prep guide and repeat it verbatim.
Re:A Senior Engineer's Guide to the System Design Interview by emodendroket on Thursday March 02, 2023 @11:33PM (#35005843)
> The purpose of the system design interview is to convince me that you could build, or lead a team to build, the thing that we are discussing at a company like the one where you are interviewing. I feel like I can easily see through interview practice or coaching, so I don't recommend spending more than an hour or so "preparing" for system design interviews. Unless you are interviewing somewhere with very inexperienced interviewers...

I don't agree with this because I bombed a couple interviews till I spent the time going through the prep. I distinctly remember an interview completely going off the rails because I was asked about a retail Web site and said "honestly I'd usually start with a simple app server - SQL database setup to start; no reason for the complexity of anything more complex till you've proven you've got the traffic and have scaling problems." That's my honest opinion, borne out by real-world experience, for most people thinking of doing something like that, but it obviously wasn't what the interviewer wanted to hear and it's not like I would have been capable of giving a fancier design; I just didn't know how you're supposed to approach the question.

Re:A Senior Engineer's Guide to the System Design Interview by okdood64 on Friday March 03, 2023 @01:12AM (#35006397)
> I just didn't know how you're supposed to approach the question.

For your example, I would ask them to define what scale we were operating in, and what the footprint would be, etc.

Then you could start off with a more complex design (which seems like what the interviewer wanted.)

Re:A Senior Engineer's Guide to the System Design Interview by emodendroket on Friday March 03, 2023 @02:48AM (#35006914)
Well I know that now because I watched a bunch of prep videos and then easily passed interviews despite my experience not significantly changing in the intervening time. But that's my point; if you don't prep you have to just guess which aspects of this pretend scenario you're meant to be taking seriously and which are OK to gloss over. I had other interviews go off the rails because I offered too much detail about a smaller part of the system and discussion of it dominated the entire time but even though the interview kept engaging this discussion it wasn't actually what they wanted. "Don't bother prepping because you can't; it's just measuring your real abilities" is just not right, in my opinion.
Re:A Senior Engineer's Guide to the System Design Interview by paxys on Thursday March 02, 2023 @11:36PM (#35005860)
Yup, exactly this. The easiest way to weed out bad candidates is to listen to all the buzzwords they rattle out and ask them to explain a single one and how it is relevant to their design.
Re:A Senior Engineer's Guide to the System Design Interview by time_to_smile on Friday March 03, 2023 @09:29AM (#35010039)
> although it could be condensed quite a bit.

I bet I can condense this even more: it's yet another FAANG shibboleth that starts out as a good idea and the evolves into a ritual candidates need to perform until they can identify that they are "in".

Google Algorithm/Whiteboard interview has morphed into a generation of robots that can solve any LC problem but are incapable of building anything real in software.

Currently I think there are more books out there on "passing the systems design interview" than there are on actual systems design.