How do you spot it? I've had candidates with stellar performance on system design interviews, and I've never had any suspect of them (and the resumé also seemed pretty strong in those cases). But now I'm curious if I missed something.
The main tells are a mismatch between apparent practical experience and apparent knowledge of the design space. I always dig into a few specific technical aspects of the design as far as possible to see how deep the candidate can go.
For example, if there's a queue I'll ask what would happen if in production the queue starts to fill, and how to mitigate. Most candidates reflexively suggest making the queue bigger, some suggest adding more downstream capacity to drain faster. People with real experience will have encountered this problem and know that you have to spend time investigating to find the root cause of the backpressure, otherwise you could add capacity to random things without reducing the queue occupancy. Are your queue consumers getting throttled by some downstream service? Are your workers CPU bound? The "right" answer here is basically "we should figure out why the queue is filling up before changing anything", but very few candidates answer that way.
Another kind of mismatch is the red flag I listed, where candidates will basically list buzzwords or ideas regardless of whether or not they are related to the question I asked. For example if I ask a question where the user-facing API is very simple (ie, write opaque data to a log) then they start talking about the tradeoffs between graphql vs REST, that's a bad sign.
It’d be fine to just up the queue capacity if the input is just inconsistent/spikey but averages to the input, but under steady state (or steadily increasing growth) higher input, you must increase the output rate or you will outgrow whatever capacity the system has
This is the opposite of what "people with real experience" do, at least in production-down situations. The first step is to mitigate, the second step is to root cause. If there's some no-brainer step that has a chance of alleviating the issue while you root-cause, and is unlikely to make things worse, you should take it.
You and OP are actually agreeing with "the correct answer is it depends and now let's discuss context". This echoes my experience as interviewer too. It's a red flag when the candidate responds with "the correct answer". That's what OP is calling out.
I'm replying here because I get the impression you're looking for "the right answer" as you see it: "the first step is to mitigate then do root cause". You're right! But it also could be too adversarial.
Most interviewers are adversarial.
Let's be honest - if an interviewer wants you to pass a system design interview, they'll make it work. I see this with particular candidates all the time. If we want the person to make it through - we'll let them get through. If we don't want them to get through - no amount of correct and behaviorally appropriate answers are gonna make them get through.
Also making a queue bigger is a bad reflexive response to queues being full. I was very involved in the SEV review (postmortem) process at Facebook and witnessed lots of cases where bad situations were made much worse by misguided hasty responses.
I agree though, hasty response is the mark of a junior eng
Depends on the function of the queue. If it's a dead-letter queue, there's literally no downside besides a likely-trivial amount of cost. If the problem is upstream of the queue, and the queue is actively feeding well-functioning consumers, yeah, it could make things way worse. This is where being an experienced engineer comes in. Also having 2-person approval like you would for any code changes. Point still stands that you should take low-risk actions to mitigate if they're available to you before root-causing.
I take your word for it that they’re trying to trick you or something - it’s not my story. But it kinda comes off like you’re denigrating people for not knowing something.
The key is that the response should reflect the actual design we're discussing, it should be more specific than something that could be said in any interview.
The leetcode grinder would almost certainly ask clarifying questions since it’s heavily emphasized as part of the rubric in pretty much all system design interview prep material.
Of course, the non leetcode grinder might ask those questions, but even the social cues that you’re allowed to ask clarifying questions might not be understood.
In my experience, I’ve seen a lot of people say that systems design test the seniority of a candidate, but then stick to a rubric that the leetcode grinder has memorized and the senior engineer might miss several important points on despite having the experience because they didn’t understand all the unwritten rules of the song and dance.
But TLDR: the hollowness of each step of the approach does become apparent in my experience.
An interview system with a high false-negative rate is not seem as problematic if the false-positive rate is also low. The cost of not hiring a very good candidate is way lower than the cost of hiring a bad one (even considering the opportunity cost in most cases).
We should add a cache to reduce latency tho…
More common than I'd like to admit.
No doubt with a follow up on how the variable would be best named.
Caching is an easy thing to blurt out in an interview setting, but not all problem spaces benefit from caching and caching often isn't the only available solution, or the most ideal one.
I honestly can't decide which would be worse: a developer who literally doesn't know what a cache is or one who installs caches with bad invalidation policies.
Usually adding caches helps by reducing load on the service behind the cache.
Caches usually reduce load on the thing behind the cache. They can sometimes reduce latency in ways that matter, but often don't. For example if the p99 requests all miss in cache, then the cache won't help.
P(cache_hit) * cache_read_latency + (1 - P(cache_hit)) * cache_miss_latency
Assuming your scenario of 99% cache misses, that means you need a cache whose read latency is less than 1% of the cost of a cache miss. There are plenty of ways to design such a cache so that you still get a net performance benefit, even in your greatly exaggerated scenario.
One very simple example of an incredibly cheap cache that is almost certainly going to cost less than 1% of the cache miss latency is a bloom filter. Bloom filters can be tuned to be incredibly space efficient as well.
https://en.wikipedia.org/wiki/Bloom_filter
Please don't be so dismissive of people who answer questions by giving generally good and standard advice, just because you have some silly corner case that you want to play "gotcha" on.
EDIT: I recommend reading the Google SRE book or the famous "tale at scale" article.
I used to work at Google long ago in the platforms division, in fact I worked on the BigTable cache (among other things, mostly related to performance). It would be very sad indeed if today's SRE book dismisses caching as a vital and standard optimization strategy and instead plays all kinds of gotchas with potential candidates as I believe you're doing.
With that said, you article you linked does not in any way support your claim about caching, on the contrary it hardly discusses caching at all. It's as if you just wanted to dump a document you thought I wouldn't read as a way to be dismissive.
And yeah this is exactly my point. If you present a "standard and common" solution that isn't applicable to the question I actually asked, and if you blindly apply solutions without thinking about what problem is actually being solved, then that's bad in an interview.
Average latency is usually not the thing you want to reduce because it's not representative of what any actual user is experiencing.
With that said, to the best of my knowledge it is the same as my HackerNews handle, so you are welcome to find whatever info you'd like on that, but please understand your request is very creepy and akward.
In fact, one of the key projects that is being worked on is to improve caching to reduce latency both at median and at tail.
I'm very surprised by this recommendation which doesn't match my experience at all. First, lots of candidates don't have experience building such systems, so they need to learn by reading books and articles. Second, the candidates need to be familiar with the structure of the interview and what's expected from them, and that takes practice too.
I think it only takes an hour or so to become familiar with the structure. I can tell during the interview if that's the problem and I will be very patient in explaining what I'm looking for in that sense.
Now some interview prep might actually help a lot of people, but it will help you better present your existing knowledge, not replace it.
Also, you can build knowledge during your preparation. It's not only about "faking" it.
You are looking for people who have solved specific system design problems. That means you are already being overlay narrow in your selection criteria.
I have a background in quant dev and HFT for example. When I was doing system design for big tech (at L6 level), I found that the kind of system design questions asked had very little overlap with my own experience. Because the fields are different and the problems are different.
I absolutely had to prepare by spending a lot of time familiarizing myself with big data/internet scale type stuff. Like spending months studying the DDIA book. Which is a lot different from low latency systems. And there's no way I would have passed those system design interviews without doing so.
The stark reality is most of the people passing these have prepared heavily and you probably don't pick up on that - because part of the practice (e.g. mock interviews) is to make you sound natural.
Now that I'm in the system, the other stark reality is that little of this system design matters. Most of the challenge is understanding the internal technologies and figuring out who to talk to to get stuff done (like pretty much every big company).
I think ultimately system design, just like LC interviews, are a proxy for "can you study hard and pass exams". Which is supposed to be a loose proxy for what makes you successful on the job. I have mixed feelings on whether it gets the right people for the job or not.
Big companies like Google have to sometimes choose mediocre processes that are Lindy and agreed on. System design interviews are less bad than coding interviews IMO, and they are much more scalable for Google.
I think I'm better at spotting genuine experience vs preparation than most interviewers because I've done a lot of interviews. On the other hand I don't think I can tell 100% of the time, and it's not black and white to begin with.
Another point, your specific background may not matter for L6 roles. I want to know if you can build the sort of thing you will be building in the role you're interviewing for. I don't care if you're good at building something else. This is less true for L4 and L5.
Why would you narrow your pool so much, especially when specific experience is usually not what defines higher engineering levels but rather architecture, leadership and communication skills?
If someone is actually skilled in system design they would be able to come into a new domain, understand the constraints and design a system within those constraints.
People seem to superficially pattern match on experience that looks similar, but doesn't really have that much overlap a lot of the time.
My thinking is that most things are far more specific than we realize, so why not mostly forget about specific domain knowledge and focus on applicable high level skills? Domain knowledge is a nice bonus, but you're almost certainly going to have to learn things on the job regardless.
What's being called "bias" here may simply be "experience" and a fundamentally different understanding of the System Design interview's purpose.
Side note: Systems Design interviews are reserved for "senior" level candidates (L5+). It is a significant inflection point for expectations, as Senior-level employees are expected to navigate through and resolve ambiguity. These interviews are not about determining if the candidate can or has solved a particular problem. When a candidate has a particular solution in mind for the presented problem, they better be prepared to explain and justify why.
(Take everything I say with a grain of salt, as there's no guarantee that an arbitrary interviewer you encounter shares the following understandings)
While Google's interviewing process for tech ladders is deliberately designed to minimize any potential interviewer bias in the process (i.e. the interviewer's role is structured to ask, as a starting point, approved questions and take notes --sometimes verbatim-- on the candidate's response for the hiring committee to make a hiring decision, the Systems Design interview is the one type of interview that does and --should-- rely on the interviewer's judgment.
The same Systems Design interview question can be given to candidates across a range of target levels.
What may be frustrating for junior candidates in particular is that unlike leetcode or cracking the coding interview questions, these questions are not intended to be or can be "completely solved." There are no specific "correct" answers. There is no book of solutions to be memorized for such questions. This is deliberate, knowing that people try to memorize answers for interviews. This does not mean a candidate can not, nor should not practice how to show their experience.
The tenets of systems design questions in engineering interviews are to:
0) Foster collaboration. The interview is about understanding how the candidate goes about solving a problem and understanding their experience solving problems (with others).
1) Give the candidate an open-ended question that is not meant to be memorizable, nor exhaustively solvable within the allotted time. This allows an interviewer/hiring committee to observe and judge a candidate's problem solving approach and experience, versus memorization.
2) Give the candidate an opportunity to show their experience -- The question and approach should be sufficiently broad to allow the candidate to surface areas where they have particular depth from their past work experience, and the interviewer to probe/explore those depths.
The interviewer's evaluation of the candidate responses and performance during a systems design interview should include the interviewer's expectations of what a candidate would at least ask or address with the presented problem. Better yet, the interviewer should include what they would expect a candidate for a given target level to address, and further specify what additional things an L+1, L+2, etc would have addressed.
Ultimately systems design interviews are not about a candidate's answers for "What" or "How" to build ______, but surfacing a candidate's judgement skills and understanding of the "Whys" along the way.
I suppose this is one more sad byproduct of the title/level-inflation or skill-dilution that has been happening across industry, as well as the 'gamification' of the interview process on both sides.
Some interviewers can also end up being lazy as well.
Sure, they can train you to learn Python, but they’re specifically looking for a Python developer. They’re not looking for a generic “person that can learn stuff” when there are many Python developers out there that can pick up the job day one.
I had a high level eng join a domain specific team I was on and he didn’t do well because he didn’t know the domain very well. He had general skills around leadership and communication that someone of his level should have, and it kept him afloat, but for a high level technical IC you need to be at worst fluent in the domain and at best a technical leader in the group.
At lower levels of IC generalist is fine, the blast radius of your technical decisions is smaller so it’s less risky to learn on the job.
But yeah, not knowing the domain you’re going to join, whether it’s a type of work or a programming language, you won’t be as effective.
I think this is a bit of a special case since these languages are largely restricted to specialized domains these days. I don't think many people are deciding between building a Web app with Python or C.
Having been on the "support" side of an engineering (supporting teams building product features) the thing I've noticed is that if you don't maintain a certain "density" of people with these system design skills things start to go off the rails. People start showing up with designs resting on some very fundamental misunderstandings or designed with no eye towards simplicity and things turn dysfunctional quickly.
It's worth noting that there are also better and worse designed system design interviews. Good ones give space to demonstrate "taste" as well as technical chops.
A few years ago, I did my round of "big tech" interviews as a manager and was somewhat naive in that I did zero prep work, but I did have almost 2 decades of actually building systems. I found system design to be the "easy" and fun part of the process (including at Google). However, if I didn't have experience with scaling and redundancy in web applications, it wouldn't have been nearly as fun.
That said, once I was in the door (different big tech co), there was absolutely zero need for any of that knowledge. Now that I'm back at a startup, it's all relevant again!
This is something interesting that often is skipped due to "sterile" nature of system design questions(solve puzzle). 90% of system design problems are tightly connected with how your org works internally. Can you depend on particular team or is it better to provide requested functionality internally. This is very often neglected but this moves or breaks stuffs.
Imagine you have to build a complete traffic system architecture for the city that will start settling in a year? It will likely be ongoing thing for you, you will observe how street layout works and adjust, build new streets, make city highway from point A to point B etc. but over time. Doing this apriori is going to go horribly bad.
I ask people to design a system but the system itself is very write heavy. Like 50 writes per read imbalanced. People will often suggest a cache because it's in the standard interview prep but not actually think about why they need it. They just go right to "I need a cache somewhere".
I mean, it's a cool thought process to go "Ooh, maybe I can store up repeated things so I don't need to repeat an expensive access process often!" It's a byproduct of modern computing hardware that we frequently have the luxury of not thinking about caching very often.
I don't think you should punish people for considering a cache. Write caching is a thing, it might just not be applicable for the system you're considering.
I don't agree with this because I bombed a couple interviews till I spent the time going through the prep. I distinctly remember an interview completely going off the rails because I was asked about a retail Web site and said "honestly I'd usually start with a simple app server - SQL database setup to start; no reason for the complexity of anything more complex till you've proven you've got the traffic and have scaling problems." That's my honest opinion, borne out by real-world experience, for most people thinking of doing something like that, but it obviously wasn't what the interviewer wanted to hear and it's not like I would have been capable of giving a fancier design; I just didn't know how you're supposed to approach the question.
For your example, I would ask them to define what scale we were operating in, and what the footprint would be, etc.
Then you could start off with a more complex design (which seems like what the interviewer wanted.)
I bet I can condense this even more: it's yet another FAANG shibboleth that starts out as a good idea and the evolves into a ritual candidates need to perform until they can identify that they are "in".
Google Algorithm/Whiteboard interview has morphed into a generation of robots that can solve any LC problem but are incapable of building anything real in software.
Currently I think there are more books out there on "passing the systems design interview" than there are on actual systems design.
Most of the advice here is good, although it could be condensed quite a bit.
The biggest red flag for me as an interviewer is when candidates list off concepts or technologies they don't understand. DO NOT say things like "we can't do this because of the CAP theorem" or "we should add a cache to reduce latency" unless what you are saying is true and you can explain why this is true (the guide mentions this).
The biggest green flag is when the candidate obviously has thought deeply about related problems and made design decisions based on real world constraints and requirements. If you bring up a lot of related things you have worked on, explain why you made the decisions you did in those scenarios, if its obviously related to the question, and if you don't spend too much doing it, then you will definitely get a positive recommendation from me.
The purpose of the system design interview is to convince me that you could build, or lead a team to build, the thing that we are discussing at a company like the one where you are interviewing. I feel like I can easily see through interview practice or coaching, so I don't recommend spending more than an hour or so "preparing" for system design interviews. Unless you are interviewing somewhere with very inexperienced interviewers...