To be really good at those interviews you have to practice to be really good at those interviews. It takes a tremendous amount of time to get good enough that you can't possibly fail (especially for people like me, who are incapable of thinking clearly when they're asked to perform in front of others).

My brother hires for a consulting firm and he was talking about how there really is a shortage of talent out there. That's true, but in an earlier era, companies would hire a bunch of people out of college and spend resources training them. I know there are a lot of intelligent people out there who would be more than capable of performing if the entry bar wasn't set so high. Unfortunately, I guess that model doesn't work in the era of job hopping.

> My brother hires for a consulting firm and he was talking about how there really is a shortage of talent out there.

Exceptional talent is rare. There's only a shortage if you're unreasonable. Lots of employers are some combination of apathetic, unreasonable, or plain ol' lazy. It costs them nothing to fish for talent for 6-12 months, accept hundreds of applications, and then complain when a unicorn didn't come along who was willing to take at or below market rate.

Anytime somebody complains that there is a “shortage of talent” you need to immediately ask them 1. What level of talent they are looking for and 2. What level of compensation they are offering. Often the answers are 1. Top 25 percentile and 2. Bottom 25 percentile, so it’s no wonder they can’t find anyone.

I take your point, and it might not cost them anything to leave a position unfilled for a year, but I'm sure weeding through hundreds or thousands of applications costs them money. The amazing part about it is that they're OK with wasting time and money knowing they're not going to find anyone for the job.

On the contrary that sounds quite expensive

You'd be surprised how many businesses survive in spite of themselves.

I would never dare to suggest writing a new database in a professional context, even less during an interview. Probably as a joke at the coffee machine because it's ridiculous but otherwise no.

It's so many distributed databases around there, no one is better than one good PostgreSQL for the majority of uses cases.

Google's requirements differ. Also if the interview question was about distributed systems, "I'll just use Postgres" is not super constructive

Google has good in-house database software that fit their requirements. I wouldn't say that I can do better in an interview.

Yes but that in-house database software did not fall from sky. Google developers wrote it. Remember their interview process is generalised. Even for candidates not expected to implement databases, they test for the awareness of tradeoffs posed by distributed databases: read-after-write/eventual consistency, linearizability, how many masters /quorum/leader election, etc. "Just use Postgres" does not demonstrate any of that, it's not what the interviewer was after.

That's true. We don't know how was the interview, perhaps it was about implementing a distributed database from scratch, perhaps it was about storing as many personal data as you can.

I would hope the answer to "how would you write a distributed database from scratch" isn't "I'd use PostgreSQL."

Actually, I'd probably do the same.

> (where the interviewer got hung up on the fact that I said I would use postgres instead of making my own database

Huh? Google thinks it's valuable for their SREs to be hand-writing databases in place of industry standards? SREs specifically?

They were probably looking for something non-relational, not suggesting you write your own.

The exact message I got back was "Attempted to use magic software solution", so I believe they intended for me to say "some kind of relational" or "some kind of non-relational" database and maybe some key criterion about what kind of access pattern instead of me internalising the problem and pulling out a ready-made solution.

When I worked on the infrastructure parts of Google I worked on systems with millions of QPS, does PostgreSQL really handle that kind of load gracefully?

Postgres was just one part; I was describing a sharding solution that was using Postgres as long term storage underneath with a memory-based distributed message queue for ingestion and sharded cache layer for egress.

Sure, PostgreSQL scales relatively nicely on single nodes but I chose it because it has a write-ahead log, strong transaction isolation and b-tree indexing, which would have been useful given the question I had.

Did you put all the data in a single postgres? Since a million queries per second is more than postgres can handle. And if you don't put them all in one database then what is the point of postgres features like transactions or indexing? At least on the teams I saw people did these things in code, or they used a solution some other engineer at Google had written, I don't think there are any public databases which handles these things nicely without costing ridiculous amounts to run. So yes, writing a database is a part of the responsibilities a SWE at Google could be expected to handle. And this isn't some kind of special role, the people working on these things were normal L3-L5 engineers.

Edit: Note that there were SRE's working on these things as well, infrastructure teams are often mixes of SWE's and SRE's and their roles overlap somewhat, sometimes SRE's builds entire things themselves because they understand the production environment better.

No I did not put all my data into a single postgres, even if the TX/s would have scaled (they wouldn't have) the data volume would have exceeded the limits of what a single server can provide.

My solution was dependent on splitting the data into sub-categories; for the bulk of the data I was going to use idempotent sharding based on a unique key, I said I would have implemented it as a SHA1 of a userID modulus'd by 512, with 512 being the upper bound on the number of shards/machines, (or a multiple of that; at the scale I was given it would have worked).

I then went into detail about how much a single machine would need to ingest and my own experience with postgresql performance, I also spoke at length about what the maximum theoretical volume of data was for a single DC (however, that was "not important" the recruiter indicated I had a magic datacenter that did not have problems with cross-connecting many, many hundreds of GB/s in a mesh).

Frankly, I already build global solutions in my day job, sure they're not google scale, but they're built to order, quite cost effective and what's more important: they function very well and are engineered to the point where we know beyond reasonable doubt that they will perform as needed on day 1. (I work with always-online video games, the first day is the worst day, scalability wise)

> the recruiter indicated I had a magic datacenter that did not have problems with cross-connecting many, many hundreds of GB/s in a mesh

Well, then this is different than your original description, I'd need to get more details about the problem but he is right that machine to machine connections in a data center doesn't scale very well. This might not be a problem at the scales you are used to but it is a problem at Google scale. This is a very common problem that is not obvious at first when you work with data centers, I guess he just assumed that you would know this. Knowing your background you would probably adapt to it quickly on the job, but I guess they just asked the same question to every experienced SRE they got?

Edit: Another problem with your solution is that you used a static sharding strategy and didn't consider that increasing demand in the future would force you to reshard the database. Downtime might be accepted in the video game industry, and there you most likely wont even get much more demand than day 1, but using sharding strategies which lets you reshard in real time without downtime is more or less a must on the projects I worked on.

I had previously asked about load and the interviewer said that it would not grow. However that was a very real drawback in my design.

Ok, then I guess he just kept quiet about the intra datacenter bandwidth issue during the interview. Personally I'd have told you and discussed it, worst case I'd recommend down leveling stating that you don't have enough experience with systems at this scale but could be a good candidate otherwise, but I probably wouldn't have rejected you outright due to it. But hey, not every interviewer can be reasonable.

I don’t believe the interviewer wasn’t reasonable. I gathered as many requirements as I could and gathered as much info about my constraints as I could and thought up a solution on the spot. It wasn’t perfect and I was rushing through calculus that I would normally spend a lot of time validating.

The interviewer probably saw my use of technology as a cop-out and wanted to challenge it. When I told him why I chose it he probably just took it as unwillingness to change my position.

It is very possible that I poorly communicated the fact that my position was completely mutable.

EDIT: the magic datacenter was the interviewer removing that as a constraint. I did not indicate that _I_ had a magic datacenter and I kept challenging the limits of networking, which was handwaved away.

The same things, but in august 2019. I spent about a month to prepare for the on-site. The sections time-limit was about 45 minutes(not the 1hour) Failed on one coding section and on Googliness. HR coordinator said that I've slightly below the acceptable hiring level. Also, the free seminars for NALSD finished the two weeks before my interview.

Also, I heard that I have only two remaining attempts of on-site at Google

> I have only two remaining attempts of on-site at Google

LOL WAT.

This is a good enough reason for me to stay away tbh. I did not hear this from my recruiter.

It was said "unofficially"