News for hackers. Stuff that matters.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by rvnx on Tuesday November 02, 2021 @03:17PM (#29086132)
I tend to agree with the green dude :|

It's normal to have a production service replicated on 2 availability regions.

The green guy is annoying, because reality is annoying, and reliability is not about luck, but is about a properly calibrated and tested process.

Yes, you need to write monitoring, you cannot run only with "hope".

Yes it sucks that a DC can go down. Your particular service is not important if it's down, but having a copy of the production data is essential in case of a catastrophe.

Except for the tests that are probably unnecessary, everything else seems to make sense.

The peer bonuses are an issue though.

Re:Google's infamous internal 2010 “I just want to serve 5TB... by novok on Tuesday November 02, 2021 @03:48PM (#29086526)
Green guy should be making a all of that a one click process to start up a service shell that does all of that for you although. Then as you write it up an automatic linting & rules engine will highlight what is missing before you make a final pull request to get the necessary human approvals, ONCE.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by midasuni on Tuesday November 02, 2021 @03:23PM (#29086227)
Depends what the problem you’re trying to solve is. In my experience the vast majority of business problems do not need that kind of reliability, and if they do they don’t need it deployed in such a Byzantian way.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by zaphar on Tuesday November 02, 2021 @03:40PM (#29086429)
You say that but then the system goes down and the CTO is walking up to your desk asking why it's down and exactly when can they expect it to be up and don't you know we are bleeding money right now?

What you call byzantine an SRE calls necessary complexity to meet the needs of your business.

Re:Google's infamous internal 2010 “I just want to serve 5TB... by midasuni on Tuesday November 02, 2021 @07:05PM (#29088722)
No it’s not. There are of course some systems like that, but refit the majority of systems for the majority of companies that’s not the case. If an invoice can’t be paid for 6 hours the world doesn’t collapse. If the CFO can’t get the statistics for his quarterly powerpoint at 3am, it’s not a major problem.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by gopher_space on Tuesday November 02, 2021 @04:00PM (#29086650)
A lot of it feels like premature optimization. Like I'm laying down a heavy infrastructure to support change but it's already locking me into certain ways of looking at problems.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by Jensson on Tuesday November 02, 2021 @06:25PM (#29088313)
It isn't premature though, a service is as robust as its weakest link, so if you let people write crappy services that easily goes down and are hard to get back when they do then you will get a huge amount of outages in major services since they depend on so many small ones.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by loxias on Tuesday November 02, 2021 @04:23PM (#29086893)
Great observation. Perhaps the term "premature infrastructure architecture optimization".
Re:Google's infamous internal 2010 “I just want to serve 5TB... by loxias on Tuesday November 02, 2021 @04:22PM (#29086881)
All of this is true, but I'd wager there's 10, at MOST, entities on the planet that are large enough to warrant this level of ... "architectural overkill". The other 99.99% of us don't need it.

I CERTAINLY don't debate that Google, or Amazon, or Facebook, or Netflix, or the phone system, or anything else that touches a noticeable percentage of the human race needs architecture like this to provide "5 9s".

But, just like when "big data" became a buzzword, and many people thought their problem needed "big data" approaches to solve, the thought that all but a small minority of entities need this is Simply Wrong.

I am reminded of a client doing something with genomics about 9 years ago. They had some over-complicated "new tool" infested approach to solve their "big data" problem, but the run times were taking too long. I was brought in as a consultant to improve it. After I was done, a data processing run that took hours (causing employees to run them overnight) before took minutes, or seconds. What did I do? I got rid of all the complexity. I replaced their expensive cluster with one studly provisioned machine. I replaced their collection of networked Java microservices with 1 non-networked multithreaded C program. I replaced their XML based format for data at rest with something I whipped up, tuned to what they actually needed.

Once their "we need big data!" >10TB data set could fit in a single machine's memory, the rest was easy. What used to "require" a cluster of machines and overnight processing could be done interactively, and quickly enough for the scientists to get into a much more productive "flow", doing dozens of runs per day.

tl;dr: unless you're google (or google scale) you don't need all this crap. :)

Re:Google's infamous internal 2010 “I just want to serve 5TB... by ridaj on Tuesday November 02, 2021 @03:23PM (#29086224)
I think the challenge is you'd expect a company like Google to have more of the setup be automated. If replication and monitoring are such universally good ideas, then why don't they come out of the box?