Re:A Google Cloud support engineer solves a tough DNS case
by sitkack on Tuesday May 19, 2020 @02:58PM
(#23238698)
There needs to be some level of conformity between instances, they stop being a heard and more of a zoo if the skew is too large. The workloads running on the instances shouldn't be able to tell which instance type they are running on, or your workloads should be written such that it doesn't matter (but at some point it will). Things that grow and move together, wear together, so you will end up with a system that is designed against the empirical contract, not the stated one.
[ Parent
| on HN ]
Re:A Google Cloud support engineer solves a tough DNS case
by jeffbee on Tuesday May 19, 2020 @03:43PM
(#23239243)
My approach has always been that somewhere in my fleet there is a heat sink that fell off and a CPU running at 400MHz. My last two jobs have started with me sitting down at my desk on day 1 and demonstrating this fact. After concluding that the zoo is unavoidable, the only thing left to do is write the software accordingly.
Re:A Google Cloud support engineer solves a tough DNS case
by saagarjha on Tuesday May 19, 2020 @05:23PM
(#23240418)
> After concluding that the zoo is unavoidable, the only thing left to do is write the software accordingly.
What do you do to write software accordingly? Make it detect when it's running on a dud? Have it run as best as it can anyways?
Re:A Google Cloud support engineer solves a tough DNS case
by jeffbee on Tuesday May 19, 2020 @05:43PM
(#23240650)
Suicide is a good solution, if some higher-level thing will notice and move the task to another machine. Batch frameworks can kill slow shards, or re-assign their work to faster shards. Clients of online services can direct more traffic to working shards and less or none to slow or broken ones.