How They SRE - Hackdot

Re:How They SRE by RcouF1uZ4gsC on Tuesday February 16, 2021 @12:28AM (#26151227)

Along the same lines, if you can avoid a distributed architecture, things get a lot more reliable. You can get a crazy amount of RAM, SSD, CPU cores on a single machine. If you run your system on a powerful machine with some other ones on hot standby, a lot of complexity goes away.

[ Parent | on HN ]

Re:How They SRE by epic9x on Tuesday February 16, 2021 @01:39AM (#26151539)

If you can run your system on a single machine, you don't need an SRE.

If you have hundreds or thousands of machines, that's an indicator that you /may/ have the complexity that requires the disciplines that can come from dedicated SRE. The tough thing is conflating filling operations problems with a role named SRE, versus actually using the best practices that will help you scale and improve reliability.

[ Parent | on HN ]

Re:How They SRE by nine_k on Tuesday February 16, 2021 @12:45AM (#26151305)

Your reservation rate goes to 200% though (a full hot standby) instead of, say, 120% to accommodate for some nodes becoming unavailable.

If your hot standby is a$100/mo VM, it's not noticeable. If it's $5000/mo, less so.

To say nothing of scaling up and down with the load — which, of course, you only need if you are a pretty large operation.

[ Parent | on HN ]

Re:How They SRE by weehoo on Tuesday February 16, 2021 @01:03AM (#26151374)

If you’re paying Bay Area prices for engineers, $5K a month is a steal to not have to pay people to deal with sharding.

[ Parent | on HN ]

Re:How They SRE by cobookman on Tuesday February 16, 2021 @01:22AM (#26151458)

Not even Bay Area prices. A Jr. SWE after overhead (benefits, HR, laptop, office-space,...etc) is easily costing the company 150k+/year in most markets.

[ Parent | on HN ]

Re:How They SRE by ocdtrekkie on Tuesday February 16, 2021 @01:52AM (#26151580)

Indeed, but take it a step further. Two ten thousand dollar servers in your basement with UPS and some rudimentary failover configuration is basically fire and forget. Remote in monthly and install updates. Done.

It'll run for ten years for next to nothing.

[ Parent | on HN ]

Re:How They SRE by Cthulhu_ on Tuesday February 16, 2021 @10:38AM (#26154850)

Until there's a power outage, flooding, malice, etc.

I think the main issue is that the cloud providers don't publish much about outages that don't affect the end-user. I mean a failed hard drive happens all the time, but S3 is never affected by that.

[ Parent | on HN ]

Re:How They SRE by nine_k on Tuesday February 16, 2021 @02:05AM (#26151628)

Depends on your bandwidth requirements. Also, if you want even higher reliability, you might consider getting two independent internet links into your basement, which is pretty doable in an urban setting.

[ Parent | on HN ]

Re:How They SRE by Mauricebranagh on Tuesday February 16, 2021 @08:18AM (#26153549)

But they wont have diverse routing :-) all it takes is a navvy with a back hoe digging in the wrong place.

And you also need to have diverse routing for power coming in and generator / battery room set up.

[ Parent | on HN ]

Re:How They SRE by snovv_crash on Tuesday February 16, 2021 @03:05AM (#26151890)

Run the backup from your friend's basement in the next town over using a different ISP. You can run the backup for them.

[ Parent | on HN ]

Re:How They SRE by ocdtrekkie on Tuesday February 16, 2021 @08:08AM (#26153488)

For a long time, my off-site backup was at my grandmother's house because it was the furthest geographic location I could give someone a box who would leave it plugged into their Internet. ;)

[ Parent | on HN ]