News for hackers. Stuff that matters.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by nik736 on Sunday August 12, 2018 @11:23AM (#17744717)
So what? I would rather have Google/Amazon employees on the issue than some random DevOps dude.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by i_r7al on Sunday August 12, 2018 @04:41PM (#17746319)
The random dude is your employee and his/her mani job to get it to work and it’s first proirity. You will never be AWS/GCP/Azure first priority.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by tobylane on Tuesday August 14, 2018 @11:06AM (#17758579)
How could I be their second priority? For any one service and availability zone isn't it all or nothing (or slow).
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by zzzcpan on Sunday August 12, 2018 @04:04PM (#17746156)
> So what? I would rather have Google/Amazon employees on the issue than some random DevOps dude.

This is fine if three nines of availability is all you need. Doesn't matter much if you prefer a big brand employee fixing things or a small brand employee. It doesn't change the outcome.

However there are a lot of things that simply cannot live with crappy three nines availability. And the only way to do better is to stop relying on any single cloud, which inevitably requires infrastructure engineers aka random devops dudes.

Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by throwawaymath on Sunday August 12, 2018 @01:05PM (#17745257)
In fairness one "random DevOps dude" might be equally capable and less expensive for your infrastructure. Generally speaking any software company can succeed without a cloud provider's infrastructure, it's just a matter of cost and developing that competency in-house. There are many site reliability engineers who specialize in high availability and downtime resolution on baremetal hardware. StackExchange notably has this competency internally.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by user5994461 on Sunday August 12, 2018 @02:46PM (#17745787)
Site reliability is a new fancy name for the sucker who is on call.

They will change career after being forced to work on week ends and holidays a few times. Incidentally, today is a Sunday AND the most taken holiday of the year.

Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by throwawaymath on Sunday August 12, 2018 @03:00PM (#17745852)
Not all site reliability engineers have a bad work life balance.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by user5994461 on Sunday August 12, 2018 @03:18PM (#17745939)
The ones at AWS and Google, maybe not.

The ones who are the single "random DevOps dude" at a small company trying to emulate AWS and Google, do have zero balance.

Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by xd on Sunday August 12, 2018 @02:38PM (#17745728)
Well I guess the whole essence of HN is now lost... no room for some "random" startup dude to do anything that can be trusted.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by verletx64 on Sunday August 12, 2018 @03:04PM (#17745877)
I think you can concede that if we're talking a five nines foot race, you're unlikely to beat out AWS etc.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by zzzcpan on Sunday August 12, 2018 @03:48PM (#17746095)
Five nines means you can't rely on a single infrastructure provider for anything ever and it doesn't actually matter whether you use AWS at all.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by apple4ever on Sunday August 12, 2018 @12:50PM (#17745188)
I wouldn’t. Hire the right person and you have immediate response instead of waiting or somebody else. A large reason we are not going cloud for our new infrastructure.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by nik736 on Sunday August 12, 2018 @01:03PM (#17745249)
And this one person never sleeps and is 24/7 on-call, right?
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by geofft on Sunday August 12, 2018 @01:07PM (#17745269)
and isn't about to quit once they have a little more money saved up?
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by hluska on Sunday August 12, 2018 @01:38PM (#17745406)
Beware of the bus factor! Going in house is great, as long as everything is well documented and your company has good backup resources.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by user5994461 on Sunday August 12, 2018 @02:46PM (#17745781)
It's Sunday. Please wait for Monday office hours for your immediate response.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by zaarn on Sunday August 12, 2018 @02:51PM (#17745807)
At work we have a 24/7 20min response time clause. If the phone for work emergency calls we are ready to help in 20 minutes at any time around the clock even on Sunday.

Why would you do anything else for your sysop/sysadmin?

Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by user5994461 on Sunday August 12, 2018 @03:30PM (#17746006)
You surely realize that no human being can be available 24/7 within 20 minutes. It's beyond slavery to expect that from any employee.

You need at least 10 sysop/sysadmin to achieve anything close to that SLA, with a sustainable rota. Contrary to the parent posters who believe it can be done with THE right guy.

Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by greenleafjacob on Monday August 13, 2018 @06:58AM (#17749227)
With 3 people you can have a "follow the sun" rotation during business hours which takes care of the entire week, and I don't think you would need 7 more people for the weekend.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by zaarn on Monday August 13, 2018 @04:06AM (#17748664)
We manage it with 3 sysop/sysadmins. I think you vastly overestimate how many people you need to keep a 20 minute SLA.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by Spooky23 on Monday August 13, 2018 @09:07PM (#17754889)
That’s a shitty life.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by zaarn on Tuesday August 14, 2018 @12:09AM (#17755534)
I find it rather insulting that you think it must be a shitty job.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by Spooky23 on Tuesday August 14, 2018 @07:25AM (#17756904)
Expectation of 24/7, 20 minute response to a callout is bullshit. If that’s not shitty, you’re a victim of some sort of employee Stockholm syndrome.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by zaarn on Tuesday August 14, 2018 @07:32AM (#17756941)
Not all 3 have to respond all the time. It's a rotating schedule for having the overnight phone and the weekend phone (during weekend nights the SLA is relaxed to 1 hour).

And keep in mind, 20 minute response doesn't mean you fix the problem in 20 minutes, it means you respond in 20 minutes to the callout.

I think you're victim of an easy-going startup culture.

Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by Spooky23 on Tuesday August 14, 2018 @12:54PM (#17759593)
I misread it.

I’ve been oncall for escalations for like 15 years. That’s miserable enough, IMO frontline guys need fixed schedules and rotation if the volume is high.

I definitely cast my perspective on this and apologize if I came on too strong,

Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by user5994461 on Tuesday August 14, 2018 @01:27PM (#17759926)
Sorry to say, you're right and he has Stockholm syndrome. It's a very aggressive schedule, must be missing holidays half of the time to keep the phone.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by simonjgreen on Tuesday August 14, 2018 @02:07AM (#17755897)
Why would you hire just one person?
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by zacharycohn on Sunday August 12, 2018 @12:58PM (#17745221)
person
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by jrs95 on Sunday August 12, 2018 @11:44AM (#17744831)
This just doesn’t make sense. Google/Amazon employees basically are some random DevOps “dudes”. Whereas your own people would be...whoever you decided to hire to work on your infrastructure.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by leot on Sunday August 12, 2018 @11:49AM (#17744859)
Infrastructure (ops, software, hiring, social) matters.

Hiring some dude won't give you that.

Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by devwastaken on Sunday August 12, 2018 @12:12PM (#17744989)
Problem being that Infrastructure is made to be very, very compicated because they're selling tons of managed features that have reliance upon each other. I don't know why this idea of having your own devops is suddenly now bad. renting your own bare hardware and managing solutions yourself is still very much a thing and something you have to do if you're using lots of bandwidth. Dropbox for example made their own infrastructure to get off of AWS [1].

[1] https://techcrunch.com/2017/09/15/why-dropbox-decided-to-dro...

Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by hluska on Sunday August 12, 2018 @01:40PM (#17745414)
I don't think that anyone thinks having your own infrastructure is bad, or if they do, they likely don't have much experience. Rather, I think it can be a nightmare if it's not properly managed, and it's hard to develop the skill to properly manage the process unless you've been burned in the past.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by module0000 on Sunday August 12, 2018 @11:54AM (#17744883)
Hiring the right dude will. The trouble is, most of them are happily employed elsewhere.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by allannienhuis on Sunday August 12, 2018 @12:02PM (#17744925)
at AWS/Google? :P
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by user5994461 on Sunday August 12, 2018 @02:51PM (#17745808)
The right dude if both on week end AND on holidays right now.

Please wait until Monday 20th to get a status on the issue. Thank you.

Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by lameiam on Sunday August 12, 2018 @12:05PM (#17744939)
yeah, by google and amazon.
Re:Ask HN: Plenty of large sites down; Reddit.com, GNU.org, ... by bootloop on Sunday August 12, 2018 @11:48AM (#17744856)
It's like an insurance. You wouldn't be able to hire as many and as highly paid random DevOps on your own. So you and others pay a 3th person to do it, just in case you need a competent person.