News for hackers. Stuff that matters.
Re:Lessons Learned from Twenty Years of Site Reliability Eng... by dilyevsky on Friday October 27, 2023 @12:17PM (#38040481)
Afair Google just ran irc on their corp network which was completely separate from prod so I wouldn’t be surprised if it was in a small server room in the office somewhere.

> very broadly applicable. I don't see any "this would only apply at Google" in here.

One thing I haven’t even heard of anyone else doing was production panic rooms - a secure room with backup vpn to prod

Re:Lessons Learned from Twenty Years of Site Reliability Eng... by DaiPlusPlus on Friday October 27, 2023 @03:11PM (#38042622)
> Afair Google just ran irc on their corp network which was completely separate from prod

I thought Google didn't have a "corp" network because of their embrace of zero-trust in BeyondCorp?

Re:Lessons Learned from Twenty Years of Site Reliability Eng... by dilyevsky on Friday October 27, 2023 @03:17PM (#38042681)
I don't think zero-trust prohibits network segmentation for redundancy or due to geographical constraints. It's mainly about how you gain access.
Re:Lessons Learned from Twenty Years of Site Reliability Eng... by Moto7451 on Friday October 27, 2023 @03:48PM (#38043061)
Correct. At an old job we did zero trust corp on a different AWS region and account. The admin site was a different zero trust zone in prod region/account and was supposed to eventually become another AWS account in another region (for cost purposes).

I can’t say if any of this was ideal but it did work unobtrusively.

Re:Lessons Learned from Twenty Years of Site Reliability Eng... by shanemhansen on Friday October 27, 2023 @06:27PM (#38044825)
They do. But I'd say most employees go their whole career without needing to do anything that requires a VPN.

It's basically all web based access through what is, at the end of the day, a http proxy.

SREs need to be ready for stuff like "hey, what if the big proxy we all use to access internal resources is down?".

Re:Lessons Learned from Twenty Years of Site Reliability Eng... by foobazgt on Saturday October 28, 2023 @04:08AM (#38048090)
Way back when, for a while, our local (Google) office's internet access ran off the same physical lines as the local prod datacenter traffic. So, any time there was a datacenter traffic outage of any kind, our office was also out. There weren't a lot of outages of that variant, but we knew immediately when one was happening. It's not particularly fun to have all of your access go out concurrently with a prod outage.
Re:Lessons Learned from Twenty Years of Site Reliability Eng... by jeffbee on Friday October 27, 2023 @05:26PM (#38044193)
When I was there, the main IRC ran in prod. But it was intentionally a low-dependency system, an actual IRC server instead of something ridiculous like gIRC or IRC-over-stubby.
Re:Lessons Learned from Twenty Years of Site Reliability Eng... by dilyevsky on Friday October 27, 2023 @06:54PM (#38045042)
i think it had a corp dns label but i'm fuzzy on that. yes it could've been a prod instance which would mean you'd need to go to panic room in case corp was down but maybe that was the intention.