News for hackers. Stuff that matters.
Re:Google outage – resolved by shawkinaw on Monday December 14, 2020 @09:03AM (#25417602)
Engineers shouldn’t deploy to prod directly, but sometimes it’s necessary to SSH into an instance for logs, stack dumps, etc. Source: worked for 2 big to very big tech cos.
Re:Google outage – resolved by alias_neo on Monday December 14, 2020 @09:22AM (#25417827)
For a large or v large tech co you should probably be aggregating logs to a centralised location that doesn't require access to production systems in this way. Stack dumps should also be collected safely off-system if necessary.

Perhaps my industry is a little more security conscious (I don't know which industry you're talking about), but this doesn't seem like good practice.

Re:Google outage – resolved by shawkinaw on Monday December 14, 2020 @09:54AM (#25418206)
Let me be clear, I agree it should not be normal to SSH into a prod box. Our logs are centrally aggregated. But it’s one thing to say it’s not normal, but quite another to say engineers shouldn't have access, because I totally disagree with that.
Re:Google outage – resolved by throwaway201103 on Monday December 14, 2020 @11:00AM (#25419016)
What normally (should) happens in that unusual case is that the engineer is issued a special short-lifetime credential to do what needs to be done. An audit trail is kept of when and to whom the credential was issued, for what purpose, when it was revoked, etc.
Re:Google outage – resolved by adwww on Monday December 14, 2020 @09:54AM (#25418200)
Who fixes the centralised log system when that needs debugging?

Unless prohibited in something like banking, following best practice to the letter is sometimes unacceptably slow for most industries.

Re:Google outage – resolved by mlnj on Monday December 14, 2020 @09:38AM (#25418031)
There should be tools that allow the team to gather such logs. Direct prod access is a recipe for disaster.
Re:Google outage – resolved by SteveNuts on Monday December 14, 2020 @09:22AM (#25417826)
Not having those things centralized is also a huge operational failure regardless of company size.