Re:Log4Shell update: second Log4j vulnerability published
by plafl on Wednesday December 15, 2021 @03:22AM
(#29563309)
> _can't reboot their servers_ because they simply don't know how to bring the app back up
A hundred years later IT priests will demand human sacrifices to appease the server gods
Re:Log4Shell update: second Log4j vulnerability published
by HowTheStoryEnds on Wednesday December 15, 2021 @12:35PM
(#29569053)
So firms will need Ceremoniously Executed Offerings, eh?
Re:Log4Shell update: second Log4j vulnerability published
by staticassertion on Wednesday December 15, 2021 @03:40AM
(#29563423)
Server uptime used to be a point of pride. Now people try to shut them down as often as they can. Funny how things change.
Re:Log4Shell update: second Log4j vulnerability published
by Bluecobra on Wednesday December 15, 2021 @08:19AM
(#29565505)
Indeed, I remember going though great lengths to keep the Solaris hosts up that I used to administer. The graybeards I worked with treated rebooting as last resort, like admitting defeat. It was a dirty word, only reserved for lesser operating systems like Windows 95, that needed rebooting often. There was an art to systems administration that seems to be lost now. Or maybe I am just looking back though rose colored glasses. :)
Re:Log4Shell update: second Log4j vulnerability published
by count on Wednesday December 15, 2021 @12:43PM
(#29569156)
To be fair, it also used to take sometimes an hour+ to reinit a system.
All those SCSI busses had to be enumerated, all the RAM checked for error, ah, the old days...
Re:Log4Shell update: second Log4j vulnerability published
by viraptor on Wednesday December 15, 2021 @05:38AM
(#29564135)
We moved a lot of things from "that one server that's setup just-so by hand" to automated and repeatable setups. If we had more/ cheaper servers before, we'd likely get there earlier.
Re:Log4Shell update: second Log4j vulnerability published
by Sitio on Wednesday December 15, 2021 @04:50AM
(#29563842)
My server had 80kbyte bandwidth and doing very little in a much slower time.
Kernel patches were not on top of my head 15 years ago.
Re:Log4Shell update: second Log4j vulnerability published
by joshstrange on Wednesday December 15, 2021 @12:00PM
(#29568589)
That seems to go hand-in-hand with the "Cattle not pets" concept, which I'm a huge fan of.
Re:Log4Shell update: second Log4j vulnerability published
by NaturalPhallacy on Wednesday December 15, 2021 @01:11AM
(#29562675)
I would run screaming from such an organization.
Re:Log4Shell update: second Log4j vulnerability published
by shakna on Wednesday December 15, 2021 @02:52AM
(#29563166)
I've helped out orgs where they had a listing of servers, but couldn't tell me what purpose, if any, they served. You have multiple entire architecture rebuilds, deprecating various parts, whilst people come and go and take their knowledge with them leaving documentation behind that is outdated enough to be more harmful than helpful.
Reality is a messy thing.
Re:Log4Shell update: second Log4j vulnerability published
by kubanczyk on Wednesday December 15, 2021 @03:57AM
(#29563541)
Usually there is no universal way to find the latest version of internal documentation across teams, hence updating such documentation is barely worth the time. People notice the fact.
Re:Log4Shell update: second Log4j vulnerability published
by nefitty on Wednesday December 15, 2021 @04:52AM
(#29563853)
In your mind, do hosted wikis and KBs like Stack Exchange, Wikimedia or Confluence address the discovery problem? If not, is there anything close?
Re:Log4Shell update: second Log4j vulnerability published
by nitrogen on Wednesday December 15, 2021 @01:38PM
(#29569914)
Not the OP, but I am a fan of leaving breadcrumbs everywhere. That is, every bit of documentation should point to other bits of documentation. Code comments point to the corporate wiki and vice versa, commit messages point to issues, issues refer to commit hashes, log messages might say who wrote a service, bespoke server and container images could have a README in /, internal DNS can have text records, etc.
We have to accept that we will sometimes play digital archaeologist. That's a good thing! It means that the technology we're studying was successful. So given that, it's better to have out of date documentation than zero documentation, so that we can pick up the trail somewhere.
Re:Log4Shell update: second Log4j vulnerability published
by nisegami on Wednesday December 15, 2021 @05:54AM
(#29564245)
In cases like these, I'm a strong proponent of turning it off and seeing who yells.
Re:Log4Shell update: second Log4j vulnerability published
by shakna on Wednesday December 15, 2021 @06:25AM
(#29564491)
In an org that named their servers after planets, we turned off "Mars" (which was actually completely and utterly undocumented) and discovered it was the bridge between the intranet and the internet. However, upon turning it back on... Nothing came up again.
There were no logs to try and work out which processes needed to launch for it to do its job, causing several days of website downtime (not a simple website - dozens of interconnected services). We ended up taking the phone off the hook because we kept getting calls of "is it fixed yet?"
Re:Log4Shell update: second Log4j vulnerability published
by jatone on Wednesday December 15, 2021 @10:07AM
(#29566935)
future reference: next time just firewall the network and see what blows up rofl.
Re:Log4Shell update: second Log4j vulnerability published
by hamburglar on Wednesday December 15, 2021 @11:46AM
(#29568366)
A bit of network analysis to identify services that are being called before turning things off would probably be wise, too.
Re:Log4Shell update: second Log4j vulnerability published
by shakna on Wednesday December 15, 2021 @09:08PM
(#29574015)
The story with Mars is quite a bit more complicated than I've written, but we did scan it, and then try and isolate it, for a few weeks before we tried the restart. Let's just say there was a lot of redundancy that only broke once it couldn't ping at a very specific time.
Re:Log4Shell update: second Log4j vulnerability published
by stickfigure on Wednesday December 15, 2021 @01:18AM
(#29562702)
I run screaming towards such organizations. Higher bill rates!
Re:Log4Shell update: second Log4j vulnerability published
by hvgk on Wednesday December 15, 2021 @04:12AM
(#29563631)
This person gets it. As always if it’s a shit show, charge by the hour and get double time! If it’s easy charge by the job and spend two months getting paid to play solitaire!
Re:Log4Shell update: second Log4j vulnerability published
by iovrthoughtthis on Wednesday December 15, 2021 @03:10AM
(#29563237)
the pyramids of our time
I've been surprised (and horrified) to learn that there are companies out there that _can't reboot their servers_ because they simply don't know how to bring the app back up. For them... live patching is the only options.
Blows my mind, but that's reality (apparently)!