News for hackers. Stuff that matters.
Re:Cold restart whole system after total outage by js2 on Wednesday July 19, 2023 @08:12PM (#36795346)
It's 1995 or so. I'm at U.F. getting my CS degree. Our facilities guy is giving a tour of the department's server room to some big wigs.

For some reason, he decides to demo the UPS cut-over switch. I have no idea why. But he manages to toggle it the wrong way and instead of switching the entire room full of servers to the UPS, he manages to cut power to _all of them_.

My recollection is that the cooling went out too and the room was suddenly very silent. But in retrospect that doesn't make sense.

What I do remember is that it was non-trivial to bring all our Unix servers back up because over the years they had been setup with NFS mounts in a loop such that for A to boot, it needed B to be up, which needed C to be up, which needed A to be up.

Oops.

So it took a lot of manual intervention to bring everything back up.

During that time, none of the CS professors could use their workstations since they were all dumb terminals. Same for many of the CS lab machines. Email was down for cis.ufl.edu. So was NNTP.

It took most of the day (as I recall), but eventually we got everything back up.

I say "we" because besides getting my CS degree, I was also working for the CS department as a lab assistant/sys op.

Re:Cold restart whole system after total outage by rout39574 on Thursday July 20, 2023 @02:24PM (#36804554)
I was in that room! Poor fellow. It was the demo of our new UPS; I can still remember the gradual fading fans and clicks of capacitors.

He was really nervous showing the new install off to us, and he was just talking through the positions of the switch; but he actually moved the switch to each of them as he did it.

Back in the days the conslops ran the asylum. :)

Any old school UF types looking at this, The Dog list lives on, and is again meeting from time to time. :) But at quieter venues. Our ears have unaccountably gotten old.

Re:Cold restart whole system after total outage by zamadatix on Thursday July 20, 2023 @03:54PM (#36805864)
The heck is a conslops besides the food at the prison?
Re:Cold restart whole system after total outage by djbusby on Thursday July 20, 2023 @04:58PM (#36806664)
Console Operators
Re:Cold restart whole system after total outage by rout39574 on Thursday July 20, 2023 @11:14PM (#36809591)
Consultant / operator.

We were student employees, who were nonetheless in charge of all the systems. It was an absolutely awesome experience.

Re:Cold restart whole system after total outage by js2 on Thursday July 20, 2023 @11:22PM (#36809636)
At least as valuable as my actual CS classes.
Re:Cold restart whole system after total outage by dunno7456 on Friday July 21, 2023 @09:14AM (#36813370)
s/capacitors/relay/

Sorry, couldn't hold myself :)

Re:Cold restart whole system after total outage by dunham on Thursday July 20, 2023 @03:01PM (#36805176)
We had a blackout (seemed to happen every fall when the students arrived - the university had a power station), the NFS server needed NIS to boot, and NIS server required NFS. We managed to manually get the NIS service running in single user mode, brought up everything else, and then rebooted NIS.
Re:Cold restart whole system after total outage by jen729w on Thursday July 20, 2023 @03:40PM (#36805672)
Vaguely related, in 2015 we were building a new platform based on Cisco UCS and NetApp filers.

Cisco had a virtual router. N1000 perhaps, the hardware was a C220 and it had some sort of appliance running on it. Which depended on some sort of LUN from NetApp for its storage. But you couldn’t stand the LUN up until the vCenter was up because UCS provisioned the LUN, it didn’t work if you did it manually, and that ran on vCenter, and the vCenter depended on the router so that it could reach the LUN. It was pure circular logic hell.

There was a path to make this work, but it literally took half a dozen very ~~clever~~ expensive Cisco and NetApp engineers a couple of days in a room with a whiteboard to figure it out. It was absurd.

Re:Cold restart whole system after total outage by spc476 on Thursday July 20, 2023 @05:12PM (#36806814)
I've got two such stories. Around 2000, I was working at a huge web hosting company (third shift, monitoring the network). Suddenly the power went out. Turns out, the building management (separate company) decided to run a UPS test and the memo got lost. Fun time that night.

Second story, around 2005---I, along with a friend and my father, were in Las Vegas eating lunch at one of the major casinos when the power went out completely. It was eerily silent and dark! (and then slowly, we started hearing the groans of slotzombies rising among us) I'm sure someone lost their job for a UPS cut-over failure.

Re:Cold restart whole system after total outage by DoneWithAllThat on Friday July 21, 2023 @11:35AM (#36815031)
I remember a day when a disgruntled employee shut off an entire Exodus data center by slapping the EPO switch. Took a long time and a lot of work to get everything in our cage back up.