News for hackers. Stuff that matters.
Re:That UPS you bought for your home server may not be as us... by mprovost on Monday August 10, 2020 @12:56PM (#24111367)
This reminded me of James Hamilton's (the AWS datacentre guru) writeup of the power outage at the 2013 SuperBowl. Specifically how the default configuration of backup generators is often to protect themselves, even if that means shutting down the things they are meant to keep running. In my own experience running a reasonably sized datacentre I was often surprised at how UPSs and generators reacted to adverse conditions. One thing with power that you don't want is to be surprised. But it's difficult to test this equipment so it's mostly a learn as you go experience.

https://perspectives.mvdirona.com/2013/02/the-power-failure-...

Re:That UPS you bought for your home server may not be as us... by jcrawfordor on Monday August 10, 2020 @01:09PM (#24111543)
Some military equipment, especially naval, has something called a "battleshort" where electrical protections can be disabled during combat, where losing equipment could be more dangerous than electrical hazards. It's interesting to think about whether this could be applied to DC environments, but honestly, I think few DCs actually have this kind of risk aversion for downtime. Alternatives like precautionary replacement of generators that have been oversped probably come at too high of a cost, not to mention that many of those false-positive-prone protective devices are there to reduce risk of fire.
Re:That UPS you bought for your home server may not be as us... by btown on Monday August 10, 2020 @06:18PM (#24114720)
https://en.wikipedia.org/wiki/Battleshort

> According to Allied Ordnance Publication AOP-38-3,[1] a NATO publication, a battleshort is "The capability to bypass certain safety features in a system to ensure completion of the mission without interruption due to the safety feature." It also says, "Examples of bypassed safety features are circuit overload protection, and protection against overheating".

> For example, the electrical drives to elevate and traverse the guns of a combat warship may have "battleshort" fuses, which are simply copper bars of the correct size to fit the fuse holders, as failure to return fire in a combat situation is a greater threat to the ship and crew than damaging or overheating the electrical motors.

> Battleshorts have been used in some non-combat situations as well, including the Firing Room/Mission Control spaces at NASA during the manned Apollo missions — specifically the Moon landings.

As a side note, with great apology to those who may have lost their lives due to the consequences of real-world battleshorts - this is actually a really interesting analogy for the "war room" scenarios we find ourselves in at startups. Risking damage to morale and productivity can be acceptable if the alternative is irrecoverable loss of the startup's reputation. But this also can't be a sustained state of affairs - these types of procedures are meant for a battle, and risk compounds if you don't return to a steady state. Perhaps adding "battleshort" to our lexicon would make it clear that "doing things that don't scale" is often necessary but not without its costs.

Re:That UPS you bought for your home server may not be as us... by vosper on Monday August 10, 2020 @10:33PM (#24116638)
> Perhaps adding "battleshort" to our lexicon would make it clear that "doing things that don't scale" is often necessary but not without its costs.

... And 6 months later marketing people will have commandeered it, without regard to the original meaning, and they’ll be asking people to battleshort so they can really be agile and lean in on providing air cover via the updated sales deck.

Re:That UPS you bought for your home server may not be as us... by blaser-waffle on Tuesday August 11, 2020 @09:19AM (#24119982)
That sentence triggered me like 3 times.

Doubly so because I used to be in the US military but now live in a different country, and it's crazy to see how those terms and acronyms infect a lot of US business culture.

A decade ago I would have been fine, even glad, that I could wrap myself in verbiage that would jive with poor career choices I made at 18. Now it just feels incongruous or jingoistic.

Re:That UPS you bought for your home server may not be as us... by btown on Tuesday August 11, 2020 @02:28PM (#24123916)
It's the good old "military-industrial complex" - except with the psychoanalytical sense of the word "complex."
Re:That UPS you bought for your home server may not be as us... by btown on Tuesday August 11, 2020 @08:29AM (#24119593)
“It’s okay team we’ll just push hard and discuss how to avoid this in the retrospective”
Re:That UPS you bought for your home server may not be as us... by xmprt on Tuesday August 11, 2020 @12:39AM (#24117211)
This made me sad. I was so excited to find this new term that I was looking forward to using.
Re:That UPS you bought for your home server may not be as us... by vosper on Tuesday August 11, 2020 @02:46AM (#24117776)
Ahh, please don’t let my cynicism stop you!
Re:That UPS you bought for your home server may not be as us... by p_l on Tuesday August 11, 2020 @03:31AM (#24118038)
Too late man, too late.

We're now sharpening pitchforks to use on the next salestrooper that comes close to the SalesBell™

/s

Re:That UPS you bought for your home server may not be as us... by helldritch on Monday August 10, 2020 @06:31PM (#24114855)
This is one of those comments I so rarely see where what I already know is put in to words I've never been able to find.

Thank you for this, I'll be updating our documentation to include these concepts first thing tomorrow.

Re:That UPS you bought for your home server may not be as us... by nine_k on Monday August 10, 2020 @05:43PM (#24114348)
Most of military equipment is built to be able to operate outside the safety bounds, because the enemy in a battle is actively trying to push it outside these safety bounds and outright destroy it. Burning down your afterburner chamber and winning a fight is better that preserving it and losing the whole aircraft.

In a datacenter though your bets are usually much lower. Nobody is going to die, or lose many millions, if some of your equipment shuts down to prevent it from being damaged, catching a fire, etc.

There are though high-stake situations where you want exactly that: spend the entire amount of the resources of certain hardware to prevent a loss of life, or of untold millions, when you are powering a surgery chamber, or a large stock trading operation.

People who realize that do over-provision and pay top dollar for that when they can afford that. I remember that when a major fire occurred in one of the skyscrapers in the financial district of NYC, traders of a particular financial company were evacuated with their laptops into helicopters on the roof, and ferried to a spare office across Hudson river, in NJ. To minimize the impact of that, they connected to the corporate network via their phones as hotspots, and kept trading while airborne.

Of course one cannot hope to pull such an operation off without extensive preparation and likely regular drills.

Not preparing to a black swan event during a high-stakes event, like translation of a Superbowl match, sounds like either not having enough paranoia which is professionally required, or, more likely, as a cost-cutting after a wrong assessment of risks.

Re:That UPS you bought for your home server may not be as us... by brianwawok on Monday August 10, 2020 @09:18PM (#24116247)
That sounds a lot harder than having half of each team in two separate buildings. If one needs to shut down the secondary team just takes the full load...
Re:That UPS you bought for your home server may not be as us... by nine_k on Tuesday August 11, 2020 @01:58AM (#24117546)
(1) I don't think the other building was an office they kept full-time; it could even have been a hotel.

(2) Think about hiring twice as much senior programmes, just in case. Good day traders are even more expensive.

Re:That UPS you bought for your home server may not be as us... by blaser-waffle on Tuesday August 11, 2020 @09:25AM (#24120028)
> (2) Think about hiring twice as much senior programmes, just in case. Good day traders are even more expensive.

Day traders are expensive? I thought a lot of those folks keep what they kill, and when they screw up they don't get to eat. "Coffee for closers," etc.

Re:That UPS you bought for your home server may not be as us... by nine_k on Tuesday August 11, 2020 @12:19PM (#24122088)
If they trade for themselves, sure. (It's like school kids who can build a funny enough game in Scratch.)

If they trade for a large corporation, with the leverage provided by that corporation, it's a whole another ballgame.

Re:That UPS you bought for your home server may not be as us... by imgabe on Monday August 10, 2020 @07:55PM (#24115666)
Electrical codes in the US are very clear that they care much, much more about preventing fires than preventing loss of power. The one exception is the fire pump that provides water pressure to the sprinkler system, since if that's running there's already a fire.

The owner of the datacenter's aversion to downtime is not going to be a concern to the building inspector or the fire marshal.

Re:That UPS you bought for your home server may not be as us... by zaarn on Tuesday August 11, 2020 @09:14AM (#24119937)
Fire supression systems basically run their own electrical code.

At one of my previous workplaces I learned a bit about our local firecode in relation to electrical code. Mostly that firecode supersedes electrical code. The fire alarm is running of a 230V line (standard) without grounding (non-standard) via a special line from the upstream power transformer that has higher amperage tolerance.

All the fire detectors are powered and wired from this at 40-70V, no ground or if there is a ground, it's grounded at the control panel of the fire alarm system.

In the event of a fire, the entire system is rated about 10X over it's standard ratings. The wires that are allowed to handle 1 Amp before are now allowed to go to 10 Amps, just in case there is a short this might burn it out and allow other equipment to operate.

I've seen some systems that are "optionally fused", they have an internal relais that's held by external power and if the fused lines fail, it automatically switches to unfused power, where the alarm system can evaluate the condition and decide to go back to the fused line if power returns. That's on top of having it's own backup batteries. Some of the more modern systems have watchdogs systems built in so that in case the fire alarm computer is on fire, it won't switch to the unfused line once the computer is no longer operational.

Re:That UPS you bought for your home server may not be as us... by angry_octet on Wednesday August 12, 2020 @01:41AM (#24128900)
I think you will find that those circuits use isolation transformers and have no path to earth normally -- if you touch active and normal building earth there is no circuit.

At to amperage ratings, don't you just mean that they have a 10x safety factor? Probably using current limiting supplies that can handle a short circuit. Fire circuits (and the feed from the main panel) should use fire rated cabling -- generally mineral insulated copper sheathed cables. Most of the standards (NFPA 75/76) are moving away from being prescriptive to being risk or performance based though, so it is possible they used normal conduit if the overall risk was small.

Incidentally, hospitals also make extensive use of isolation transformers for wet areas (like emergency / ICU), to provide a local point for resetting breakers (electronically), identifying which point is tripping, etc, see IEEE 602-2007. Much better filtering for medical equipment too. You also tend to see positive retention plugs (in the US, green dot UL 817).

Re:That UPS you bought for your home server may not be as us... by zaarn on Wednesday August 12, 2020 @02:42AM (#24129192)
The cables have fairly high ratings but in the even of a short circuit, the ratings have a safety margin of about 10x that the cable can last about 6 hours under, above the normal rating under standard conditions (ie, not fire).
Re:That UPS you bought for your home server may not be as us... by wvh on Monday August 10, 2020 @05:19PM (#24114119)
Many years back I had the opportunity to visit one of those data centres at the heart of Europe's financial operations. In case of fire, you had a minute or so to get out, because then the place is pumped full of argon. I guess that's a DC variation of your "battleshort"...
Re:That UPS you bought for your home server may not be as us... by robaato on Monday August 10, 2020 @06:11PM (#24114629)
Worked on a project in late 80s for an Italian bank. We implemented a SWIFT (financial) interface (on Stratus for reliability reasons). A few fun and games:

- The Stratus supposedly had fault tolerant duplexing of boards - pull one out and the other keeps working. A famous demo where local consultant pulls out the only board which wasn't duplicated!

- The machine room was also fire proofed with Halon gas (if I recall correctly), but due to security concerns the door was locked from inside by operators when present (luckily as programmers we weren't expected to work inside with doors locked!). Luckily also, it was my first exposure to "paired programming"/"paired operating" as in there were invariably 2 of them.

- A couple of the operators became too interested in how to work things, and wrote a quite comprehensive manual. They were "moved on" within the bank, because for security, management only wanted "people who could follow instructions", not "who can think for themselves".

- the whole headquarters building was in a guarded compound, where ultimate protection against "red brigade terrorists" (https://en.wikipedia.org/wiki/Red_Brigades) were the armed guards. We used to meet them in the ground floor cafe every mid-morning, with a revolver on their hip, while they drank an espresso with brandy chaser. Really inspired confidence that they would be there to protect us should the need arise!

Re:That UPS you bought for your home server may not be as us... by consp on Monday August 10, 2020 @07:03PM (#24115164)
> They were "moved on" within the bank, because for security, management only wanted "people who could follow instructions", not "who can think for themselves"

And this is why we cannot have nice IT security practices.

Re:That UPS you bought for your home server may not be as us... by frabert on Monday August 10, 2020 @05:52PM (#24114435)
Interesting, one of the guys that worked on my Uni's DC told us that the fire suppressor system is interlocked: to enter the DC, you need to open a door using a key. Upon opening the door, the fire extinguishing system is disabled, and is only enabled when the door is closed. The people working in the DC are supposed to keep the key on themselves. Also, they used Halon 1301 instead of Argon as the gas.
Re:That UPS you bought for your home server may not be as us... by mark-r on Monday August 10, 2020 @06:16PM (#24114695)
Halon is a CFC and isn't legal anymore due to international agreements.
Re:That UPS you bought for your home server may not be as us... by myself248 on Monday August 10, 2020 @11:38PM (#24116943)
Yeah, you can't install new Halon 1211 systems, but it's still out there. New systems are usually FM-200 or Inergen, which I believe can use the same plumbing as Halon.

Older systems, before Halon was invented, are typically CO2 or Argon. They require a lot more agent to accomplish the job, but they're very simple.

I worked briefly in a CO2 datacenter. It dated from the mainframe era, and had some gorgeous mementos on the walls. There were SCBA packs in the hallway outside, and everyone got a brief safety lecture before being allowed into the facility. The point I remember was basically "The alarm will sound intermittently for twenty seconds, then go solid for five, before discharge. If it goes off, do not stop to save your files, just run for the door. If you're not outside yet when the solid tone sounds, hold your breath and run on what's in your lungs."

Apparently the CO2 is considerably nastier than Halon to get a lung-full of...

Re:That UPS you bought for your home server may not be as us... by mark-r on Tuesday August 11, 2020 @12:42AM (#24117223)
Yes, carbon dioxide is not biologically inert like Halon is. It's the signal your body uses to say it's out of oxygen. You will definitely react negatively to getting a lung full of it.
Re:That UPS you bought for your home server may not be as us... by Doxin on Tuesday August 11, 2020 @03:23AM (#24117986)
For reference: It's not a lack of oxygen that causes the sensation of asphyxiation. It's a surplus of CO2. It'd essentially be the gas-based equivalent of waterboarding.
Re:That UPS you bought for your home server may not be as us... by zaarn on Tuesday August 11, 2020 @09:23AM (#24120018)
A large surplus of CO2 is also toxic, regardless of how much oxygen you have (see: Apollo 13).
Re:That UPS you bought for your home server may not be as us... by frabert on Monday August 10, 2020 @06:24PM (#24114779)
I must have confused it with some other fire suppression system I heard of, then!
Re:That UPS you bought for your home server may not be as us... by mark-r on Monday August 10, 2020 @06:31PM (#24114858)
Oh Halon was very popular for a long time. I don't know how long they could be grandfathered, but recharge supplies must be impossible to get by now.
Re:That UPS you bought for your home server may not be as us... by chipsa on Tuesday August 11, 2020 @12:02AM (#24117056)
https://en.wikipedia.org/wiki/Clean_agent_FS_49_C2

Looks like there's a drop in replacement for Halon.

Re:That UPS you bought for your home server may not be as us... by sbierwagen on Tuesday August 11, 2020 @09:34PM (#24127800)
Amusingly, tetrafluoroethane and pentafluoroethane are themselves being phased out in refrigeration applications because of their sky-high global warming potential, thousands of times worse than CO2.
Re:That UPS you bought for your home server may not be as us... by dividedbyzero on Monday August 10, 2020 @05:30PM (#24114218)
What would they do in case of injured/incapacitated people in there? Seems conceivable that some things that could start a fire might cause that as well.
Re:That UPS you bought for your home server may not be as us... by brewdad on Monday August 10, 2020 @05:40PM (#24114324)
Name a server rack after those who gave the ultimate sacrifice.
Re:That UPS you bought for your home server may not be as us... by captn3m0 on Monday August 10, 2020 @02:54PM (#24112698)
Re:That UPS you bought for your home server may not be as us... by Unklejoe on Monday August 10, 2020 @04:46PM (#24113788)
We use something similar on turbocharged drag (grudge) racing cars.

The boost controller has a "scramble" button which overrides the preset boost targets and allows you to run a higher (and unsafe) boost level as long as the button is pressed.

This is useful when you're in the middle of a $10,000 race and you're losing by a few feet.

Re:That UPS you bought for your home server may not be as us... by segfaultbuserr on Monday August 10, 2020 @05:33PM (#24114256)
What, this is real?! I already thought it was a Hollywood movie plot.
Re:That UPS you bought for your home server may not be as us... by Unklejoe on Monday August 10, 2020 @10:14PM (#24116548)
Yes, definitely real! Although what you're probably thinking of is nitrous oxide injection. One popular manufacturer of nitrous oxide systems is "NOS", which was featured heavily on the Fast & Furious movies.

The "scramble" button I'm talking about is for "boost", which is the amount of positive air pressure inside the intake manifold.

Both "boost" and nitrous can dramatically increase power by essentially getting more oxygen into the combustion chamber, but nitrous is a much more dramatic effect.

Re:That UPS you bought for your home server may not be as us... by bluGill on Monday August 10, 2020 @08:42PM (#24116000)
All motorcycle mechanics have a sign "speed costs money, how fast do you want to go". It isn't a joke, most of them can get as much power out of a 500cc motor as industrial engines get out of 19 liters. The catch is that you have to rebuild the engine every 10 hours of operation, while the industrial engine will run non-stop for many years.
Re:That UPS you bought for your home server may not be as us... by blaser-waffle on Tuesday August 11, 2020 @09:27AM (#24120047)
Those industrial engines also don't turn you into a meat crayon if you make a minor screw-up.
Re:That UPS you bought for your home server may not be as us... by bluGill on Tuesday August 11, 2020 @02:45PM (#24124191)
Sure they do, run away engines are a real problem. They will sometimes start using their own oil as fuel, and at that point you better run. (modern engines have safety air shutdowns)
Re:That UPS you bought for your home server may not be as us... by kitteh on Monday August 10, 2020 @09:05PM (#24116162)
A320 and B737 had special thrust bump modes for short fields like KSNA. Going back many years to the P&W JT8D you could operate in the red provided you did it for only a certain amount of seconds and logged it down.
Re:That UPS you bought for your home server may not be as us... by repiret on Monday August 10, 2020 @10:45PM (#24116691)
In general aviation aircraft it’s quite common for there to be time limits on the highest power setting.
Re:That UPS you bought for your home server may not be as us... by p_l on Tuesday August 11, 2020 @03:37AM (#24118065)
There's a maximum power mode selectable on the FADEC interface, which gives you at least 30 minutes of engine life - any use means the engine needs total overhaul.

This mode is pretty much reserved for go around or post V1 takeoff when one engine on twin engined airliner had failed.

Re:That UPS you bought for your home server may not be as us... by irontinkerer on Monday August 10, 2020 @04:32PM (#24113655)
“ the P-51H Mustang was rated at 1,380 hp, but WEP would deliver up to 2,218 hp”

Wow!

Re:That UPS you bought for your home server may not be as us... by linsomniac on Monday August 10, 2020 @04:44PM (#24113764)
Indeed. I forget where I read it, but it provides that power increase for (IIRC) some small double digits number of seconds before you need to throttle back, or the engine eats itself. If you are able to make it back to base, and that WEP wire is broken, the engine needs a rebuild before it flies again. Definitely an impressive power boost though.
Re:That UPS you bought for your home server may not be as us... by tomatotomato37 on Monday August 10, 2020 @07:29PM (#24115439)
The MiG-25 also have something similar where the engine redline is marked around Mach 2.8, but there's no system stopping the pilot from keeping it maxed to somewhere above Mach 3 but damaging the engine in the process.

Funnily enough I believe the SR-71 has the opposite problem, where the engines would merrily go above Mach 3.5 with no issue but the airframe and everything attached to it would get torn apart by the intense heat if the pilot tried to go any faster

Re:That UPS you bought for your home server may not be as us... by chipsa on Tuesday August 11, 2020 @12:11AM (#24117090)
No, SR-71s are still limited by engine temperature. They measure the Compressor Inlet Temperature, but I suspect that actual limitation is the turbine inlet temp. That said, it'll merrily try to go above that without issue if you don't throttle back.

Additionally, there's the Mach Cone. The shock wave off the nose forms a cone shape, with the angle determined by the speed of the aircraft. From the nose to the wingtip forms an angle of about 17.5 degrees, which corresponds well with the max speed from the CIT of Mach 3.3.

That both methods of determining max speed agree shouldn't be surprising. Skunk Works was filled with good engineers.

Re:That UPS you bought for your home server may not be as us... by p_l on Tuesday August 11, 2020 @03:40AM (#24118086)
MiG-31 has limiter on engine controls connected to airspeed and mach number indicators, because the engine will happily run above Ma 3, with no issues.

The airframe, however, will get bent from temperature and you better hope the missile cooling loops don't fail.

The power isn't dialed down at lower speeds in order to provide thrust for maneuvering and climb.

Re:That UPS you bought for your home server may not be as us... by eigenvector on Monday August 10, 2020 @01:32PM (#24111783)
I work in the power generation industry. We have a lot of primary/secondary as well primary/standby systems.

The cardinal rule is - test everything. Test it when you install and commission it, test it when part of the system changes, test it periodically during maintenance shutdowns, test it when you have a convenient time to do so without losing production (such as an outage to a different system that necessitates your system being offline).

Test components individually - at the factory and in the field - and test systems end-to-end.

Test with the most adverse possible conditions, not the most optimal. Test beyond your normal operating envelope.

When I participate in design reviews as a maintenance engineer my primary line of inquiry is: how will I test this stuff during operations? Has it been designed in a way that makes testing impossible without an expensive outage? Can workers safely gain access to components that need routine testing without having to de-energize other, unrelated parts of the system?

Now not every industry has the budget available to test to the extent that we do. But even in our industry, I often see a penny-wise, pound-foolish approach of "assume this thing works perfectly from the factory then act surprised years later when an abnormal condition occurs and it fails".

If something was not tested, it may not work. It might be for a very simple reason (like mis-configuration of a single setting during installation) or a very complex reason. But either way, wouldn't the company rather know their equipment doesn't work -before- putting it into service than after?

Re:That UPS you bought for your home server may not be as us... by a1369209993 on Monday August 10, 2020 @02:56PM (#24112725)
Apropos of this, in TFA's case they apparently never tested "rip the battery out while the system is running". (There are other, harder-to-test ways for a battery to fail, but that seems like a obvious and easily-tested failure mode.)
Re:That UPS you bought for your home server may not be as us... by mprovost on Tuesday August 11, 2020 @01:23AM (#24117396)
That's a good point about testing. I think the key point is where the testing is happening. I'm sure the UPS (and other power equipment) is tested by the manufacturer for many different failure scenarios. The issue is that to the end user, the response is often surprising. So I'm sure to the APC engineer, having the unit shut down when the battery fails (according to some metric) but while there is still mains power available is something that they tested. But the consumer hasn't tested a battery failure, and in fact it may be difficult to test the exact failure mode (low voltage etc) without access to some spare, old, batteries. So when it happens and you're surprised and you call the manufacturer, they (almost always) will say that it operated according to how it was designed and tested. It just didn't do what you wanted or expected. At least that was always my experience doing a post mortem on a datacentre outage.
Re:That UPS you bought for your home server may not be as us... by kitteh on Monday August 10, 2020 @09:07PM (#24116176)
The telco Central Office I worked in had weekly tests where we'd run the generator for an hour. Everyone's desk computer required a UPS as they were on utility. Comically after a few years you'd have to get a new UPS at your desk as theyd die from continual use.
Re:That UPS you bought for your home server may not be as us... by myself248 on Monday August 10, 2020 @11:18PM (#24116837)
Most of the COs I worked in did monthly generator "dry runs" without putting load on it, and then every 6 months they'd do a "drawdown test":

Disable the turbine. Kill AC power (turn off all the rectifiers and HVAC) and let the office run on battery for a while. Closely monitor the batteries, and the temperature in critical areas.

After a good while on battery (an hour or two, if I recall), walk leisurely over to the turbine and enable it. Let it start and warm up, then transfer the rectifiers and HVAC over to generator power.

Closely monitor the turbine while the batteries recharge, the HVAC while on unusual power, and the humans who are really glad to have air conditioning back.

When satisfied that all is copacetic, transfer things back to utility AC, shut down the turbine, and call the fuel company to top off the tank.

Most of the batteries in these offices were between 10 and 20 years old, with a couple instances of the beautiful cylindrical "Bell Cells" still in service. They were sized to not break a sweat running the whole building, and impeccably maintained, so they didn't tend to fail prematurely.

Re:That UPS you bought for your home server may not be as us... by blaser-waffle on Tuesday August 11, 2020 @11:36AM (#24121471)
Same basic idea. We'd exercise the generators weekly/bi-weekly, depending on the site and configuration.

Once a quarter we'd do a live load transfer, and watch all the PDUs and UPS systems do their thing. Those could be stressful...

Re:That UPS you bought for your home server may not be as us... by laurent92 on Monday August 10, 2020 @06:29PM (#24114836)
How do you test diesel generators which back up a nuclear reactor? I understand you do it during the plant downtime, but don’t you require something at the other end of the network that consumes the 1GW you generate?
Re:That UPS you bought for your home server may not be as us... by gh02t on Monday August 10, 2020 @08:06PM (#24115737)
A combination of routing the power elsewhere in the grid when possible and banks of gigantic (shipping container sized) dummy loads. As other people have noted, you don't need to dissipate the full amount of power all at once, you test the generators individually.

http://resistive-loadbank.sell.everychina.com/p-109417602-35... is the first example I pulled up, and there are bigger ones.

Re:That UPS you bought for your home server may not be as us... by a9h74j on Monday August 10, 2020 @07:52PM (#24115639)
The diesel generators can back up power to safety systems, so closer to MWs than GWs. Some run with left-handed and right-handed engines coupled face-to-face for redundancy; thus you might AFAIK shut down fuel injection to one to prove the other is supplying torque.
Re:That UPS you bought for your home server may not be as us... by bluGill on Monday August 10, 2020 @08:48PM (#24116056)
You test them live: someone turns off the outside power switch and you watch them start and take over automatically. Everyone is watching the gauges and if there is a sign of trouble you reengage mains power, then fix the problem and retry. This test is done monthly.

I don't have actual knowledge of nuclear power plants, that is classified beyond the public. However hospitals do the above, because if thing dont work on a bright sunny day how do you think they will work when a tornado has not only taken out power, but also filled the er with server trauma cases.

Re:That UPS you bought for your home server may not be as us... by solstice on Monday August 10, 2020 @07:45PM (#24115579)
Naive idea: Putting a big copper rod in whatever body of water the plant uses for cooling and connecting power to it should do it no?
Re:That UPS you bought for your home server may not be as us... by Scoundreller on Monday August 10, 2020 @10:04PM (#24116489)
Not copper. You want resistance to be highest at your heat dump. Otherwise all your wires melt.
Re:That UPS you bought for your home server may not be as us... by generatorguy on Monday August 10, 2020 @01:03PM (#24111465)
Electrical Engineers would determine the settings for the electrical protection of a generator. Mechanical Engineers would determine the settings for mechanical protection of a motor. To operate the equipment beyond thresholds would result in reduced lifespan or immediate and permanent damage to the equipment. If this is what the customer requires they have to specify that to the Engineer so that when the equipment blows up the Engineer is not liable.

Diesel generators for hospitals and water pumps for fire suppression systems are generally set up with very loose settings as it is clearly worse for a patient to die or a building to burn down than to destroy a machine, and this requirement is in the specifications.

Re:That UPS you bought for your home server may not be as us... by brudgers on Monday August 10, 2020 @05:36PM (#24114286)
Diesel generators for hospitals and water pumps for fire suppression systems

Those systems also require regular testing by qualified personnel. NFPA 25 requires monthly testing of fire pumps and annual flow testing. For low rise buildings with inadequate municipal water pressure, a water tower might be cheaper and easier. NFPA 99 - Standard for health care facilities, requires emergency generators to be tested 12 times per year.

Re:That UPS you bought for your home server may not be as us... by generatorguy on Monday August 10, 2020 @07:34PM (#24115474)
The testing would consist of running the diesel generators under load but not operating in abnormal conditions.
Re:That UPS you bought for your home server may not be as us... by bluGill on Monday August 10, 2020 @08:51PM (#24116073)
True, but the easiest test is to pull the main power to the hospital and watch the system automatically take over. If anything fails you know what is wrong. This is scheduled so that those in surgery (or other worrisome situations) have extra help and can signal get back to mains power now.
Re:That UPS you bought for your home server may not be as us... by generatorguy on Monday August 10, 2020 @11:29PM (#24116907)
ideally the hospital’s generator control system or transfer switch is able to start and synchronize the generators and turn up the throttle until no power is being imported from the utility prior to opening the utility feed, thereby transferring the load from the utility to the generators. This test is only performed when there are no surgeries scheduled and it is cancelled if there are any emergencies coming in all of the sudden. If the test is already in progress incoming ambulances with patients requiring surgery are diverted to another hospital. To finish the test the hospital is synchronized to the utility, the throttle on the generators turned down until no power is flowing through the generator breaker which is then opened.

I’ve built and programmed control systems that do all this with multiple generators, multiple utility feeds, and multiple tiers of loads of different priorities.

Re:That UPS you bought for your home server may not be as us... by jimmygrapes on Monday August 10, 2020 @11:34PM (#24116925)
That is a good way to test the generator without interruption, but at some point you still need to test the ATS under real world circumstances: unstable or suddenly missing utility power. Throwing the main covers one of those, and the other requires a little more planning and resources.
Re:That UPS you bought for your home server may not be as us... by generatorguy on Tuesday August 11, 2020 @11:05AM (#24121043)
Honestly I am struggling to remember an instance where we purposefully plunged an operating hospital in to darkness.

You can test the system’s ability to detect loss of power or poor quality power by lifting sensing wires or injecting out of frequency or voltage range power with a test set, and load pickup and load testing we used a load bank. The actual hospital load is fed through bypass breakers during these tests so if the utility went out we would have to manually operate breakers to disconnect the load bank and swing the vital loads over to the generators.

If I am setting up electrical protection on a generator I don’t put a short circuit on the generator, I show the protection relay what it would see by injecting current and voltage with a relay test set and make sure the relay gives the signal to open the breaker when it should.

Greenfield hospital generation sites are straightforward to open the utility feeds since the vital loads are still fed through the old power system.

Re:That UPS you bought for your home server may not be as us... by bluGill on Tuesday August 11, 2020 @02:44PM (#24124171)
You can (and should) do all that. However until you actually disconnect from the utility power you don't know for sure that the systems all work. The generator powers a load, but turns out your hospital is more load than you thought. Or the generator starts but the switch to connect the hospital to the generator is broke and you didn't test that...

In the end you WILL test what happens when the hospital plunges into darkness. Would you prefer the first test to be a a bright sunny day when all the staff is ready for something (and you can turn the utility power back on quickly if something fails), or when someone with a backhoe/chainsaw has an accident with your power feed. The latter will happen, I don't know how or when, but at some point in your future trees will take down power lines, backhoes will cut something, major weather events will take out power for days, maybe a large blackout...

You should have course test everything separately often. However if you wouldn't let a random electrician (electrician only because they will know how to not kill themselves) turn off your utility connection with only a day of warning, your system isn't trusted to handle real events which are typically much worse than a clean turn off the switch.

Re:That UPS you bought for your home server may not be as us... by generatorguy on Tuesday August 11, 2020 @03:55PM (#24125207)
I remember some complication with the elevators not working or homing properly at one place so we must have had an outage. it was probably at 3am, scheduled, and with incoming patients diverted. All our tests are and I’m in the electrical room so they all looks the same to me. An actual outage to vital loads would be done once and then you shut the door and walk away as there was never any question that it was going to work.

FYI where I’ve worked the tiers of loads are metered individually and picked up in order of priority and with a check to verify the combined capacity of the generators which are online is great enough to pick up the load. Likewise loads are shed in order if there isn’t enough generation such as one generator tripping.

I don’t really see that much difference between opening a breaker and bad quality power. It is the same sensing and logic in both cases: is the magnitude and frequency of the phase to phase voltages in the acceptable range or not?

Re:That UPS you bought for your home server may not be as us... by bluGill on Tuesday August 11, 2020 @04:43PM (#24125751)
The important part is that whatever bad happens to mains power, the backup systems detect it and properly take over. Bad frequency or power out are both bad situations that the backups need to handle. You need to have confidence that your systems will handle both. If you only pull power and never test bad frequency, one phase missing, brown outs, over voltage, (list not complete because I don't know everything) then you have not done your job. If engineering determines that your test also covers another bad scenario then I'm fine with not testing it.
Re:That UPS you bought for your home server may not be as us... by brudgers on Tuesday August 11, 2020 @08:50PM (#24127588)
Electric utilities provide appropriately higher levels of service and communication to hospitals. Hence there are protocols insuring hospitals don’t lose power in the way you are imagining. The people involved are serious competent professionals with extensive relevant experience, training, and education focused on safety and in touch with the specific facts of actual hospitals and utility grids.
Re:That UPS you bought for your home server may not be as us... by bluGill on Wednesday August 12, 2020 @08:34AM (#24131387)
There isn't much that they can do about storms knocking down wires, squirrels chewing a wire and knocking out a phase (other than taking down all phases). Which is why hospitals have several backup systems.
Re:That UPS you bought for your home server may not be as us... by mulmen on Monday August 10, 2020 @03:42PM (#24113222)
My grade school neighbored a hospital. I recall occasional loud rumblings of the backup generators during recess, presumably they were tests. Legend was they came from a WWII submarine! Neat!

Years later I had a tech support job in that hospital and all the PCs had small UPSs to keep them alive for the 30-60 seconds it took the big generators to come online.

I never did get a chance to see the generator room.

e: Another personal anecdote that is more relevant to the topic.

I was working for a VOIP provider back when Hurricane Sandy hit NYC. Part of our production network was housed in a telecom building in Manhattan. We received daily updates on how many floors the flooding was from our equipment and about the fuel deliveries to the building backup generators. Apparently the generators were on the roof? I'd love to know more about the setup in that building because we had zero downtime.

Re:That UPS you bought for your home server may not be as us... by Cyphase on Monday August 10, 2020 @07:50PM (#24115620)
Your last paragraph reminded me of the datacenter that famously rode out Hurricane Katrina: https://www.wired.com/2005/11/hurricane-heroics/

The blog where they posted regular updates is still up; here's the first post related to the storm: https://interdictor.livejournal.com/2005/08/27/

"Hmm. This could actually be a nasty storm."

Re:That UPS you bought for your home server may not be as us... by jasonjayr on Monday August 10, 2020 @11:28PM (#24116897)
That blog is an amazing insight into internet operations + culture from that time frame. Windows Media for streaming! URLS with perl/script.pl in them! AIM/ICQ! It helps the writer has a bit of a flair for storytelling.

Very good read!

Re:That UPS you bought for your home server may not be as us... by mulmen on Tuesday August 11, 2020 @04:16AM (#24118278)
Just getting around to really reading this. The bit rot is incredible.
Re:That UPS you bought for your home server may not be as us... by mulmen on Monday August 10, 2020 @08:12PM (#24115794)
I’ve never heard this story, thanks for sharing!
Re:That UPS you bought for your home server may not be as us... by Fogest on Monday August 10, 2020 @03:48PM (#24113261)
Working in a 911 call center this is how it worked for us. Except we had larger UPS's that could last probably 30-60 minutes in the building and then a large external generator in a small building beside. You'd see stuff like the lights that weren't on the UPS's turn off for a few seconds before the main generator fired up.
Re:That UPS you bought for your home server may not be as us... by mark-r on Monday August 10, 2020 @06:24PM (#24114778)
My current job refuses to replace old UPSs that have died, because they have a generator to keep the building powered in case of loss of power. But that doesn't keep your PC running for the moments it takes the generator to spin up.
Re:That UPS you bought for your home server may not be as us... by Fogest on Monday August 10, 2020 @07:17PM (#24115308)
Oh yeah, if we had that where we worked people would definitely be in danger. Most likely all of our systems would be down if we didn't have the UPS+Generator setup. It often takes 5-30 seconds for the generator to kick in and be running at full capacity. Normally 5 seconds, but sometimes there is a bit of a delay.
Re:That UPS you bought for your home server may not be as us... by mark-r on Thursday August 13, 2020 @10:35PM (#24152577)
The other part of the equation is that they're recommending that everybody switch to laptops, which are capable of running for a long time without power. My job involves a lot of long compiles, so I want to stick with a desktop PC to get the maximum compile speed.
Re:That UPS you bought for your home server may not be as us... by Fogest on Thursday August 13, 2020 @11:08PM (#24152817)
Considering how old the software we used was it definitely could run well on a laptop. However it does mean you have to manage the battery life on a lot of different laptops versus one central area.
Re:That UPS you bought for your home server may not be as us... by mark-r on Friday August 14, 2020 @11:45PM (#24165730)
Don't get me wrong, the company pushes laptops for reasons other than eliminating UPSs. Even a nearly dead laptop battery should tide you over for the minute it takes for the building generator to get online. I don't think managing laptop batteries is a big concern for our IT department.
Re:That UPS you bought for your home server may not be as us... by linsomniac on Monday August 10, 2020 @04:49PM (#24113822)
Everything involved in 911 calls had special tagging to indicate it. Failure to go through appropriate review and approval processes before doing work on tagged equipment would create a Resume Generation Event.
Re:That UPS you bought for your home server may not be as us... by Scoundreller on Monday August 10, 2020 @10:09PM (#24116517)
Nowadays, some hospitals, especially those with heating loads, are doing the opposite:

Run a combined-cycle natural gas generator 24/7, and use the grid as your backup. Gets good efficiency. Runs loops on the coolant, oil and exhaust for water/building heat, and can switch over instantly in the event of failure.

https://www.solidwastemag.com/feature/talking-bout-cogenerat...

Re:That UPS you bought for your home server may not be as us... by dn3500 on Monday August 10, 2020 @10:00PM (#24116472)
WWII submarine generators produced DC, so they couldn't have been used directly as backup power for a hospital. Maybe they were refitted with alternators.
Re:That UPS you bought for your home server may not be as us... by userbinator on Monday August 10, 2020 @11:18PM (#24116838)
I wouldn't be surprised if it was the engines that were from the WWII era.
Re:That UPS you bought for your home server may not be as us... by cactus2093 on Monday August 10, 2020 @03:48PM (#24113264)
> it's difficult to test this equipment

Can't you just flip the circuit breaker or just pull the plug out? Or does a real power outage usually look different to that in some way, like a surge followed by a shutoff?

Re:That UPS you bought for your home server may not be as us... by gerdesj on Monday August 10, 2020 @07:36PM (#24115497)
If done right it is routine.

In our small data centre we have a board with five lights and a key. You turn the key and the mains is cut off. The lights give the status. The UPSs beep for a few seconds "on battery", the genny fires up in the boiler room and when it stabilises, it takes the load. We leave it running for 5-15 minutes, go out and check the fuel levels and top up accordingly. It's diesel so safe to hot refuel. When done, turn the key back and then the UPSs take the load again and then hand back to the mains. The genny runs for another 15 minutes and then shuts down. That should avoid flip flopping.

You then fill in the test form on our wiki and job done. We do it weekly to fortnightly. An ESP8266 based thingie reports status back to the central monitoring (GPIOs and ESPHome -> Home Assistant)

Re:That UPS you bought for your home server may not be as us... by bradstewart on Monday August 10, 2020 @04:27PM (#24113606)
It depends. We built and deployed some IoT devices (controlling and reporting status of lights, HVAC systems, that sort of thing) to a bunch of greenhouses--one of which ran on generator power.

The customer was running the generators right at the maximum capacity, and we'd see really, really odd things on the power lines. Some times a surge like you mention, sometimes changes to the frequency, sometimes changes to the voltage. We discovered all sorts of weird brownout conditions in our devices.

Re:That UPS you bought for your home server may not be as us... by jcrawfordor on Monday August 10, 2020 @04:30PM (#24113643)
It's difficult in practice to transfer load to generators for test especially if there's load shedding involved as is common in larger facilities (higher-power-consumption items like part of air conditioning capacity are cut off when transferred to generators). To help address this a lot of more critical facilities are equipped with a test load (really just a very large outdoor resistor with active cooling) that allows for running the generators at rated load for test and exercise without actually transferring. Of course, these tests do not always detect problems with the transfer equipment, and if things aren't designed and maintained well the generator may experience load under real conditions that behaves differently from the test load.
Re:That UPS you bought for your home server may not be as us... by avh02 on Monday August 10, 2020 @06:55PM (#24115084)
Iirc (top of my head), some datacenters go off the grid regularly at peak (provider) load times and get more favorable rates for doing it. I guess it ensures your fuel doesn't go stale and all your transfer systems work.
Re:That UPS you bought for your home server may not be as us... by bluGill on Monday August 10, 2020 @08:54PM (#24116102)
Good point. Many power companies will give you a nice discount year round if you will disconnect from mains on demand. If you are on the fence about a good backup generator this will more than make up the difference.
Re:That UPS you bought for your home server may not be as us... by zzz61831 on Monday August 10, 2020 @04:03PM (#24113409)
You can put your equipment on a voltage relay to make failure modes as simple as flipping a circuit breaker. It will turn off once voltage drops too much and will wait for it to stabilize for specified amount of time to turn it on.
Re:That UPS you bought for your home server may not be as us... by WalterBright on Monday August 10, 2020 @03:52PM (#24113306)
Around here the typical failure pattern is the power goes out for a second or two, then comes back on for a few seconds, then goes out for good.
Re:That UPS you bought for your home server may not be as us... by myself248 on Monday August 10, 2020 @11:29PM (#24116900)
That's a recloser operating.

Initially, there's a fault, say, a tree contacting a wire. This conducts a bunch of current to ground, the voltage in part of the network sags a fair bit, and a protective relay senses the overcurrent and opens the circuit. This happens in a second or two, and your power goes out. You may notice lights dimming or flickering in the instant beforehand.

But, many faults are transient. Perhaps that tree branch finished falling, and is no longer against the wire. So the protective relay performs a "reclose" operation, where it turns the power back on for a moment and measures the current. There's normally a huge starting inrush as motor-driven appliances restart, so it waits a moment before sampling the current, and this is the period when the power comes back on for a few seconds. (It's probably pretty nasty power during this interval, too.)

If things are good, hey, you're good! A brief outage while waiting for the fault to clear itself, and then everything's back. The operation will be reported to the dispatch center, and someone may come out to inspect the area in case there's damage.

But if the fault is still there, and after a moment the current is still way higher than it should be, the recloser opens again, and this time it stays open. Gonna need a crew to move that tree, and then they'll manually try again.

Re:That UPS you bought for your home server may not be as us... by copperx on Monday August 10, 2020 @02:17PM (#24112288)
> But it's difficult to test this equipment

You could just, you know, pull the plug and see what happens?

Re:That UPS you bought for your home server may not be as us... by CarbyAu on Monday August 10, 2020 @07:54PM (#24115655)
Yeah, but any enterprise that warrants this gear likely also has a bunch of risk averse bureaucracy saying "Unplug that, you lose your job."
Re:That UPS you bought for your home server may not be as us... by bluGill on Monday August 10, 2020 @08:56PM (#24116108)
That is a mid level of risk adverse. High levels make you prove everything works by unplugging everything monthly.
Re:That UPS you bought for your home server may not be as us... by yjftsjthsd-h on Monday August 10, 2020 @09:50PM (#24116416)
Risk aversion comes in shallow and deep variants.