From http://status.fogcreek.com/2012/10/fog-creek-services-update...:

"Here's the physical situation:

The generators are on a high floor in the building and the pumps supplying the generators with fuel are submerged. The best option at this point is for people to physically lug diesel up over a dozen floors, or make other arrangements for pumping fuel to that high floor. "

Yes, a generator not in the basement / ground level makes sense. When a flood happens, it usually happens from the ground floor up ;) Thus why generators are at the upper levels.

Even in this case, using a pump solution makes little sense. Assuming you have electricity to power the pump, I suppose you could power the pump with the electricity from the remaining fuel, but a 17 floor pipe containing fuel would probably be heavier (and more dangerous) than a long mains power cord, and it wouldn't make sense to have a 17 floor cable to power the pump with the pump at the bottom of the run than it would to have just the pipe of fuel running the length of 17 floors and the pump at the top of the run.

Time for a winch? I don't think Home Depot stocks a 17 story pipe, and relocating the generator probably will take more time than just winching / carrying up the barrels.

I believe having the pumps at the bottom makes sense because you only have ~14PSI of atmospheric pressure to use to pull the fuel up if the pump were at the top, but you can generate as much pressure as you need at the bottom to push the fuel up.

Generators higher up makes sense, but 17 stories high? I suppose real-estate concerns limit where you can put them though.

In this case, I think you are correct. Need lots of exhaust so have to be on the roof. One police plaza just tore the side of their building off to make room for generator exhaust. It's a warzone in lower manhattan right now.

Wow, that is pretty nuts. Did they install temporary generators, or did they run into unanticipated ventilation issues with existing generators?

Why not store the fuel on the same floor as the generators?

I understand that this might be a fire-safety issue. In which case I would agree that a winch should have been installed.

(I'm possibly missing something else obvious here!)

If you'd read the previous articles about the hurricane's effects on datacenters, you'd know that the diesel pumps in many buildings were wiped out by flooding.

"buy a fucking pump"

I suspect this is good advice, but the image that popped into my head was an elevated generator, a ground-level tank, and an electric pump reliant on either mains or generator power. "It's always worked during our monthly tests!" Whoops!

Probably the elevated generator should be installed with at least a small gravity-fed tank, easily refilled once the pump is operating. I doubt anyone wants a giant fuel depot anywhere inside or on the roof of the building.

Keep in mind that for heights taller than 3 floors (and practically, the limit is shorter than that), pumps have to be located on the same level as the fuel supply because the atmospheric column only has so much mass. Pumps and their electrical inputs should probably be installed with waterproofing if flooding is considered likely.

The fuel is stored in the basement of the datacenter. That flooded, taking the pump that was supposed to pump up the fuel stored in the basement to the 17th floor.

This has nothing to do with buying a pump, that was already there, this has to do with the fact that the diesel tanks and the pumps for them are under water.

In the future I hope they put the pumps on higher floors, and make sure the fuel tanks are completely sealed, so even if the tanks are flooded the pump can keep pumping fuel out of it.

Mind you they are in lower Manhattan, not some random suburban office park.

That was probably their first mistake.

How so? This storm was a whopper. To the point that buoys were reporting waves 5 times taller than anything on record. Historically, Manhattan has been a fine place for a data center. Moreover, Peer1 survived the 2003 blackout, the only other recent event of this magnitude, without service disruption.

You can armchair quarterback this all you want, but unless you're actually making the decisions, perhaps you should consider the information available.

Well sure, and everything is fine until it isn't. It would be impossible for Manhattan to have any kind of Internet without peering and NAPs, but for hosting? I'll choose Nowheresville anytime, which I have done before.

I'm sure there are situations where you need to get someone to the DC quickly, so a Manhattan DC makes a lot of sense for an NYC based company. The current solution already worked the VAST majority of the time too. If they'd hosted it in central nowhere, people would criticize them for not planning appropriately for the times they need someone at the DC.

LOL I agree. There are a few small data center in Manhattan. Actually, if they choose data center like Amazon they may expect outrage once in a while....

In most buildings, the generator is at the top floor, just saying. Source: I worked @ cbc on Queen street in ottawa,ON, and the generator is on the top floor.

Makes sense actually, probably in case of flooding.

Doesn't make that pump any less sensible though!

Apparently the pumps got flooded: http://status.fogcreek.com/2012/10/fog-creek-services-update... (near the bottom)

Well, now I do feel like a cunt.

Past my bedtime, so didn't connect this with that wee bit of weather the East coast USA experienced recently.

Sincerely hope lugging diesel remains the least of their worries.

This is reminiscent of what happened at Fukushima Daiichi, although with less dire consequences.

I suspect that's for ventilation reasons