Tilted Forum Project Discussion Community - View Single Post

vanblah · 07-19-2007, 07:07 AM

Long story ... but big, unusual problems:

About 3 years ago we moved our IT department and server room into a brand-new state of the art building on campus. This building cost around 43 million dollars and has all the bells and whistles.

The server room is nothing different than most server rooms: separate A/C with backup; 15Kva power conditioner and UPS; natural gas generator with diesel backup. Nothing can go wrong ... right?

Well, during the first month we lost power to the room because the utility company was doing some work and severed one of the main feeds to the campus. No biggie right? The UPS and generator will keep us up and running. Nope. It all failed. Nothing like trial by fire. Room goes down hard. We go through and unplug everything because we don't know what's going to happen when power is restored ... good thing too.

We get the techs out here and they "fix" everything so nothing like this will ever happen again.

Power goes out again a few weeks later (rainstorm). Generator kicks in; but the vibration from the generator itself trips the earthquake valve which cuts off natural gas. No big deal - diesel should kick in any minute which sends an alarm to physical plant to call us and give us an hour to get back to campus and shutdown right? Nope. UPS batteries are depleted in 45 minutes. The UPS is supposed to send a graceful shutdown signal to the servers when it reaches 20% ... didn't happen. The room goes down hard.

Power is restored. The room comes up just as hard. We lose three servers because of the subsequent surges. Why is the power conditioner failing; shouldn't it be inline and protect us from surges? You'd think so. But it seems that the power conditioner fails "closed" when batteries are depleted. So when power is restored the servers are NOT protected but the power conditioner itself IS protected from surges.

The technicians for the power conditioner tell us that the device will protect itself from surges after batteries are depleted because it would "cost us a lot of money to replace it."

Hmmm, let's think about the logic here. We have paid around $30,000 for a power conditioner to protect $200,000 worth of equipment; but they have designed the conditioner to protect itself rather than the much more costly equipment downstream?

To make a long story short it took us two more power outages to get everything working the way it should. And I still don't trust any of it.

07-19-2007, 07:07 AM	#5 (permalink)
vanblah Junkie	Long story ... but big, unusual problems: About 3 years ago we moved our IT department and server room into a brand-new state of the art building on campus. This building cost around 43 million dollars and has all the bells and whistles. The server room is nothing different than most server rooms: separate A/C with backup; 15Kva power conditioner and UPS; natural gas generator with diesel backup. Nothing can go wrong ... right? Well, during the first month we lost power to the room because the utility company was doing some work and severed one of the main feeds to the campus. No biggie right? The UPS and generator will keep us up and running. Nope. It all failed. Nothing like trial by fire. Room goes down hard. We go through and unplug everything because we don't know what's going to happen when power is restored ... good thing too. We get the techs out here and they "fix" everything so nothing like this will ever happen again. Power goes out again a few weeks later (rainstorm). Generator kicks in; but the vibration from the generator itself trips the earthquake valve which cuts off natural gas. No big deal - diesel should kick in any minute which sends an alarm to physical plant to call us and give us an hour to get back to campus and shutdown right? Nope. UPS batteries are depleted in 45 minutes. The UPS is supposed to send a graceful shutdown signal to the servers when it reaches 20% ... didn't happen. The room goes down hard. Power is restored. The room comes up just as hard. We lose three servers because of the subsequent surges. Why is the power conditioner failing; shouldn't it be inline and protect us from surges? You'd think so. But it seems that the power conditioner fails "closed" when batteries are depleted. So when power is restored the servers are NOT protected but the power conditioner itself IS protected from surges. The technicians for the power conditioner tell us that the device will protect itself from surges after batteries are depleted because it would "cost us a lot of money to replace it." Hmmm, let's think about the logic here. We have paid around $30,000 for a power conditioner to protect $200,000 worth of equipment; but they have designed the conditioner to protect itself rather than the much more costly equipment downstream? To make a long story short it took us two more power outages to get everything working the way it should. And I still don't trust any of it.