We had some excitement here in at work last week. On Thursday I was sitting in my office trying to set up a server when suddenly the server stopped responding. I was a bit worried, because I hadn’t done anything that could have caused it to hang up, so I tried to reach it another way and realized that I’d lost my network connectivity altogether. Right about then I got a knock on my office door and one of my employees said, “Um… there’s water pouring out of the server room…”
What?!
Sure enough, when I checked, there was about a quarter of an inch of water on the floor outside the server room and it was quickly spreading. All of our network equipment is in that room, so I quickly opened the door and found that it was raining
(rain-forest raining, you know where it’s so humid the water just seems to be forming into droplets out of the air around you) down directly on the rack that ouses our equipment. I ran in there, jumping as I grabbed the power cords to ensure that I didn’t electrocute myself, getting drenched the whole time, and managed to get the power cut off to all of our equipment, and then had to go kick about a hundred students out of the lab, telling them that they wouldn’t be able to print, access email, or really anything else. Some had no way of saving what they were working on, so I started scrounging up floppy disks to help them out.
Before we found out what had caused the indoor monsoon, the networking guys got here and started pulling out the equipment, and water poured out of every single thing they took off the rack. We pulled them all into my office and opened them up to let them dry out, and started trying to figure out what was going on.
Turns out a custodian had broken a pipe directly over our server room while cleaning, and instead of telling someone about it, he decided to hold on to it for 10 minutes and see if he could somehow stop the water with his bare hands (Typing that makes me think of that song, “Jackie-chan-jackie-chan-jackie-jackie-chan-chan….” for some reason). That ten minutes allowed the water to pool in the ceiling above our server room and start leaking down through it. Had the dude told someone immediately, we’d probably have been able to cover our equipment and get it shut down and only been out of commission for a couple of hours.
As it was, the lab was out of commission for the rest of the day about 8 hours) waiting for the drenched equipment to dry out enough for us to test it. Once that had happened we started firing things up and out of 4 switches ($1000-$1500 each), 1 UPS ($3000) 1 router ($2500) and 1 server ($2500), we only lost two switches. Everything else fired up again. We decided to let the room air out and dehumidify over night, and got everything put back in and all 120 computers reconnected by about noon the next day.
This was in one of the most used computer labs on campus, and my staff were spending all their time turning people away and explaining why the computers didn’t work. It was quite the ordeal, though I can *almost* laugh about it now.
