[geeks] OH YEAH

Jonathan C. Patschke jp at celestrion.net
Fri Mar 15 21:57:19 CST 2002


> >From what I heard (it was my first day, actually) it wasnt a SWBT tech, it
> was a new employee.  They made us all go through little "when and when not to
> hit the big red switch" "training" about a week later.

No no, that was the -second- time.  The two incidents were about two
months apart, IIRC.  The first time, it was a SW Bell tech who got scared
shitless when he set the alarm off by not badging out.  He panicked,
slammed the EPO with his fist.

That didn't kill the alarm (and he didn't think to just walk out), and the
dude went ape-shit.  When the found him, he was trying to yank the lockbox
off the FMS-120 release.

I heard this from the manager of the datacenter (back when it was
$OtherCompany), and I still have the email[1] confirming what he told me
via phone the next day in my Eudora hive.

They didn't think to lock down the EPO until the lady who worked for
$DataCenter made the same (or similar) mistake.  After that, they made
-everyone- take the little training course, even us customers.  They
reassured us that we needn't go psycho when we hear the alarm.

> Nothing like hearing two huge datacenters go completely SILENT.

They weren't silent for long.  I can only imagine that what happened when
I noticed was the same thing that happened at all $DataCenter's other
customers:
   Me: *calls $Colo*
 Tech: $Colo.  How can I help you?
   Me: Hi, this is Jonathan with $Company.  We have machines at your 
       center.
 Tech: Oh, hi, Jonathan.  This is $Manager.  What's up?
   Me: Not our servers.  What happened?
 Tech: I don't know.  It's dark here.
   Me: Dark?
 Tech: Yeah.
   Me: As in, "no power"?
 Tech: Uh, erm, yeah.
   Me: Shit.  *Click*
       $Boss, I'm heading to Austin.

By the time I got there (40 minutes later, thanks to rush-hour traffic),
it looked like a sci-fi convention with about 5 dozen bearded sysadmins
and PFYs milling about and cursing.

--Jonathan
[1] If I can find it, and sufficiently mangle it to protect the
    innocent/guilty, I'll post it.  It's a rather entertaining read.



More information about the geeks mailing list