[SunRescue] Re: Best wishes for Bill...

[Bill Bradford]

On Sat, Nov 04, 2000 at 07:23:07PM -0500, Kurt Huhn wrote:
> I know the feeling - my products will be rolled out to the customer sites.
> If any one of them barfs, we lose millions.  The responsibility is like a
> drug - you want more, more MORE!
> Kurt

My first day on the job, a new intern accidentally hit the Big Red Switch.
The one that kills ALL the power to the entire datacenter.  Kills the UPSes
and air conditioning too.

Everyone was sitting there and noticed that our sessions had froze.  We
then noticed the LACK of "AC noise" and roar from the other side of the
wall.

All the sysadmins slowly stood up and started walking back to the
datacenter.  We kinda hung our heads.  Wasnt our fault, but the rest
of the day was going to be VER interesting.

Walked into the datacenter, and the only noise was the vent. fans on
the EMC units slowly turning off one by one as the built-in UPSes shut
everything down and turned themselves off.

Multi-million-dollar, large, datacenter and colocation facility.
Totally quiet.

We walked around turning all the power switches off, and in 5 minutes the
power came back on.  Spent the rest of the day bringing systems back up.

Total 15 minute outage, and I heard it cost the company about $1M (including
the free-month-of-colocation we gave to all the colo customers).

The intern in question?  Turns out a door got stuck open and a "intrusion
alarm" went off - she started hitting every button by the door trying to
turn it off.  Sees big red button. *poof*.

A couple days after this. everyone in the office was required to go to
at least TWO sessions where they explained the purpose of the button (we
knew, of course) and when to push it and NOT push it.  A week later, the
"break glass to push button" covers appeared as well, and safety pins on
the not-halon fire extinguishing system.  Everyone was required to sign
a statement that they'd been to the class, and that they understood that
setting off the halon (er, whatever it is) system without good cause or
hitting the BRS without reason would result in their immediate escort from
the premises and they'd turn in their badge.

The poor girl who did it?  Since she didnt KNOW what the button was for
when she pushed it, she didnt get punished - altho I'm sure she felt like
shit for a week or two.  I think the kidding from her coworkers was enough.

Bill

-- 
Bill Bradford
KD5LQR - Austin, TX
mrbill at sunhelp.org
mrbill at mrbill.net