[SunRescue] exploding sun psus

[Bill Bradford]

On Thu, Dec 07, 2000 at 12:44:24PM -0800, Christopher Byrne wrote:
> Anyway they said that it was having trouble and asked if I could look at it.
> So the fired it up, and we went to the mike to have a test, my friend
> talked, it crackled for a bit then BOOM, the damned thing toasts itself. I
> can believe it was five Farad because the thing was so large(about the size
> of a quaker oats cannister according to the circuit diagram).

Most impressive burnup I've seen was at ioNET, my first ISP job.  (Since
bought by PSI, now part of inter.net... in Oklahoma City).

One night, we're working late, 2-3 of us on staff.  Me, the other head
admin, one tech support guy.  All of a sudden, the building goes dark.  No
problem, say we - we've got UPSes.

Run into the NOC, everything looks fine.  UPS is making funny noises
(you know that "wonk" noise that Sony monitors make when degaussing themselves
 at power-up?  It was making that noise, every five seconds), but all the
equipment is fine.  

After a minute or two, and no power-back-up, we go outside to see if its just
us, or the neighborhood.  One of the guys looks up, and sees a LARGE fireball
on top of the electric pole that feeds our building - the transformer on top
of said pole has shat itself, and is arcing over.  Loudly, and with a rather
pretty arc/fireball, as well.

We go back into the building, call 911 for the fire department (flaming
things were falling from the pole by this time), and proceeded to just YANK
power for non-essential boxes, so we could have enough time left to properly
shut down the really important ones.  Turns out the "wonk" noise was the UPS
reacting to its incoming power line arcing over/surging.

We managed to get everything shut down before the batteries ran out, put a
recorded message on the phone system (which was on its own huge UPS, and
had plenty of power left), and went outside to watch Oklahoma Gas & Electric
replace the transformer.. since there wasnt much we could do to keep the 
equipment running by this time.

2 hours later, they had the transformer replaced, and we were able to spend
the next hour bringing everything back up properly.  It was kinda funny
sending out the "DEFCON 1" page to all on-call personnel and having EVERYONE
show up, literally, in less than five minutes.  "ioNET Emergency Response Team".

UPS still worked after this, but we called out APC the following Monday; they
took a look at the unit, and replaced the entire thing under warranty.  It had
done its job, but used up all of its nine lives in the process. 8-)

(other stories eventually.. such as the time we found out we were exceeding
 the rated capacity of our breaker/fuse box and the line coming into the
 building by 2x.... (we found out when we smelled burning insulation!) and
 the time that somehow, some way, five full racks full of USR Total Control
 dialup equipment got plugged into the wall.. all through ONE big yellow
 extension cord.. (found this out when the cord melted itself)).

Ahh, the early days of the ISP industry (this was '95-'96).

Bill

-- 
Bill Bradford
mrbill at mrbill.net
Austin, TX