[rescue] Trade?

[Skeezics Boondoggle]

On Fri, 10 Oct 2003, Bill Bradford wrote:

> One power cord per enclosure (it was 10 or 12 modems if I remember
> correctly), one phone cord per modem.  (these was the pre-lines-on-PRI
> days).  The only thing you saw in the front of the enclosure was the red
> LEDs on the front edge of the Courier PCBs.

Yeah, we chose the Microcoms at the time because they could do 16 modems
per chassis, and we'd stack four of those and two Telebit Netblazers in
each rack.  (The USR TCRs were *insanely* expensive at the time - way, way
more than we could afford, and the big Ascend boxes that took in 672 lines
over a T3 didn't exist yet.)  The first row were some older DEC 4-rail
racks, bolted together.  The rows behind that were taller, cheaper relay
racks, which held more modem chassis.[1]

So with 256 Microcoms facing that window, with the soft feeping green and
red lights, it cast this eery, aurora-like glow into that viewing room.  
Very soothing.  And very cool.

Of course, when it was time to do scheduled downtimes we loved to shut
down the servers, then use the BRS (labeled "History Eraser Button!") to
kill power the the room and cut off the stragglers.  That's when the Angry
Crickets came out - 1100 modem relays clicking off all at once!  Then that
strange silence that creeps out all sysadmins, that sound of a quiet
datacenter...

> You can see some of them in this 1996-vintage picture of me at T.N.  
> they're above and below the Portmaster.
> 
> http://www.mrbill.net/me/older-pics/1996/bill.gif

Cool!  I'm pretty sure I've tracked down the negatives of every picture
taken of me and buried the photographers, their offspring and pets - since
I'll be running for president in '04, need to make sure my past is all
tidied up. :-)  (I figure "who better than a guy named Boondoggle to run
the government!?")  But I digress.

> Imagine nothing but four or five full 72" racks full of these in the
> back of the room, with the lights off.  VERY impressive, but man, so was
> the heat all those Couriers and power supplies put off. 8-)

Hell yeah, those modem racks kicked out the heat.  It was insane; we had
something like 600 square feet and the server farm took up two folding
tables (stacks of SS5s and 20s, disk enclosures on the floor, one Sparc
with a monitor acting as a serial console server) and one rack (Cisco +
CSU/DSUs, NetApp, crappy Xyplex FDDI concentrator, Ethernet switch).  The
entire rest of the room was modem racks and Netblazers.  Most of the A/C
was to cool the modem pool!

> Ahh, the golden days of ISPs.

Yeah, there probably aren't many rooms with 1100 analog modems racked up
like that in one room any more... we joked about having a guy on roller
skates to zip up and down the rows to find and reset stuck modems, but
some clever and insane friend of ours wrote some expect scripts to walk
through the modem pool and diagnose modems that didn't appear to be
answering and reset them.  We had standing orders for 32 new POTS lines
every week for almost six months.  When I left we had something like
1400-1500 phone lines in 10 cities, 4 T1s out to the 'net and 2 more to
tie into the frame cloud for our outlying POPs, and a guy who'd drive
around in an old busted VW van to service things.  Totally a seat of the
pants operation, and a huge amount of fun despite the ridiculous hours,
but crazy as hell.

Those were the days!

-- Chris

[1] Those relay racks weren't bolted down (landlord was fussy); since the
weight was at the back, my ingenious plan was to just put the racks back
to back so they leaned against each other, with a long threaded rod at the
top to connect the them and provide enough space for one skinny admin to
slip between the rows and access plugs and things.  It was Very Nutty(tm).