[rescue] Thoughts on Suits, unemployment, and 3-year-old hardware

[Skeezics Boondoggle]

Begging the Rescuers' forgiveness for a wee bit of indulgence; it's a
lousy day, and I need to vent a bit. :-/  And, it's semi-topical. :-)

This morning I had to push the DNS updates to take my company's web-based
product offline.  (Probably not supposed name names since there are suits
involved, and litigation pending, and this is a public archive...)

Last Monday was the 4-year anniversary of my hiring, as employee #9, at my
current company.  Longest I've ever worked at once place in an
uninterrupted stretch.  And easily the most fun *and* the most challenge
I've had at any job, ever.  I took this gig mainly because I got the
chance to start with a completely clean slate - the requirements were:  
"We've settled on Solaris and Oracle.  All the other details are your
concern.  And it has to scale."

Our first product was technically a smashing success - but didn't generate
enough income to sustain us.  After a couple of years of slogging it out,
we made the difficult decision to sell it off, shrink the company down to
just an engineering core (8 of us - and NO suits!) and develop something
completely new and have someone else slap their name on it and do the
sales/marketing.

Just over one year ago, after a breakneck development cycle and a complete
reshuffling of our existing hardware (very little budget for upgrades or
expansion) we renamed and reinvented ourselves and launched the new
product - on-time.  "80 hours a week and loving it!"

At our peak, we've been handling 12 billion HTTP hits/month on four Sun
420Rs behind a Cisco LD417 load balancer.  (These weren't static hits,
either; almost all dynamic content.)  During "peak times" (about 16 hours
a day, with a 2-3 hour lull and very steep ramp-up/ramp-down - you'd
boggle at our Cricket graphs) we'd sustain anywhere from 5000-8000
connections per second - often higher - and if our latency ever exceeded
200ms I'd get paged about it.  Our aggregate external bandwidth was
typically over 70 megabits/sec (30 in, 40 out), with even higher bursts.

You could think of it as "design a system to withstand a denial-of-service
attack that we're going to inflict on ourselves." :-)

Processing the data from the web farm in near-real-time were four more
420Rs, an E4500, and two NetApps.  We figured our overall end-to-end
application-level utilization was less than 40% - so on THREE-YEAR-OLD
HARDWARE, with no money in the budget for upgrades, we were already
handling twice the traffic we figured on initially, and could likely have
doubled it again, since our software guys and my highly tuned Solaris
boxes were exceeding our initial expectations by such wide margins.  The
Ciscos would have topped out before our processing pipeline did.

(So, when someone says "What can you do with 400Mhz?!" just send them my
way...)

Now I have to shut it all down.  The tide finally came in on our little 
dot-com sandcastle.

I'm going to spend April selling off the company assets, drinking heavily,
sitting on the couch for hours staring at the walls, screaming a lot,
sleeping in, wondering how my children are suddenly taller than I am, and
thinking about what I'm going to do next.  I've been out of circulation so
long, it seems there are painfully few Solaris gigs around here these
days, and certainly none paying anywhere *close* to what the SANS salary
surveys say a SAGE Level 3+ geek with 20 years experience and my skill set
in the Pacific NW region is supposedly making.  $97,500 _average_?  Yeah,
RIGHT, maybe if I sold crack with every Jumpstart install.

Bleah.

So, I've got some great Sun/NetApp/Cisco hardware for sale, and some Dells
(that I admit, do run Solaris pretty reliably).  Perhaps the handful of
Rescuers in the Portland/Seattle area might be interested?  I'm trying to
find local buyers first - local pickups highly preferable to dealing with
Ebay and shipping/billing hassles.  At some point, though, I may have to
post stuff on Ebay, and I'd be happy to have the list get a look at it
before the unwashed masses (and clueless lowballing liquidators).

The Bloody Marys from breakfast have worn off.  Ah!  One pint of Duvel
still stashed in the desk drawer!  "Alc. 8,5% by vol."  Right on!

Cheers...