[geeks] my head just went explodey

[Jonathan C. Patschke]

On Sun, 3 Apr 2005, Charles Shannon Hendrix wrote:

> I'm fairly certain I can boast that I have seen the worst written, most
> bloated WWW application in world history.
>
> Or has someone else seen something worse?

Dude.  You have No Fracking Idea.

$ork is a government agency that handles medical data.  Through
judicious use of Google, you could figure out who they are, but, for
obvious reasons, I'm not going to mention their name here.

$ork is staring its death in the face because the legislature thinks we
do a pretty awful job at what we do, and has the numbers to justify
that opinion (I have no opinion, as I'm just a sysadmin and have no idea
of how any of that medical insurance stuff works).  In a last-ditch
effort to save our pork-barreling hides, the "business process
improvement" folks decided to replace all our paperwork with ePaperwork!
On our runs-like-a-champ mainframe?  Oh, hell no, it has to be a eWeb
Application for the New iMillennium.net!

When we delivered the first phase of the application to the public, we
had[0]:

    * Two IBM quad-processor RS64-IV database servers running Oracle.
    * Two IBM dual-processor POWER4  application servers running
      WebSphere.
    * Two IBM uniprocessor POWER3-II web servers running Apache.
    * A couple of Dells that generate and print PDFs for mailing.

And all, that, with 8GB of memory on the DB servers, 4GB on the app
servers, and 2GB on the web servers (each, mind you) would suppport...

    Four concurrent users.

Any more and you'd see unbearable lag, to the point of reaching the
application's session timeout.

Why?  Well, you see, with the advent of "Rapid Application Development",
it was pontificated from those on high that no one working on the
project would need to learn any programming language[1].  So, there were
pointy-clicky tools for crapping out Java and HTML code, pointy-clicky
tools for oozing ad hoc SQL queries for pasting into aforementioned Java-
crapper, and other pointy-clicky tools for doing schema design for
accepting the queries from the aforementioned query-oozer.

Now, this application has fairly simple goals.  It models insurance
coverage and claims.  When the first phase was delivered, all it did was
manage the list of doctors who participated in a government program;
that's -IT-.  A well-trained Perl monkey could make something like that
in a week or two.  With PHP it'd be trivial.

With this "Rapid Application Development" stuff, it took two years,
millions of dollars of hardware, millions of dollars of software, and
the result fell flat on its face.

How?  Well, when the pointy-headed intellectual non-programmers used
their pointy-clicky tools to generate the DB schema, they created
HUNDREDS of tables.  A full UML of the schema in a readable font size
consumes SIXTY US letter-sized pages.  Across those tables, some of
which have hundreds of thousands of rows, you have six-dimensional and
seven-dimensional FULL JOINS (as generated by the pointy-clicky SQL
tool) which are squirted via JDBC ad-hoc (as generated by the
pointy-clicky Java tool).  You just Can't Do That with any system of
reasonable size and expect reasonable performance.

Also, there was the requirement that there be no "system-specific" code.
They -rewrote- lpr, lpd, zip, unzip, cron, and numerous other utilities
in a smattering of TCL, Java, Perl, and Python.  God only knows how they
managed to crap them out while pretending to know now programming
languages.  Actually, having seen the code therein, it's not much of a
mystery; one could write that and still claim ignorance.

Through a -lot- of tuning[2], the real DBA they hired and I got the
application so that it'd actually run under realistic loads while only
bringing the system to its knees instead of outright crashing it.

And if we're not to the "head explody" point yet, IBM purchased the
conslutting group that was responsible for this trainwreck and put THIS
on their website touting how lovely their "solution" was:

   http://www-106.ibm.com/developerworks/rational/library/2282.html

Still not sick enough yet?

The project manager was brought back to deliver "Phase 2" which is, if
you can believe it, even MORE wasteful and bloated and bletcherous to
the point that it makes me physically ill.

"Phase 2" is when the thing goes fully public, not just to-
participating-doctors public.  I fully expect to find a pile of smoking
slag where my servers were before the day folks actually start using it.



[0] It should be noted that the DB servers now have 32GB of memory each,
     the app servers were augmented with two dual-processor POWER4+
     systems, and a Large POWER5 system was added to the mix.  No, it
     still isn't fast, and, no, the load average almost never drops below
     10.
[1] For the purposes of this farce, HTML is considered a programming
     language.
[2] A sizable amount of which involved killing their rewritten Unix
     tools and wrapping the real ones in Very Small Shell Scripts(tm)
     to translate the arguments and flags.
-- 
Jonathan Patschke  )   "Once again, `digital' is proving to be a
Elgin, TX         (     synonym for `crap'."
USA                )                    --Andrew Orlowski on DRM