[geeks] Re: [rescue] clustered file storage

[Jonathan Katz]

Josh noted:
> So, let me see if I foloow you.  One company wants to be able to increase
> storage and CPU power to Oracle (or what ever else).  So, they have oracle
> using data stored on a file server instead of locally, and they have
> oracle running clustered.  Thus they can add more disks to the file
> server, or they can drop another box on the network and add it to the
> oracle cluster.

That happens. Big sites will want to build horizontical and vertical
scability into their environments at as many levels as possible.

Horizontal scalability usually means the growth of an existing
service in order to distribute load. This would mean a second firewall
or second webserver. Horizontal scalability can be incorrectly
summed up as saying that the growth which is required to service
demand is met by adding systems to perform the same service.

Vertical scalability is the ability to migrate services and grow them
independantly of one another. By having a webserver and a dbserver
on the same physical system you can't really grow one independantly
of the other without moving a specific service to a different system.
Veritcal scalability can be (semi-incorrectly) summed up by saying
that the growth which occurs is within a system to augment an
existing service.

The tradeoffs are the more scalable a setup becomes the more difficult
it is to maintain, as there are more "moving parts." However, if an
environment isn't scalable enough necessary upgrades may exceed any
possible budget (we know how tight IT budgets are) and manpower
available. For ease I'm going to abuse the webserver analogy for an
explanation. You can replace webserver with application server or
"whatever."

Initially, webservers and associated hardware were simple (circa 1995)
<pre>

{internet} - [webserver/dbserver w/ packet-filter]=[direct attach storage]

</pre>

This is completely non-scalable. In order to increase capacity the entire
setup would need to go off-line. Everything on one-host is a SPOF. With
a lack of segregation between layers (public internet, db server, webserver)
security issues arise as well as issues in troubleshooting.

Let's dissect a typical early (circa 1997) webfarm:
<pre>

                                  [ webserver 1 ] 
{internet} - [ router w/ ACLs ] <[ webserver 2 ] --- [ dbserver ]-[direct attach storage]
                                  [ webserver 3 ]

</pre>

Traffic would be balanced between webservers using something like
round-robin DNS. A single network connection would be a single point
of failure  (SPOF.) Adding storage to the dbserver would mean shutting
down the server, adding another host adapter, adding an array, etc.
That would mean downtime. Filesystem space on the dbserver isn't
readily scalable. In this model the one thing which is easily scalable
are the webservers. You can easily add more, drop the new IPs in the
DNS for round-robin and your load is further distributed.

Today, if I were asked (and had a big budget!) to build a webfarm, it would look like:
<pre>
                                  [ webserver 1 ]                                      [ $STORAGE ]
{internet} - [ router w/ ACLs ]   [ webserver 2 ] - - [ dbserver ] [Brocade or Vicom] < 
               |BGP|            X [ webserver 3 ]  X    |OPS?|    X    { SAN }         X
{internet} - [ router w/ ACLs ]   [ webserver 4 ] - - [ dbserver ] [Brocade or Vicom] <
                                  [ webserver 5 ]                                      [ $STORAGE ]
</pre>

$STORAGE can be Sun T3s, EMC, Hitachi, or combinations thereof.

This is a completely scalable and high availabilty (HA) setup.
o Multiple internet connections allow for redundancy and load balancing.
o Multiple webservers are loadbalanced by webserver software (iPlanet 4.x does this) If the webserver software doesn't, imagine a few F5s or Alteons in front of the webservers.
o Multiple DB servers ensure the DB is always up and helps to distribute the load. If the load gets to be too much another DB server can be plugged in.
o With the SAN multiple switches can break out into as many storage farms as are needed. I have two here, as one is a mirror of the other (full data availability on at least one of the SANs)

The same can be said for internal SAP-like servers, too. Instead
of a router doing BGP multiple connections are made to different
points in the corporate network. Instead of a webserver SAP
application (I think they call this a broker?) would be at
that second layer. The DBserver layer remains the same.

Now the cool thing is through the miracle of multiple domains
I don't even need a seperate system for this-- I can have one
huge frame (E10K, 6800) splice it up and have an entire farm
in one rack!

Hopefully this helps sum things up. Maybe this should be
migrated to geeks@ (as I'm CC'ing there?)

-Jon