[geeks] The Idea(tm)... Homebrew Terabytes Cheap

[Jody Stephens]

On Fri, Apr 13, 2001 at 02:13:27PM -0700, Chris Byrne wrote:
> Jody,
> 
> First of all you work for tattered cover? Kick ass, that is the coolset book
> store. I always go every time I'm in Denver, even though the parking is a
> pain (or at least it was last time I was there about a year ago). I think we
> had to park in the safeway down the street and walk over.

Groovy, you might have missed the parking garage attached to the store (but
maybe it was full). So yes I do, it is a labor o' love (I think people at
the burger king across the street make more than I do). 

> 
> Anyway, managing a huge amount of storage isn't as simple as it seems. While
> I admit they cost far more than they should, there really is a reason that
> large scale storage arrays cost mucho bucks. The biggest problems here are
> bus bandwidth and filesystem management.
> 
> There's just no way you can split up the limited badwidth of a. the system
> bus and b. the drive controllers over that many drives efficiently. I've
> tried sticking 16 drives on a single general purpose system and have it max
> out at less than 5% of the sustained throughput capabilites of each drive.

Well the way I am looking at it, I'll be writing ~8GB sequentially once a night, 
assuming I were to stripe it correctly I don't think that is going to kill 
the system. And even if it was terribly slow, so what? It will still probably
be faster than tape. This isn't for streaming video or anything.

> 
> And that assumes your bus functions properly with that many devices attached
> to it, and that you don't exceed your cable length limit. I've never tried
> putting that many devices on a single bus before, but my gut feeling is that
> 90+ devices (80 drives, 10 plexes, plus all the SCSI cards and any other
> ancillary cards) isn't going to work.

Haven't gotten this far yet. But I think I would divide the drives between
several busses. I imagine that it would be possible to arrange each logical
array to go over multiple busses that way I wouldn't be killing the plex.

> 
> Also you are gonna need a HONKING power supply in each box. Up until
> recently I would have said that'd be a problem, but enermax just came out
> with a truly kick ass 650 watt PS with two fans and 16 power taps.

Haven't gotten this far either, but it seems I shouldn't have to many problems
(heck I got a 3/160 chassis with a 900 watt PS for $5, but anyways). And 
these things are not 15KRPM scsi drives, they (the Western digitals I'm looking
at) are 5400RPM ide drives.

> 
> The other issue is filesystem/volume management (actually add scheduling of
> backups to maximize performance to that as well)
> That's mostly an administrative issue on the order of "do I want to take the
> admin bandwidth to handle volume and backup management manually, or do I
> want to buy expensive volume management software ala veritas, or do I want
> to hack an opensource tool to work in this somewhat exceptional
> environment".

Yeah this is really the kicker. But I don't have it to hard, as my requirements
are not at all complex (basically big array, bunch of directories with 
dates and machine names). I'm probably living a fantasy anways because what
this thing is really for is backup and I don't think I could risk the stores
data on some hare-brained scheme such as this. I just really really would
like to find some sort of backup media that is not so frikken exspensive and
slow and fraught with failure (~10% failure rate on our tapes, and then some
months >%30 (I guess IBM doesn't make a good tape drive). It seems like it is
cheap, fast, big, choose 2. 

> 
> I went through this last year with StorageWay, and we came to the conclusion
> that with all of the factors weighd in, it's actually cheaper to buy a
> serious storage array than to try and build and maintain one yourself.

Probably for you, but I make chicken feed :) and, I'm sorry, but the bonus 
points for building your own >TB range array is just insane... 

> 
> Now if they woyuld jsut make one that didnt cost $250k
> 
> Actually the good news is, you can often find mid range storage arrays on
> EBay for pretty reasonable prices. Add a veritas license, and a machine to
> control the thing, and for the same $20k you were talking about you can have
> a hitachi or EMC (probably a 1 TB array rather than a 3 TB)

Well I might look at that, but we don't really need more than a few hundred GBs
except for archival purposes. And then we would have to get a maintence agreement
as there is no way I could maintain something like that. 

> 
> 
> Chris Byrne
> 
> 
[snip]
Thanks for the thoughts though, and if you are ever in denver again drop me a
note, I get a good discount (one of the only benefits of working here). 

-- 
                Jody Stephens -- (303)322-1965 x2608
                      jodys at tatteredcover.com
                     Tattered Cover Book Store
                          1628 16th Street
                         Denver, Co. 80202
                       (303)322-1965 Offices
(303)436-1070 LoDo Store         (303)629-1704 Office and LoDo Fax
(303)322-7727 Cherry Creek Store (303)399-2279 Cherry Creek Fax