[rescue] Sun 711

[Chris Byrne]

> -----Original Message-----
> From: rescue-admin at sunhelp.org [mailto:rescue-admin at sunhelp.org]On
> Behalf Of George Adkins
>
> This one was mpeg video, but it was not compressed.
>

The reason for using compressed files is because they are guaranteed to be
dense. Many other file types incdlue large amounts of empty space or
contigous blocks of identical data that can cause filesystems (not actualy
storage hardware) to change the way a file is dealt with. Others use large
random numbers stacked together, but I find compressed files do jsut as well
and it's easier to make a large compressed file. Just take a few iso images
and zip them together.

>
> this was a 1 gig file, and it took just longer than 10 seconds.  To get 1
> full minute, I's probably have to cat together 8 or 9 gigs of file.  The
> system has 1 gig of RAM and the controller has 32 Megs of cache,
> but I don't
> see how that would falsely affect this test since it's straight read
> performance (no write caching).
>

Mostly this is important for write testing not read testing, but It depends
on what kind of read ahead optimizations your filesystem has and how you do
the testing. Rather than using straight tic tock time some benchmarks use
system timing facilities which measure the time used in processing the file
by the kernel or the filesystem driver etc... Some controllers will do a
huge read ahead into cache and then burst out to the file system which can
be a lot faster than feeding the filesystem little chunks.

Using a file that is much larger than the combination of RAM and cache
minimizes the effects these optimizations have on transfers.

Now this is onl a factor when you are attempting to measure actual spindle
throughput rather than  doing realworld testing of applications. It's always
better to test with the application and data you will actually be using.

For example I had a client a few months ago who was running oracle 8i. He
had several 2-500 gig databases with an average record size of 2k, and a lot
of empty table space (large sparse files). His performance was average, not
great but not bad. About what I would expect considering the data layout and
ram of the systems.

The NT side of the shop started compaling about performance problems so he
decided he was going to do some tests. The way he did this was to cat up
some random junk into a 2 gb file and start copying it and then reading it
back all over the place. His results were extremely inconsistent showing
anywhere from 4MB/s all the way up to 24MB/s, and the time command shoed
times all over the place (the max sustained of the array was 25MB/s per
client)and he took this to mean it was a problem with his storage system.

This was obviously not the case. the problem was with his testing method. It
was inconsistent, sometimes he had load sometimes he didnt. there were
differnt filesystem types involved, differnt spindle loadings, etc... etc...

Anyway we profiled his system using bonnie and IOzone and guess what we
found? For the usage patterns of his applications (large files, small record
sizes) his performance was excellent. Both read and write performance
climbed until it peaked at a file size of just under 256mb and a record size
of 32 mb and then plateaud until right around a gigabyte, then started going
down from there. The reason? the array had 256mb of cache, and the machine
had 1 gb of ram. Once you exceeded that combined total you were going on
spindle throughput alone which was pretty poor because they had layed out
their data and filesystems badly. There was a lot of controller contention
and a lot of spindle contention going on.

We redesigned the file systems, and rearranged the controllers to minimize
contention issues. Their application performance got a little bit better,
because it was OK in the first place, and their benchmarking got quite a bit
better because of the reduced contention.

Oh and the NT performance problems? It turns out that security had installed
new virus scanning software on every NT system (clients and servers) that
was scanning every file on every access. And they wonder why things slowed
down...


> would not be
> using IOZone (Don't do Winders...).
>

IOzone is a UNIX utility (though it will compile and run properly on windows
if you have cygwin). IOMeter is a similar tool for windows.

Chris Byrne