[SunHELP] I/O size tuning : VxVM/EMC Symmetrix/FC-AL/Solaris/Oracle

[DAUBIGNE Sebastien - BOR ( SDaubigne@bordeaux-bersol.sema.slb.com )]

Our customer has a DSS (OLAP) running on a E10K/Solaris 2.6/VxVM3.2/VxFS3.4
/ Oracle RDBMS 8.1.7, with 4 FC-AL JNI cards to an EMC Symmetrix 8730 array.
The EMC disks are internally stripped (RAID 0+1), with meta-volumes of 4 HVE
(stripe-unit=960Kb, full-stripe = 4x960Kb).

After reading many VxVM/Oracle/Sun/JNI documents (which concludes that 1 Mb
I/O is optimal for sequential I/O on modern disks because 1 I/O of 1 Mb is
faster than 8 I/O of 128Kb due to average positioning time), we were
convinced that sequential I/O should issue 1 Mb I/O to the Symmetrix to get
optimal performances, so we told our customer to tune the following Oracle /
kernel parameters :

-	Solaris maxphys to 1 Mb
-	VxVM vol_maxio to 1 Mb
-	Oracle db_multiblock_read_count (for seq I/O) to 1 Mb

Hence, Oracle sequential I/O (full-scan) are issued with a size of 1 Mb
instead of 128 Kb (due to maxphys default), which is what Oracle/Sun/VxVM
tell to do in many tuning papers.

But now, EMC Professionnal Services made some analysis on the Symmetrix
(with EMC Worlload Analyzer) and said that *average* I/O size are too high
(200 Kb with 800 Kb peaks, which is less than 1 Mb because it is an average
with mixed sequential (1Mb) and random (16 Kb) Oracle I/O), because it leads
to large Front End (FA) queues (30 to 75 ms), with busy devices and many
locks in the array cache memory.
So EMC advice is to remove kernel parameters tuning to reduce I/O to the 128
Kb Solaris default.

We don't agree with EMC and think that large queues are an expected
consequence of large I/O size which is the best size for seq I/O. Reducing
I/O size will reduce average wait queue BUT will increase number of I/O,
with poor sequential throughput. 

I feel like EMC's conclusions are based on "old school" analysis (old disks,
small I/O) and/or OLTP load.
 
Could someone share any similar experiences, and give advice ?

---
Sebastien DAUBIGNE 
sdaubigne at bordeaux-bersol.sema.slb.com
<mailto:sdaubigne at bordeaux-bersol.sema.slb.com>  - (+33)5.57.26.56.36
SchlumbergerSema - SGS/DWH/Pessac