Patch Name: PHKL_15729 Patch Description: s700 10.10 UFS, VxFS, buffer cache cumulative patch Creation Date: 98/06/24 Post Date: 98/06/26 Hardware Platforms - OS Releases: s700: 10.10 Products: N/A Filesets: OS-Core.CORE-KRN Automatic Reboot?: Yes Status: General Release Critical: Yes PHKL_15729: HANG PHKL_15499: HANG PANIC PHKL_10286: PANIC PHKL_8325: CORRUPTION PHKL_8175: HANG PHKL_7795: HANG Path Name: /hp-ux_patches/s700/10.X/PHKL_15729 Symptoms: PHKL_15729: Under heavy I/O, after the buffer cache allocation has reached its system defined maximum, the system eventually hangs when all I/O processes sleep on waiting for buffer cache. PHKL_15499: This patch fixes two defects : - SR: 1653251934 DTS: JAGaa01592 The System hangs under heavy I/O involving VxFS type of filesystems. When the buffer cache virtual space is heavily fragmented and a readhead is being done, system hangs. - SR: 4701375956 DTS: DSDe441059 In a mixed UFS/VxFS environment, the system panics with a dirty invalid buffer which was associated with a UFS filesystem that has since been unmounted. The typical stack trace of the panic looks like the following: panic+0x14 brelse+0x1ec getnewbuf+0x6b0 ogetblk+0x104 getblk1+0x258 vx_getblk+0x5c PHKL_10286: Panic trap 15 in bwrite() under heavy disk I/O stress. PHKL_8325: Data loss with UFS files using fragments. PHKL_8175: K-Series 2-way stopped processing I/O's on a certain VxFS. PHKL_7795: Heavily used Vxfs with snapshot hangs. Defect Description: PHKL_15729: When buffer cache allocation reaches the defined maximum, the allocation of new buffers from system memory stops. When buffers are freed, their physical memory spaces, instead of being returned to the system, are saved in a pool to be reused for new buffers. A bug in the getnewbuf() routine disallows the reclaiming of these available memories when bufpages reaches the ceiling. Furthermore, another bug in the routine that frees up space for the buffer cache resource map erroneously releases more buffers than necessary. The prohibition of reusing the memory from the reserved pool and the loss of free buffers results in running out of buffers in the freelist, causing all I/O processes hung in sleep wait. PHKL_15499: This patch fixes two defects : -SR: 1653251934 DTS: JAGaa01592 An Uniprocessor system hangs when heavy I/O is performed. When the buffer cache virtual space is heavily fragmented and we are doing a readahead, system hangs. The allocbuf1() won't do a bcfeeding_frenzy() to de-frag because readahead sets BX_NONBLOCK|BX_NOBUFWAIT flags. Instead it returns an error to ogetblk(). A bug there keeps us looping instead of returning to the original caller. Reproduce: On a UP K-series system with 64Mb, create a 400 Mb VxFS file system mounted at /new_fs. Make sure that /usr is also of type VxFS type. The following code will produce the hang. while true do rm -rf /new_fs/* cp -r /usr/* /new_fs done & while true do date sleep 10 done & The system will hang in about 15 minutes. - SR: 4701375956 DTS: DSDe441059 Essentially, in getnewbuf, we might set a B_DELWRI buffer to B_BUSY, but later decide to NOT write it out (in fixing some deadlock problem). Rather, we will simply brelse it. If a umount of the device or filesystem comes in between the time of setting B_BUSY and brelse(), it will skip the buffer thinking that someone else is flushing it out. Later when it calls binval() assuming all buffers should have been written out, it may invalidate a dirty buffer. PHKL_10286: A buffer arrives in bwrite() with B_ASYNC/B_DELWRI and B_INVAL on and bp-vp == 0 (buffer disowned). On attempting to complete the write, VOP_STRATEGY resolution results in a trap 15 due to null vp. PHKL_8325: There was a code path where dirty buffers could possibly be dropped (not flushed) when extending UFS files using fragments. PHKL_8175: Currently, biodone() sets B_DONE and drops the spinlock for the buffer before making a callback to an I/O completion routine, if one has been set up. This means that biowait() can return while the callback is in progress, and the buffer might be released. The solution is to always set B_DONE in finish_biodone() instead of biodone(). This fix has been implemented in PHKL_7957 (700) and PHKL_7958 (800). implemented in PHKL_8175 (700 10.10) PHKL_8176 (800 10.10) PHKL_7795: When Vxfs calls getnewbuf() to acquire a buffer, it passes VX_NONBLOCK in bxflags to avoid potential deadlock. However, getnewbuf(), when finding a B_DELWRI buffer, proceeds to call bwrite() to flush the buffer without checking the bxflags. This causes deadlock in the following scenario: 1. getnewbuf() tries to flush a buffer belongs to Vxfs. The Vxfs strategy layer finds the buffer involves in an uncommitted transaction and decides to flush the log buffer first. 2. This file system has a snapshot and the region of the log we are about to overwrite happens to change for the first time. So the snapshot strategy layer needs to copy the old data from primary disk to the snapshot before flushing the log. In order to do so, it asks for another buffer. 3. getnewfs() is called again and yet another Vxfs dirty buffer needs to be flushed. 4. Vxfs strategy layer decides the current log buffer must be flushed before any dirty buffer. Since the current log buffer is locked, it sleeps and waits, for a lock that is owned by itself. The fix is that if a B_DELWRI buffer with VX_NONBLOCK is chosen, getnewbuf() will return NULL instead of flushing and returning the buffer. SR: 1653166496 1653173922 1653251934 4701327338 4701333419 4701348359 4701375956 5003407619 Patch Files: /usr/conf/lib/libhp-ux.a(ufs_mchdep.o) /usr/conf/lib/libhp-ux.a(vfs_bio.o) what(1) Output: /usr/conf/lib/libhp-ux.a(ufs_mchdep.o): ufs_mchdep.c $Date: 98/06/24 16:52:23 $ $Revision: 1.14.89.15 $ PATCH_10.10 (PHKL_15729) /usr/conf/lib/libhp-ux.a(vfs_bio.o): vfs_bio.c $Date: 98/06/24 16:51:33 $ $Revision: 1.2 1.89.21 $ PATCH_10.10 (PHKL_15729) cksum(1) Output: 3257756306 10880 /usr/conf/lib/libhp-ux.a(ufs_mchdep.o) 1199100366 28412 /usr/conf/lib/libhp-ux.a(vfs_bio.o) Patch Conflicts: None Patch Dependencies: None Hardware Dependencies: None Other Dependencies: None Supersedes: PHKL_7795 PHKL_8175 PHKL_8325 PHKL_10286 PHKL_15499 Equivalent Patches: PHKL_15730: s800: 10.10 Patch Package Size: 100 KBytes Installation Instructions: Please review all instructions and the Hewlett-Packard SupportLine User Guide or your Hewlett-Packard support terms and conditions for precautions, scope of license, restrictions, and, limitation of liability and warranties, before installing this patch. ------------------------------------------------------------ 1. Back up your system before installing a patch. 2. Login as root. 3. Copy the patch to the /tmp directory. 4. Move to the /tmp directory and unshar the patch: cd /tmp sh PHKL_15729 5a. For a standalone system, run swinstall to install the patch: swinstall -x autoreboot=true -x match_target=true \ -s /tmp/PHKL_15729.depot 5b. For a homogeneous NFS Diskless cluster run swcluster on the server to install the patch on the server and the clients: swcluster -i -b This will invoke swcluster in the interactive mode and force all clients to be shut down. WARNING: All cluster clients must be shut down prior to the patch installation. Installing the patch while the clients are booted is unsupported and can lead to serious problems. The swcluster command will invoke an swinstall session in which you must specify: alternate root path - default is /export/shared_root/OS_700 source depot path - /tmp/PHKL_15729.depot To complete the installation, select the patch by choosing "Actions -> Match What Target Has" and then "Actions -> Install" from the Menubar. 5c. For a heterogeneous NFS Diskless cluster: - run swinstall on the server as in step 5a to install the patch on the cluster server. - run swcluster on the server as in step 5b to install the patch on the cluster clients. By default swinstall will archive the original software in /var/adm/sw/patch/PHKL_15729. If you do not wish to retain a copy of the original software, you can create an empty file named /var/adm/sw/patch/PATCH_NOSAVE. Warning: If this file exists when a patch is installed, the patch cannot be deinstalled. Please be careful when using this feature. It is recommended that you move the PHKL_15729.text file to /var/adm/sw/patch for future reference. To put this patch on a magnetic tape and install from the tape drive, use the command: dd if=/tmp/PHKL_15729.depot of=/dev/rmt/0m bs=2k Special Installation Instructions: None