1    Release Notes for Visual Threads Version 2.2

1.1    User Interface Changes from Previous Release

The following user interface changes have been made in Visual Threads Version 2.2 relative to Visual Threads Version 2.1:

1.2    Operating System Requirement

Visual Threads Version 2.2 runs on Compaq Tru64 UNIX Version 4.0D and higher.

For information about Visual Threads and older versions of Tru64 UNIX, see Section 1.5.

1.3    VTI Configuration

1.3.1    VTI Interactions with the Pthreads Library

The Visual Threads Instrumentation tool (vti) is configured at installation time to work with a particular pthreads library.

If you upgrade the version of pthreads, or are testing another version of pthreads, the installed version of vti may not work correctly. In particular, vti is sensitive to the boundary version number "3.16" for pthreads. If vti is configured for pthreads 3.15 or earlier, it cannot be used with pthreads 3.16 without reconfiguring vti. You can determine the version number of the threads library by doing the following command:

/bin/what /usr/shlib/libpthread.so | grep V
DECthreads version V3.15-421 Jul 20 1999

If the pthreads library being used is not in /usr/shlib, substitute the correct location in the above command to determine the version. Note that version 3.16 of the pthreads library is the version that ships automatically with Compaq Tru64 UNIX Version 5.0A.

To avoid problems with vti, you must reinstall Visual Threads whenever the pthreads library version changes.

1.3.2    VTI and Inline Locks

With the most recent versions of the libpthreads library (versions 3.16 and above, shipping with Tru64 UNIX Version 5.0A), the Visual Threads Instrumentation tool (vti) will not work with programs that have been compiled using pthread inline locks. That is, previous versions of the pthreads library supported programs being compiled with the preprocessor symbol _PTHREAD_USE_INLINE to provide better performance (although users were warned in the header file that they might have to recompile in the future if they used this feature). With vti configured to work properly with pthreads library 3.16 (and above), vti will give an error similar to this if it encounters an inline lock:

Error: inline lock found at 0x12000185c, but current pthreads library
does not support inline locks

If you get this error, you must recompile your application without the use of _PTHREAD_USE_INLINE. VTI cannot detect inline locks when using versions of libpthreads equal to or higher than 3.16.

1.4    Running Visual Threads After Installation

You can run Visual Threads in several ways:

1.4.1    Possible Problems Preventing Visual Threads Startup

If you are unable to run Visual Threads, possible problems preventing startup are the following:

1.4.2    Using the Online Help

All the documentation for Visual Threads is contained in the online Help. (There is also a manpage called dxthreads and two related manpages called vti and vttrace.)

Before you start using Visual Threads, you should choose Help: Contents from the Control Panel. Then read all the topics in the online book "Getting Started" for a quick introduction that will save you time in learning Visual Threads.

In particular, you might save yourself some time if you read these important help topics:

1.5    Information About Tru64 UNIX Versions 4.0D, 4.0E and 4.0F

1.5.1    Support for libpthreads Libraries for Versions 4,0D, 4.0E and 4.0F

In order to analyze threaded applications, Visual Threads requires a debugging-enhanced version of several system libraries (libpthreads.so, libpthreadsdebug.so, libpthread.so, libcmalib.so, libc.so, and libexc.so). These enhanced libraries are included in Patch Kit 4 (or later) for Tru64 UNIX Version 4.0D, Patch Kit 2 (or later) for Tru64 UNIX Version 4.0E, and Patch Kit 4 (or later) for Tru64 UNIX Version 4.0F. You must install one of these patch kits before installing Visual Threads. Or, you can upgrade to Tru64 UNIX Version 5.0 or above.

If your system does not include the enhanced libraries, you will receive the following error message at installation time for V4.0D and V4.0E systems:

Error: the threads libraries on this system do not include support
for enhanced debugging.  You must install a recent patch kit or
upgrade to a newer version of the operating system (see the Visual
Threads Release Notes for more details).

On V4.0F systems, you do not receive an installation error message, but you get an unresolved symbol message when you invoke Visual Threads.

1.6    Instrumentation Issues

1.6.1    Visual Threads Instrumentation Tool

Some rules (in particular, the Data Protection rules and most of the Events rules) require your application to be instrumented before you analyze it. Visual Threads uses the Visual Threads Instrumentation tool (VTI), an Atom-based tool, to perform this instrumentation. VTI is invoked when you enable a rule requiring instrumentation, or when you enable the "NUMA data locality analysis" check box in the Run Dialog box.

You can also run VTI as a standalone process from the command line. For more information about VTI, see the vti manpage after installation.

1.6.2    Atom Software for Tru64 UNIX Versions 4.0D and 4.0E

VTI works best with versions of Atom that fully support the -gp, -Xnodisable, and -Xfork options. During installation and during instrumentation, warning messages are generated if the version of Atom doesn't support all of these options. You can still use VTI, but with these limitations:

If you receive any of these installation or instrumentation warnings, you should consider getting a newer version of Atom with the Compaq Developers' Toolkit Supplement for Compaq Tru64 UNIX.

1.6.3    Atom Exits with Insufficient Disk Space

If Atom doesn't have enough disk space to create instrumented files, it exits without giving an error status. In this situation, Visual Threads displays an error during instrumentation with no further information. In order to instrument your images, make sure that you have at least 10 MB of space in /tmp or in the directory path pointed to by the TMPDIR environment variable.

1.6.4    Debug Limitations with an Instrumented Application

When Visual Threads instruments your application, it specifies an Atom option to include application-level debugger support (-gp option or equivalent) in the resulting binary. However, even though this support is included, sometimes the application code and variables will not be visible when you debug the instrumented program. This is because of the stack changes that occur in the interface between the application code and the Atom-inserted analysis code.

Consider the following scenario:

stopped at [ __VisualThreadsAlarm(...) 0x3ffbfff34d4] (ladebug) where
>0  0x3ffbfff34d4 in __VisualThreadsAlarm(0x38001082318, 0x3800108234f,
0x38001082318, 0x3800108234f, 0x380010859c0, 0x30)
"DebugInformationStrippedFromFile0"
#1  0x3ffbfff3818 in __VisualThreadsSync(0x38001082318, 0x3800108234f,
0x38001082318, 0x3800108234f, 0x380010859c0, 0x30) in
/usr/opt/VisualThreads/shlib/libpthreadevent_vt.so
#2  0x3ffbfff4a00 in vt_event_write(0x38001082318, 0x3800108234f,
0x38001082318, 0x3800108234f, 0x380010859c0, 0x30) in
/usr/opt/VisualThreads/shlib/libpthreadevent_vt.so
#3  0x12001a260 in /tmp_mnt/var/users/sharris/ins1/test1.vti

Information:  An  type was presented during execution of the
previous command.  For complete type information on this symbol,
recompilation of the program will be necessary.  Consult the compiler
man pages for details on producing full symbol table information
using the -g (and -gall for cxx) flags.

The top-level entries in the call stack are in Visual Threads, but the interesting point is at call stack entry #3 in the example above. This represents the Atom analysis routine that detected the violation. This routine was actually invoked as a result of executing some routine in the application, which was instrumented to include a "call" to an Atom analysis routine. But the application routine is not visible to the debugger due to the stack changes at this "call" interface (what should be call stack #4). In general, application-level code and variables are visible only if you do not have an Atom analysis routine on the current call stack.

Although you may not be able to use the debugger to fully view or navigate the stack in this situation, you can use Visual Threads to get the call stack representing the call that generated the alarm. Before invoking the debugger, you can click MB3 on the function name in the table (fourth column) to see the stack; double-clicking on the function name takes you to the source. If you are already in the debugger, the Alarm box is no longer visible. However, you can recall it by choosing View: Events from the Visual Threads Control Panel. In the Events window, click MB3 and select "Event Details" on the alarm event (which should be the last event shown).

1.6.5    "Assertion Failed" Instrumentation Error

Under rare circumstances, Atom might issue an error like the following:

Assertion failed: 0, file
../../../../../../src/usr/ccs/lib/libmld/cmrlc_consume.c, line 1057
atom: Error: Command '/tmp/atomAAAaapzja/phil.tool' terminated with
receipt of SIGABRT signal.
atom: Error: Run atom with the -debug switch to see if the error is in
the instrumentation code.

One possible reason for this error is that the application being instrumented was created on an incompatible version of the operating system. For example, you might be trying to instrument on a Version 4 system an image that was created on a Version 5 system. If this isn't the cause of the error, consult your Atom documentation or submit a problem report to Visual Threads with full information about the version of Atom you were using, the version of Visual Threads, the environment variables, and a copy of the binary being instrumented.

1.7    Must Reinstall Visual Threads If You Upgrade

You must reinstall Visual Threads if you patch or upgrade to a newer version of the following:

1.8    Trace File Issues

1.8.1    Cannot Record and Play a Trace File at the Same Time

Visual Threads cannot record a trace file and play a trace file at the same time. Therefore, if you are playing a trace file and you choose the rule action "Start recording a trace file," Visual Threads will ignore the action.

1.8.2    Trace Files and System Run-Time Library Internals Toggle

If you create a trace file while the Collect events for all system run-time library internals toggle is turned off, and then play the trace file later with the toggle turned on, you might get violation conditions involving internal synchronization objects from system libraries due to partial information being collected. Examples of violation conditions you might see are unlockedCVWait and invalidUnlock. (You will not see these violation conditions if these rules are not enabled.)

1.9    Attach Capability and Restrictions

Visual Threads can enable an ongoing event stream when attaching to a process. You should be aware of the following restrictions:

  1. By default, most rules are suppressed during this processing. This is because it is dificult for Visual Threads to determine a consistent state from which to apply its rule processing. After fully attaching, individual rules can be enabled from the Rule Setup dialog box. In particular, the following rules are suppressed:

    Deadlock Rule: inconsistentOrder

    Programming Errors Rule: unlockedCVWait

    Deadlock Template: lockOrder

    Programming Errors Rule: invalidUnlock

    Deadlock Rule: deadlock

    Programming Errors Rule: relock

    Programming Errors Rule: threadLeak

    Programming Errors Rule: missingUnlock

    Deadlock Rule: priorityInversion

    Programming Errors Rule: potentialPriorityInversion

  2. Like most debuggers, Visual Threads must intercept the execution of the running application to insert its data gathering techniques. Unlike traditional debuggers, these data collection probes cannot be completely reversed when detaching from the process. Memory and other resources that support tracing of the application cannot be completely deallocated. Therefore, Compaq recommends that you use Attach only with non-production applications that can be restarted.

  3. Visual Threads must have knowledge of the specific version of the POSIX Threads Library in order to provide this support, and therefore this capability may not be available on all patch levels of Tru64 UNIX. If a particular version of the POSIX Threads Library is not known to Visual Threads, you will get a warning message and only a snapshot of current thread states will be collected. A snapshot will show you only the state of the application at the time the attach command was executed. It is useful mainly for letting you view objects (for example, by using the Object Browser window, or the Monitor window). You will not receive events from your process, but can re-execute the Attach operation using the Play button to refresh the snapshot over time.

    Contact Visual.Threads@hp.com for information about support for the particular version of the POSIX Threads Library you are using.

  4. Due to a bug in the pthreads library on systems prior to Tru64 UNIX V5.1, pthread_mutex_lock() may incorrectly return a failure status if tracing is dynamically enabled. This can be avoided if the "Collect system events" toggle is turned off.

1.10    CPU Time Statistic May Be Inaccurate for Systems With Mixed CPU Speeds

If you run Visual Threads on a system with multiple CPUs where the CPUs run at different speeds, Visual Threads calculates the statistic "CPU Time" based on a single global CPU speed.

1.11    Extraneous Inconsistent Order Violations Reported on Fork Operations on V5.1A Systems

If you have selected the Data Collection Option "Collect events for all system runtime library internals", and your application performs fork() operations, then on Tru64 UNIX V5.1A systems, you may see about 30 "inconsistent lock order" violations which refer to the popen mutex. These violations can be ignored.

1.12    Source Line Imprecision with Java 1.1.8.*

Visual Threads may not always pinpoint the exact source line when showing source from a Java program using Java 1.1.8. This is due to Java imprecision in symbolization and does not occur with Java 1.1.7 versions or Java 1.2 versions. The problem can be seen when running the example program philjava; when the deadlock alarm fires, the callstacks and source locations for the lock operations may highlight the statement that precedes the actual statement of the lock, which is a "synchronized" statement.

1.13    Key Rundown Causes a Thread to Not Be Listed as Terminated

If a thread defines a key, the user's rundown function (if any) is called after the thread has been reported as terminated. This causes the thread state to go from terminated back to running/blocked/etc. during the processing of the key rundown procedure. After completion of the rundown routine, the thread will be left in the running state until the thread is deleted.

A future release of Tru64 UNIX will correct this problem by invoking the rundown prior to reporting the thread as terminated.

1.14    Debugger Issues

See Section 1.6.4 for debug limitations with an instrumented application.

1.14.1    How Visual Threads Stops the Debugger

If you are running an application not synchronized (that is, with the Synchronize the tool with the application toggle turned off) and you invoke the debugger, Visual Threads starts the debugger, specifying that it should attach to your application. To stop the debugger, Visual Threads sends a SIGINT to the process to cause the debugger to stop.

If you are running an application that is synchronized (that is, with the Synchronize the tool with the application toggle on) and you invoke the debugger, Visual Threads stops the debugger using either of two methods:

1.14.2    Must Exit Ladebug Explicitly

Due to a bug in Ladebug, Ladebug does not terminate when the terminal goes away. Therefore, you need to exit Ladebug explicitly after debugging your application.

1.14.3    Using the TotalView Debugger with Visual Threads

Visual Threads provides built-in support for launching TotalView on a threaded process, and setting up the necessary breakpoint to cause it to stop when debugging is requested from a Visual Threads alarm as described in Section 1.14.1. The built-in support requires TotalView V4.0 (or greater), and that VT_DEBUG must be set to the value totalview. The totalview command must be on the path, or VT_DEBUG must contain the full path to the totalview command.

1.15    Cannot Bring Up Some Windows If They Are Minimized

Some Visual Threads windows allow you to bring up only one instance of the window at one time. If you minimize one of these windows and try to bring it up again using the menus on the Control Panel, you will not be able to bring it up. To bring it up again, just click on the icon.

1.16    Restriction for Applications That Make Use of the OpenMP API

If you plan to use Visual Threads with applications that make use of the OpenMP API, you should be aware of at least one known restriction.

If you use the unguardedData rule with this type of application, Visual Threads might report many race conditions that are not race conditions at all. This is because these applications use indirect synchronization mechanisms that are undetectable by the Visual Threads Instrumentation tool (VTI). VTI is designed for applications that use the following synchronization mechanisms to directly protect data locations:

See the vti manpage for details about the algorithm used by VTI.

1.17    X Server Support

The Visual Threads user interface is implemented in Java and therefore inherits restrictions on the X server and window managers that are supported.

Visual Threads is supported only with X servers provided by Compaq. This includes those contained in eXcursion, Tru64 UNIX and OpenVMS. Similarly, you must also use one of the following supported window managers: mwm, CDE, or eXcursion.

When used with other X servers or window managers, Visual Threads may not function correctly.

1.18    Visual Threads on AlphaServer GS Series Systems

1.18.1    Home RAD Changes

On Tru64 UNIX V5.1 on an AlphaServer GS Series system, Visual Threads is not notified about Home RAD changes caused by pthread_rad_attach() or pthread_rad_bind(). If the application changes the Home RAD of a thread using these functions or some other mechanism, Visual Threads cannot accurately determine on which RAD the thread is executing. This can cause the CPU time to be attributed to the incorrect data being displayed in the CPU Utilization window. Similarly, the "Percent remote accesses" statistic of synchronization objects, and the Object Access pie chart may be incorrect. This problem has been corrected in V5.1A. These problems should not occur if the Home RAD of a thread is not forcibly changed.

1.18.2    Java Problems

Visual Threads uses Java to provide the user interface. There are some known problems in the Java implementation that may cause occasional failures of Visual Threads during startup. These problems are more frequent when running Visual Threads on a system with more than four processors (in particular, AlphaServer GS Series systems), and when using a PC X Server display such as eXcursion. There are several behaviors that may be observed when starting Visual Threads:

In either of the above cases, simply restarting Visual Threads usually corrects the problem as it is intermittent. After Visual Threads completely initializes, seldom are Java problems encountered.

1.19    Submitting Problem Reports

If you need to submit a problem report about Visual Threads, if possible include the following with your submission:

You can submit the problem report using your normal support mechanism, if you have one, or send the report in e-mail to Visual.Threads@hp.com.

1.20    Visual Threads Email Address and Web Page

If you have comments, questions, or feedback about Visual Threads that you'd like to send to Compaq, send an e-mail message to Visual.Threads@hp.com.

For the latest information, see the Web page at: http://www.compaq.com/visualthreads/.

1.21    Intermittent Problems

1.21.1    eXcursion and eXceed Cause Problems in Rules Setup Dialog Box

When eXcursion or eXceed is used for displaying, the Overview tab in the Rules Setup dialog box has occasional problems. The checkbox text does not always line up, and if you click on a tab, a rectangle is sometimes displayed on the current checkbox or on the wrong checkbox.

1.21.2    Dialog Boxes Might Obscure Each Other

It is possible for two dialog boxes to obscure each other, particularly with window managers that implement "raise window on mouse enter." This is because all dialog boxes associated with a frame are at the same level in the "stacking order." If a dialog box (or the application itself) is unresponsive, make sure that one dialog box is not obscuring another (modal) dialog box.

1.21.3    Event Buffering Might Cause Internal Error

If you run Visual Threads on a system with multiple CPUs and you have the Disk space used for event buffering field set to anything other than None, you might experience an internal error. If this happens, go to the Data Collection tab in the Options dialog box and set the field to None. This should eliminate the problem from recurring.