Äîêóìåíò âçÿò èç êýøà ïîèñêîâîé ìàøèíû. Àäðåñ îðèãèíàëüíîãî äîêóìåíòà : http://www.parallel.ru/sites/default/files/ftp/mpi/doc/mpeguide.ps Äàòà èçìåíåíèÿ: Wed Nov 2 11:53:59 2011 Äàòà èíäåêñèðîâàíèÿ: Tue Oct 2 03:11:58 2012 Êîäèðîâêà: IBM-866

ANL/MCSíTMíANLí98/xx
User's Guide for MPE: Extensions for MPI Programs
by
William Gropp and Ewing Lusk
ARGONNE
NATIONAL
LABORATORY
UNIVERSITY OF
CHICAGO
. .
MATHEMATICS AND
COMPUTER SCIENCE
DIVISION

Contents
Abstract 1
1 Introduction 1
2 The MPE library of useful extensions 1
2.1 Logfile Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2.2 Logfile Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.3 Parallel X Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.4 Other MPE Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.5 Profiling Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.5.1 Accumulation of Time Spent in MPI routines . . . . . . . . . . . . . 3
2.5.2 Automatic Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.5.3 Customized Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.5.4 RealíTime Animation . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.6 Logfile Viewers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.6.1 Upshot and Nupshot . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.6.2 Jumpshotí2 and Jumpshotí3 . . . . . . . . . . . . . . . . . . . . . . 5
2.7 Accessing the profiling libraries . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.8 Automatic generation of profiling libraries . . . . . . . . . . . . . . . . . . . 8
2.9 Tools for Profiling Library Management . . . . . . . . . . . . . . . . . . . . 8
3 Using MPE 10
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2 Directory Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.3 Example Makefile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.4 Environmental Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.5 Utility Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.5.1 Log Format Converters . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.5.2 Log Format Print Programs . . . . . . . . . . . . . . . . . . . . . . . 12
3.5.3 Display Program Selector . . . . . . . . . . . . . . . . . . . . . . . . 13
3.6 Using MPE in MPICH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.6.1 Compilation and Linkage . . . . . . . . . . . . . . . . . . . . . . . . 13
3.6.2 Inheritance of Environmental Variables . . . . . . . . . . . . . . . . 13
3.6.3 Viewing Logfiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4 Debugging MPI programs with builtíin tools 14
4.1 Error handlers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.2 Contents of the library files . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
A Installing MPE 16
A.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
A.1.1 Configuration Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
A.1.2 Build Options and Features . . . . . . . . . . . . . . . . . . . . . . . 17
A.2 Installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
A.2.1 Configuring as part of the MPICH configure . . . . . . . . . . . . . . 19
A.2.2 Configuring as part of an existing MPI implementation . . . . . . . 19
A.3 Install/Uninstall Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
iii

B Installing Java for Jumpshots 21
B.1 viewers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
C Automatic generation of profiling libraries 23
C.1 Writing wrapper definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
D Manual Pages 27
Acknowledgments 28
iv

Abstract
The MPE extensions provide a number of useful facilites for MPI programmers.
These include several profiling libraries to collect information on MPI programs, iní
cluding logfiles for postímortum visualization and realítime animation. Also included
are routines to provide simple X window system graphics to parallel programs. MPE
may be used with any implemenation of MPI.
1 Introduction
The Message Passing Interface (MPI) [4] provides a strong basis for building parallel
programs. One of its design goals was to enable the construction of parallel software
libraries, thus helping to solve the problem of developing large parallel applications. The
MPE (MultiíProcessing Environment) library exploits the features of MPI to provide a
number of useful facilities, including performance and correctness debugging, graphics, and
some common utility routines.
The MPE library was developed for the MPICH [1] implementation of MPI (and
is included with the MPICH distribution), but can and has been used with any MPI
implementation. Installation instructions for MPE are in appendix ??.
2 The MPE library of useful extensions
Currently the main components of the MPE are
ffl A set of routines for creating logfiles for examination by various graphical visualization
tools : upshot, nupshot, Jumpshotí2 or Jumpshotí3.
ffl A sharedídisplay parallel X graphics library.
ffl Routines for sequentializing a section of code being executed in parallel.
ffl Debugger setup routines.
2.1 Logfile Creation
MPE provides several ways to generate logfiles that describe the progress of a computation.
These logfiles can be viewed with one of the graphical tools distributed with MPE. In
addition, you can customize these logfiles to add applicationíspecific information.
The easiest way to generate logfiles is to link your program with a special MPE library
that uses the profiling feature of MPI to intercept all MPI calls in an application. To find
out how to link with a profiling library that produces log files automatically, see Section 2.7.
You can create customized logfiles for viewing by calls to the various MPE logging routines.
For details, see the MPE man pages. An example is shown in Section 2.5.3.
To be added in later editions of this User's Guide:
1

ffl All MPE logging routines
ffl An example logfile
2.2 Logfile Format
MPE currently provides 3 different logfile formats, they are ALOG, CLOG and SLOG.
ALOG is provided for backward compatibility purpose and is no longer developed. CLOG
is a simple collections of singly timestamped events and is understood by Jumpshotí2.
CLOG is the first logfile format in MPE to be read by a Java based visualization tool.
SLOG is an abbreviation for Scalable LOGfile format and is based on doubly timestamped
states. The scalability of SLOG comes from separating all states in the logfile into
frames of data, each is small enough to be processed efficiently by the display program,
Jumpshotí3. At the same time, when two adjacent frames are compared sides by sides,
states and arrows will be showed leaving the earlier frame and then entering into later
frame seamlessly. SLOG and Jumpshotí3 are capable to handle logfile in the Giga Byte
range. And a simple statistics of state activties is kept by SLOG and displayed by the
Preview window of Jumpshotí3 to guide user to locate interesting frame. See document
`mpe/viewers/jumpshotí3/doc/TourStepByStep.pdf' for more details.
2.3 Parallel X Graphics
MPE provides a set of routines that allows you to display simple graphics with the X
Window System. In addition, there are routines for input, such as getting a region defined
by using the mouse. A sample of the available graphics routines are shown in Table 1. For
arguments, see the man pages.
You can find an example of the use of the MPE graphics library in the directory
mpich/mpe/contrib/mandel. Enter
make
mpirun ínp 4 pmandel
to see a parallel Mandelbrot calculation algorithm that exploits several features of the MPE
graphics library.
2.4 Other MPE Routines
Sometimes during the execution of a parallel program, you need to ensure that only a few
(often just one) processor at a time is doing something. The routines MPE×Seq×begin and
MPE×Seq×end allow you to create a ``sequential section'' in a parallel program.
The MPI standard makes it easy for users to define the routine to be called when an
error is detected by MPI. Often, what you'd like to happen is to have the program start
a debugger so that you can diagnose the problem immediately. In some environments, the
error handler in MPE×Errors×call×dbx×in×xterm allows you to do just that. In addition,
2

Control Routines
MPE Open graphics (collectively) opens an X display
MPE Close graphics Closes a X11 graphics device
MPE Update Updates an X11 display
Output Routines
MPE Draw point Draws a point on an X display
MPE Draw points Draws points on an X display
MPE Draw line Draws a line on an X11 display
MPE Draw circle Draws a circle
MPE Fill rectangle Draws a filled rectangle on an X11 display
MPE Draw logic Sets logical operation for new pixels
MPE Line thickness Sets thickness of lines
MPE Make color array Makes an array of color indices
MPE Num colors Gets the number of available colors
MPE Add RGB color Add a new color
Input Routines
MPE Get mouse press Get current coordinates of the mouse
MPE Get drag region Get a rectangular region
Table 1: MPE graphics routines.
you can compile the MPE library with debugging code included. (See the ímpedbg configure
option.)
2.5 Profiling Libraries
The MPI profiling interface provides a convenient way for you to add performance analysis
tools to any MPI implementation. We demonstrate this mechanism in MPICH, and give
you a running start, by supplying three profiling libraries with the MPICH distribution.
MPE users may build and use these libraries with any MPI implementation.
2.5.1 Accumulation of Time Spent in MPI routines
The first profiling library is simple. The profiling version of each MPI Xxx routine calls
PMPI Wtime (which delivers a time stamp) before and after each call to the corresponding
PMPI Xxx routine. Times are accumulated in each process and written out, one file per
process, in the profiling version of MPI Finalize. The files are then available for use in
either a global or processíbyíprocess report. This version does not take into account nested
calls, which occur when MPI Bcast, for instance, is implemented in terms of MPI Send and
MPI Recv.
3

2.5.2 Automatic Logging
The second profiling library is called MPE logging libraries which generate logfiles, they
are files of timestamped events for CLOG and timestamped states for SLOG. During
execution, calls to MPE Log event are made to store events of certain types in memory, and
these memory buffers are collected and merged in parallel during MPI Finalize. During
execution, MPI Pcontrol can be used to suspend and restart logging operations. (By
default, logging is on. Invoking MPI Pcontrol(0) turns logging off; MPI Pcontrol(1) turns
it back on again.) The calls to MPE Log event are made automatically for each MPI call.
You can analyze the logfile produced at the end with a variety of tools; these are described
in Sections 2.6.1 and 2.6.2.
2.5.3 Customized Logging
In addition to using the predefined MPE logging libraries to log all MPI calls, MPE logging
calls can be inserted into user's MPI program to define and log states. These states are
called UseríDefined states. States may be nested, allowing one to define a state describing
a user routine that contains several MPI calls, and display both the userídefined state and
the MPI operations contained within it. The routine MPE×Log×get×event×number has to
be used to get unique event numbers (this is important if you are writing a library that uses
the MPE logging routines) from the MPE system. The routines MPE×Describe×state and
MPE×Log×event are then used to describe userídefined states.
int eventID×begin, eventID×end;
...
eventID×begin = MPE×Log×get×event×number();
eventID×end = MPE×Log×get×event×number();
...
MPE×Describe×state( eventID×begin, eventID×end, ''Amult'', ''bluegreen'' );
...
MyAmult( Matrix m, Vector v )
--
/* Log the start event along with the size of the matrix */
MPE×Log×event( eventID×begin, mí?n, (char *)0 );
... Amult code, including MPI calls ...
MPE×Log×event( eventID×end, 0, (char *)0 );
Ý
The logfile generated by this code will have the MPI routines within the routine MyAmult
indicated by a containing bluegreen rectangle.
If the MPE logging library, `liblmpe.a', are NOT linked with the user program,
MPE×Init×log and MPE×Finish×log need to be used before and after all the MPE calls.
Sample programs `cpilog.c' and `fpi.f' are available in MPE source directory
`contrib/test' or the installed directory `share/examples' to illustrate the use of these
MPE routines.
4

2.5.4 RealíTime Animation
The third library does a simple form of realítime program animation. The MPE graphics
library contains routines that allow a set of processes to share an X display that is not
associated with any one specific process. Our prototype uses this capability to draw arrows
that represent message traffic as the program runs.
2.6 Logfile Viewers
There are 4 graphical visualization tools distributed with MPE, they are upshot, nupshot,
Jumpshotí2 and Jumpshotí3. Out of these 4 Logfile Viewers, only 3 viewers are built by
MPE. They are upshot, Jumpshotí2 and Jumpshotí3.
2.6.1 Upshot and Nupshot
One tool that we use is called upshot, which is a derivative of Upshot [3], written in Tcl/Tk.
A screen dump of Upshot in use is shown in Figure 1. It shows parallel time lines with
process states, like one of the paraGraph [2]. The view can be zoomed in or out, horizontally
or vertically, centered on any point in the display chosen with the mouse. In Figure 1, the
middle window has resulted from zooming in on the upper window at a chosen point to show
more detail. The window at the bottom of the screen show a histogram of state durations,
with several adjustable parameters.
Nupshot is a version of upshot that is faster but requires an older version of Tcl/Tk.
Because of this limitation, Nupshot is NOT built by default in current MPE.
2.6.2 Jumpshotí2 and Jumpshotí3
There are 2 versions of Jumpshot distributed with the MPE. They are Jumpshotí2 and
Jumpshotí3, which have evolved from Upshot and Nupshot. Both are written in Java,
are graphical visualization tools for interpreting binary tracefiles which displays them
onto a canvas object, such as the one depicted in Figure 2. For Jumpshotí2, See [5]
for more screenshots and details. For Jumpshotí3, See file
`mpe/viewers/jumpshotí3/doc/TourStepByStep.pdf' for a brief introduction of the tool.
As the size of the logfile increases, Jumpshotí2's performance decreases, and can
ultimately result in Jumpshotí2 hanging while it is reading in the logfile. It is hard to
determine at what point Jumpshotí2 will hang, but we have seen it with files as small as
10MB. When CLOG file is about 4MB in size, the performance of Jumpshotí2 starts to
deterioate significantly. There is a current research effort that will result in the ability to
make the Java based display program significantly more scalable. The results of the first
iteration of this effort are SLOG which supports scalable logging of data and Jumpshotí3
which reads SLOG.
5

Figure 1: A screendump from upshot
Figure 2: Jumpshotí1 Display
6

2.7 Accessing the profiling libraries
If the MPE libraries have been built, it is very easy to access the profiling libraries. The
configure in the MPE directory determines the link path necessary for each profiling library
(which varies slightly for each MPI implementation). These variables are first substituted
in the Makefile in the directory `mpe/contrib/test'. The Makefile is then installed into
directory `share/examples' during the final installation process. This information is placed
in the following variables:
ffl PROF LIBS í The compiler flag needed to link with the mpe library only. The link
path is ílmpe ílpmpich or ílmpe ílpmpi depending on the MPI implementation.
ffl LOG LIBS í The compiler flag needed to link with the logging libraries. The logging
libraries log all MPI calls and generate log file. The link path is íllmpe $PROF LIB.
ffl TRACE LIBS í The compiler flag needed to link with the tracing library. The
tracing library will trace all MPI calls. Each MPI call is preceded by a line that
contains the rank in MPI×COMM×WORLD of the calling process, and followed by another
line indicating that the call has completed. Most send and receive routines also
indicate the values of count, tag, and partner (destination for sends, source for
receives). Output is to standard output. The link path is íltmpe $PROF LIB.
ffl ANIM LIBS í The compiler flag needed to link with the animation library. The
animation library produces a realítime animation of the program. This requires the
MPE graphics, and uses X11 Window System operations. You may need to provide
a specific path for the X11 libraries (instead of ílX11). The link path is ílampe
$PROF LIB ílX11.
ffl F2CMPI LIBS í The compiler flag needed to link Fortran to C MPI wrapper
library with all the above mentioned libraries. For MPICH, this should be ílfmpich.
Otherwise, it could be ílmpe f2cmpi, MPE's own Fortran to C MPI wrapper library.
ffl FLIB PATH í The full compiler flag needed to link Fortran MPI programs with the
logging library.
As part of the make process, small programs `cpi.c' and `fpi.f' are linked with each
profiling library. The result of each link test will be written as part of the make output. If
the link test is successful, then these link paths should be used for user programs as well.
If the MPI implementation being used is MPICH, then adding compiler flag ímpilog to
MPICH's mpicc/mpif77 will automatically link user program with MPE's logging libraries
(íllmpe ílmpe). Library link path ílpmpich is also linked with the MPI profiling interface
when ímpilog flag is used
If a Fortran MPI program is linked with MPICH, it is necessary to include the library
`ílfmpich' ahead of the profiling libraries. This allows C routines to be used for impleí
menting the profiling libraries for use by both C and Fortran programs. For example, to
generate log files in a Fortran program, the library linkage flag is ílfmpich íllmpe ílmpe
ílpmpich. Using mpif77 ímpilog will automatically link with all these libraries.
7

If the MPI implementation being used is not MPICH, it is necessary to include the
library `ílmpe f2cmpi' (MPE's own Fortran to C MPI wrapper library) instead of library
`ílfmpich'. Again this has to be linked before any of the profiling libraries. So the compiler
linkage flag will be ílmpe f2cmpi íllmpe ílmpe ílpmpi ílmpi.
It is possible to combine automatic logging with manual logging. Automatic logging will
log all MPI calls and is achieved by linking with $LOG LIBS. Manual logging is achieved by
the user inserting calls to the MPE routines around MPI calls. This way, only the chosen MPI
calls will be logged. However, if a combination of these two types of logging is preferred, then
the user must NOT call MPE Init log and MPE Finish log in the user program. Because
in the linked logging library, MPI Init will call MPE Init log and MPI Finalize will call
MPE Finish log.
2.8 Automatic generation of profiling libraries
For each of these libraries, the process of building the library was very similar. First,
profiling versions of MPI Init and MPI Finalize must be written. The profiling versions
of the other MPI routines are similar in style. The code in each looks like
int MPI×Xxx( . . . )
--
do something for profiling library
retcode = PMPI×Xxx( . . . );
do something else for profiling library
return retcode;
Ý
We generate these routines by writing the ``do something'' parts only once, in schematic
form, and then wrapping them around the PMPI calls automatically. It is thus easy to
generate profiling libraries. See the README file in mpich/mpe/profiling/wrappergen or
Appendix C.
Examples of how to write wrapper templates are located in the mpe/profiling/lib
subdirectory. There you will find the source code (the .w files) for creating the three
profiling libraries described above. An example Makefile for trying these out is located in
the mpe/profiling/examples directory.
2.9 Tools for Profiling Library Management
The sample profiling wrappers for MPICH are distributed as wrapper definition code. The
wrapper definition code is run through the wrappergen utility to generate C code (see
Section 2.8. Any number of wrapper definitions can be used together, so any level of
profiling wrapper nesting is possible when using wrappergen.
A few sample wrapper definitions are provided with MPICH:
timing Use MPI×Wtime() to keep track of the total number of calls to each MPI function,
8

and the time spent within that function. This simply checks the timer before and
after the function call. It does not subtract time spent in calls to other functions.
logging Create logfile of all pt2pt function calls.
vismess Pop up an X window that gives a simple visualization of all messages that are
passed.
allprof All of the above. This shows how several profiling libraries may be combined.
Note: These wrappers do not use any mpichíspecific features besides the MPE graphics
and logging used by `vismess' and `logging', respectively. They should work on any MPI
implementation.
You can incorporate them manually into your application, which involves three changes
to the building of your application:
ffl Generate the source code for the desired wrapper(s) with wrappergen. This can be a
oneítime task.
ffl Compile the code for the wrapper(s). Be sure to supply the needed compileíline
parameters. `vismess' and `logging' require the MPE library (`ílmpe'), and the
`vismess' wrapper definition requires MPE GRAPHICS.
ffl Link the compiled wrapper code, the profiling version of the mpi library, and any
other necessary libraries (`vismess' requires X) into your application. The required
order is:
$(CLINKER) !application object files...? ``
!wrapper object code? ``
!other necessary libraries (ílmpe)? ``
!profiling mpi library (ílpmpi)? ``
!standard mpi library (ílmpi)?
To simplify it, some sample makefile sections have been created in
`mpich/mpe/profiling/lib':
Makefile.timing í timing wrappers
Makefile.logging í logging wrappers
Makefile.vismess í animated messages wrappers
Makefile.allprof í timing, logging, and vismess
To use these Makefile fragments:
1. (optional) Add $(PROF×OBJ) to your application's dependency list:
myapp: myapp.o $(PROF×OBJ)
9

2. Add $(PROF×FLG) to your compile line CFLAGS:
CFLAGS = íO $(PROF×FLG)
3. Add $(PROF×LIB) to your link line, after your application's object code, but before
the main MPI library:
$(CLINKER) myapp.o íL$(MPIR×HOME)/lib/$(ARCH)/$(COMM) $(PROF×LIB) ílmpi
4. (optional) Add $(PROF×CLN) to your clean target:
rm íf *.o *~ myapp $(PROF×CLN)
5. Include the desired Makefile fragment in your makefile:
include $(MPIR×HOME)/mpe/profiling/lib/Makefile.logging
(or
#include $(MPIR×HOME)/mpe/profiling/lib/Makefile.logging
if you are using the wildly incompatible BSD 4.4íderived make)
3 Using MPE
3.1 Introduction
The MultiíProcessing Environment (MPE) attempts to provide programmers with a comí
plete suite of performance analysis tools for their MPI programs based on post processing
approach. These tools include a set of profiling libraries, a set of utility programs, and a
set of graphical tools.
The first set of tools to be used with user MPI programs is profiling libraries which
provide a collection of routines that create log files. These log files can be created manually
by inserting MPE calls in the MPI program, or automatically by linking with the appropriate
MPE libraries, or by combining the above two methods[see section.2.5.3]. Currently, the
MPE offers the following 3 profiling libraries.
ffl Tracing Library í This library traces all MPI calls. Each MPI call is preceded by a
line that contains the rank in MPI×COMM×WORLD of the calling process, and followed by
another line indicating that the call has completed. Most send and receive routines
also indicate the values of count, tag, and partner (destination for sends, source for
receives). Output is to standard output.
ffl Animation Libraries í A simple form of realítime program animation that requires
X window routines.
10

ffl Logging Libraries í The most useful and widely used profiling libraries in MPE.
They form the basis to generate log files from user MPI programs. There are currently
3 different log file formats allowed in MPE. The default log file format is CLOG. It
is basically a collection of events with single timestamps. And there is ALOG which
is provided for backward compatibility reason and it is not being developed. And
the most powerful one is SLOG, stands for Scalable LOGfile format, which can be
converted from CLOG file after CLOG file has been generated (preferred approach),
or can be generated directly when MPI program is executed (through setting the
environmental variable MPE LOG FORMAT to SLOG).
The set of utility programs in MPE includes log format converter (e.g. clog2slog),
logfile print (e.g. slog print) and logfile viewer wrapper, logviewer, which selects the
correct graphical tool to display the logfile based on the logfile's file extension.
Currently, MPE's graphical tools includes 3 display programs, upshot for ALOG,
Jumpshotí2 for CLOG and Jumpshotí3 for SLOG. The logviewer script eliminates the
need for user to remember the relationship between logfile formats and display programs.
3.2 Directory Structure
The final install directory contains the following subdirectories. In terms of usage of MPE,
user usually only need to know about the files that have been installed in `include/', `lib/'
and `bin/'.
include/ contains all the include files that user program needs to read.
lib/ contains all the libraries that user program needs to link with.
bin/ contains all the utility programs that user needs to use.
sbin/ contains the MPE uninstall script to uninstall the installation.
share/ contains user readíonly data. Besides `share/examples/', user usually does NOT
need to know the details of other subdirectories.
3.3 Example Makefile
`share/examples' contains some very useful and simple example programs and `Makefile'
which illustrates the usage of MPE routines and the linkage of MPE libraries to generate
logfiles. In most cases, users can simply copy the `share/examples/Makefile' to their
home directory, and do a make to compile the suggested targets. Users don't need to copy
the `.c' and `.f' files when MPE has been compiled with a make that has VPATH support.
The created executables can be launched with mpirun provided by the MPI implementation
to generate sample logfiles.
11

3.4 Environmental Variables
There are 2 environmental variables, TMPDIR and MPE LOG FORMAT, that user may
need to set before the generation of logfiles :
MPE LOG FORMAT : determines the format of the logfile generated from the exí
ecution of application linked with MPE logging libraries. The allowed value for
MPE LOG FORMAT are CLOG, SLOG and ALOG. When MPE LOG FORMAT
is NOT set, CLOG is assumed.
TMPDIR : specifies a directory to be used as temporary storage for each process. By
default, when TMPDIR is NOT set, `/tmp' will be used. When user needs to generate
a very large logfile for longírunning MPI job, user needs to make sure that TMPDIR
is big enough to hold the temporary logfile which will be deleted if the merged logfile
can be created successfully. In order to minimize the overhead of the logging to the
MPI program, it is highly recommended user to use a *local* file system for TMPDIR.
Note : The final merged logfile will be written back to the file system where process
0 is.
3.5 Utility Programs
In `bin/', user can find several useful utility programs when manipulating logfiles. These
includes log format converters, e.g. clog2slog, log format print programs, e.g. slog print,
and a script to launch display program, logviewer.
3.5.1 Log Format Converters
clog2slog í a CLOG to SLOG logfile converter. Since the automatic generation of SLOG
file through setting of environmental variable MPE LOG FORMAT to SLOG may
NOT work for some non wellíbehaved MPI programs, using the logfile format converter
can generate extra diagnostic information about the condition of the logfile. Also the
converter allows one to adjust certain parameters of the logfile, like frame size which is
the segment of the logfile to be displayed by Jumpshotí3's Time Line window. For non
well behaved MPI program, one may need to increase the frame size from the default
64KB to a bigger value. For more information about the converter, do clog2slog íh.
clog2alog í a CLOG to ALOG logfile converter. It is not being developed. It is provided
here for backward compatibility purpose.
3.5.2 Log Format Print Programs
slog print í a stdout print program for SLOG file. It serves to check the content of the
logfile. If the SLOG file is too big, it may NOT be useful to use slog print. Also,
when slog is NOT complete, slog print won't work. So it serves as a simple test to
check if the SLOG file is generated completely.
clog print í a stdout print program for CLOG file.
12

3.5.3 Display Program Selector
logviewer í the script which involves appropriate viewer based on the file extension of
logfile. For instance, if the logfile is foo.slog, logviewer will invoke Jumpshotí3
to display the logfile. Jumpshotí3 resides in `share/'. For more information of
logviewer, do logviewer íhelp to list all available options.
3.6 Using MPE in MPICH
MPE has been seamlessly integrated into MPICH distribution, so user may find it easier to
use MPE when using it with MPICH. Here are the differences of using MPE with MPICH
and with other MPI implementations.
3.6.1 Compilation and Linkage
MPICH provides scripts to help users to compile and link C/C++ and F77/F90 programs.
They are mpicc for C programs, mpiCC for C++ programs, mpif77 for F77 and mpif90 for
F90 programs. In addition, these 4 scripts allows special options to be used to link with
MPE profiling libraries. These options are :
ímpitrace í to compile and link with tracing library.
ímpianim í to compile and link with animation libraries.
ímpilog í to compile and link with logging libraries.
For instance, the following command creates executable, fpilog, which generates logfile
when it is executed.
mpif77 ímpilog ío fpilog fpilog.f
For other MPI implementations, user needs to compile and link their application with
MPE profiling libraries explicitly as shown in the example makefile.
3.6.2 Inheritance of Environmental Variables
MPE relies on certain environmental variables (e.g. TMPDIR). These variables determine
how MPE behaves. It is important to make sure that all the MPI processes receive the
intended value of environmental variables. The complication of this issue comes from the
fact that MPICH contains many different devices for different platforms, some of these
devices have a different way of passing of environmental variables to other processes. The
often used devices, like ch p4 and ch shmem, do not require special attention to pass the
value of the environmental variable to spawned processes. The spawned process inherits
the value from the launching process as long as the environmental variable in the launching
process is set. But this is NOT true for all the devices, for instance, the ch p4mpd device
13

may require a special option of mpirun to be used to set the environmental variables to all
processes.
mpirun ínp N cpilog íMPDENVí MPE×LOG×FORMAT=SLOG
In this example, the option íMPDENVí is needed to make sure that all processes have
their environmental variable, MPE LOG FORMAT, set to SLOG.
For other MPI implementations, how environmental variables are passed remains uní
changed. User needs to get familar with the environment and set the environmental variables
accordingly.
3.6.3 Viewing Logfiles
MPE's install directory structure is the same as MPICH's. So all MPE's utility programs
will be located in MPICH's `bin/' directory. To view a logfile, say `fpilog.slog', do
logviewer fpilog.slog
The command will select and invoke Jumpshotí3 to display the content of SLOG file if
Jumpshotí3 has been built and installed successfully.
4 Debugging MPI programs with builtíin tools
Debugging of parallel programs is notoriously difficult, and we do not have a magical solution
to this problem. Nonetheless, we have built into MPICH a few features that may be of use
in debugging MPI programs.
4.1 Error handlers
The MPI Standard specifies a mechanism for installing one's own error handler, and specifies
the behavior of two predefined ones, MPI×ERRORS×RETURN and MPI×ERRORS×ARE×FATAL. As
part of the MPE library, we include two other error handlers to facilitate the use of dbx in
debugging MPI programs.
MPE×Errors×call×dbx×in×xterm
MPE×Signals×call×debugger
These error handlers are located in the MPE directory. A configure option (ímpedbg) includes
these error handlers into the regular MPI libraries, and allows the commandíline argument
ímpedbg to make MPE×Errors×call×dbx×in×xterm the default error handler (instead of
MPI×ERRORS×ARE×FATAL).
14

4.2 Contents of the library files
The directory containing the MPI library file (`libmpich.a') contains a few additional files.
These are summarized here
libmpe.a contains MPE graphics, logging, and other extensions (PMPE×Xxxx)
libmpe nompi.a contains MPE graphics without MPI
libampe.a contains MPE Animation interface
liblmpe.a contains MPE Logging interface
libtmpe.a contains MPE Tracing interface
libmpe f2cmpi.a contains MPE Fortran to C MPI wrapper interface
mpe prof.o Sample profiling library (C)
15

Appendices
A Installing MPE
A.1 Configuration
MPE can be configured and installed as an extension to most MPI standard compliant MPI
implementations, e.g. MPICH, LAM, SGI's MPI, HPíUX's MPI and IBM's MPI. It has
been integrated seamlessly into MPICH distribution, so MPE will be installed automatically
during MPICH's installation process.
A.1.1 Configuration Model
MPE is designed to be used as an extension to an existing MPI implementation, so its
configuration model assumes a general MPI development environment. Here are the some
of the variables that MPE configure reads, some are read as environmental variables and
some are read from the command line arguments to configure.
CC : C compiler used to create serial executable,
e.g. xlc for IBM MPI.
MPI×CC : C compiler used to compile MPI program and to create parallel
executable,
e.g. mpcc for IBM MPI, or mpicc for MPICH.
MPE×CFLAGS : CFLAGS for CC and MPI×CC.
F77 : F77 compiler used to create serial executable,
e.g. xlf for IBM MPI.
MPI×F77 : F77 compiler used to compile MPI program and to create parallel
executable,
e.g. mpxlf for IBM MPI, or mpif77 for MPICH.
MPE×FFLAGS : FFLAGS for F77 and MPI×F77.
MPI×INC : compiler's include flag (with prefix ''íI'') for MPI×CC/MPI×F77,
e.g. ''íI/usr/include'' for mpi.h on IRIX64.
MPI×LIBS : compiler's library flag (with prefix ''íL'' for library path and
prefix ''íl'' for each library name) needed by MPI×CC/MPI×F77,
e.g. ''íL/usr/lib ílmpi'' for libmpi.a on IRIX64.
F2CMPI×LIBS : compiler's library flag for Fortran to C MPI wrapper library,
e.g. ''ílfmpich'' when MPI×CC=mpicc & MPI×F77=mpif77 for MPICH.
Among above listed variables, CC, MPI CC, F77 and MPI F77 are usually set by
the corresponding environmental variables. The rest can be set through command line
arguments to configure. In some MPI implementations, like HPíUX's, MPI CC and
MPI F77 are reserved for use by the MPI implementation, use the configure options to
set MPI CC and MPI F77 instead.
16

A.1.2 Build Options and Features
MPE's configure is written using autoconf 2, and supports VPATH style install process. It
means the actual source directory and the building directory can be in 2 different locations.
This allows the same source directory to be used to build multiple versions of MPE with
different options and still won't mess up the original source. It is highly recommended that
user should do a VPATH build. Also MPE involves several different independent packages,
in order to create a tightly integrated environment for user, it is recommended that user
should do a make install to install the MPE in a separate directory after the build is
done. The benefit is that all utility programs will be in `bin/', all libraries will be in `lib/'
and all graphic tools will be nicely organized in `share/' ...
There are 2 types of configure options.
1) MPI implementation and User options
2) Generic configure flags supplied by autoconf 2
For a list of flags/switches for type 1 (not type 2) in MPE, use the script
configureííhelp.
The following is not a complete list but some of the more important ones. Generic flags:
ííprefix=INSTALL×DIR Specifies the final install directory for
''make install''. All libraries, utility
programs, graphic programs and examples
are installed in a standard directory
structure without files created in the
building process.
ííxíincludes=X×INC Specifies the directory where X include
files are located. This is used when
configure has trouble in locating X in
user system.
ííxílibraries=X×LIBS Specifies the directory where X libraries
are located. This is used when configure
has trouble in locating X in user system.
MPI implementation Options:
ííwithímpicc=MPI×CC Specify MPI C compiler to generate parallel
executable, e.g. mpcc for AIX.
ííwithímpif77=MPI×F77 Specify MPI F77 compiler to generate parallel
executable, e.g. mpxlf for AIX.
ííwithícflags=MPE×CFLAGS Specify extra CFLAGS to the C and MPI×CC
compilers, e.g. ''í64'' for IRIX64 C compiler
17

ííwithífflags=MPE×FFLAGS Specify extra FFLAGS to the F77 and MPI×F77
compilers, e.g. ''í64'' for IRIX64 F77 compiler
ííwithímpiinc=MPI×INC Specify compiler's include flag for MPI
include directory,
e.g. ''íI/pkgs/MPI/include'' for mpi.h
ííwithímpilibs=MPI×LIBS Specify compiler's library flag for MPI
libraries,
e.g. ''íL/pkgs/MPI/lib ílpmpich ílmpich''
ííenableíf77 Enable the compilation of routines that
require a Fortran compiler. If configuring
with MPICH, the configure in the topílevel
MPICH directory will choose the appropriate
value for you. However, it can be overridden.
The default is yes, ííenableíf77.
ííenableíf2cmpilib Enable the building of MPE's internal Fortran
to C MPI wrapper library. The default is yes,
ííenableíf2cmpilib
ííwithíf2cmpilibs=F2CMPI×LIBS Specify compiler's library flags for Fortran
to C MPI wrapper library. Using this option
will force íídisableíf2cmpilib.
e.g. ''ílfmpich'' when configuring MPE for MPICH
Other User Options:
ííenableíecho Turn on strong echoing. The default is no,
íídisableíecho.
ííwithímpelibname=MPE×LIBNAME Specify the MPE library name instead of the
default 'mpe'. e.g. if MPE×LIBNAME=''MPE'',
then the libraries generated will be libMPE.a,
liblMPE.a, libtMPE.a, libaMPE.a and
libMPE×f2cmpi.a. This option is necessary
when configuring MPE for a existing and older
version of MPICH which has MPE installed.
ííenableímpe×graphics Enable the building of MPE graphics routines.
If disabled, then the MPE routines that make
use of X11 graphics will not be built. This
is appropriate for systems that either do not
have the X11 include files or that do not
support X11 graphics. The default is
enable=yes.
18

ííenableíviewers Enable the build of all the available log
viewers. The default is enable=yes
ííwithíjava=JAVA×HOME Specify the path of the topílevel directory
of the Java, JDK, installation. If this
option is not given, configure will try to
locate JDK for you to build Jumpshotí2 and
Jumpshotí3. JDK 1.1.6 to JDK 1.1.8 can be
used to build both Jumpshots.
ííwithíwishloc=WISHLOC This switch specifies the name of Tcl/Tk wish
executable. If this switch is omitted,
configure will attempt to locate a version.
This is used only for upshot.
Note: Because Tcl and Tk keep changing in
incompatible ways, we will soon be dropping
support for any tool that uses Tcl/Tk.
A.2 Installation Instructions
As noted earlier, the MPE library can be installed as part of the MPICH configure or as
an extension of an existing MPI implementation. Below are instructions and examples for
typical installation of MPE on popular MPI implementations.
A.2.1 Configuring as part of the MPICH configure
The configure in the MPICH directory will try to determine the necessary information and
pass it to the MPE configure. If no options are given, the MPE will automatically be
configured by default. However, the user can provide extra configuration information to
MPE through MPICH configure with the following options:
ímpe×opts=MPE×OPTS
where MPE OPTS is one or more of the choices in section A.1.2. Multiple instances of
ímpe opts are allowed in MPICH configure to specify different options for the MPE. The
following is a configure option which specifies a nonídefault location of JDK installation.
ímpe×opts=ííwithíjava=/sandbox/chan/java/1.1.8
A.2.2 Configuring as part of an existing MPI implementation
The following are some examples for configuring MPE for an existing MPI implementation.
ffl For SGI MPI, e.g. denali.mcs.anl.gov, do the following for default ABI, ín32
19

setenv MAKE gmake
$--MPE×SRC×DIRÝ/configure ííwithímpilibs=ílmpi ``
ííwithíjava=/usr/javaí1.1.6/usr/java
make
make install PREFIX=$--MPE×INSTALL×DIRÝ
for 64 bits ABI, add options --withícflags=í64 and --withífflags=í64 to the configure
options.
ffl For IBM MPI, e.g. quad.mcs.anl.gov, do
setenv MPI×CC mpcc
setenv MPI×F77 mpxlf
$--MPE×SRC×DIRÝ/configure ííwithíjava=/homes/chan/pkgs/java/J1.1.8
make
make install PREFIX=$--MPE×INSTALL×DIRÝ
ffl For HPíUX's MPI, do
$--MPE×SRC×DIRÝ/configure ííwithímpicc=mpicc ``
ííwithímpif77=mpif77 ``
ííwithíflib×path×leader=''íWl,íL''
make
make install PREFIX=$--MPE×INSTALL×DIRÝ
MPE's Fortran support on HPíUX's MPI is NOT working yet. So to get MPI Fortran
code to generate logfile, you could use HPíUX's libfmpi.a if it is there. Here is the
configure options.
$--MPE×SRC×DIRÝ/configure ííwithímpicc=mpicc ``
ííwithímpif77=mpif77 ``
ííwithíflib×path×leader=''íWl,íL'' ``
ííwithíf2cmpilibs=ílfmpi
make
make install PREFIX=$--MPE×INSTALL×DIRÝ
ffl For existing MPICH, like prebuilt version of MPICHí1.1.2 or MPICHí1.2.0, e.g.
donner, do
setenv MPI×CC $--MPICH×INSTALL×DIRÝ/mpicc
setenv MPI×F77 $--MPICH×INSTALL×DIRÝ/mpif77
$--MPE×SRC×DIRÝ/configure ííwithíf2cmpilibs=ílfmpich ``
ííwithímpelibname=newMPE ``
ííwithíjava=/sandbox/jdk117×v3
make
make install PREFIX=$--MPE×INSTALL×DIRÝ
20

It is important to use the configure option --withímpelibname to specify a different
MPE library name than the default ''mpe'' when configuring MPE for older MPICH.
Without this option, the linkage tests in MPE would most likely use the old MPE
libraries in the MPICH instead of the newly built MPE libraries in resolving the MPE
symbols. Also the option --withíf2cmpilibs forces MPE to use the Fortran to C MPI
wrapper library in previous version of MPICH.
ffl For LAM, do
setenv MPI×CC $--LAM×INSTALL×DIRÝ/bin/hcc
setenv MPI×f77 $--LAM×INSTALL×DIRÝ/bin/hf77
$--MPE×SRC×DIRÝ/configure ííwithímpilibs=''íL$--LAM×INSTALL×DIRÝ/lib ílpmpi'' ``
ííwithíjava=/sandbox/jdk117×v3
make
make install PREFIX=$--MPE×INSTALL×DIRÝ
Using MPE with LAM for fortran MPI program is not working until recently, i.e.
MPE in MPICHí1.2.1 or later. Configure options listed above enable MPE's internal
Fortran to C MPI library. 1 To use LAM's Fortran to C MPI library in LAM 6.3.3
or later, liblamf77mpi.a, do
setenv MPI×CC $--LAM×INSTALL×DIRÝ/bin/hcc
setenv MPI×f77 $--LAM×INSTALL×DIRÝ/bin/hf77
$--MPE×SRC×DIRÝ/configure ííwithímpilibs=''íL$--LAM×INSTALL×DIRÝ/lib ílpmpi'' ``
ííwithíf2cmpilibs=íllamf77mpi ``
ííwithíjava=/sandbox/jdk117×v3
make
make install PREFIX=$--MPE×INSTALL×DIRÝ
A.3 Install/Uninstall Scripts
A mpeinstall script is created during configuration. If configuring with MPICH, then
the mpiinstall script will invoke the mpeinstall script. However, mpeinstall can also
be used by itself. This is only optional and is of use only if you wish to install the MPE
library in a public place so that others may use it. Final install directory will consist of
an `include', `lib', `bin', `sbin' and `share' subdirectories. Examples and various logfile
viewers will be installed under `share'. The `sbin' in installed directory contains an MPE
uninstall script mpeuninstall.
B Installing Java for Jumpshots
B.1 viewers
MPE includes a `viewers' subdirectory, which is an independent package, includes 2 versions
of Jumpshots, Jumpshotí2 and Jumpshotí3. Both needs to be built with Java Development
1 if you have MPE in MPICHí1.2.1, you need to download a patch from MPICH's web server to fix MPE
before building MPE with LAM. The URL is http://wwwíunix.mcs.anl.gov/mpi/mpich/buglistítbl.html
21

Kit (JDK), i.e. Java distribution that includes a Java compiler, javac. Theoretically,
Jumpshots can be distributed with precompiled byte code instead of source code that needs
to be compiled. The main reason is that there exists JDK which isn't compatible with
Swing and one of the easiest ways to detect this problem is to compile and link the code to
see if things are fine. We are still planning to distribute byte code in later version.
Jumpshotí2 and Jumpshotí3 are both developed based on Sun's JDK. Jumpshotí2 is
developed based on JDKí1.1/Swingí1.0.3. So it can only be built with JDKí1.1, not JDKí
1.2 or later. On the other hand, Jumpshotí3 is developed based on JDKí1.1/Swingí1.1.1. It
can be built by both JDKí1.1 and JDKí1.2. So it is recommended that user should use the
latest JDKí1.1, i.e. JDKí1.1.8, to build both Jumpshots when configuring MPE or MPICH.
If one would like to use different version of JDK to build each Jumpshot, one needs to build
these two tools separately.
We will list the status of the JDK distributions on some of the popular UNIX platforms
that we have tested and what is needed to build the JDK properly for Jumpshots. The
URLs provided below are updated on 09/23/2000.
ffl Linux/i386: Linux running on intel x86 processor has many choices of JDKs.
Blackdown.org has released many different versions of JDK for Linux on this platform,
including both JDKí1.1 and JDKí1.2. You can download them by locating the closest
FTP site at `http://www.blackdown.org', one of the popular download sites in US
is `ftp://metalab.unc.edu/pub/linux/devel/lang/java/blackdown.org/'. Pick
the JDK distribution that has the correct `libc' for your platform.
Sun also distributes a JDKí1.2 for linux. Here is the URL for download.
`http://www.javasoft.com/products/jdk/1.2/downloadílinux.html'.
As soon as the package is unpacked, it should be ready to compile Jumpshots.
ffl Linux/alpha: Linux running on alpha processor has limited choice of JDK. The
only versions that have been tested to be able to compile Jumpshots is JDKí1.1.8
from alphalinux.org. Here are the URL:
`ftp://ftp.alphalinux.org/pub/java/javaí1.1/r2/jdk118 RH60 alpha bin 21164 v2.tgz'
or
`ftp://ftp.alphalinux.org/pub/java/javaí1.1/r3/jdk118 RH60 alpha bin 21164 v3.tgz'
Since the distribution does NOT come with classes file. User needs to download the
classes file separately. The URL is
`ftp://ftp.alphalinux.org/pub/java/javaí1.1/r2/jdk118 alpha classes v2.tgz'
Since Jumpshots need file `classes.zip', so after unzipped the file, be sure to do the
following to generate the `classes.zip'.
cd jdk118/classes
zip í0 ../lib/classes.zip *
As opposed to JDK for linux, `libawt.so' is dynamical linked instead of statically
linked. So when running Jumpshot, it will complain missing of `libXm.so' if your
system doesn't have Motif installed. We installed Lesstif which seems to resolve the
issue.
22

Bugs: The ''nesting'' of states in Jumpshotí3 are NOT working in this setup.
ffl Solaris: We have tested JDKs as old as JDKí1.1.6 on Solaris box, e.g. Soí
laris JDK 1.1.6 03. 2
ffl IRIX64: We have tested JDKí1.1.6, JDKí1.1.7 and JDKí1.1.8 from SGI, they all
seem to work fine with Jumpshots. JDKí1.1.8 seems to work best with Jumpshots on
IRIX. You can download them at
`http://www.sgi.com/developers/devtools/languages/java.html'.
ffl AIX: Only JDKí1.1.8 for AIX have been tested with Jumpshots. You can download
them at
`http://www.ibm.com/java/jdk/aix/index.html'
ffl HPíUX: HP distributes JDK for its HPíUX OS. None of JDK for HPíUX has been
tested with Jumpshots because of lack of access to the platform. Here is the URL for
HP's JDK:
`http://www.unixsolutions.hp.com/products/java/index.html'
C Automatic generation of profiling libraries
The profiling wrapper generator (wrappergen) has been designed to complement the MPI
profiling interface. It allows the user to write any number of `meta' wrappers which can be
applied to any number of MPI functions. Wrappers can be in separate files, and can nest
properly, so that more than one layer of profiling may exist on indiividual functions.
Wrappergen needs three sources of input:
1. A list of functions for which to generate wrappers.
2. Declarations for the functions that are to be profiled. For speed and parsing simplicity,
a special format has been used. See the file `proto'.
3. Wrapper definitions.
The list of functions is simply a file of whitespaceíseparated function names. If omitted,
any forallfn or fnall macros will expand for every function in the declaration file.
C.1 Writing wrapper definitions
Wrapper definitions themselves consist of C code with special macros. Each macro is surí
rounded by the ff gg escape sequence. The following macros are recognized by wrappergen:
----filenoÝÝ
2 Swing 1.1.1 requires at least JDK 1.1.7, so in principle Jumpshotí3 may have problem with JDKí1.1.6.
23

An integral index representing which wrapper file the macro came from. This
is useful when declaring fileíglobal variables to prevent name collisions. It is
suggested that all identifiers declared outside functions end with ×----filenoÝÝ.
For example:
static double overhead×time×----filenoÝÝ;
might expand to:
static double overhead×time×0;
(end of example).
----forallfn !function name escape? !function A? !function B? ... ÝÝ
...
----endforallfnÝÝ
The code between ----forallfnÝÝ and ----endforallfnÝÝ is copied once for every
function profiled, except for the functions listed, replacing the escape string
specified by !function name escape? with the name of each function. For
example:
----forallfn fn×nameÝÝstatic int ----fn×nameÝÝ×ncalls×----filenoÝÝ;
----endforallfnÝÝ
might expand to:
static int MPI×Send×ncalls×1;
static int MPI×Recv×ncalls×1;
static int MPI×Bcast×ncalls×1;
(end of example)
----foreachfn !function name escape? !function A? !function B? ... ÝÝ
...
----endforeachfnÝÝ
----foreachfnÝÝ is the same as ----forallfnÝÝ except that wrappers are written
only the functions named explicitly. For example:
----forallfn fn×name mpi×send mpi×recvÝÝ
static int ----fn×nameÝÝ×ncalls×----filenoÝÝ;
----endforallfnÝÝ
might expand to:
static int MPI×Send×ncalls×2;
static int MPI×Recv×ncalls×2;
(end of example)
24

----fnall !function name escape? !function A? !function B? ... ÝÝ
...
----callfnÝÝ
...
----endfnallÝÝ
----fnallÝÝ defines a wrapper to be used on all functions except the functions
named. Wrappergen will expand into a full function definition in traditional
C format. The ----callfnÝÝ macro tells wrappergen where to insert the call
to the function that is being profiled. There must be exactly one instance
of the ----callfnÝÝ macro in each wrapper definition. The macro specified by
!function name escape? will be replaced by the name of each function.
Within a wrapper definition, extra macros are recognized.
----vardecl !type? !arg? !arg? ... ÝÝ
Use vardecl to declare variables within a wrapper definition. If
nested macros request variables through vardecl with the same names,
wrappergen will create unique names by adding consecutive integers to
the end of the requested name (var, var1, var2, ...) until a unique name
is created. It is unwise to declare variables manually in a wrapper
definition, as variable names may clash with other wrappers, and the
variable declarations may occur later in the code than statements from
other wrappers, which is illegal in classical and ANSI C.
----!varname?ÝÝ
If a variable is declared through vardecl, the requested name for
that variable (which may be different from the uniquified form that
will appear in the final code) becomes a temporary macro that will
expand to the uniquified form. For example,
----vardecl int i dÝÝ
may expand to:
int i, d3;
(end of example)
----!argname?ÝÝ
Suggested but not neccessary, a macro consisting of the name of one
of the arguments to the function being profiled will be expanded to
the name of the corresponding argument. This macro option serves
little purpose other than asserting that the function being profilied
does indeed have an argument with the given name.
----!argnum?ÝÝ
Arguments to the function being profiled may also be referenced by
number, starting with 0 and increasing.
25

----returnValÝÝ
ReturnVal expands to the variable that is used to hold the return
value of the function being profiled.
----callfnÝÝ
callfn expands to the call of the function being profiled. With nested wrapper
definitions, this also represents the point at which to insert the code for any
inner nested functions. The nesting order is determined by the order in which
the wrappers are encountered by wrappergen. For example, if the two files
`prof1.w' and `prof2.w' each contain two wrappers for MPI Send, the profiling
code produced when using both files will be of the form:
int MPI×Send( args...)
arg declarations...
--
/*preícallfn code from wrapper 1 from prof1.w */
/*preícallfn code from wrapper 2 from prof1.w */
/*preícallfn code from wrapper 1 from prof2.w */
/*preícallfn code from wrapper 2 from prof2.w */
returnVal = MPI×Send( args... );
/*postícallfn code from wrapper 2 from prof2.w */
/*postícallfn code from wrapper 1 from prof2.w */
/*postícallfn code from wrapper 2 from prof1.w */
/*postícallfn code from wrapper 1 from prof1.w */
return returnVal;
Ý
----fn !function name escape? !function A? !function B? ... ÝÝ
...
----callfnÝÝ
...
----endfnallÝÝ
fn is identical to fnall except that it only generates wrappers for functions
named explicitly. For example:
----fn this×fn MPI×SendÝÝ
----vardecl int iÝÝ
----callfnÝÝ
printf( ''Call to ----this×fnÝÝ.``n'' );
printf( ''----iÝÝ was not used.``n'' );
printf( ''The first argument to ----this×fnÝÝ is ----0ÝÝ``n'' );
----endfnÝÝ
26

will expand to:
int MPI×Send( buf, count, datatype, dest, tag, comm )
void * buf;
int count;
MPI×Datatype datatype;
int dest;
int tag;
MPI×Comm comm;
--
int returnVal;
int i;
returnVal = PMPI×Send( buf, count, datatype, dest, tag, comm );
printf( ''Call to MPI×Send.``n'' );
printf( ''i was not used.``n'' );
printf( ''The first argument to MPI×Send is buf``n'' );
return returnVal;
Ý
A sample wrapper file is in `sample.w' and the corresponding output file is in `sample.out'.
D Manual Pages
The MPE routines can be divided into six classes:
Logging routines MPE×Describe×event MPE×Describe×state MPE×Finish×log
MPE×Init×log MPE×Log×event MPE×Log×get×event×number MPE×Log×receive
MPE×Log×send MPE×Start×log MPE×Stop×log
X window graphics MPE×Add×RGB×color MPE×CaptureFile MPE×Close×graphics
MPE×Draw×circle MPE×Draw×line MPE×Draw×logic MPE×Draw×point
MPE×Draw×points MPE×Draw×string MPE×Fill×circle MPE×Fill×rectangle
MPE×Get×mouse×press MPE×Iget×mouse×press MPE×Line×thickness
MPE×Make×color×array MPE×Num×colors MPE×Open×graphics MPE×Update
Sequential Section MPE×Seq×begin MPE×Seq×end
Shared counter MPE×Counter×create MPE×Counter×free MPE×Counter×nxtval
Tag management MPE×GetTags MPE×ReturnTags MPE×TagsEnd
Miscellaneous MPE×Comm×global×rank MPE×Decomp1d MPE×Errors×call×debugger
MPE×IO×Stdout×to×file MPE×Print×datatype×pack×action
MPE×Print×datatype×unpack×action MPE×Ptime MPE×Signals×call×debugger
MPE×Wtime
27

Acknowledgments
The work described in this report has benefited from conversations with and use by a large
number of people. We also thank those that have helped in the implementation of MPE,
particularly Edward Karrels.
Debbie Swider had maintained MPE in MPICHí1.2.0 and before. Omer Zaki did the
first implementation of Jumpshot, Jumpshotí2. Abhi Shandilya did the original clog2slog
converter. Anthony Chan implemented SLOGíAPI, extended the Jumpshotí2 to Jumpshotí3
to support SLOG, integrated SLOGíAPI, clog2slog, Jumpshotí2 and Jumpshotí3 into
MPE.
References
[1] William Gropp, Ewing Lusk, Nathan Doss, and Anthony Skjellum. A highíperformance,
portable implementation of the MPI messageípassing interface standard. Parallel Comí
puting, 22:789--828, 1996.
[2] M. T. Heath. Recent developments and case studies in performance visualization
using ParaGraph. In G. Haring and G. Kotsis, editors, Performance Measurement and
Visualization of Parallel Systems, pages 175--200, Amsterdam, The Netherlands, 1993.
Elsevier Science Publishers.
[3] Virginia Herrarte and Ewing Lusk. Studying parallel program behavior with upshot.
Technical Report ANL--91/15, Argonne National Laboratory, Argonne, IL 60439, 1991.
[4] Message Passing Interface Forum. MPI: A messageípassing interface standard. Interí
national Journal of Supercomputer Applications, 8(3/4), 1994.
[5] Omer Zaki, Ewing Lusk, William Gropp, and Deborah Swider. Toward scalable perí
formance visualization with Jumpshot. High Performance Computing Applications,
13(2):277--288, Fall 1999.
28