Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.parallel.ru/sites/default/files/ftp/cluster/scali/Scali_LibGuide_30.pdf Дата изменения: Wed Nov 2 11:53:58 2011 Дата индексирования: Tue Oct 2 03:57:54 2012 Кодировка: Поисковые слова: m 2-9

Scali Library User 's Guide


Copyright © 1999-2002 Scali AS. All rights reserved.

Acknowledgement
The development of ScaMPI has benefited greatly from the work of people not connected to Scali. We wish especially to thank the developers of MPICH for their work which serv ed as a reference w hen implementing the first version of ScaMPI. The list of persons contributing to algorithmic ScaMPI improvements is impossible to compile here. We apologise to those who remain unnamed and mention only those who certainly are responsible for a step forw ard. Scali is thankful to Rolf Rabenseifner for the improved reduce algorithm used in ScaMPI.


Table of contents
Chapter 1 Introduction .............................................................................................. 1.1 Scali Library Suite .................................................................................................... 1.1.1 ScaMPI ........................................................................................................ 1.1.2 ScaShmem ................................................................................................... 1.1.3 ScaIP ........................................................................................................... 1.1.4 ScaMAC ....................................................................................................... 7 7 7 7 7 7

Chapter 2 Using ScaMPI ............................................................................................ 9 2.1 Setting up a ScaMPI environment ........................................................................... 9 2.1.1 ScaMPI environment variables ................................................................. 9 2.2 Compiling and linking .............................................................................................. 9 2.2.1 Compiler support ...................................................................................... 10 2.2.2 Compiler flags ........................................................................................... 10 2.2.3 Linker flags ............................................................................................... 10 2.3 Running ScaMPI programs .................................................................................... 10 2.3.1 Nam ing convention ................................................................................... 10 2.3.2 mpimon - monitor program ...................................................................... 11 2.3.3 mpirun - wrapper script ........................................................................... 16 2.4 Useful tools .............................................................................................................. 19 2.4.1 Debug ging with a sequential debugger ................................................... 19 2.4.2 Usefull builtin-tools for debugging .......................................................... 19 2.4.3 Profiling ScaMPI applications ................................................................. 20 2.4.4 Profiling with ScaMPE ............................................................................. 26 2.5 An example program ............................................................................................... 28 2.5.1 Hello-world.c - source in C ....................................................................... 28 2.5.2 Hello-world.f - source in Fortran ............................................................. 28 2.5.3 Compiling .................................................................................................. 29 2.5.4 Linking ...................................................................................................... 29 2.5.5 Running ..................................................................................................... 29 2.6 MPI test programs .................................................................................................. 29 2.6.1 Producer - a producer-consumer MPI test program ............................... 29 2.6.2 Bandwidth - a bandwidth MPI test program .......................................... 30 2.6.3 Bidirect - a bidirectional MPI test prog ram. ........................................... 30 Chapter 3 3.1 General 3.1.1 3.1.2 Description of ScaMPI . description ........................ ScaMPI libraries .............. ScaMPI executables ......... ... ... ... ... .. .. .. .. ... ... ... ... .. .. .. .. .. .. .. .. ... ... ... ... .. .. .. .. ... ... ... ... .. .. .. .. ... ... ... ... .. .. .. .. .. .. .. .. ... ... ... ... .. .. .. .. ... ... ... ... .. .. .. .. ... ... ... ... .. .. .. .. .. .. .. .. ... ... ... ... .. .. .. .. ... ... ... ... .. .. .. .. ... ... ... ... .. .. .. .. ... ... ... ... .. .. .. .. .. .. .. .. ... ... ... ... .. .. .. .. 31 31 31 31

Scali Library User's Guide Version 3.0

3


3.2 Starting ScaMPI application programs ................................................................. 3.2.1 Application start-up - phase 1 .................................................................. 3.2.2 Application start-up - phase 2 .................................................................. 3.2.3 Application start-up - phase 3 .................................................................. 3.3 Stopping ScaMPI application programs ................................................................ 3.4 Communication protocols........................................................................................ 3.4.1 Inlining protocol ........................................................................................ 3.4.2 Eagerbuffering protocol ............................................................................ 3.4.3 Transporter protocol ................................................................................. 3.5 Communication resources ....................................................................................... 3.5.1 Channel buffer .......................................................................................... 3.5.2 Eagerbuffer buffer..................................................................................... 3.5.3 Transporter buffer .................................................................................... Chapter 4 Tips & Tricks for ScaMPI .................................................................... 4.1 Application program notes...................................................................................... 4.1.1 MPI_Probe() and MPI_Recv() ................................................................... 4.1.2 Unsafe MPI programs............................................................................... 4.2 Namespace pollution ............................................................................................... 4.3 Error and warning messages .................................................................................. 4.3.1 User interface errors and warnings ......................................................... 4.3.2 Fatal errors ............................................................................................... 4.4 When things don't work - troubleshooting ............................................................. 4.4.1 Standard input and ScaMPI .................................................................... 4.4.2 Why doesn't my program start to run? .................................................... 4.4.3 Why doesn't mpid start ............................................................................. 4.4.4 Interconnect problem s .............................................................................. 4.4.5 Why does my program terminate abnormally? ....................................... 4.4.6 How do I control SC I and local shared memory usag e? .......................... 4.5 H ow to optimize MPI performance......................................................................... 4.5.1 Performance analysis................................................................................ 4.5.2 Using MPI_Isend(), MPI_Irecv(). ............................................................. 4.5.3 Using MPI_Bsend(). .................................................................................. 4.5.4 Avoid starving mpi-processes - fairness. ................................................. 4.5.5 Using processor-power to poll. ................................................................. 4.5.6 Communication buffer adaption .............................................................. 4.5.7 Reorder network traffic to avoid conflicts................................................ 4.6 Benchmarking ......................................................................................................... 4.6.1 How to get ex pected performance ............................................................ 4.6.2 Memory consum ption increase after warm-up ........................................

3 3 3 3 3 3 3 3 3 3 4 4 4

2 2 2 3 4 5 6 7 8 9 0 1 2

43 43 43 44 44 45 45 45 45 45 45 46 46 47 48 49 49 49 49 49 50 50 50 51 51 51

4

Scali Library User's G uide Version 3.0


Ch 5.1 5.2 5.3

apter 5 ScaShmem ................................................................................................ Description .............................................................................................................. Application porting to ScaShmem.......................................................................... Features and limitations ........................................................................................ 5.3.1 Communication initialization and termination ...................................... 5.3.2 Runtime requirements ............................................................................. 5.3.3 Datatypes / porting ................................................................................... 5.3.4 Dynamic memory allocation..................................................................... 5.3.5 ScaShmem environment variables .......................................................... 5.4 Compiling and linking ............................................................................................ 5.5 Running y our application ....................................................................................... Ch 6.1 6.2 6.3 6.4 Ch 7.1 7.2 7.3 7.4 apter 6 ScaIP - IP for SCI ........................................................... Introduction ..................................................................................... Simplified netw ork model ............................................................... Config uration .................................................................................. ScaIP package installation ............................................................. .. .. .. .. .. ... ... ... ... ... .. .. .. .. .. ... ... ... ... ... .. .. .. .. .. ... ... ... ... ... .. .. .. .. .. .. .. .. .. .. ... ... ... ... ... .. .. .. .. ..

53 53 53 54 54 54 54 54 55 55 56 57 57 57 58 59 61 61 61 62 63 63 64 65 66 67 67 67 67 67 68 68 68

apter 7 ScaMAC .................................................................................................... Introduction ............................................................................................................. The scimac driv er .................................................................................................... Setting up the scimac driv er .................................................................................. The ScaMAC utilities .............................................................................................. 7.4.1 macstat - display scimac driver status .................................................... 7.4.2 macping - check reachability of remote scimac drivers .......................... 7.4.3 macctl - set the debug level of the scimac driver .................................... 7.5 ScaMAC packag e installation ................................................................................ Ch 8.1 8.2 8.3 8.4 8.5 8.6 8.7 apter 8 Support ..................................................................................................... Feedback .................................................................................................................. Scali mailing lists .................................................................................................... ScaMPI FAQ............................................................................................................ ScaMPI release documents ..................................................................................... Problem reports ....................................................................................................... Platforms supported ............................................................................................... Licensing .................................................................................................................

Chapter 9 Related documentation ........................................................................ 69 9.1 References ............................................................................................................... 69

Scali Library User's Guide Version 3.0

5


6

Scali Library User's G uide Version 3.0


Chapter 1

Introduction

A Scali System is a set of SCI interconnected nodes, where each node is a multiprocessor w orkstation running Linux. To help y ou use the full potential power of such a system we have developed a suite of libraries.

1.1 Scali Library Suite
1.1.1 ScaM PI ScaMPI is a high performance MPI implementation. The prog ramming environment for ScaMPI provides a variety of options and tools for tuning and debugging. ScaMPI utilises shared memory on intranode communication, and the fast SCI interconnect on internode communication. Any parallel MPI-conforming application can be run with ScaMPI and benefit from the SCI performance. The chapters conserning ScaMPI is written for users w hich have a basic understanding of MPI [1, 2, 3], and some basic knowledge of the C and/or Fortran programming language. gcc and bash are used for all examples.

1.1.2 ScaShmem ScaShMem is an implementation of the Cray/SG I Shmem abstraction. It is built on top of ScaMPI 1.1.3 ScaIP ScaIP is a version of IP based on SCI. 1.1.4 ScaM AC ScaMA C, Scali Media Access Control driv er for SCI, includes a kernel mode driver and some utilities to allow fast transfer of data on SCI.

Scali Library User's Guide Version 3.0

7


Chapter 1 Introduction

8

Scali Library User's G uide Version 3.0


Chapter 2

Using ScaMPI

This chapter describes the setup, compile, link and run of a prog ram using ScaMPI. Furthermore some useful tools for debugging and profiling are briefly discussed. Please note that the "ScaMPI release notes" are av ailable in the /opt/scali/doc/ScaMPI directory.

2.1 Setting up a ScaMPI environment
2.1.1 ScaM PI environment variables The use of ScaMPI requires that some environment v ariables are defined. These are usually set in the standard startup scripts (e.g. .bashrc when using bash), but they can also be defined manually.

Name
MPI_HOME

Description
Installa tion directory. For a standa rd installation, the variable should b e set as:

export MPI_HOME=/opt/scali
LD_LIBRARY_PATH Path to dyna mic link libraries. Must b e set to include the pa th to the directory where these libraries can b e found:

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$MPI_HOME/lib
PATH Path variab le. Must be updated to include the pa th to the directory where the MPI binaries can be found:

export PATH=${PATH}:$MPI_HOME/bin

Table 2-1: Environment variables Normally, the ScaMPI library's header files mpi.h and mpif.h reside in the $MPI_HO ME/include directory.

2.2 Compiling and linking
MPI is an "Application Programming Interface"(API) and not an "Application Binary Interface"(ABI). This means that you as a main rule should recompile and relink your application when starting to use ScaMPI. But since the MPICH-implementation is

Scali Library User's Guide Version 3.0

9


Chapter 2 Using ScaMPI

widely and Sc linked linked

used we have made ScaMPI ABI compatible depending on versions of MPICH aMPI. Please check "ScaMPI release notes" for details. Having an application with mpich you should be able to just change library-path if it is dynamically or you have to relink it if you have linked it statically.

2.2.1 Compiler support ScaMPI is a C++ library built using the GNU compiler.This implies that you have to link with the G NU runtime library. Depending on the compiler used, the way to link with the ScaMPI libraries varies. Check the "ScaMPI release notes" for information on supported compilers and how linking is done. Please note that the GNU compiler, or a similar version of the C++ compiler, must be installed on your system. The G NU compilers are included in the ScaFegcs package, available for dow nload at http://www.scali.com/download. 2.2.2 Compiler flags The follow ing string must be included as compile flags (bash syntax):
"-D_REENTRANT -I$MPI_HOME/include"

2.2.3 Linker flags The follow ing string outlines the setup for the necessary link flags (bash syntax):
"-L/opt/scali/lib $CRT_BEGIN -lmpi $CRT_END"

The runtime setup CRT_BEG IN and CRT_END libraries are defined for some compilers. Please note that when linking a Fortran main prog ram, the Fortran interface library libfmpi must be included before CRT_B EGIN.

2.3 Running ScaMPI programs
Note that executables issuing ScaMPI calls cannot be started directly from a shell prompt. ScaMPI programs can either be started using the MPI monitor program mpimon, the wrapper script mpirun, or from the Scali Universe GUI [4]. 2.3.1 Naming convention When an application program is started, ScaMPI is modifying argv[0]. The following convention is used for the executable, reported on the command line using the Unix utility ps: -(mpi:@) where:

10

Scali Library User's G uide Version 3.0


2.3 Running ScaMPI programs

is the name of the application program. is the application's mpi-process rank number. is the Unix process identifier of the monitor program mpimon. is the name of the node where mpimon is running. Note that ScaMPI requires a homogenous file system image, i.e., a file system providing the same path and program names on all nodes of the Scali System. 2.3.2 mpimon - monitor program The control and start-up of an ScaMPI application is monitored by mpimon. The program mpimon has several options which can be used for optimising ScaMPI performance. Normally it should not be necessary to use any of these options. However, unsafe MPI programs [3] mig ht need buffer adjustments to solve deadlocks. Trading performance by changing communication space is best av oided if there is no compelling reason to do so. 2.3.2.1 Basic usage Normally the program is invoked as:
mpimon -- [] [ []]...

Parameter
-[ []] Name of a pplication program.

Description

Program options for the application program. Separator, ma rks end of user progra m options. Name of node and the number of mpi-processes to run on that node. The option can occur several times in the list. Mpi-processes will be given ra nks sequentia lly according to the list of node-number pairs.

Table 2-2: Basic options to mpimon 2.3.2.2 Advanced usage The complete syntax for the program:
mpimon []... [-- ]...

Scali Library User's Guide Version 3.0

11


Chapter 2 Using ScaMPI

Parameter
-

Description
-- []... []... Name of application program. Program options for the application program. Separator, signa ls end of user progra m options. [] Name of node. Given either a s a node name or as an Internet a ddress expressed in the Internet standard dot notation. Number of mpi-processes to run on node. If is omitted, one mpiprocess is started on each node specified.



Table 2-3: mpimon parameters Numeric values can be given as mpimon options in the following way: Option


Description
| : = * 1024 : = * 1024 * 1024

Table 2-4: Numeric input

12

Scali Library User's G uide Version 3.0


2.3 Running ScaMPI programs

A complete lists of available mpimon options

:

mpimon option
--automatic

Description
Separa tor, ma rks end of user program options. Set automatic-mode for process(es). Default: none Legal: `n,m,o..' = (list) or `n-m' = (range) or `all' Set numb er of barrier fanin reads. Default: 8 Set numb er of barrier fanout reads. Default: 8 Set debug-mode for process(es). Default: none Legal: `n,m,o..' = (list) or `n-m' = (range) or `all' Set debugger to start in debug-mode. Disable process timeout. Set display to use in deb ug-/manual-mode. Set dryrun-mode. Default: none Legal: `none' or `totalview' Define how to export environment. Default: export Legal: `expor t' = all or `mpi' = MP I_?? or `none' Set exact-match-mode. Set path to internal executables. Display availab le options. Set installa tion-directory. Handling of immedia tes. Default: lazy Legal: lazy, threaded, automatic Inherit userdefinable limits to processes. Initialise MPI_COMM_WORLD at startup (all channels are created).

-bar rier_fanin

-bar rier_fanout

-debug

-debugger -disable-timeout -display -dryrun

-environment

-exact_matc h -execpath -help -home -immediate_handling

-inherit_limits -init_comm_world

Table 2-5: Complete list of mpimon options

Scali Library User's Guide Version 3.0

13


Chapter 2 Using ScaMPI

mpimon option
-inter_adapters

Description
Set list of sci adapters for inter-communication. Defa ult: all Legal: `n,m,o..' = (list) or `n-m' = (range) or `all' Set threshold for inlining (in bytes) per inter-channel. Defa ult: 1030 Set buffer size (in bytes) per inter-cha nnel. Defa ult: 4K Legal: Powers of 2 Set chunk-size for inter-communication. Defa ult: 512k Legal: Multiplum of pages Set numb er of b uffers for eager inter-protocol. Defa ult: 2 Set buffer size (in bytes) for eager inter-protocol. Defa ult: 256K Legal: Powers of 2 Set threshold (in bytes) for eager inter-protocol. Defa ult: 32K Legal: Powers of 2 Set buffer-pool-size for inter-communication. Defa ult: 32M Legal: Multiplum of pages Set numb er of b uffers for tra nsporter inter-protocol. Defa ult: 4 Legal: Powers of 2 Set buffer size (in bytes) for tra nsporter inter-protocol. Defa ult: 256K Set threshold for inlining (in bytes) per intra-channel. Defa ult: 560 Set buffer size (in bytes) per intra -channel. Defa ult: 4K Legal: Powers of 2 Set chunk-size for intra-communication. Defa ult: 1M Legal: Multiplum of pages Set numb er of b uffers for eager intra -protocol. Defa ult: 2

-inter_channel_inline_threshold -inter_channel_size

-inter_chunk_size

-inter_eager_count

-inter_eager_size

-inter_eager_threshold

-inter_pool_size

-inter_transporter _count

-inter_transporter _size

-intra_channel_inline_threshold -intra_channel_size

-intra_chunk_size

-intra_eager_count

Table 2-5: Complete list of mpimon options

14

Scali Library User's G uide Version 3.0


2.3 Running ScaMPI programs

mpimon option
-intra_eager _size

Description
Set buffer size (in bytes) for eager intra-protocol. Default: 128K Legal: Powers of 2 Set threshold (in bytes) for eager intra-protocol. Default: 1M Legal: Powers of 2 Set buffer-pool-size for intra-communication. Default: 4M Legal: Multiplum of pages Set numb er of buffers for transporter intra -protocol. Default: 4 Legal: Powers of 2 Set buffer size (in bytes) for transporter intra -protocol. Default: 64K Set manual-mode for process(es). Default: none Legal: `n,m,o..' = (list) or `n-m' = (range) or `all' Read parameters from the named file. Default: none Ena ble sepa rate output for process(es). Filename: ScaMPIoutput_host_pid_rank Default: none Legal: `n,m,o..' = (list) or `n-m' = (range) or `all' Applica tion use Cra y ShMem library. Set debug-mode for submonitor(s). Default: none Legal: `n,m,o..' = (list) or `n-m' = (range) or `all' Set manual-mode for submonitor(s). Default: none Legal: `n,m,o..' = (list) or `n-m' = (range) or `all' Ena ble trace for submonitor(s). Default: none Legal: `n,m,o..' = (list) or `n-m' = (range) or `all' Ena ble statistics. Distrib ute standard in to process(es). Default: none Legal: `n,m,o..' = (list) or `n-m' = (range) or `all'

-intra_eager _thr eshold

-intra_pool_size

-intra_transpor ter_c ount

-intra_transpor ter_size

-manual

-read

-separate_output

-shmem -sm_debug

-sm_manual

-sm_trace

-statistic s -stdin

Table 2-5: Complete list of mpimon options

Scali Library User's Guide Version 3.0

15


Chapter 2 Using ScaMPI

mpimon option
-timeout

Description
Set timeout (elapsed time in seconds) for run. Legal: Positive number Ena ble trace for process(es). Defa ult: none Legal: `n,m,o..' = (list) or `n-m' = (range) or `all' Display va lues for user-options. Display version of monitor. Set working directory. Set xterm to use in debug-/manual-mode.

-trace

-verbose -Ver sion -working_directory -xterm

Table 2-5: Complete list of mpimon options

2.3.3 mpirun - wrapper script mpirun is a wrapper script for mpimon, giving MPICH [10] style startup for ScaMPI applications. Instead of the mpimon syntax, where a list of pairs of node name and number of mpi-processes is used as startup specification, mpirun uses only the total number of mpi-processes. Using scaconftool (see [4]), mpirun attempts to generate a list of operational nodes. Note that only operational nodes are selected. If no operational node is available, an error message is printed and mpirun terminates. If scaconftool is not available, mpirun attempts to use the file /opt/scali/etc/ScaConf.nodeidmap for selecting the list of operational notes. In the generated list of nodes, mpirun evenly divides the mpiprocesses among the nodes.

16

Scali Library User's G uide Version 3.0


2.3 Running ScaMPI programs

2.3.3.1 mpirun usage
mpirun []

Parameter
m pirun options

Description

Options passed on to mpimon Name of a pplication program to run. Program options passed on to the application program.

Table 2-6: mpirun format mpirun option
-np -npn -pbs -pbsparams <"params"> -p4pg

Description
Total number of mpi-processes to be started, default 2. Maximum number of mpi-processes pr. node, default np /nodes. Submit job to PBS queue system Specify PBS scasub pa rameters Use mpich compatible pgfile for program, mpi-process a nd node specification. pgfile entry: <#procs> Program name given a t comma nd line is additiona lly sta rted with one mpi-process at first node V erbose. Debug a ll mpi-processes using the GNU debugger gdb. Limit runtime to minutes. Take the list of possible nodes from Do not use scaconftool for genera ting nodelist. Ignore the /opt/ scali/etc/ ScaConf.nodear chmap file (which describes each node). Specify nodename of front-end running the scaconf server. Distribute stdin to mpi-process(es). : all (defa ult), none, or mpi-process number(s). Use nodes from pa rtition

-v -gdb -maxtime|-cpu -machinefile -noconftool -noarc hfile

-H -mstdin

-part

Scali Library User's Guide Version 3.0

17


Chapter 2 Using ScaMPI

mpirun option
-q -t

Description
Keep quiet, no mpimon printout. test mode, no mpi program is started Parameters not recognized are passed on to mpimon.

Table 2-7: mpirun options

18

Scali Library User's G uide Version 3.0


2.4 Useful tools

2.4 Useful tools
Debugging with a separate debugging session for each mpi-process requires no parallel debugger. However, debugging several mpi-processes in separate debugging sessions, may become a time consuming and tedious task. 2.4.1 Debugging with a sequential debugger ScaMPI applications can be debugg ed using a sequential debugger. By default, the GNU debugger gdb is invoked by mpimon. If another debugger is to be used, specify the debugger using the mpim on option -debugger . To set debug -mode for one or more mpi-processes, specify the mpi-process(es) to debug using the mpimon option -debug . In addition, note that the mpimon option -display should be used to set the display for the xterm terminal emulator. An xterm terminal emulator, and one debugg er, is started for each of the mpi-processes being debugged. For example, to debug two mpi-processes with rank 0 and 1 using the default gdb debugger: mpimon -display my_pc:0.0 -debug 0,1 Initially, for both mpi-process 0 and mpi-process 1, an xterm w indow is opened. Next, in the upper left hand corner of each xterm window, a message containing the application program's run parameter(s) is displayed. Typically , the first line reads Run parameters: run . The information following the colon, i.e., run is needed by both the debugger and the ScaMPI application being debugged. Finally, one debugger is started for each session. In each debugg er's xterm window , do whatever debugging action that is appropriate before the mpiprocess is started. Then, when ready to run the mpi-process, paste the run into the debugger to start running. 2.4.2 Useful built-in-tools for debugging 2.4.2.1 Using built-in segment protect violation handler If you have an application that terminates with a SIGSEGV-signal it is often useful to be able to freeze the situation instead of ex iting w hich is normal behaviour. The builtin SIGSEGV-handler can be made to do this by defining the environment-variable SCAMPI_INSTALL_SIGSEGV_HAND LE R. Legal option are: 1 The handler dump all registers and start looping. Attaching w ith a debug ger w ill then give the possibility to examine the situation giving the segment protect violation.

Scali Library User's Guide Version 3.0

19


Chapter 2 Using ScaMPI

2 the handler dump of registers but all processes will exit afterwards. All other values will disable the installation of the handler. 2.4.2.2 Using built-in sanity-check of data. If you are unsure of the quality of you SCI-network is nice to be able to run ScaMPI with extra sanity-checking of data. This is a slower mode where we make a checksum of all data both at the sender and the receiver and compare. This mode is controlled by the environment-variable SCAMPI_DATACH ECK_ENABLE and has the following options: 1 w hen error is detected report and loop 2 w hen error is detected report and exit All other value disables the sanity check. 2.4.3 Profiling ScaMPI applications When developing MPI programs it is difficult to do performance analysis. There are different tools av ailable that can be useful in detecting / analysing performance bottlenecks: · ScaMPI has built-in proprietary trace and profiling tools · Freeware that uses the standard MPI profiling interface such as MPE which is developed by the MPICH-implementors. It is part of a MPICH-distribution and we have also made it available as a part of ScaMPI-distribution · Commercial tools that collect information during a run and postprocesses and presents afterwards. One ex ample of this is Vampir from Pallas GmbH . See http://www.pallas.de for more information. The main difference between these tools is that the ScaMPI tools can be used with an existing binary while the other tools require reloading with extra libraries. 2.4.3.1 Using ScaMPI built-in trace To use built-in trace-facility you need to set the env ironment-v ariable SCAMPI_TRACE specifying what options you want to apply. The following options can be specified: (<...-list> is a semicolon-separated list of Posix-regular-expressions.)

Name
-b -s -S

Description
Trace beginning and end of ea ch MPI_call Start trace after seconds End tra ce after seconds

Table 2-8: O ptions for SCAMPI_TRACE

20

Scali Library User's G uide Version 3.0


2.4 Useful tools

Name
-c -C -p -t

Description
Sta rt trace after MPI_calls End trace a fter MPI_calls Ena ble for process(es): 'n,m,o..' = (list) or 'n-m' = (range) or 'all' Ena ble for MPI_calls in . MPI_call = 'MPI_ca ll' | 'call' Disable for MPI_calls in . MPI_call = 'MPI_ca ll' | 'call' Define format: 'timing', 'arguments', 'rate' Verbose Print this list of options

-x

-f -v -h

Table 2-8: Options for SCAMPI_TRACE By default only one line is written per MPI-call. The "-b" option is useful when trying to pinpoint what MPI-call that are started but not completed (deadlocks). The "-s/-S/c/-C" -options are nice to hav e if you have an application that runs ok for a long er period and then stop or if y ou want to have a closer look at some part of the ex ecution of the application. From time to time it is feasible to trace only one or a few of the processes. Specifying the "-p" options allows you to pick the processes y ou want to trace. All MPI-calls are enabled for tracing by default. If you want to look only on a few calls you could do that by specify ing a "-t " option or if you want to exclude some call you add a "-x " option. The "-t" will disable all tracing and then enable those calls that matches the . The matching is done using "regular-posixexpression"-syntax. "-x" will to the opposite; First enable all tracing and then disable those call matching . Examples: "-t MPI_Irecv" : Trace only immediate recv (MPI_Irecv) "-t isend;irecv;wait" :Trace only MPI_Isend, MPI_Irecv and MPI_Wait "-t MPI_[b,r,s]*send" : Trace only send-calls (MPI_Send, MPI_Bsend, MPI_Rsend, MPI_Ssend) "-t i[a-z]*" : Trace only calls beginning with MPI_I As you can see calls can be specified with or without the "MPI_"-prefix . You can also use upper- or lower-case when specifying calls. The default format of the output has the following parts: : _ where

Scali Library User's Guide Version 3.0

21


Chapter 2 Using ScaMPI

Field


Description
is rank within MPI_COMM_WORLD is name of MPI-call is name of communicator is rank within communicator used

Table 2-9: Fields in output from built-in trace

This format can be extended by using the "-f"-option. Adding "-f arguments" w ill give some more information concerning length of messages. If "-f timing " is given you get some timing-info between the and -fields. It has the following format: + S where "-f rate" w ill add some rate-information. The rate is calculated by dividing the numberName


Description
is elapsed time in seconds since returning to the a pplication from MPI_Init is elapsed execution time for current call



Table 2-10: Timespec in output from built-in trace of-by tes transferred by the elapsed time to execute the call. All parameters to -f can be abbreviated and can occur in any mix. The verbose-option(-v) will print information about which options you have selected. Normally you will not get any error-messag es concerning the options you have given. But if you add -verbose as command-line option to mpimon, errors will be printed. Rate-measurements on Alpha-processors will be inaccurate due to low-resolutiontimers.

22

Scali Library User's G uide Version 3.0


2.4 Useful tools

2.4.3.2 Using ScaMPI built-in timing To use the built-in timing you need to set the environment variable SCAMPI_TIMING specifying what options you want to apply. The following options can be specified: (<...-list> is a semicolon-separated list of Posix-regular-ex pressions.) Name
-s -c -p -f

Description
Print for intervals of seconds Print for intervals of MPI_calls Ena ble for process(es) 'n,m,o..' = (list) or 'n-m' = (ra nge) or 'all' Print a fter MPI_ca lls in : MPI_call = 'MPI_call' | 'call' Verbose Print this list of options

-v -h

Table 2-11: Options for SCAMPI_TIMING Printing of timing-information can be either at a fix ed time-interval if y ou specify "-s " or for a fixed number-of-calls-interval if you use "-c ". You can also get output after specific MPI-calls if using "-f "; See above for details how to write . The output has two parts; First a timing-part followed by a buffer-statistics-part. The first part has the following layout: All lines starts with : where : is rank w ithin MPI_COMM_WORLD. This part is included to facilitate separation of output (grep). The rest of the format has the follow ing fields: where Name
is name of MPI-call is number of calls to since last printout is sum of execution-time for ca lls to since last printout

Description

Table 2-12: Fields in output from built-in timing

Scali Library User's Guide Version 3.0

23


Chapter 2 Using ScaMPI

Name


Description
is average time-per-call for ca lls to since la st printout is number of calls to is sum of execution-time for calls to is average time-per-call for ca lls to



Table 2-12: Fields in output from built-in timing After all detail-lines (one per MPI-call which has been called since last printout), there will be a line with the sum for all calls follow ed by a line giving the overhead introduced when obtaining the timing-measurements. The second part containing the buffer-statistics has two types of lines; one for receives and one for sends. "Receiv e-lines" has the following fields: recv from (): where Name


Description
is communicator b eing used is rank within is rank within is rank within MPI_COMM_WORLD

Table 2-13: Fields in"recv"-lines from built-in timing

24

Scali Library User's G uide Version 3.0


2.4 Useful tools

"Send-lines" has the following fields: send to (): where Name


Description
is communica tor b eing used is ra nk within is ra nk within is ra nk within MPI_COMM_WORLD

Table 2-14: Fields in"send"-lines from built-in timing The are as follows: !!!!!!! where Name


Description
is number of sends/receives is average length of messages in bytes is number of messa ges sent/received using zero-bytes-mechanism is number of messa ges sent/received using inline-mecha nism is number of messages sent/received using eagerbuffer-mechanism is number of messages sent/received using transporter-mechanism





Table 2-15: Common filelds in output from built-in timing For more details on the different mechanisms, see "ScaMPI Description". Timing-measurements on Alpha-processors will be inaccurate due to low-resolutiontimers.

Scali Library User's Guide Version 3.0

25


Chapter 2 Using ScaMPI

2.4.3.3 Using ScaMPI built-in cpu-usage To use built-in cpu-usag e-timing you need to set the environment variable SCAMPI_CPU_USAGE . The information displayed is collected with the system-call "times"; see man-pages for more information. The output has two different blocks. The first block contains cpu-usage by the submonitors on the different nodes. One line is printed for each submonitor followed by a sum-line and an average-line. The second block consists of one line per process follow ed by a sum-line and an average-line. 2.4.4 Profiling with ScaMPE The ScaMPE libraries are adapted versions of the MPE libraries from MPICH [10]. An executable program linked with one of the ScaMPE libraries libtmpi, liblmpi or libampi collects performance data during runtime. Normally, the libraries are installed in the directory /opt/scali/contrib/lib, and the upshot tool, described below, is installed in /opt/scali/contrib/bin. The main components of ScaMPE are: · A set of routines for creating log files for examination by the visualization tool upshot. · Trace or real time animation of MPI calls. · A shared display parallel X graphics library. 2.4.4.1 Linking an ScaMPI application Profiling using one of the ScaMPE libraries is achieved by linking with the appropriate ScaMPE library before the standard ScaMPI library libmpi. · Trace MPI calls - library libtmpi To trace all MPI calls, apply -ltmpi. Each MPI call is preceded by a line that contains the rank in MPI_COMM_WOR LD of the calling process, and followed by another line indicating that the call has completed. Most send and receive routines also indicate the v alues of count, tag, and partner (destination for sends, source for receives). Output is to standard output stdout. · Generate log file - library liblmpi To generate an upshot style log file of all MPI calls, apply -llmpi. When the application is about to finish, an information message is printed to stdout, and the trace data is written to a log file for post-processing. The name of the log file, with suffix .alog, is created based on the argument provided in argv[0]. Note that when an application program is started, ScaMPI is modify ing argv[0], as described in section 2.3. However, the log file name is always created as executablename-.alog. For example, if the program being

26

Scali Library User's G uide Version 3.0


2.4 Useful tools

profiled is sendrecv, the generated log file is sendrecv-.alog. · Real time animation - library libampi To produce a real-time animation of the program, apply -lampi -lmpe -lm -lX11. Note that this requires the MPE graphics in libmpe, and that X11 Window System operations are used. To link the X11 libraries (libX11), it may be necessary to provide a specific path for the libraries. In addition, note that to resolve some mathematics references used, the standard library libm must be included in the link command line. For a description of the MPE graphic routines, see the MPICH documentation [10]. Notes for Fortran users For a Fortran prog ram, it is necessary to include the Fortran wrapper library libfmpi ahead of the profiling libraries. This allow s C routines to be used for implementing the profiling libraries for use by both C and Fortran programs. For example, to generate an upshot style log file in a Fortran prog ram, the libraries are included in the order lfmpi -llmpi -lm.

2.4.4.2 Examine the generated log file - upshot To examine a log file generated using liblmpi, the parallel program visualization tool upshot can be used to analyse the program performance. Note that upshot uses the environment variable $D ISPLAY to select the display to use. Start the visualization tool:
/opt/scali/contrib/bin/upshot

When started, browse and select the appropriate log file to be analysed. For more information, see the document named README_UPSHOT in the directory /opt/scali/contrib/doc/ScaMPE. If upshot is not av ailable, any other visualization tool, e.g., nupshot, that understands the log file format can be used instead. For more information, see the MPICH documentation [10].

Scali Library User's Guide Version 3.0

27


Chapter 2 Using ScaMPI

2.5 An example program
When the ScaMPItst package has been installed, the source code and the executable code, for both the hello-world example prog ram and a number of test programs, are located under the /opt/scali/examples/src and the /opt/scali/examples/bin directories. A description of each program in the package can be found in the READ ME file, located in the /opt/scali/doc/ScaMPItst directory.

As ex amples, the MPI program named hello-world is used. It exists as a C program in the file hello-world.c, and as a Fortran program in the file hello-world.f. They are compiled and linked using G NU compilers. Before compilation, it is assumed that the BASH shell environment variable has been properly defined. In addition, ScaMPI must have been installed and function correctly. 2.5.1 Hello-world.c - source in C
#include #include "mpi.h" void main(int argc, char** argv) { int rank; int size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); printf("Hello-world, I'm rank %d; Size is %d\n", rank, size); MPI_Finalize();

2.5.2 Hello-world.f - source in Fortran
program hello_world implicit none include 'mpif.h' integer rank,size,ierr call call call write & "; call end mpi_init(ierr); mpi_comm_rank(MPI_COMM_WORLD,rank,ierr); mpi_comm_size(MPI_COMM_WORLD,size,ierr); (*,'(A,I3,A,I3)') "Hello-world, I'm rank ",rank, Size is ",size mpi_finalize(ierr);

28

Scali Library User's G uide Version 3.0


2.6 MPI test programs

2.5.3 Compiling
% gcc -c -D_REENTRANT -I$MPI_HOME/include hello-world.c % g77 -c -D_REENTRANT -I$MPI_HOME/include hello-world.f

2.5.4 Linking
% gcc hello-world.o -L$MPI_HOME/lib -lmpi -o hello-world % g77 hello-world.o -L$MPI_HOME/lib -lfmpi -lmpi -o hello-world

2.5.5 Running Start the hello-world program on the 3 nodes named nodeA, nodeB and nodeC.
% mpimon hello-world -- nodeA 1 nodeB 1 nodeC 1

The hello-world program should produce the following output:
Hello-world, I'm rank 0; Size is 3 Hello-world, I'm rank 1; Size is 3 Hello-world, I'm rank 2; Size is 3

2.6 MPI test programs
The ScaMPItst package contains a collection of MPI test programs for ScaMPI. The following sections give a brief description of some of the test programs, which can be used to measure basic MPI performance. To re-compile any of the test programs, you may use the included Makefile found in the appropriate /opt/scali/exam ples/src directory. 2.6.1 Producer - a producer-consumer MPI test program Producer is a simple producer-consumer program. Mpi-processes w ith rank 0, 1, 2, ..., n/2-1 send data w hile mpi-process n/2, n/2+1, ...,n-1 receive data. Mpi-process 0 will send to mpi-process n-1, mpi-process 1 will send to mpi-process n-2, and so on. The producer program parameters are: -l i i is the loop count. -n j j is the number of bytes to transfer for each send operation. As a first test, run producer between any pair of two nodes, nodeX and nodeY:
% mpimon producer -l 1 -n 1024 -- nodeX nodeY

Scali Library User's Guide Version 3.0

29


Chapter 2 Using ScaMPI

A sing le mpi-process is started on each node, and a single messag e of size 1024 bytes are transferred from the mpi-process on nodeX to the mpi-process on nodeY. The program should return TEST CO MPLETE. Repeat the test for all pairs of nodes. N is the number of nodes (must be an even number for this test).
% mpimon producer -l 1 -n 1024 -- ...

The program should return TEST CO MPLETE. 2.6.2 Bandwidth - a bandwidth MPI test program Bandwidth is a prog ram to measure bandwidth for various message sizes between two mpi-processes. First one-way bandwidth and the latency for a zero byte message are measured, then the ping-pong (two-way) bandwidth and latency are measured. Measure the bandw idth between any pair of nodes, nodeX and nodeY, by running:
% mpimon bandwidth -- nodeX nodeY

2.6.3 Bidirect - a bidirectional MPI test program. Bidirect tests uni- and bi-directional traffic between a given number of nodes. The program may be run between two nodes, nodeX and nodeY, using the run_bidirect script as:
% run_bidirect nodeX nodeY

or between a given set of nodes, nodeX nodeY... nodeZ, using another script as:
% run_permutated_bidirect nodeX nodeY... nodeZ

The run_permutated_bidirect script will test uni- and bi-directional traffic betw een all permutations of node combinations.

30

Scali Library User's G uide Version 3.0


Chapter 3
3.1 General description

Description of ScaMPI

ScaMPI consists of libraries to be linked and loaded with the user application program(s) and a set of executables which control the start-up and execution of the user application program(s). 3.1.1 ScaM PI libraries Name
libmpi libfmpi

Description
Standard library containing the C A PI. Library containing the Fortra n API wrappers.

Table 3-1: Libraries 3.1.2 ScaM PI executables A number of executable programs are included in ScaMPI. 3.1.2.1 mpimon - monitor program mpimon is a monitor program which is the user's interface for running the application program. 3.1.2.2 mpisubmon - submonitor program mpisubmon is a submonitor program which controls the execution of application programs. One submonitor program is started on each node per run. 3.1.2.3 mpiboot - bootstrap program mpiboot is a bootstrap program used when running in manual-/debug-mode. 3.1.2.4 mpid - daemon program mpid is a daemon program running on all nodes that can run ScaMPI. mpid is used for starting the mpisubmon programs (to avoid using Unix facilities like the remote shell rsh). mpid is started automatically when a node boots, and must run at all times.

Scali Library User's Guide Version 3.0

31


Chapter 3 Description of ScaMPI

3.2 Starting ScaMPI application programs
ScaMPI uses socket communication for control purposes. Schematically, start-up of application programs in a Scali System is performed as described in the following sections. 3.2.1 Application start-up - phase 1 · Parameter control. mpimon does as much control of the specified options and parameters as possible. The userprog ram names are checked for validity, and the nodes are, using sockets, contacted to ensure they are responding and that mpid is running. · Connecting to nodes. mpimon establishes a connection to the mpid daemon on each node specified, and transfers basic information to enable the daemon to start the submonitor mpisubmon.

mpimon

mpid node

mpid node

mpid node

Figure 3-1: Application start-up - phase 1 3.2.2 Application start-up - phase 2 · Starting submonitors. On each node, mpid starts the submonitor mpisubmon. · Transferring control information. Each submonitor establishes a connection to mpimon. C ontrol information are exchanged between each mpisubmon and mpimon to enable mpisubmon to start the specified userprograms (mpi-processes). · Creating shared memory On each node, mpisubmon creates memory segments to be shared between the interconnected nodes.

32

Scali Library User's G uide Version 3.0


3.2 Starting ScaMPI application programs

mpimon

mpisubmon

mpisubmon

mpisubmon

mpid node

mpid node

mpid node

Figure 3-2: Application start-up - phase 2

3.2.3 Application start-up - phase 3 · Starting m pi-processes. On each node, mpisubmon starts all the mpi-processes to be executed. Processes start and enter MPI_Init(). · Mpi-processes synchronize. Upon receipt of all control information, the processes will via the local mpisubmon inform mpimon that they are ready to run. When all processes are ready, mpimon will return a `start running' message to all the processes. · MPI-processes return from MPI_Init() and start to run. The user program(s) takes control.

Scali Library User's Guide Version 3.0

33


Chapter 3 Description of ScaMPI

mpim on

mpisubmon

mpisubmon

mpisubmon

process process process process process mpi-process node

process process process process process mpi-process node

process process process process process mpi-process node

Figure 3-3: Application start-up - phase 3

3.3 Stopping ScaMPI application programs
Termination of application programs in a Scali System are performed as outlined below. · Mpi-processes enters MPI_Finalize(). Each process signals, via its local mbisubmon, to mpimon that it has entered MPI_Finalize(), and it is now w aiting . · Mpi-processes synchronize Processes wait for an "all stopped message" from mpimon. The message is transmitted via mpisubmon when all processes are waiting in MPI_Finalize(). · Mpi-processes leave MPI_Finalize(). Processes terminate, each mpisubmon releases shared memory segments and exits, and finally mpimon terminates.

34

Scali Library User's G uide Version 3.0


3.4 Communication protocols

3.4 Communication protocols
In ScaMPI, the communication protocol (inlining, eagerbuffering, transporter) used to transfer data between a sender and a receiv er depends on the size of the messag e to transmit, see figure below.

Increasing message size

Transporter protocol

Eagerbuffering protocol Inlining protocol Figure 3-4: Thresholds for different communication protocols

The various communication protocols used, are briefly outlined in the following sections.

Scali Library User's Guide Version 3.0

35


Chapter 3 Description of ScaMPI

3.4.1 Inlining protocol The inlining protocol is used when small messages are to be transferred.

Sender

Receiver

Header ring buffer (Channel)

Figure 3-5: Inlining protocol

When the inlining protocol is used, the application's data is included in the message header. The inlining protocol utilizes one or more channel ringbuffer entries. The actual threshold for the inlining protocol can be set as described in section 3.5.1. The inlining protocol is selected when: 0 <= messag e size <= channel_inline_threshold.

36

Scali Library User's G uide Version 3.0


3.4 Communication protocols

3.4.2 Eagerbuffering protocol The eagerbuffering protocol is used when medium size messages are to be transferred.

EagerBuffer pool

Sender

Receiver

Header ringbuffer (Channel)

Figure 3-6: Eagerbuffering protocol

The protocol uses a scheme where the buffer resources, being allocated by the sender, are released by the receiver, without any explicit communication between the two communicating partners. The eagerbuffering protocol utilizes one channel ring buffer entry for the message header, and one eagerbuffer for the application data being sent. The eagerbuffering protocol is selected when: channel_inline_threshold < message size <= eag er_size.

Scali Library User's Guide Version 3.0

37


Chapter 3 Description of ScaMPI

3.4.3 Transporter protocol The transporter protocol is used when large messages are to be transferred. Step 1: Sender Receiv er

Header ringbuffer (Channel) Step 2: Sender Receiv er

Transporter selection field Step 3: Sender Transporter ringbuffer Receiv er

Figure 3-7: Transporter protocol

Initially (step 1), the protocol only transmits the message header. Once the receiv er is ready to accept data (step 2), the sender is informed. Finally (step 3), the application's data is transferred from the sender to the recipient in the transporter ringbuffer. The transporter protocol utilizes one channel ringbuffer entry for the message header, and transporter buffers for the application data being sent. The transporter protocol provides for fragmentation and reassembly of large messages, if necessary, for messag es whose size is larger than the size of the transporter ringbuffer-entry (transporter_size). The transporter protocol is selected when: message size > eager_size.

38

Scali Library User's G uide Version 3.0


3.5 Communication resources

3.5 Communication resources
All resources (buffers) used by ScaMPI reside in shared memory, and are allocated by mpisubmon on demand from the sender mpi-process. ScaMPI uses a on demand scheme for allocating resources. On demand means that buffers are not allocated until needed. To get a list of the resource settings, pass the -verbose option to mpimon. mpisubmon operates on two separate buffer pools suitable for sharing - both pools in shared memory. One pool (local shared memory) provides resources for intra-node communication, and the other pool (SCI shared memory) provides resources for internode communication. The siz e of each buffer pool, and the size of each chunk may be set using options to mpimon. The pool size limits the total amount of shared memory, and the chunk size limits the maximum block of memory that can be allocated as a single buffer. To set the pool size and the chunk size limits, use mpimon and specify: -intra_pool_size -intra_chunk_size -inter_pool_siz e -inter_chunk_size to co to co to co to co set mm set mm set mm set mm the buffer unication the chunk unication the buffer unication the chunk unication pool size for intra-node size for intra-node pool size for inter-node size for inter-node

Sections 3.5.1 - 3.5.3 outlines the various types of resources (channel, eagerbuffer, transporter) being used, and lists the mpimon options used to enforce specific buffering. Howev er, it is normally not necessary to set the buffer parameters. Automatic buffer manag ement is performed by ScaMPI, as described in the `automatic buffer management' section.

Scali Library User's Guide Version 3.0

39


Chapter 3 Description of ScaMPI

3.5.1 Channel buffer For a sender-receiver pair, one channel ring buffer is used for each communicator.

(5 com municators) Channel Channel Cha er Ringbuffnnel l Channe Ringbuffer Channel Ringbuffer Ringbuffer Ringbuffer

Sender

Receiver

F igure 3-8: Channel resource

The ringbuffer is divided into equally sized entries. The size varies differs for different architectures/SCI-hardw are; see "ScaMPI release notes" for details. An entry in the ringbuffer, which is used to hold the information forming the message envelope, is reserved each time a message is being sent, and is utilized by both the inline protocol, the eagerbuffering protocol, and the transporter protocol. In addition, one ore more entries are utilized by the inline protocol for application data being transmitted. To force the channel resource definitions, use mpimon and specify: -intra_channel_size to set the ringbuffer size (in bytes) per intra-channel -inter_channel_size to set the ringbuffer size (in bytes) per inter-channel To set the channel threshold definitions, use mpimon and specify : -intra_channel_inline_threshold to set threshold for inlining per intrachannel -inter_channel_inline_threshold to set threshold for inlining per interchannel

40

Scali Library User's G uide Version 3.0


3.5 Communication resources

3.5.2 Eagerbuffer buffer For a sender-receiver pair, one, and only one, eagerbuffer buffer is used.

Eagerbuffer Sender Receiver

Usedflags

Figure 3-9: Eagerbuffer buffer

An eagerbuffer buffer is allocated when medium size messages are to be transferred, and is utilized by the eagerbuffering protocol. To chang e the eagerbuffer resource definitions, use mpimon and specify: -intra_eager_size -intra_eager_count -inter_eager_size -inter_eager_count to co to co to co to co set the buffer size mmunication set number of buf mmunication set the buffer size mmunication set number of buf mmunication (in bytes) for intra-node fers for intra-node (in bytes) for inter-node fers for inter-node

Scali Library User's Guide Version 3.0

41


Chapter 3 Description of ScaMPI

3.5.3 Transporter buffer For a sender-receiver pair, one, and only one, transporter buffer set is used.

Transporter Ringbuffer Sender Receiver

Selection

Figure 3-10: Transporter buffer

A transporter buffer is allocated when large messages are to be transferred, and is utilized by the transporter protocol. The buffer consists of equally sized entries arranged as a ringbuffer. To change the transporter resource definitions, use mpimon and specify: -intra_transporter_size -intra_transporter_count -inter_transporter_size -inter_transporter_count to set the bufferentry size (in bytes) for intra-node communication to set number of entries in buffer for intranode communication to set the bufferentry size (in bytes) for internode communication to set number of entries in buffer for internode communication

42

Scali Library User's G uide Version 3.0


Chapter 4

Tips & Tricks for ScaMPI

This chapter is the place to start when something seems to go wrong running your ScaMPI programs. If you have any problems with ScaMPI, first check the (not yet complete) list of common errors and their solutions. An updated list of ScaMPI Frequently Asked Questions are posted in the Support section at http://www.scali.com. If you cannot find a solution to the problem(s), please read this chapter before contacting support@scali.com. Currently, the following sections are by no means complete. Problems reported to Scali will eventually be included in appropriate sections. Thus, please send your relevant remarks by e-mail to support@scali.com.

4.1 Application program notes
4.1.1 MPI_Probe() and MPI_Recv() During development and test of ScaMPI, we have run into several application programs with the follow ing code sequence: while (...) { MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, comm, sts); if (sts->MPI_TAG == SOME_VALUE) { MPI_Recv(buf, cnt, dtype, MPI_ANY_SOURCE, MPI_ANY_TAG, comm, sts); doStuff(); } doOtherStuff(); } For MPI implementations that have one, and only one, receiv e-queue for all senders, the program's code sequence works ok. However, the code will not work as expected with ScaMPI. ScaMPI utilizes one receive-queue per sender (inside each mpi-process). Thus, a message from one sender can bypass the messag e from another sender. In the time-gap between the completion of MPI_Probe() and before MPI_Recv() matches a message, another new message from a different mpi-process could arrive, i.e., it is not certain that the message found by MPI_Probe() is identical to one that MPI_Recv() matches.

Scali Library User's Guide Version 3.0

43


Chapter 4 Tips & Tricks for ScaMPI

To make the program work as expected, the code sequence should be corrected to: while (...) { MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, comm, sts); if (sts->MPI_TAG == SOME_VALUE) { MPI_Recv(buf, cnt, dtype, sts->MPI_SOURCE, sts->MPI_TAG, comm, sts); doStuff(); } doOtherStuff(); } 4.1.2 Unsafe MPI programs Because of different buffering behaviour, some programs may run with MPIC H, but not with ScaMPI. Unsafe MPI programs may require resources that are not always guaranteed by ScaMPI, and deadlock might occur (since ScaMPI use spinlocks, these might seem to be livelocks). If you want to know more about how to write portable MPI programs, see for example [2]. A typical example that will not work with ScaMPI (for long messages): while (...) { MPI_Send(buf, cnt, dtype, partner, tag, comm); MPI_Recv(buf, cnt, dtype, MPI_ANY_SOURCE, MPI_ANY_TAG, comm, sts); doStuff(); } To get this example to work with ScaMPI, the MPI_Send() must either be replaced by using MPI_Isend() and MPI_Wait(), or the whole construction should be replaced using MPI_Sendrecv() or MPI_Sendrecv_replace().

4.2 Namespace pollution
The ScaMPI library , being written in C++, have all its class names prefixed with MPI_. Depending on the compiler used, the user may run into problems if he/she has C++ code using the same prefix MPI_. In addition, there exist a few global variables that could cause problems. All these functions and v ariables are listed in the include files mpi.h and mpif.h. Normally , these files are installed in /opt/scali/include.

44

Scali Library User's G uide Version 3.0


4.3 Error and warning messages

Due to the fact that ScaMPI doesn't have fixed its OS routines to specific libraries, it will be good programming practise to avoid using OS functions as application function names. Naming routines or global variables as send, recv, open, close, yield, internal_error, failure, service or other OS reserved names may result in an unpredictable and undesirable behaviour.

4.3 Error and warning messages
4.3.1 User interface errors and warnings User interface errors are problems with the environment setup causing difficulties for mpimon when starting a ScaMPI program. mpimon will not start before the environment is properly defined. These problems are usually easy to fix, by giving mpimon the correct location of some executable. The error message provides a straight forward indication of what to do. Thus, only particularly troublesome user interface errors will be listed here. Using the -verbose option enables mpimon to print more warnings. 4.3.2 Fatal errors Upon a fatal error, ScaMPI prints an error message before calling MPI_Abort() to shut dow n all mpi-processes.

4.4 When things don't work - troubleshooting
This section is meant as a starting point to help debugging . The main focus is on locating and repairing faulty hardware and software setup, but can also be helpful in getting started after installing a new system. For a description of the Scali Universe GUI, see the Scali System Guide [4]. 4.4.1 Standard input and ScaM PI The -stdin option specifies which mpi-process rank should receiv e the input. You can in fact send stdin to all the mpi-processes with the all argument, but this requires that all mpi-processes read the exact same amount of input. The most common way of doing it is to send all data on stdin to rank 0: mpimon -stdin 0 myprogram -- node1 node2 ... < input_file Note that default for -stdin is none. 4.4.2 Why doesn't my program start to run? mpimon: command not found. Include /opt/scali/bin in the PATH env ironment v ariable.

s
J

Scali Library User's Guide Version 3.0

45


Chapter 4 Tips & Tricks for ScaMPI

mpimon can't find mpisubmon. Set MPI_HOME=/opt/scali or use the -execpath option. The application has problems loading libraries (libsca*). Update the LD_LIBRARY_PATH to include /opt/scali/lib.

Incompatible m pi versions. mpid, mpimon , mpisubmon and the libraries all have version variables that are checked at start-up. Set the environment variable MPI_HOME correctly Restart mpid because a new version of ScaMPI is installed w ithout restart of mpid Reinstall ScaMPI because a new v ersion of ScaMPI was not cleanly installed on all nodes.

4.4.3 Why doesn't mpid start mpid opens a socket and assigns a predefined mpid port number, see /etc/services, to the end point. If mpid is terminated abnormally, the mpid port number cannot be reused until a system defined timer has expired. Use netstat -a | grep m pid to observe when the socket is released. When the socket is released, restart mpid ag ain. 4.4.4 Interconnect problems 4.4.4.1 Routing The program terminates with an ICMS_NO_RESPONSE error message This happens when one or more mpi-processes are unable to create a remote memory mapping to another node within a (long ) period of time. Check if all relevant nodes are alive by issuing any command with scash, e.g., /opt/scali/bin/scash -p nodename.

s s s s s s s
46

J J J J J J J J J J

Set workingdirectory failed ScaMPI assumes homogenous file-structure. If you start mpimon from a directory that is not available on all nodes you must set SCAMPI_WORKING_DIRECTORY to point to a directory that are available on all nodes. ScaMPI uses wrong interface for tcp-ip on frontend with more than one interface Set SCAMPI_NODENAME to hostname of correct interface. MPI_Wtime gives strange values ScaMPI uses a hardware-supported highprecision timer for MPI_Wtime. This timer can be disabled by SCAMPI_DISABLE_HPT=1

Scali Library User's G uide Version 3.0


4.4 When things don't w ork - troubleshooting

Check if SCI network routing is properly set with /opt/scali/sbin/scaconftool (command: sciping OK), or use the Scali Universe GUI.

4.4.4.2 Bad clean up A previous ScaMPI run has not terminated properly. Check for mpi-processes on the nodes using /opt/scali/bin/scaps. Use /opt/scali/sbin/scidle Use /opt/scali/bin/scash to check for leftover shared memory segments on all nodes (ipcs for Solaris and Linux ). Note that core dumping takes time... 4.4.4.3 Space overflow The application have required too much SCI or shared memory resources. Your mpimon pool-size specifications are too large. Number of communicators in the program is higher than expected when doing automatic buffer calculations. Since memory by default is allocated in large chunks, try to reduce the chunk-size parameter to mpimon (use mpimon -verbose to get current buffer setting s).

4.4.5.2 SCI interconnect failures The program terminates with an ICMS_* message An SCI problem has occurred, find out more using the SCI diagnostics helper: /opt/scali/bin/sciemsg . Reloading of SCI drivers and rerouting y our sy stem may be necessary. Contact your local System Administrator if assistance is needed. The interconnect diagnostic in the Scali Universe G UI and the SCI documentation in the Scali System Guide may help you locate the problem. Problems and fixes will be included in the FAQ on http://www.scali.com . If there is a SCI problem needing attention, please contact support@scali.com.

4.4.5.3 General problems Are you reasonable certain that your algorithms are MPI safe? Check if ev ery send has a matching receive.

s s s s s
J J J J J J J J J

J

4.4.5 Why does my program terminate abnormally? 4.4.5.1 Core dump The application core dumps. Use a debugg er to locate the point of violation. The application may need to be recompiled to include symbolic debug information (-g for most compilers). Define SCAMPI_INSTALL_SIGSEGV_HANDLER=1 and attach to the failing process w ith debugger.

Scali Library User's Guide Version 3.0

47


Chapter 4 Tips & Tricks for ScaMPI

The program just hangs Try starting the program with -init_comm_world specified; if it doesn't start, there is a buffer allocation problem. Further information is available in the `How do I control SCI and local shared memory usage?' section. If the application has a large degree of asynchronicity, try to increase the channelsize. Further information is available in the `H ow do I control SCI and local shared memory usage?' section. Are y ou really sure that your alg orithms are MPI safe? The program terminates without an error message Investigate the core file, or rerun the program in a debug ger.

4.4.6 How do I control SCI and local shared memory usage? Adjusting ScaMPI buffer sizes Note that forcing size parameters to mpimon is usually not necessary. This is only a means of optimising ScaMPI to a particular application, based on knowledge of communication patterns. For unsafe MPI programs it may be required to adjust buffering to allow the program to complete. How do I control SCI and local shared m emory usage? The eager buffers are used for small messages, while the transporter buffers are used for handling large messages (larg er than eag er size). The channel buffers is a send queue where each entry is 64 bytes, i.e.in a 8k buffer there is room for 128 outstanding requests. The function of the various buffers is outlined in section 3.5. All buffers are created when needed (i.e., when tried used for the first time), or at start up when -init_comm _world is specified. The buffer space required by a communication channel is approximately: channel = (2 * channel-size * communicators) + (transporter-size * transporter-count) + (eager-size * eager-count) + 512 (give-or-take-a-few-bytes)

s s s s
48

J J J

Note: Messages up to 560 bytes (the upper limit can be set using the option channel_inline_threshold to mpimon) g et inlined in the channel buffer. If you frequently use short messages, increasing the channel-size beyond 4k bytes might be a good idea. 4.4.6.1 Automatic buffer management The communicators parameter depends on the application (assumed to be two in the automatic approach). If more communicators than expected by the buffer size calculations are used, the application may run out of shared memory. To overcome this, reduce the chunk-size. The pool-size is a limit for the total amount of shared memory.

Scali Library User's G uide Version 3.0


4.5 How to optimize MPI performance

The automatic buffer size computations is based on a full connectivity , i.e., all communicating with all others. If all mpi-process P in a program communicate with all the other mpi-processes, each mpi-process will communicate with P_intra mpiprocesses intra node (it-self inclusive) and (P - P_intra) mpi-processes inter node. Given a total pool of memory dedicated to communication, each communication channel will be restricted to use a partition of only: inter_partition = inter_pool_size / (P_intra*(P-P_intra)) intra_partition = intra_pool_size / (P_intra * P_intra) The automatic approach is to downsize all buffers associated with a communication channel until it fits in its part of the pool. The chunk size sets the size of each individual allocated memory segment. The automatic chunk size is calculated to wrap a complete communication channel.

4.5 How to optimize MPI performance
There is no universal recipe for getting good performance out of a message passing program. Here are some do's and don't's for ScaMPI. 4.5.1 Performance analysis. Learn about the performance behaviour of your MPI application on a Scali System by using a performance analy sis tool. The freely available ScaMPE profiling library may be used with ScaMPI. For more information, please see section 2.4.3. 4.5.2 Using MPI_Isend(), MPI_Irecv(). If communication and calculations does not overlap, using immediate calls, e.g., MPI_Isend() and MPI_Irecv(), are usually performance ineffective. 4.5.3 Using MPI_Bsend(). Using buffered send, e.g., MPI_Bsend(), usually degrade performance significantly compared to their unbuffered relatives. 4.5.4 Avoid starving mpi-processes - fairness. MPI programs may, if not special care is taken, be unfair and may starve mpiprocesses, e.g., by using MPI_Waitany() as illustrated for a client-server application in example 3.15 & 3.16 in the MPI 1.1 standard [1]. Fairness can be enforced, e.g ., by use of several tags or separate communicators.

Scali Library User's Guide Version 3.0

49


Chapter 4 Tips & Tricks for ScaMPI

4.5.5 Using processor-pow er to poll. ScaMPI is implemented using poll when w aiting for communication to terminate. This is efficient when this period is short or if you don't have anything else to use the processorpow er for. In threaded application w ith irregular communication patterns you probably hav e other threads that could make use of the processor. In this case performance may increase if you enable the backoff-polling-strategy built into ScaMPI. It functions like this: After waiting a short period (idle time) we start backing off using systemcall nanosleep to release processor. The nanosleep period starts at a minimum and it doubles for each call until it reaches a maximum. It is controlled by a set of environment-variables: SCAMPI_BACKOFF_ENABLE SCAMPI_BACKOFF_IDLE SCAMPI_BACKOFF_MIN SCAMPI_BACKOFF_MAX turns the mechanism =n defines idle-period Default 20 ms = n defines minimum Default 10 ms = n defines maximum Default 100 ms on t o n ms backoff-time in ms backoff-time in ms

4.5.6 Communication buffer adaption If the communication behaviour of the application is known, explicitly giving buffersize setting s to mpimon to match the requirement of the application, will in most cases improve performance. Example: Application sending only 900 by tes messages. Set channel-inline-threshold 964 (64 added for alignment) and increase the channel-size significantly (32-128 k). Note: the channel-inline-threshold can not be increased beyond 1023. Setting eager-size 1k and eager-count high (16 or more). Note: If all messages can be buffered, the transporter-{size, count} can be set to low values to reduce shared memory consumption. 4.5.7 Reorder network traffic to avoid conflicts Many-to-one communication may introduce bottlenecks. Zero byte messag es are low-cost. In a many-to-one communication, performance may improve if the receiver sends ready-to-receive tokens (in the shape of a zero-byte messag e) to the mpi-process wanting to send data.

J J

50

Scali Library User's G uide Version 3.0


4.6 Benchmarking

4.6 Benchmarking
Benchmarking is that part of performance evaluation that deals with the measurement and analysis of computer performance using various kinds of test programs. Benchmark figures should always be handled with special care when compared to similar results. 4.6.1 How to get expected performance · Improving performance for short runs. By default, communication buffers are allocated w hen requested the first time. To eliminate this startup time from your measurement either run a warm-up phase before doing the actual measurement or use the parameter -init_comm _world to mpimon to allocate communication buffers between all pairs of mpi-processes. · Caching the application program on the nodes. For benchmarks with short execution time, total execution time may be reduced when running it repetitive. For large configurations, copy ing the application to the local file system on each node will reduce startup latency and improve disc bandwidth. · The first iteration is (very) slow. The mpi-processes in an application are not started simultaneously. Inserting an MPI_Barrier() before the timing loop will eliminate this. To reduce setup time after MPI_Init(), specify the parameter -init_comm_world to mpimon. 4.6.2 Memory consumption increase after warm-up Remember that group operations (MPI_Comm _{create, dup, split }) may involve creating new communication buffers. If this is a problem, decrease the chunk-size as described in section 4.4.

Scali Library User's Guide Version 3.0

51


Chapter 4 Tips & Tricks for ScaMPI

52

Scali Library User's G uide Version 3.0


Chapter 5
Scali's compatibility library for Cray /SGI ShMem.

ScaShmem

5.1 Description
The Scali's compatibility library covers most of the Cray/SG I ShMem application programmers interface (exceptions are listed in release-notes). The library is made to enable running of applications previously limited to Cray/SGI machines in a workstation environment. With the favourable price and availability of memory for workstations, memory intensive applications may in particular benefit from this library. This implementation of Scaliґs Cray/SGI ShMem compatibility library is layered on top of ScaMPI imposing some restrictions on the usage. The ShMem communication layer uses ScaMPI's thread-hot and -safe feature and creates an additional serv er thread to handle remote requests, i.e. a client- server architecture. For new applications it is therefore recommended to use ScaMPI as your communication library to utilize the full performance potential of Scali products.

5.2 Application porting to ScaShmem
Using of the Shmem compatibility library means running on another architecture than the Cray/SGI machine which your application initially was made for. The siz e of data types on your Cray/SGI system and y our cluster may differ; e.g. processors employed in the T3E are true 64 bit processor while x86 PCs are 32 bit, and changes may hav e to be made to the source code before it can run on your new architecture. Make sure that your ShMem code is tolerant to these differences. Some examples of do & don't: · the T3E is 64 bit, all such variables must be replaced with on a 32 bit machine to operate with the same precision · make consistent use of header files (shmem.h vs. shmem.fh) · use portable maths when comparing with T3E (i.e. -K ieee -pc 64) · enable flag s related to the T3E implementation as the source code may support other architectures as well. · Fortran code with careful declaration of variables' size (kind=4 etc according to Cray compiler practice) and corresponding naming of the shmem procedure calls simplifies the porting process

Scali Library User's Guide Version 3.0

53


Chapter 5 ScaShmem

· Zero initialization is not supplied by all Fortran and C compilers for workstations.

5.3 Features and limitations
Please refer to the ScaShMem release notes (/opt/scali/doc/ScaShMem/RELEASE_NOTES) for further details which may not have made it into this manual. 5.3.1 Communication initialization and termination Since Cray/SGI ShMem does not have explicit start and stop function calls, the MPI and the server thread is started when the first call to a shmem_* function is performed. 5.3.1.1 Communication initialization If your application has processes that rely on remote data before the corresponding process have performed a communication call, and the corresponding process have performed a communication call, you will hav e to either call the nonstandard function shmem_start() or do a shmem_barrier_all(). 5.3.1.2 Communication termination All application processes are stopped when the first process terminates. It is therefore recommended to end y our application with a shmem_barrier_all(). Your application will terminate with messages aka
-- mpimon --- Aborting run after process- terminated abnormally Childprocess exited with exitcode 216 ---

5.3.2 Runtime requirements Since we impose a client/server architecture w e recommend having two processors per shmem process, i.e. one process per dual processor workstation. Running on single processor workstations will hav e serious negativ e impact on performance. 5.3.3 Datatypes / porting The default data types size for put and g et are 64 bit. Beware that integers for all architectures, and long's for x86, are 32 bit, or even better, use the shmem calls with specified size of data ty pe. 5.3.4 Dynamic memory allocation Unlike on a machine from SGI Inc. or Cray Inc., using standard Unix memory allocation to memory used in ShMem communication will not work. Dynamic memory visible to ShMem communication have to be allocated w ith one of the following:

54

Scali Library User's G uide Version 3.0


5.4 Compiling and linking

void *shmalloc(size_t size); void *shrealloc(void *ptr, size_t size); void *shmemalign(size_t alignment, size_t size); POINTER (addr, A(1)) INTEGER (length, errcode, abort) CALL shpalloc(addr, length, errcode, abort);

Freeing ShMem capable memory must be done with:
void shfree(void *ptr);

No stack manipulation issues are implemented. 5.3.5 ScaShmem environment variables The processes are allocated statically at startup. Standard Cray/SGI ShMem environment variables to control the placement of the ShMem application on the sy stem are not recognized.

5.4 Compiling and linking
Fortran (be careful with your data size and initialization):
g77 -D_REENTRANT -I/opt/scali/include -c appl.f g77 -o appl appl.o -L/opt/scali/lib -lshmem -lfmpi -lmpi -lpthreads

C:
gcc -D_REENTRANT -I/opt/scali/include -c appl.c gcc -o appl appl.o -L/opt/scali/lib -lshmem -lfmpi -lmpi -lpthreads

Note: Libraries shall always come after the object files on the linkage line.

Scali Library User's Guide Version 3.0

55


Chapter 5 ScaShmem

5.5 Running your application
To run your application you may use shmemrun like this
/opt/scali/bin/shmemrun -np -coherence appl

For details run shmemrun -h. If you prefer to use mpirun or mpimon please use the -shmem mpimon option (ScaMPI 1.11.9 or newer).

56

Scali Library User's G uide Version 3.0


Chapter 6

ScaIP - IP for SCI

This chapter describes the Scali Internet Protocol driver ScaIP. Please note that the ScaIP release notes and other readable information are available in the /opt/scali/doc/ScaIP directory.

6.1 Introduction
The ScaIP package contains the kernel mode driver scip, which w hen it is loaded, exists as a kernel-resident Internet Protocol network interface (supporting the 'inet' address family). When properly configured, the scip driver provides support for the upper layered module Linux IP to transmit and receive Internet datagram packets over SCI.

6.2 Simplified network model
The network architecture spans both the user-level and the kernel-level, as shown in "Fig ure 6-1: Simplified network model". The application layer is executed at the userlevel, and is utilizing the network services provided at the transport layer in the kernel. The most common predefined interface between the transport layer and the application layer is the Berkely socket interface. The transport layer protocols (e.g . TCP and UDP) use services offered by the netw ork layer (IP) to send messages to a destination node and to receive messages from other nodes. The network layer handles the transfer of data packets between the connected nodes using the serv ices provided by the link layer software (e.g. the Ethernet driver or the ScaIP/ScaMAC/ScaSCI modules). The link layer includes the network interface hardware and the software device driver software that controls the network interface.

Scali Library User's Guide Version 3.0

57


Chapter 6 ScaIP - IP for SCI

Client

C lient

Application layer

User- level

TCP

UDP

Tr ansport layer

IP

Network layer

Kernel-level

Ethernet driver

ScaIP scip

Link layer

ScaMAC scimac

I ssci

ScaSCI

SCI interconnect Ethernet

Physical layer

Figure 6-1: Simplified network model

6.3 Configuration
The ScaIP network interface driver scip is configured using the standard Linux maintenance command /sbin/ifconfig, or any other suitable Linux command or script. To display information about the ScaIP interface, the standard Linux command /bin/netstat can be used. Thus, standard Linux utilities are to be used both to config ure ScaIP and to retrieve network interface information from the scip driver. The name of the ScaIP interface is the driver name (scip) followed by the unit number (0 for the first ScaIP interface), for example scip0.

58

Scali Library User's G uide Version 3.0


6.4 ScaIP packag e installation

The address being assigned to the ScaIP interface (inet family) is an IP address (e.g. in a class-C network, 192.168.4.1 w here 192.168.4 is the network number and 1 is the node's unique host number). By default, the Address Resolution Protocol (ARP) is enabled to implement mapping between the IP address and the hardware address for the netw ork interface. The use of the ARP protocol can be disabled. To manually create the address mapping, the Linux maintenance command /sbin/arp can be used. If, for one reason or another, there is a need to enforce a statically (and permanently) creation of address mapping entries for nodes accessible via the ScaIP interface, consult the text file '/opt/scali/kernel/scip/arptable.example' and follow the instructions contained therein. The hardware address of the ScaIP interface is associated with the interface when the scip driver is loaded and attached to the ScaMAC module (i.e., the scimac driver). It is not permitted to change the hardware address of the interface. The scip driver does not provide fragmentation of datagram packets. If transmission of a datagram larger than the Maxim um Transm ission Unit (MTU) is attem pted, the datagram packet is dropped. The physical MTU size of the ScaIP interface cannot be set using '/sbin/ifconfig' (or any other utility program). The MTU setting is determined by a configuration property (scimac_max_ebuf_size) for the ScaMAC module; the actual MTU of the ScaIP interface is 'scimac_max_ebuf_size + 86'. The scip driver is initialized when the ScaIP module is loaded into Linux kernel space, and is de-initialized when the ScaIP module is unloaded from the Linux kernel. Error messages from the driver are printed to the system's log file (e.g. /var/log/messages).

6.4 ScaIP package installation
The scip driver cannot be used to transfer packets ov er SCI until the ScaIP package (distributed as a Linux RPM file) has been installed on each processing node. To install the ScaIP package you should use the Scali Softw are Platform (SSP) installation program. The SSP installation prog ram provides y ou with the option to install the ScaIP package on each of the selected processing nodes. The scip driver depends on the scimac driver (the Scali Media Access Control driver for SCI) included in the ScaMAC package, which in turn depends on the scasci driver (the Scali PCI SCI driver) included in the ScaSCI packag e.

Scali Library User's Guide Version 3.0

59


Chapter 6 ScaIP - IP for SCI

The chapter Softw are Installation in the Scali System Guide offers detailed information on how to use the SSP installation prog ram. Another way to install (or update) the ScaIP package is interactively by hand using the Linux RPM package manager program rpm . Note that the operation should be performed on each of the processing nodes. For a description of how to update the ScaIP packag e using the rpm program, please consult the text file /opt/scali/doc/ScaIP/INSTALL.

60

Scali Library User's G uide Version 3.0


Chapter 7

ScaMAC

This chapter describes the Scali Media Access Control package ScaMAC.

7.1 Introduction
The ScaMA C package has been dev eloped to prov ide an efficient way to pass data packets betw een SCI interconnected nodes. The package includes the kernel mode driver scimac and some utility programs. The scim ac driver is written as a multi-threaded loadable driv er module supporting unicast data transfer over the SCI interconnect. In the current implementation neither broadcasting to all nodes nor multicasting to a group of nodes on the interconnect are supported. Please note that the ScaMAC release notes and other readable information are available in the /opt/scali/doc/ScaMAC directory. For a brief description of the network model, see chapter 6.2 on page 57.

7.2 The scimac driver
The scim ac driver provides Application Programming Interface services for other upper lay ered modules (e.g. ScaIP) to transmit and receive data packets over SCI. The API contains methods for other modules to attach/detach to scimac, and methods to exchange data packets with similar modules on other SCI interconnected nodes. The scim ac drivers provide a reliable connection oriented (node-to-node) transfer of data packets over SCI, and guarantees that the data is sent and received in order. Upon SCI interconnect problems, data packets are, by default, retransmitted until retransmission timeout. On timeout, the data packet is dropped by scimac. Upon error, scimac error messag es and warnings are printed to the system log file (e.g. /var/log/messages ).

Scali System Guide Version 3.0

61


Chapter 7 ScaMAC

7.3 Setting up the scimac driver
The scimac driver can be explicitly started, stopped and restarted by user root using the command:
# /sbin/service/scimac [start|stop|restart]

Note that the scimac driver depends on the ssci driver (Scali PC I SCI driver) to be running. If the Scali PCI SCI driver is not loaded and started, the scim ac driver will fail to start. The file /opt/scali/kernel/scimac/scimac.conf contains configuration variables for the scimac driver. If a variable is chang ed, the new value will override the default value when the scimac module is installed in the kernel. Note that a modification to scimac.conf will not be effective until the scimac driver module is reloaded. Table 7-1 lists the config uration variables together with their default values.

Variable
scimac_max_no_hdrs

D efault
32

Description
The maximal number of scim ac packet headers to be used by the driver The maximal number of eager buffers to be used by the scimac driver The maximal size in bytes of each eager buffer used Enable use of a kernel thread to d efer reception of packets Enable use of the immediate bottom half to defer reception of packets The maximal number of packets queued for transfer p er connection path at any one time The packet retransmit time in milliseconds The p acket`s maximal retransmit time in millisecond s

scimac_max_no_ebufs

8

scimac_max_ebuf_size

32768

scimac_use_ulevel_recv

1

scimac_use_sw_interrupts

0

scimac_max_send _queuelen

2000

scimac_pkt_rexmit_time scimac_max_rexmit_time

200 5000

Table 7-1: scimac config uration variables

62

Scali System G uide Version 3.0


Section: 7.4 The ScaMAC utilities

Variable
scimac_min_nodeid_number

Default
0x100

Description
The lowest valid node identif ier in the Scali system The largest valid node identifier in the Scali system The incremental node identifier step in the Scali system

scimac_max_nodeid _number

0xff00

scimac_nod eid_increment

0x100

Table 7-1: scimac configuration variables

Normally, there should not be necessary to change any of the configuration parameters.How ever, the configuration is by default optimized for a Linux 2.4.x kernel. If unex pected performance problems occur when using a Linux 2.2.x kernel, you should change two of the default configuration values. For a Linux 2.2.x kernel, edit the file: /opt/scali/kernel/scimac/scimac.conf by setting:
scimac_max_ebuf_size = 16384

7.4 The ScaMAC utilities
The following is a description of the ScaMac utility prog rams which can be used to retrieve and display various scimac driv er interface information, and to check the reachability of remote scimac drivers. The utilities are located in the bin and the sbin directories of the Scali installation. Normally they are of limited interest to the ordinary user. Currently, no standard Linux utilities can be used to collect and display information about the scimac driv er interface. 7.4.1 macstat - display scimac driver status The program macstat displays information gathered from the driver's data structures. The information printed is controlled by the option you select. Usage:
/opt/scali/bin/macstat -{a|c|i|s|R|p} []

Options:
-a

Display the state of all remote node connections, and connection configuration related information.

Scali System Guide Version 3.0

63


Chapter 7 ScaMAC

Show the scimac driver's configuration v ariables (current values), as defined by the configuration properties listed in the tex t file /opt/scali/kernel/scimac/scimac.conf. -i Display the state of all remote node connections, and connection traffic statistics. -s Show a summary of accumulated driver traffic. -R Reset the traffic statistics (privileged, for user root only). -p Show the distribution of data packet sizes being transferred. Access the specified scim ac instance (physical point of attachment), instead of 0 (default).
-c

7.4.2 macping - check reachability of remote scimac drivers The program macping can be used to check the reachability and connectivity of scimac driv ers on SCI interconnected nodes. When the program is activated, the scimac driver sends a short out-of-band request packet via the SCI interconnect to the scimac driver(s) on the nodes specified at the command line. If a node responds, macping computes the round-trip time and prints a summary of information. Otherwise, if a node does not respond, macping will print a timeout message.

64

Scali System G uide Version 3.0


Section: 7.4 The ScaMAC utilities

Usage:
/opt/scali/bin/macping [-n ] [[ ...] | [: []]]

Options>
-n

Access the specified scimac instance (phy sical point of attachment), instead of 0 (default). The SCI node identifier of one or more nodes to probe. Node identifier increment.

If no node identifier is given, macping attempts to send a ping request to each of the SCI connected nodes currently known by the scimac driver (as displayed by '/opt/scali/bin/macstat -a'). 7.4.3 macctl - set the debug level of the scimac driver The program macctl will set or get the debug level of the scimac driver. By default, all printing of debug information is disabled. When scimac debug is enabled, scimac driver information is printed to the system log file (e.g. /var/log/messages). Note that if debug is enabled in the driver, it w ill automatically slow down the transfer of data packets, and may lead to network congestion or loss of packets. Usage:
/opt/scali/bin/macctl -d []

Options:
-d The debug level to be enabled (0 to disable debug code).

If no option is specified, macctl prints a list of available debug level values and their meaning. Debug level values (in any hexadecimal (0x ) combination, or 0 to disable): S S S S S S S S S C C C C C C C C C IM IM IM IM IM IM IM IM IM AC_WARN AC_INFO AC_EP AC_PT AC_Q FULL AC_INTR AC_IPATH AC_RPATH AC_MSG 0x1 0x2 0x4 0 x8 0x10 0x20 0 x4 0 0x80 0x100 / / / / / / / / / * * * * * * * * * Error conditions (WARNING style) */ Useful information about events */ Device driver entry and exit points */ Management service requests/responses */ Full rcv/snd queue log ging */ Interrupt information */ Setup of paths and initial communication */ Tracking of path referencing */ Handling of scimac messages */

Scali System Guide Version 3.0

65


Chapter 7 ScaMAC

S S S S S S S S S S S S

C C C C C C C C C C C C

IM IM IM IM IM IM IM IM IM IM IM IM

AC AC AC AC AC AC AC AC AC AC AC AC

_PMS _PPA _FLOW _XFER _EBUF _THRE AD _ALLOC _MEM _CH _TIME _INEMPTY _RCVH DRS

0x200 0x400 0x800 0x 1000 0x 2000 0x 4000 0x 8000 0x10000 0x20000 0x40000 0x80000 0x100000 0x200000 0x400000 0x800000 0x1000000

/* /* /* /* /* /* /* /* /* /* /* /* /* /* /* /*

SCIMAC _XMTHDRS SCIMAC _SCAMEM SCIMAC _PING SCIMAC _URG

Promiscuous mode logg ing */ Allocation and (de)reference of ppa */ Flow control logging */ Packet transfer logging */ Ebuf usage */ Information on the thread operations */ Kernel memory alloc & free operations */ Memory copy operations */ Channel setup */ Timing of different parts of the code */ Tracking empty receive buffer */ Rcv ring buffer headers (if no new pkt found) */ Local copy of xmt headers (if remote rcv full) */ ScaMem operations */ Probing reachability of a remote scimac driv er */ Broadcasting URG request from ScaIP */

7.5 ScaMAC package installation
The scimac driver and the ScaMAC utilities cannot be used for data transfer over SCI until the ScaMAC package (distributed as a Linux RPM file) has been installed on each processing node. To install the ScaMA C package you should use the Scali Software Platform (SSP) installation program. The SSP installation prog ram will install the specified software packages on each of the processing nodes. The ScaMAC package is automatically installed when you decide to install the ScaIP packag e. The scimac driv er depends on the ssci driver (Scali PCI SCI driver) included in the ScaSCI package. The ScaSCI package is automatically installed during an SSP installation. The chapter Software Installation in the Scali System Guide offers detailed information on how to use the SSP installation program. Another way to install (or update) the ScaMAC package is interactively by hand using the Linux RPM package manag er program rpm. Note that the operation should be performed on each of the processing nodes. For a description of how to update the ScaMAC package using the rpm program, please consult the tex t file /opt/scali/doc/ScaMAC/INSTALL .

66

Scali System G uide Version 3.0


Chapter 8
8.1 Feedback

Support

Scali appreciates any sug gestions to improve both this Scali Library User's Guide and the software described herein. Please send your comments by e-mail to support@scali.com. The user of parallel tools software using ScaMPI on a Scali System, is encouraged to provide feedback to the National H PCC Softw are Exchange (NHSE) - Parallel Tools Library [9]. The Parallel Tools Library provides information about parallel sy stem software and tools, and, in addition, it provides for communication between the software author and the user.

8.2 Scali mailing lists
We have developed mailing lists being available on the Internet. For instructions on how to subscribe to a mailing list (e.g., scali-announce or scali-user), please check out the Mailing Lists section at http://www.scali.com/support.

8.3 ScaMPI FAQ
The ScaMPI Frequently Asked Questions are posted on our Web site at http://www.scali.com. Please check out the ScaMPI FAQ section at http://www.scali.com/support. In addition, the FAQ is, when ScaMPI has been installed, available as a text file in /opt/scali/doc/ScaMPI/FAQ .

8.4 ScaMPI release documents
When ScaMPI has been installed, a number of small documents like FAQ, RELEASE NOTES, README, SUPPORT, LICENSE_TERMS, INSTALL are available as text files in the /opt/scali/doc/ScaMPI directory.

Scali Library User's Guide Version 3.0

67


Chapter 8 Support

8.5 Problem reports
Problem reports should, whenev er possible, include both a description of the problem, the software versions, the computer architecture, an example, and a record of the sequence of ev ents causing the problem. Any information that you can include about what triggered the error will be helpful. The report should be sent by e-mail to support@scali.com.

8.6 Platforms supported
ScaMPI is available for a number of platforms. For up-to-date information, please check out the ScaMPI section at http://www.scali.com/products. For additional information, please don't hesitate to contact Scali at sales@scali.com.

8.7 Licensing
ScaMPI is licensed using the Scali license manager system. In order to run ScaMPI a valid demo or a permanent license must be obtained. To obtain the appropriate license, please send an inquiry to sales@scali.com. Any technical issues should be addressed to support@scali.com.

68

Scali Library User's G uide Version 3.0


Chapter 9
9.1 References

Related documentation

[1] MPI: A Message-Passing Interface Standard The Message Passing Interface Forum, Version 1.1, June 12, 1995, Message Passing Interface Forum, http://www.mpi-forum.org. [2] MPI: The complete Reference: Volume 1, The MPI Core Marc Snir, Stev e W. Otto, Steven Huss-Lederman, David W. Walker, Jack Dongarra. 2e, 1998, The MIT Press, http://www.mitpress.com. [3] MPI: The complete Reference: Volume 2, The MPI Extension William G rop, Steven Huss-Lederman, Ewing Lusk, Bill Nitzberg, W. Saphir, Marc Snir, 1998, The MIT Press, http://www.mitpress.com. [4] Scali System Guide Scali AS, http://ww w.scali.com/ [5] ScaMPI Data sheet Scali AS, http://ww w.scali.com/ [6] Scali Free Tools Scali AS, http://www.scali.com./ [7] Review of Performance Analysis Tools for MPI Parallel Programs UTK Computer Science Department, http://www.cs.utk.edu/~brow ne/perftoolsreview/. [8] Debugging T ools and Standards HPDF - High Performance Debug ger Forum, http://www .ptools.org/hpdf/. [9] Parallel Systems Software and Tools NH SE - National HPCC Software Exchange, http://ww w.nhse.org/ptlib. [10] MPICH - A Portable Implementation of MPI The MPICH home page, http://www.mcs.anl.gov/mpi/mpich/index.html.

Scali Library User's Guide Version 3.0

69


Chapter 9 Related documentation

[11] MPI Test Suites freely available Argone National Laboratory, http://www-unix.mcs.anl.gov/mpi/mpi-test/ tsuite.html

70

Scali Library User's G uide Version 3.0


List of figures
3 3 3 3 3 3 3 3 3 3 6 -1 -2 -3 -4 -5 -6 -7 -8 -9 -10 -1 Application start-up - phase 1 ....................................................................... Application start-up - phase 2 ....................................................................... Application start-up - phase 3 ....................................................................... Thresholds for different communication protocols ...................................... Inlining protocol............................................................................................. Eagerbuffering protocol ................................................................................. Transporter protocol ...................................................................................... Channel resource ........................................................................................... Eagerbuffer buffer ......................................................................................... Transporter buffer ......................................................................................... Simplified network model ............................................................................. 32 33 34 35 36 37 38 40 41 42 58

Scali Library User's Guide Version 3.0

71


72

Scali Library User's G uide Version 3.0


List of tables
2-1 2-2 2-3 2-4 2-5 2-6 2-7 2-8 2-9 2-10 2-11 2-12 2-13 2-14 2-15 3-1 7-1 Environment variables ....................................................................................... 9 Basic options to mpimon .................................................................................. 11 mpimon parameters ......................................................................................... 12 Numeric input ................................................................................................... 12 Complete list of mpimon options ..................................................................... 13 mpirun format................................................................................................... 17 mpirun options .................................................................................................. 18 Options for SCAMPI_TRACE .......................................................................... 20 Fields in output from builtin trace .................................................................. 22 Timespec in output from builtin trace ............................................................. 22 Options for SCAMPI_TIMING ........................................................................ 23 Fields in output from builtin timing ................................................................ 23 Fields in"recv "-lines from builtin timing ......................................................... 24 Fields in"send"-lines from builtin timing ........................................................ 25 Com monfields in output from builting timing ................................................ 25 Libraries ............................................................................................................ 31 scim ac configuration variables ........................................................................ 62

Scali Library User's Guide Version 3.0

73


74

Scali Library User's G uide Version 3.0


Index
B Benchmarking ScaMPI ................................................................................................. 51 C Communication protocols in ScaMPI ........................................................................... 35 Eagerbuffering protocol ......................................................................................... 37 Inlining protocol ..................................................................................................... 36 Transporter protocol .............................................................................................. 38 Communication resources in ScaMPI .......................................................................... 39 Channel buffer ....................................................................................................... 40 Eagerbuffer buffer ................................................................................................. 41 Transporter buffer ................................................................................................. 42 Compiling ScaMPI ..................................................................................................................... 9 ScaMPI example program ..................................................................................... 29 ScaShmem .............................................................................................................. 55 D Debugging ScaMPI ................................................................................................................... 19 E Environment variables ScaMPI ..................................................................................................................... 9 ScaShmem .............................................................................................................. 55 L libfmpi ............................................................................................................................ 31 libmpi ............................................................................................................................. 31 Linking ScaMPI ................................................................................................................... 10 ScaMPI example program ..................................................................................... 29 ScaShmem .............................................................................................................. 55 M MPI ............................................................................................................................7, 69 mpi.h ..........................................................................................................................9, 44 mpiboot .......................................................................................................................... 31 MPICH ...........................................................................................................................69 mpid ............................................................................................................................... 31 mpif.h .........................................................................................................................9, 44 mpimon ....................................................................................................................11 , 31 Advanced usage ..................................................................................................... 11

Scali Library User's Guide Version 3.0

75


Basic usage ............................................................................................................. 11 List of available options ......................................................................................... 13 mpirun ........................................................................................................................... 16 mpisubmon .................................................................................................................... 31 O Optimize ScaMPI performance..................................................................................... 49 P Profiling nupshot ................................................................................................................... 27 ScaMPE libraries ................................................................................................... 26 ScaMPI ................................................................................................................... 20 upshot ..................................................................................................................... 27 R Running ScaMPI ................................................................................................................... 10 ScaMPI example program ..................................................................................... 29 ScaShmem .............................................................................................................. 56 S ScaIP ................................................................................................................................ 7 Scali Universe GUI ....................................................................................................... 10 ScaMAC ........................................................................................................................... 7 ScaMPE ..........................................................................................................................26 libampi .................................................................................................................... 26 liblmpi ..................................................................................................................... 26 libtmpi..................................................................................................................... 26 upshot ..................................................................................................................... 26 ScaMPI ............................................................................................................................. 7 Builtin-cpu-usage ................................................................................................... 26 Builting-sanitycheck-of-data ................................................................................. 20 Builtin-segment-protect-violation-handler ........................................................... 19 Builtin-timing ......................................................................................................... 23 Builtin-trace ........................................................................................................... 20 Environment............................................................................................................. 9 Example program ................................................................................................... 28 Executables ............................................................................................................ 31 Libraries ................................................................................................................. 31 Test programs ........................................................................................................ 29 SCAMPI_BACKOFF_ENABLE, backoff-mechanism ................................................. 50 SCAMPI_BACKOFF_IDLE,backoff-mechanism ......................................................... 50 SCAMPI_BACKOFF_MAX, backoff-mechanism ......................................................... 50 SCAMPI_BACKOFF_MIN, backoff-mechanism ......................................................... 50 SCAMPI_CPU_USAGE , builtin cpu-usage facility ..................................................... 26

76

Scali Library User's G uide Version 3.0


S S S S S S S S

CAMPI_DATACHE CK_ENABLE, builtin sanity-check of data .............................. 20 CAMPI_DISABLE_HPT, disable high precision timer ............................................ 46 CAMPI_INSTALL_SIGSEGV_HAND LE R, builtin SIGSEGV handler ............19 , 47 CAMPI_NODENAME, set hostname ........................................................................ 46 CAMPI_TIMING , builtin timing-facility ................................................................... 23 CAMPI_TRACE, builtin trace-facility ....................................................................... 20 CAMPI_WORKING_DIRECTORY, set working directory ...................................... 46 caShmem ....................................................................................................................... 7 Compiling ............................................................................................................... 55 Environment variables .......................................................................................... 55 Features and Limitations ...................................................................................... 54 Linking ................................................................................................................... 55 Porting .................................................................................................................... 53 Running .................................................................................................................. 56 T Troubleshooting ScaMPI .............................................................................................. 45

Scali Library User's Guide Version 3.0

77


78

Scali Library User's G uide Version 3.0