This directory contains a number of simple programs for testing and launching 
a mixed set of coupled applications using IBM's Parallel Environment and 
LoadLeveler.

To allocate a set of nodes on an SP for simultaneous execution of multiple 
applications, it is necessary to define the POE job as MPMD.  In these 
circumstances, POE creates a single global MPI communicator which will be 
shared among all the participating applications.  Unfortunately, this will 
cause quite a bit of confusion if these applications were not designed with 
this in mind (such as any P++ application).  To ameliorate this situation, 
each application must split off its own private communicator using the 
MPI_Comm_split call.  An example of this is shown in either orange.c or red.c.

A second issue which arises when setting the POE job to MPMD is that non-MPI 
(serial) applications never call MPI_Init, which is expected of every process
in the MPMD job.  This issue is resolved by creating a wrapper application 
which makes the appropriate MPI calls and 'exec's the serial job.  An example
of this code is mpish.c.

A complete example involving four single process applications is presented in
the source files red.c, orange.c, yellow.c, and green.c.  Two of the programs,
red.c and orange.c are MPI jobs which call MPI_Comm_split.  The other two 
programs, yellow.c and green.c are serial applications which must be launched
by mpish.c.  All of the programs join a single PVM group, which is similar 
to what they would need to do if they were actually using InterComm. Once the
codes are built, they can be launched using the LoadLeveler script pvm.cmd.  
The LoadLeveler script has the pvmmaster script (found in the 'scripts' 
directory) as its executable, which is responsible for setting up and running
PVM daemons on the allocated nodes.  The pvmmaster script in turn calls on 
poe to start a MPMD job and uses its first argument as the name of the 
command file.  The command file is listed in pvm.cmd as the argument to 
pvmmaster and is currently set to poe.cmd.  The poe.cmd file contains a list 
of all the applications to run, with parallel applications listed multiple 
times for multiple tasks.  The poe.cmd currently only runs each of the four 
programs as a single process, with the serial jobs being launched indirectly 
with mpish.

To run the complete application set, first be sure that all the mentioned 
scripts and applications are in your PATH (and in your PATH on LoadLeveler 
allocated nodes), then the only command that is needed is:

>llsumbit pvm.cmd

If this fails, it is most likely that the pvm scripts or the sample 
applications were not found in your PATH.
