lf90lowlevel.f90 and rf90lowlevel.f90 can be run under LoadLever by getting a 
single allocation of computational resources for all the applications that are 
being coupled.

It must be submitted to LoadLever as follows:

llsubmit l2r.cmd

This kind of execution uses multiple scripts in order to allow setting up
the PVM virtual machine and running the applications that will talk to each
other and exchange data. Additionally, the environmental variable MP_PGMMODEL
must be set to 'mpmd' by the LoadLever script to allow starting up multiple
parallel applications.  

l2r.cmd is the LoadLever script. This script can request 1 nodes (2 tasks in
the same node) or 2 nodes (1 task in each node) where the PVM tasks will be
hosted. These configurations are set by the following variables:

# @ node = 1
# @ tasks_per_node = 2

or

# @ node = 2
# @ tasks_per_node = 1

A file named l2r.out will contain the output generated by the execution
and typically will look like the following:

Starting up PVM...
This is the PVM configuration...
adding new host [bs2501sw.]
adding new host [bs2401sw.]
16 tasks will run on 2 processors...
executing: rm -f /home/bluesky/hcma/.ssh/known_hosts.bak
executing: mv /home/bluesky/hcma/.ssh/known_hosts /home/bluesky/hcma/.ssh/known_
hosts.bak
executing: touch /home/bluesky/hcma/.ssh/known_hosts
executing: ssh-keyscan -t rsa,dsa bs2501sw. >> /home/bluesky/hcma/.ssh/known_hos
ts
executing: ssh bs2501sw. 'hostname; rm -f /tmp/pvm* > /dev/null; echo halt | pvm
'
executing: ssh-keyscan -t rsa,dsa bs2401sw. >> /home/bluesky/hcma/.ssh/known_hos
ts
executing: ssh bs2401sw. 'hostname; rm -f /tmp/pvm* > /dev/null; echo halt | pvm
'
executing: cat /home/bluesky/hcma/hostfile.60944.WE | pvm
executing: rm -f /home/bluesky/hcma/.ssh/known_hosts
executing: mv /home/bluesky/hcma/.ssh/known_hosts.bak /home/bluesky/hcma/.ssh/kn
own_hosts
---> PVM started
pvmd already running.
pvm> conf
2 hosts, 1 data format
                    HOST     DTID     ARCH   SPEED       DSIG
                bs2501en    40000     RS6K    1000 0x0658eb59
               bs2401sw.    c0000     RS6K    1000 0x0658eb59
pvm> quit

Console: exit handler called
pvmd still running.
Running the POE job
   0: Taskname is: '/home/bluesky/hcma/research/InterComm-1.0/test/f90/lf90lowle
vel'
   0: lf90lowlevel: task tid 262152 running on hostid 262144
   0:[t40009]  lf90lowlevel: initializing pvm
   0:[t40009]  lf90lowlevel: the applications are using the following host (bs24
01sw. with hostid 786432)
   0:[t40009]  lf90lowlevel: the applications are using the following host (bs25
01en with hostid 262144)
   0:[t40009]  lf90lowlevel: task tid 262153 running on hostid 262144
   0:[t40009]  lf90lowlevel: initializing InterComm -- running 8 tasks on 2 host
s
   0:[t4000a]  lf90lowlevel: task tid 262154 running on hostid 262144
   0:[t4000b]  lf90lowlevel: task tid 262155 running on hostid 262144
   0:[tc0005]  lf90lowlevel: task tid 786437 running on hostid 786432
   0:[tc0006]  lf90lowlevel: task tid 786438 running on hostid 786432
   0:[tc0007]  lf90lowlevel: task tid 786439 running on hostid 786432
   0:[tc0008]  lf90lowlevel: task tid 786440 running on hostid 786432
   0:[t40009]  lf90lowlevel: waiting for remote party
   0:[t40009]  lf90lowlevel: remote party has started
.
.
.
   1: Taskname is: '/home/bluesky/hcma/research/InterComm-1.0/test/f90/rf90lowle
vel'
   1: rf90lowlevel: task tid 786433 running on hostid 786432
   1:[t40005]  rf90lowlevel: initializing pvm
   1:[t40005]  rf90lowlevel: task tid 262149 running on hostid 262144
   1:[t40005]  rf90lowlevel: initializing InterComm -- running 8 tasks on 2 host
s
   1:[t40004]  rf90lowlevel: task tid 262148 running on hostid 262144
   1:[t40003]  rf90lowlevel: task tid 262147 running on hostid 262144
   1:[tc0002]  rf90lowlevel: task tid 786434 running on hostid 786432
   1:[tc0003]  rf90lowlevel: task tid 786435 running on hostid 786432
   1:[tc0004]  rf90lowlevel: task tid 786436 running on hostid 786432
   1:[t40006]  rf90lowlevel: task tid 262150 running on hostid 262144
   1:[t40005]  rf90lowlevel: waiting for remote party
   1:[t40005]  rf90lowlevel: remote party has started
   1:[t40005]  rf90lowlevel: defining regions
   1:[t40005]  rf90lowlevel: creating schedule
   1:[t40005]  rf90lowlevel: array before data is received
   1:[t40005]  rf90lowlevel: receiving data
   1:[t40005]  rf90lowlevel: array after data is received
   1:[t40005]  rf90lowlevel: cleaning up
.
.
.
.

If execution errors are generated they will be stored in the l2r.error file.

l2r.pl is the script used by l2r.cmd to set up the PVM virtual machine that
will be used to provide the communication infrastructure to lf90lowlevel and
rf90lowlevel. These applications themselves are in the POE script file named
l2r.programs.
