Downloading and building LULESH

You can get LULESH by cloning its git repository as follows:

        git clone https://github.com/LLNL/LULESH.git
        

You can use CMake to build LULESH on zaratan by following these steps:

        mkdir build

        cd build

        cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_CXX_FLAGS="-g -O3" -DMPI_CXX_COMPILER=`which mpicxx` -DWITH_OPENMP=Off -DWITH_SILO=Off ..

        make
        

This should produce an executable lulesh2.0 in the build directory.

Running LULESH

Lets say you want to run LULESH on 8 processes for 10 iterations/timesteps. This would be the mpirun line:

        mpirun -np 8 ./lulesh2.0 -i 10 -p
        

Using HPCToolkit and Hatchet

HPCToolkit is available on zaratan via the hpctoolkit/gcc module. You can use HPCtoolkit to collect profiling data for a parallel program in three steps.

1. Step I: Creating a hpcstruct file (used in Step III) from the executable (named exe here)
   hpcstruct exe
   This will create a file called exe.hpcstruct
2. Step II: Running the code (LULESH) with hpcrun:
   mpirun -np <num_ranks> hpcrun ./exe <args>
   This will generate a measurements directory.
3. Step III: Post-processing the measurements directory generated by hpcrun:
   mpirun -np 1 hpcprof-mpi --metric-db=yes -S exe.hpcstruct -I <path_to_src> <measurements-directory>
   This will generate a database directory.

Hatchet can be used to analyze the database directory generated by hpcprof-mpi using its from_hpctoolkit reader.

You can install Hatchet on zaratan using:

        module load python

        pip install --user multiprocess

        pip install --user hatchet==1.2.0
        

OR If you want to use the latest (development) version of hatchet on zaratan or your local computer, you can install it by following these steps:

        module load python

        git clone https://github.com/hatchet/hatchet.git
        

Then add the directory where you installed hatchet to your PYTHONPATH and then do:

        cd hatchet/

        ./install.sh
        

Assignment Tasks

1. Task 1: You will run LULESH on 1, 8 and 27 MPI processes in the default (weak scaling) mode (with the parameters suggested above), and compare the performance of various executions. Identify the functions/statements that the code spends most of its time in. Identify the functions/code regions that scale poorly as you run on more processes.
2. Task 2: You will run LULESH on 1, 8 and 27 MPI processes with the additional argument -s 45 and compare the performance of these executions with those in the default mode. Identify the functions/code regions where the code spends disproportionately more time compared to the default mode in task 1.
3. Task 3: You will run LULESH on 1, 8, and 27 MPI processes in the strong scaling mode (use additional arguments, -s 45, -s 22, and -s 15 respectively), and compare the performance of various executions. Identify the functions/code regions that scale poorly as you run on more processes in this strong scaling mpde. Compare the results with the functions you identified in task 1.

What to Submit

You must submit the following files and no other files:

• Python scripts that use hatchet for the analyses: task1.py, task2.py, and task3.py.
• Database directory generated on 1 process for Task 1 renamed to lulesh-default-1process.
• A report that describes what you did, and identifies the main bottlenecks in the source code in the various scenarios above (does not need to exceed 1 page per task). Which function consumes most of the time?

You should put the code, and report in a single directory (named LastName-FirstName-assign2), compress it to .tar.gz (LastName-FirstName-assign2.tar.gz) and upload that to gradescope. Do not include irrelevant files in the tarball. Replace LastName and FirstName with your last and first name, respectively.

Tips

• Don't follow the build and running instructions in this assignment blindly. The goal is for you to learn to compile and run parallel code, and learn how to use HPCToolkit and Hatchet.
• Helpful resources: HPCToolkit user manual and Hatchet User Guide
• If you have questions about using these tools or Python and pandas, try using Google first.

Grading

The project will be graded as follows:

Component	Percentage
Analysis 1	30
Analysis 2	30
Analysis 3	30
Writeup	10