HPCS Productivity Study

The Experimental Software Engineering Group at the University of Maryland is conducting a study in this class on how hard it is to write parallel programs using different technologies. This study is part of a much larger project that is related to the development of the next generation of parallel machines.

We are going to collect data as you complete your programming assignments to measure how much time you spend developing and what different activities you are using. Some of this data collection will be automatic, and some will be manual.

Most of the data collection software will be installed on the class server. However, for those people who are working on their local machines, we have some downloadable software that will allow us record how much time you spend in your editor. Please go to the page on installing this instrumentation on your local machine.

Configuring your beowulf account for instrumentation

To configure our beowulf account for the instrumentation, please log in and run the following script: /home/lorin/inst/sensors/setup.sh

Note that it will change your shell to tcsh, which is required by one of the instrumentation programs.

Using the instrumentation

We are collecting data using four mechanisms:

Effort log
Compiler instrumentation

Editor instrumentation

Command-line instrumentation

Effort log

We ask that you keep a log of how much time you spend working on your assignment. These files contain a blank log, as well as an example of how to fill out the log (the example log is just intended to show you how to fill in the log: the entries were made up for illustration purposes).

Whenever you spend some time on the assignment, please make an entry in the log. We ask that you try to keep track of the effort while you work. If you forget and fill it out after you've done the work, we ask that you just indicate this in the "Backdated" field (see below). Here is a description of the fields for each entry:

Date The date you did some work

Working remotely Check this if you were working on the class server during this interval

Start time The time you started working

Stop time The time you stopped working

Breaks (minutes) Approximately how much time you spent not working on this assignment during this interval (e.g. eating lunch)

Question Which question in the assignment you are working on

Language/approach Which programming language (e.g. Matlab,C) and which parallel programming approach (serial,STARP,MPI,OpenMP) you are using

Activity Fill in one of the two-letter activity codes from the list at the top of the form

Comments Anything extra you want to tell us about this work interval

Backdated Check this if you filled in this entry from memory, as opposed to while you were working

Date	The date you did some work
Working remotely	Check this if you were working on the class server during this interval
Start time	The time you started working
Stop time	The time you stopped working
Breaks (minutes)	Approximately how much time you spent not working on this assignment during this interval (e.g. eating lunch)
Question	Which question in the assignment you are working on
Language/approach	Which programming language (e.g. Matlab,C) and which parallel programming approach (serial,STARP,MPI,OpenMP) you are using
Activity	Fill in one of the two-letter activity codes from the list at the top of the form
Comments	Anything extra you want to tell us about this work interval
Backdated	Check this if you filled in this entry from memory, as opposed to while you were working

Activities

There are a number of activities listed under the effort log. The following is a description of these categories.

Thinking: Any work done on the assignment that does not involve using the computer. This includes: thinking about a solution to the problem, planning/designing your implementation .
Serial coding: Writing the code for the serial version of your solution
Parallelizing the code: Writing the code for the parallel version of your solution
Testing code: Verifying that the answer produced by your code is correct, timing your code for performance evaluation, etc.
Debugging: Identifying the reason for a failure and fixing it
Tuning code: the code to improve performance
Experimenting with the environment: Work not done specifically on one of the assignments (e.g. writing "hello world", learning how to use the simulator)
Other: Any other activity not listed above

Note that these activities are not listed in a particular order, and some of these may activities may not apply in your particular case.

Compiler instrumentation

When you compile, one or both of the following two questions will sometimes appear:

How long (in minutes) have you been working before this compile?
(Hit enter if you have been working continuously since last compile)
>

Reason for recompile:
1. Serial coding
2. Parallelizing the code
3. Testing
4. Debugging
5. Tuning
6. Experimenting with environment
7. Other (you will be asked to supply reason)
>

The first question only needs to be answered if you have just returned to the compiler after having taken a break (or are compiling for the first time). If you have been working on the assignment since the last compile, just hit enter. The second question relates to the type of activity you are doing. These are the same activities as on your effort form.

Editor instrumentation

The editor instrumentation will track how much time you spend editing files. It is not interactive. Depending on your editor, you may see status messages related to "Hackystat", which is the data collection system we are using.

Command-line instrumentation

If you are working on the remote machine, the commands that you input to the shell will be logged. The command-line instrumentation only works with tcsh, so we ask that you we ask that you please use tcsh as your shell. (Note that if you are working on your local machine, we do not collect this data).

This instrumentation is not interactive: there is nothing you need to do. When you log out, you will see a message that indicates the command-line data has been collected.

If you have any questions, please contact Lorin Hochstein (lorin@cs.umd.edu)