CMSC421 Project Two: Reinforcement Learning

 

This project has two components.  Your team (2-3 members) will create:

Each of these is described below.

 

The Environment

In your environment, there will be 4 states/percepts and 4 actions.  Rewards should be in the range from -10 to 10.

Your environment will be described by two text files:

We will provide an environment simulator which will be able to read in your environment and test. 

We will constrain everyone to use the same abstract representation so that learning agents will be able to interact with all of the environments created by the class.

Environments which show some thought or creativity are encouraged.

Environment Turn in Procedure

You will turn in the following text files:

  1. state transition file
  2. reward file
  3. README file which includes the name of your environment, the names of your team
    members and a detailed description. The README file can be txt or pdf.

Put these in a tar file called 'env.tar'.

To submit electronically 'env.tar', execute following command from one of your team member's account:
/afs/csic.umd.edu/submit/fall2005/cmsc421/bin/submit 2 env.tar
(Note that the project number is 2)

When your team submits env.tar,  please email jylee@cs.umd.edu which account your team will use to submit (cs421xxx) and names of your team members.

The Learning Agent

This is the core of the project.  We will provide skeleton code for both learning and evaluating your agent.  Your job is to write an agent which will interact with the environment and will learn the function Q(S,A), which is the value of executing action A in state s.  Your learning agent will output the Q function that it learns.  We provide a simulator that will take as input this Q function and evaluate the function the agent has learned.

Skeleton code is available in:
/afs/csic.umd.edu/projects/cmsc421/common/pa2/SKELETON-CPP
for the C++ code, and
/afs/csic.umd.edu/projects/cmsc421/common/pa2/ SKELETON-JAVA
for the Java code.

LEARN directory: Files in this directory are used to implement your learning agent.  There is an Agent_shell which will take as input the environment files, number of learning epochs and number of steps in each epoch and will output a text file with the Q function values.

You are allowed to modify ONLY Agent.cpp/Agent.java (and Agent.h). You will turn in ONLY the Agent files and we will test your with the original Agent_shell file. Note that you can not change parameters of function Agent.Learn() or it will not work with our Agent_shell.cpp. DO NOT MODIFY OTHER FILES.


EVAL directory : Files in this directory are used to evaluate your agent learning. After learning (by running Agent_shell), it will output Q values in a file (above <Q-value-file>).
The evaluator code takes the Q-file as input and evaluates the learning result.  The evaluator will take as input the environment files, the Q values, number of epochs and number of steps in each epoch.  It will compute the total discounted reward received by the agents.
 

Once all the environments are turned in, we will be making these available for you to test your agent. 

You will turn in two things: your code, which we will evaluate on several of the class environments and a report.  The evaluation of your code will be based on the reward received during learning and the reward received during the evaluation phase. 

For your writeup, we will expect a short description of your environment, a short description of your learning algorithm and, most importantly, some experiments which illustrate design decisions that you explored.  The most obvious of these would be comparing different strategies for setting the learning rate parameter alpha in the program.  These experiments should show, on two or more environments, the varying results.  Think about what parameters you will hold fixed (number of steps and epochs?  Amount of time?  reward?) and which you will measure (total reward?  average reward? number of steps/epochs?)  You should present these in a graph or table.  You should include a discussion of your findings.

Learning Agent Turn in Procedure

  1. Code: You will turn in the following files.
    1. Agent.cpp[h]/Agent.java - ONLY SUBMIT SOURCE CODE, NO OBJ/EXE
    2. README file which include the names of your team members

    Put these in a tar file called 'agent.tar'

    To submit electronically 'agent.tar', execute following command from one of your team member's account:
    /afs/csic.umd.edu/submit/fall2005/cmsc421/bin/submit 3 agent.tar
    (Note that the project number is 3).
    Please email the ta when you have turned in the code.

    The code is due ** 5PM MONDAY DECEMBER 12 **.  We will be attempting to evaluate these as soon as we get them so that we can discuss some of the results in class on Tuesday, so you are highly encouraged to turned the code in on time.
     
  2. Report: Please turn in a hard copy of your report by 5PM Wednesday DECEMBER 14 ** to Prof. Getoor's office.  Slide under her door, if she is not there.  Please clearly indicate the group member names, and the login of the account that was used to turnin the environment, code and report.

    Please also turn in an electronic copy of your report:
    /afs/csic.umd.edu/submit/fall2005/cmsc421/bin/submit 4 name-report.pdf/doc
    (Note that the  is 4).
    also by 5PM Wednesday December 14.
  • *
*

Web Accessibility