Test Types for CS Courses
In our courses we have three types of tests for class
projects/assignments: public, release, and secret. Keep in mind
some projects may only have a subset of them. Additional information
about each kind of test is provided below.
- Public tests →
You have access to these tests. That means
you know what the test is about, and what are the expected
results. This information is provided when the project is
posted, and in some projects the actual code for the test is
provided along with the results. The submit server executes
these tests, and you should always verify you pass the
tests in the submit server, even you passed them locally
(e.g., in Eclipse or in the grace cluster).
- Release tests →
Unlike public tests no information
is provided about these tests. That means you do not have
access to information that describes what the test is about,
and what are the expected results. In some projects the
instructor might provide a meaningful name for a test and
that can help you in the process of figuring out what is
wrong with your code, but often the names used are not descriptive.
There are several goals for release tests. First, it allows
you to verify the functionality of your project. Second,
it encourages you to work early on your projects. Release tests
results, unlike public tests results, can only be seen a limited
number of times. For example, when a project is said to have
three tokens (assuming a 24-hour period) it means that you
have three opportunities to see (in the submit server )the
results of release tests in a 24-hour period. Notice that after
you use a token (i.e., after you do release testing) you need
to wait 24 hours for that token to be used again. For example,
suppose you submit your project and do release testing at 1 pm.
After release testing you realized you are not passing one
of the tests. You then modify your code, submit, and
release test again at 2:15 pm. After release testing for
the second time you still do not pass all tests. Once again, you
modify your code and release test at 3:30 pm. At this point
you will not be able to release test again until 1 pm (the
first token you used would have regenerated by then).
As you can see, the sooner you start working on your projects, the
more opportunities you have to do release testing.
Notice that the results of release tests are always
computed and used to generate your project grade whether you
release test your project or not. If for some
reason you forgot to do release testing, we will run the tests
anyways. Of course, you could have fixed problems had you
release tested your project; you just lost the opportunity
to do better in the project (in case something went wrong
with release tests).
Notice that in most projects you can only perform release
testing after you pass all the public tests.
- Secret tests →
These are tests you don't have any access
to. You do not know what the tests are about and you will
not be able to see their result. You will know whether you
passed these tests once you receive the grade for your project.
After you received a grade for project, you are welcome to
see a TA during office hours to find out additional information about
release and secret tests you may have failed.