Revised February 23, 2002.

© 2002 by Charles Lin. All rights reserved. You must receive explicit written permission to copy information on this webpage. Updated in 2002.

© 2001 by Charles Lin. All rights reserved. You must receive explicit written permission to copy information on this webpage.

Background

For example, most beginning programmers believe "I must spend all my time coding, because time spent planning and testing code is a waste of time". Yet, in many software companies, you may spend more than half your time planning how to write code. This involves sitting down and determing what you want the code to do. You may think about the "actors" (objects that do things) and what they do ("actions"). Actors translate to classes and actions to methods in those classes.

Once those decisions have been made, and coding has begun, you need to think of how to test the code. Companies often have separate groups of people: those who write code (developers) and those who test code (testers).

At this point, you aren't ready to work in teams, so some of that kind of testing is postponed to a course in software engineering.

Nevertheless, it's a good idea to test your code. The question is: how?

Primary Input and Primary Output

Primary Outputs: Are They a Crutch?

Students seem fearful of writing their own test files before using primary inputs and outputs. They literally do not trust themselves to write their own test files successfully, and want to see an "official version". But think about it this way: if you're writing your own code, who's going to give you a primary input? Doesn't it make sense to learn how to write your own test code? This means not only "feeding" your program with your input, but also determining what output ought to be produced by your input.

Don't underestimate the importance of learning to write your own test files. Doing so forces you to determine what a reasonable output should be. Many CMSC 106 students are, obviously, inexperienced programmers. Some of them try to match primary output. When they fail to match primary output, they scratch their heads and head to a TA.

A typical encounter goes like this: "TA, TA! My output doesn't match primary". "Student, what's wrong? Why doesn't it match?" "TA, TA! I don't know! I looked at my output, and it's different from the primary." "Stduent, if I took away the primary output, could you tell me what output the primary input would produce?" "TA, TA! Don't touch my primary output! How else am I supposed to know what's going on?"

OK, so the scenario is artificial, but not too far from what happens. Students often come to office hours, and have no idea how the primary output is produced. All they know is that their own output and the primary output are different. In effect, these students have become human "diff" machines, unaware of why their output is different.

You may believe that matching the primary output is the quickest way to get finished. Perhaps it is. However, you may be hurting yourself in the long run, if you don't learn how to write your own test files. And you may also be hurting yourself if you don't test your classes before writing test files.

Testing Classes

Some people believe in writing as much code as possible, without a thought to testing the code. They do so even as they've been told to test after each method. Such people believe in the "code first, test (much) later" approach to code writing. There is some virtue to this approach. You learn to write lots of code. You learn to deal with hundreds of errors messages at a time, without flinching. You also may be spending more time debugging than you need to.

There's an approach you can take that may seem rather radical. You can actually write a "test driver" prior to coding the class. Yes, before coding the class. Of course, you might wonder how it's possible to do this. This doesn't happen until you plan the class ahead of time. This means writing down methods, data members, etc.

We'll look at how to try this idea out.

You can still even write such test drivers after the fact. You've written your class, and now you want to test it. After all, you should check to see if your class works, right?

runTestDriver()--Organizing the Test Driver Code

Where should you write the testing code? One possibility is to create a file, such as FooTester.cpp, have a main() and put all the code there.

However, there's a nicer place to put the testing code (to be called test driver)---in the class itself.

In the class that you plan to test, write a static method called runTestDriver(). Why static? A static method is one that doesn't belong to an object. It's basically like a standalone function, but resides inside a class. Thus, you are forced to declare variables of the class, etc., inside this method.

Let's look at the details:

1. First, in your .h file, write

   static void runTestDriver();
   

   in the public section of your class. This will allow the test driver to be run in any function (say, main()) as follows:

   int main() {
       Foo::runTestDriver();  // runs test driver for Foo object
       return 0;
   }
   

   Since the method is static, you need to call it by prefacing the method call with Foo:: or the name of the class you are testing, followed by two colons.

   Convention Call the function you want to run tests on runTestDriver().

   If you need to have helper functions, you should preface them with testDriver. For example, suppose you need to do some operation such as "count", then you can write a static (yes, static) method called testDriverCount(). This method, like all helper functions, can be placed in the PRIVATE section of the class header file.

2. Second, write the method in your .cpp file, as in:

   void Foo::runTestDriver() {
      // put your testing code here
   }
   

   Notice that they word static is no longer here. Just some C++ rule.

   OK, now you know where to put the code, let's look at an example.

   Suppose you're starting to write a class. You want to test a method at a time. Which methods should be implemented first, so that it makes testing easier? Here's how I would do it:

   1. Write the default constructor and print() method first (which can be used to implement the output operator).

      You want to start simple, and the simplest place to start is the default constructor. You also need a way of determing whether the object is "correct" and one way to do this is to be able to print it out.

      Here's how I suggest you start out (this is in Fract.cpp):

      #include <iostream>
      #include "Fract.h"
      
      using namespace std;
      
      void FractDriver::runTestDriver()  
      {
         Fract f1;  // default fraction
      
         cout << "Testing default constructor" << endl;
         cout << "---------------------------" << endl;
         cout << f1 << endl;
         cout << "[ANSWER] 0 / 0" << endl << endl;
      }
      

      This can be called in main() in, say, main.cpp.

      #include <iostream>
      #include "Fract.h"
      
      using namespace std;
      
      int main()
      {
         FractDriver::runTestDriver();  // calls driver
      }
      

      It's easy, it's simple. But there are important features. First, there are some output lines to indicate what kind of test is being run. In this case, it's the default constructor being run.

      Then, the object is printed. Then, the "expected answer" is printed. It's useful to print the expected answer, so you can anticipate what you think the output is. It's often a good idea to know what you think the output is and see if it is indeed the output.

   2. You may wish to test other constructors (except the copy constructor, which is tested later)

      #include <iostream>
      #include "Fract.h"
      
      using namespace std;
      
      void Fract::runTestDriver()
      {
         Fract f1( 2, 6 );  // fraction 2/6 
      
         cout << "Testing next constructor" << endl;
         cout << "------------------------" << endl;
         cout << f1 << endl;
         cout << "[ANSWER] 1 / 3" << endl << endl;
      }
      

   3. If you've written the copy constructor and assignment operator, you may want to test that next.

      If you haven't written it because you intend to use the implicit version created by the compiler, then testing the copy constructor and assignment operator may be silly. Still, it doesn't hurt. When you test these methods, create an "original" object, make a copy (one copy using the copy construtor and another using assignment operator), modify the original, and see if the copy is modified. If it's a deep copy, then the copy should be the copy of the original PRIOR to the original being modified. Here's a sample test driver.

      #include <iostream>
      #include "Fract.h"
      
      using namespace std;
      
      void FractDriver()
      {
         Fract orig( 2, 5 ); 
         Fract copyOfOrig = orig; // copy constructor 
         Fract secondCopy;
      
         secondCopy = orig; // assignment operator 
      
         orig.add( 1, 5 ); // modify orig 
      
         cout << "Testing copy constructor" << endl;
         cout << "------------------------" << endl;
         cout << orig << endl;
         cout << "[ANSWER] 3 / 5" << endl;
         cout << copyOfOrig << endl;
         cout << "[ANSWER] 2 / 5" << endl << endl;
      
         cout << "Testing operator=" << endl;
         cout << "-----------------" << endl;
         cout << orig << endl;
         cout << "[ANSWER] 3 / 5" << endl;
         cout << secondCopy << endl;
         cout << "[ANSWER] 2 / 5" << endl << endl;
      }
      

      As you may notice, it may not be a good idea to test the copy constructor and assignment operator right away. The reason? You need methods to modify the original object. If you only have a default constructor, the object won't have changed much. Especially for objects that store dynamic memory (linked lists, trees), you need methods, such as add() so the object will be complex enough to fully test the copy constructor/assignment operator.

      Rule of Thumb For complex objects, test copy constructor and assignment operator later. Get simpler methods to work first.

      This is especially true if you are testing linked lists. You need a way to create something besides an empty linked list to properly test the copy constructor/assignment operator.

   4. Test the other methods.

      There should be several other methods, and you should write tests for them.

   How We Can Check What Your Test Driver Does

   While you can testing as throughly or as leniently as you want, we should be able to see the results of your test by calling your test driver. Thus, we should be able to make a call to Foo::runTestDriver() (where Foo is replaced by the class you are testing), and it should print information.

   Drawbacks of Printing

   Other than using a debugger, the most common way to debug code is to print. Usually, one uses a debugger to locate an existing bug. Writing test drivers is a preventive measure. You try to find the bugs, before they find you.

   The advantage of printing is that it's easy to write the code. However, printing output has drawbacks. It's hard to automate. You, the programmer, must sit down and determine whether your code passed the test or not by visually inspecting line after line of output.

   Wouldn't it be nice if you could somehow automate the steps so that it only prints when something has failed? That would be convenient.

   Alas, it takes more time to write automated tests, than simply printing everything. But that time may be well worth it (if you have lots of time). You can often run test after test, with very little effort. You don't have to recompile code.

   To write such tests, you have to be reasonably clever to come up with tests to fit whatever class you are testing. Let's look at an example. Suppose you wish to test whether a sorted linked list is sorting correctly or not. You can create an input file that looks like:

   add 4
   add 3
   add 10
   add 8
   add 5
   answer 4 3 5 8 10
   

   Your test driver processes the commands on each line. When the test driver sees "add", it adds the number as a new element of a linked list. When your test driver sees "answer", it will double check to see if the answer is correct. The "answer", in this case, is 4 (indicating the size of the list), followed by the four numbers in sorted order. In effect, it's what's supposed to be in the linked list. The test driver will check the expected answer (from the input file) to the values in the linked list, and see if they are the same.

   For example, here's a way to to write a test driver that can handle the above input file.

   #include <iostream>
   #include "List.h"
   
   using namespace std;
   
   void List::runTestDriver()
   {
      List list;
      string cmmd;
   
      while ( cin >> cmmd )
       {
         if ( cmmd == "add" )
           {
               int num;
               cin >> num;
               list.add( num );
           }
         else if ( cmmd == "answer" )
           {
               Iterator iter = list.iterator();
               int size;
               cin >> size;
               if ( size != list.size() )
                  cout << "Error: sizes do not match" << endl;
   
               int *arr = new int[ size ];
               int index = 0;
               for ( iter.goFirst(); iter.inList(); iter.goNext() )
                  {
                      if ( iter.getCurrent() != arr[ index ] )
                        {
                            cout << "Error at index " << index << endl;
                            cout << "Expected " << arr[ index ] 
                                 << " but see " << iter.getCurrent() << endl;
                        }
                      index++;
                  }
           }
         else
            cout << "Invalid command: " << cmmd << endl;
       }
   
   }
   

   This isn't the cleanest test driver (there could be many improvements). However, it gives you some idea how you could write a test driver which can "automatically" check if the answers are correct, and only print errors when something is wrong.

   Admittedly, the input file requires that you write the solution down, but once you do, it's easy to create many variations of the input, without having to recompile the code.

   Writing Debugging Methods

   Sometimes, it's even useful to write methods whose sole purpose is to help you debug the class. For example, suppose you have a sorted linked list. You might want to write a verify() method.

   This method might have a prototype that looks like:

       bool debugVerify( int arr[], int size );
   

   The word "debug" is placed in front so that it's easy for the person looking at the class that it's a "debug" method, and not really meant for use by the user of the class.

   This method takes an array and the array's size as input. It assumes the array is sorted with exactly the same contents as the linked list. Again, you should be able to see how this method can be used with the driver in the previous method (which checked for the "answer").

   While this method is not useful to the "user" of the class, it is useful to you, the developer of the class.

   Figuring out Test Cases

   For complicated classes, like linked lists, you may need to create very thorough tests. For example, to properly test a linked list, you should create all possible ways to insert three values (e.g., largest, middle, smallest, OR smallest, largest, middle, etc). If you can get them all correct, then it probably works on larger inputs sizes.

   The key is to make simple tests first, which test all possible situations, then work your way up to more complicated tests. Some programmers pick very difficult cases first (often relying on the sample input or output provided), but refuse to try simple cases (complaining it takes too much time to write their own test cases). Start with simple input cases, debug that first, then work your way to more complicated cases. You'll discover that fixing problems in small test cases often means avoiding problems in large test cases.

   Code Coverage

   Although we won't use this method much, another way to test code, is to create tests based on code coverage. There are two rules for writing test cases based on code coverage: one for "if-else" statements and one for loops.

   if-else statements

   Suppose you have an if-else statement like the one below.

   if ( <cond 1> )
     {
       cout << "In body 1" << endl;
       // body 1 
     } 
   else if ( <cond 2> )
     {
       cout << "In body 2" << endl;
       // body 2 
     } 
   else if ( <cond 3> )
     {
       cout << "In body 3" << endl;
       // body 3 
     } 
   

   You will need to write four test cases. The first test case should cause "In body 1" to print. The second should cause "In body 2" to print. The third should cause "In body 3" to print. The fourth should cause nothing to print (since there is no final else case).

   Loops

   while( <cond> )
     {
       cout << "In while body" << endl;
       // body  
     } 
   

   For a loop, you need to write one test that causes the body to print, and one test for it not to print at all.

   Combining Loops and if-else

   Using the two rules above, you should be able to combine them to determine all possible paths through your method, and generate a test for each possible path. Unfortunately, if your method is very long, this may require many, many tests. This is one reason to keep your methods short (by calling helper functions)---it makes it easier to test.

   There are some software tools that check for test coverage. Basically, you use the tool to "instrument" your executable (which modifies the executable with special testing code). Then, you run a series of test files. The tool checks to see which lines of code are run and which are not run.

   If a certain line of code is not run, then there are two possibilities. First, you did not create a test case that would make the code run (and thus, that piece of code was untested), or second, it's impossible to get to that piece of code (because of various conditions that are impossible to satisfy), in which case, the code is not important and shouldn't be there.

   Unfortunately, it's hard to tell which of the two cases has occurred. All you know is that some code wasn't run, so it wasn't tested. Based on that information, you need to figure out if it's possible to write a test case to make the code that didn't run, run.

   We won't be using such tools in this course, but it's useful to know that they exist.