This page describes the results from evaluations of the technique.
The example-based technique was identified as an effective strategy in our environment. While we cannot say in all cases that an example-based learning approach would be superior to one based on the class hierarchy and model of interaction, the indication of this study was that for novice users, the examples were a more effective way to learn. Several of our subjects had recognized benefits of the HB technique but were unable to apply it due to their lack of familiarity. They expressed this in comments during the interviews such as: "The HB procedure was more similar to what I normally do, but…" or "I found the examples limiting in some ways and thought the HB procedure would address this problem, but…" It is possible that, if our subjects had had more experience with the framework, the HB procedure would have proven better suited to their needs. Indications are that hierarchy-based procedure required more experience with the framework to be used effectively.
Since the subjects of our study only had significant experience with the EB technique, we can report only on the characteristics of example-based strategies. We identified two main types of example-based strategies: strictly adaptive and ad hoc adaptive, each with its own strengths and weaknesses. The relative effectiveness of each seems to be most strongly determined by how closely the object model of the system to be developed corresponds with the existing applications. We observe that when the functionality called for by the object model is well-contained in the set of existing applications, just about any example-based technique should be helpful. However, a strictly adaptive technique can't take the developer far beyond what is provided by the existing applications themselves. In a situation in which the set of applications is sparse and does not contain the necessary functionality, an ad hoc technique may be more appropriate. On the other hand, if the set of applications is particularly large, then a strict adaptation technique may be most helpful. Despite its weaknesses, such a technique in procedural form was shown to guide the developer toward implementing the object model "as is" and away from "gold-plating," or spending time providing extra features that seem nice but are not necessary.
From the experiences of our beginning learners, we also have evidence about other characteristics that are required by example-based techniques in order to be successful. For instance, comments from our learners seem to indicate that future studies need to be undertaken to determine if we can add better guidance for helping developers find functionality in existing example applications. Their experiences in this study also indicate that example-based techniques should guide developers to begin their implementation from an existing application, if a suitable one can be found, and to stay closely to the original object model once implementation has begun.
The results of this evaluation study are explained in more detail in a technical report and a conference proceedings. Our raw data is also available on the web, in the form of a summary spreadsheet (in tab-delimited text format) and our qualitative notes on each of the teams.