Creating and deploying DCQA processes presented challenges for which we have created the following novel solutions:
  • Modeling the QA subtask configuration space: We formally model aspects of the QA subtasks and underlying software that will be varied under control of the distributed process. This includes not only process and software configuration parameters, but also constraints among them. To do this, we developed a general representation with configuration options, option settings, and inter-option constraints. We also developed the notion of temporary inter-option constraints to help us reduce configuration space size artificially in certain situations.
  • Exploring the configuration space: The configuration space of a QA process for a performance-intensive infrastructure system can be quite large. Even with a large pool of user-supplied resources, brute-force approaches may be infeasible or simply undesirable. Consequently, we developed techniques to explore/search the configuration space. We developed a general search strategy based on uniform sampling of the configuration space and supplemented it with customized adaptation strategies to allow goal-driven process adaptation.
  • Feedback: As subtasks are scheduled and executed in parallel at several sites, feedback (subtask results) is collected. QA processes can analyze this feedback and modify their behavior based on it. We have developed techniques for automatically characterizing such feedback, visualizing it, and adapting the QA process. 
  • Unpredictable resource availability: Since QA subtasks are assigned to remote machines, volunteered by end users, we cannot know when resources will be available. Moreover, some volunteers may wish to maintain some control of how their resources will be used; for example limiting which version of a system can undergo QA on their resources. In such cases, it is impossible to pre-compute QA subtask schedules. Therefore, we have developed scheduling techniques that adapt based on a variety of factors including resource availability.
  • Process management: We developed a new process, called the Skoll process, that provides a flexible framework to integrate the above mentioned QA techniques and tools.
Some recent articles: