Content

    Emerging trends and challenges

    Solution approach: Distributed Continuous QA

Emerging Trends and Challenges

Software quality assurance (QA) tasks are typically performed in-house by developers, on developer platforms, using developer-generated input workloads. One benefit of in-house QA is that programs can be analyzed at a fine level of detail since QA teams have extensive knowledge of, and unrestricted access to, the software. The shortcomings of in-house QA efforts, however, are well-known and severe, including (1) increased QA cost and schedule and (2) misleading results when the test-cases, input workload, software version and platform at the developer's site differ from those in the field. These problems are magnified in performance-intensive software, such as that found in high-performance computing systems, distributed real-time and embedded systems and the accompanying systems software (e.g., operating systems, middleware, and language processing tools). This is because this software is increasingly subject to the following trends:

These three trends present new challenges to developers. One major new challenge is the explosion of the software configuration space. To support customizations demanded by users, performance-intensive software must run on many hardware and OS platforms and typically have
many options to configure the system at compile- and/or run-time. For example, web servers (e.g., Apache), object request brokers (e.g., TAO), and databases (e.g., Oracle) have dozen or hundreds of options. While this flexibility promotes customization, it creates many potential system configurations, each of which deserves extensive QA.

When increasing configuration space is coupled with shrinking software development resources, it becomes infeasible to handle all QA in-house. For instance, developers may not have access to all the hardware, OS, and compiler platforms on which their software will run. In this environment developers are forced to release software with configurations that have not been subjected to extensive QA.
Moreover, the combination of an enormous configuration space and tight development constraints mean that developers must make design and optimization decisions without precise knowledge of the consequences in fielded systems.
 

Solution Approach: Distributed Continuous QA

To address these challenges, we are conducting a collaborative research project called Skoll. Skoll is a general, continuous, feedback-driven process, supported by automated tools to carry out around-the- orld, around-the-clock QA. Our approach divides QA processes into multiple subtasks that are intelligently distributed to client machines around the world, executed by them, and their results returned to central collection sites where they are fused together to complete the overall QA process.
 
Creating Skoll presented challenges for which we have created the following novel solutions: