|To provide designers with deeper information about their
system's fielded performance, we are developing and evaluating a new
class of performance analyses. Starting with some designer-defined
system space, our techniques recast certain performance analyses as
large-scale statistically-designed experiments that are performed
collaboratively, and in parallel, on a wide variety of fielded
resources. The experimental analysis identifies a subset of variability
dimensions that most affect performance. Designers can then focus on
this much smaller system space when reasoning about certain future
design and development decisions.
This approach isn't meant to replace all in-house performance analyses. In particular, since we do not fully control the fielded resources on which the performance measurement tasks run, our approach can't provide completely noise free measurements. Instead, we seek to identify general trends and detect relative performance differences that will be experienced by eventual end users.
Some of our recent work is described in: