MSDN Magazine - June 2008 - (Page 60)

Identifying Test Data and reporting Metrics For parallel applications, test data is defined by identifying the scalability aspects of the scenario. The scalability exhibited by the application really depends on the parallelizability of the algorithm used. Certain algorithms may produce super-linear speedup for certain problem domains (for example, a search through a non-uniform distribution). Other algorithms may not produce embarrassingly parallel speedup due to limited parallelizability, resulting in total runtime speed increases that aren’t proportional to the number of CPUs. Still other algorithms (such as sorts) won’t scale linearly across data set size. As you can see, it is important to understand how the performance of the application scales across the following categories: The number of Threads or CPu Cores Typical user expectation is that the parallel application will scale linearly as the number of CPUs or threads increases. Dataset Size Typical user expectation is that the performance of the parallel application should not decrease when the size of the input dataset increases. application is greatly influenced by its algorithm’s parallelizability aspects, amount of sequential parts in the program, the parallelization overhead and the data/workload characteristics. Conducting performance testing allows stakeholders to understand and analyze the performance characteristics of parallel applications. The general methodology for conducting performance testing of parallel applications remains the same as that of performance testing sequential applications. The following section describes how some of the key performance testing steps are to be adapted for concurrent applications. For an understanding of the most helpful factors to measure, see the “Identifying Test Data and Reporting Metrics” sidebar. Execution Workload As the execution progresses, the amount of work done by the thread may not remain constant. It can increase or decrease, randomly vary, or be normally distributed. The variations in workload as the execution progresses can impact the performance characteristics of the application. The types of workload variations as execution progresses include: constant workload, increasing workload, decreasing workload, random workload and normally distributed workload. The execution time is the most common metric used for reporting. Other reporting metrics include speedup achieved in comparison with sequential application and speedup across the scalability categories. It’s important to define the reporting metric and how it is going to be measured. Some good questions to start with include: What values have to be reported? How will they be reported? Will the instrumentation used to take the measurement affect the performance? Further, when possible, take measurement in parts so it is possible to understand the performance of the subparts of the system. The most important step in performance testing is defining the objectives of the test. The most important step in performance testing is defining the objectives or purpose of the test. For parallel applications, valid test objectives include understanding the parallelizability/scalability of the algorithm used, understanding the performance characteristics of various design alternatives, discovering the synchronizations and communication overheads, and validating that performance requirements are met. It should be noted that the objective and scope of the performance testing could also vary based on who is conducting the test. For example, customers deploying a parallel application would perform performance testing to ensure that business needs are met, while the development team would be interested in conducting 60 msdn magazine exhaustive performance testing to identify bottlenecks and to improve the parallelizability of the program. Further, the test objectives can vary throughout the software development cycle. During the design and implementation phase, the test objective might be to understand and improve the performance characteristics of the application, while during testing and release phases (stabilization phases), the objective might be to ensure that performance does not regress across builds. Defining test scenarios and setting goals for them is another important step in the performance testing process. Development teams interested in understanding the performance characteristics of a parallel application should define three types of test scenarios—customer scenario tests, key performance indicator (KPI) tests, and micro benchmark (MBM) tests. The relationships between these different types of performance tests can be considered as equivalent to that which exists between system tests, integration tests, and unit tests. In other words, a customer scenario tests spawns a set of KPI tests and a KPI test spawns a set of MBM tests. Understanding the relationship between these different test types enables developers to understand performance relationships among various sub-parts of a parallel application. Gaining this understanding also permits better prioritization of performance bugs and, more importantly, allows the development team to respond to customer performance complaints with resultoriented suggestions. Setting performance goals for any non-trivial application is hard, let alone for parallel applications. One approach in setting goals is to derive the goal by triangulating metrics obtained from an expected algorithmic or theoretical performance metric for each component and sub-component of the application, from comparative analysis (comparison candidates include similar competitor applications) and from profiling the existing implementation/application (if one exists). Concurrency

Table of Contents Feed for the Digital Edition of MSDN Magazine - June 2008

MSDN Magazine - June 2008 Contents Toolbox CLR Inside Out Cutting Edge Patterns In Practice SAAS Concurrency Robotics Form Filler GUI Library Service Station Foundations Windows With C++ Concurrent Affairs Going Places { End Bracket }

MSDN Magazine - June 2008

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.