MSDN Magazine - April 2008 - (Page 20)

Figure 3 Using a Debugger to Validate Performance Results Operation Measured MethodCalls: EmptyStaticFunction() [count=1000 scale=10.0] MethodCalls: aStructWithInterface.Interface() [count=1000 scale=10.0] Median 1.000 0.031 Mean 0.964 0.029 StdDev 0.102 0.012 Min 0.857 0.021 Max 1.196 0.039 Samples 10 10 lows me to choose the most natural alternative, knowing I was not sacrificing performance to do so. Validating Your Performance Results The MeasureIt application makes collecting data for a broad variety of benchmarks very easy. Unfortunately, MeasureIt does not address an important aspect of using benchmark data: validation. It is extremely easy to measure something other than what you thought you were measuring. The result is data that is simply wrong, and worse than useless. The old adage “if it sounds too good (or bad) to be true, it probably is” definitely applies to performance data. It is imperative that you validate data that you use in any important design decision. What does it mean to validate performance results? It means collecting other information that also will predict the performance result and seeing if the two methodologies agree. For very small micro-benchmarks, inspecting machine instructions and making an estimate based on the number of instructions executed is an excellent check. In a debugger like Visual Studio, it should be as easy as setting a breakWhat does it mean to point in your benchmark validate performance code and switching to the results? It means collecting disassembly window (Deother information that also bug -> Windows ->Diswill predict the performance assembly). Unfortunately, result and seeing if the two the default options for Visual Studio are designed to methodologies agree. simplify debugging, not to do performance investigations, so you need to change two options to make this work. First, go to Tools | Options… | Debugging | General and clear the Suppress JIT Optimization checkbox. This box is checked by default, which means that even when debugging code that should be optimized, the debugger tells the runtime not to do so. The debugger does this so that optimizations don’t interfere with the inspection of local variables, but it also means that you are not looking at the code that is actually run. I always uncheck this option because I strongly believe that debuggers should strive to only inspect, and not to change the program being debugged. Note that unsetting this option has no effect on code that was compiled for debugging since the runtime would not have optimized that code anyway. Next, clear the Enable Just My Code checkbox from Tools | Options | Debugging | General dialog. The Just My Code feature instructs the debugger not to show you code that you did not write. Generally, this feature removes the clutter of call frames that are 20 msdnmagazine CLR Inside Out Validating Micro-Benchmark Data with a Debugger often not of interest to the application developer. However, this feature assumes that any code that is optimized can’t be yours (it assumes your code is compiled using the debug configuration or suppressed JIT Optimizations is turned on). If you allow JIT optimizations but don’t turn off Just My Code, you will find that you never hit any breakpoints because the debugger does not believe your code is yours. Once you have unchecked these options, they remain unchecked for ALL projects. Generally this works out well, but it does mean that you don’t get the Just My Code feature. You may find yourself switching Just My Code on and off as you go from debugging to performance evaluation and back. As an example of using a debugger to validate performance results, you can investigate an anomaly in the data shown in the excerpt in Figure 3. This data shows that calls to an interface method of a C# structure is many times faster than a call to a static method. This certainly seems odd, given that you would expect a static method call to be the most efficient type of call. To investigate this, you set a breakpoint in this benchmark and run the application. Switch to the disassembly window (Debug -> Windows -> Disassembly) and see that the whole benchmark consists of just the following code: aStructWithInterface.InterfaceMethod(); 00000000 ret What this shows is that the benchmark (which is 10 calls to an interface method) has been inlined away to be nothing. The ret instruction is actually the end of the delegate body that defines the whole benchmark. Well, it is not surprising that doing nothing is faster than doing method calls, so this shows the reason for the anomaly. The only mystery is why static methods don’t get inlined, too. This is because for static methods, I specifically went out of my way to suppress inlining with the MethodImplOptions.NoInlining attribute. I intentionally “forgot” to put this on this interface call benchmark to demonstrate that the JIT compiler can make certain interface calls as efficient as non-virtual calls (there is a comment mentioning this above the benchmark). Wrapping Up To reiterate, it is very easy to measure something other than what you intended, especially when measuring small things that are subject to JIT compiler optimizations. It is also very easy to accidentally measure non-optimized code, or measure the cost of JIT compilation of a method rather than the method itself. The MeasureIt /usersGuide command will bring up a user’s guide that discusses many of the pitfalls you might encounter when creating benchmarks. I strongly recommend that you read these details when you are ready to write your own benchmarks.

Table of Contents Feed for the Digital Edition of MSDN Magazine - April 2008

MSDN Magazine - April 2008 Contents Toolbox CLR Inside Out Basic Instincts Cutting Edge Foundations Test Run Service Station Windows with C++ Going Places { End Bracket }

MSDN Magazine - April 2008

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.