Code Coverage and Hotspot Tutorial

This tutorial gives an overview of using the Coverage and Hotspot Analyzer to check the code coverage of a test suite, and to optimize code by identifying hotspots.

The sample data is available in the Analyzer's default database (example_db) which includes experiment data for the Ana-Find program and the Data::Dump module.

Example Ana-Find

This project illustrates the use of hotspot analysis to optimize code. The Ana-Find script parses dictionary files looking for anagrams. The project contains numerous experiments which contain coverage and hotspot information gathered from the command 'perlcov-perl ana-find >/dev/null' (perlcov-perl is the replacement interpreter used by the Analyzer to collect statistical data about your program at runtime). Experiments in this project get gradually faster as hotspots are identified and the code is revised.

Viewing the Ana-Find Experiment Data

Expand the project tree for the Example Ana-Find project by clicking the "+" on the left. A number of experiments appear sorted by project name, which happens to be the file's source code control change number.

With each successive experiment, the runtime (displayed in seconds in the Runtime column) decreases. The number of subroutines (the # Subs column) decreases from seven in the first experiment to only two in the last.

One of the most significant performance improvements in the Ana-Find project can be seen between the experiments "Change 269172" and "Change 269173". We can see that the test (four repetitions of the command 'perlcov-perl ana-find >/dev/null') took 129.391 seconds for Change 269172 and only 90.519 seconds for Change 269173. Ana-Find went from 34 lines of code (# LOC) to 33, and from five subroutines (# Subs) to four. We can find out where in the code this optimization happened by comparing the two versions.

Comparing Experiments

Click on Change 269173 to select it, then click "Mark for Compare" in the toolbar, View menu or right-click context menu. This marks the experiment with a check. Now click on Change 269172 to select it, then click "Compare Hotspots". While an experiment is selected, applicable keyboard shortcuts (like space bar for "Mark for Compare") are displayed in the status bar.

The Comparison View window appears showing Change 269172 on the left and Change 269173 on the right.

The Sub column displays a list of subroutines in the code. The T1 (selected experiment) and T2 (marked for comparison experiment) columns show the time taken by each subroutine (in seconds).

In the Sub list we see that the 'signature' subroutine has no time information in the T2 column (corresponding to Change 269173). This is the missing subroutine we noticed in the Experiment View. Click on the signature subroutine to jump to the relevant code in the Change 269172 version.

The "%" column on the left indicate the percentage of time spent on each line of code, in this case sub signature took 17% of the total runtime of the script.

There is no corresponding sub in the Change 269173 version (the left code display pane is blank), so removing this subroutine appears to have saved us 9.149 seconds. However, the program still needs that join to work. Click on the 'find_anagrams' sub in the top pane to see where it went.

Change 269172 has the following on lines 47 through 49:

Change 269173 incorporates the join in line 43, so the signature sub is not needed:

Adding the join to find_anagrams increased the time spent on that line from 22% to 24% of the total runtime, but this negligible compared with the time saved by not calling a separate signature subroutine.

Example Data-Dump

This project shows the evolution of the test suite for the Data::Dump module. Click the "+" next to Example Data-Dump to expand the Project tree and view the experiments.

The experiments in this project were generated by running `make test` against successive versions of Data::Dump. In version 0.01 only 75 of 190 lines of code was covered, but by version 1.06 this had increased to 333 of 378 lines.

To see if the code coverage of Data::Dump's test suite has improved in a more recent version, we will download version 1.07 of the module, and create a new experiment on this codebase.

Data::Dump 1.07 can be downloaded here:

http://cpan.org/authors/id/G/GA/GAAS/Data-Dump-1.07.tar.gz

Unpack the tarball in a convenient directory, and run perl Makefile.PL there to prepare the package's Makefile. Then, in the Analyzer, click the Add Experiment button.

To keep the naming scheme consistent, we will call this experiment "1.07", but you could specify any name you choose here. For "Working Directory", select the directory where you have unpacked Data-Dump-1.07.tar.gz.

Unlike the Ana-Find experiments, perlcov-perl does not need to be specified in the experiment's "Run Command" field. The Analyzer automatically uses it when 'make test' is invoked. Click OK to run the experiment.

The "Active Runs" dialog box appears displaying the standard output of 'make test'. If anything goes wrong with the specified command, you can see the it here. Click "Dismiss" to close the dialog box.

A new experiment named "1.07" should appear which shows us an overview of the coverage information for the version we've added. We can see it has 11 files, 10 invocations of perl, 13 subroutines, and 389 lines of code in total -- 379 of which were covered by 'make test'.

Select experiment 1.07 and click "View Coverage".

The Experiment View window appears showing us the coverage information for experiment 1.07. We see a larger version of the Coverage histogram shown in the Project View. Though almost all the code has been covered in the experiment, by clicking on the light green part of the histogram at the far left (or by sliding the black triangle over to it), we can focus on the subroutines where coverage is not quite complete. With "0-89%" selected in the histogram, our Sub list is limited to "Data::Dump::tied_str" and "Global code in t/quote-unicode.t" -- the only parts of the module with less than 98% coverage.

Select "Data::Dump::tied_str" to jump to sub tied_str in the code. Lines 339 and 340 are marked in light blue (indicating "cold" areas of the code). If we're curious about why these lines were not covered by the test, we can hover the mouse pointer over the bolded if statement on line 335.

A hover tip appears telling us that the else was never hit because the initial test was always true. In this particular case, it is probably not worth changing the test suite to cover this condition.

Have a look through previous versions of Data::Dump to see how the test suite has been improved over the course of several revisions. The experiments can be compared the same way we did when examining different versions of Ana-Find, highlighting where test coverage has improved.