Gregory M. Kapfhammer

Evaluating Test Suite Prioritizations with Coverage Effectiveness

Introduction

Test suite prioritization techniques reorder a test suite in order to improve the effectiveness of the testing process. This short tutorial explains how to calculate the coverage effectiveness (CE) of a test suite, regardless of the coverage criterion or the test automation framework. The current coverage calculator is implemented in the R programming language. Please follow the below steps in order to learn how to calculate CE values for a simple test suite. The R source code is currently licensed under the GNU General Public License, Version 2. Please contact Dr. Gregory M. Kapfhammer if you have questions about the tutorial, the source code, or the software license.

Download and Installation

1. Download and install the R programming language. This tutorial was tested with R version 2.5.1 (2007-06-27). The demonstration should work properly regardless of whether you execute R on the command line or in the Emacs Speaks Statistics (ESS) module. However, this tutorial was preprepared by running the provided R program in Emacs and ESS 5.3.4.

2. Download the file CoverageEffectiveness.R. This is an R program that can calculate the coverage effectiveness value for a test suite prioritization. Note that this source code assumes that the gtools package is installed into your R distribution. You may want to learn more about gtools and subsequently install it using the install.packages function provided by R. You can type help(install.packages) at the R prompt if you want to learn more about how to use this function.

3. Download the files CoverageData.txt and TestTimingData.txt. These files contain the coverage information and test case execution times for a test suite that consists of three test cases. The tool paper Test Suite Reduction and Prioritization with Call Trees, published at the IEEE/ACM International Conference on Automated Software Engineering 2007 describes the example test suite in more detail.

Calculating Coverage Effectiveness

1. After executing R and entering the R environment, run the below command in the directory where you saved CoverageEffectiveness.R. For all of these examples, the > symbol corresponds to the R prompt that you see in Emacs/ESS or the R environment. You should see the following output when you execute the ls() command. The second block of output shows all of the function names and global variables that are provided by CoverageEffectiveness.R.

   > source("CoverageEffectiveness.R")
         

   > ls()
    [1] "AllCoverageEffectiveness"        "AllUnitCoverageEffectiveness"      
    [3] "CalculateCoverageEffectiveness"  "CalculateUnitCoverageEffectiveness"
    [5] "CoverageDataFile"                "CoverageEffectiveness"             
    [7] "DissCoverageDataFile"            "DissCoverageDataFileLarge"         
    [9] "DissTimingDataFile"              "DissTimingDataFileLarge"           
   [11] "GlobalCoverage"                  "GlobalTiming"                      
   [13] "PrintGlobals"                    "ReadCoverageData"                  
   [15] "ReadTimingData"                  "SaveGraphic"                       
   [17] "SaveGraphicPaper"                "SaveGraphicWeb"                    
   [19] "SetGlobals"                      "TestTimingDataFile"                
   [21] "UnitCost"                        "UnitCoverageEffectiveness"         
   [23] "libraries"                       "reloadCoverageEffectiveness"    
         

2. Next, we must read in the test coverage and execution time data from the file system. In order to read in the inputs and check that these steps worked properly, you should execute the following commands. The output indicates that test case T1 covers the requirements R1 and R2. We also see that test T2 takes ten seconds to execute.

   > Coverage = ReadCoverageData(CoverageDataFile)
   > Coverage
     NameCD Requirement
   1      1           1
   2      1           2
   3      2           1
   4      2           2
   5      2           3
   6      2           5
   7      3           1
   8      3           4
   9      3           5
   
   > Timing = ReadTimingData(TestTimingDataFile)
   > Timing
     NameTTD Time
   1       1    5
   2       2   10
   3       3    4
        

3. At this point, we can create a tentative ordering of the test suite and calculate the corresponding CE value. We will call the CalculateCoverageEffectivness function that accepts four different parameters. If you know that you are going to use the same coverage and timing data for many function calls, you may consider setting these as global variables and then using the simpler CoverageEffectiveness function. The following example shows the CE value for the test suite ordering T1, T2, T3. The FALSE parameter indicates that we do not want to visualize the cumulative coverage of the test requirements.

   > CalculateCoverageEffectiveness(c(1,2,3), Coverage, Timing, FALSE)
   [1] 0.3789457
   

4. Next, we will try to calculate a coverage effectiveness value, visualize the coverage function, and then save the result in a graphic that can be used in a research paper or a Web site. The function SaveGraphicPaper will make an encapsulated postscript (EPS) file that can be placed into a LaTeX paper while the SaveGraphicWeb creates a Portable Network Graphics (PNG) file that can be incorporated into a Web site, as seen below. The default SaveGraphic function is currently implemented to create an EPS file.

    
   > CalculateCoverageEffectiveness(c(1,2,3), Coverage, Timing, TRUE)
   > SaveGraphicWeb("Prioritize123.png")
   > SaveGraphicWeb("Prioritize123.eps")
   

   
5. When the test suite has a small number of test cases, it is possible to automatically calculate the CE value for all of the possible prioritizations. For example, the following function calls will make the collection of graphs that are seen below. Note that this example shows that the orderings T3, T2, T1 and T3, T1, T2 have the highest coverage effectiveness values. Therefore, these orderings are judged to be the best according to the CE metric.

   > libraries()
   > AllCoverageEffectiveness(Coverage, Timing, AllGraphs=TRUE)
     Ordering        CE
   1  1, 2, 3 0.3789457
   2  1, 3, 2 0.5052638
   3  2, 1, 3 0.3789432
   4  2, 3, 1 0.4315752
   5  3, 1, 2 0.5789495
   6  3, 2, 1 0.5789423
   > SaveGraphicWeb("AllPrioritizations.png")
   > SaveGraphicPaper("AllPrioritizations.eps")
   

   

Conclusion

Calculating coverage effectiveness is important because it enables both researchers and practitioners to assess the usefulness of a test suite prioritization technique. The CE metric only requires the provision of coverage and test timing information and it does not require fault seeding. However, if timing data is not available, then it is possible to calculate the unit coverage effectiveness value. The approach assumes that each test case executes in the same amount of time. You are welcome to experiment with CoverageEffectiveness.R and leverage this system during your new experimental study of regression testing algorithms. While this tutorial does not aim to be a comprehensive introduction to either the calculation of coverage effectiveness or the R source code, it should serve as a basis upon which to build further investigations.

Related Papers:

• Adam Smith, Joshua Geiger, Gregory M. Kapfhammer, and Mary Lou Soffa. Test Suite Reduction and Prioritization with Call Trees (Tool Paper). In the Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, Atlanta, Georgia, November 2007.

• Gregory M. Kapfhammer and Mary Lou Soffa. Using Coverage Effectiveness to Evaluate Test Suite Prioritizations. In the Proceedings of the ACM International Workshop on Empirical Assessment of Software Engineering Languages and Technologies, Atlanta, Georgia, November 2007.