Gregory M. Kapfhammer

Associate Professor of Computer Science

http://www.cs.allegheny.edu/~gkapfham/

  
View  Edit  Uploads  History  Top  Changes  Search

Software Unit Test Coverage and Adequacy


  • Zhu et al. assert that software test data adequacy criterion are normally used for two different purposes. What are these purposes and how are they related?

the two purposes are:
  1. to specify a particular software testing requirement, and the test cases to satisfy that requirement. it uses test case selection criterion or test data adequecy criterion.
  2. to determine the observations that should be made during the testing process. it uses statement coverage, path coverage, and mutation score.
they are related in that one must first determine what one wants to observe in the program, and then decide what testing requirement and test cases to use.
Deb Wright
Uses of software test data adequacy criterion:
  • It can be used as a stoping rule to decide when enough test cases have been performed.
  • To demonstrate the code is covered and thus adequate.
Both help provide a software dependability assessment.

Elizabeth Zehner
Software test data adequacy criterion are normally used for the following two different purposes. First, it ensures that before testing the objectives and goals of the test are known. It is also important that the goals of the tests are capable of being measured. In other words they should be “quantified, reasonable, and achievable.” Secondly, software test data adequacy criterion helps testers manage the software testing process so the quality of software is ensured by the tests. This allows the tests not to be redundant and unnecessary. Together these two create a confidence in the system and provide an assessment of the software.
Noosh Moussavi
  • In this article, Zhu et al. state that adequacy criterion are normally based on one of two information sources. What are these information sources? Please provide one or two examples of each of these information sources.
  1. Specification based – test set is adequate if all the identified features have been fully exercised. Black box testing, structural testing
  2. Program based – test is adequate if program has been fully exercised. White box testing, error based testing

Deb Wright

The two uses of adequacy criterion are specified by their name:
  • Specification-based: Use feature specifications to measure test adequacy. The test set is adequate if all the specifications are in place and work correctly.
  • Program-based: If the program has been thoroughly exercised, the test set is considered adequate.

These can be used together as combined specification- and program based-criteria. This just uses both adequacy criteria in determining test set adequacy.
Elizabeth Zehner
The two ways that you can categorize adequacy criteria: specification-based and program-based. Specification-based specifies the required testing in terms of identified features of the specification or the requirements of the software. This allows a test set that is adequate if all the required and recognized features are implemented. Program-based specifies testing requirements in terms of the program being tested. It decides if a test is adequate depending on whether or not the program has been completed implemented.
Noosh Moussavi
  • What are the three basic approaches to testing? If you are using JUnit to write test cases, which type of testing approach are you employing?
  1. Structural testing: specifies testing requirements in terms of the coverage of a particular set of elements in the structure of the program or the specification.
  2. Fault-based testing: focuses on detecting faults in the software, it has some measurement of the fault detecting ability of test sets.
  3. Error-based testing: requires test cases to check the program on certain error-prone points according to our knowledge about how programs typically depart from their specifications.
If you are using Junit, you are using Fault-based testing.

Deb Wright
The three basic approaches to testing are:
  • Structural Testing: Testing requirements are in terms of the code coverage of the program.
  • Fault-based Testing: Testing focuses on finding faults in the software. This is the type of testing JUnit supports.
  • Error-based Testing: Test cases check error-prone portions of the code to ensure programs behave correctly if errors occur.

Elizabeth Zehner
There are three basic approaches to software testing:

1) Structural testing specifies testing requirements in terms of a the structure of the program or its specification.
2) Fault based testing tests the detected faults in the software. Fault based is a measurement of the fault detecting ability of the tests.
3) Error based testing specifies tests which check the program at certain points that are known to be faulty or error prone.
JUnit uses the fault based approach because it has the ability to pick up faults in the system and you must write tests that are capable of doing this.
Noosh Moussavi
  • Select one test data adequacy criterion and describe it. Please select an adequacy criterion that has not already been selected by someone else. If possible, please state how your test adequacy criterion is related to other criteria.
Error-Based Adequacy Criteria
Error based testing methods require test cases to check programs on certain error-prone points. it does this by partitioning the input-output behavior space into subdomains so that the behavior of the software on each subdomain is equivalent.more than one test case is often used for each subdomain. the space is partitioned in two ways, using Specification-Based input space partitioning and Program based input space partioning.
Specification-Based input space partitioning is when a subset of data is considered as a subdomain if the specification requires the same function on the data. there is no general mechanically applicable method to derive partitions from specifications, even if there is a formal functional specification, however systematic approaches to analysing formal functional specifications and deriving partitions have been proposed.
Program based input space partioning: in this case, two input data belong to the same subdomain if they cause the same "computation" of the program. the subdomains correspond to the paths of the program, and when the paths comtain loops, a particular subset tof the paths is selected according to some criterion.
the following are two types of domain testing:
Boundary analysis: this is to test if the borders if a subdomain are correct.
Functional analysis: emphasises the correctness of computation on each subdomain.
Deb Wright
Fault-based Adequacy Criteria
These measure the ability of a test set to detect faults. Methods include error seeding, program mutation testing, and perterbation testing.
Error seeding introduces artificial faults into the program randomly. Then the test program detects these faults. It is based on the assumption that artificial faults are as difficult to detect as natural ones. This assumption generally proves to be false.
Program mutation testing systematically generates many mutants through mutation operators. The success of this criteria depends on the mutation operators. This is fairly successful, but compuationally expensive.
Perterbation testing tries to find differences between the current program and the hypothetical perfect program. The adequacy of a test set is defined by its ability to limit the error speace defined in terms of a set of functions.

Elizabeth Zehner

There are two main subgroups of program based structural test adequacy criteria: control-flow criteria and data-flow criteria. These two have been combined so that to create and extend the dependence coverage criteria. The criteria for both of these groups are based on the flow graph model of the program. Flow graph models, in most cases especially with procedural programming languages, can be generated automatically. It is created when the program code is decomposed into a set of disjoint blocks of statements.

Since statements correspond to nodes, control-flow criteria can be broken down in terms of flow graphs. There exists the following types of flow graphs: statement coverage criterion, branch coverage criterion, path coverage criterion, cyclomatic-number criterion, and multiple condition coverage.

Data-flow criteria takes into the account data flow information which is then taken into account in the definition of testing requirements. Data flow testing methods are based on the investigation of the ways in which values are associated with variables and how these associations can effect the execution of the program. There are 4 types of criterion under data flow criteria: all definitions criterion, all uses criterion, ordered context coverage criterion, and context coverage criterion.
Noosh Moussavi

Link to this Page