Gregory M. Kapfhammer

Associate Professor of Computer Science

http://www.cs.allegheny.edu/~gkapfham/

  
View  Edit  Uploads  History  Top  Changes  Search

How to Misuse Code Coverage

  • Marick states that programmer's can introduce a fault of omission into their programs. What is this type of fault? How is it related to the concept of "misusing code coverage"?

Faults of Omission - occur when the programmer has left out a part of that code that is needed to make the program work correctly. If this fault is present in a program then the programmer is useing fault coverage incorrectly if he sees that all of his tests pass. He thinks that the program is more correct than is actually is. There is also a problem when the programmer just writes a test that will cover the missed code with out thinking about where the actual error is coming from.
Jim Clause
Marick states that one cannot necessarily say that they are done with testing their software if all the code that exists in their program has been exercised during testing. This type of testing can only tell one how the code that exits performs, it cannot specify how the code that should exist performs. This is what Marick means by faults of omission: errors that are fixed by adding additional code. Thus, by one assuming that their code is correct because it passes code coverage tests, they are misusing code coverage by expecting it to be a "tell all", informing them of all the potential mistakes in their code. Marick warns that if one's code is weak in a way that coverage tests can detect, it is also likely weak in a way that the tests cannot detect.
Matt Rummel
The author quotes Glass that faults of omission are "code not complex enough for the problem." That is, faults of omission are only fixed by adding more code. It is related to missing code coverage because a coverage tool will probably say that something near the area with the fault of omission was not covered. Code coverage tools are not meant to give quick fix commands. Instead they point out uncovered code as a hint that there is probably a deeper problem in that specific area. Misusing code coverage is when the developer uses the code coverage flag as a quick fix command and doesn't look for underlying problems.
Elizabeth Zehner

Fault of Omission is as Glass states it, "code not complex enough for the problem". These bugs can be usually fixed only by adding new code. It is the programmers fault not to see the that the bug is related deeper than what code coverage gives you. They expect code coverage will tell them the mistakes they have but will only tell them things in their code that they have not tested. It is related to the misuse of code coverage on the same point that programmers expect code coverage to tell them what is wrong with their code and how to fix it, but it really only tells them what they didn’t test for.
John Pacino
  • What is the "most common coverage mistake"? How can we ensure that our development team for Juggernaut does not commit this mistake?

The most common coverage mistake is writing test cases to satisfy the coverage tool instead of fixing the bugs in the code. I think that we can try and avoid this error by not running the test suite until all the tests have been generated. This would prevent people from running a few tests seeing what's not covered and then writing test cases to cover those uncovered areas.
Jim Clause
The most common coverage mistake is to resolve the errors that the coverage tests give by editing the tests so that they conform to making the program pass. We can avoid such a situation in our Juggernaut project by having a person who was not as deeply involved in the actual coding be involved in the testing. It is my experience that when someone has been working on program for a while they are more prone to create a "quick and dirty test" that satisfies the testing tool than someone who is testing someone else's code.
Matt Rummel
The most common coverage mistake is to write tests that cover the code more fully without looking for faults of omission in the code. The Juggernaut development team has a few options to make sure this doesn't happen. I think requiring a (Swiki) lab notebook for each team where they detail each fault of omission that occured and how they dealt with it would be the best solution because it requires that each team is accountable to the group. Making the team stop staring at the code and sit down and write about the problem would probably make it easier for the team to identify faults of omission.
Elizabeth Zehner

The most common coverage mistake I believe is to make tests that do not fully test your code. That is to take the easy way out and test only code that you know will pass. We can ensure this by not looking over fault of omission, looking at the test and seeing if the bug is deeper than what you have and if it can be satisfied by adding some new code.
John Pacino
  • What steps can be taken to ensure that code coverage is used in an appropriate fashion?

I think the main thing we need to do it understand that having 100% code coverage does not ensure that the code is correct or even close to correct. Code coverage errors should not be fixed by writing bew tests only they must also be fixed by thinging about why the occured and if it is possible to fix them by adding new code to the program
Jim Clause

One way to ensure that we are using code coverage correctly is to not set a certain level of coverage as a "shipping gate" for the product. Marick notes that when such a criterion is used to denote the completion of a project, those working on the project have a tendency to optimize their performance to meet that target of coverage. Another way that we can ensure that we are using coverage appropriately is by not relying on code coverage to tell us all the mistakes in our programs. Certainly code coverage can be a helpful tool in testing our software, but should not be the only type of testing utilized: a complete test of a software system should include a variety of testing techniques. Thus, when we are applying code coverage techniques to test our code, we must keep in mind that the point of testing is not to be an added hindrance to software development but a tool to point out potential weak points. We should not try to write tests to satisfy requirements that have been set for our work, but to help in developing the best possible product.

Matt Rummel
Stress that Marick's point that "If a part of your test suite is weak in a way that coverage can detect, it's likely also weak in a way coverage can't detect." Encourage responsibility for code coverage fixes by making them public and open to inspection. Students or developers who do not make quick fixes and find faults of omission should be rewarded approprately for the difficult task through good grades or bonuses.
Elizabeth Zehner

Marick states in this article that "If a part of your test suite is weak in a way that coverage can detect, it’s likely also weak in a way coverage can’t detect". Then he goes on and talks about 4 steps that can be taken:
1) What feature in the interface does that condition corresponds to.
2) Rethink how that feature should have been tested and do not concentrate on satisfying the coverage condition. Mainly think about the test.
3) Run new test and check coverage.
4) Repeat for other features.
John Pacino

Link to this Page