Gregory M. Kapfhammer

Associate Professor of Computer Science

http://www.cs.allegheny.edu/~gkapfham/

  
View  Edit  Uploads  History  Top  Changes  Search

An Experimental Determination of Sufficient Mutant Operators

  • Briefly describe the mutation testing process. Why is (or, why isn't) mutation testing an appropriate technique for measuring the quality of a test suite?

  • What is an equivalent mutant? How can an equivalent mutant affect the mutation score for a given program module?

  • Why is mutation testing costly? Please postulate about some techniques that could be used to reduce the cost of mutation testing.

  • What is selective mutation?

  • Describe the experiment(s) that you would propose if you wanted to determine which object-oriented mutation operators in Juggernaut were the most efficient.

Chris Howell
1. Mutation testing is the process of writing tests that should ultimately fail. The test will either result in the correct output or incorrect output. If it is the incorrect output, that means the testis a mutant. The faults introduced into the program are ones that are single changes to a program. The faulty programs are the mutant ants and if the test fails, the mutant is killed.
Mutation testing is an inappropriate way of measuring the quality of a test suite since there are many of qualities of a test suite and not just if the program doesn't have any mutants. Mutation testing is also very expensive since for large systems, there are many many tests that need to be done.

2. An equivalent mutant is one that always produces the same output as the original program, so no test case can kill it. The mutation score for a set of test data is the percentage of non-equivalent mutants killed by that data. If the mutants are equivalent, then the mutation score of would less tests calculated into the equation.

3. Mutation testing is costly since if all of the programs in a system have to be tested, and only one part of the program is changed each time, then the cost of running the tests will be costly. Even small programs, could be very costly because of the number of times it needs to be run. If the programs were to be changed somehow, so that not every line of code with a certain mutation is done. Or if there was a way to distribute the mutations accross a network so that multiple operations can be done at once.

4. Selective mutation is reducing the number of mutants a program created by getting rid of the operator mutations which create the most mutants. Being more selective on the mutations being created.

5. I would propose we run through the tests on test programs and see which took the longest and which were correctly implemented and worked the best. Then I would see which one kiss the mutatns the best and which ones didn't. I would compare the results and see which one actually did the best.

Brandon Taylor
1.
The mutation testing process is where a test suite is tested to see if its test adequately detects failure. This is done by making many alterations of a program, to be processed through the tests. The idea is you create code that should fail and hope that the programs tests detect this failure. This way of testing your tests can be very powerful, because it shows faults in your tests that you may not have realized exist. Though it is a good measure of weather tests are good or not, mutation testing bears a large computational cost.

2.
An equivalent mutant is a mutant that is created that is in essence the same as the original code. For example if you had y= x 1 an equivalent mutant could be y= –X -1. This can effect the mutation score, because equivalent mutants can not be killed, and thus show up as still being alive. Even though these mutants should not be killed, when the score is calculated these show up as alive and thus decrease the mutant kill rate.

3.
Mutation testing is so costly because a new source program must be tested for each mutant test, and then run against the test suite. Some possible ideas on how to reduce the computational expense of mutants is possibly only mutate objects that are returned rather than every line of code. By doing this you still are testing what you really care about, the final result), and reduce the number of mutants dramatically. Another option is only test a given segment of code (i.e. Function) until failure, then move to the next segment, since a failure is already known.

4.
Selective mutation is where the number of mutants created is reduced. This is done by getting a way from operator mutations, and focusing on areas that programmers commonly screw up.

5.
The testing method I would propose would be to create several different programs that were designed to generate one specific type of mutant. I would then run all of these and get the time of execution. Along with the time, I would get the kill score. You then can make a kill/time relationship, and compare the effectiveness of each method.
Joe Zumpella
1.
Mutation testing is a technique that we use to test software. We right these test cases in a way that they should fail. We find test cases that will either result in correct or incorrect output, where the correct output demonstrates quality or causing the test case to fail revealing a fault. These cases that fail are the mutants of the original and a mutant gets killed when one of these test cases causes it to fail. Mutation testing is very expensive due to the mass amounts of test cases needed. This leads me to lean more towards the inapproiate side. Some repetitive test cases cause this large cost that is often unneeded.

2.
Equivalent mutants will always produce the same output as the original program. This means that no test cases will ever kill them.
These mutants will never be killed and decrease the mutant kill rate. So since the mutation score is the percentage of nonequivalent mutants killed by that data, this would cause the rate to go down.

3.
Mutation testing is costly because the number of mutants generated can be very large. It was found to be closely related to the product of the number of data references times the number of data objects, which can be a very large number for even small systems. Testing every line of code can be redundant, if this problem could be resolved it would decrease the cost. Once a particular block of code is tested and fails then it should not be tested again.

4.
Selective mutation is simply reducing the number of mutants. This is done by eliminating operator mutations. They are the mutations that generally create the most mutants.

5.
I think that we would need to create a program to go through juggernaut and come up with kill scores to measure each mutation operator. I think this would be pretty simple and keep the system from getting more complicated. This is a quick and effective way to determine which operators are working at all and which operators are working effectively.

Link to this Page