Gregory M. Kapfhammer

Associate Professor of Computer Science

http://www.cs.allegheny.edu/~gkapfham/

  
View  Edit  Uploads  History  Top  Changes  Search

GUI Testing: Pitfalls and Process

  • Briefly review the status of GUI testing. Why is it difficult to test GUIs? How is testing a GUI different than testing a program?
Brenda Gruber
GUI's are more frequently used in software applications than ever; therefore adequate testing of GUI's has become of more and more importance. Currently GUI testing techniques are incomplete. Usually record-playback techniques are used (a tester interacts with a GUI and records the events, storing the session and replaying it with different inputs). This is difficult because it relies on the tester to perform a variety of interactions with the GUI. It is impossible to guarantee that every possible interaction is testeed.

Another common technique is to release a beta version to limited useres and to get their feedback. This is also limited because, again it is not guaranteed that every possible interaction was attempted, so there may still remain some bugs.

Testing GUI's is different from testing regular software because typical software testing uses test coverage criteria as a guideline, but this does not work with GUI's because the GUI software is different from the underlying application code and mapping GUI events to the application code is not simple or straightforward.

The current verification tool to verify that a GUI executes correctly is a test oracle (a program that generates expected results for a test case and compares them with the actual results). In traditional software testing the oracle is invoked after the test case executes and then compare the final output with the oracle output.

In GUI testing the oracle is invoked during the test execution because the GUI state can end up with an unexpected screen, which may make further testing impossible or useless until the current problem is solved. Also, if the oracle does not verify the GUI after each execution step it might be impossible to find where an error actually occured. By using the oracle during test execution it complicates GUI testing.
GUI testing currently relies on record - playback techniques. With this method, one performs sample interactions with the GUI giving it several different types of mouse and keyboard events. The tool records those events, usually as a script, and the tester later plays back the recorded sessions to recreate the events with different inputs. GUI testers often find it impractical to generate all possible test cases for all parts of the GUI. The selection of a subset of test cases usually occurs during test case generation. It is difficult to anticipate test cases fault detection capabilities making it difficult to select the most effective subsets.
A traditional verification tool is a test oracle. A test oracle is a separate program that generates expected results from a test case and the compares them to the actual results. GUI's test cases are different from the traditional test oracle approach. The GUI testing requires that the execution of the oracle take place with test case execution, because an incorrect GUI state will lead to a wrong screen and thus all future tests will fail. Thus unless the oracle verifies the GUI state after each step in the testing process, finding out exactly which error occurred first to cause the cascade of resulting errors is impossible.
The importance of GUI testing is emphasized by the fact that GUIs account for as much as sixty percent of an applications total code. Until an effective way to test GUIs is found, the majority of code in much of today's software will go inadequately tested. The property of GUI testing that sperates it from most software testing is that it is difficult to specify a criterion for determining when the GUI has been adequately tested. Additionally, the fact that tests must be run in isolation as errors will cause a cascade of additional errors makes it difficult to run a series of tests on the code as one would in traditional software testing.
Matt Rummel

Although GUIs make software easy to use for a user, it is not the same for a programmer. GUIs can be 60% of the application’s code. Currently GUI testing techniques, as the author states are "incomplete, ad hoc, and largely manual". The tests are labor intensive because it is the responsibility of the tester to come up with interesting test for GUI interaction. One alternative that is used is to send out the application with the GUI to a small group and let them give feedback on what’s wrong and bugs with the GUI.

Testing GUIs is different than testing a program because when you test a program, you have guidelines to follow such as code coverage. You can not use code coverage with testing GUIs because there can be problematic interactions between the GUI’s user events and the application.

John Pacino
  • Memon reccomends that a tester follow certain steps when they are testing a GUI. What are these steps? How are these steps related to those that would be taken during the testing of more convential software?
Brenda Gruber
Memon suggests that a tester follow the same testing steps for conventional software, but require uniques testing techniques. These steps are:
1. Determine what to test by defining coverage criteria.
2. Generate test case inputs from software specifications and structure.
3. Generate expected output to compare with actual output.
4. Execute test cases and verify output.
5. Determine whether the GUI was adequately tested.

The overall process that Memon recommends for testing follows that of most traditional testing techniques. The first step is to determine a definition for coverage criterion so that you can determine when testing is completed. Next, you generate the test case inputs from software specifications and structure. For GUIs, these would be things like mouse clicks, menu selection, etc. Then, you generate expected output and compare it to the actual output from your program. This will usually consists of screen shots or window positions. The next step is to execute your test cases and verify that the output does indeed match the expected output generated from your oracle. Finally, you must decide if you have adequately tested your GUI to determine which parts were actually implemented. This analysis usually checks events and resulting GUI states.

Although Memon states the process it similar to traditional testing techniques there are some differences. An adequate coverage criterion for GUIs seems harder to determine than traditional testing. Therefore, although the techniques for testing GUIs are similar to traditional testing methods, the implementation process is quite different.
Matt Rummel

Tester Steps:
1) Determine what to test by coverage criteria.
2) Generate test case inputs from software specifications and structure.
3) Generate expected output to compare with actual output.
4) Execute test cases and verify output.
5) Determine whether the GUI was adequately tested.

These steps can be related that they go through the same ideas of testing software but I think the author tries to go out of his way to say that GUI testing in much harder and requires unique testing techniques.

John Pacino

Link to this Page