Of late, one doesn’t have to travel too far back in time to find instances where improper test coverage led to catastrophic failures in the software. The disastrous opening of Terminal 5 at London Heathrow airport, in 2008, was the prime example of engineers failing to account for real-life instances and disregarding proper test coverage. What resulted over the next 10 days was that 42,000 bags failed to travel with their owners, and the airport had to cancel over 500 flights. At Kualitee, we ensure that errors of this magnitude don’t come to light.
In this article, we’ll go in-depth with understanding test coverage, its importance and some of the metrics used to measure it.
What is Test Coverage?
Test coverage can be regarded as a general software testing metric that defines test density. To put it simply, test coverage provides a general idea about the comprehensiveness of the QA process and consequently lets all the stakeholders know whether a product has been properly tested or not. That is, whether all its parameters have been checked thoroughly.
Advantages of Measuring Test Coverage
Having some metrics on your side is always better than having none at all. You can get the following benefits by measuring your test coverage
- Even though test coverages aren’t always 100% accurate, they are decent at showing how good your current test coverage is which will help to gauge how much time you’ll be required to finish with project tests.
- If your test coverage shows any gaps, you will have the opportunity to add new required tests which will enhance the quality of the project.
- Based on the test coverage statistics, you may even notice tests that do the same job and reduce their number, thus ending up saving some additional time and resources.
- Based on set test coverage, it is relatively easier to make decisions on whether exit criteria have been met or not.
- Test coverage also makes it easier to control the scope of work and schedule.
Disadvantages of Measuring Test Coverage
Like most other things, test coverage measurement also has its unique challenges that we can refer to as its main drawbacks:
- You cannot fully automate test case analysis. This, when compared to code coverage, is a drawback as code coverage tools can automatically generate a detailed report when test coverage is not available.
- If most actions are manual, it can take a considerably long time to note the right requirements and test cases and properly study and analyze them.
- Test coverage data is never 100% accurate since there is a high probability of errors and omissions.
Measuring Project Test Coverage
Test coverage by feature – This is perhaps the simplest one. You’ll begin by preparing a list of all features which are available at your project and then verify whether all of them were covered with tests. This method of test coverage isn’t data-rich in terms of how good those features are tested but it will eventually show if you don’t forget to cover any features with appropriate tests. It can be seen as the first step in proper test coverage. You first make sure that all the features are covered and then move on to how well they’re covered.
Test coverage by instrumentation – This method is relatively more complex than the first one. It uses a special tool that will verify what part of code was executed while you perform a certain action in the system. This will show you the percentage of code that was executed during your set of tests. These are usually automation tests. An important point to note is that even if the code was executed, it doesn’t necessarily mean that the expected result was achieved during this action. So just like in the first instance, you don’t really know how well a project was tested with this method.
Test coverage by GUI – Most development projects today is to be displayed on desktop PCs, browsers or mobile devices. This means they would normally have some sort of user interface for the consumer to interact with. This user interface can, in turn, have lots of screens, pop-ups, drop-downs, tooltips, buttons, links, scroll bars and other elements which also should be covered with your tests. This is because these elements are the ones your users will be interacting with the most and you may want to check if they will work exactly as required.
Test coverage by risks – A software can’t be considered completely tested if associated risks and their causes haven’t been tested carefully. Albeit all app functions work properly, it’s important to predict possible app failures associated with a connection failure, a huge number of concurrent users, etc. QA team should check app behaviour in risky situations and its ability to face up to. In a nutshell, the technique implies listing the risks associated with software work and reassuring they’re appropriately covered with tests.
Test coverage by transition – This method counts all number of “ways” or “paths” which the user may attempt to use to get the desired results. It’s similar to going out in a car, train or jet and return home from your job in any of these ways. This means that all those ways should be tested well. It’s really important to hide this for web applications where users can also navigate through URLs or links and not only by buttons and options in drop-down menus.
Even at Kualitee we recognise that it’s almost impossible to get a 100% accurate way to measure your product test coverage. But you can get close to the number by combining different metrics and tools that may help you understand the coverage of your tests better. The methods of test coverage measurements which you should choose entirely depend upon your project functionality and available resources and time. By having such data you’ll be ready to have good test coverage levels of all project aspects during all times of development and support which can significantly improve the standard of your product!
