Code coverage is an important aspect of software testing, and can be considered fundamental to the overall system testing of a component. The motivation behind coverage tooling is simply to give you (as a developer or tester) more insight into the areas of code that are being exercised by a set of test cases. This information is useful because you can then use it to devise new test cases to achieve adequate coverage.
The IBM® Rational® Code Coverage feature is a tool that integrates with IBM® Rational® Application Developer. You can use it to generate and analyze coverage statistics for your Java applications. The tooling generates statement coverage statistics for the application under test (that is, the number or percentage of lines in your application that have been executed).
Instrumentation
In order to properly analyze the coverage statistics in the Rational Code Coverage feature, it is important to understand the technology used behind the scenes.
Rational Code Coverage uses an instrumentation engine to manipulate the bytecode of a class and inject custom calls to the coverage data collection engine. Figure 1 provides a high-level overview of the process:
Figure 1. Overview of the Rational Code Coverage execution environment
Basic blocks versus executable units
The instrumentation engine operates on units of bytecode called executable units. The definition of executable unit is slightly different than the traditional definition of basic block, but the differences are important to take into account when the results are analyzed.
By definition, a basic block is a set of instructions that cannot be branched into or out of. The key idea here is that when the first instruction runs, all of the subsequent instructions in that block are guaranteed to be executed without interruption. It follows that a basic block can be conceptually considered a single group or block of instructions. In general, basic blocks end on branch, call, throw or return statements.
An executable unit begins at the start of every basic block and at any instruction that corresponds to a line of source code that is different than the previous instruction. What differentiates an executable unit from a basic block is the condition that triggers the end of the executable unit. For example, the divide instruction is not considered to be the end of an executable unit despite the fact that it can throw an exception.
The instrumentation engine in Rational Code Coverage is used to inject custom code at the start of every executable unit. Consequently, you can customize the Rational Code Coverage feature to report statistics down to the executable unit level of granularity (in other words, block coverage). Figure 2 provides an overview of how the instrumentation engine modifies the bytecode to support code coverage.
Figure 2. Overview of the bytecode instrumentation
Generating coverage statistics in RAD
One of the major advantages of the Rational Code Coverage feature is that you can enable it on any Java project in RAD by navigating to the Code Coverage panel in the project Properties, as shown in Figure 3.
Figure 3. Code Coverage panel in the project Properties
Select the Enable code coverage check box in Figure 3 to enable code coverage for the project, and to instrument the classes in your project under the covers. You can also use this panel to customize acceptable coverage levels for each step of granularity. The supported levels of granularity are described following:
· Type coverage: the percentage of types covered in a class
· Method coverage: the percentage of methods covered in a class
· Line coverage: the percentage of lines covered in the class file
· Block coverage: the percentage of blocks covered in a class file. Note that a block refers to an executable unit (as described previously)
You can also specify custom filters, and they are used to control what gets instrumented in your project. By default, all of the classes in your project are instrumented, but you can create custom filters to exclude target packages or specific types, if there is a need to restrict the results.
Package Explorer
After you enable code coverage on a project, coverage statistics will be generated the next time that the application is launched. Note that statistics will not be generated for all types of launch configurations automatically. Table 1 displays the launch types that are supported from within RAD.
|
Launch Type
|
|
Java Applet
|
|
OSGi Framework
|
|
JUnit
|
|
JUnit Plug-in Test
|
|
Java Application
|
|
Eclipse Application
|
|
Standard Widget Toolkit (SWT) Application
|
Table 1. Supported launch configurations
A sample application has been provided (see link at the bottom of this article) and will be used throughout this article. The application is a simple representation of different vehicles (car, van, motorcycle, and so on) and the various parts associated with each vehicle. The UML diagram outlining the structure of this application is displayed in Figure 4.
Figure 4. UML diagram for the sample application
There are two JUnit tests already defined in the project: TestCar.java and TestCarImproved.java. As the names suggest, these tests target the Car.java class. While in the Java perspective in RAD, you can start the TestCar.java test by right-clicking TestCar.java and selecting Run As > JUnit test. The results of the JUnit test will appear in the JUnit view as normal. The coverage results are integrated into the RAD UI, and you can analyze them by switching back to the Package Explorer. Figure 5 displays a sample result set for the TestCar.java test.
Figure 5. Coverage results for TestCar.java displayed in the Package Explorer
By default, the UI is annotated with only the line coverage information; however, you can change this in the workbench Preferences, and optionally choose to include coverage for packages, types, and blocks. The percentage beside each Java item is a breakdown of the line coverage for the last execution. You can drill down into the various Java artifacts (for example, classes, types, and methods) in the Package Explorer to get coverage statistics at a lower level of granularity.
The results are color-coded depending on the success rate: by default, red indicates that the acceptable coverage level has not been met while green indicates that the appropriate coverage level was achieved. Naturally, the goal of the test is to reach an acceptable coverage level on the classes of interest.
Based on the results shown in Figure 5, the first test was inadequate: the Car class (and abstract parents AbstractFourWheelVehicle and Vehicle) did not reach appropriate coverage level. Luckily, you have a second attempt to execute: TestCarImproved.java. Again, you can execute the test as a normal JUnit and the results are automatically updated in the Package Explorer (Figure 6).
Figure 6. Code coverage results for TestCarImproved.java displayed in the Package Explorer
Java editor
Line coverage results are also displayed and marked in the Java editor, and you can use it to give a more precise indication of which lines are covered in each class. After coverage statistics have been generated, you can open any class in your project with the Java editor, and the left ruler bar in the editor shows the coverage information. Figure 7 displays the results for Vehicle.java:
Figure 7. Coverage results displayed in the Java editor
The color indicators are the same as they are in the Package Explorer. That is, by default a green line was covered and a red line was not covered. There is a slight advantage in viewing the results in the Java Editor because it also indicates the partially covered lines. Partially covered lines can occur when there is more than one executable unit on a line of source code, but only one of them has been executed. As an example, look at the first line of code in the setTargetSpeed(int speed) method shown in Figure 7: the first executable unit is the if statement, and the second executable unit is the return statement. By default, a partial line is colored in yellow.
Generating reports
You can compile the code coverage results into reports and view them in RAD, or save them to the file system for future analysis. You can generate two different types of reports: Workbench reports (Eclipse-based) and HTML reports. To generate a report, select Run > Code Coverage > Generate Report. Figure 8 shows the report generation dialog.
Figure 8. Report generation dialog
You can create and view a report in RAD using the Quick View option on the dialog, or save it to the file system using the Save Report option.
Workbench reports
The workbench reports (also known as Eclipse-based reports) provide a consolidated view of all of the coverage statistics for your project, and contain coverage information for all of the classes in your project at execution time. Figure 9 shows a populated Eclipse-based report.
Figure 9. Coverage results in an Eclipse-based report
Workbench reports have the added advantage of being integrated in RAD, so you can use them as a quick tool to provide insight into the parts of your code that require improved test coverage. As Figure 9 shows, the statistics in a workbench report contain coverage information for all levels of granularity: from a package to a method. Right-clicking any of the Java artifacts displays a pop-up menu with two additional actions: Show in Package Explorer and Open in Java Editor. These are useful tools for identifying and investigating areas of code with low coverage, because they highlight the selected area of code by opening it in the appropriate viewer or editor.
HTML reports
HTML reports display the same type of information provided in the Eclipse-based report, but in HTML format. These reports are particularly useful when saved to the file system, because they provide a way for the coverage results to be analyzed independently of RAD, shared with team mates, or published to a Web site for viewing.
Generating statistics outside of the workbench
One of the major features of the Rational Code Coverage tool is its ability to generate statistics outside of RAD. This provides extra flexibility and enables you to customize your environment to take advantage of the Rational Code Coverage feature in your systems. For example, one natural combination would be to set up a nightly build environment and generate statistics with JUnit tests on the nightly driver.
You can integrate the Rational Code Coverage feature into your environment by performing the following three steps: instrumentation, execution, and report generation.
Step 1. Instrumentation
There are two different approaches that you can use to instrument your application. The first is to use the instrument.bat/sh script provided in the <RAD_HOME>/plugins/com.ibm.rational.llc.engine_<date>/scripts directory. This article does not focus on this script, but you can reference the RAD documentation for more information if necessary. The second approach is to use the instrumentation Ant task provided by the Rational Code Coverage feature. Listing 1 shows an example usage of the instrument task configured to target the sample application in this article.
<target name="instrument">
<taskdef name="instrument"
classname="com.ibm.rational.llc.engine.instrumentation.
anttask.InstrumentationTask"
classpath="{path to com.ibm.rational.llc.engine plugin}"/>
<instrument saveBackups="true"
baseLineFile="project.baseline"
buildPath="VehicleProject"
outputDir="VehicleProjectInstr"/>
</target>
Listing 1. Example usage of instrument Ant tasks on the sample application in this article
A quick overview of the expected parameters is outlined in Table 2, following.
|
Parameter
|
Description
|
|
buildPath
|
The path to the project on the file system
|
|
outputDir
|
(Optional) The output directory of the instrumented project. If not specified, the classes in the buildPath will be instrumented in place.
|
|
baseLineFile
|
(Optional) The output location of the baseline project index file. See the following paragraph for more information on this file.
|
|
saveBackups
|
(Optional) Set to true if the original class files should be backed up before instrumenting.
|
Table 2. Input parameters for instrumentation tasks
Both approaches to instrumentation will output a baseline file. A baseline file is a notion specific to the Rational Code Coverage feature. The baseline file contains an index of all of the classes in your project, and maintains additional metadata about each class. This file is used at the reporting step (Step 3 following) to determine which classes in your application were not covered. This step is necessary because the Rational Code Coverage data collection engine is only notified of a class when it is loaded by Java™ Virtual Machine (JVM), and so a list of the classes that were not executed cannot be determined without additional metadata. If the baseline file is not present at reporting time, the classes that were not loaded will be absent from the report.
Step 2. Execution
In order to execute the instrumented classes, the Java environment must be configured correctly at launch. The two specific parameters needed for execution are explained below:
· -Dcoverage.out.file=<absolute path to output file>: the file specified by this JVM argument is the output location of the coverage statistics
· Add the <Rational Application Developer HOME>/plugins/com.ibm.rational.llc.engine_<date>/RLC.jar to the classpath: because the code has been instrumented with callbacks to the the Rational Code Coverage data collection engine, the RLC.jar file needs to be on the classpath at runtime.
These parameters can be supplied to a JUnit Ant task. Listing 2 provides example usage.
<target name="run">
<junit showoutput="true" fork="yes">
<jvmarg value="-Dcoverage.out.file={absolute path to the output file}"/>
<classpath>
<pathelement location="{absolute path to the
<Rational Application Developer HOME>/plugins/com.ibm.rational.llc.engine_<date>
/RLC.jar file}"/>
<pathelement location="{path to the project classes}"/>
<pathelement path="{absolute path to the junit.jar}" />
</classpath>
<test name="com.ibm.vehicles.tests.TestCar" outfile="TestCar" />
</junit>
</target>
Listing 2. Example of how to specify the the Rational Code Coverage feature arguments in an Ant launch
Step 3. Report generation
You can generate reports using another Ant task provided by the Rational Code Coverage feature. This task uses the reporting functionality provided by the BIRT Eclipse.org project, and thus requires that you download the BIRT V2.3.2 Reporting Engine standalone offering. This can be done by navigating to http://www.eclipse.org/birt/download, select the V2.3.2 release and downloading the Report Engine offering. Note that this Ant task can only produce HTML reports.
Listing 3 provides sample usage of the reporting Ant task. Note that, as input, it requires the coveragedata file generated in Step 2 and (optionally) the baseline file generated in Step 1.
<target name="generate-report">
<path id="lib.path">
<pathelement location="{absolute path to the
<Rational Application Developer HOME>/plugins/
com.ibm.rational.llc.common_<date>.jar plugin}"/>
<pathelement location="{absolute path to the
<Rational Application Developer HOME>/plugins/
com.ibm.rational.llc.report_<date> plugin}"/>
<pathelement location="{absolute path to the
<Rational Application Developer HOME>/plugins/
org.eclipse.equinox.common_<date>.jar plugin}"/>
<fileset dir="{absolute path to the BIRT ReportEngine directory}\lib" includes="*.jar"/>
</path>
<taskdef name="code-coverage-report"
classname="com.ibm.rational.llc.report.birt.adapters.ant.ReportGenerationTask"
classpathref="lib.path"/>
<code-coverage-report
outputDir="{absolute path to the report output directory}"
coverageDataFile="{absolute path to the coveragedata file generated in step 1}"
baseLineFiles="{absolute path to the baseline file generated in step 1}"/>
</target>
Listing 3. Example usage of the report generation Ant task on the sample application in this article
An example HTML report is displayed in Figure 10. Generating HTML reports using the Ant task provides a means by which users can view the statistics generated in an Ant environment independently of RAD.
Figure 10. Coverage results in an HTML report
Try it out!