Martin Probst's weblog

Java Unit Test Coverage

Sunday, September 4, 2005, 11:16 — 1 comment Edit

I’ve spent one and a half day last week setting up a Java Unit Test code coverage system. This was somewhat surprising to me, I don’t think something like that should take that long. The major problem was the state of the available tools. I wanted to find if there exist any usable open source tools first, so I avoided Clover, JCover & Co. Instead I tried:

So now we have something that is somewhat working. Somewhat because I ran into (presumably) a bug of ant (1.6.3) where custom junit task result formatters don’t get their extension passed along if the <junit/> task is set to forkmode=‘once’. This currently makes it impossible to view the results of the unit tests if they are run with code coverage enabled, and by that makes it quite difficult to hunt down errors. I still have to check if that bug is fixed in a later version of ANT.

The forkmode=‘once’ also lead to quite a number of errors on our side, as our test machinery relies on static class fields in several places, and those might be set to something wrong after a test. That’s probably an error on our side, but annoying nonetheless. The forkmode=‘once’ is necessary though, as anything else slows down the testing horribly.

In the aftermath coverage testing is quite nice, and the results are not as horrible as I expected. In most packages we have a coverage of over 90%. Most of the untested code is in generated classes. I presume most of it is untestable and not used at all. Code coverage in terms of lines or blocks is of course a very bad criterium for test completeness, path coverage wouldn’t be that much better too, but it can at least give you good pointers to areas that are under- or untested. Another step to better software development ;-)

PS: Also a plus for EMMA is that it’s self contained, only two jars, as opposed to other projects which require 6-8 libraries to be on your classpath. This is generally just a little more work to do when setting up, but wait until tool A requires a different version of a library than tool B. DLL hell for java, but that’s another story …

I would like to share my interest in a great tool like Emma, which I recently discovered, and to contradict you that from my point of view, it is not a drawback that Emma does not have sharper colors, as the report it provides I find to be more complex and helpful.