Mini-review of Java Bug Finders

Here is a mini-review of four Open Source program checkers for Java, based on using them recently to check a large, multi-threaded desktop application with hundreds of classes.

Every six months, we audit our code base using various Java code-checking tools; I go through the results (which contain a lot of dross) and forward the interesting ones to the programmers for fixes or justifications. I find this has four benefits:

• It finds bugs and risky code;
• It gives "objective" feedback to programmers to help them recognize where they have been excellent or lazy;
• It gives me, as team leader, an opportunity to study the code, design and team from a different perspective; and
• By slicing off whole classes of faults, we can concentrate more on design shortcomings.

We already develop from inside Eclipse and some of our programmers routinely use extra code-checking plugins, and much of the code base has been through this audit three or four times before, though continual improvements outdate confidence. So I was interested to see which code-checking tools seemed to provide most value for a project at this stage.

Bottom Line

It seems that JLint and FindBugs cover pretty orthogonal areas: I would recommend them as the basic bug finding utilities; they found about a dozen good bugs. PMD and CheckStyle are mostly concerned with infelicities of design or style, but they had a really low hit rate for detecting bugs: I put them in the prevention is better than cure school of tools. Maybe it is the luck of the draw, or maybe we had already fixed those bugs in a previous run, or maybe they were not the kinds of bugs our guys make; I cannot say.

JLint

What a beauty. JLint has two command-line tools:

• antic finds syntax problems and to so extent is redundant when using an IDE like Eclipse. Best tests: detect = instead of == in a test; detect case statements without breaks.
• JLint checks through class files rather than source files. Best tests: detect potential null references, detect certain deadlocks.

I use Artho's version of JLint, which is a slight update on Russian Konstantin Knizhnik's original version.

FindBugs

FindBugs is available on the command line and as an Eclipse plugin. Best tests: detect exception paths where streams are not properly closed, detect certain synchronization problems.

"http://www.cs.umd.edu/~pugh/java/bugs/"
>Findbugs is explained in a paper with a great
title:
"http://www.cs.umd.edu/~pugh/java/bugs/docs/findbugsPaper.pdf"
>Finding Bugs is Easy (PDF).

PMD

PMD is an Eclipse plug-in which is notable because you can write your own rules using XPath expressions over an XML version of the parse tree: shades of Schematron! This might be particularly useful to enforce in-house policy such as "names of private fields must start with 'my' and no other name may start with 'my'", or to detect that some JRE-, platform- or library-specific workaround has been used. Best tests: detect empty catch blocks, detect unused fields, imports etc.

Another aspect of PMD that may be useful on some projects is its ability to detect near-duplicate code: where a programmer has cut-and-pasted code from one place into another. I suppose there would be three reasons for wanting to know this: to check whether refactoring into a method or class would be better, to check whether a fix in one place should also be applied in other places, and to check whether code has been copied contrary to licensing.

Enabling all rules sets in PMD makes it seem like a nanny on the verge of a nervous breakdown: the warning levels are all too high (compared to the default levels that other plugins provide) and there are too many complaints about trivial matters. But that is just a matter of configuration. Out-of-the-box, PMD is probably better for continual use when coding new classes to promote good practises.

CheckStyle

Oliver Burn's CheckStyle is an Eclipse plugin that covers pretty similar ground. Best tests: String comparison using equality, check that an overriding finalize() method includes a call to super.finalize(), check classes that override equals() also override hashcode().

Checkstyle makes the normal mistake when calculating cyclomatic complexity that it includes top-level case statements even if they don't use fallthrough: that is structure not complexity! There is a wonderful Indian product JStyle, which reports metrics, bugs and style issues, does a much better job with complexity metrics than any of these tools. In previous audits we looked at complexity more; for this audit I just picked the top ten most complex methods for refactoring.

Any tips on good bug-finding tools?