More on testing: test coverage
June 1, 2010
As my Twitter followers will no doubt know by now, I joined Mozilla two weeks ago. I’m working on the Firefox Sync (née Weave Sync) project which allows Firefox users to synchronize personal browsing data securely with the cloud and thus with other devices.
When I joined, the team was working hard on getting the 1.3 release out, which meant I wasn’t really able to help out. So I spent most of my time so far improving the tests (see bugs 566575, 557590, 557596, 557588 if you’re interested). This not only benefits the project in the long run of course, it’s also an excellent way of getting to know the code.
It turns out, the debugger service actually has what’s called an interrupt hook that is called for every expression the interpreter encounters. Normally you would use this in stepping mode to step through individual expressions, but it may just as well be used to do code coverage. Mind you, it’s not fast. I believe invoking the debugger already turns of the JIT, and with the interrupt hook in place, tests run about an order of magnitude slower. Ugh.
But it worked well enough for a proof of concept. So I hacked together a small patch for the test harness to dump the coverage data it collects as a JSON file. This is then picked up by a small analytics tool written in Python which generates a simple HTML report like this one. When I started writing tests we were at 49% line-by-line coverage, now we’re at 58%. Of course these numbers have to be taken cum grano salis since line-by-line coverage is just one metric. It doesn’t say anything about the quality of your tests. For all you know, some piece of code may be calling a whole bunch of code, thus making it appear covered, but you haven’t actually tested the relevant behaviour.