More on testing: test coverage

June 1, 2010

As my Twitter followers will no doubt know by now, I joined Mozilla two weeks ago. I’m working on the Firefox Sync (née Weave Sync) project which allows Firefox users to synchronize personal browsing data securely with the cloud and thus with other devices.

When I joined, the team was working hard on getting the 1.3 release out, which meant I wasn’t really able to help out. So I spent most of my time so far improving the tests (see bugs 566575, 557590, 557596, 557588 if you’re interested). This not only benefits the project in the long run of course, it’s also an excellent way of getting to know the code.

As I’ve written in my last blog post, the lack of a decent code coverage tool for JavaScript is bothering me a bit when writing unit tests in JS and this time was no different. Fortunately, the xpcshell test harness that we’re using runs with much different privileges than what your average webapp’s JavaScript runs with. That means we get access to things like the JavaScript debugger service. That, after all, has features like breakpoint and stepping, so surely one should be able to (ab)use it somehow to record code coverage, right?

It turns out, the debugger service actually has what’s called an interrupt hook that is called for every expression the interpreter encounters. Normally you would use this in stepping mode to step through individual expressions, but it may just as well be used to do code coverage. Mind you, it’s not fast. I believe invoking the debugger already turns of the JIT, and with the interrupt hook in place, tests run about an order of magnitude slower. Ugh.

But it worked well enough for a proof of concept. So I hacked together a small patch for the test harness to dump the coverage data it collects as a JSON file. This is then picked up by a small analytics tool written in Python which generates a simple HTML report like this one. When I started writing tests we were at 49% line-by-line coverage, now we’re at 58%. Of course these numbers have to be taken cum grano salis since line-by-line coverage is just one metric. It doesn’t say anything about the quality of your tests. For all you know, some piece of code may be calling a whole bunch of code, thus making it appear covered, but you haven’t actually tested the relevant behaviour.

Anyway, have a look at the bug in the issue tracker if you’re interested in the patch and the report generator. Feedback is, of course, most welcome. It would also be fun to apply this or a similar technique to Firebug (it looks like somebody has tried in the past) so you could get code coverage on your webapp’s JavaScript without having to resort to crude code rewriting tools.


2 Responses to “More on testing: test coverage”

  1. Interesting! I’m not sure how JSD interacts with the JIT these days, that’d be interesting to find out, since bugs/differences between the JIT and interpreter could change your test results. Regardless, I think it’d be great to have this functionality built-in to our test harnesses so you could produce a JS coverage report from any of our test suites.

  2. […] I mentioned in a previous blog post, I joined the Services team at Mozilla, which is mainly responsible for the Firefox Sync […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: