It’s been more than a month since my last status report. Here’s what happened since then:

Firefox 4.0b8, Firefox Mobile 4.0b3, Sync 1.6

We’re getting ready to ship Firefox 4.0b8 and Firefox Mobile
4.0b3. Along with those beta releases we will ship Sync 1.6 as an add-on for Firefox 3.5/3.6. It will match the version of Sync built-in to the betas so that you can sync between Firefox 3.5/3.6 and the betas releases. Here’s what’s new in these releases and what we’ve been working on these past weeks:

Easy setup

Up until now, setting up Sync on a new computer has been a bit of a chore. With the wizard redesign by Alex Faaborg we reduced the number of necessary steps, but you still have to type your account name, password and Sync Key — particularly annoying on mobile devices.

We looked at how other systems cope with this problem in terms of usability. Bluetooth is a good example where you pair another device with an existing one while it’s often hard or even impossible to type on one of them. Obviously for Sync we need something that works on all platforms where Firefox is supported, ranging from desktop computers to mobile phones. Moreover, it needs to be secure so that Sync’s mission of protecting the user’s privacy isn’t compromised in any way.

The solution was PAKE (password-authenticated key agreement), a cryptographic mechanism for two parties to agree upon a strong key based on a weak shared secret, e.g. a Bluetooth-like PIN. In our case it allows a desktop computer that’s already connected to Sync to securely transfer the credentials to, say, a mobile phone. The mobile phone displays a random PIN that simply has to be entered on the desktop computer. Then both devices will go through the PAKE algorithm (J-PAKE in our case) to agree upon a strong key, communicating through a simple server via HTTPS. Once it’s verified on both sides, the desktop will send the credentials to the mobile phone. This way, you can set up a mobile phone for Sync with zero typing on the phone itself. Of course, this also works for setting up another desktop or laptop computer.

This project included design and implementation work at all levels: designing and implementing the server API and communication protocol with my colleagues, making a prototype implementation of the J-PAKE algorithm, refactoring it to be based on the J-PAKE implementation Brian Smith created for NSS, and implementing the easy setup UI. It was challenging, sometimes outside my comfort zone, but possibly the most fun I’ve had in years.

Simpler crypto

Then and now

Up until now, Sync would protect your personal data as follows:

  • Your Sync Key (a.k.a. secret phrase) is turned into a 256 bit AES key using PBKDF2.
  • That key is used to encrypt/decrypt the private key of an RSA key pair that’s stored on the server.
  • The RSA key pair is used to encrypt/decrypt 256 bit AES bulk keys, one per collection, that are used to encrypt/decrypt your private data.

The original vision was that indirection introduced by the RSA key pair as well as the simple PKI we had on the server would allow for easy sharing mechanisms. Alas these weren’t and aren’t a priority — rightly so IMHO — and the complexity of the setup had some serious impact on client and server performance. It also had some shortcomings security-wise.

We now have a much simpler setup that still guarantees the security and privacy of your personal data (better in some places, in fact).

  • The Sync Key is now always a 128 bit random key, user generated Sync Keys are a thing of the past. This means you will no longer have to come up with a secure secret phrase of some kind, let alone understand what it means for a secret phrase to be cryptographically secure (fact is, most users don’t). When shown in the UI we encode the Sync Key in a modified base32 alphabet, making it 26 characters long. Fortunately you will rarely have to type those, thanks to the new easy setup mechanism.
  • A 256 bit AES key and a 256 bit HMAC key are derived from the Sync Key using an HKDF.
  • This key bundle is used to encrypt/decrypt a special keys record on the server. It contains the 256 bit AES and HMAC bulk keys that are used to encrypt/decrypt your personal data. By default there’s only one pair of AES and HMAC keys.

What gives?

So instead of storing at least half a dozen individual records on the server for the various keys, we only have one server record to fetch at the beginning, and we have plans to even cache that locally in the password manager. With 256 bit keys everywhere this is much more viable now.

On average the simplified crypto setup reduces the number of request per sync by ca. 15%. Our ops guys love us for this. On the client it makes the first sync per browser session much faster because there are fewer requests to the server and no RSA crypto is involved — particularly important on mobile devices. It also makes the first sync ever much faster because no RSA key pair has to be generated. Moreover, we can now optimize our crypto code for the simpler setup much better. Initial investigations have shown that a ~25% speed up and ~50% improvement in memory usage might be possible.

I can’t take all the credit for this. Our resident cryptographer Brian Smith helped us design much of the system and, following the proud Mozilla tradition of giving newbies projects that end up touching every single piece of code in the system, our new colleague Richard Newman implemented nearly all of the code.

Tighter integration with Places

Sync interacts a lot with Places, Firefox’s history and bookmark database. Now that Sync is being integrated into Firefox proper, we can couple them more tightly and gain better performance as a result.

For instance, Places will assign the GUIDs as used by Sync to history records and bookmarks as they are created, so that by the time Sync needs to deal with them, they will already be there. Having Places take over the disk I/O here means it will be not on the main thread and it will be part of operations that Places has to do anyway, so it’s less disk I/O overall.

We’re also considering adding more asynchronous APIs to Places, specifically with Sync’s usage in mind.

Coming up

Simplified crypto and easy setup were the last big changes we had lined up for Firefox 4. We still have a lot of work to do, but it’s mostly polish and paper cuts from here on.

There’s also increasing interest from third party developers in building clients for other browsers. Marcus Wolschon, for instance, has written a Firefox Sync client for the popular Androind browser Dolphin HD. Unfortunately, our API and developer docs are currently scattered across many pages in different locations of the Mozilla wiki. As we get closer to Firefox 4, I would like to spend some time collecting and updating that info and putting it in its own section on MDC/MDN.


Sync in Firefox 4.0 beta 7

November 12, 2010

Update: If you’ve opened this article because you want know how to file a Sync bug, please read: How to file a good Sync bug. Thank you!

Under the hood, the Sync feature shipped in Firefox 4.0 beta 7 is on par with Firefox Sync 1.5. The new setup wizard is also there, now generating accounts solely based on email addresses and the new 20 character Sync Key. There are a two of differences compared to 4.0 beta 6 and the add-on, though:

Toolbar button

As 4.0 beta 7 removed the status bar, we had to find a new place for the Sync button. Our philosophy is to make Sync a background service that you shouldn’t have to interact with. So want to impose as little UI as possible on users, but we do realize that some users like to be able to sync manually as well from time to time. So we’ve made the Sync button an optional toolbar button. It is not in the toolbar by default, but if you right-click on the toolbar and select Customize, you can drag it in any toolbar (the tabs bar, the navigation toolbar or even the add-on toolbar).

Sync toolbar button in the navigation toolbar

Sync toolbar button in the tabs toolbar


Traditionally, Firefox Sync has kept a log of its actions under about:sync-log. This has been very handy for debugging problems when things go wrong. However, most of the time the logging to disk creates I/O overhead that we’d rather not have, especially on mobile devices. Logging to disk is therefore disabled by default now. You can re-enable it, for instance when you’re trying to track down a problem:

  1. Go to about:config
  2. Do right-click, select New -> Boolean
  3. Give it the name services.sync.log.appender.debugLog.enabled (case matters!)
  4. Set it to true
  5. Restart Firefox

(Due to an oversight the services.sync.log.appender.debugLog.enabled preference doesn’t exist by default in beta 7. That will be fixed in the next beta, though.)

Covert BarTab-like feature in Firefox 4.0 beta 7

A few weeks ago Paul O’Shannessy implemented cascading session restore. By doing that he took a big item off my to-do list for BarTab and made add-ons like Load Tabs Progressively redundant. Not only that, the number of concurrently loaded tabs can be tweaked through the browser.sessionstore.max_concurrent_tabs preference in about:config. If set to 0, you get instant BarTab-like behaviour. Great success!

In the meantime I have fixed a few Firefox 4.0 compatibility issues in BarTab. These fixes are in the latest BarTab beta, available from the BarTab beta channel. I’m very busy working on Firefox Sync these days so I can’t promise if I’ll be able to maintain BarTab and if it will ever officially support Firefox 4.0. But thanks to Paul, one of the biggest reasons for having BarTab installed just went away!

Vertical Tabs 0.7

Vertical Tabs has received a few updates as well and is now at version 0.7. Compatibility with Firefox 4.0 beta 7 (now the minimum required version) and recent nightlies has been restored. The styling now works better when Personas (lightweight themes) are installed. Since Firefox removed the status bar, Vertical Tabs no longer rolls a custom status bar on left, either. Instead the tabs toolbar has been moved there, and it’s fully customizable. You can, for instance, put your Sync toolbar button down there! Lastly, the Tabs on Top menu item has been removed since that choice does not make any sense once Vertical Tabs is installed.

Try out Firefox 4.0 beta 7 now!

Status report for October

November 9, 2010

It’s been over a month since my last mega-update, so here’s an update of what I’ve been up to since then:

Shipped: Firefox Sync 1.5

We shipped Firefox Sync 1.5 which is also part of Firefox 4.0b7 and Firefox Mobile 4.0b1. It contains various performance and stability fixes, but the most visible change is the greatly simplified (and prettier) setup wizard.

Sync Key

The “secret phrase” has been renamed to Sync Key, a term that we think will be understood better by less technical people. By default it’s an auto-generated string of 20 alphabetical characters. Users are encouraged to securely back this key up by either saving or printing a document with the key on it, though we’re planning some further additions to the wizard that will hopefully make this unnecessary for most cases (see below.)

Storage version bump

Version 1.5 also included a storage version bump, IIRC the second one so far in Sync’s history. These changes to the storage structure are necessary once in a while usually for optimization reasons (in this case server migration), but the downside is that they introduce a clear compatibility cut. Since both Firefox 4.0b6 and Firefox Mobile 4.0b1 are lacking the necessary UI notifications to tell users to upgrade Sync, the storage bump became a bit of a hassle for users trying out the Firefox 4 beta. Fortunately this has been fixed for both Firefox and Firefox Mobile.

Coming up: Firefox Sync 1.6

Sync 1.6 represents most if not all upcoming improvements that we want to include in Firefox 4.0 and are working on right now. Note that the majority of these will only be available to Firefox 4 users because they depend on changes to the underlying platform.

Encryption off the main thread

This should help UI responsiveness a lot during syncs because CPU intensive encryption and decryption processes no longer run on the main thread. We had an initial version of this working using threads but recent changes in the JägerMonkey JavaScript engine have made threads with shared state impossible. We are now looking at implementing this using ChromeWorkers, a privileged version of HTML5’s web workers.

Less synchronous I/O activity

This is an ongoing effort to improve the app’s responsiveness when Sync is tracking changes to the profile (e.g. new history visits), as well as during syncs, by making a lot of the I/O asynchronous. This includes Sync’s own I/O as well as queries to Places, Firefox’s bookmark and history database, to which we will probably be adding asynchronous APIs.

Easy setup for secondary devices

Setting up Sync on a second computer requires a lot of typing, pretty cumbersome especially on a mobile phone. We’re working on ways to improve that experience by securely transferring your Sync credentials from one device to another. One of the methods we’re exploring uses a PAKE (password-authenticated key agreement) algorithm whereby you enter a relatively small code that’s shown on one device into the other, and then the two devices agree on a stronger key that they’ll use to encrypt and then transfer your credentials with. Another method we’re looking at for Firefox Home users is 2d barcodes.

Crypto simplification

Right now Sync uses a PKI-like storage setup for the keys that the data is encrypted with. It allows for a lot of flexibility, but it also makes things unnecessarily complicated. What’s more, the additional cryptography is expensive and every additional key stored on the server means more network traffic. So reducing the number of keys and crypto steps will be good for both the client and the server. Going forward we’d therefore like to get rid of the PKI layer. We’d also like to always auto-generate the Sync Key, now with higher entropy. This means that users will no longer be able to choose their own phrase for the Sync Key, and that the length will grow from 20 to likely 25 alphabetical characters. Fortunately this will be mostly transparent to users as they won’t have to deal with the Sync Key most of the time anyway, thanks to the easy setup option.

This means another storage bump, so Sync 1.6 will likely not be backwards-compatible with Sync 1.5, but we hope that this will be the last bump for a while, so that it will be forward-compatible with Firefox 4.

Better sync timing

Right now Sync mostly syncs at predefined intervals while it could be more clever about when it syncs. Syncing more reactively after changes to the profile (e.g. user added new bookmarks) would improve the user experience a lot.

In other news

Relocated to the Bay Area

After having worked remotely for the first few months, I relocated to the Bay Area two weeks ago. I’m excited to join many of my colleagues at the Mountain View office. It has already paid off in the short time I’ve been here, particularly since I’m in the same (native) timezone as most of them now.

Podcast with Netzpolitik

While in Berlin I met Markus Beckedahl of Netzpolitik, a German blog on internet culture and politics. He wanted to interview me about my job at Mozilla in a podcast. In the podcast we talked a bit about what Mozilla is and what it’s like working in the Mozilla project. Of course we also talked about Firefox 4, Firefox Sync and various other things. Nothing an avid reader of Planet Mozilla wouldn’t already know, but if you speak German and like to listen to me answer questions (probably not always super well) for about 50 minutes, stay tuned. The podcast will be online soon, I’m told.

It’s 17 September which means I’ve been working for Mozilla for exactly 4 months now. I’ve been so fantastically bad about doing regular status updates — unlike some of my more disciplined colleagues — I thought this would make a good opportunity to do one. But since I’ll have to cover so much time, this’ll be fairly high level.

Firefox Sync

As I mentioned in a previous blog post, I joined the Services team at Mozilla, which is mainly responsible for the Firefox Sync infrastructure. Right now this entails

  • the Firefox Sync add-on and — as of Firefox 4 and Fennec 2 — core functionality which I work on,
  • the Firefox Home iPhone app,
  • the server backend (PHP for now, Python in the future),
  • and the database cluster

Even if my work mostly involves traditional Firefox frontend development, the whole infrastructure provides an fun mix of technology and our team an equally fun mix of skills and personalities.

When I joined the team was getting ready to release Firefox Sync 1.3, so for the first few weeks I mostly worked on improving the test coverage — a great way to get to know the code base. After the 1.3 release the priority was to get Sync integrated into Firefox proper. We reorganized the code base of the add-on code so that we could periodically merge the client library for Sync to mozilla-central where Firefox lives. The UI has been tightly integrated with Firefox’s UI: We completely revamped the setup wizard to simplify the sign up process, there’ll be a menu item in the Firefox menu (as well as the traditional menu bar) and an optional toolbar button for those who like to invoke Sync manually.

Landing Sync in Firefox proper has been a fun ride and with a few more 4.0 betas to come, it’s not over yet.

BarTab and the future of tabs

Ever since I’ve been hired I haven’t had much time to work on BarTab, or even read the bugmail I get for BarTab. Apologies to everyone who reported issues. They’re not forgotten, just on the backburner.

The good news is that at the Mozilla Summit a bunch of us sat down together and discussed tabs. Firefox Panorama, or my much humbler Vertical Tabs, is making it very easy to get lots and lots of tabs opened in your browser. The need for throttling the session restore workload was clearly recognized and is being worked on. It might even (secretly) support BarTab’s behaviour. 😉

Fun with js-ctypes

JavaScript is undoubtedly an important corner stone of the Mozilla platform and I find its abilities amazing, not just within the browser. The js-ctypes library is a platform feature that was borrowed from Python. It allows platform (as opposed to web) JavaScript to load dynamic C  libraries and call their functions. We use it in Firefox Sync to interface with NSS, Mozilla’s crypto library and alternative to OpenSSL, without having to write glue code in C or C++ and to compile it for every platform we want to ship Firefox Sync on.

Much like with the Python equivalent, you first have to declare your types and functions for js-ctypes. This be a bit tedious as you essentially have to convert large parts of a header file to declarations in JavaScript code. You’d think it’d be possible to automate this… And it is!

Enter gcc-xml, an XML backend to gcc. It can parse files and spit out something like an Abstract Syntax Tree in XML format which you can the process. Though you can also cut out the middle man and use pygccxml, a small Python library that gives you pretty good object representation of what gcc-xml spits out. Thanks to that I was able to hack together a script that can walk through the header files of OS X frameworks and generate the corresponding js-ctypes declarations. See my “OSXTypes” project on github for the script, its output and a much shorter rewrite of the original Add to iPhoto example.

I’m sure this script can be refined. Making it work on other platforms with generic libraries, not just OS X frameworks, would be pretty cool… Other examples would also be awesome. Feel free to submit patches or even better, fork!

As my Twitter followers will no doubt know by now, I joined Mozilla two weeks ago. I’m working on the Firefox Sync (née Weave Sync) project which allows Firefox users to synchronize personal browsing data securely with the cloud and thus with other devices.

When I joined, the team was working hard on getting the 1.3 release out, which meant I wasn’t really able to help out. So I spent most of my time so far improving the tests (see bugs 566575, 557590, 557596, 557588 if you’re interested). This not only benefits the project in the long run of course, it’s also an excellent way of getting to know the code.

As I’ve written in my last blog post, the lack of a decent code coverage tool for JavaScript is bothering me a bit when writing unit tests in JS and this time was no different. Fortunately, the xpcshell test harness that we’re using runs with much different privileges than what your average webapp’s JavaScript runs with. That means we get access to things like the JavaScript debugger service. That, after all, has features like breakpoint and stepping, so surely one should be able to (ab)use it somehow to record code coverage, right?

It turns out, the debugger service actually has what’s called an interrupt hook that is called for every expression the interpreter encounters. Normally you would use this in stepping mode to step through individual expressions, but it may just as well be used to do code coverage. Mind you, it’s not fast. I believe invoking the debugger already turns of the JIT, and with the interrupt hook in place, tests run about an order of magnitude slower. Ugh.

But it worked well enough for a proof of concept. So I hacked together a small patch for the test harness to dump the coverage data it collects as a JSON file. This is then picked up by a small analytics tool written in Python which generates a simple HTML report like this one. When I started writing tests we were at 49% line-by-line coverage, now we’re at 58%. Of course these numbers have to be taken cum grano salis since line-by-line coverage is just one metric. It doesn’t say anything about the quality of your tests. For all you know, some piece of code may be calling a whole bunch of code, thus making it appear covered, but you haven’t actually tested the relevant behaviour.

Anyway, have a look at the bug in the issue tracker if you’re interested in the patch and the report generator. Feedback is, of course, most welcome. It would also be fun to apply this or a similar technique to Firebug (it looks like somebody has tried in the past) so you could get code coverage on your webapp’s JavaScript without having to resort to crude code rewriting tools.

This past week I’ve spent some time here and there improving my Weave Sync client for Google Chrome. It all started out as a proof of concept, so initially I only hacked together enough to get it to show some lights on the screen. I decided that if I were to take this any further, I should man up and write some tests for it, because refactoring code or adding features without knowing for sure your stuff still works freaks me out.

I guess nowadays you don’t have to defend the whole test malarkey anymore. I’ve never really minded writing tests as I my code grows anyways. I’ve done TDD on a couple of projects before so I’m used to writing tests before or, when not in TDD mode, shortly after writing the code. When fixing a bug, for instance, I tend to always write the test first. This week, though, I was paying the price of graduating a toy project into something a bit more serious. Writing nothing but tests for a while is not that much fun.

QUnit, JSpec

I decided to go with QUnit as the testing framework. I’ve never really liked JUnit-based frameworks anyway, and especially in JavaScript the particular flavour of object-orientendness provided by JsUnit felt out of place. QUnit is from the jQuery folks — it’s used to test jQuery itself. I can highly recommend it: it’s a neat little and nimble library, much like jQuery itself.

The only other contender for me was JSpec, a JavaScript interpretation of RSpec. Their idea is basically to make your tests look like real language. It’s a bit like the doctest concept that exists in the Python world. Doctests combine documentation and unit test code which is great when you want to write well-documented tests that can also serve as an API example document to developers. RSpec and clones go further by merging the natural language and unit test into one. I find this approach intriguing and I want to try it out some time, but for now I decided against it. That said, JSpec also provides a JavaScript API which reads quite nicely. So maybe that’s a good stepping stone…

A mock XMLHttpRequest implementation

The Weave client is obviously all about talking to a server, so I needed a way to mock this XMLHttpRequests. JsUnit has a mock implementation of it but it’s pretty much useless. JSpec has a better one, but it’s all tied into JSpec and it doesn’t support Progress Events handlers, something that Mozilla and WebKit based browsers do nowadays. Because frankly the readystatechange event is just silly.

So I decided to roll out my own mock XMLHttpRequest implementation, MockHttpRequest. It tries to follow the XMLHttpRequest and Progress Events specs where possible and practical. Of course it itself is 100% tested. Try it, fork it, improve it, critique it and let me know when you do! 🙂

Code coverage

When you’re writing tests for your code, it’s always a good idea to track code coverage. That way you can make sure your tests hit every code path. This is especially, but not only, useful when you’re writing tests after the fact.

Sadly it seems there aren’t that many tools for that out there. There’s an abandoned Firefox extension that hooks into spidermonkey (Mozilla’s JavaScript interpreter) and an abandoned Firebug extension that hooks into Firebug’s profiling functionality. Hooking into an interpreter or profiler makes a lot of sense since you don’t have to change the code in question. Hopefully one of these projects will be revived one day or a similar one will come along. If somebody wants to pick this up, I could perhaps be bothered to help. 🙂

For now your best shot for JavaScript code coverage seems to be JSCoverage. It takes your existing code tree, makes a copy while inserting code to keep track of code coverage (essentially a counter that’s increased on every line of code). It’s crude but effective, so for now I’m happy to get a feel for how complete my tests are.