Firefox 4.0 beta 7 updates for BarTab and Vertical Tabs

Covert BarTab-like feature in Firefox 4.0 beta 7

A few weeks ago Paul O’Shannessy implemented cascading session restore. By doing that he took a big item off my to-do list for BarTab and made add-ons like Load Tabs Progressively redundant. Not only that, the number of concurrently loaded tabs can be tweaked through the browser.sessionstore.max_concurrent_tabs preference in about:config. If set to 0, you get instant BarTab-like behaviour. Great success!

In the meantime I have fixed a few Firefox 4.0 compatibility issues in BarTab. These fixes are in the latest BarTab beta, available from the BarTab beta channel. I’m very busy working on Firefox Sync these days so I can’t promise if I’ll be able to maintain BarTab and if it will ever officially support Firefox 4.0. But thanks to Paul, one of the biggest reasons for having BarTab installed just went away!

Vertical Tabs 0.7

Vertical Tabs has received a few updates as well and is now at version 0.7. Compatibility with Firefox 4.0 beta 7 (now the minimum required version) and recent nightlies has been restored. The styling now works better when Personas (lightweight themes) are installed. Since Firefox removed the status bar, Vertical Tabs no longer rolls a custom status bar on left, either. Instead the tabs toolbar has been moved there, and it’s fully customizable. You can, for instance, put your Sync toolbar button down there! Lastly, the Tabs on Top menu item has been removed since that choice does not make any sense once Vertical Tabs is installed.

Try out Firefox 4.0 beta 7 now!

Advertisements

Status report for October

It’s been over a month since my last mega-update, so here’s an update of what I’ve been up to since then:

Shipped: Firefox Sync 1.5

We shipped Firefox Sync 1.5 which is also part of Firefox 4.0b7 and Firefox Mobile 4.0b1. It contains various performance and stability fixes, but the most visible change is the greatly simplified (and prettier) setup wizard.

Sync Key

The “secret phrase” has been renamed to Sync Key, a term that we think will be understood better by less technical people. By default it’s an auto-generated string of 20 alphabetical characters. Users are encouraged to securely back this key up by either saving or printing a document with the key on it, though we’re planning some further additions to the wizard that will hopefully make this unnecessary for most cases (see below.)

Storage version bump

Version 1.5 also included a storage version bump, IIRC the second one so far in Sync’s history. These changes to the storage structure are necessary once in a while usually for optimization reasons (in this case server migration), but the downside is that they introduce a clear compatibility cut. Since both Firefox 4.0b6 and Firefox Mobile 4.0b1 are lacking the necessary UI notifications to tell users to upgrade Sync, the storage bump became a bit of a hassle for users trying out the Firefox 4 beta. Fortunately this has been fixed for both Firefox and Firefox Mobile.

Coming up: Firefox Sync 1.6

Sync 1.6 represents most if not all upcoming improvements that we want to include in Firefox 4.0 and are working on right now. Note that the majority of these will only be available to Firefox 4 users because they depend on changes to the underlying platform.

Encryption off the main thread

This should help UI responsiveness a lot during syncs because CPU intensive encryption and decryption processes no longer run on the main thread. We had an initial version of this working using threads but recent changes in the JägerMonkey JavaScript engine have made threads with shared state impossible. We are now looking at implementing this using ChromeWorkers, a privileged version of HTML5’s web workers.

Less synchronous I/O activity

This is an ongoing effort to improve the app’s responsiveness when Sync is tracking changes to the profile (e.g. new history visits), as well as during syncs, by making a lot of the I/O asynchronous. This includes Sync’s own I/O as well as queries to Places, Firefox’s bookmark and history database, to which we will probably be adding asynchronous APIs.

Easy setup for secondary devices

Setting up Sync on a second computer requires a lot of typing, pretty cumbersome especially on a mobile phone. We’re working on ways to improve that experience by securely transferring your Sync credentials from one device to another. One of the methods we’re exploring uses a PAKE (password-authenticated key agreement) algorithm whereby you enter a relatively small code that’s shown on one device into the other, and then the two devices agree on a stronger key that they’ll use to encrypt and then transfer your credentials with. Another method we’re looking at for Firefox Home users is 2d barcodes.

Crypto simplification

Right now Sync uses a PKI-like storage setup for the keys that the data is encrypted with. It allows for a lot of flexibility, but it also makes things unnecessarily complicated. What’s more, the additional cryptography is expensive and every additional key stored on the server means more network traffic. So reducing the number of keys and crypto steps will be good for both the client and the server. Going forward we’d therefore like to get rid of the PKI layer. We’d also like to always auto-generate the Sync Key, now with higher entropy. This means that users will no longer be able to choose their own phrase for the Sync Key, and that the length will grow from 20 to likely 25 alphabetical characters. Fortunately this will be mostly transparent to users as they won’t have to deal with the Sync Key most of the time anyway, thanks to the easy setup option.

This means another storage bump, so Sync 1.6 will likely not be backwards-compatible with Sync 1.5, but we hope that this will be the last bump for a while, so that it will be forward-compatible with Firefox 4.

Better sync timing

Right now Sync mostly syncs at predefined intervals while it could be more clever about when it syncs. Syncing more reactively after changes to the profile (e.g. user added new bookmarks) would improve the user experience a lot.

In other news

Relocated to the Bay Area

After having worked remotely for the first few months, I relocated to the Bay Area two weeks ago. I’m excited to join many of my colleagues at the Mountain View office. It has already paid off in the short time I’ve been here, particularly since I’m in the same (native) timezone as most of them now.

Podcast with Netzpolitik

While in Berlin I met Markus Beckedahl of Netzpolitik, a German blog on internet culture and politics. He wanted to interview me about my job at Mozilla in a podcast. In the podcast we talked a bit about what Mozilla is and what it’s like working in the Mozilla project. Of course we also talked about Firefox 4, Firefox Sync and various other things. Nothing an avid reader of Planet Mozilla wouldn’t already know, but if you speak German and like to listen to me answer questions (probably not always super well) for about 50 minutes, stay tuned. The podcast will be online soon, I’m told.

Mozilla, four months later. A status update.

It’s 17 September which means I’ve been working for Mozilla for exactly 4 months now. I’ve been so fantastically bad about doing regular status updates — unlike some of my more disciplined colleagues — I thought this would make a good opportunity to do one. But since I’ll have to cover so much time, this’ll be fairly high level.

Firefox Sync

As I mentioned in a previous blog post, I joined the Services team at Mozilla, which is mainly responsible for the Firefox Sync infrastructure. Right now this entails

• the Firefox Sync add-on and — as of Firefox 4 and Fennec 2 — core functionality which I work on,
• the Firefox Home iPhone app,
• the server backend (PHP for now, Python in the future),
• and the database cluster

Even if my work mostly involves traditional Firefox frontend development, the whole infrastructure provides an fun mix of technology and our team an equally fun mix of skills and personalities.

When I joined the team was getting ready to release Firefox Sync 1.3, so for the first few weeks I mostly worked on improving the test coverage — a great way to get to know the code base. After the 1.3 release the priority was to get Sync integrated into Firefox proper. We reorganized the code base of the add-on code so that we could periodically merge the client library for Sync to mozilla-central where Firefox lives. The UI has been tightly integrated with Firefox’s UI: We completely revamped the setup wizard to simplify the sign up process, there’ll be a menu item in the Firefox menu (as well as the traditional menu bar) and an optional toolbar button for those who like to invoke Sync manually.

Landing Sync in Firefox proper has been a fun ride and with a few more 4.0 betas to come, it’s not over yet.

BarTab and the future of tabs

Ever since I’ve been hired I haven’t had much time to work on BarTab, or even read the bugmail I get for BarTab. Apologies to everyone who reported issues. They’re not forgotten, just on the backburner.

The good news is that at the Mozilla Summit a bunch of us sat down together and discussed tabs. Firefox Panorama, or my much humbler Vertical Tabs, is making it very easy to get lots and lots of tabs opened in your browser. The need for throttling the session restore workload was clearly recognized and is being worked on. It might even (secretly) support BarTab’s behaviour. 😉

Fun with js-ctypes

JavaScript is undoubtedly an important corner stone of the Mozilla platform and I find its abilities amazing, not just within the browser. The js-ctypes library is a platform feature that was borrowed from Python. It allows platform (as opposed to web) JavaScript to load dynamic C  libraries and call their functions. We use it in Firefox Sync to interface with NSS, Mozilla’s crypto library and alternative to OpenSSL, without having to write glue code in C or C++ and to compile it for every platform we want to ship Firefox Sync on.

Much like with the Python equivalent, you first have to declare your types and functions for js-ctypes. This be a bit tedious as you essentially have to convert large parts of a header file to declarations in JavaScript code. You’d think it’d be possible to automate this… And it is!

Enter gcc-xml, an XML backend to gcc. It can parse files and spit out something like an Abstract Syntax Tree in XML format which you can the process. Though you can also cut out the middle man and use pygccxml, a small Python library that gives you pretty good object representation of what gcc-xml spits out. Thanks to that I was able to hack together a script that can walk through the header files of OS X frameworks and generate the corresponding js-ctypes declarations. See my “OSXTypes” project on github for the script, its output and a much shorter rewrite of the original Add to iPhoto example.

I’m sure this script can be refined. Making it work on other platforms with generic libraries, not just OS X frameworks, would be pretty cool… Other examples would also be awesome. Feel free to submit patches or even better, fork!

More on testing: test coverage

As my Twitter followers will no doubt know by now, I joined Mozilla two weeks ago. I’m working on the Firefox Sync (née Weave Sync) project which allows Firefox users to synchronize personal browsing data securely with the cloud and thus with other devices.

When I joined, the team was working hard on getting the 1.3 release out, which meant I wasn’t really able to help out. So I spent most of my time so far improving the tests (see bugs 566575, 557590, 557596, 557588 if you’re interested). This not only benefits the project in the long run of course, it’s also an excellent way of getting to know the code.

As I’ve written in my last blog post, the lack of a decent code coverage tool for JavaScript is bothering me a bit when writing unit tests in JS and this time was no different. Fortunately, the xpcshell test harness that we’re using runs with much different privileges than what your average webapp’s JavaScript runs with. That means we get access to things like the JavaScript debugger service. That, after all, has features like breakpoint and stepping, so surely one should be able to (ab)use it somehow to record code coverage, right?

It turns out, the debugger service actually has what’s called an interrupt hook that is called for every expression the interpreter encounters. Normally you would use this in stepping mode to step through individual expressions, but it may just as well be used to do code coverage. Mind you, it’s not fast. I believe invoking the debugger already turns of the JIT, and with the interrupt hook in place, tests run about an order of magnitude slower. Ugh.

But it worked well enough for a proof of concept. So I hacked together a small patch for the test harness to dump the coverage data it collects as a JSON file. This is then picked up by a small analytics tool written in Python which generates a simple HTML report like this one. When I started writing tests we were at 49% line-by-line coverage, now we’re at 58%. Of course these numbers have to be taken cum grano salis since line-by-line coverage is just one metric. It doesn’t say anything about the quality of your tests. For all you know, some piece of code may be calling a whole bunch of code, thus making it appear covered, but you haven’t actually tested the relevant behaviour.

Anyway, have a look at the bug in the issue tracker if you’re interested in the patch and the report generator. Feedback is, of course, most welcome. It would also be fun to apply this or a similar technique to Firebug (it looks like somebody has tried in the past) so you could get code coverage on your webapp’s JavaScript without having to resort to crude code rewriting tools.

On testing… QUnit, JSpec, MockHttpRequest, JSCoverage

This past week I’ve spent some time here and there improving my Weave Sync client for Google Chrome. It all started out as a proof of concept, so initially I only hacked together enough to get it to show some lights on the screen. I decided that if I were to take this any further, I should man up and write some tests for it, because refactoring code or adding features without knowing for sure your stuff still works freaks me out.

I guess nowadays you don’t have to defend the whole test malarkey anymore. I’ve never really minded writing tests as I my code grows anyways. I’ve done TDD on a couple of projects before so I’m used to writing tests before or, when not in TDD mode, shortly after writing the code. When fixing a bug, for instance, I tend to always write the test first. This week, though, I was paying the price of graduating a toy project into something a bit more serious. Writing nothing but tests for a while is not that much fun.

QUnit, JSpec

I decided to go with QUnit as the testing framework. I’ve never really liked JUnit-based frameworks anyway, and especially in JavaScript the particular flavour of object-orientendness provided by JsUnit felt out of place. QUnit is from the jQuery folks — it’s used to test jQuery itself. I can highly recommend it: it’s a neat little and nimble library, much like jQuery itself.

The only other contender for me was JSpec, a JavaScript interpretation of RSpec. Their idea is basically to make your tests look like real language. It’s a bit like the doctest concept that exists in the Python world. Doctests combine documentation and unit test code which is great when you want to write well-documented tests that can also serve as an API example document to developers. RSpec and clones go further by merging the natural language and unit test into one. I find this approach intriguing and I want to try it out some time, but for now I decided against it. That said, JSpec also provides a JavaScript API which reads quite nicely. So maybe that’s a good stepping stone…

A mock XMLHttpRequest implementation

The Weave client is obviously all about talking to a server, so I needed a way to mock this XMLHttpRequests. JsUnit has a mock implementation of it but it’s pretty much useless. JSpec has a better one, but it’s all tied into JSpec and it doesn’t support Progress Events handlers, something that Mozilla and WebKit based browsers do nowadays. Because frankly the readystatechange event is just silly.

So I decided to roll out my own mock XMLHttpRequest implementation, MockHttpRequest. It tries to follow the XMLHttpRequest and Progress Events specs where possible and practical. Of course it itself is 100% tested. Try it, fork it, improve it, critique it and let me know when you do! 🙂

Code coverage

When you’re writing tests for your code, it’s always a good idea to track code coverage. That way you can make sure your tests hit every code path. This is especially, but not only, useful when you’re writing tests after the fact.

Sadly it seems there aren’t that many tools for that out there. There’s an abandoned Firefox extension that hooks into spidermonkey (Mozilla’s JavaScript interpreter) and an abandoned Firebug extension that hooks into Firebug’s profiling functionality. Hooking into an interpreter or profiler makes a lot of sense since you don’t have to change the code in question. Hopefully one of these projects will be revived one day or a similar one will come along. If somebody wants to pick this up, I could perhaps be bothered to help. 🙂

For now your best shot for JavaScript code coverage seems to be JSCoverage. It takes your existing code tree, makes a copy while inserting code to keep track of code coverage (essentially a counter that’s increased on every line of code). It’s crude but effective, so for now I’m happy to get a feel for how complete my tests are.

On refactoring… — BarTab 2.0 is on its way

In the excellent Coders at Work book, Doug Crockford advises programmers to rewrite their stuff every six months or so. He says rewrite, but I don’t think he actually means that. Developers love rewriting stuff and most of the time it’s absolutely pointless — I know, I’ve been there.

I think what he means is refactoring. Basically streamlining the good parts and getting rid of cruft. In an ideal world, refactoring can be done in small, atomic steps. It should create no or few incompatibilities. And most crucially, it shouldn’t in any way affect the product’s shipping date.

This past week I have given BarTab this treatment. Since its creation in late January, it has grown organically. After a few months of fixing bugs, adding features, releasing early and often, and observing it “in the wild,” it was time to step back and clean it up. And boy did it need cleaning up.

My precondition for this refactoring was that I wouldn’t add any new features. Nada. Zip. Even though it would’ve been very tempting at various stages. I did manage to fix a few lingering bugs, though. In the end I turned over almost every line of code, some even twice. It was a deeply satisfying experience, and I’m glad I stuck by my no-new-features rule. It wasn’t an ideal refactoring in the sense that it was completely backwards compatible. The old API was horrible, it is now much more symmetric and free of horrible puns.

So BarTab 2.0 (available now as beta) is leaner, meaner, less invasive (no eval() hacks!) and more compatible with other Firefox add-ons. In a lot of ways it’s the BarTab that I should always have written. But you know as well as I do, that’s not how it works. Very few people write perfect code the first time round.

To me, BarTab is the perfect example of why Release Early and Often and the occasional Refactoring works extremely well for small, self-contained pieces of code such as a library, plug-in or extension. It isn’t by far the first time I’ve done things this way, but it’s certainly turned out very nicely this time.

Tabs, tabs, tabs

Choosing the web browser as the application for most of your daily tasks means that it better well be usable. In my case that primarily means dealing with lots of tabs. So it happens that much of the time I’m spending hacking Firefox is on improving the tab story.

Lazy tab loading FTW

A few months ago I was fed up with the long (re)start times of a tab encumbered Firefox and came up with BarTab, a Firefox add-on that changes tab loading behaviour so that they’re loaded only when they’re accessed. As I wrote in a blog post then, it was initially intended for tab addicts like myself, but thanks to lots of feedback from all kinds of users — tab addicts and novices alike — BarTab has continually improved to serve related use cases as well.

Firefox after a restart. The three tabs on the left haven't been loaded yet, hence their dim appearance.

I’m really overwhelmed by BarTab’s popularity. It’s been reviewed by various online journals such as LifeHacker, scrutinized for its memory and CPU cycle conserving powers, featured on Rock Your Firefox and suggested for integration into Firefox proper. That’s a steep career for an add-on that’s barely three months old!

Making bookmarks obsolete

On the other hand, three months was enough time to take a step back, observe my own interaction with the browser and analyze the user feedback. I think there’s some real usability potential in lazy tab loading.

Novice users typically don’t keep a lot of open tabs. What many of them do is bookmark regularly visited sites. This behaviour is encouraged by the bookmark toolbar being visible by default in Firefox. However, with bookmark-related UI elements in the location bar, a toolbar, the sidebar, and the menubar, I’m sure you’ll agree that the usability of the current implementation could definitely be improved. Or in the words of a Reddit user:

“It’s like someone taped a bunch of cats together.”

I would even go as far as challenging the whole concept of bookmarks. I think they require too much user interaction. Tabs, on the other hand, are easy to open, close and discover. They’re a reasonably simple concept and one that’s part of the browser already anyway. When the brower supports lazy-loading tabs via a BarTab-like mechanism, tabs can essentially serve as bookmark substitutes. You simply remember a site by keeping its tab open. The difference to bookmarks is that you don’t need to do anything to “bookmark” them. In fact, you have to do one thing less: close them. Memory wastage won’t even be an issue with smart unloading. BarTab already supports this, albeit in a crude manner.

Improving tab discoverability

Eradicating bookmarks in favour of tabs will, at least to a certain degree, make everybody a “tab addict.” This has a few scalability implications. There’s of course the challenge of making the tab mechanism itself scale to hundreds, possibly thousands of tabs. That certainly is a tricky one to say the least, but I’ll leave that aside for now. Another real issue is the tab UI.

First there’s the tab arrangement. There are very good reasons for having a vertical arrangement to the side of the browsing window. Most computers have widescreen displays these days, so vertical screen real estate is more valuable than horizontal one. More to the point, a vertical arrangement provides more space for more tabs, so it scales better to a higher number of tabs. When I maximize Firefox on my laptop, I can see about 14 tabs in a horizontal and 40 in a vertical arrangement. That’s a factor of about 3 or half an order of magnitude more.

About 14 tabs are visible in the horizontal tab bar.

With a vertical tab arrangement one can overlook about 40 tabs.

Of course not everybody needs to or wants to see that many tabs. The novice users just have a few sites they visit regularly. In this case, one could still make good use of all that vertical space, for instance by showing thumbnails for each site. This scales well even to tens of tabs and helps users visually recognize their sites.

But even tab addicts might be interested just seeing a selection of all tabs available. The current tool of choice here is Tree Style Tab. It not only displays your tabs vertically, it also allows you to arrange them in arbitrary trees. In addition to that it has about a million other features. Don’t get me wrong, it’s a fabulous and brilliant add-on, but I think it’s a bit too complicated. And the complication comes at a price: Yes, thanks to BarTab restarting Firefox with about 100 tabs is quite fast — until you enable Tree Style Tab and have it restore your tree hierarchy.

This is why I’ve started work on Vertical Tabs (code on GitHub). Its proposed feature set is simple and much constrained: arrange tabs vertically in such a way that looks native on all platforms and allow grouping of tabs. In a second iteration there might be room for some additional group features such as previews and automatic unloading of tabs (via BarTab) when groups are collapsed. But all in all I’d like to keep it simple, fast and open to as many types of users as possible.

Tabs arranged with Tree Style Tab. A bit too much structure, methinks.

Tabs arranged in groups with Vertical Tabs.

Outlook to history

To sum up, a simple experiment with tabs has shown that the browser UI and is concepts can yet be improved. Merging tabs with bookmarks to a unified concept, both in terms of the UI as well as the underlying mechanics, is without doubt a challenge. But conceptually I think it makes a lot of sense.

What’s more, this way tabs become invariably linked to the browsing history. They essentially represent an open window or link to a certain item in that history. A loaded tab is an active link, an unloaded tab is a inactive link that just happens to be represented in the UI as a tab.

This, no doubt, should make us think about how to represent the browsing history as a whole in the UI. Of course the browsing history is riddled with information that you’re normally not interested in, so you need some sort of filter. This is what the tab bar could eventually become: a filered view on your browsing history.