MapGate

September 23, 2012

I haven’t used iOS 6 in anger yet. I don’t know whether the maps are really that horrible. I just feel like a lot of entries on The Amazing iOS 6 Maps tumblr are a bit pathological, as if people have been looking for screwed up data. Yes, some of those satellite pictures look funny and walking directions that involve swimming are lol, but it’s not like Google Maps doesn’t have or had similar problems. And how much is this going to affect the usability of the app really?

As an engineer who’s involved in getting a crazily ambitious project off the ground, within an organization that has absolutely no prior credentials in the respective field, I don’t find all this very encouraging. And it’s not just because it’s Apple… Remember when Microsoft launched Bing? I already can’t wait for how much shit Mozilla is going to get when Firefox OS launches, just because this or that is not as good as on the iPhone. Can’t wait.

I actually think it’s kind of nice to see Apple trying to create a competitive maps solution. Seeing Apple be the newcomer that’s struggling against the big guy. In a strange way, this is an Apple I can relate to. Much like the Apple from the 90s. I want to pat them on the back and encourage them.

May it also teach them some humility. They’re doing this to prevent Google service lock-in for them and their users. Maybe they’ll figure out that reciprocity is a good thing. Creating choice means that your users stay with you for the right reasons, and not because you’re holding them hostage. When you don’t take your users for granted, you try to make better products. Especially since in this business, it’s not uncommon to be relying on your competitors.

Really, all of this, not even Clock Gate, seems all that dramatic to me. I just hope it will make Apple realize that their shit don’t stink less than others. Maybe then their users will be more forgiving when they make bold moves. C’mon Apple, be the underdog again. It suits you better, anyway.

There are many success stories like this in open source, but this is such a neat one, I have to call it out. It basically goes like this:

Webdev makes awesome app (I have seen it. It’s is truly awesome. For any app, really, but *especially* for a web app). But awesome app is not as awesome as it could be in places. Webdev investigates. Finds a bottleneck in Firefox. Files bug. Goes through the seven circles of hell (XPCOM). Submits a patch. Goes through several review iterations. Patch gets committed.

Bobby Holley tells the whole story in his blog post. It’s short and worth a read.

This contribution is a testament to open source and Mozilla’s open development style. I wish we had more contributions like this (duh), but you’re probably not surprised to hear that this is pretty rare. Sure, it has to do with the level of complexity of some of the code. But, there are tons of relatively easy-to-approach parts in Firefox.

So I ask, have you ever come across a bug in Firefox that you really wanted to fix but didn’t/couldn’t? If so, what stopped you and what could perhaps have helped you?

My take on webOS and Mozilla

December 11, 2011

HP announced yesterday that they’re going to open source webOS. No matter what one may think of webOS (or HP), this is great news. It’s an opportunity, but it remains to be seen what HP and others will do with it.

Several people I’ve spoken to or chatted with are wondering whether (or even suggesting that) Mozilla should embrace webOS. On the surface, it makes a lot of sense. “webOS” is about the “web”, right?

Unfortunately, I don’t think it’s as simple as that. webOS is a technologically interesting stack, but it’s just a stack. Sure, it happens to be one that is very portable and might have a low entry barrier for developers. But WebKit and V8 do not a web make. And an app written in JS does not a web app make.

Enter Mozilla’s Boot to Gecko (B2G) project. On the surface, this too sounds like just another stack: you’re booting into Gecko instead of WebKit — so what’s the difference?

Well, B2G’s goal is about moving the entire web forward to a place where it can run a mobile phone — not run on a mobile phone but run the phone. Sure, we’re using Gecko to do this, but this is just a means to an end. Just like most of our other efforts that drive the web forward also use Gecko and Firefox as a carrier pigeon. Mozilla’s mission, after all, is to move the internet and the web forward, not make a browser or a rendering engine.

So what does driving the web rather than just particular stack forward mean? It means introducing and standardizing APIs to control just about every bit of hardware on modern phones, from the various radios to the cameras and storage devices. The idea is to level the playing field: there’s not just one stack and there’s not just one vendor in control. Just like on the web.

As a result, carriers, OEMs, and most importantly users, will be able replace and/or improve bits of the stack as they see fit and there’s also absolutely no red tape that keeps them from doing so (except for broadcasting/networking regulations, of course). This is quite different from, say, Android. It also nicely illustrates the difference between “open source” and “open”. Android is just one of those two, and it remains to be seen what webOS will be like.

I think herein lies webOS’s opportunity. The mobile landscape already has enough red tape stacks and it’s starting to disenfranchise people, and I’m sure companies too, at a large scale. If one could get anybody engaged in something new, it would be them. But not with another proprietary stack. With one that’s open.

If HP wants to give webOS the web credentials it doesn’t deserve right now, they should join Mozilla at the standards table and make webOS a “Boot to WebKit”. Competition and choice is what made the web great. Let’s do it again. And again.

The future of BarTab

December 2, 2011

BarTab has a faithful user base. Every day I get emails that look approximately like this:

Thing is, I made BarTab when I was a grad student because it was something I and Firefox needed at the time. Things have changed. I no longer am a grad student with lots of disposable time (yeah, right) and frankly, there are more important things at stake in Firefox land.

Firefox is stepping up

But! The good news is that Firefox is slowing assimilating BarTab’s feature set. That’s right! Let me show you how:

Since Firefox 8, you can tell Firefox to only load your tabs on demand, just like with BarTab. You can find this setting in the Firefox options/preferences dialog:

What’s more, my tireless colleagues Paul O’Shannessy and Tim Taubert are working on bringing even more BarTab-like features, e.g. the auto-unloading.

Who wants the keys?

As far as BarTab’s future is concerned, there might be hope. People have forked it on GitHub and apparently made it work (great work, whoever you are!), other people have made XPIs with fixes available. This is awesome, but I can’t just merge their work and release it. At least I don’t want to put my name on something that I haven’t thoroughly reviewed and tested. But, if somebody else wants to take than on, I’d be more than happy to hand the keys to BarTab over. Please get in touch with me if you’re interested!

It’s been a year and a half since I joined Mozilla and the Sync team. We’ve done exciting things since then: integrated Sync into Firefox, simplified the cryptography scheme to get rid of expensive RSA encryption and auto-generate a key for the user, provided an easy device pairing mechanism, improved performance and memory consumption, made sync scheduling smarter, error reporting less and less obnoxious, tons of reliability fixes under the hood, … the list goes on.

At the same time we’ve grown a team of three engineers to a team of a dozen or so, plus a product manager, QA, and an Ops team. There’s a roadmap and weekly client and server release trains ensure a tight and timely engineering/QA/deployment cycle. And while we’ve hit some bumps along the road, it feels like we’re doing proper software development and stuff. More importantly it feels like proper, feel-good teamwork.

With this many resources are devoted to Sync, I am happy to announce that for a while I’m going to devote most of my attention to some new stuff happening at Mozilla right now: Boot to Gecko. This rather geeky name stands for something incredibly exciting: developing a truly open and transparent phone stack that is for and of the web. It ranges from creating a phone’s complete UI and functionality with HTML/CSS/JS down to writing a GSM stack in JavaScript. I expect to dabble with all of it here and there and learn a ton from a bunch of rather intelligent people. As you can imagine, I’m pretty excited about that.

That said, taking a break from Sync, even if it may be temporary, won’t be easy. I will sorely miss Mike Connor’s mentorship, sharing two halves of a brain with Richard Newman, collaborating with battle-hardened server engineers and ops people, and being a mentor to a bunch of incredibly talented people that are all going to be much better at this stuff than I am right now. Thank you all.

Now, if you don’t mind, I’m going to put the intarwebs in ur phonez. Kthxbai.

About half a year ago Mozilla bought me a Nexus S. It is a close cousin to the Samsung Galaxy S marketed by Google. The difference is a slightly different shell, the lack of a card reader, and a plain vanilla Android 2.3 “Gingerbread”, as opposed to Samsung’s own hacked up version of Android 2.2 “Froyo” (I think). But I don’t really want to review this phone so much as the experience, which is a lot due to the operating system, Android. I haven’t used Android versions that were customized by manufacturers much, but they all disappointed in one way or another.

That’s not to say Gingerbread doesn’t have problems. Visual feedback and haptics aren’t as refined as on iOS, particularly when scrolling. It can be quite slow and drain the battery. The built-in browser is even more useless as Mobile Safari. The App Store application is a UX clusterfuck. I could go on…

So it’s not perfect. But I have to say, I prefer it to iOS, mostly for one reason: the back button. It makes going from, say, email app to the browser to the Twitter app and back a piece of cake. It’s incredibly predictable and is exactly what I want: follow a link in an email or a tweet and get back to where I was when I’m done.

It still surprises me that iOS doesn’t really have a solution for this at all. Apple poorly retrofit multitasking into the UI and the solution they came up with is throw back to 1990s: the double click. Instead of having multiple apps work together like on Android, iOS apps are silos. The Twitter client, News Reader, etc. all contain a little web browser. I want a real browser, not a lobotomized web view!

Android apps, on the other hand, have many hooks that allow them to work together, not just the back button. My favourite one is the Share feature. Apps can register themselves as share providers and other apps can share things through them. It’s so brilliant, I wish the web would work this way. (And indeed here at Mozilla we are working on making it so.)

You also get a real choice of web browsers. This doesn’t sound very important, until you actually try to open more than one web page at a time with built-in browser. Help is at hand with a variety of web browsers that have UIs geared towards power users, such as Dolphin HD and others. But unlike on iOS, these are not restricted to using the built-in WebKit, which is getting on a bit in Android.

Indeed, they’re not restricted to using WebKit at all, which means you can use other, possibly more modern browser engines like Mozilla or Opera. I’m biased for sure, but it’s probably safe to say that the most modern Andorid browser these days is Firefox Mobile. Its startup time could be a bit shorter and it can be a bit of a memory hog (help is on its way.) The browsing performance itself is pretty fast, though.

But the real game changer is Sync. You’re probably laughing even harder now since I work on Sync, so I’m outrageously biased. But up until I got the Nexus S, I had only ever used it between desktop computers. Hand on heart, it changed my mobile web experience by multiple orders of magnitude. Having access to all my browsing history, passwords, and bookmarks from my other computers means I can use the web on my mobile device exactly like on my desktop. I just can’t tell you how of a difference this makes.

Anyway, back to Android itself. The choice of web browsers is just an example of the way the device feels to me as a power user. The smell of choice is unmistakable. I can install my favourite web browser. I can tether whenever I want to how many devices I want. I realize not many people may care about this. I do and others might too.

Lastly, another positive surprise was the keyboard. I thought the iOS one was already pretty good, but Android’s keyboard (at least the stock one in Gingerbread on the Nexus S) is even better. As you type, it already comes up with suggestions. Most of the time you don’t even have to finish typing a word, it will already be in the list of suggestions and you can save yourself a lot of tapping. On iOS, on the other hand, you have to tap on the suggestion to dismiss it. It still confuses the heck out of me, even though I know how it works.

The upshot: I really like Android, so much in fact that I want an Android tablet. I’m just not sure they’re there yet. Maybe I should try out the new Galax Tab II. But that might mean I’d have to put up with a badly hacked up version of Android at which point an iPad might be the better choice.

Recently we analyzed Firefox Sync’s performance and found that it created a shockingly large number of objects during an initial sync, i.e. when all the data on the server is downloaded and applied locally. Millions of objects in fact. This caused us to not only allocate more memory than we really needed, it also meant that the garbage collection (GC) would have to step in a lot and block the main thread for several hundred milliseconds.

Here are some tips and lessons learned. (Many thanks to Andreas Gal for guiding me through the JS engine internals!)

Profiling

You can use operating system level debug and profiling tools such as Apple Instruments, but they will usually not produce useful data that will help you optimize your JavaScript. Fortunately there are several ways to instrument Spidermonkey’s GC at various levels.

How often and for how long does GC happen?

Set the javascript.options.mem.log preference to true. This will write a log message to the Error Console every time a GC happens. It will also tell you how long each GC run took, e.g.:

GC timestamp: 1298535350097039, duration: 321 ms.

This is a good way to find out if your GC is overworked, but really not much more.

What does the GC spend its time on?

To find out more, you can recompile Firefox with --enable-gctimer which defines MOZ_GCTIMER. This will dump a file called gcTimer.dat into your current working directory. This is what it looked like for a first sync before we started our optimization work:

  AppTime,  Total,   Mark,  Sweep, FinObj, FinStr,  Destroy,  newChunks, destroyChunks
      0.0,  222.7,  141.5,   59.1,   45.9,    4.4,      8.8,          7,          0 
  11805.8,  301.9,  176.2,   64.8,   52.7,    7.3,      4.8,          1,          0 
  24492.8,  258.9,  185.1,   63.4,   49.5,    7.3,      6.6,          1,          0 
  35807.6,  233.2,  195.7,   26.8,   18.5,    4.5,      3.8,          0,          0 
  48767.3,  230.5,  189.5,   26.3,   19.6,    3.8,      2.9,          0,          0 
 113195.6,  281.6,  203.6,   54.4,   32.3,   15.4,      6.7,          1,          0 
 125005.3,  338.1,  217.5,   94.8,   69.1,   14.9,     10.8,          1,          0 
 137759.5,  374.5,  199.8,  131.7,   85.8,   22.8,     23.1,          5,          0 
 149515.7,  312.2,  211.2,   75.6,   48.8,    8.8,     18.0,          0,          0 
 201876.5, 1384.0,  243.5,  754.7,  488.4,   46.1,    220.3,         40,          0 
 258690.2, 1525.4,  300.9,  822.2,  556.4,   47.6,    218.1,          0,          0 
 309871.2, 1336.6,  242.1,  713.5,  450.0,   50.3,    213.2,          0,          0 
 362310.4, 1363.3,  248.1,  731.4,  465.5,   47.0,    218.9,          1,          0 
 417596.2, 1332.4,  264.1,  702.4,  450.2,   45.4,    206.7,          0,          0 
 472853.8, 1399.9,  254.4,  762.7,  486.2,   50.1,    226.5,          0,          0 
 540223.6, 1548.2,  328.8,  848.4,  486.1,   72.0,    290.3,          0,          0 
 599972.6, 1363.9,  269.3,  729.1,  466.6,   51.3,    211.2,          0,          0 
 655834.2, 1384.6,  265.1,  737.2,  468.2,   52.5,    216.5,          0,          0 
 764844.8,  811.9,  241.0,  363.6,  233.7,   50.8,     79.1,          0,          0 
 768540.4,  213.5,  170.4,   34.1,   24.2,    5.3,      4.6,          0,          0 
 768905.3,  372.0,  134.2,  230.1,  218.1,    4.4,      7.7,          0,          0 
 769290.4,  303.0,   13.6,  258.8,  220.1,    5.0,     33.7,          0,          0 
 769742.4,   58.7,    4.9,   50.0,   24.5,    2.9,     22.6,          0,         30 
 769802.3,   12.1,    5.5,    6.1,    4.0,    2.0,      0.1,          0,          0 

This can give you a good idea of when and where the GC spends its time.

Note: the numbers in the first couple of columns are in some JS engine internal time format (they’re really some sort of cycle count, expressed in millions I believe). The Mark and Sweep times will roughly add up to the Total time. The FinObj and FinStr times indicate how long it took to free objects and strings, respectively. These numbers will roughly add up to the Sweep time.

Also, each chunk in the two rightmost columns is 1MB large, just to give you an idea of how much data we’re dealing with here.

Last resort: printf and gdb

As a last resort, you can instrument js_NewGCObject and the like in js/src/jsgcinlines.h, for instance by adding the following lines of code to the top of the function:

    if (cx->hasfp() && cx->fp()->isScriptFrame()) {
        printf("NewGCObject %s:%d\n", cx->fp()->script()->filename,
               js_PCToLineNumber(cx, cx->fp()->script(), cx->fp()->pc(cx)));
    }

Before you run Firefox with this change, make sure you disable the JIT though by setting the javascript.options.tracejit.* and javascript.options.methodjit.* preferences to false.

Also, this change will make Firefox very chatty. Best you direct this output to a file, grep for the resource or chrome URLs you’re interested in and then use a simple script to accumulate and sort the data. I ended up with something like this (output shortened a bit):

18187 resource://services-sync/engines/history.js:505
18187 resource://services-sync/engines/history.js:506
18533 resource://services-sync/record.js:264
18578 resource://services-sync/record.js:259
18613 resource://services-sync/ext/Sync.js:185
18613 resource://services-sync/ext/Sync.js:186
18615 resource://services-sync/util.js:609
18616 resource://services-crypto/WeaveCrypto.js:522
18616 resource://services-crypto/WeaveCrypto.js:523
56393 resource://services-sync/util.js:229
56393 resource://services-sync/util.js:234
56393 resource://services-sync/util.js:235
56393 resource://services-sync/util.js:236
192622 resource://services-sync/log4moz.js:361
192622 resource://services-sync/log4moz.js:365
260624 resource://services-crypto/WeaveCrypto.js:713
674818 resource://services-sync/util.js:408

Now you have a list of top offenders and you can look at them one by one. As a bonus, you also see which places in your code are called how often. In the above example, I was syncing ca. 18,000 history records, so it was easy to see which places were called once, twice or many more times for each record.

Often it will be obvious which expression on a given line of code creates an object and how to avoid (not always as obvious, more on that below.) However, in some cases the line will look completely innocent. Then you can use gdb to see what the interpreter is doing:

  1. Add a Math.sin(0) call or similar just before the line in question.
  2. Start Firefox in gdb (firefox -g) and set a breakpoint on the math_sin function.
  3. Run your code. Once you hit the math_sin breakpoint, set a breakpoint for js_NewGCObject and continue. That way you will (hopefully!) break exactly on the object creation you can’t identify.
  4. Look at the backtrace. You might see things like NewArguments which means you accessed the arguments object on that line for the first time (more on that below). Or you might see something entirely different. Fun!

How to avoid creating objects

Here are some of the patterns I encountered that create objects when not necessary. This isn’t a complete list by far and I invite you all to contribute more. Ed Lee used the above profiling techniques on his code already and came up with some useful tips. Also note that all this only applies to code that gets called a massive number of times, in a short period of time, thus putting lots of pressure on the GC.

  • Put constant things in global consts. This one is pretty “duh,” but you often don’t think about it. We had lots of code like this:
    function somewhere_in_sync() {
      let data = queryDatabase(sql_statement, ["column1", "column2", "column3"]);
      ...
    }
    

    That array of strings can obviously be pulled out of the function. Not a big deal if you call this once. Big deal if you call this 20,000 times.

  • Similarly, avoid creating arrays just to use their join method. We had lots of logging code do this:
    log.debug(["Associated", record.id, "with", record.guid].join(" "));
    

    Also, as a rule of thumb, creating strings rather than objects is better. The GC timer data as shown above has proven this as well, where string sweeping time typically amounts to a fraction of object sweeping time.

  • Reuse arrays. Avoid Array.map, Array.filter, Array.concat, Array.reduce, Array.slice, Array.splice because they create new arrays. Instead just modify the existing array in-place. As a bonus you also get to avoid a closure (see below.) For instance, we had code like this:
    let records = Array.filter(function(record) {
      return determine_if_record_should_stay(record);
    });
    

    We replaced this with

    let i, k;
    for (i = 0, k = 0; i < records.length; i++) {
      let record = records[k] = records[i]; // replace filtered items
      if (determine_if_record_should_stay(record)) {
        k += 1;
      }
    }
    records.length = k; // truncate
    

    Pretty it’s not. But avoids creating new arrays.

  • Similarly, reuse objects and functions. Lots of APIs have you pass in listener or observer objects, e.g.:
    function doStuff() {
      let value1, value2;
      someAPICall({
        onSuccess: function onSuccess() {
          // do stuff with value1, value2
        },
        onError: function onError() {
          // do stuff with value1, value2
        }
      });
    }
    

    If you can, only create this object and the functions once and put the values they need on the object, e.g.:

    const gSomeHandler = {
      onSuccess: function onSuccess() {
        // do stuff with this.value1, this.value2
      },
      onError: function onError() {
        // do stuff with this.value1, this.value2
      }
    };
    function doStuff() {
      gSomeHandler.value1 = value1;
      gSomeHandler.value2 = value2;
      someAPICall(gSomeHandler);
    }
    

    Of course this only works if you can guarantee that this function won’t be called multiple times in parallel. But even then you can at least save on creating the functions by using the global object as a prototype:

    function doStuff() {
      let handler = {};
      handler.__proto__ = gSomeHandler;
      handler.value1 = value1;
      handler.value2 = value2;
      someAPICall(handler);
    }
    
  • The magic arguments object is created lazily on the first access. Once created, it has to be garbage collected, though. Thus avoid patterns that have you do things like
      func.apply(this, arguments);
    

    or even inspect the contents or length of arguments. If you must, use

      func.call(this, arg1, arg2, ...)
    
  • Tracemonkey will try to optimize closures, so even if you define a function within a function that gets called many times, it recognizes all those functions to be the same. The problem arises when you mutate those functions, e.g. set an attribute on them. Then Tracemonkey has to clone the function object so that you end up with a unique one (Brendan explain this very well in this episode of A Minute With Brendan Eich.)

    So try to avoid mutating closure functions. Actually, as Ed points out, try to avoid closures anywhere in code that gets called a lot.

  • Lastly, when designing new XPCOM APIs, optimize for the least number of objects necessary in JavaScript. For instance, instead of requiring JS code to create an nsIURI object, accept a simple string and do the validation in native code. Moreover, allow JavaScript to opt into callbacks rather than forcing it to pass no-ops.

Conclusions

By using the profiling techniques and modifying the code as described above, we managed to eliminate more than 75% of all object allocations during an initial sync. It has been a lot of fun hunting down and killing object allocations, but there are some serious lessons learned here:

  • We need better instrumentation tools. Instrumenting functions within the JS engine isn’t available to everyone, particularly to authors of cross-browser JS frameworks. As a first step it would be great to get the number of object allocations per source code line, much like I was able to produce with printf and a little bit of script code.
  • Many async patterns involve creating new objects one way or the other, either implicitly through func.apply(this, arguments) or through generator, deferred or event objects. Neither pattern will be available to us in the guts of Sync as we’ll have to guarantee that it remains optimized for tens of thousands of calls. That’s ok, though. There’s nothing wrong with simple callbacks. We’ll just have to keep this in mind whenever we feel tempted to use syntactic sugar to replace ease the use of asynchronous APIs.

This past week I’ve spent some time here and there improving my Weave Sync client for Google Chrome. It all started out as a proof of concept, so initially I only hacked together enough to get it to show some lights on the screen. I decided that if I were to take this any further, I should man up and write some tests for it, because refactoring code or adding features without knowing for sure your stuff still works freaks me out.

I guess nowadays you don’t have to defend the whole test malarkey anymore. I’ve never really minded writing tests as I my code grows anyways. I’ve done TDD on a couple of projects before so I’m used to writing tests before or, when not in TDD mode, shortly after writing the code. When fixing a bug, for instance, I tend to always write the test first. This week, though, I was paying the price of graduating a toy project into something a bit more serious. Writing nothing but tests for a while is not that much fun.

QUnit, JSpec

I decided to go with QUnit as the testing framework. I’ve never really liked JUnit-based frameworks anyway, and especially in JavaScript the particular flavour of object-orientendness provided by JsUnit felt out of place. QUnit is from the jQuery folks — it’s used to test jQuery itself. I can highly recommend it: it’s a neat little and nimble library, much like jQuery itself.

The only other contender for me was JSpec, a JavaScript interpretation of RSpec. Their idea is basically to make your tests look like real language. It’s a bit like the doctest concept that exists in the Python world. Doctests combine documentation and unit test code which is great when you want to write well-documented tests that can also serve as an API example document to developers. RSpec and clones go further by merging the natural language and unit test into one. I find this approach intriguing and I want to try it out some time, but for now I decided against it. That said, JSpec also provides a JavaScript API which reads quite nicely. So maybe that’s a good stepping stone…

A mock XMLHttpRequest implementation

The Weave client is obviously all about talking to a server, so I needed a way to mock this XMLHttpRequests. JsUnit has a mock implementation of it but it’s pretty much useless. JSpec has a better one, but it’s all tied into JSpec and it doesn’t support Progress Events handlers, something that Mozilla and WebKit based browsers do nowadays. Because frankly the readystatechange event is just silly.

So I decided to roll out my own mock XMLHttpRequest implementation, MockHttpRequest. It tries to follow the XMLHttpRequest and Progress Events specs where possible and practical. Of course it itself is 100% tested. Try it, fork it, improve it, critique it and let me know when you do! :)

Code coverage

When you’re writing tests for your code, it’s always a good idea to track code coverage. That way you can make sure your tests hit every code path. This is especially, but not only, useful when you’re writing tests after the fact.

Sadly it seems there aren’t that many tools for that out there. There’s an abandoned Firefox extension that hooks into spidermonkey (Mozilla’s JavaScript interpreter) and an abandoned Firebug extension that hooks into Firebug’s profiling functionality. Hooking into an interpreter or profiler makes a lot of sense since you don’t have to change the code in question. Hopefully one of these projects will be revived one day or a similar one will come along. If somebody wants to pick this up, I could perhaps be bothered to help. :)

For now your best shot for JavaScript code coverage seems to be JSCoverage. It takes your existing code tree, makes a copy while inserting code to keep track of code coverage (essentially a counter that’s increased on every line of code). It’s crude but effective, so for now I’m happy to get a feel for how complete my tests are.

Tabs, tabs, tabs

May 1, 2010

Choosing the web browser as the application for most of your daily tasks means that it better well be usable. In my case that primarily means dealing with lots of tabs. So it happens that much of the time I’m spending hacking Firefox is on improving the tab story.

Lazy tab loading FTW

A few months ago I was fed up with the long (re)start times of a tab encumbered Firefox and came up with BarTab, a Firefox add-on that changes tab loading behaviour so that they’re loaded only when they’re accessed. As I wrote in a blog post then, it was initially intended for tab addicts like myself, but thanks to lots of feedback from all kinds of users — tab addicts and novices alike — BarTab has continually improved to serve related use cases as well.

Firefox after a restart. The three tabs on the left haven't been loaded yet, hence their dim appearance.

I’m really overwhelmed by BarTab’s popularity. It’s been reviewed by various online journals such as LifeHacker, scrutinized for its memory and CPU cycle conserving powers, featured on Rock Your Firefox and suggested for integration into Firefox proper. That’s a steep career for an add-on that’s barely three months old!

Making bookmarks obsolete

On the other hand, three months was enough time to take a step back, observe my own interaction with the browser and analyze the user feedback. I think there’s some real usability potential in lazy tab loading.

Novice users typically don’t keep a lot of open tabs. What many of them do is bookmark regularly visited sites. This behaviour is encouraged by the bookmark toolbar being visible by default in Firefox. However, with bookmark-related UI elements in the location bar, a toolbar, the sidebar, and the menubar, I’m sure you’ll agree that the usability of the current implementation could definitely be improved. Or in the words of a Reddit user:

“It’s like someone taped a bunch of cats together.”

I would even go as far as challenging the whole concept of bookmarks. I think they require too much user interaction. Tabs, on the other hand, are easy to open, close and discover. They’re a reasonably simple concept and one that’s part of the browser already anyway. When the brower supports lazy-loading tabs via a BarTab-like mechanism, tabs can essentially serve as bookmark substitutes. You simply remember a site by keeping its tab open. The difference to bookmarks is that you don’t need to do anything to “bookmark” them. In fact, you have to do one thing less: close them. Memory wastage won’t even be an issue with smart unloading. BarTab already supports this, albeit in a crude manner.

Improving tab discoverability

Eradicating bookmarks in favour of tabs will, at least to a certain degree, make everybody a “tab addict.” This has a few scalability implications. There’s of course the challenge of making the tab mechanism itself scale to hundreds, possibly thousands of tabs. That certainly is a tricky one to say the least, but I’ll leave that aside for now. Another real issue is the tab UI.

First there’s the tab arrangement. There are very good reasons for having a vertical arrangement to the side of the browsing window. Most computers have widescreen displays these days, so vertical screen real estate is more valuable than horizontal one. More to the point, a vertical arrangement provides more space for more tabs, so it scales better to a higher number of tabs. When I maximize Firefox on my laptop, I can see about 14 tabs in a horizontal and 40 in a vertical arrangement. That’s a factor of about 3 or half an order of magnitude more.

About 14 tabs are visible in the horizontal tab bar.

With a vertical tab arrangement one can overlook about 40 tabs.

Of course not everybody needs to or wants to see that many tabs. The novice users just have a few sites they visit regularly. In this case, one could still make good use of all that vertical space, for instance by showing thumbnails for each site. This scales well even to tens of tabs and helps users visually recognize their sites.

But even tab addicts might be interested just seeing a selection of all tabs available. The current tool of choice here is Tree Style Tab. It not only displays your tabs vertically, it also allows you to arrange them in arbitrary trees. In addition to that it has about a million other features. Don’t get me wrong, it’s a fabulous and brilliant add-on, but I think it’s a bit too complicated. And the complication comes at a price: Yes, thanks to BarTab restarting Firefox with about 100 tabs is quite fast — until you enable Tree Style Tab and have it restore your tree hierarchy.

This is why I’ve started work on Vertical Tabs (code on GitHub). Its proposed feature set is simple and much constrained: arrange tabs vertically in such a way that looks native on all platforms and allow grouping of tabs. In a second iteration there might be room for some additional group features such as previews and automatic unloading of tabs (via BarTab) when groups are collapsed. But all in all I’d like to keep it simple, fast and open to as many types of users as possible.

Tabs arranged with Tree Style Tab. A bit too much structure, methinks.

Tabs arranged in groups with Vertical Tabs.

Outlook to history

To sum up, a simple experiment with tabs has shown that the browser UI and is concepts can yet be improved. Merging tabs with bookmarks to a unified concept, both in terms of the UI as well as the underlying mechanics, is without doubt a challenge. But conceptually I think it makes a lot of sense.

What’s more, this way tabs become invariably linked to the browsing history. They essentially represent an open window or link to a certain item in that history. A loaded tab is an active link, an unloaded tab is a inactive link that just happens to be represented in the UI as a tab.

This, no doubt, should make us think about how to represent the browsing history as a whole in the UI. Of course the browsing history is riddled with information that you’re normally not interested in, so you need some sort of filter. This is what the tab bar could eventually become: a filered view on your browsing history.

Right now we’re seeing stuff happening in the webbrowser that starts to go beyond the traditional web application. I’m talking about things that make either the browser itself or the interaction between different web applications a richer experience.

On the browser side, the basket of technology commonly referred to as HTML 5 falls into this category. It makes the browser a much more powerful client, thus rebalancing a lot of weight between web server and client to an IMHO much more natural equilibrium.

On the web application side we are seeing are starting to see lots of interesting application mashups, thanks to technology like OAuth and cross-origin AJAX hacks. Facebook’s recent innovation, the social graph and the ability to embed it into other applications, is a powerful example.

As many have noted, there are huge problems with this and they all have to do with security and privacy. Virtually none of the technologies coming forward under the HTML 5 umbrella help with security. Yes there are some attempts to fix one of the most backwards security policies in the browser. But these are mere sticking plasters over oozing flesh wounds.

Weaving a richer web

As I’ve written here before, I think we need a browser designed for the cloud. Back then I was mostly speaking in terms of usability and explicitly ignored the privacy issue. It think it’s time to come back to that now. With HTML 5, we’re giving the browser many ways to be a richer client. But in terms of user data, we’re still treating it as a dumb terminal.

Weave, a Mozilla Labs project, has a different vision. Its broad idea is to create a richer web experience while still having users control their data. Concretely, the Weave Sync service allows you to synchronize your tabs, history, bookmarks, settings and passwords between different browsers. And here’s the thing: the data is all encrypted, on the client. All that Mozilla is storing on their storage nodes (which you don’t have to use, by the way) is a bunch of encrypted JSON.

Sure, you may be saying, that’s great for making Firefox a richer browser, but how does that help the general web?

Well, it turns out that doing RSA and AES cryptography in JavaScript isn’t such a far fetched idea at all. With some inspiration from an abandoned project, I was able to hack together a (very rough) version of Weave Sync for Google Chrome. Since it’s entirely written in JavaScript, it actually works in any browser.

See for yourself. (To try it out, you need to create a Weave account with some tab data in it, e.g. by installing the Weave Sync add-on in your Firefox and syncing your tabs).

Sure, you may be thinking, encryption is great for dealing with personal data, but it would be impossible in a social web. What if you wanted to share your bookmarks with other people?

Well, is it really that impossible? Let’s look at what Weave does. In a nutshell, it encrypts your private data with a symmetric “bulk key.” This bulk key is stored along with data on the server, but in encrypted form: encrypted with your public key. That means to get to your data you’ll need your private key to decrypt the bulk key which in turn can then decrypt your private data.

If I now wanted to share my bookmarks with you, I could simply give you my bulk key by encrypting it with your public key. Job done. You can see my data (and only the data I’ve encrypted with the particular bulk key that I’m sharing with you), but nobody else can. Not even Mozilla.

I know, sharing bookmarks is so 1998. But it’s essentially the same thing as the Like button (or the LUUV button). Or your address book. Or your whole social graph. Point is, we no longer need the server to do the heavy lifting for us because the browser environment is getting richer — be it the HTML templating, session storage or even cryptography. The server can become a dull data storage that we can scale the heck out of and, more crucially, potentially ditch for different one if you like. While all the data is in the client’s hands and leaves it only in encrypted form.

This is the kind of cloud I can definitely get on board with.

Follow

Get every new post delivered to your Inbox.