MapGate

September 23, 2012

I haven’t used iOS 6 in anger yet. I don’t know whether the maps are really that horrible. I just feel like a lot of entries on The Amazing iOS 6 Maps tumblr are a bit pathological, as if people have been looking for screwed up data. Yes, some of those satellite pictures look funny and walking directions that involve swimming are lol, but it’s not like Google Maps doesn’t have or had similar problems. And how much is this going to affect the usability of the app really?

As an engineer who’s involved in getting a crazily ambitious project off the ground, within an organization that has absolutely no prior credentials in the respective field, I don’t find all this very encouraging. And it’s not just because it’s Apple… Remember when Microsoft launched Bing? I already can’t wait for how much shit Mozilla is going to get when Firefox OS launches, just because this or that is not as good as on the iPhone. Can’t wait.

I actually think it’s kind of nice to see Apple trying to create a competitive maps solution. Seeing Apple be the newcomer that’s struggling against the big guy. In a strange way, this is an Apple I can relate to. Much like the Apple from the 90s. I want to pat them on the back and encourage them.

May it also teach them some humility. They’re doing this to prevent Google service lock-in for them and their users. Maybe they’ll figure out that reciprocity is a good thing. Creating choice means that your users stay with you for the right reasons, and not because you’re holding them hostage. When you don’t take your users for granted, you try to make better products. Especially since in this business, it’s not uncommon to be relying on your competitors.

Really, all of this, not even Clock Gate, seems all that dramatic to me. I just hope it will make Apple realize that their shit don’t stink less than others. Maybe then their users will be more forgiving when they make bold moves. C’mon Apple, be the underdog again. It suits you better, anyway.

There are many success stories like this in open source, but this is such a neat one, I have to call it out. It basically goes like this:

Webdev makes awesome app (I have seen it. It’s is truly awesome. For any app, really, but *especially* for a web app). But awesome app is not as awesome as it could be in places. Webdev investigates. Finds a bottleneck in Firefox. Files bug. Goes through the seven circles of hell (XPCOM). Submits a patch. Goes through several review iterations. Patch gets committed.

Bobby Holley tells the whole story in his blog post. It’s short and worth a read.

This contribution is a testament to open source and Mozilla’s open development style. I wish we had more contributions like this (duh), but you’re probably not surprised to hear that this is pretty rare. Sure, it has to do with the level of complexity of some of the code. But, there are tons of relatively easy-to-approach parts in Firefox.

So I ask, have you ever come across a bug in Firefox that you really wanted to fix but didn’t/couldn’t? If so, what stopped you and what could perhaps have helped you?

My take on webOS and Mozilla

December 11, 2011

HP announced yesterday that they’re going to open source webOS. No matter what one may think of webOS (or HP), this is great news. It’s an opportunity, but it remains to be seen what HP and others will do with it.

Several people I’ve spoken to or chatted with are wondering whether (or even suggesting that) Mozilla should embrace webOS. On the surface, it makes a lot of sense. “webOS” is about the “web”, right?

Unfortunately, I don’t think it’s as simple as that. webOS is a technologically interesting stack, but it’s just a stack. Sure, it happens to be one that is very portable and might have a low entry barrier for developers. But WebKit and V8 do not a web make. And an app written in JS does not a web app make.

Enter Mozilla’s Boot to Gecko (B2G) project. On the surface, this too sounds like just another stack: you’re booting into Gecko instead of WebKit — so what’s the difference?

Well, B2G’s goal is about moving the entire web forward to a place where it can run a mobile phone — not run on a mobile phone but run the phone. Sure, we’re using Gecko to do this, but this is just a means to an end. Just like most of our other efforts that drive the web forward also use Gecko and Firefox as a carrier pigeon. Mozilla’s mission, after all, is to move the internet and the web forward, not make a browser or a rendering engine.

So what does driving the web rather than just particular stack forward mean? It means introducing and standardizing APIs to control just about every bit of hardware on modern phones, from the various radios to the cameras and storage devices. The idea is to level the playing field: there’s not just one stack and there’s not just one vendor in control. Just like on the web.

As a result, carriers, OEMs, and most importantly users, will be able replace and/or improve bits of the stack as they see fit and there’s also absolutely no red tape that keeps them from doing so (except for broadcasting/networking regulations, of course). This is quite different from, say, Android. It also nicely illustrates the difference between “open source” and “open”. Android is just one of those two, and it remains to be seen what webOS will be like.

I think herein lies webOS’s opportunity. The mobile landscape already has enough red tape stacks and it’s starting to disenfranchise people, and I’m sure companies too, at a large scale. If one could get anybody engaged in something new, it would be them. But not with another proprietary stack. With one that’s open.

If HP wants to give webOS the web credentials it doesn’t deserve right now, they should join Mozilla at the standards table and make webOS a “Boot to WebKit”. Competition and choice is what made the web great. Let’s do it again. And again.

The future of BarTab

December 2, 2011

BarTab has a faithful user base. Every day I get emails that look approximately like this:

Thing is, I made BarTab when I was a grad student because it was something I and Firefox needed at the time. Things have changed. I no longer am a grad student with lots of disposable time (yeah, right) and frankly, there are more important things at stake in Firefox land.

Firefox is stepping up

But! The good news is that Firefox is slowing assimilating BarTab’s feature set. That’s right! Let me show you how:

Since Firefox 8, you can tell Firefox to only load your tabs on demand, just like with BarTab. You can find this setting in the Firefox options/preferences dialog:

What’s more, my tireless colleagues Paul O’Shannessy and Tim Taubert are working on bringing even more BarTab-like features, e.g. the auto-unloading.

Who wants the keys?

As far as BarTab’s future is concerned, there might be hope. People have forked it on GitHub and apparently made it work (great work, whoever you are!), other people have made XPIs with fixes available. This is awesome, but I can’t just merge their work and release it. At least I don’t want to put my name on something that I haven’t thoroughly reviewed and tested. But, if somebody else wants to take than on, I’d be more than happy to hand the keys to BarTab over. Please get in touch with me if you’re interested!

It’s been a year and a half since I joined Mozilla and the Sync team. We’ve done exciting things since then: integrated Sync into Firefox, simplified the cryptography scheme to get rid of expensive RSA encryption and auto-generate a key for the user, provided an easy device pairing mechanism, improved performance and memory consumption, made sync scheduling smarter, error reporting less and less obnoxious, tons of reliability fixes under the hood, … the list goes on.

At the same time we’ve grown a team of three engineers to a team of a dozen or so, plus a product manager, QA, and an Ops team. There’s a roadmap and weekly client and server release trains ensure a tight and timely engineering/QA/deployment cycle. And while we’ve hit some bumps along the road, it feels like we’re doing proper software development and stuff. More importantly it feels like proper, feel-good teamwork.

With this many resources are devoted to Sync, I am happy to announce that for a while I’m going to devote most of my attention to some new stuff happening at Mozilla right now: Boot to Gecko. This rather geeky name stands for something incredibly exciting: developing a truly open and transparent phone stack that is for and of the web. It ranges from creating a phone’s complete UI and functionality with HTML/CSS/JS down to writing a GSM stack in JavaScript. I expect to dabble with all of it here and there and learn a ton from a bunch of rather intelligent people. As you can imagine, I’m pretty excited about that.

That said, taking a break from Sync, even if it may be temporary, won’t be easy. I will sorely miss Mike Connor’s mentorship, sharing two halves of a brain with Richard Newman, collaborating with battle-hardened server engineers and ops people, and being a mentor to a bunch of incredibly talented people that are all going to be much better at this stuff than I am right now. Thank you all.

Now, if you don’t mind, I’m going to put the intarwebs in ur phonez. Kthxbai.

About half a year ago Mozilla bought me a Nexus S. It is a close cousin to the Samsung Galaxy S marketed by Google. The difference is a slightly different shell, the lack of a card reader, and a plain vanilla Android 2.3 “Gingerbread”, as opposed to Samsung’s own hacked up version of Android 2.2 “Froyo” (I think). But I don’t really want to review this phone so much as the experience, which is a lot due to the operating system, Android. I haven’t used Android versions that were customized by manufacturers much, but they all disappointed in one way or another.

That’s not to say Gingerbread doesn’t have problems. Visual feedback and haptics aren’t as refined as on iOS, particularly when scrolling. It can be quite slow and drain the battery. The built-in browser is even more useless as Mobile Safari. The App Store application is a UX clusterfuck. I could go on…

So it’s not perfect. But I have to say, I prefer it to iOS, mostly for one reason: the back button. It makes going from, say, email app to the browser to the Twitter app and back a piece of cake. It’s incredibly predictable and is exactly what I want: follow a link in an email or a tweet and get back to where I was when I’m done.

It still surprises me that iOS doesn’t really have a solution for this at all. Apple poorly retrofit multitasking into the UI and the solution they came up with is throw back to 1990s: the double click. Instead of having multiple apps work together like on Android, iOS apps are silos. The Twitter client, News Reader, etc. all contain a little web browser. I want a real browser, not a lobotomized web view!

Android apps, on the other hand, have many hooks that allow them to work together, not just the back button. My favourite one is the Share feature. Apps can register themselves as share providers and other apps can share things through them. It’s so brilliant, I wish the web would work this way. (And indeed here at Mozilla we are working on making it so.)

You also get a real choice of web browsers. This doesn’t sound very important, until you actually try to open more than one web page at a time with built-in browser. Help is at hand with a variety of web browsers that have UIs geared towards power users, such as Dolphin HD and others. But unlike on iOS, these are not restricted to using the built-in WebKit, which is getting on a bit in Android.

Indeed, they’re not restricted to using WebKit at all, which means you can use other, possibly more modern browser engines like Mozilla or Opera. I’m biased for sure, but it’s probably safe to say that the most modern Andorid browser these days is Firefox Mobile. Its startup time could be a bit shorter and it can be a bit of a memory hog (help is on its way.) The browsing performance itself is pretty fast, though.

But the real game changer is Sync. You’re probably laughing even harder now since I work on Sync, so I’m outrageously biased. But up until I got the Nexus S, I had only ever used it between desktop computers. Hand on heart, it changed my mobile web experience by multiple orders of magnitude. Having access to all my browsing history, passwords, and bookmarks from my other computers means I can use the web on my mobile device exactly like on my desktop. I just can’t tell you how of a difference this makes.

Anyway, back to Android itself. The choice of web browsers is just an example of the way the device feels to me as a power user. The smell of choice is unmistakable. I can install my favourite web browser. I can tether whenever I want to how many devices I want. I realize not many people may care about this. I do and others might too.

Lastly, another positive surprise was the keyboard. I thought the iOS one was already pretty good, but Android’s keyboard (at least the stock one in Gingerbread on the Nexus S) is even better. As you type, it already comes up with suggestions. Most of the time you don’t even have to finish typing a word, it will already be in the list of suggestions and you can save yourself a lot of tapping. On iOS, on the other hand, you have to tap on the suggestion to dismiss it. It still confuses the heck out of me, even though I know how it works.

The upshot: I really like Android, so much in fact that I want an Android tablet. I’m just not sure they’re there yet. Maybe I should try out the new Galax Tab II. But that might mean I’d have to put up with a badly hacked up version of Android at which point an iPad might be the better choice.

Recently we analyzed Firefox Sync’s performance and found that it created a shockingly large number of objects during an initial sync, i.e. when all the data on the server is downloaded and applied locally. Millions of objects in fact. This caused us to not only allocate more memory than we really needed, it also meant that the garbage collection (GC) would have to step in a lot and block the main thread for several hundred milliseconds.

Here are some tips and lessons learned. (Many thanks to Andreas Gal for guiding me through the JS engine internals!)

Profiling

You can use operating system level debug and profiling tools such as Apple Instruments, but they will usually not produce useful data that will help you optimize your JavaScript. Fortunately there are several ways to instrument Spidermonkey’s GC at various levels.

How often and for how long does GC happen?

Set the javascript.options.mem.log preference to true. This will write a log message to the Error Console every time a GC happens. It will also tell you how long each GC run took, e.g.:

GC timestamp: 1298535350097039, duration: 321 ms.

This is a good way to find out if your GC is overworked, but really not much more.

What does the GC spend its time on?

To find out more, you can recompile Firefox with --enable-gctimer which defines MOZ_GCTIMER. This will dump a file called gcTimer.dat into your current working directory. This is what it looked like for a first sync before we started our optimization work:

  AppTime,  Total,   Mark,  Sweep, FinObj, FinStr,  Destroy,  newChunks, destroyChunks
      0.0,  222.7,  141.5,   59.1,   45.9,    4.4,      8.8,          7,          0 
  11805.8,  301.9,  176.2,   64.8,   52.7,    7.3,      4.8,          1,          0 
  24492.8,  258.9,  185.1,   63.4,   49.5,    7.3,      6.6,          1,          0 
  35807.6,  233.2,  195.7,   26.8,   18.5,    4.5,      3.8,          0,          0 
  48767.3,  230.5,  189.5,   26.3,   19.6,    3.8,      2.9,          0,          0 
 113195.6,  281.6,  203.6,   54.4,   32.3,   15.4,      6.7,          1,          0 
 125005.3,  338.1,  217.5,   94.8,   69.1,   14.9,     10.8,          1,          0 
 137759.5,  374.5,  199.8,  131.7,   85.8,   22.8,     23.1,          5,          0 
 149515.7,  312.2,  211.2,   75.6,   48.8,    8.8,     18.0,          0,          0 
 201876.5, 1384.0,  243.5,  754.7,  488.4,   46.1,    220.3,         40,          0 
 258690.2, 1525.4,  300.9,  822.2,  556.4,   47.6,    218.1,          0,          0 
 309871.2, 1336.6,  242.1,  713.5,  450.0,   50.3,    213.2,          0,          0 
 362310.4, 1363.3,  248.1,  731.4,  465.5,   47.0,    218.9,          1,          0 
 417596.2, 1332.4,  264.1,  702.4,  450.2,   45.4,    206.7,          0,          0 
 472853.8, 1399.9,  254.4,  762.7,  486.2,   50.1,    226.5,          0,          0 
 540223.6, 1548.2,  328.8,  848.4,  486.1,   72.0,    290.3,          0,          0 
 599972.6, 1363.9,  269.3,  729.1,  466.6,   51.3,    211.2,          0,          0 
 655834.2, 1384.6,  265.1,  737.2,  468.2,   52.5,    216.5,          0,          0 
 764844.8,  811.9,  241.0,  363.6,  233.7,   50.8,     79.1,          0,          0 
 768540.4,  213.5,  170.4,   34.1,   24.2,    5.3,      4.6,          0,          0 
 768905.3,  372.0,  134.2,  230.1,  218.1,    4.4,      7.7,          0,          0 
 769290.4,  303.0,   13.6,  258.8,  220.1,    5.0,     33.7,          0,          0 
 769742.4,   58.7,    4.9,   50.0,   24.5,    2.9,     22.6,          0,         30 
 769802.3,   12.1,    5.5,    6.1,    4.0,    2.0,      0.1,          0,          0 

This can give you a good idea of when and where the GC spends its time.

Note: the numbers in the first couple of columns are in some JS engine internal time format (they’re really some sort of cycle count, expressed in millions I believe). The Mark and Sweep times will roughly add up to the Total time. The FinObj and FinStr times indicate how long it took to free objects and strings, respectively. These numbers will roughly add up to the Sweep time.

Also, each chunk in the two rightmost columns is 1MB large, just to give you an idea of how much data we’re dealing with here.

Last resort: printf and gdb

As a last resort, you can instrument js_NewGCObject and the like in js/src/jsgcinlines.h, for instance by adding the following lines of code to the top of the function:

    if (cx->hasfp() && cx->fp()->isScriptFrame()) {
        printf("NewGCObject %s:%d\n", cx->fp()->script()->filename,
               js_PCToLineNumber(cx, cx->fp()->script(), cx->fp()->pc(cx)));
    }

Before you run Firefox with this change, make sure you disable the JIT though by setting the javascript.options.tracejit.* and javascript.options.methodjit.* preferences to false.

Also, this change will make Firefox very chatty. Best you direct this output to a file, grep for the resource or chrome URLs you’re interested in and then use a simple script to accumulate and sort the data. I ended up with something like this (output shortened a bit):

18187 resource://services-sync/engines/history.js:505
18187 resource://services-sync/engines/history.js:506
18533 resource://services-sync/record.js:264
18578 resource://services-sync/record.js:259
18613 resource://services-sync/ext/Sync.js:185
18613 resource://services-sync/ext/Sync.js:186
18615 resource://services-sync/util.js:609
18616 resource://services-crypto/WeaveCrypto.js:522
18616 resource://services-crypto/WeaveCrypto.js:523
56393 resource://services-sync/util.js:229
56393 resource://services-sync/util.js:234
56393 resource://services-sync/util.js:235
56393 resource://services-sync/util.js:236
192622 resource://services-sync/log4moz.js:361
192622 resource://services-sync/log4moz.js:365
260624 resource://services-crypto/WeaveCrypto.js:713
674818 resource://services-sync/util.js:408

Now you have a list of top offenders and you can look at them one by one. As a bonus, you also see which places in your code are called how often. In the above example, I was syncing ca. 18,000 history records, so it was easy to see which places were called once, twice or many more times for each record.

Often it will be obvious which expression on a given line of code creates an object and how to avoid (not always as obvious, more on that below.) However, in some cases the line will look completely innocent. Then you can use gdb to see what the interpreter is doing:

  1. Add a Math.sin(0) call or similar just before the line in question.
  2. Start Firefox in gdb (firefox -g) and set a breakpoint on the math_sin function.
  3. Run your code. Once you hit the math_sin breakpoint, set a breakpoint for js_NewGCObject and continue. That way you will (hopefully!) break exactly on the object creation you can’t identify.
  4. Look at the backtrace. You might see things like NewArguments which means you accessed the arguments object on that line for the first time (more on that below). Or you might see something entirely different. Fun!

How to avoid creating objects

Here are some of the patterns I encountered that create objects when not necessary. This isn’t a complete list by far and I invite you all to contribute more. Ed Lee used the above profiling techniques on his code already and came up with some useful tips. Also note that all this only applies to code that gets called a massive number of times, in a short period of time, thus putting lots of pressure on the GC.

  • Put constant things in global consts. This one is pretty “duh,” but you often don’t think about it. We had lots of code like this:
    function somewhere_in_sync() {
      let data = queryDatabase(sql_statement, ["column1", "column2", "column3"]);
      ...
    }
    

    That array of strings can obviously be pulled out of the function. Not a big deal if you call this once. Big deal if you call this 20,000 times.

  • Similarly, avoid creating arrays just to use their join method. We had lots of logging code do this:
    log.debug(["Associated", record.id, "with", record.guid].join(" "));
    

    Also, as a rule of thumb, creating strings rather than objects is better. The GC timer data as shown above has proven this as well, where string sweeping time typically amounts to a fraction of object sweeping time.

  • Reuse arrays. Avoid Array.map, Array.filter, Array.concat, Array.reduce, Array.slice, Array.splice because they create new arrays. Instead just modify the existing array in-place. As a bonus you also get to avoid a closure (see below.) For instance, we had code like this:
    let records = Array.filter(function(record) {
      return determine_if_record_should_stay(record);
    });
    

    We replaced this with

    let i, k;
    for (i = 0, k = 0; i < records.length; i++) {
      let record = records[k] = records[i]; // replace filtered items
      if (determine_if_record_should_stay(record)) {
        k += 1;
      }
    }
    records.length = k; // truncate
    

    Pretty it’s not. But avoids creating new arrays.

  • Similarly, reuse objects and functions. Lots of APIs have you pass in listener or observer objects, e.g.:
    function doStuff() {
      let value1, value2;
      someAPICall({
        onSuccess: function onSuccess() {
          // do stuff with value1, value2
        },
        onError: function onError() {
          // do stuff with value1, value2
        }
      });
    }
    

    If you can, only create this object and the functions once and put the values they need on the object, e.g.:

    const gSomeHandler = {
      onSuccess: function onSuccess() {
        // do stuff with this.value1, this.value2
      },
      onError: function onError() {
        // do stuff with this.value1, this.value2
      }
    };
    function doStuff() {
      gSomeHandler.value1 = value1;
      gSomeHandler.value2 = value2;
      someAPICall(gSomeHandler);
    }
    

    Of course this only works if you can guarantee that this function won’t be called multiple times in parallel. But even then you can at least save on creating the functions by using the global object as a prototype:

    function doStuff() {
      let handler = {};
      handler.__proto__ = gSomeHandler;
      handler.value1 = value1;
      handler.value2 = value2;
      someAPICall(handler);
    }
    
  • The magic arguments object is created lazily on the first access. Once created, it has to be garbage collected, though. Thus avoid patterns that have you do things like
      func.apply(this, arguments);
    

    or even inspect the contents or length of arguments. If you must, use

      func.call(this, arg1, arg2, ...)
    
  • Tracemonkey will try to optimize closures, so even if you define a function within a function that gets called many times, it recognizes all those functions to be the same. The problem arises when you mutate those functions, e.g. set an attribute on them. Then Tracemonkey has to clone the function object so that you end up with a unique one (Brendan explain this very well in this episode of A Minute With Brendan Eich.)

    So try to avoid mutating closure functions. Actually, as Ed points out, try to avoid closures anywhere in code that gets called a lot.

  • Lastly, when designing new XPCOM APIs, optimize for the least number of objects necessary in JavaScript. For instance, instead of requiring JS code to create an nsIURI object, accept a simple string and do the validation in native code. Moreover, allow JavaScript to opt into callbacks rather than forcing it to pass no-ops.

Conclusions

By using the profiling techniques and modifying the code as described above, we managed to eliminate more than 75% of all object allocations during an initial sync. It has been a lot of fun hunting down and killing object allocations, but there are some serious lessons learned here:

  • We need better instrumentation tools. Instrumenting functions within the JS engine isn’t available to everyone, particularly to authors of cross-browser JS frameworks. As a first step it would be great to get the number of object allocations per source code line, much like I was able to produce with printf and a little bit of script code.
  • Many async patterns involve creating new objects one way or the other, either implicitly through func.apply(this, arguments) or through generator, deferred or event objects. Neither pattern will be available to us in the guts of Sync as we’ll have to guarantee that it remains optimized for tens of thousands of calls. That’s ok, though. There’s nothing wrong with simple callbacks. We’ll just have to keep this in mind whenever we feel tempted to use syntactic sugar to replace ease the use of asynchronous APIs.
Follow

Get every new post delivered to your Inbox.