Bas Schouten

So, it seems my little demo of a pre-alpha firefox build with Direct2D support has generated quite some attention! This is good, in many ways. Already users trying out the builds have helped us fix many bugs in it. So I'm already raking in some of the benefits. It has also, understandably, led to a lot of people running their own tests, some more useful than others, some perhaps wrong, or inaccurate. In any case, first of all I wanted to discuss a little on how to analyze D2D performance with a simple firefox build.

Use the same build
Don't compare a random nightly to this build in performance. The nightly builds contain updates, no two nightly builds will be the same, and therefor perform the same. I don't continuously keep my D2D builds up with the latest repository head. Additionally the build flags used may very well not be the same, for example some builds(like the nightlies) may be built with something called Profile Guided Optimization, which means the compiler analyzes hotspots, and optimizes them. This significantly improves their JavaScript performance. My test build is not build with PGO, although I might release a build using PGO as it gets more stable. This is probably causing some of the differences some people testing have been seeing. Please keep in mind that because this is not a final, complete build, it should probably not be compared too enthusiastically to other browsers either.

If you want to properly compare performance, use my build, and switch on the forcegdi pref. Go to 'about:config' and look for the font.engine.forcegdi pref, and set that to true. After that, you will have a build using GDI only.

Obviously I should have mentioned this in my previous post, so people would not have wasted their time on inaccurate performance analysis. I apologize for that, it is partially because I had not expected so much publicity.

Focus on what (should be) different
When you do run tests, it is of course always valuable to get measurements on the overall performance of the browser. If there's some other part of the browser than rendering showing performance decreases, I'm doing something wrong and please do let me know! There is however, a lot of more involved with displaying and parsing a webpage than just the rendering. If you want to get a really good idea on what it does to your rendering speed, you'll want to measure solely the time it takes to do the actual drawing. We have a non-cross-browser tool in Mozilla in my build that will allow you to do this. You can add a 'bookmarklet' that runs this test, just add a bookmark, and make it point to 'javascript:var r = window.QueryInterface(Components.interfaces.nsIInterfaceRequestor).getInterface(Components.interfaces.nsIDOMWindowUtils).redraw(500); alert(r + " ms"); where the number inside 'redraw' is the number of rendering runs you want to time. It will then pop up an alert box which will tell you the time in milliseconds it took to execute the redraws. Keep in mind this still only analyzes static rendering performance.

Know what you're measuring, and how you're doing it
If you are using an independent measurement benchmark. Be sure you understand well exactly what it measures, and how it does it. This is a very important step. Something spewing out a number and then listing 'higher is better' for a certain part of functionality is great, but it only becomes useful information when you know how the measurement is executed, what it's margin of error is, and what the overhead is it adds to the whatever it is testing. For a lot of all-round browser benchmarks, rendering is only a small part of what's tested. And the total test result differences between D2D enabled and disabled, may not reflect the actual difference in user experience.

Considerations on static page-loading measurements
Since obviously just a rendering improvement if it doesn't actually function better for the end user is practically useless, keep in mind that when you're timing how long it takes to load a page, you're timing all the aforementioned overhead. When interacting with the page without switching to another page, a lot of this overhead does not occur, and a large part of time might be spent in actual rendering. This means that during dynamic interaction (like scrolling), improvements may be more noticeable, although harder to measure.

And finally, thanks!
It's been great to see how many people are trying this, and as I mentioned earlier in this post, it has already greatly assisted us. It's also great that people are working on their own tests of performance and such things, it is always a good thing to have independent performance tests ran. And this too, will help us improve on the build in areas where our own testing may have been lacking. So, thanks, to everyone who is directly or indirectly helping to hopefully provide a great new feature!