Monday, March 30th 2009

Beta Browser Battle 2: Page-load times

This is part two of my comparison of the latest browsers. (Part one is here.) This time I compared page-loading times, just as Betanews recently did.

The browsers I’m comparing are …

I compared the browsers on five different sites / web pages:

Results

Let’s just get straight to the results. I’ll go through my methods later.

Graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta

In the graph above, the average page-load times for all five web pages have been added together, as have the 95% confidence intervals. All in all, this graph is based on 500 page loads.

Chrome and Firefox are tied for first place – their confidence intervals overlap. Safari and Internet Explorer are tied for third, and Opera is fifth.

Method

For each combination of browser and web site I did a total of 20 page-loads. I measured one web page at a time, working my way through the five browsers.

Since network traffic and page weight can vary over time, I did them in two sets of ten measurements. First I did ten measurements with the browsers in one order: A, B, C, D and E. Then I did ten measurements in the opposite order, starting with browser E. I also rotated the five browsers between A, B, C, D and E for the five different web pages.

Before timing the page-loads, I shift+reloaded (or the equivalent ctrl+reload in IE) the web page ten times to saturate any network cache and to get the browser warmed up. I did this for each browser, before each set of ten measurements. (Ten reloads might sound excessive, but I started off doing only three, which turned out to be too little to reach the shortest load times.)

Between each page-load I cleared all browser data (cookies, cache, etc.). Except for Facebook, where I kept cookies and secure sessions to be able to time the Facebook home page when logged in.

To time the page-loads I used this Javascript page-load timer. As the Microsoft white-paper on testing browsers says, this could introduce an observer effect. But I think we can assume that the Javascript that is being executed is pretty simple and shouldn’t affect the times noticeably.

This test showed that Google Chrome 2 beta is not 100% stable. It hung twice (in 100 page loads) and produced load times of over 30 seconds. I decided to remove these values and replace them with new ones.

Results in detail

In these graphs, each bar shows the average of 20 page-loads. The error bars represent 95% confidence intervals.

Chart or graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta on youtube.com.

For youtube.com, Firefox and Chrome are tied for first. Safari and Internet Explorer are tied for third. Opera is last.

Chart or graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta on the Facebook home page.

The Facebook home page loads fastest in Firefox and Chrome, whose confidence intervals only just overlap. The other three browsers are significantly separated.

Perhaps it is the fairly Javascript-heavy nature of Facebook that makes it load so slowly in IE8?

Chart or graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta on msn.com.

Msn.com: Chrome and Internet Explorer are tied for first. Firefox and Safari are tied for third. Opera is last, again.

Chart or graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta on a Wikipedia article.

I decided to test the browsers on a long Wikipedia article with lots of images. I looked up Munich, which turned out to be a good candidate.

Chrome and Firefox are tied for first place. Safari is third, Opera fourth and IE fifth.

Chart or graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta on ebay.com.

Finally, ebay.com: Chrome, IE and Firefox are all tied for first place. Safari is tied with Firefox but slower than Chrome and IE. Opera is last.

Conclusions

Chrome sucks web pages off the Internet like an Electrolux. So does Firefox. In this test I haven’t managed to separate them significantly. As we all can see, Chrome has a lower average sum than Firefox, and perhaps with more data it would be possible to separate them statistically.

Opera is the slowest of the lot, which surprises me. Opera was also slowest in the start-up test. Perhaps though we should cut it some slack – it’s labelled alpha after all. Performance might improve when it reaches beta and final status. Opera also has a turbo feature in the works, but that is kind of cheating since it will lower image quality by tougher compression.

Obviously, this test could be made better in mainly two ways. I could test on more web sites, and I could do more page loads for each web site. But this test was, all in all, 500 timed page-loads and 500 non-timed page-loads. It took me more than a day to complete.

It’s also worth noting that this test is pretty much consistent with Betanews’ page load test, where Chrome 2 beta wins and Firefox 3.1 beta 3 is second.

This test was done with clean browser cache. I’m considering doing the same test but without clearing cache and cookies for each page load. After all, that’s how most page loads are done in the real world. A user who visits any of these five sites will most likely have been there many, many times before. I just need to figure out a good set-up for such a test.

2 Responses to this post:

  1. Anonymous says:

    IMHO, any future test should include (and vary) certain important settings that influence the actual and perceived page load times, especially on Firefox and OperaFF: something that could be achieved partly with the Fasterfox and Configuration mania extensions; about:config settings for:content.interrupt.parsingnglayout.initialpaint.delaycontent.* , if parsing is trueprettu much all settings listed at http://kb.mozillazine.org/Category:Tweaking_preferencesEffects of extensions (test firefox in safe mode vs. same Firefox /w some vital extensions enabled)Test more than one Firefox version;at least the latest stable (3.0.8),latest branch 3.5b4prelatest trunk 3.6a1preWith and without Tracemonkey enabledTest K-Meleon vs. Firefox as well (with same gecko and same prefs.js)Opera: prefs/advanced/browsing/Loading:redraw: Instantly/after x seconds/after page has finished loadingChrome: test latest chromium builds as well( http://build.chromium.org/buildbot/snapshots/chromium-rel-xp/ , chrome-win32.zip in portable chrome/chromiumTrident based browsers: test IE7/IE8 vs eg. Avant/Maxthon

  2. Anonymous says:

    important for me:-page load (of course)ANDone thing opera does very well, the back-button: when i hit back, i AM back, NOT like IE, that goes on to reload the page, i only left about 2 sec ago(which makes no sense, the page does not change within that short period)Is that better with IE8 or chrome?