David Naylor: Blog

This is part two of my comparison of the latest browsers. (Part one is here.) This time I compared page-loading times, just as Betanews recently did.

The browsers I’m comparing are …

• Firefox 3.1 beta 3
• Safari 4 beta
• Internet Explorer 8
• Opera 10 alpha
• Chrome 2 beta

I compared the browsers on five different sites / web pages:

• http://www.youtube.com/
• http://www.facebook.com/home.php (logged in)
• http://www.msn.com/
• http://en.wikipedia.org/wiki/Munich
• http://www.ebay.com/

Results

Let’s just get straight to the results. I’ll go through my methods later.

In the graph above, the average page-load times for all five web pages have been added together, as have the 95% confidence intervals. All in all, this graph is based on 500 page loads.

Chrome and Firefox are tied for first place – their confidence intervals overlap. Safari and Internet Explorer are tied for third, and Opera is fifth.

Method

For each combination of browser and web site I did a total of 20 page-loads. I measured one web page at a time, working my way through the five browsers.

Since network traffic and page weight can vary over time, I did them in two sets of ten measurements. First I did ten measurements with the browsers in one order: A, B, C, D and E. Then I did ten measurements in the opposite order, starting with browser E. I also rotated the five browsers between A, B, C, D and E for the five different web pages.

Before timing the page-loads, I shift+reloaded (or the equivalent ctrl+reload in IE) the web page ten times to saturate any network cache and to get the browser warmed up. I did this for each browser, before each set of ten measurements. (Ten reloads might sound excessive, but I started off doing only three, which turned out to be too little to reach the shortest load times.)

Between each page-load I cleared all browser data (cookies, cache, etc.). Except for Facebook, where I kept cookies and secure sessions to be able to time the Facebook home page when logged in.

To time the page-loads I used this Javascript page-load timer. As the Microsoft white-paper on testing browsers says, this could introduce an observer effect. But I think we can assume that the Javascript that is being executed is pretty simple and shouldn’t affect the times noticeably.

This test showed that Google Chrome 2 beta is not 100% stable. It hung twice (in 100 page loads) and produced load times of over 30 seconds. I decided to remove these values and replace them with new ones.

Results in detail

In these graphs, each bar shows the average of 20 page-loads. The error bars represent 95% confidence intervals.

For youtube.com, Firefox and Chrome are tied for first. Safari and Internet Explorer are tied for third. Opera is last.

The Facebook home page loads fastest in Firefox and Chrome, whose confidence intervals only just overlap. The other three browsers are significantly separated.

Perhaps it is the fairly Javascript-heavy nature of Facebook that makes it load so slowly in IE8?

Msn.com: Chrome and Internet Explorer are tied for first. Firefox and Safari are tied for third. Opera is last, again.

I decided to test the browsers on a long Wikipedia article with lots of images. I looked up Munich, which turned out to be a good candidate.

Chrome and Firefox are tied for first place. Safari is third, Opera fourth and IE fifth.

Finally, ebay.com: Chrome, IE and Firefox are all tied for first place. Safari is tied with Firefox but slower than Chrome and IE. Opera is last.

Conclusions

Chrome sucks web pages off the Internet like an Electrolux. So does Firefox. In this test I haven’t managed to separate them significantly. As we all can see, Chrome has a lower average sum than Firefox, and perhaps with more data it would be possible to separate them statistically.

Opera is the slowest of the lot, which surprises me. Opera was also slowest in the start-up test. Perhaps though we should cut it some slack – it’s labelled alpha after all. Performance might improve when it reaches beta and final status. Opera also has a turbo feature in the works, but that is kind of cheating since it will lower image quality by tougher compression.

Obviously, this test could be made better in mainly two ways. I could test on more web sites, and I could do more page loads for each web site. But this test was, all in all, 500 timed page-loads and 500 non-timed page-loads. It took me more than a day to complete.

It’s also worth noting that this test is pretty much consistent with Betanews’ page load test, where Chrome 2 beta wins and Firefox 3.1 beta 3 is second.

This test was done with clean browser cache. I’m considering doing the same test but without clearing cache and cookies for each page load. After all, that’s how most page loads are done in the real world. A user who visits any of these five sites will most likely have been there many, many times before. I just need to figure out a good set-up for such a test.

A few days ago I compared the four different releases of Firefox for start-up time (cold and warm) and page loading time. It got quite a lot of attention so today I decided to compare the five latest preview releases from the big five:

• Firefox 3.1 beta 3
• Safari 4 beta
• Internet Explorer 8 (since there is no IE9 beta)
• Opera 10 alpha
• Chrome 2 beta

This time I did things a little more scientifically, following Justin’s suggestion in the comments. I made a batch file for each browser to print the exact time, then launch the browser, opening a page with a script showing the exact time again. The time difference equals the launching time.

It should be noted that this method requires me to opt out of Chrome’s default “new tab” page, with suggested sites. If this affects the results in any real way is unknown, but personally I doubt it. The new tab page in Chrome loads very quickly.

Cold start-ups (directly after booting your computer) are the ones that can feel like an eternity some times. For that reason I think it is more important to have a fast cold start-up than a relatively speaking fast warm one (which generally are about 5-10 times faster anyway). So let’s start with cold start-ups.

I did ten measurements for each browser. A fairly big sample size which gives tiny 99% confidence intervals, which are visible in the graph below.

IE8 is the winner here (2.40 secs), slightly faster than Chrome (2.66 secs). All browsers are, with a 99% probability, significantly different (none are tied). However, this comparison was done on my Windows (XP) computer so IE8 has an unfair advantage – who knows how large part of Internet Explorer is pre-loaded with the operating system? That makes Chrome’s performance all the more impressive.

Safari is marginally faster than Firefox (4.98 vs 5.19 secs). Surprisingly, Opera (7.14 secs) is roughly two seconds slower than Firefox and Safari. I actually thought it would be at least as fast.

Now let’s have a look at warm start-up times. I launched the browsers four times before starting the timing. Then I did 15 measurements for each browser.

Here, Chrome is in a league of its own with an average of 0.247 secs. Firefox and Opera are tied. They took on average 0.530 and 0.531 secs respectively, and their confidence intervals overlap. IE8 averaged 0.575 seconds and Safari came in last with 0.617 seconds.

Conclusion

Chrome impresses the most, even if IE8 launches slightly faster after reboot. Firefox and Safari are pretty similar, while Opera clearly is the slowest for cold start-ups.

These results really explain (and justify) Chrome’s good reputation for speed.

I’m curious if the differences are as large when it comes to page-loading. I’m planning on doing such a comparison too, I just need to work out a good solid method. So stay tuned if you like this kind of stuff.