This is part two of my comparison of the latest browsers. (Part one is here.) This time I compared page-loading times, just as Betanews recently did.
The browsers I’m comparing are …
- Firefox 3.1 beta 3
- Safari 4 beta
- Internet Explorer 8
- Opera 10 alpha
- Chrome 2 beta
I compared the browsers on five different sites / web pages:
Results
Let’s just get straight to the results. I’ll go through my methods later.
In the graph above, the average page-load times for all five web pages have been added together, as have the 95% confidence intervals. All in all, this graph is based on 500 page loads.
Chrome and Firefox are tied for first place – their confidence intervals overlap. Safari and Internet Explorer are tied for third, and Opera is fifth.
Method
For each combination of browser and web site I did a total of 20 page-loads. I measured one web page at a time, working my way through the five browsers.
Since network traffic and page weight can vary over time, I did them in two sets of ten measurements. First I did ten measurements with the browsers in one order: A, B, C, D and E. Then I did ten measurements in the opposite order, starting with browser E. I also rotated the five browsers between A, B, C, D and E for the five different web pages.
Before timing the page-loads, I shift+reloaded (or the equivalent ctrl+reload in IE) the web page ten times to saturate any network cache and to get the browser warmed up
. I did this for each browser, before each set of ten measurements. (Ten reloads might sound excessive, but I started off doing only three, which turned out to be too little to reach the shortest load times.)
Between each page-load I cleared all browser data (cookies, cache, etc.). Except for Facebook, where I kept cookies and secure sessions to be able to time the Facebook home page when logged in.
To time the page-loads I used this Javascript page-load timer. As the Microsoft white-paper on testing browsers says, this could introduce an observer effect
. But I think we can assume that the Javascript that is being executed is pretty simple and shouldn’t affect the times noticeably.
This test showed that Google Chrome 2 beta is not 100% stable. It hung twice (in 100 page loads) and produced load times of over 30 seconds. I decided to remove these values and replace them with new ones.
Results in detail
In these graphs, each bar shows the average of 20 page-loads. The error bars represent 95% confidence intervals.
For youtube.com, Firefox and Chrome are tied for first. Safari and Internet Explorer are tied for third. Opera is last.
The Facebook home page loads fastest in Firefox and Chrome, whose confidence intervals only just overlap. The other three browsers are significantly separated.
Perhaps it is the fairly Javascript-heavy nature of Facebook that makes it load so slowly in IE8?
Msn.com: Chrome and Internet Explorer are tied for first. Firefox and Safari are tied for third. Opera is last, again.
I decided to test the browsers on a long Wikipedia article with lots of images. I looked up Munich, which turned out to be a good candidate.
Chrome and Firefox are tied for first place. Safari is third, Opera fourth and IE fifth.
Finally, ebay.com: Chrome, IE and Firefox are all tied for first place. Safari is tied with Firefox but slower than Chrome and IE. Opera is last.
Conclusions
Chrome sucks web pages off the Internet like an Electrolux. So does Firefox. In this test I haven’t managed to separate them significantly. As we all can see, Chrome has a lower average sum than Firefox, and perhaps with more data it would be possible to separate them statistically.
Opera is the slowest of the lot, which surprises me. Opera was also slowest in the start-up test. Perhaps though we should cut it some slack – it’s labelled alpha after all. Performance might improve when it reaches beta and final status. Opera also has a turbo feature in the works, but that is kind of cheating since it will lower image quality by tougher compression.
Obviously, this test could be made better in mainly two ways. I could test on more web sites, and I could do more page loads for each web site. But this test was, all in all, 500 timed page-loads and 500 non-timed page-loads. It took me more than a day to complete.
It’s also worth noting that this test is pretty much consistent with Betanews’ page load test, where Chrome 2 beta wins and Firefox 3.1 beta 3 is second.
This test was done with clean browser cache. I’m considering doing the same test but without clearing cache and cookies for each page load. After all, that’s how most page loads are done in the real world. A user who visits any of these five sites will most likely have been there many, many times before. I just need to figure out a good set-up for such a test.
Full-text blog feed
