Monday, March 30th 2009
The browsers I’m comparing are …
I compared the browsers on five different sites / web pages:
Let’s just get straight to the results. I’ll go through my methods later.
In the graph above, the average page-load times for all five web pages have been added together, as have the 95% confidence intervals. All in all, this graph is based on 500 page loads.
Chrome and Firefox are tied for first place – their confidence intervals overlap. Safari and Internet Explorer are tied for third, and Opera is fifth.
For each combination of browser and web site I did a total of 20 page-loads. I measured one web page at a time, working my way through the five browsers.
Since network traffic and page weight can vary over time, I did them in two sets of ten measurements. First I did ten measurements with the browsers in one order: A, B, C, D and E. Then I did ten measurements in the opposite order, starting with browser E. I also rotated the five browsers between A, B, C, D and E for the five different web pages.
Before timing the page-loads, I shift+reloaded (or the equivalent ctrl+reload in IE) the web page ten times to saturate any network cache and to get the browser
warmed up. I did this for each browser, before each set of ten measurements. (Ten reloads might sound excessive, but I started off doing only three, which turned out to be too little to reach the shortest load times.)
Between each page-load I cleared all browser data (cookies, cache, etc.). Except for Facebook, where I kept cookies and secure sessions to be able to time the Facebook home page when logged in.
This test showed that Google Chrome 2 beta is not 100% stable. It hung twice (in 100 page loads) and produced load times of over 30 seconds. I decided to remove these values and replace them with new ones.
In these graphs, each bar shows the average of 20 page-loads. The error bars represent 95% confidence intervals.
For youtube.com, Firefox and Chrome are tied for first. Safari and Internet Explorer are tied for third. Opera is last.
The Facebook home page loads fastest in Firefox and Chrome, whose confidence intervals only just overlap. The other three browsers are significantly separated.
Msn.com: Chrome and Internet Explorer are tied for first. Firefox and Safari are tied for third. Opera is last, again.
I decided to test the browsers on a long Wikipedia article with lots of images. I looked up Munich, which turned out to be a good candidate.
Chrome and Firefox are tied for first place. Safari is third, Opera fourth and IE fifth.
Finally, ebay.com: Chrome, IE and Firefox are all tied for first place. Safari is tied with Firefox but slower than Chrome and IE. Opera is last.
Chrome sucks web pages off the Internet like an Electrolux. So does Firefox. In this test I haven’t managed to separate them significantly. As we all can see, Chrome has a lower average sum than Firefox, and perhaps with more data it would be possible to separate them statistically.
Opera is the slowest of the lot, which surprises me. Opera was also slowest in the start-up test. Perhaps though we should cut it some slack – it’s labelled alpha after all. Performance might improve when it reaches beta and final status. Opera also has a turbo feature in the works, but that is kind of cheating since it will lower image quality by tougher compression.
Obviously, this test could be made better in mainly two ways. I could test on more web sites, and I could do more page loads for each web site. But this test was, all in all, 500 timed page-loads and 500 non-timed page-loads. It took me more than a day to complete.
It’s also worth noting that this test is pretty much consistent with Betanews’ page load test, where Chrome 2 beta wins and Firefox 3.1 beta 3 is second.
This test was done with clean browser cache. I’m considering doing the same test but without clearing cache and cookies for each page load. After all, that’s how most page loads are done in the real world. A user who visits any of these five sites will most likely have been there many, many times before. I just need to figure out a good set-up for such a test.