Wednesday, March 25th 2009
A few days ago I compared the four different releases of Firefox for start-up time (cold and warm) and page loading time. It got quite a lot of attention so today I decided to compare the five latest preview releases from the big five:
This time I did things a little more scientifically, following Justin’s suggestion in the comments. I made a batch file for each browser to print the exact time, then launch the browser, opening a page with a script showing the exact time again. The time difference equals the launching time.
It should be noted that this method requires me to opt out of Chrome’s default “new tab” page, with suggested sites. If this affects the results in any real way is unknown, but personally I doubt it. The new tab page in Chrome loads very quickly.
Cold start-ups (directly after booting your computer) are the ones that can feel like an eternity some times. For that reason I think it is more important to have a fast cold start-up than a relatively speaking fast warm one (which generally are about 5-10 times faster anyway). So let’s start with cold start-ups.
I did ten measurements for each browser. A fairly big sample size which gives tiny 99% confidence intervals, which are visible in the graph below.
IE8 is the winner here (2.40 secs), slightly faster than Chrome (2.66 secs). All browsers are, with a 99% probability, significantly different (none are tied). However, this comparison was done on my Windows (XP) computer so IE8 has an unfair advantage – who knows how large part of Internet Explorer is pre-loaded with the operating system? That makes Chrome’s performance all the more impressive.
Safari is marginally faster than Firefox (4.98 vs 5.19 secs). Surprisingly, Opera (7.14 secs) is roughly two seconds slower than Firefox and Safari. I actually thought it would be at least as fast.
Now let’s have a look at warm start-up times. I launched the browsers four times before starting the timing. Then I did 15 measurements for each browser.
Here, Chrome is in a league of its own with an average of 0.247 secs. Firefox and Opera are tied. They took on average 0.530 and 0.531 secs respectively, and their confidence intervals overlap. IE8 averaged 0.575 seconds and Safari came in last with 0.617 seconds.
Chrome impresses the most, even if IE8 launches slightly faster after reboot. Firefox and Safari are pretty similar, while Opera clearly is the slowest for cold start-ups.
These results really explain (and justify) Chrome’s good reputation for speed.
I’m curious if the differences are as large when it comes to page-loading. I’m planning on doing such a comparison too, I just need to work out a good solid method. So stay tuned if you like this kind of stuff.