Archive for March, 2009

Monday, March 30th 2009

Beta Browser Battle 2: Page-load times

This is part two of my comparison of the latest browsers. (Part one is here.) This time I compared page-loading times, just as Betanews recently did.

The browsers I’m comparing are …

  • Firefox 3.1 beta 3
  • Safari 4 beta
  • Internet Explorer 8
  • Opera 10 alpha
  • Chrome 2 beta

I compared the browsers on five different sites / web pages:

Results

Let’s just get straight to the results. I’ll go through my methods later.

Graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta

In the graph above, the average page-load times for all five web pages have been added together, as have the 95% confidence intervals. All in all, this graph is based on 500 page loads.

Chrome and Firefox are tied for first place – their confidence intervals overlap. Safari and Internet Explorer are tied for third, and Opera is fifth.

Method

For each combination of browser and web site I did a total of 20 page-loads. I measured one web page at a time, working my way through the five browsers.

Since network traffic and page weight can vary over time, I did them in two sets of ten measurements. First I did ten measurements with the browsers in one order: A, B, C, D and E. Then I did ten measurements in the opposite order, starting with browser E. I also rotated the five browsers between A, B, C, D and E for the five different web pages.

Before timing the page-loads, I shift+reloaded (or the equivalent ctrl+reload in IE) the web page ten times to saturate any network cache and to get the browser warmed up. I did this for each browser, before each set of ten measurements. (Ten reloads might sound excessive, but I started off doing only three, which turned out to be too little to reach the shortest load times.)

Between each page-load I cleared all browser data (cookies, cache, etc.). Except for Facebook, where I kept cookies and secure sessions to be able to time the Facebook home page when logged in.

To time the page-loads I used this Javascript page-load timer. As the Microsoft white-paper on testing browsers says, this could introduce an observer effect. But I think we can assume that the Javascript that is being executed is pretty simple and shouldn’t affect the times noticeably.

This test showed that Google Chrome 2 beta is not 100% stable. It hung twice (in 100 page loads) and produced load times of over 30 seconds. I decided to remove these values and replace them with new ones.

Results in detail

In these graphs, each bar shows the average of 20 page-loads. The error bars represent 95% confidence intervals.

Chart or graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta on youtube.com.

For youtube.com, Firefox and Chrome are tied for first. Safari and Internet Explorer are tied for third. Opera is last.

Chart or graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta on the Facebook home page.

The Facebook home page loads fastest in Firefox and Chrome, whose confidence intervals only just overlap. The other three browsers are significantly separated.

Perhaps it is the fairly Javascript-heavy nature of Facebook that makes it load so slowly in IE8?

Chart or graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta on msn.com.

Msn.com: Chrome and Internet Explorer are tied for first. Firefox and Safari are tied for third. Opera is last, again.

Chart or graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta on a Wikipedia article.

I decided to test the browsers on a long Wikipedia article with lots of images. I looked up Munich, which turned out to be a good candidate.

Chrome and Firefox are tied for first place. Safari is third, Opera fourth and IE fifth.

Chart or graph showing page-load times for Firefox 3.1 beta 3, Safari 4 beta, Internet Explorer 8, Opera 10 alpha and Chrome 2 beta on ebay.com.

Finally, ebay.com: Chrome, IE and Firefox are all tied for first place. Safari is tied with Firefox but slower than Chrome and IE. Opera is last.

Conclusions

Chrome sucks web pages off the Internet like an Electrolux. So does Firefox. In this test I haven’t managed to separate them significantly. As we all can see, Chrome has a lower average sum than Firefox, and perhaps with more data it would be possible to separate them statistically.

Opera is the slowest of the lot, which surprises me. Opera was also slowest in the start-up test. Perhaps though we should cut it some slack – it’s labelled alpha after all. Performance might improve when it reaches beta and final status. Opera also has a turbo feature in the works, but that is kind of cheating since it will lower image quality by tougher compression.

Obviously, this test could be made better in mainly two ways. I could test on more web sites, and I could do more page loads for each web site. But this test was, all in all, 500 timed page-loads and 500 non-timed page-loads. It took me more than a day to complete.

It’s also worth noting that this test is pretty much consistent with Betanews’ page load test, where Chrome 2 beta wins and Firefox 3.1 beta 3 is second.

This test was done with clean browser cache. I’m considering doing the same test but without clearing cache and cookies for each page load. After all, that’s how most page loads are done in the real world. A user who visits any of these five sites will most likely have been there many, many times before. I just need to figure out a good set-up for such a test.

Monday, March 30th 2009

Röyksopp – Junior

Röyksopp - Junior on Spotify

Röyksopp released their new album a few days ago, Junior. If you like Yello, you will probably like this. If you have Spotify installed you should be able to click this link to open the album.

Thursday, March 26th 2009

BetaNews Beat Me To It

I was writing yesterday about perhaps doing a page-load test with the latest browsers. Minutes after posting I saw that Betanews already did one.

To cut a long story short: Chrome 2 beta wins, Firefox 3.1 beta 3 is second, Safari 4 beta is third and Internet Explorer 8 is fourth.

Compare that with Microsoft’s own test where (surprisingly!) Internet Explorer 8 wins, basically testing the same sites as betanews.

I might try to do test all the same though, for fun. And because more tests give a more complete picture of reality.

Wednesday, March 25th 2009

Beta Browser Battle: Start-up Times

A few days ago I compared the four different releases of Firefox for start-up time (cold and warm) and page loading time. It got quite a lot of attention so today I decided to compare the five latest preview releases from the big five:

  • Firefox 3.1 beta 3
  • Safari 4 beta
  • Internet Explorer 8 (since there is no IE9 beta)
  • Opera 10 alpha
  • Chrome 2 beta

This time I did things a little more scientifically, following Justin’s suggestion in the comments. I made a batch file for each browser to print the exact time, then launch the browser, opening a page with a script showing the exact time again. The time difference equals the launching time.

It should be noted that this method requires me to opt out of Chrome’s default “new tab” page, with suggested sites. If this affects the results in any real way is unknown, but personally I doubt it. The new tab page in Chrome loads very quickly.

Cold start-ups (directly after booting your computer) are the ones that can feel like an eternity some times. For that reason I think it is more important to have a fast cold start-up than a relatively speaking fast warm one (which generally are about 5-10 times faster anyway). So let’s start with cold start-ups.

I did ten measurements for each browser. A fairly big sample size which gives tiny 99% confidence intervals, which are visible in the graph below.

Graph showing cold start-up times for Firefox 3.1 beta 3, Chrome 2 beta, Safari 4 beta, Opera 10 alpha and Internet Explorer 8.

IE8 is the winner here (2.40 secs), slightly faster than Chrome (2.66 secs). All browsers are, with a 99% probability, significantly different (none are tied). However, this comparison was done on my Windows (XP) computer so IE8 has an unfair advantage – who knows how large part of Internet Explorer is pre-loaded with the operating system? That makes Chrome’s performance all the more impressive.

Safari is marginally faster than Firefox (4.98 vs 5.19 secs). Surprisingly, Opera (7.14 secs) is roughly two seconds slower than Firefox and Safari. I actually thought it would be at least as fast.

Now let’s have a look at warm start-up times. I launched the browsers four times before starting the timing. Then I did 15 measurements for each browser.

Graph showing warm start-up times for Firefox 3.1 beta 3, Chrome 2 beta, Safari 4 beta, Opera 10 alpha and Internet Explorer 8.

Here, Chrome is in a league of its own with an average of 0.247 secs. Firefox and Opera are tied. They took on average 0.530 and 0.531 secs respectively, and their confidence intervals overlap. IE8 averaged 0.575 seconds and Safari came in last with 0.617 seconds.

Conclusion

Chrome impresses the most, even if IE8 launches slightly faster after reboot. Firefox and Safari are pretty similar, while Opera clearly is the slowest for cold start-ups.

These results really explain (and justify) Chrome’s good reputation for speed.

I’m curious if the differences are as large when it comes to page-loading. I’m planning on doing such a comparison too, I just need to work out a good solid method. So stay tuned if you like this kind of stuff.

Tuesday, March 24th 2009

Why Can't My Stats Always Look Like This?

Browser statistics by StatCounter

My Firefox benchmarking post has just been twittered by MozillaFirefox with 15,000 followers, and a few people at Mozilla.

Monday, March 23rd 2009

Firefox 3 Is Overtaking IE7 in Europe

Firefox 3 is quickly becoming the most popular browser in Europe. At the moment StatCounter shows it to be just a about half a percentage point below Internet Explorer 7.

Graph showing Firefox 3 and IE7 usage in Europe

Above is a graph showing what has happened since July 2008. Within one or a couple of weeks now, Firefox 3 should have passed IE7. This can be compared to North America where IE7 still has roughly twice as many users as Firefox 3. (Let’s not talk about Asia, where IE6 still is king.)

Saturday, March 21st 2009

Once More: Firefox 3 is Not Bloated

Just in from the insane-browser-prophecies department:

Despite the fact it’s not really ready for human consumption, Chrome has won. Firefox is already dead. The only way the situation can be altered is for Mozilla to slam on the brakes, lean out of the window of the truck, apologize for going the wrong way, and turn around. But that’s unimaginable.

Kier Thomas

*Snicker*

Well, that will only be true if all Firefox users migrate to Chrome. Why would they do that? Chrome does not provide any advantages that seem significant enough for a long-time Firefox user to switch.

If you consider how many Firefox users that have special extensions installed the above scenario seems even more unlikely. Even if Chrome did have equivalents for all the Firefox extensions, it doesn’t provide enough benefits to motivate the hassle of swapping out Firefox.

And what has Mozilla to apologise for? Ripping up Microsoft’s monopoly? Opening the doors for standards based coding?

Kier and others are making out that Firefox has become bloated and slow. So I decided to do a quick comparison of the different releases of Firefox on my four year old AMD 3200+ Windows XP PC.

I installed Firefox 1.0.8, Firefox 1.5.0.12, Firefox 2.0.0.20 and Firefox 3.0.7.

Installation files for Firefox 1, 1.5, 2 and 3.

I set up a clean profile for each version and set them to open a blank page by default. Then I made sure to close all other programs and launched each version of Firefox five times in a row, timing the launches with a stopwatch. (From hitting Enter to seeing the big white browser window.) And then I repeated the whole process so I got ten values for each browser.

I tested my reaction time which was 0.23 seconds on average and subtracted this from all the measured times.

So here is a visual comparison of the launch time for the different versions of Firefox since 1.0 (averages of ten measurements):

Start-up times for Firefox 1.0, 1.5, 2.0 and 3.0 on a Windows XP AMD 3200+ system.

Firefox 1 took just over 0.5 seconds on average. Firefox 3 takes 0.6 seconds. That’s a difference of 0.1 seconds on a four year old system. On a newer system the difference will be even smaller.

I also tested the cold launch times, by rebooting windows between each startup. These are averages of five start-ups:

Cold launch times for Firefox 1.0, 1.5, 2.0 and 3.0

Here the pattern is the opposite. Firefox 2 launches faster than 1.5 which launches faster than 1.0. Firefox 3 launches slightly slower than version 2 but is still a few seconds better than version 1 and 1.5.

I also tested average page loading times for DN.se, a fairly heavy page with plenty of Flash and images.

Average page load times for Firefox 1.0, 1.5, 2.0 and 3.0

Again I did five measurements for each version. Then I repeated the procedure, just to make sure no version was being helped by network caching. (I also loaded the page a few times before starting the test to make sure no browser was disadvantaged by being first.)

Between each page load I cleared all history, browser cache and cookies. So these values should be pretty representative for cold page load times for pages with plenty of images and Flash.

Conclusion

So, Firefox 3 takes roughly 20 percent longer to launch than Firefox 1 for warm starts, which equals at most one or a few tenths of a second. For cold starts (first start after booting your computer) Firefox 3 launches about 30 percent faster than Firefox 1. Also, each page load in Firefox 3 is probably saving you several seconds compared to Firefox 1.

We already know that Firefox’s memory consumption has gone down a whole lot and Javascript speed was improved by a factor of 3 or 4 for version 3.

Bloated?

So could someone explain to me how Firefox 3 is bloated? Is Firefox bloated because it lets you find visited pages easily from the location bar? Is it bloated because it has an industry leading automatic update system? Because it lets you rearrange tabs as you like, because it passes the Acid2 test or because it can remember your tabs from session to session?

Yes, Firefox has added many features since version 1.0. But it just hasn’t gotten bloated in the sense of unnecessary features that get in the user’s way.

Quite the opposite is true in fact. The Firefox developers have thoughtfully added many capabilities to Firefox without forcing mums and grannies to jump through hoops when they want to go on-line. At the same time they have made it load web pages much faster, and cold starts are much quicker. Warm starts are marginally slower.

Friday, March 20th 2009

Microsoft (Isn’t) Lying About IE8 CSS 2.1 Compliance

Microsoft boldly claims

We believe that IE8 has the first complete implementation of CSS 2.1 in the industry and it is fully compliant with the current CSS 2.1 test suite.

Jason Upton

Now, correct me if I’m wrong, but don’t the following CSS 2.1 tests look very much like fails in IE8 (and Firefox btw):

CSS 2.1 test suite test

CSS 2.1 test suite test

CSS 2.1 test suite test

CSS 2.1 test suite test

As I said, if I have misunderstood something, like for instance if the above tests aren’t considered part of the CSS 2.1 test suite, or if they for some reason should be ignored, do correct me. But if not, it seems like Microsoft are somewhat exaggerating IE8’s capabilities in the CSS 2.1 area.

Edit: Removed a few of the tests which turned out to be fails because I had ClearType activated.

Edit 2: Arron Eicholz of Microsoft has explained to me that the above fails are due to faulty tests. That would explain why Firefox also fails them.

Monday, March 16th 2009

Eurovision Song Contest 2009: Sweden

Finally, my favourite contestant won the Swedish final for the Eurovision Song Contest. Can’t remember when that happened last.

Obviously, I have to do my part to promote the song, so here goes.

Malena Ernman won with the song La Voix, which is half disco/techno/pop and half opera. A cool mix, and I hope it is memorable enough to do well in Moscow. At least better than Sweden has done the last couple of years. (18th place both 2007 and 2008.)

Here is her performance in the Swedish final, where I think she sang better than in the qualifying rounds. The powerful bits are less harsh or screamy if you will.

Thursday, March 12th 2009

Firefox 3.1 beta 3: Faster but Slower

A couple of days ago I tested how good the latest unreleased browsers from Opera, Microsoft, Mozilla, Apple and Google are at running Javascript.

Today Mozilla released Firefox 3.1 beta 3. So I gave it a spin on the Sunspider and V8 Javascript benchmarks.

I was obviously expecting it to do better than beta 2 in both tests, but that didn’t quite turn out to be the case. Below are averages from three runs of each combination of browser and benchmark, using a clean Firefox profile. (Not safe mode, since that will reduce Javascript speed in beta 3!)

Edit: I just added the data from my previous test for the non-Firefox browsers, for comparison. I also made a nice new graph.

To make the two benchmarks more comparable I’ve normalized the scores (inverting the time data from Sunspider, turning it into speed). The winner of each benchmark is given 100 and the others are given relative scores.

Graph comparing Javascript performance for Mozilla Firefox 3.1 beta 2, Mozilla Firefox 3.1 beta 3, Opera 10 alpha, Internet Explorer 8 Release Candidate 1, Safari 4 beta and Chrome 1.0

And here is the raw data (averages for three runs):

Browser Sunspider
(lower is better)
V8
(higher is better)
Firefox 3.1 b2 1,737ms 133
Firefox 3.1 b3 2,150ms 201
Opera 10 a 6,188ms 137
IE 8 rc1 7,994ms 47
Safari 4 b 1,633ms 1,056
Chrome 1.0 1,761ms 1,136

That’s pretty weird stuff. Beta 3 runs the V8 benchmark 51% faster than beta 2. But it chugs through the Sunspider benchmark 19% slower. (Or taking 24% longer, if you wish … 2150/1737 = 1.238 and 1/1.238 = 1-0.192)

Does anyone happen to know why this is the case? Was this a strategic move from the Firefox developers? I mean, did they introduce this slight performance regression knowingly, intending to fix it for final release?

Thursday, March 12th 2009

Fun with Google

Google Suggest can be pretty funny …

screenshot of Google suggesting funny options

Saturday, March 7th 2009

Safari and Chrome are JavaScript Speedfreaks

I decided to have a go at comparing JavaScript execution speed in some upcoming browsers. Actually, that’s just almost true. The version of Chrome that I tested was 1.0 which obviously is a final release. There is no newer alpha or beta release from them at the moment. (As far as I can see.)

The tested browsers are (all on Windows XP):

  • Internet Explorer 8 Release Candidate 1
  • Firefox 3.1 beta 2 (which will become 3.5)
  • Safari 4 beta
  • Opera 10 alpha
  • Chrome 1.0.154.48

I tested the browsers with two different benchmark suites. Google’s V8 and Apple’s SunSpider.

So, lets get straight to the results:

Graph showing results for pre-release versions of Opera, Firefox, Internet Explorer, Safari and Chrome for the Google V8 JavaScript Benchmark.

As you can see, I have normalized all the results for both tests so that the winner of each test gets exactly 100. This is to make a comparison possible between the two benchmarks.

I ran all benchmarks three times in each browser to get more reliable averages.

Graph showing results for pre-release versions of Opera, Firefox, Internet Explorer, Safari and Chrome for the SunSpider JavaScript Benchmark.

Safari and Chrome are as tied as can be. They both get a total score of 193. Opera and Internet Explorer are both a lot slower than the two winners, with 38 and 24 respectively.

The interesting thing is that Firefox does so well with SunSpider (93), and so poorly with V8 (12). What is the big difference between these benchmarks? Not that I’d understand it if someone explained it to me …

Obviously there’s still room for improvement to Firefox’s JavaScript engine (currently known as TraceMonkey).

Wednesday, March 4th 2009

Statcounter releases global statistics

Statcounter have just launched a tool for graphing their global web traffic statistics.

Statcounter Global Screenshot

The interesting thing is that it puts Firefox at around 28 percent right now, while NetApplications measures Firefox to be roughly 21 percent.

I’m guessing that Statcounter to a greater extent is used by individuals on blogs and smaller websites than NetApplications measured sites. The personal sites get more visitors using their home computers where they to a higher degree use Firefox (than at work).

Statcounter says Internet Explorer has a mere 63 percent, which equals 2.25 IE users per Firefox user. Pretty cool how much has changed since 2004.

Slowly but surely.