also known as... I Tried To Do a Comprehensive BlackBerry Bold vs. iPhone 3G Web Browser Download Speed Test, but After Four Days Of Trying I'm Still Trying.
It's been a week since the BlackBerry Bold was released by Rogers in Canada. A lack of in-store availability on launch day meant I had to wait until day two to actually buy one, but buy one I did and I unboxed it on video with nearly uncontrollable excitement (can you blame me?!). I followed up soon after with another video showing off the device's media capabilities via the photos, movies, music and ringtones that come preloaded on the Bold.
With those two clips out of the way, it was time to move onto my next task - the BlackBerry Bold browser overview and Bold vs. iPhone 3G browser speed test showdown. I know some of you are getting sick of the Bold vs. iPhone which is better/comparison stuff, but there are a LOT of people out there who want to see this browser speed test happen (including the editors of some major tech sites that have been urging me to get this showdown done right). It was only a few minutes after I set out to begin the definitive Definitive DEFINITIVE Bold/iPhone browser speed test showdown that the frustrations that have consumed the past four days of my life began...
So here was my master plan. Visit eight popular websites on a commercially purchased, full release BlackBerry Bold and iPhone 3G under the four basic browsing conditions used by most people...
Websites to be visited:
Browser Conditions for Speed Test:
All told each phone would be timed over 32 website visits and compared under the same browser settings/conditions. The phones would not be "raced" side by side so as to eliminate variables (wouldn't want them competing for the same WiFi signals now would we?!). Rather, they would be individually time trialed. The timing device would be the stopwatch on my BlackBerry KickStart (err Pearl) 8220. Cache would be cleared, the appropriate settings would be picked per the four browsing states listed above, and then the eight sites would be visited in a row (timing each site individually, short break between each visit as the new URL is typed in). Once testing was complete for the BlackBerry Bold, I'd put it down, pick up the iPhone 3G, and visit the same eight sites under that same browser setting state. Once testing on a setting state was complete, cache would be cleared, settings would be changed, and the tests would be done under the next condition. And so it would go until each phone was timed visiting eight sites, four times over, each time under a different state. Also, the tests would be carried out late at night so that the sites being visited wouldn't be getting updated or change while the speed tests were taking place.
The other issue I wanted to be consistent and within regular operating norms on was signal strength. Though I own an iPhone 3G, I don't use it much or know that much about it to check those settings, but on the BlackBerry I used the ALT + NMLL shortcut to switch my signal strength to numbers. Throughout the 3G tests it registered at -70. For the WiFi tests, I checked the connection strength (Options > WiFi Connections > Menu Key > Network Details) prior to each test and and it was always between -35 and -40 dBm. So though I don't know the numbers for the iPhone 3G, at least it was always used in the exact same location. Lastly, I decided that the radios would be left on during the WiFi tests so as to duplicate real life use. I think most people still want the ability to receive calls even when browsing via WiFi.
All in all, about as fair a trial as I could think up or conduct within my work environment with the tools and smartphones at hand.
Special Note Regarding the BlackBerry Bold's Web Browsers
Seemingly Innocent Browser Settings...that for me lead to TROUBLE!
When set to BlackBerry, if you visit a site like NYTimes.com or WSJ.com, their servers will detect the BlackBerry setting and serve you up a mobile-optimized site instead of the full html site. A nice touch, especially if content is your only concern, but for this test since we want to compare HTML sites the default settings wouldn't work. So for all of the browser tests, the Bold would be set to "Internet Explorer" (which I later would find out from a Rogers Rep and RIM Support Rep was the recommended selection for this kind of test).
Preamble to the SUPER DUPER TEST
So it was after midnight on Sunday when the rest of the blogosphere was hopefully sleeping (didn't want anybody posting stories and changing the sites to be visited) that I got set up to do the testing. Video camera ready, digital camera ready, notebook and pen ready, stopwatch ready, iPhone 3G and a shiny new BlackBerry Bold ready.
Before getting to the official time trials per the rules above, I figured I'd do some general testing and browsing so I'd have a sense of what was to come. Both phones had their WiFi turned off, and their browsers set to factory defaults (minus Browser Identification on the Bold set to Internet Explorer). I was pleased to see the Bold beat the iPhone for pulling up the first three sites I visited - CrackBerry.com, Engadget.com and Gizmodo.com. It was only faster by a few seconds, but it was faster, so I took this as hopeful sign of good things to come. The Bold's browser doesn't do quite as a good of job at rendering pages as the iPhone 3G's browser, but it's pretty darn close (I'd argue 95% as good). There are some sites, like NationalGeographic.com, where the Bold really messes up the tables, and it also seems to have something against the header on CrackBerry.com. Regardless, the Bold pulled up these sites about a bajillion times faster than they would have on my 8320 Curve on Edge and looked about a gajilion times better than the browser on OS4.2.2. A HUGE leap forward for RIM.
LET THE RACES & FRUSTRATIONS BEGIN
The Bold pulled up the majority of sites in the 29 to 40 second range. Not too bad I figured, considering some of the sites (like CrackBerry and theIphoneBlog are actually pretty beasty). The Bold seemed to be off to a good start.
When set on the same browser settings, in this round of speed trials the iPhone 3G definitely seemed to hold the upper hand. Round 1 goes to the iPhone 3G.
HOUSTON... errr.. WINNIPEG... we have a PROBLEM
After a while I got used to the Bold's Browser HATING me
At this point I'm thoroughly baffled. I try browsing websites over WiFi on the Bold for the better part of an hour. Unfortunately, that doesn't equal many sites. I eventually gave up. Something must be wrong. Images would begin to load, but then the blue status bar of death and hour glass would take over. It would take several tries of cache clearing and rebooting to get a site to load, and when it did, it was in the minutes. Either this phone is broken (yet everything else about the device seems to be in perfect working order) or the Bold hates me, or RIM released a device with some Gremlins living in the code, or maybe a bit of all three. 4am. Time for bed.
LET THE FRUSTRATIONS CONTINUE
Test Day 2 - Monday
I take care of my regular Monday morning CrackBerry duties then get back to the Bold. I do a few more tests. Same deal as the night before. I call Rogers for support. While the tech guy I got was super friendly and we examined the device and browser settings and options, everything looked good on my end. He thought maybe it was hardware problem and suggested returning the device since I just bought it a few days prior.
Test Day 3 - Tuesday
Please Please Please just let my first Bold have been a dud. I love the device. I just want it to work properly the way it should.
I call back in to RIM Technical. I get a really helpful guy who has a lot patience with me and the situation at hand. Juggling both Bolds back forth, we go through different speed tests under different settings. He's taking notes. He gives me a task list of more tests to try and he gets started on doing some testing of his own. I email him my results. He emails me back an hour or so later that he's escalated the testing on to their Research department and somebody from that team will get back to me.
At this point I get an email from one of CrackBerry.com's forum moderators telling me to check out a forum post. It seems a few members are beginning to wonder where I'm at... nothing new on the Bold, not too many blog posts from me this week, no new podcast.. so I post where I'm at in the forums along with a little RANT!
I give up for now. Two new BlackBerry Bolds that have been three years in the making that I've been waiting months for to get released and I can't even get done the little browser speed test I've been trying to complete since Sunday night. I'm going to bed.
Test Day 4 - Wednesday
I email back RIM technical support first thing in the morning looking for an update. They respond and want more questions:
make and model of the router I've been trying to use WiFi on
- the exact steps I have been going through in conducting my speed tests.
That was enough to get BBCool onboard to try and figure out what's going on here. I emailed Doug my spreadsheet with sites to visit and my timing data (plus we added two more great sites to the test - blackberrycool.com and quicklybored.com) and he picked up the gauntlet. I have heard grumblings that there could be some "geographic" issues at play here with the issues I'm experiencing, and while I don't understand why or how that would be the case, with Doug and Simon being in Ottawa we could at least maybe validate or throw out my results data.
Test Day 5 - Thursday
WHAT ARE YOUR RESULTS?
You've seen what I'm trying to do. How do your test results compare? Post them in the blog comments or on this dedicated forum thread. Or email it in to Kevin at CrackBerry.com. Are you experiencing any of the same issues? Are you trouble free? Let me know! Follow the methodology in this post with the same sites being visited. I'll follow up with a Call for Bold Browser Data blog post shortly that lays out the exact steps.
CONCLUSIONS FOR NOW
I'm sorry guys and gals. I failed you. I set out to put together an awesome test comparing the browsing speeds of the Bold and iPhone 3G and I couldn't get it done. The premise was there, but the equipment is not up to snuff yet to draw any definitive conclusions. I say that because I don't think this is a fair battle just yet. I may have bought the Rogers BlackBerry Bold (well, TWO of them) over the counter, but the Bolds I'm rocking are obviously still in training - not ready for the full-out browser speed test war just yet.
So here are my conclusions for NOW:
- there has to be some sort of issue with the Bold and WiFi/HotSpot Browser that pops up under specific circumstances. I don't know what is yet, but I'm sure there's something up. I know I'm a CrackBerry Addict and not a BlackBerry Expert, but I don't *think* it's user error on my part in this case.
- when the Bold browser is working it's pretty fast and works well. Not quite as fast as the iPhone's Safari browser, but fast. Word has it that RIM still has a lot of performance tweaking to do to extract more speed out of the browser and I believe it. There's no doubt the hardware is capable of sucking down the data quick, so this showdown may be pretty tight in a month or two as new code gets released. For now I'd be happy just having a native browser that worked as intended. I've got video footage of this whole debacle/massacre/mess or whatever you want to call, but I'm going to trash it in hopes a better day will be here soon. I'll turn the video recorder on once the Bolds' browsers are no longer buggered.
- if you're in the USA waiting for the Bold to come to AT&T, don't be mad, be GLAD. I'm
sure hope by the time you get the device RIM will have addressed these issues and you won't have to be frustrated like ME.
If you've read this far you are hardcore and truely a smartphone geek. Be proud of it. I've spent the better part of a week in frustration dealing with this. I think me and the Bolds are officially now on a break. Just for a bit. At least until this issue gets resolved and I actually know which one of the Bolds I'm going to keep as my daily driver, my SIM is going back in my Curve.
No 3G or Awesome Display or Fast Processor, but my Curve is Still Sexy
Good thing I just did the Colorware thing and got it painted white. I'm still attracted to her and have a lot of love... call me a BlackBerry Polygamist, it's the truth!!