When surveys go wrong ...

BlackBerry booth
By Chris Umiastowski on 13 Jul 2012 09:50 am EDT

Ok, so the folks at Baird Equity Research apparently did a study about developer sentiment on different platforms. AllThingsD has already written up another BlackBerry death certificate as a result of this survey.

Whatever happened to questioning the quality of these surveys?  I'm not going to try to pick apart this particular survey because I have no idea how it was conducted.

But I will say this - just look at the picture.  There's something obviously wrong with it.  The title says, "How do you view the long term outlook for the following platforms".

RIM Survey

The fact that Windows Phone 7 even shows up (ranking 4th) tells you there is a critical flaw somewhere in how this survey was executed.

The title asks for a ranking about a long term outlook.  Windows Phone 7 is in worse shape than BlackBerry 7 from this perspective. It's being replaced by Windows 8 later this year and has no long term future.  BlackBerry 7, at least, has a few years of life yet as it tackles the emerging markets (although I can see why devs wouldn't be very interested in it).  

This serves to remind everyone that a picture isn't always worth a thousand words.  Sometimes it's just plain useless.

To start analyzing the numbers of a poorly executed survey is akin to looking for a pattern in the coffee grinds that I throw into my trash can.

Reader comments

When surveys go wrong ...


If I were an average-level dev, it would be a daunting task to stand out and compete in the iOS or Android store. Developing for bb10 actually looks like an attractive proposition in that regard. Plus RIM is aware of the issues and is pouring resources into dev support.

As an average level dev I can assure you this is the case. It's easy to release a pretty good solitaire app for a less saturated market. Try releasing a solitaire app for iOS that stands out from the crowd. That being said as a platform you want the top devs, the average level devs will only get your platform so far.

+1 ppl need to start doing the math and crossing data, like that billions downloads crapple claims to have looks good alone but if you divide that by that also good looking number of 600000 apps you'll see a damn low number. So the questing is: how many downloads per app?


I am a standalone developer too.

It is a daunting task to compete in the iOS realm, as the competition is huge. Add to the fact that unless you are already an Apple developer, the initial cost to even start developing an iOS application by acquiring MacBook hardware is a disadvantage.

And to Chris' point, WP7 is already dead. If you attend WP conferences, they themselves are pushing WP8.

I agree. I'm going to put effort into BB and WP development. There are way too many apps in iOS and Android for me to make a dent - so it's not me worth putting the money/time into developing for those. This takes into account that no one knows who I am or have ever heard of my apps.

If they asked big developers such as Rovio, Gameloft or EA then the above chart closer to what you get. They already have a brand and people will be looking at what they are bringing to the table next.

As a small developer shop, unless I have the next big idea, BB and WP is where to go.

Therein likes the rub. Unless developers have the next big idea, they should question what the value proposition or differentiator of their app is. If devs just want to get an app out there, they might start coding apps they know they can code vs. coding apps that potential customers need or want. That might be fine if they're just trying to gain experience or what-not, but the value to the platform and its customers is questionable (other than jacking up the # of apps, which leads to positive press because of surveys like this).

Choice is good up to a point, but when alternatives no longer provide real value other than being an alternative, there's a diminishing return on the choice. With BB and WP, there definitely is a greater opportunity to be one of the first apps of a certain type on the platform. There's also a good chance that those platforms will appeal to, and be marketed to, certain target audiences, such as business users, which means devs can target more specific needs.

Yep, just another sign that this survey is complete garbage. Surveying on 200 people gives you an extremely small chance of your numbers being accurate.

Most of the time, surveys take a relatively small sample of the population to get a 'feel' for the opinions of the 'masses'. This is standard protocol. If you ever watch the news, and hear about these political polls, that's what they do as well. And it can quite literally drive the direction of general opinion. These are typically quite biased, and have little basis on reality. The larger the sample however, the more acurate the results depending on the questions asked. Ultimately, its just a shell game in most cases.

Your right to some degree; however, in order to determine if the "200" is representative of the total population of "developers"; then you would need to know the total number of people who are developers on the platforms in the article. Likewise, you would need to know whether or not the sample was selected randomly or was this a convenience sample. And lastly, you also need to report the total number of surveys sent so that the response rate is known.

Without this information, any survey statistics should be questioned.

Hmm sounds like an article I read last week. The author had gone to IO confrence & asked devs what their fav OS was...unsuprisingly it was Android. The author also asked what they thought about BB & found a way lower number of supportive replies. They tried to say that this showed that BB was dead & had no dev support.

That's like going into McDonalds & asking the cusomers in there if they like Big Macs. Then using thgosr responses to say that almost everyone likes Big Macs.

iOS and Android don't know anything about BB10, or WP7, hence the skewed results. Most "mobile" developers right now are ON ANDROID AND IOS, hence the results are what they are. I'd prefer those platforms too if I developed for them.

It's All Things D! They're so far up Apple's a$$, it's not even funny. The author of that article (John Paczkowski) only writes death articles on RIM. Have a look at his past articles for the past couple of years. I used to frequent that website until I had enough of it.

I do not really understand why BB10 is on that list. I mean we cannot make a good enough judgement on how futureproof BB10 will be when it hasn't even been released.
Also I completely agree that BBOS7 has atleast 2-3 years left in it for different markets. Far more potential than WP7. In fact BBOS7 should be 3rd seeing as how WP8 hasn't been released either and it is a start from 0 kind of OS. The whole survey seems to be something that was done for a high school project.

In statistics, we all know what happens when you ask a "specific" set of people what they "think". You will be clutching at straws if you call that a credible survey.

It hasn't been released and what is of Blackberry is crap compared to the competition that's why the bad mark up also probably you know little things like market share and surveys of how many people would actually buy a blackberry over an iPhone or Android which isn't a lot.

A comma every now and then does wonders...

It's the difference between "Let's go eat, Grandpa!" and "Let's go eat Grandpa!"

Engadget seems to be growing a touch more fair on RIM lately, and to my knowledge they are owned by AOL. However that's simply my opinion. :)

Let's not forget for a moment how they compare version of Windows Phone and BBOS, but fail to do the same with iOS and Android. If you don't start by comparing Apples to Apples how can you compare anything at all...

BGR already posted on the death of RIM developers too based on this survey. Good to see that bloggers actually right credible blog posts anymore. At least all of the bloggers on Crackberry seem to get the facts straight, whether they are good or bad.

I think we should start an online donation site so that when BB10 comes out and is a success, we have a huge online party to rub it into BGR and All things D's faces!

I can't stand those sites anymore. I just deleted them off my news feeds and twitter feed. They are absolute garbage.

I actually did the same, because i have a mind for myself an know crap when i see it. I am a huge BB Fan, nobody can tell me about another Platform, but i am also a clear thinker and identify crap when i see it. I will not defend RIM when they doing bullshit.

I also ask people to do something on your phone and i will show you I can do it too. I can make a phone call, i can launch an app, i can watch a video, i can post a facebook status, I can take pictures, i can view pictures, i can take HD Video, I can transfer file to my phone wirelessly, i can read and respond to emails, i can listen to music, i can instant messenger, i can browse the web, i can get turn by turn voice navigation, i have bb travel to manage my travels and so much more. So really what would be the benefit of getting iOS or Android when I can do all these things already on my 9900.

I will be buying 2 BB10 phones at launch. One for me and one for my wife. I support RIM/BlackBerry and if they have to close their doors, i would be the last one to put down my BlackBerry.

BlackBerry for LIFE, BlackBerry by CHOICE

I am pretty sure that I was one of the devs in this survey. And despite being a BlackBerry first developer, was honest about there not being much of a long term future for the classic BlackBerry OS. (Which is what happens when the company that makes the OS announces that they are going to do something new).

Chris is right about the 78M BlackBerry 7 users outlasting the 2M Windows Phone 7 users, but my guess is that most developers answering the survey have not actually developed for WinPhone7 or Windows8, and are unaware of the small differences.

Also, how is Bada not a 0.0% ?

Lets do a survey of just the BB Devs now and than again in December.

Would be interesting.

One thing to remember is RIM is spending big bucks and getting tons
of face time with the Devs.
This alone indicates they are concerned about having Devs and their applications
on board.
They would not have to do this if they did not have to.

Retaining customers and Devs, who have been abused by RIM for over 2 years is going
to be tough. No matter what a survey says.


Every company does this to get developers on board.

Microsoft is the most developer focus platform AFAIK. And they are spending big bucks to get facetime with developers with WP also. I recently was invited to a Nokia event to develop with WP.

I turned it down as BB10Jam was a week later and had to choose one event, blame it on the day job).

My point is, developers will go where money can be made and where you can exercise your creativity. With the new development model RIM has adopted, they actually are doing the right things.

And the fact that they are spending the money to show that is another good thing!

You can prove just about anything you want with "statistics" . . . too many factors contribute to an accurate result; factors that an outfit like the website in question probably aren't too careful about imho.

Just the same ol stuff...if it has the name "Blackberry" in it now people will bash whether they research or are educated or not. It's just the way it is until something is released. "Blackberry" is synonymous with "going out of business", "battery pull", "Old", "boring"....at least that's what I've been hearing from my friends. My coworker said yesterday that Blackberry 10 devices won't matter because it's the same "BBOS". You just can't force people to be fair and go do their own research if they don't want to. It's a shame though because it's the same people saying the same things to their friends. It'll be RIM's job to market and spread the word to quelch the negativity.

So these guys apparently did a selective survey; “Baird surveyed 200 developers culled from a sample set of 4,300”. So when you see this sample size given the potential developers in the world and I can’t seem to find any details on how they did the cull, I am reminded about a famous quote my statistics prof. used to repeat often:

"There are three kinds of lies: lies, damned lies and statistics." - Mark Twain

Good job Chris in debunking this crap...the people need to know what's what's. Team Blackberry need to set them straight!

I think I did this survey and remember thinking something like that. Android and iOS aren't divided by version numbers, but Windows Phone is (and BlackBerry is, but that's different because it actually is an entirely different platform). I bet most people read Windows Phone 7 and Windows 8 as the same since they'll be carrying their WP7 apps forward.

Yes, all WP7 apps will run on WP8, so from a developer standpoint WP7 is not dead because you will still be able to develope one app that will run on both. You can choose to build a native app that will only run on WP8. No BB7 apps will carry forward to BB10 so there is a big difference.

WP7 apps MUST be recompiled to run on WP8 because WP8 has more in common with Windows 8 than WP7. So... any devs thinking there will be no difference are in for a very rude awakening.

This is a portion of the reason why no WP7 phones will receive an update to WP8, only WP7.8.

BB10 can at least run Java and HTML5 apps, so if people are coding such apps for BB7, there will (if memory serves me correctly) be fewer changes that need to be done to bring the same app over from BB7 to BB10.

The vast majority of developers develop for either iOS or Android. That includes the majority of average and poor apps. What is important are the the high end apps. RIM needs to be sure that they have those high end apps on board.

The Baird study is certainly flawed but it's interesting that it's data almost perfectly mirrors Appcelerstor's data. I talk to Appcelerator about the dev survey it does with developers (in conjunction with IDC's research arm) almost every quarter (my questions to Appcelerator have even made their way into the survey from time to time) and I can say their survey is absolutely well done, especially for its target market (they survey smaller app developers and developers who build apps for their companies or on contract for others -- in other words, lots of enterprise types).

What we've been seeing for more than 2 years is a high developer interest in iOS (iPhone and iPad) and high interest in Android phones. Over the last year and a half, we've seen high interest in mobile-optimized HTML5 web apps (and that interest continues to grow) and interest but not a lot of motivation for Android tablets.

BlackBerry interest (and also Playbook interest) is on the decline, and the decline just over the last 12 months is worrisome -- you can't spin your way around it, interest is near-gone. Windows Phone interest is increasing and since WP8 will run the NT kernel, that interst should continue to grow.

The thing is, interest doesn't equal development -- but user numbers don't either. You want to know why more developers will spent money on developing a WP7 or 8 app over BlackBerry (7 or 10), it's multifaceted:

1. BBOS is seen as a dead-end. It has shit tools for building apps and the memory limitations make it hard to build a modern app for the platform. BB10 still has unfinished tools and devs are rightly worried about RIM's longevity. People got burned on webOS, they don't want to invest time and money in a platform with no future.

2. Even with the low userbase, Windows Phone has a huge advantage in Microsoft. Developers almost assume that at some point, it will have a large userbase. And Microsoft has the money to support the platform.

3. Microsoft has huge ties to enterprise -- far larger for software devs than RIM has ever had for mobile. That means that you have an army of developers who build and code apps for their business using Windows and Windows tools. Microsoft can leverage that in encouraging devs to target Windows Phone.

4. Microsoft is hands-down the best company in the world at developer relations. They treat devs like gold, have great tools and will pay to get key apps made. That permeates even outside of the enterprise. Now that porting to WP8 is going to be so much easier, expect apps to increase.

5. You only have so much time and money for so many targets. Even with app frameworks, supporting multiple platforms is expensive. iOS and Android are the two "must-have" platforms. For the rest, the goal is to see if they can build a mobile-optimized web app to support users on other platforms.

RIM should spend $100m on dev relations. They should have done it years ago. But they didn't. Now it's an uphill battle to convince people to build on a platform. This is why you see such a focus on web technologies. RIM knows it isn't going to be a must-have native platform.

I liked some of what I saw at BBJam 10, but the dev tools still suck and cascades have negligible value until more core APIs are opened up. I wish them luck.

For some reason I'm not sure if you've been paying attention to what's been going on with BB/RIM for the past five months.

I read something somehwere (I believe Boostmobile) had a statement that said 90% of all apps downloaded only get run about 10 times.

COFFEE GRINDS?!? what are you doing with coffee grinds in your garbage - K-Cups - check it out - as always nice work by you and the CB Team

Chris, I don't think you understand. WP7 is not being replaced with Windows8 and all apps have a path forward. WP7 is being replaced with WP8 and all apps written for WP7 will continue to work. Not to mention MS offers free development tools and Visual Studio that are both top notch in the industry. As a developer, it would be foolish not to download and mess around with them.

Conversly, BB7 is dead with no path forward, and BB10 does not exist yet.

Yes the apps written for WP7 will work with WP8. But not vice-versa. Especially when talking about high-speed processing data which WP7 doesn't support dual-core processor.

Let me put this into another way. The graph said WP7, WP7 also means WP8. Windows 8 and Windows Phone 8 are NOT the same. Not even Windows RT and WP8 are the same. They may have the same kernel but you can't immediately run one app on another platform without any code changes. I also believe that this is absolutely correct(I don't know how many people were surveyed as that could change how I feel) as Windows Phone has always had more developer interest than Blackberry. Nobody should even debate that. It's a Microsoft product. Microsoft is the corporate holy land for developers.

I also see that their app store is growing at a much faster pace than App World. Windows Phone started in 2010 while Blackberry was 2009 and Windows Phone has more apps and especially more apps when if you don't count e-books and other junk that Windows Phone doesn't have (I checked).

Blackberry's developer interest is going anywhere much for now. Even though I love how RIM got developer devices out for their next gen platform whilst Windows Phone 8/Microsoft has absolutely nothing yet. No developer phones, no emulators (at least not yet) and the whole works.

American finance, which is the real backbone of the tech blogging industry, is really just a bully in the media, but Canada has a habit of standing up to America when it's being a dick.

200 years later, this is the war of 1812 all over again.

We're going to burn your major corporations to the ground.

And because of this is why RIM (NASDAQ: RIMM -1.62%) is down today after two positive days? Damn, viruses spread like the air...


Nothing is more frustrating than the misuse of research design in order to proselytize an agenda. Without an Y-Axis label, it is impossible to tell what type of data is actually being reported here. Is this simply a count? Is this the average that is being reported? It's ambiguous as currently presented.

Presumably, it's a count considering that the question being asked is nominal in nature; that is, it is nominal-type data. But if that is the case, there should not be the words "poor" or "excellent" in the Y-Axis. If the rating scale was, for example, "poor", "neutral", "good", "excellent"; then you would never construct your graph in the manner that is shown - it is misleading and inaccurate.

On the other hand, if you are attempting to construct this question so that it generates interval-type data, i.e., a rating scale, that would take into account "poor" and "excellent" on the Y-axis, but then fails to explain why a 10-point scale was used in which a true neutral point is impossible. A 5-point Lykert scale should have been used (see Cozby, 2012 for an excellent review of test construction rules).

The problem with these types of so-called surveys and results is that they fail to include the entire data set, including the specific questions and rating scales being used, so that the public can scrutinize it in order to determine whether or not what is being reported is truly accurate.

i just want more choices, choosing between android and apple isnt really a choice...

i want to look at my options and make a choice based on whats in front of me, right now between the two choices i have, android wins, though when bb10 is released i will look into its performance, apps, specs, features, yes how it compares to my other two choices

BB is Bold for not giving up...

Apps launched to fanfare on BB10 can be ported to Android and iOS. Apps launched on iOS and Android either cost a whole lot of up front cash to publicize them, or they are discovered and word of mouth has to build before they rank above 400,000 i.e. no one is even going to know your app exists unless they are full-time paid "Google Play" or "App Store" analysts.

Launching a new app on BB10 makes a lot more sense than trying to launch on Android or iOS right now, because if you have a winner, people will sit up and take notice, and we all know that the BB10 launch is going to be VERY CLOSELY WATCHED by the media at large, and if you have a good app, you'll get coverage, plain and simple... and you'll have iFans and Andrones waiting for YOUR app to be ported to THEIR platform... just make sure your key concepts are patent-protected.

There is no way a small time, indie developer, or one-man shop can make an app that provides any sort of return on investment in terms of time, resources or upside on iOs or Android.

BlackBerry 10 & Playbook is the way to go. Although I would agree, that big time development houses make a ton more money simply because of name recognition.

There is a separate circle in the Hell for those who make bar charts not staring from zero. On that chart it looks like BB10 bar is 6 times shorter than iOS bar while in reality it's just 2.5 times shorter. Of course there are also numbers there clearly showing that but the main perception is made by the chart (that's why the chart is there instead of just plain numbers, right?). Here is the corrected one: http://farm8.staticflickr.com/7252/7562486494_21985c3da6_z.jpg.

I still cant believe that people still buy into servays and stats as a whole. Although states are more accurate servays have always and always will be biased and flawed. People got to realize that servays are a real crappy way to determine anything other then hear say. Lets wait and see how BB10 really does in the app department and then we can have our opinions if its a seccuss or failure. Till then all we can do is wait and see for ourselves and not what stupid media tells us.

Blech, I think the data is skewed. There are lots of devs for the BB platform, I think it is doing better in marketshare than WP 7 and certainly Windows 8, I don't know why people are so quick to bury BlackBerry. App World has over 3 billion downloads! and the competition is far less fierce. Plus, Android market is filled with free apps, making it impossible for devs to get paid well. I hear BlackBerry apps make the most money on average, even above iOS apps!