News & Rumors

North America and LATAM BlackBerry services experiencing an outage [UPDATED]

News & Rumors

BlackBerry Always Gooder Than iPhone!

News & Rumors

Some EMEA networks compensating BlackBerry users following outage

News & Rumors

Listen to the RIM BlackBerry Service Press Conference replay

News & Rumors

CrackBerry Asks: What would you like to see, if anything, from RIM in terms of reparations for the BlackBerry service outage?

News & Rumors

RIM says Global BlackBerry Services have been fully restored

News & Rumors

Research In Motion Founder and Co-CEO Mike Lazaridis addresses BlackBerry service outage!

News & Rumors

CrackBerry Poll: BlackBerry Outage or BlackBerry Outrage? How ticked off are you over this disruption in BlackBerry services?

News & Rumors

BlackBerry Service Update - Letter from Robin Bienfait Chief Information Officer, RIM

Special Coverage

BlackBerry Service Update Press Conference Live Blog! [Now Ended]

News & Rumors

BlackBerry Service Update Press Conference [updated: Audio download available]

News & Rumors

Official BlackBerry Service Updates from Research In Motion

News & Rumors

BlackBerry services still down EMEA, North America, elsewhere... [updated]

News & Rumors

BlackBerry services in Europe, the Middle East and Africa screech to a halt, UK server to blame [updated]

News & Rumors

BIS outage across multiple North American Carriers

News & Rumors

Having Issues With Your BlackBerry Data Service Today?

News & Rumors

Another Data Outage For BlackBerry Users?

Annoyed by the BlackBerry outage? Read This!

Understanding Reliability
By Kevin Michaluk on 12 Oct 2011 10:29 am EDT
12
loading...
0
loading...
143
loading...

BlackBerry outages happen. They suck. But they happen. Those of us who have been longtime BlackBerry users have dealt with some major outages in the past. We're dealing with one now (I feel really bad for those who have been affected for literally days now). And the odds are this won't be the last BlackBerry service outage you endure, assuming of course you stick with BlackBerry. 

Some of you cope (and are currently coping) with the service interuption quite well. Others, like my friend BlackLion15 think enough is enough and that someone has to go over to Waterloo and slap those in charge at RIM. I usually try and take these kinds of things in stride, but on a day like today when iMessage becomes available in the Apple world, I have to admit the timing of this BlackBerry service outage couldn't be worse.

This outage got me thinking about BlackBerry reliability, which prompted me to re-read an article we posted here on CrackBerry back in 2008 about Understanding Reliability. Reading the article again today (posted below), most of the points ring true... especially that point about wishing RIM would do a better job of communicating what's going on to people. At least RIM is doing a slightly better job this time around, with updates being posted to their website. Though I do think in the three + years since this article was written that the competition has heated up and people's tolerance for service downtime is less than it once was. If you've never read the article before, take a few minutes now and read it again. And let's hope everybody affected by the outage has their service returned to normal soon. It's hard to get work done when I just keep glancing over at my 9900 hoping the red light will start flashing again.

Understanding Reliability

Written by Neil Sainsbury, originally posted on March 3rd, 2008

With all the recent BlackBerry network outages, I thought it might be a good time to sit back and reflect on how reliable the BlackBerry actually is, what a reasonable expectation of reliability should be, and also to address the criticism some people have that RIM isn’t doing enough to notify people of outages.

The first statement to be made when discussing reliability of modern day technology is an obvious one, but it needs to be said: “Nothing is 100% reliable.” Later I’ll be discussing in more detail why this is the case and also how this is exactly the trade-off that the market has had to make to bring you what you want at a price you can afford. For now however, it’s important to keep this little tidbit of information always in the back of your mind.

And really, the fact that technology isn’t 100% reliable should be obvious. At a past workplace, I had more downtime because the office air-conditioner was broken than I have because the BlackBerry network was down. I’ve spent countless hours sitting in gridlock on the highway because somebody’s car broke down and I’ve also probably spent days correcting problems caused by some software problem on my PC that shouldn’t have happened. Unreliability of technology is with us and its here to stay. Does that mean we should not strive to make technology more reliable? No. Does it mean we should resignedly give up when technology fails us? No. What is does mean is that we have to adjust our expectations, acknowledge that problems will happen and move on. Really, in terms of reliability, the BlackBerry is probably one of the most reliable pieces of technology in my life – no small feat, given the large number of independent components (device, carrier network, RIM NOCs, email servers) that comprise “the BlackBerry package.” I think if you were to do the math and work out uptime percentage, you would probably find the same.

So why is technology unreliable? Good question! It’s because we’re cheap. In general, markets are spectacularly good at giving people exactly what they want within fiscal constraints. The reason why technology isn’t 100% reliable is because we, as consumers, are not willing to pay the price for that level of reliability. There comes a point when a company is evaluating their plans to ensure reliability of services when things start getting a little bit ‘kooky’ from an outsider’s perspective. “When part A fails,” they say “we have three engineers stationed within 100KM who can be there in around an hour.” But what if the traffic is jammed? “We’ll have a standby helicopter that can get them there quicker.” Who’s the helicopter pilot? “Bob. He’s ace, but he is often out all night at the casino so we better get a backup pilot too.”

Don’t laugh. I’ve heard discussions like this.

The money involved in making all these provisions for outlandish circumstances is phenomenal but it can mean the difference between 99.9% (three nines) and 99.999% (five nines) reliability. In a competitive marketplace, a company that tries to provide the golden five nines is going to incur significant costs which they will ultimately pass on to the consumer. The company that doesn’t will be able to offer services cheaper for the end-user and in today’s markets tends to be the one that survives while the others go bankrupt.

At the end of the day, the old adage “you get what you pay for” rings true.

If there is one area however where I think you could level a valid complaint against RIM, its notifying people of outages. While you might not realize it, this is actually a PR problem. The question that RIM must ultimately answer when an outage occurs is “Do we actually want to tell people there was a problem?” It’s a tough question to answer too. Every time there is an outage there would be a fairly significant chunk of people who may not have been affected and would never even know. Do you risk telling them something bad and sending down false alarms? That could have terrible image consequences especially for a company that sells itself on reliability. It’s a balancing act and thus far RIM has erred on the side of “image preservation” preferring to release little, if any information. The idea being that if RIM was to make a lot of noise about an outage that would draw unwanted attention yet if they say nothing “everything might just blow over.”

I personally think RIM may now be finding that this approach is not the best and I have a little solution I humbly offer up for consideration. I think a nice solution would be to provide SMS notification to affected subscribers. The majority of these outages seem to be occurring at a software level at RIM’s NOC. So, technically they could build in independent tracking to determine if data requests from devices are being serviced. If a request & response (for emails, web browser data, etc.) does not make a full round trip in from carrier network back out to carrier network at least several consecutive times over a reasonable period of time, an SMS is instead sent to that person notifying them of a problem (simultaneously sounding warning claxons in “the war room” J). By doing this, you keep the individual up to date and also avoid sending out mass notifications which would inevitably reach people that may not even have a problem (bad PR).

I’m sure RIM has considered this approach too and may have even discredited it internally for some reason I’m not aware of but I hope that they could find a similar-in-concept approach to solve this notification problem rather than just do nothing.

In conclusion, next time the BlackBerry network goes down - and believe me, it will happen again - relax. Your BlackBerry has not simply suddenly turned in to a block of wood. It’s still a phone. It can still send SMSs. In today’s technologically bustling sprawl, there’s also a good chance you’ll find a PC with Internet access within 500 metres of where you are.

76 comments

BBThemes

sadly the `saying nothing` idea was a bad one. you can only get away with that when its something minor, when its affects entire continents, then much as apple demonstrated with antennagate, being proactive and vocal is the right way.

considering this was written 3 years ago, id suggest EVERY RIM exec should attend kevins devcon speech on how to deal with the media.

jwhelan

I pre-ordered iPhone on the 7th. This outage confirms I made the correct decision. I hope I can came back to blackberry in the future!

cjterminator

I really do not understand why people are comparing iPhone with Blackberry service. After all iPhone (read Apple) itself does not provide any communication services, in each case it is the individual operator's service which you will choose. Every operator (in USA) I can think of had outage once in a while be it Verizon, AT&T, Sprint or T-mobile. People should be comparing services here. Now that Apple is jumping into iCloud service, we will see how reliable that comes to be.
Nothing is 100% in this world.
At the end of it, I completely agree it can be very frustrating to not have services properly for days.

Laaz

They do have Apple Push Notification Service, not to be compared though, but they do have communication services.

dupuy

As a MobileMe subscriber, I can tell you that the Apple iCloud stuff it is built on also has some serious outages (in fact, it's having a lot of them right now).

The important point this article missed is that whether you plan for three-nines or five-nines, you also have to make sure that the system fails "gracefully" and avoid single points of failure.

Right now, my BlackBerry Pearl 3G is *entirely useless* because I (foolishly, but who knew?) rebooted it after an app upgrade this morning, and it is stuck on the orange AT&T screen waiting for some sort of "okay to proceed" from the BlackBerry network. It won't even work as a phone?!?!

One of the big problems with the BlackBerry architecture is the excessive dependence on RIM servers to provide application-level connectivity. Apple's move to iCloud is a win for users when it works (no need to sync with your Mac/PC any more) but will also suffer from this kind of problem (can you run your Angry Birds app if it can't pull your old high scores off the cloud?). Still, I suspect that Apple will never go quite as far as where BlackBerry already is.

Skezza

This is it for me, only half way through my contract, but ordered an Android replacement yesterday, love the build quality of BB, but don't like the outdated OS and how slow they run

So its Bye Bye BB

pfluger

QNX will fix this all

ps4rh

Why? Are they going to throw BIS under a bus and replace it with POP/IMAP/EAS because they can't get the classic BlackBerry environment to work?

SBMobile3

Ahhh, BIS & BES don't work with QNX at the moment. How do you propose this will happen?

kbaggs

On many points I agree with you. But when I check both @blackberry and @blackberryhelp and see that they haven't posted an update in 17 hours. That is just poor management of customer service.

shebmiller

Hmmm I'm not too worried but I do agree with many that this situation and they're way of handling it is hurting their brand; which is already pretty hurt as it is.

djdonnyk

Thanks for posting the article, it was very imformative and I agree with the contents. It's true that we as consumers think we are paying alot of money for our BB service but we just have to chill when it comes to outages. As stated above I'm sure other smartphones have outage issues as well.

jamilliiee

Very dumb article. How does this fix anything. Im soo glad i pre ordered an iPhone. After all every one knows the iPhone is a better phone, it is only for the sake of BBM that many people still have blackberrys. But its rather unfortunate that Imessenger arrives in a bit. I'd like to see what RIM's market will be like next year this time...! Shame on RIM.

shebmiller

You assume too much. I hope you're truely happy with your iPhone; but it is NOT a better phone. A better app launcher maybe, but PHONE? No; and that's no assumption.

SBMobile3

Name "3" things that "actually" make BB's better than iPhone's! I think buyers have spoken on this topic in the last 4 years. iOS users have gone up to 220M+, while BB users have either fallen or have stagnated. Since 1998, RIM has ONLY accumulated 70M+ users. That's right! The BIG, AMAZING RIM took over 12 years to gain 25% of Apple's user-base. That's pathetic!

zoryad

Go and buy your iPhon and become another victim of commercial. You will remember that you wrote when you will trying to dictate complaints to the Apple support.

shebmiller

So, using the same argument, McDonalds is clearly the best source for food out there, right?

but since you asked...

1. I can sync/transfer without going through iTunes/iCloud
2. I can create as many shortcuts as the keyboard offers to increase productivity and decrease my time on the device. Get in, do what I have to do, get out.
3. Sure, not currently, but I get my emails faster than anything on my BB.

(just a few more)
4. I don't have to buy a case to make my antenna functional
5. I can remove/replace my battery if ever it dies and I can't recharge it right away.
6. I know my information is secure and I don't have to worry that my phone is gathering location data without my "permission"

Don't bother replying, I wont be wasting any more of my time on you.

crwblyth

Absolute win of a comment. +1

kidmac13

+1

And something i rarely see as a "plus" for BB's. word substitution. i text, email, and all social network, ALOT. and that comes in ridiculously handy.

knightfall

1. Physical keyboard
2. Variety of devices
3. Security
4. Canadian (But that one is why -I- like BB more than other brands.

t bizkit

I do not agree with this post. When I think of Blackberry, unfortunately the word "unreliable" comes to mind first. I am not the only one that thinks this. Look at their plummeting stocks... I unfortunately do not think we will see the light of day of QNX Smartphones if Rim doesn't act fast enough.

walnuts1024

I dont know how you can call BB unreliable when outages like this are so rare. If you want to call the company unreliable with regards to promises made then go for it. Most trust that their BB will get them the information that they need.

SBMobile3

This is the fifth outage this year, alone! I guess in your world, bad news is good news! lol! Keep drinking that RIM-kool-aid! Must taste amazing! SMH

shebmiller

Why are you even here commenting?

menaknow

"Look at their plummeting stocks..."

I don't think stock price indicates how a company is operating. Just more what analyst and market folks think.

Look at Microsoft. They actually have some great products, Windows 7, Exchange etc.

And everyone knows they make money...

But where is their stock price? Yep, in the 20's...

doug1010

I'm still wondering about hacker shenanigans, relating back to the london riots in august. i wonder if they'd say something if this is the case.

danimalchil

This is just the worst possible time for this to be happening. I'm watching CNBC right now and they are laughing at RIM. Damn.

shebmiller

Haters gotta hate, I'm just keeping a positive attidute and hoping for the best next week.

Oh adnwe do have SOME communication...

http://www.rim.com/newsroom/service-update.shtml

BBThemes

haha communicating with your customers 6 times in 3 days worldwide wouldnt be something id be shouting from the rooftops about

shebmiller

That's why I'm quietly whispering it from the corner of the 3rd basement of an underground complexe...

Akainzo

When the power outage happened in New York, how many people changed city because of the incident?

jamilliiee

Thats a dumb statement though.. why would you make such a comparison ? C'mon son.

Akainzo

I can grandpa, when people can make statements of jumping ship just because of an outage.

Or was it not a power outage?

WillieLee

What a bunch of children. Go ahead and switch to another device but if you think they aren't going to witness service failures you are being delusional. As more services are provided in the "cloud" you will see more outages. It has already happened, Google has had several Gmail problems, carriers see outages several times a year, cable stations as well. Amazon's EC2 went down but was that declared the end of the world?

They had a major hardware failure, that happens. And considering that the regular massive amount of data is now being piled on by "OMG! Is this working?" messages it will just take longer for the system to return to normal.

shankeith

+111

and what are these people hoping to switch to..."o screw bbm, it never works...i'll switch to android which has.....txting?"

SBMobile3

Show me a link where any of Apple's iOS services didn't work at any given time!

mikeconnors81

From September 30th of this year........Seems as though Apple iOS services went down but maybe you missed that one.....

http://www.pcmag.com/article2/0,2817,2393914,00.asp#fbid=FS5pTmndh6X

walnuts1024

This is unfortunately bad timing for RIM. Everyone that I work with that has gone from BB to iphone in the business environment absolutely hates it. They have fun with it for a few days. Then they spend the next few trying to find ways to make it function like a BB.

Many have either gone back to BB or moved to the android platform. So before people start jumping ship do to a systems (which happens to ALL systems) remember no device will give you the exact functionality that you have with BB so choose your replacement carefully.

Skezza

Blackberrys are better phones, Android is a better OS, I think if they updated to say BB OS 15 tomorrow they might catch up, of course they need phones with decent speed processor

jamilliiee

Outages will occur, yes!. but for 3 days straight ?!? come on now. thats just something to complain about.

daveshiz

Agree with a few comments on here but come on 3 days here in uk !! Even the most loyal bb fan would admit that's bloody shocking !!

twinmomma416

Sure. Outages occur. If you own a Blackberry. When I finally gave up the ghost and moved to iPhone, the only outages that occur are the ones when I'm trying to communicate with Blackberry owners.

shebmiller

What? How does that even work? If you're on iOS and you're communicating with someone on a BB then you're txting or calling them.

twinmomma416

Not if you're using an internet based messaging system like whatsapp.

theKHMERboy

Man it seems like the blackberry server has been down a lot these days.. Been happening 3 times over the past few month.. Making me think twice about rim service and how reliable they really are

El Platanero

There was no statement where it was an optimistic outlook. It was all about being realistic. I have yet to earn a device that didn't give me some kind of issue. If you guys are so jumpy that you would jump ship over an outage then I suggest you check your priorities

How many of you switched companies when there was a power outage, moved out of state because a power line went out, switched wives because yours got sick for three days or switched dogs because yours bit a burgler but didn't finish the hob?

I understand how some are being affected by this and weather you switch or not does not consern me but I will say that a bunch of you sound like people that live around excuses on which if everything is not perfect around you it is someone else's fault. No offense but grow up. Blackberry, android or apple...it doesn't matter...just grow up.

jl1884

im pretty annoyed by the fact that it seems like RIM cannot get its shit in order, i am a huge blackberry fan but my patients is running pretty dry. Get it together and come out with a great phone and service w/ tons of apps!!! if things arent better by the time iphone 5 comes out i will jump ship to where it now seems the grass is def greener!!!!

robertdusa

I think we're way under the 99.9% uptime mark at this point.

IMHO, the argument to accept downtime presented in this article is no longer (if it ever was) relevant, at least not when it comes to BlackBerry. RIM isn't just selling us a phone and then kicking in BBM and the rest for craps and giggles. They charge us for the service, maybe not a lot, but they still charge us. RIM markets the service as a major (if not the only one) reason for buying the phone, and subscribing to said service.

Most importantly, RIM relies on that subscription fee to stay alive as a company. Without it, they can't keep the lights on.

When your core business is built around a service and a subscription fee that you can't survive without, then you can't afford a service outage that's longer than the market or your customers can bear. What does that mean? Simply, it means if your company is besieged daily by bad press, sliding stock value, and competitors with "better" product offerings, your customers and the market can't (or won't) bear that outage for long.

My two cents.

newshutr

Just wait until the new iCloud goes belly up...and things are lost. Probably not forever..but enough for people to whine like what people are doing now.. Enjoy that new iPhone smell....cause it's going to smell like fail once in a while. And that is a given.

jzvi12

That means a iphone 4s loaded with tons of apps =)

Sent on the Sprint® Now Network from my BlackBerry®

gemini9109

It's weird how everyone is bashing RIM about it's unreliability & other things when BBM, a RIM program, was the ONLY wireless communication service still working when the earthquake happened in August . Outages or not, NO phone is better than the other . I still have a lot of faith in RIM, & I'm damn happy that I bought one 2 years ago . I'm sick of reading/hearing the comparisons b/c the petty debates remind me of middle school . If you're going to switch to an iPhone, Android or even two tin cans connected together by a string please do it in a mature manner, & stop posting how superior you THINK your Apple, Google, etc. product is on a RIM-oriented site !

br14

Well pity help RIM if Neil Sainsbury still feels the same way.

Despite what Mr Sainsbury says, reliability to 5 9's is quite possible within the budget RIM has for their NOC's etc.

The problem is one of design, and having the balls to make big decisions. At one time RIM made those big decisions, but in those days the impact was never as great. Now they have no one who apparently will take a risk.

RIM complained that a backup switch failed. I find it hard to believe that they didn't have a backup for the backup. In systems as mission critical as those RIM's NOC supports, they should have 2 and possible 3 layers of redundancy.

As for the delays, I know better than most what happens when you shut down a service handling RIM's volume for several days - like traffic on the freeway when an accident is cleared, the hold up can still last for hours. However, surely they could build capacity into the system (perhaps again based on redundancy) to handle the potential backed up data, engaging the backup servers to assist in clearing load.

RIM is testing customer loyalty to the extreme. Heads should roll at RIM for this shambles. I nominate their CIO as a useful start.

MrTherapy

this iPhone vs BB verbal tirade is actually entertaining as I have nothing else to do since my BB is down!

MrTherapy

Any Android or Windows lovers wanna step into the war? How much is Apple and RIM paying you guys to argue over whose phone is better?!?! LOLOLOL

ffejie

This year, RIM is trending a lot worse than 99.9% uptime. Over half of the world is out right now, as far as I can tell. It might be closer to 75% of Blackberry users. No BIS, no BES, no BBM. Basically, well over 80% of the common features are turned off.

99.9% uptime means you're allowed 525 minutes of downtime per year per user. That's a lot of minutes. However, they've blown way past that with this one outage. In fact, they're approaching 99% uptime for the year if it stays out through the rest of today.

This is completely unacceptable. RIM is a service provider and must be held to the same standards to which an AT&T, Verizon or Sprint is held: five-9s. Yes, we need to get to 3 right now, but the target should be 5. Does Google go down for 3 days? Not even close. That's because they're on a pay per use model. If Google (search) goes down for 1 hour, how many millions of searches do they lose? How many people start switching the default to Bing? What about Facebook? Same thing. It's unacceptable that a company with the resources of RIM can't keep the one thing that makes them money (the BlackBerry network) operational even close to three-9s availability.

empressdoinalot

As much as I would love to say I'm going to get and iPhone or an Android I don't think I can do that. I've been on BlackBerry for 2 years now and this is the 2nd time its happened. The first due to my own network having problems and not BlackBerry. I feel sorry for the business contacts because they're use to receiving their emails and all that. I'm just a general user and as much as I love BBM I can do without, its only been 2 days and I'm not dying! Maybe cause I have a life?! My £5 is going a little to waste and my 2 week old upgraded curve 9300 is kind of going to waste but hey just ride out the storm. Its technology can't expect it to function all the time.

And to all those going to get iPhone's... Good luck to you! There new service may have come out but you're going to deal with problems. I had an Android on Erricsson Xperia Mini Pro and I got rid of it after a month because I just love BB too much.

However what I'm finding annoying is those idiots who send me those broadcasts 'to get your BBM working pass on this broadcast with this number'... Piss off and quit wasting my time!

chicofontaine

Only difference between now and when that article was written is that some devices, including my own are completely useless at the moment.

With that said this and a few other little things have piqued my interest in switching to android, but their phones don't offer anything I want. I've been on blackberry for about 4 yrs now and I really do like it. I just hope they put out a phone that will wow me and make me happy like my 9700 did when that came out. I thought that would be the 9900 but the lack of UMA and autofocus on the camera are 2 huge turnoffs for me personally.

tc.cararo

ahhhh ordered my iPhone 4s today. Had nothing to do with the outage, but not at all surprised by this.

MaxPayne79

the problem is that it's NOT 99.9 percent. For me since july it's been more like 80 percent. once a month there are two or three days where i have no service. doesn't matter if i have the torch or the 9780, i simply have no bis. and it's really frustrating because without bis, the blackberry is essentially useless.

mrpinkbullets

1. Can someone explain to me, in relatively simple terms, the technical (hardware/software/mechanical) reason why we experience major BIS outages every six months or so while the same thing doesn't seem to happen to iOS/Android users?

2. With most large corporations and North American federal, state, and local governments relying exclusively on BB for mobile communications, how do they put up with these outages and why don't they raise a bigger stink? If it's such a pain for me not to receive my own (fairly unimportant) emails, I have to assume it would be much more serious if policymakers, law enforcement officers, and government officials were affected by limited communications.

3. What are the technical (hardware/software/mechanical) differences between BIS and the regular data networks for iOS/Android/everyone else? What are the differences to the user? What are the benefits of BIS over regular data?

kidmac13

i may be blined by my phone, or RIM, or the fact that i am VERY productive with it, but it will take more than an outage(that doesnt actually make my phone completely useless) to change from my Berry. but over here in Australia ive only ever experienced one of these outages and that was 3 years ago.

scootnyinzer

I still got my Facebook notifications and my wife and I still chatted and flirted during the day. Made my calls and still saw emails. Wednesday afternoon I got a bunch of junkmail all at once and (maybe this is my imagination) I think my battery life was noticeably better.

Not a catastrophe.

98hundred

BIS is just too good of a package to just let it go. Any IT company can expect problems with the physical side of the runnings. I woke up just now - 7:00(gmt+2), South Africa time.and I noticed my services are back, twitter and fb refused ro update but everything else seems to be back up but slow

Fruitydc ri

Having owned few BB; since beginning of time, I've to say BB is overly heavy, yet doesn't perform accordingly. As an IT professional we get many many many complains from VIPs whose emails they cannot receive.. for days! How do we explain to them!!??
And it happens till today, 2014! I still own 1 9900 black crap berry, and the network drops for no reason many many times a day (happens to all my older models too). Blackberry welcome to the douche club and I sincerely hope Blackberry close down too, very soon as it's unfair to consumers and buyers to have such a bad bad device! Your time's up!

Fruitydc ri

Oh yes, we ordered many Blackberries for our major corporation and many of these new BB's came DOA, DEAD! How's that for reliability?