Research In Motion Founder and Co-CEO Mike Lazaridis addresses BlackBerry service outage!

By Kevin Michaluk on 13 Oct 2011 06:57 am EDT
5
loading...
0
loading...
109
loading...

An update from RIM's Founder and Co CEO Mike Lazaridis. Would write more here but gotta jump in the car to head out for an early morning radio interview on all this BlackBerry outage craziness.... watch the video and leave a comment!

RIM is holding a service update press conference live at 10AM EST - listen along with us on our live podcast.

Reader comments

Research In Motion Founder and Co-CEO Mike Lazaridis addresses BlackBerry service outage!

136 Comments

36 hours without the service, millions of people affected and you think they did a good work? Damn fanboy, you totally deserve this Blackberry mediocre service.

Every company that operates on this scale experiences service interruptions .. even Apple. And let's remember, this is only email and messaging we're talking about .. it's not life and death as there are other avenues to get your messages. Granted this was a long outage, but they've had a pretty stellar record otherwise. There is no reason to sling the fanboy label at him for his reasonable comment.

True, that was just me being frustrated, Ive counted on their reliability for years now and had a very rough day without the service, noted and I take it back.

Good morning! Up and running too @ South Florida!!!

I've tried contacting thru BBM to a few customers overseas and it seems DR is up but Venezuela still down?

Mmm, I understand the approach he took on that message, but I think a much more confident approach would be better (without beigh cocky of course), I dont want a RIM CEO that inspires pitty when you see him, I know he has to be humble about the problem but he needs to inspire confidence that he knows what he is doing... bad bad approach he took there... I mean, it's not like he is publically admitting he cheated on his wife like the US presidents do...

I live in UK and today my BB seems has been fully funtioning all day. Much improved from yesterday. Still the odd stray email coming through but no real worries.

BBM just started working again for me here in Germany... Finally after 3 days... phew! Kind of shows me how dependent I am already. :s

My 2 BBs are working again just like before. No lost emails at all. Still a Black berry fan and sticking with it. At least I didnt' loose any email at all. That's another important thing about BB.

Nice to see the right person speaking. It felt sincere to me. Would like to hear the final update and all of the details and what they will do to prevent further outages.

there were probably "sad" cues on the TelePrompTer. he's just reading that shit, some guy in "Damage Control" wrote it. ;)

And how does one come up with "sad cues" statements? Is your life so miserable that you have to come here and B****? ;)

Good job boys! git'er dun!

The CEO comes say sorry for an issue and people are still not pleased.

It seems not even Jesus and the profets would be able to save RIM from haters by now.

Just watched a news story about the outage. Then they talked about "switching fruits". Guess they can't take the time to learn that Apple's servers messed up... Gotta love the media...

Gotta laugh that he was talking about estimates I'm glad he didn't give one.

Go RIM. Gotta love 99% uptime :)

+1
I rather stick with a company that can say, yes it's our fault and we're fixing it instead of "you're holding it wrong".

YO! In South Africa, one of the first countries affected, and STILL not up and running. Fix our first FFS! BB makes up for 70% of our smart phone "population"....that's a shitload of people without phone service.

Apology not accepted.

Hi Liza,

Your BlackBerry was, and still is unable to make phone calls? I don't recall hearing that phone and SMS was affected, as those services don't rely on data.

Liza, you're probably mistaking your phone being suspended for lack of payment or your $5.00 prepaid card has expired. Email and BBM were the only issues.

LOL. Glad to see someone did. One of the questions that was asked during yesterday's conference call was whether or not RIM felt they could do more to communicate with the public about the service outages. This is a good sign. Services back up to normal for Boston, Massachusetts.

all good here in the Philippines. now bring on O.S. 2 so i can think about restoring my trust in you!

RIM should be honest of what exactly caused this issue. One network switch in Europe can't cause this problem when they have redundant network. Last time they lied and this time they better come clean.

David Yach said "multiple redundancy".

Clearly not for this switch. Which is unacceptable in a system this critical. Assuming you want to stay in business.

When you spend hundreds of thousands of dollars on large scale switches + there redundancies AND you regularly test and switch over or even load balance or HA or whatever they might have setup, yea you kind of get surprise when the crap fails.

Data center failures happen even with the best equipment. Expect more outages from lots of vendors once Cloud technology really gets going.

I'm fairly technical, but would never pretend to fully understand how this switch failure could have allowed for such a large scale outage; however, I can say with 100% certainty that RIM's network is, like, way more complicated than my D-LINK router + laptop ;)

I think many people will be expecting a more detailed, technical explanation as to what happened and what steps will be taken to lessen the chance of this happening again. These details would be needed to restore confidence.

It just goes to show that, the bigger you are, the harder you can fall.

I don't think people realise how may emails and BBM messages would be going through their network. If you've ever had anything to do with SQL servers (not you personally Rob) and replication you'll know how one problem can cascade into a MUCH bigger problem. While I'm not completely convinced one switch was the problem, I can see how it would have caned their servers trying to cope with the load the last 3 days. I appreciate the public apology from Mike. It was a good effort.

Anywho, I've only been a BB user for a year and I dig it and hope their BBX phones are fracking awesome!

I heard some comments in this site that Blackberry moves data at the Petabyte level every single day. Imagine the incredible amount of data lag if the service is interrupted. Anyway, I received all my emails at 4:03am this morning making my total outage time of around 26 hours.

When I see a problem of this magnitude, I picture a room full of switches burned by a malfunctioning generator.

Dear Blackberry, it's nice you're back, but can I ever fully trust you again? Also, while you were gone I touched a Samsung Galaxy in a bar.

Bbm and emails starting working again as of this morning. Goes to show how dependent we all are on our BBs! Hope it doesn't happen anymore

gief t-shirt!

Orange Romania, all working fine since late last night. Glad it's over, hope it will stay this way.

Good stuff from Mike L, he's gone up in my estimations, hopefully they will do this at the start of problems in futher not the end.

P.s. my BB is up and running again :)

In Croatia beerys are working well now. Yesterday was a little problems with email, but there was no problem with browsing. Anyway, now is everything back to normal ;) thanks RIM for fast repair.

My BlackBerry started receiving emails again last night around 5:30 and its been working perfectly fine ever since.

Josh Sawyer
Blackberry Bold 9930, Blackberry Playbook, and HTC Evo 3D owner

My BlackBerry started receiving emails again last night around 5:30 and its been working perfectly fine ever since.

Josh Sawyer
Blackberry Bold 9930, Blackberry Playbook, and HTC Evo 3D owner

Dear Mike,

Thank you for your sincerity to come down and speak to us. It's a relief for your customers who have been so struggled with the outage. Yes, I agree it's too soon to conclude the meltdown is resolved, but the service is getting improved. I like that statement. I hope RIM will continue to provide the reliable service we can depend on, day by day.

From Melbourne, AUS on Optus: my BBM is starting to kick off, starting to be able to send messages to friends, but still sometimes experiencing delays as well. other BIS services are working as usual. Hopefully all services will be up and running by this weekend.

Mike seems very genuine, mine, in the UK came back on this morning. It's very disappointing to have this happen, but when you think of the 'antenna-gate' that apple had last year, no company is beyond error or mistake. Let's hope that this is the end of it.

This is what people tends to forget and i guarandamntee haters/whiners do not know a thing about: as a CEO and CIO (earlier post) they dont oversee the operations of the server themselves. YET imagine what this guys have to go through:

In the middle of the night you receive a call regarding the server. damn right its not your fault, but the person on the other end says, "Sir, the best thing to do is to issue a public apology to minimise damage to our image while our team fix this" What would you feel? FML thats what. you are forced to apologize for things that you didnt do.

Yet these guys apologized and im glad they did. so those of you who whine bla bla remember they are human too and they are paid what they are paid because they have the capacity to take responsilbility and apologize while most of you would say F that im going home just like u say F BB im switching.

Could be his fault. What if the Engineering group identified that this could happen and asked for funding or additional capacity and were shot down by the CEO's in a capital plan due to high costs or something? You don't know if it is or isn't his fault, but the simple rule of management, everything is managements fault.

Anyway I'm glad that one of these two guys got out there and send "we screwed up". They need to do a little more of this not just for BB users but for those of us that bought the Playbook. I hope this is the beginning of a humbler, hard working RIM than what we have dealt with the past few years.

Oh and I'm still not buying this being the result of a single core switch failure, and if it truly is then they better hire some brighter data engineers and get some cold spares out to sites so this can be fixed quicker.

that's part of the job description. why do you think he gets paid as much as about 500 of his lower level employees combined?

I'm on Rogers in the Toronto area and am still without mail. IM services are working but I've not received any mail since yesterday at 2pm. In fact I've only received 5 emails in the past 24 hrs.

This, obviously sucks.

Hi Mike,

Thanks at last for the update.I am in the UK and mine has not worked all day so I am still waiting.

I have to add that the communications for this outage and in general from RIM are nothing short of a disgrace.If I treated my clients like this I wouldn't have a business for very long.You might have developed a great bit of kit but the backup falls well short of an acceptable level.If you are to grow your business I suggest you address this matter at the highest levels as matter of urgency

I'm glad you're not running RIM. No company is without failure at some point for some period of time, your company included. All you've proved is that your head is buried to deep in the sand to know WHEN your company falls short.

my bbm is working, had a few emails but i am unable to use "blackberry protect" this is the only problem i am currently having, very happy to have back bbm!

This realllllyyy sucks, and him up on that video doesn't help me personally. Its like come on guys.

On a side note me personally I see this as a growing issue for them. Looking to switch to Apple or Android

That was about as humble a video from an industry leader as I ever seen (despite cue cards).

Ok Mike, I forgive ya. Now lets get this show on the road and quit farting around. You got 3 major wakeup calls and RIM doesn't get a standing 8 count. Fix this damn problem and release OS2.0.

I hope RIM really, really wakes up and comes out fighting for the sake of its reputation. Perhaps this was RIM Co-CEOs CTJ (Coming To Jesus) moment that lights a fire under their ass and re-energizes their commitment toward its subscriber base?

Sometimes it takes a disaster to get things going.

Is this going to be one of the "best of" episodes in the rumored blackbery TV service that was mentioned a while back? Ha ;-)

I only saw minimal impact yesterday. Maybe I was lucky.

Nice to see Mike (and yes, I feel I can call him by his first name since I've been a BB user and abuser for so very long) issue this apology.

Looks like he hasn't had much sleep the last few days.....

There isn't a phone made by a competitor that makes me want to ditch my berry. I thought the droid pro would be that phone but my wife has it and I'm not a fan. As long as email and messaging are the priority, I'll continue to use the phone that handles this need the best way I see fit. I'm more practical than I am loyal.

Personally I didn't need an apology. They didn't do it on purpose and they do have 99.9% uptime. I'm happy with that. It just sucks that it was down for consecutive days :(

Mike apology accepted.

Now can I apologize to you for having to answer to the whiners Arggg!

The funny thing is RIM didn't make the switch that failed. But they are the ones that get shit on? WTF people

On bell and bbms arent going through. My BIS email accounts keep getting deleted. Added them 3 times today already and they keep deleting themselves. WTF.

Never really experienced any problems here in Louisiana - BBM no interuption (D and R never an issue either). All emails on BB within 30 to 60 seconds of them showing on laptop, immediate received on other end when sent from gmail and my company email from my BB

The apology was a good step for public perception, though I could have done without one. BB service is very reliable otherwise. But I need that T-Shirt :D

Up and running full speed in Ontario, Canada via the Bell Mobility network. Hope we get some sort of free apps or something to smooth over this little disruption here. :)

Blackberry for life baby!!

It takes a great deal to apologize publicly and take responsibility Mike. My BlackBerry is up and running again.
Apology accepted.

I must be lucky i did not experience any outtages. My BBM, e-mail and internet all worked fine. Only exception being some bbm messages didnt go through to friends experiencing issues with their BIS. I am 9650 Bold User. Verizon Wireless New England.

Another nail in Mike's coffin.

The kool aid is not tasting very good after the last few days. By the way, cue card reading of someone elses words does not strike me as 'genuine' or 'sincere' but hey, you guys must have to believe in something...

While I continue to use blackberry (9930 and was 'all in' with a 64gig Playbook on launch day), it can not be denied that this company continues to over-promise and underwhelm at nearly every turn. Other than the 9900/9930 (which launched with 2010 specs in 2011) the line up of products remains weak/uninspired. What a piece of crap that new all-touch blackberry is - wow. Try typing on that one for more than a few minutes. They can't even figure out how to properly name these devices - marketing fail 101. Even the Playbook was cut at the knees by inflated pricing and an all but empty ecosystem that only now is showing flickers of life (if you are a gamer that is...). Way too little too late. And imagine pinning the hopes of an entire billion+ dollar company on a software change when the first tangible evidence of it in practice on a blackberry devise led to global sales of under 250,000 in 6 months (While the Kindle Fire - stealing its design sells more units in just one week than blackberry could in 5+ months!). You want to know why Mike L looks a bit haggard - I can give you lots of reasons! He is a man who is months away from being tossed out on his ass.

If RIM is to be salvaged as a continuing entity and not split into pieces and/or sold/absorbed by a rival job one will be finding a new leader and doing away with the piss poor dual headed CEO team that have squandered every advantage bbery once owned.

One day this firm will be a case study in business school on how NOT to run a business.

OK - end of the rant. Too bad a good company with a once great product fell into such a sad state. One final thought...if/when Blackberry bites the dust in the next 18-24 months, does Kevin reinvent himself as an Apple fan boy or does he go Andriod??? Time will tell...

All I kept thinking during the outage was 'wow...this is what it feels like to have a Droid or i apple.' How do they cope? Looks like its still a 99.9% uptime to me. Much better than what my Business internet provider can say and I pay more for that!

Apology accepted. Now go take a nap! :)

my service has been back since yesterday, but you better make this up to me somehow, like maybe releasing OS2 for my playbook *cough cough*

I think it was good to put this out there like this, and not just a quick press release like before. That says a lot about how serious they are taking this situation. I think the only thing that was missing was a better explanation of what caused the outage.

I would think that their up-time percentage will drop a bit after all of this.

Vodafone, UK. I get one or two stranded BBMs that were sent to me days ago, but I still can't send or access me email servers (through setup to resend my service books). Still not up and running here, but no worries. It will eventually. People need to chill out. It's only a phone. It's pathetic how outraged people are getting everywhere and on social media. You'll get your line rental back for the down days (as per email sent to me by Vodafone). There are worse things in life than a lack of emails on a BlackBerry.

The world would be a better place if people were ranting all over the net something like "Damn you weather, now half a million people are going to die in Africa. Come on people, give them a couple of bucks so they can have something to eat!"

Thanks much RIM! My BlackBerry Internet Service in Jamaica was only affected yesterday! Now it's back today! My heart goes out to those who suffered from lack of service since Monday or so. I hope the problem has been properly resolved now. Technology can fail and I accept that!

i thought it was heart felt and very good. being in the telecommunications business myself i understand more than most when he says the failover failed. I've had that happen in my business and man we all start to scramble when that happens

Idaho, US on Verizon with Torch 9850 here. Email started coming back last night. Still a little slow but seems to be fine. I made it thru ok, but knew it was crippling to some. The bad thing was my IP was also having issues and I could recieve but not send email yesterday. Very disorienting and showed me how dependent I am on technology.

I did like the apology, and it is accepted. I have seen signs of RIM "getting it" lately and think they are stepping up. We, of course, are unaware of much of what is going on behind the scenes, but RIM would have to be dead not to get that they really have a lot to prove. I figure a lot of the teams and certainly CEOs are not getting much sleep and there is much to be finished before DevCon. They know they don't have to roll things out next week, but they have to make some concrete promises that delight us. And those promises will have to be held to. Trust is very fragile.

+1 Sincere, pragmatic, and honest.... Boy does he look tired...

Here is the real question... Who sold or architected the failover solution? Who was the vendor whose hardware failed?

They should bear the Cross to Golgotha with RIM.

I guess the riots in the UK have nothing to do with this. Funny how all this happens just before 4s is launched on the 14th. I wonder if imMessage is open to all govts. because bbm will certainly be once this is over. Poor Mike it must be killing him to not be able to voice this even if he's not sure. Then again there are forces at work here beyond our/my comprehension. Therefore wether we like it or not the screws are being tightened. If I don't hear something concrete coming out of Devcon next week I hate to say it but I'm out of here keyboard and all. The media has done it's job in discrediting this company some more. I just can't believe that this could happen to a company like RIM unless it was allowed to happen. and yes Mike was reading because they couldn't let him just express himself openly as he may have just told us the truth... My son who is only 21 and switched from a bb to the iphone only a month ago came home last night and heard me and my wife discussing how both our phones are semi down his first words were I told you bb s--ks. I spoke to 2 other friends last night and they both had the same sentiments if not the same word and they both recently switched from bb to iphones and will never come back. their words not mine. They both honestly believe the bb is dying a very slow death. The media is helping in a big way. So Mike why don't you speak up and put it on the line atleast if you have to go you go with your head held up high, no need to spin you are only hurting this company further. If people just used some critical thinking in their everyday lives the media would have very little to justify their existence. Since this is not the case RIM will continue the downward spiral that people who are aware of what is going on will have no choice but to jump ship. Unless and here is a shot in the dark. forget about google and I've heard this scenerio so I don't think it will work as well as this will. RIM and MICROSOFT together under one umbrella. The media will have won the battle but not the war. At the end of the day they both need to reboot in order to work better. I'm keeping the faith for a little while longer.

I don't think Blackberry was ever down, I think it was just the antenna runs around the phone and we were all holding it wrong! ;)

All good here in Montreal.

I appreciate RIM's attempts to inform people through their support page, twiiter, facebook, conference call, and now video from Mike L. I like the fact that Mike L was wearing a t-shirt instead of wearing a suit. It's good for image.

What frustrates me are the people that whine when they think that RIM isn't saying enough. Then they whine about RIM's 'arrogance' when they do issue statements. I don't think they can say anything about Mike L's statement. he was apologetic and certainly not 'arrogant.'

I guess the riots in the UK have nothing to do with this. Funny how all this happens just before 4s is launched on the 14th. I wonder if imMessage is open to all govts. because bbm will certainly be once this is over. Poor Mike it must be killing him to not be able to voice this even if he's not sure. Then again there are forces at work here beyond our/my comprehension. Therefore wether we like it or not the screws are being tightened. If I don't hear something concrete coming out of Devcon next week I hate to say it but I'm out of here keyboard and all. The media has done it's job in discrediting this company some more. I just can't believe that this could happen to a company like RIM unless it was allowed to happen. and yes Mike was reading because they couldn't let him just express himself openly as he may have just told us the truth... My son who is only 21 and switched from a bb to the iphone only a month ago came home last night and heard me and my wife discussing how both our phones are semi down his first words were I told you bb s--ks. I spoke to 2 other friends last night and they both had the same sentiments if not the same word and they both recently switched from bb to iphones and will never come back. their words not mine. They both honestly believe the bb is dying a very slow death. The media is helping in a big way. So Mike why don't you speak up and put it on the line atleast if you have to go you go with your head held up high, no need to spin you are only hurting this company further. If people just used some critical thinking in their everyday lives the media would have very little to justify their existence. Since this is not the case RIM will continue the downward spiral that people who are aware of what is going on will have no choice but to jump ship. Unless and here is a shot in the dark. forget about google and I've heard this scenerio so I don't think it will work as well as this will. RIM and MICROSOFT together under one umbrella. The media will have won the battle but not the war. At the end of the day they both need to reboot in order to work better. I guess I will keep the faith for a little while longer. Yes this was typed from my playbook that I paid $699.00 + Tax. 64Gb. How much are they going for now?

The CEO releasing a video to apologize, and to clearly speak out that his company messed up, is very unusual in this industry and should get highly appreciated. I certainly respect that.
What is required now in addition is to release information on what exactly happened (a failing switch failover alone is not believable as the cause for an outage of such dimensions) and what RIM is going to do in order to make sure this will not happen again.
This I would call perfect problem management and customer communication.

Apology accepted Mike, shit happens & nothing is 100% failsafe when it comes to technology, time to move forwards & try to ensure that such lengthy outages never happen again.

When Mike says things like "I/we know how much this has affected you..." yada yada yada....I just don't think they care how much it has affected us.

Mike, a great way to try an grab some confidence back.....offer every person who feels they were terribly affected a free phone of their choice. Sure, it will hurt you monetarily, and the wall street pundits will be all over it (if they are not hanging out in a tent), but it would hopefully be seen as "we care about you and we want to keep you).

Oh, and btw, next time, have your cue cards right over the camera...it's distracting having you looking over to your left all the time......

I have to say that from what I noticed, I wasn't affected here in Philadelphia. I was able to connect to the Internet and recieved all of my emails with no problems. Honestly, if I wasn't on Crackberry almost all the time, I wouldn't have even known about the outage.

I think it's awesome of Mr. Lazaridis to come out and admit the mistakes. I don't see how people can bash RIM for not addressing the issue, but then attack them as soon as they do. Makes no sense to me.

Thanks RIM for the amazing service you provide for us and best of luck with DevCon and BBX!

Would haev appreciated the apology more if it actually came sooner than later. He should have communicated to the BB community and media on Day 1 of the outage to say that they are sorry and are working on the problem. That would have gone over better than the female who did the initial press conference - that was just poor.

Strangely I did not miss a beat with all of my work email.. It is always on somewhat of a delay anyway. My G-Mail was messed up for a while though.. But I received everything later that night.

Apology Accepted Mike. You guys are good people.. Happy to be a BB user!!

Thank you for bringing my BB back to life..it feels great to see those tick with initial D and R..which means I am connected again to the world..and my hope is that for RIM to improve the server functionality..yeah sorry I admit that I really cursed a lot for d past 3 days..haha..

You know what RIM, a bunch of tweets and a video of your hack CEO's reading insincere apologies off of cue cards like a bad SNL skit doesn't cut it.

Issue one statement, offer some app store credit and get back to business.

Nobody really cares why it happened, they just want their phone to work.

just a thought about the cause, though I'm definitely no tech expert. I remember hearing a little while back the reason PB was released without native email was that their servers couldnt handle it at the time. With OS2 now being so close, perhaps the adjustments they needed to make threw their network off, or they had to take it offline to make changes and couldnt get it started back up

Nicely done clip. No BS, just straight info without any overblown promises. I wish we could get more of that type of update from RIM. Would probably help build confidence in RIM as a company and the future of BB overall.

I think, and this is just me, it's not all that bad considering all companies experience glitches. I see it as one hiccup in a so far great experience with them. I experienced a delay in messages and inability to use BB protect but that's about it so I won't speak on behalf of others. But I still looooove my rocking 9900! Proud BB owner here. I'll take one hiccup even per year over my daily dropped calls with my iPhone wc I still hang on to for entertainment wc it does it's job quite well. The idea of my BB being an entertainment device still doesn't quite sit on me aside from music and occasional YouTube. All have pros and cons and there's no use pitting them against each other. BB's are workhorses and sometimes they hit a bump on the road. Let's move on and let them get their shot at maintaining customer confidence. Everything is theirs to lose so I don't think they will be complacent or cocky about this.

I hope all y'all got your BB juices all flowing back to normal.

And sidebar, another reason I love BB is because Mike looks like a Canadian Santa. High five Mike!

This is pathetic. Someone tell that guy to hire a PR firm.

The statement should have simply been, "Most customers are not affected, we know what the issue is, and we'll fix it."

The next time we heard from the CEO it should have been, "We fixed it and we're putting safeguards in place to prevent this sort of problem again, sorry for the inconvenience."

Why the big news story? It's a cell phone - the crap hardly works half the time due to cell reception issues - you can hardly walk from East 52nd St to West 52nd St. without losing a cell signal 3 times. What's the big change?!