BlackBerry Service Update Press Conference [updated: Audio download available]

By Michelle Haag on 12 Oct 2011 03:03 pm EDT


Update: The call ended with no real answers as to when BlackBerry services will be back up and running. RIM didn't really give any answers that we didn't already know, however they did explain a bit more of exactly what is going on and what caused the issues. Have a listen to the call in the player above or visit the link on the Research In Motion website.

RIM is holding a conference call at 3pm EST to discuss the recent outage in service being reported worldwide. If you'd like to tune in to listen, you can do so at the link below, and we'll follow up for those of you that can't make it live.

Join the BlackBerry Service Update Press Conference

Michelle Haag Michelle Haag "@_Miche11e_ and C0001B3B5" 1215 (articles) 1695 (forum posts)

Reader comments

BlackBerry Service Update Press Conference [updated: Audio download available]


Those questions were weak. No one pressed for an ETA on when things would calm down or even if they have gotten the failed core switch back on-line at all. How many core switches does RIM have globally? Is there a risk of backlog saturation at which point they won't be able to continue storing message?

Aouch, from the global mail rough answers but I understand that they are heavily under pressure, but they should get cooler people communicating

They answered that: the core switch failure was followed by a huge backlog of messages / emails and slowed the entire server down..

yeah, i got that. just wondering what size of backlog is necessary to produce such a halt in the services.

considering there are 3 RIM relays in the world... I'd say half blackberry users worth of sent/recieved data over... even 1 day.... a lot

Was just listening to the call, horrible crisis management

There are thousands of people at RIM and non-engineers and you are telling us you have been too busy you can't update your users?

On a technical note: If all the messages will be delivered, where do they store them? Either they store all of our messages all the time or they have a magic storage facility that can scale up that fast and can store all messages sent through BIS for several days? Anyone has more knowledge about this?

It is a pity RIM, but as I said already on Monday, you are digging your own grave:

I was going to answer, but I am not sure how to answer that without getting in trouble. Ask me next year then I can provide more information ;).

Wow, that was awful. They didn't answer the (few) questions and didn't tell us anything. It is quite clear that they don't get customer service at all. Frighteningly inept.

You have to think that they don't yet have a clue how to solve the problem or how long it will take, or they would have said something substantial. Things are much, much worse than we thought.

or maybe its all a front to keep people from knowing that qnx is ready to go as of next week. yeah. probably not but im wishing.

Maybe even the com guys are sending back log emails....
Seriously, I dont care much about com but this was horrible...they could have send at least nice people not robots

This conference would hurt RIM more than do anything good...we already know PR of RIM is awe full but it has hit a new low now...

Best part I got was

"Data backlog causing problems catching up"

And 10000's of users are sending mass Broadcast messages to "fix their bbm". Causing MORE data back up

This. And add to that people sending "test" messages every 10 minutes just to see if it's working again.

Well, you should. The whole cynical world is eating humble pie over the iPhone 4S's stellar sales so far.
The iPhone 4 already had better specs than the Bold 9900, not to talk of the 4S. This failure on RIM's part won't help matters at all.

The only thing that's keeping me with Rim is their keyboard. iPhone is a better system without a doubt

cosign on deRusett.....the thousands of jackasses that are sending stupid broadcast messages are making it harder for things to be normal....this outage is a blow to all of us but hopefully they get it restored in time

Most people experiencing "a delay or, in some cases, service interruption." SOME cases? I don't know who is having a "delay" with this--I have had complete service interruption all day. No e-mail, no web, no nada. Way to downplay this major issue RIM--sounds like a BlackBerry PlayBook sales #s call with plenty of propaganda to go around

RIM at its finest - once again showcasing to the world and to their customers just how out of touch, antiquated, incompetent, and ill prepared this company is for the modern age.

And for god sake, get someone who can at least communicate properly with the media. As it is, you've already got major media outlets running rampant with stories of this problem; speculation ranges from anywhere that this is the death knell for the company to its CEO's being forced to resign over this to its stock value dropping so low that a takeover is guaranteed. Might be nice for RIM to step up to the plate with someone who is competent and address what is being reported at within the media.

Particularly in light of the fact that any excitement that had been building up for Devcon has literally been thrown out the window. Now the sole focus will be on what happened and why did it happen.

+1. Did anyone else get the impression that the smug girl from "Media Relations" seemed like she was about 25 years old? These people really do seem completely clueless sometimes...

Rim, if you are listening or reading these posts then get in touch, i look after the biggest datacentres in Europe and work for one of the biggest IT outsourcing companies, we deal with major cliants and yes some of them are your competitors. We work to strict SLA's and so far we achieve for our customers a 98% break fix service on a 4hr fix.
Some of our clients are your competitors but seeing as you seem to be unable to fix the problem it may be time to call in some external resourse to help and aid in getting these servers up and running. We look after data centres in Slough but so far we havent been engaged in your issue.
If you need to contact me then please do and i will supply contact details.

I'd bet you don't handle anything like the volumes of data RIM's NOC handles.

In fact, it's a pretty safe bet their data centres handle more data than anyone.

RIM will have fixed the original issue in a matter of hours, but the subsequent volume issues are enormous.

Even a few minutes of outage can take hours to resolve. And no conventional software products can handle the load.

The problem is that the individual carriers can't handle the volume either, and that the messages need to be processed in sequence as stated by David Yach.

So take the 400+ carriers, each of which needs careful flow control, plus sending all the messages in sequence, and you get some idea of the complexity.

If RIM releases the dam holding back the data, there's a good chance their carrier partner data centres would also collapse.

It would be like having multiple DoS attacks hitting your networks.

I used to be like you, thinking I could solve the problems, until I realised the scale of their solution. RIM has some of the best minds in the industry working on these problems and the failure is not technical. It's management. Just a question of dedicating a few hundred million dollars.

The message was specifically that your e-mails are being preserved for later distribution - the implication being that all BBM messages are being deleted as they are sent. The more I listen to the call the more it sounds like legal ground being prepared, and the less it seems like a customer-related exercise. If that was RIM's legal team, at least it explains why they were such god-awful PR people.

It wasn't their legal team. David Yach is CTO for Software and the other person was from their PR team.

And BBM's are also being sent. Just a few hours late if you're one of the unlucky ones.

Er, None of the BBMs I have sent since Monday at about 11:00am GMT has been transmitted, and none of those sent to me has been delivered. And that includes the time when the service was "fully restored" (sic). BBMs sent during the few hours on Tuesday morning went through OK, but none of the ones sent on Monday afternoon ever did.

Sorry, you must be on the European data centre. In North America the messages are being delivered.

That suggests the problem is rather more significant than is being admitted.

Although failure of a switch and the subsequent failure of the failover (legal case against a hardware vendor?) might indicate they lost some data. Unless data is queued in the carrier locations.

I'm located in Canada and all my messages sent since about the 10pm EST last night did not send and it has almost been a full day late. There was a brief window around 1pm where i was able to send new messages, but all prior messages did not send.

I dont have bis service for 3 days now in Romania, on Orange. so it's not only delays and slowness and whatever crap they told. It's totally dead. No mail, no bbm, no nothing. I can only use the browser and facebook/ twitter but just on WiFi.

There's a whole upside to this outage thing that we all need to consider, and I think we can all sleep easier knowing it.

At least there won't be any riots being organised and excecuted any time soon!

Badum-bum ching!

ChrisInSD - 750+ comments on the blog regarding service interruption... 70MM + BB users.... not sure that's a representative sample of EVERYONE. I had intermittent data problems for only two hours this morning and that was all. If you're affected I am sure it sucks, but don't assume the WHOLE system is affected to that degree.
As for PR, I'm in the investment business and wouldn't put much stock (no pun intended) in what analysts say about a company. A lot of them are like economists and the weather man when it comes to the accuracy of their predictions. The media loves a good story and it's even better when it is negative, hence the use of RIM as the market pinata of choice lately!!

"hence the use of RIM as the market pinata"

True enough. RIM's performance is nowhere near as bad as the share price would indicate.

But to ignore the impact PR has on the company is too stick your head in the sand.

You can bet Apple wouldn't have handled this as badly. They're so good at PR the death of their CEO made postive headline news all around the world.

I despair with RIM. They have "mulitple redundancy", yet their "fully tested" failover failed. Sorry but that's not believable (the "tested" bit at least).

I wasn't implying that the PR fallout from this or from other issues that RIM has had should be ignored and that they don't have an impact. They do and they should be addressed.
I was attempting to provide some objectivity/perspective to people who assume because something is happening to them that it must be happening to everyone and thus their reaction must be how everyone else is reacting. History should teach us that this is never a good thought process....

Perhaps you should head over to BlackBerry's Facebook page and check out the comments of people much less polite than I. Don't need lecture about assuming my situation is the same as everyone's--but thanks anyway. It is the situation for A LOT of people and that's enough--it doesn't have to be EVERY USER. Did I actually say "this is happening to everyone?" I don't believe I did. Rather, I simply pointed out that RIM decided to downplay the numbers of people affected by indicating (both in their choice of words and intonation) that a few people had actually had a complete service interruption, but rather the main thing going on is minor "delays." Given the press coverage of the event, I don't have to take my own experience as the only gauge of what's occurring--rather, I can easily confirm that my experience is being repeated all over the place. I stand by my comments--RIM "sugarcoated" the problem and the PR person seemed pretty smug about it. The last thing a frustrated customer wants to hear.

They could have handled this a lot better--but the response is par for the course for them. I am not saying they are not working as fast as they can to fix it, that it isn't a big complicated issue, that this IT guy on the call isn't super smart and competent. I, as a customer, don't care about those details--I would like the service to be restored (recall, we do PAY them for this service) and I wouldn't mind a bit more empathy and a bit less elitism. Entitled to my opinion on this topic, as are you. Cheers!

Wasn't trying to lecture or be elitist. Facebook shows 8.4MM likes (about 12% of BB's 70MM users) of which 168,000+ are "talking" about them. Even if you assume they are ALL being negative, that's still a small sample of users.
There is always a vocal minority on every issue and it usually doesn't represent the silent majority. That being said RIM certainly DOES need to do a much better job of managing damage control and media spin. Perception is reality and this is a classic case of that.

Chill out everyone. What did you want them to say, it's all fixed and everyone gets free ice cream ?

I've had to manage major technical outages and I can tell you, it's not fun. You're trying to fix the problem and get constantly being interrupted by everyone wanting to know when it will fixed. So, you end up making a "Hobson's choice", spending your time reporting to concerned parties, delaying the resolution or managing the fix activities and pissing off your users by not communicating.

What everyone should do is let them do their jobs and if you really want your pound of flesh, bitch at them after.

email not coming through in Southern California on AT&T. And WOW that was a WEAK conference call! Given the circumstances, it should have been one of the two boneheaded CEOs addressing these questions.

Nonetheless, I still love my blackberry. It's like a junkie girlfriend who you know is bad for you but who's too hot to let go.

Here in mid-western Ontario, I've had 1 bbm (the infamous broadcast one), a couple of FB notifications and no email at all since this morning. I'm not overly concerned - but I can well imagine how the enterprise sector is feeling!

Look like they got it fixed as im now receiving BBM messages and a whole load of old emails.. 5:06PM was when my emails started coming in non-stop... Im in the US...

I should not get too beat up about this disaster - it means that RiM will have to pull out all the stops to keep their customer base.

I have also read comments saying that Apple would have handled the situation better - well as an owner of Blackberry and Apple devices remember the antenna issue in the iPhone 4 and rubbish comment of how you should hold it differently etc before finally providing cases etc.

As my Blackberry is currently bust I have been trying to upgrade my iPhone to iOS 5 and guess what? The servers are overloaded.

Anyway as email etc is down I used the fax machine today - it works!

I use BIS in Northern Europe, and for today my "best thing" was that my emails i got a 4 hours later, everything else - silent...

The question is - How does it comes that big RIM can't fix it in a 3 days time (is it?) ? :(

Hope all's going to be solved by the night. :/ ( I'm at GMT + 02)

Well my e-mails are finally back to normal here in Texas. But my cell coverage has been poor around my house for the last 4 days. Must be a cell site problem.

Apple would handle this better because they do not manage email, BBM or internet traffic for there devices.
That's all done by the various carriers and Apple will not help you if you lose any data on your device or if you are missing emails.

If you really realize the amount and type of data RIM actually handles, you realize that any origination would be under enormous pressure to manage a incident like this well.

Finaly! After three days of outage my blackberry can get information via email, twitter, BeWeather.

O2, Scotland

Looks like everything is working, a bit slower than previous, but hope that the backlogs will go through during the night..

btw - pissed of because of the RIMs communication. If they would have said that - sorry, we will fix this till 12pm wednesday, and would give more status updates, saying how is the progress and so on, then the people wouldn't be so angry about this. Not the bes timing - these three days just showed me that i should really think about my next phone as iPhone..

Service update from RIM CIO
To All BlackBerry Customers:
I want to first apologize for the service interruptions and delays many of you have been experiencing this week. I also wanted to connect with you directly, give you an update on the service issues we are trying to solve, and answer some of the questions and concerns you’ve expressed.
You’ve depended on us for reliable, real-time communications, and right now we’re letting you down. We are taking this very seriously and have people around the world working around the clock to address this situation. We believe we understand why this happened and we are working to restore normal service levels in all markets as quickly as we can.
Here is the current status of service and issues for the various regions that were impacted:
For Europe, Middle East, India and Africa (EMEIA):
• Email systems are operating and we are continuing to clear any backlogged messages. Support teams are working to minimize the impact on our customers.
• BBM traffic is online and traffic is passing successfully
• Browsing is temporarily unavailable as the Support teams monitor service stability and continue to assess when this service can be safely brought online
• Support teams have added capacity to help with message delivery between regions and continents
For Canada and Latin America:
• Email systems are operating and we are continuing to clear any backlogged messages. Support teams are working to minimize the impact on our customers
• BBM and browsing services are online and traffic is passing successfully (except for three carrier networks in Latin America that are serviced by the EMEIA infrastructure – browsing is temporarily unavailable for those three carrier networks)
• Support teams are investigating reports of BBM delays
For the U.S.:
• Email systems are operating and we are continuing to clear any backlogged messages. Support teams are working to minimize the impact on our customers.
• Support teams have added capacity to help with message delivery between regions and continents
• BBM and browsing services are online and traffic is passing successfully
• Support teams are investigating reports of BBM delays
We will provide regular updates on, and via our social channels. We are doing everything in our power to restore regular service everywhere and to restore your trust in us.
Yours sincerely,
Robin Bienfait
Chief Information Officer, RIM