Join Our 3 MILLION+ Members Today! Register Here | Login
Login or Register to post comments

87 Comments

Posted by oneday116 Wednesday, Oct 12, 2011 224 days ago

there late

 
 
Posted by theHip Wednesday, Oct 12, 2011 224 days ago

They're.

 
 
Posted by LegenJERRY Wednesday, Oct 12, 2011 224 days ago

+10000 hahahahahaha

 
 
Posted by kojita Wednesday, Oct 12, 2011 224 days ago

Listening now, did not learn anything new so far

 
 
Posted by kojita Wednesday, Oct 12, 2011 224 days ago

Q&A session

 
 
Posted by Accidental Post Wednesday, Oct 12, 2011 224 days ago

Canned response so far.....

 
 
Posted by kojita Wednesday, Oct 12, 2011 224 days ago

lol no questions?

 
 
Posted by shabbs Wednesday, Oct 12, 2011 224 days ago

Those questions were weak. No one pressed for an ETA on when things would calm down or even if they have gotten the failed core switch back on-line at all. How many core switches does RIM have globally? Is there a risk of backlog saturation at which point they won't be able to continue storing message?

 
 
Posted by battery Wednesday, Oct 12, 2011 224 days ago

Just that there is no new problems causing the NA trouble - just back log from the initial problem

 
 
Posted by Ateela Wednesday, Oct 12, 2011 224 days ago

They said they'll tell us the number of people affected but they didn't...

 
 
Posted by imran.ahmed Wednesday, Oct 12, 2011 224 days ago

Great, I can't even listen to this from my Bold 9900! super fail

 
 
Posted by Soprono Wednesday, Oct 12, 2011 224 days ago

very true on the delay between communication from RIM to general public

 
 
Posted by BENZ_O Wednesday, Oct 12, 2011 224 days ago

PS3 was the worst!

 
 
Posted by kojita Wednesday, Oct 12, 2011 224 days ago

Aouch, from the global mail rough answers but I understand that they are heavily under pressure, but they should get cooler people communicating

 
 
Posted by Soprono Wednesday, Oct 12, 2011 224 days ago

shit, hardly anything valuable

 
 
Posted by maxb1ack Wednesday, Oct 12, 2011 224 days ago

time line for restoration of service?

 
 
Posted by Ateela Wednesday, Oct 12, 2011 224 days ago

Last caller asked some hard questions lol

 
 
Posted by kojita Wednesday, Oct 12, 2011 224 days ago

How can a single core switch produce such a global mess?

 
 
Posted by Ateela Wednesday, Oct 12, 2011 224 days ago

They answered that: the core switch failure was followed by a huge backlog of messages / emails and slowed the entire server down..

 
 
Posted by kojita Wednesday, Oct 12, 2011 224 days ago

yeah, i got that. just wondering what size of backlog is necessary to produce such a halt in the services.

 
 
Posted by kojita Wednesday, Oct 12, 2011 224 days ago

no absolutely nothing on timeline, before devcon though :)

 
 
Posted by kojita Wednesday, Oct 12, 2011 224 days ago

How big do you think the backlog is? That xould have been interesting to know

 
 
Posted by maxb1ack Wednesday, Oct 12, 2011 224 days ago

considering there are 3 RIM relays in the world... I'd say half blackberry users worth of sent/recieved data over... even 1 day.... a lot

 
 
Posted by BBThemes Wednesday, Oct 12, 2011 224 days ago

RIM normally moves around 20 Petabytes of data a day on average, thats 20480 Tb, so yea quite alot

 
 
Posted by kojita Wednesday, Oct 12, 2011 224 days ago

wow, I was not aware of the numbers...their uptime is quite impressive, and their services too.

 
 
Posted by Robert22 Wednesday, Oct 12, 2011 224 days ago

We have not got any new informations. Just kidding.

 
 
Posted by Hitman6267 Wednesday, Oct 12, 2011 224 days ago

did seriously no one ask about an ETA for restoration ?

 
 
Posted by BBHacker Wednesday, Oct 12, 2011 224 days ago

Was just listening to the call, horrible crisis management

There are thousands of people at RIM and non-engineers and you are telling us you have been too busy you can't update your users?

On a technical note: If all the messages will be delivered, where do they store them? Either they store all of our messages all the time or they have a magic storage facility that can scale up that fast and can store all messages sent through BIS for several days? Anyone has more knowledge about this?

It is a pity RIM, but as I said already on Monday, you are digging your own grave: http://bit.ly/rim_outage

 
 
Posted by naila_nc Wednesday, Oct 12, 2011 224 days ago

I was going to answer, but I am not sure how to answer that without getting in trouble. Ask me next year then I can provide more information ;).

 
 
Posted by ljcn Wednesday, Oct 12, 2011 224 days ago

Wow, that was awful. They didn't answer the (few) questions and didn't tell us anything. It is quite clear that they don't get customer service at all. Frighteningly inept.

 
 
Posted by ljcn Wednesday, Oct 12, 2011 224 days ago

You have to think that they don't yet have a clue how to solve the problem or how long it will take, or they would have said something substantial. Things are much, much worse than we thought.

 
 
Posted by Robert22 Wednesday, Oct 12, 2011 224 days ago

So think I.

 
 
Posted by seb.hebert Wednesday, Oct 12, 2011 224 days ago

or maybe its all a front to keep people from knowing that qnx is ready to go as of next week. yeah. probably not but im wishing.

 
 
Posted by kojita Wednesday, Oct 12, 2011 224 days ago

Maybe even the com guys are sending back log emails....
Seriously, I dont care much about com but this was horrible...they could have send at least nice people not robots

 
 
Posted by mike22 Wednesday, Oct 12, 2011 224 days ago

Just keep req sevice books over and over 9900 for sale

 
 
Posted by JPMorgan_ Wednesday, Oct 12, 2011 224 days ago

Big hit to RIM's reputation. Things not looking good.

 
 
Posted by smartbuddy Wednesday, Oct 12, 2011 224 days ago

This conference would hurt RIM more than do anything good...we already know PR of RIM is awe full but it has hit a new low now...

 
 
Posted by pistolpetefan23 Wednesday, Oct 12, 2011 224 days ago

In North America (Wisconsin) and my email and BBM were down for this morning, but about an hour ago (2 pm CST) it was restored.

 
 
Posted by laxcrespo1013 Wednesday, Oct 12, 2011 224 days ago

???

 
 
Posted by greatwiseone Wednesday, Oct 12, 2011 224 days ago

Another reason why RIM needs to ditch BIS for QNX phones...

 
 
Posted by deRusett Wednesday, Oct 12, 2011 224 days ago

Best part I got was

"Data backlog causing problems catching up"

And 10000's of users are sending mass Broadcast messages to "fix their bbm". Causing MORE data back up

 
 
Posted by john_v Wednesday, Oct 12, 2011 224 days ago

This. And add to that people sending "test" messages every 10 minutes just to see if it's working again.

 
 
Posted by SevereDeceit Wednesday, Oct 12, 2011 224 days ago

This is giving Apple even more business. I bet the die hard BlackBerry users regret their bashing of the iPhone 4S now.

 
 
Posted by BENZ_O Wednesday, Oct 12, 2011 224 days ago

I don't.

 
 
Posted by joKer_pb Wednesday, Oct 12, 2011 224 days ago

+1

 
 
Posted by texazzpete Wednesday, Oct 12, 2011 224 days ago

Well, you should. The whole cynical world is eating humble pie over the iPhone 4S's stellar sales so far.
The iPhone 4 already had better specs than the Bold 9900, not to talk of the 4S. This failure on RIM's part won't help matters at all.

 
 
Posted by maxb1ack Wednesday, Oct 12, 2011 224 days ago

funny to see nexus and 4S devices advertisement right above shopCrackBerry ... :D

 
 
Posted by imgleb Wednesday, Oct 12, 2011 224 days ago

The only thing that's keeping me with Rim is their keyboard. iPhone is a better system without a doubt

 
 
Posted by swade86 Wednesday, Oct 12, 2011 224 days ago

cosign on deRusett.....the thousands of jackasses that are sending stupid broadcast messages are making it harder for things to be normal....this outage is a blow to all of us but hopefully they get it restored in time

 
 
Posted by ChrisInSD Wednesday, Oct 12, 2011 224 days ago

Most people experiencing "a delay or, in some cases, service interruption." SOME cases? I don't know who is having a "delay" with this--I have had complete service interruption all day. No e-mail, no web, no nada. Way to downplay this major issue RIM--sounds like a BlackBerry PlayBook sales #s call with plenty of propaganda to go around

 
 
Posted by Darlaten Wednesday, Oct 12, 2011 224 days ago

RIM at its finest - once again showcasing to the world and to their customers just how out of touch, antiquated, incompetent, and ill prepared this company is for the modern age.

And for god sake, get someone who can at least communicate properly with the media. As it is, you've already got major media outlets running rampant with stories of this problem; speculation ranges from anywhere that this is the death knell for the company to its CEO's being forced to resign over this to its stock value dropping so low that a takeover is guaranteed. Might be nice for RIM to step up to the plate with someone who is competent and address what is being reported at within the media.

Particularly in light of the fact that any excitement that had been building up for Devcon has literally been thrown out the window. Now the sole focus will be on what happened and why did it happen.

 
 
Posted by ChrisInSD Wednesday, Oct 12, 2011 224 days ago

+1. Did anyone else get the impression that the smug girl from "Media Relations" seemed like she was about 25 years old? These people really do seem completely clueless sometimes...

 
 
Posted by toloy2k Wednesday, Oct 12, 2011 224 days ago

Rim, if you are listening or reading these posts then get in touch, i look after the biggest datacentres in Europe and work for one of the biggest IT outsourcing companies, we deal with major cliants and yes some of them are your competitors. We work to strict SLA's and so far we achieve for our customers a 98% break fix service on a 4hr fix.
Some of our clients are your competitors but seeing as you seem to be unable to fix the problem it may be time to call in some external resourse to help and aid in getting these servers up and running. We look after data centres in Slough but so far we havent been engaged in your issue.
If you need to contact me then please do and i will supply contact details.
Regards

 
 
Posted by q649 Wednesday, Oct 12, 2011 224 days ago

Don't hold your breath.

 
 
Posted by br14 Wednesday, Oct 12, 2011 224 days ago

I'd bet you don't handle anything like the volumes of data RIM's NOC handles.

In fact, it's a pretty safe bet their data centres handle more data than anyone.

RIM will have fixed the original issue in a matter of hours, but the subsequent volume issues are enormous.

Even a few minutes of outage can take hours to resolve. And no conventional software products can handle the load.

The problem is that the individual carriers can't handle the volume either, and that the messages need to be processed in sequence as stated by David Yach.

So take the 400+ carriers, each of which needs careful flow control, plus sending all the messages in sequence, and you get some idea of the complexity.

If RIM releases the dam holding back the data, there's a good chance their carrier partner data centres would also collapse.

It would be like having multiple DoS attacks hitting your networks.

I used to be like you, thinking I could solve the problems, until I realised the scale of their solution. RIM has some of the best minds in the industry working on these problems and the failure is not technical. It's management. Just a question of dedicating a few hundred million dollars.

 
 
Posted by dcroissant Thursday, Oct 13, 2011 223 days ago

+1

 
 
Posted by ljcn Wednesday, Oct 12, 2011 224 days ago

The message was specifically that your e-mails are being preserved for later distribution - the implication being that all BBM messages are being deleted as they are sent. The more I listen to the call the more it sounds like legal ground being prepared, and the less it seems like a customer-related exercise. If that was RIM's legal team, at least it explains why they were such god-awful PR people.

 
 
Posted by br14 Wednesday, Oct 12, 2011 224 days ago

It wasn't their legal team. David Yach is CTO for Software and the other person was from their PR team.

And BBM's are also being sent. Just a few hours late if you're one of the unlucky ones.

 
 
Posted by ljcn Wednesday, Oct 12, 2011 224 days ago

Er, actually....no. None of the BBMs I have sent since Monday at about 11:00am GMT has been transmitted, and none of those sent to me has been delivered. And that includes the time when the service was "fully restored" (sic). BBMs sent during the few hours on Tuesday morning went through OK, but none of the ones sent on Monday afternoon ever did.

 
 
Posted by br14 Wednesday, Oct 12, 2011 224 days ago

Sorry, you must be on the European data centre. In North America the messages are being delivered.

That suggests the problem is rather more significant than is being admitted.

Although failure of a switch and the subsequent failure of the failover (legal case against a hardware vendor?) might indicate they lost some data. Unless data is queued in the carrier locations.

 
 
Posted by lushpupp Wednesday, Oct 12, 2011 224 days ago

I'm located in Canada and all my messages sent since about the 10pm EST last night did not send and it has almost been a full day late. There was a brief window around 1pm where i was able to send new messages, but all prior messages did not send.

 
 
Posted by balenax Wednesday, Oct 12, 2011 224 days ago

I dont have bis service for 3 days now in Romania, on Orange. so it's not only delays and slowness and whatever crap they told. It's totally dead. No mail, no bbm, no nothing. I can only use the browser and facebook/ twitter but just on WiFi.

 
 
Posted by rappadappa77 Wednesday, Oct 12, 2011 224 days ago

I have been getting the same 5 emails from last night about 6 times now, wtf!

 
 
Posted by shabbs Wednesday, Oct 12, 2011 224 days ago

Who was doing all that typing? Heh heh...

 
 
Posted by undone Wednesday, Oct 12, 2011 224 days ago

seems like the flow is going got a glut of emails all at once.

 
 
Posted by jellico Wednesday, Oct 12, 2011 224 days ago

At the risk of being jinxed..., I have had service with no interruptions during all of this

 
 
Posted by shlee Wednesday, Oct 12, 2011 224 days ago

Michelle, I still cant find out which conference center handled the call.

 
 
Posted by jedimstr397 Wednesday, Oct 12, 2011 224 days ago

There's a whole upside to this outage thing that we all need to consider, and I think we can all sleep easier knowing it.

At least there won't be any riots being organised and excecuted any time soon!

Badum-bum ching!

 
 
Posted by flyadan Wednesday, Oct 12, 2011 224 days ago

http://mobile.nytimes.com/article;jsessionid=7DF97971FB38E638CF295FE8881...

Gotta love it when crackberry makes the NY TIMES

 
 
Posted by dellri1 Wednesday, Oct 12, 2011 224 days ago

ChrisInSD - 750+ comments on the blog regarding service interruption... 70MM + BB users.... not sure that's a representative sample of EVERYONE. I had intermittent data problems for only two hours this morning and that was all. If you're affected I am sure it sucks, but don't assume the WHOLE system is affected to that degree.
As for PR, I'm in the investment business and wouldn't put much stock (no pun intended) in what analysts say about a company. A lot of them are like economists and the weather man when it comes to the accuracy of their predictions. The media loves a good story and it's even better when it is negative, hence the use of RIM as the market pinata of choice lately!!

 
 
Posted by br14 Wednesday, Oct 12, 2011 224 days ago

"hence the use of RIM as the market pinata"

True enough. RIM's performance is nowhere near as bad as the share price would indicate.

But to ignore the impact PR has on the company is too stick your head in the sand.

You can bet Apple wouldn't have handled this as badly. They're so good at PR the death of their CEO made postive headline news all around the world.

I despair with RIM. They have "mulitple redundancy", yet their "fully tested" failover failed. Sorry but that's not believable (the "tested" bit at least).

 
 
Posted by dellri1 Wednesday, Oct 12, 2011 224 days ago

I wasn't implying that the PR fallout from this or from other issues that RIM has had should be ignored and that they don't have an impact. They do and they should be addressed.
I was attempting to provide some objectivity/perspective to people who assume because something is happening to them that it must be happening to everyone and thus their reaction must be how everyone else is reacting. History should teach us that this is never a good thought process....

 
 
Posted by ChrisInSD Wednesday, Oct 12, 2011 224 days ago

Perhaps you should head over to BlackBerry's Facebook page and check out the comments of people much less polite than I. Don't need lecture about assuming my situation is the same as everyone's--but thanks anyway. It is the situation for A LOT of people and that's enough--it doesn't have to be EVERY USER. Did I actually say "this is happening to everyone?" I don't believe I did. Rather, I simply pointed out that RIM decided to downplay the numbers of people affected by indicating (both in their choice of words and intonation) that a few people had actually had a complete service interruption, but rather the main thing going on is minor "delays." Given the press coverage of the event, I don't have to take my own experience as the only gauge of what's occurring--rather, I can easily confirm that my experience is being repeated all over the place. I stand by my comments--RIM "sugarcoated" the problem and the PR person seemed pretty smug about it. The last thing a frustrated customer wants to hear.

They could have handled this a lot better--but the response is par for the course for them. I am not saying they are not working as fast as they can to fix it, that it isn't a big complicated issue, that this IT guy on the call isn't super smart and competent. I, as a customer, don't care about those details--I would like the service to be restored (recall, we do PAY them for this service) and I wouldn't mind a bit more empathy and a bit less elitism. Entitled to my opinion on this topic, as are you. Cheers!

 
 
Posted by dellri1 Wednesday, Oct 12, 2011 224 days ago

Wasn't trying to lecture or be elitist. Facebook shows 8.4MM likes (about 12% of BB's 70MM users) of which 168,000+ are "talking" about them. Even if you assume they are ALL being negative, that's still a small sample of users.
There is always a vocal minority on every issue and it usually doesn't represent the silent majority. That being said RIM certainly DOES need to do a much better job of managing damage control and media spin. Perception is reality and this is a classic case of that.

 
 
Posted by EchoTango Wednesday, Oct 12, 2011 224 days ago

Chill out everyone. What did you want them to say, it's all fixed and everyone gets free ice cream ?

I've had to manage major technical outages and I can tell you, it's not fun. You're trying to fix the problem and get constantly being interrupted by everyone wanting to know when it will fixed. So, you end up making a "Hobson's choice", spending your time reporting to concerned parties, delaying the resolution or managing the fix activities and pissing off your users by not communicating.

What everyone should do is let them do their jobs and if you really want your pound of flesh, bitch at them after.

 
 
Posted by BerryPotter Wednesday, Oct 12, 2011 224 days ago

email not coming through in Southern California on AT&T. And WOW that was a WEAK conference call! Given the circumstances, it should have been one of the two boneheaded CEOs addressing these questions.

Nonetheless, I still love my blackberry. It's like a junkie girlfriend who you know is bad for you but who's too hot to let go.

 
 
Posted by jvictor77 Wednesday, Oct 12, 2011 224 days ago

Here in mid-western Ontario, I've had 1 bbm (the infamous broadcast one), a couple of FB notifications and no email at all since this morning. I'm not overly concerned - but I can well imagine how the enterprise sector is feeling!

 
 
Posted by PLTM Wednesday, Oct 12, 2011 224 days ago

Look like they got it fixed as im now receiving BBM messages and a whole load of old emails.. 5:06PM was when my emails started coming in non-stop... Im in the US...

 
 
Posted by boriswade Wednesday, Oct 12, 2011 224 days ago

I should not get too beat up about this disaster - it means that RiM will have to pull out all the stops to keep their customer base.

I have also read comments saying that Apple would have handled the situation better - well as an owner of Blackberry and Apple devices remember the antenna issue in the iPhone 4 and rubbish comment of how you should hold it differently etc before finally providing cases etc.

As my Blackberry is currently bust I have been trying to upgrade my iPhone to iOS 5 and guess what? The servers are overloaded.

Anyway as email etc is down I used the fax machine today - it works!

 
 
Posted by Arthur.cc Wednesday, Oct 12, 2011 224 days ago

I use BIS in Northern Europe, and for today my "best thing" was that my emails i got a 4 hours later, everything else - silent...

The question is - How does it comes that big RIM can't fix it in a 3 days time (is it?) ? :(

Hope all's going to be solved by the night. :/ ( I'm at GMT + 02)

 
 
Posted by DeeDudnath Wednesday, Oct 12, 2011 224 days ago

So my 9900's useless right now.....

 
 
Posted by rems Wednesday, Oct 12, 2011 224 days ago

You can't make calls or use wifi??

 
 
Posted by bennavere Wednesday, Oct 12, 2011 224 days ago

Well my e-mails are finally back to normal here in Texas. But my cell coverage has been poor around my house for the last 4 days. Must be a cell site problem.

 
 
Posted by honsoloswookie Wednesday, Oct 12, 2011 224 days ago

Apple would handle this better because they do not manage email, BBM or internet traffic for there devices.
That's all done by the various carriers and Apple will not help you if you lose any data on your device or if you are missing emails.

If you really realize the amount and type of data RIM actually handles, you realize that any origination would be under enormous pressure to manage a incident like this well.

 
 
Posted by malita Wednesday, Oct 12, 2011 224 days ago

Finaly! After three days of outage my blackberry can get information via email, twitter, BeWeather.

O2, Scotland

Looks like everything is working, a bit slower than previous, but hope that the backlogs will go through during the night..

btw - pissed of because of the RIMs communication. If they would have said that - sorry, we will fix this till 12pm wednesday, and would give more status updates, saying how is the progress and so on, then the people wouldn't be so angry about this. Not the bes timing - these three days just showed me that i should really think about my next phone as iPhone..

 
 
Posted by daviskacer Wednesday, Oct 12, 2011 224 days ago

Service update from RIM CIO
To All BlackBerry Customers:
I want to first apologize for the service interruptions and delays many of you have been experiencing this week. I also wanted to connect with you directly, give you an update on the service issues we are trying to solve, and answer some of the questions and concerns you’ve expressed.
You’ve depended on us for reliable, real-time communications, and right now we’re letting you down. We are taking this very seriously and have people around the world working around the clock to address this situation. We believe we understand why this happened and we are working to restore normal service levels in all markets as quickly as we can.
Here is the current status of service and issues for the various regions that were impacted:
For Europe, Middle East, India and Africa (EMEIA):
• Email systems are operating and we are continuing to clear any backlogged messages. Support teams are working to minimize the impact on our customers.
• BBM traffic is online and traffic is passing successfully
• Browsing is temporarily unavailable as the Support teams monitor service stability and continue to assess when this service can be safely brought online
• Support teams have added capacity to help with message delivery between regions and continents
For Canada and Latin America:
• Email systems are operating and we are continuing to clear any backlogged messages. Support teams are working to minimize the impact on our customers
• BBM and browsing services are online and traffic is passing successfully (except for three carrier networks in Latin America that are serviced by the EMEIA infrastructure – browsing is temporarily unavailable for those three carrier networks)
• Support teams are investigating reports of BBM delays
For the U.S.:
• Email systems are operating and we are continuing to clear any backlogged messages. Support teams are working to minimize the impact on our customers.
• Support teams have added capacity to help with message delivery between regions and continents
• BBM and browsing services are online and traffic is passing successfully
• Support teams are investigating reports of BBM delays
We will provide regular updates on BlackBerry.com, RIM.com and via our social channels. We are doing everything in our power to restore regular service everywhere and to restore your trust in us.
Yours sincerely,
Robin Bienfait
Chief Information Officer, RIM
http://www.rim.com/newsroom/service-update.shtml

 
 
Posted by dcroissant Thursday, Oct 13, 2011 223 days ago

Sky News UK

http://news.sky.com/home/technology/article/16088130