Men With BlackBerriesIt looks like the Men with BlackBerries have finally come forward and offered an explanation for the Great BlackBerry Outage of 2007.

The cause? It was the installation of an insufficiently tested piece of software that set off a chain reaction through RIM's systems which quickly escalated into leaving five million BlackBerry users in North America without wireless email and web applications. Talk about forcing CrackBerry Addicts to quit cold turkey!

As earlier reported, the problem did indeed develop at RIM's network operations center which processes every e-mail message that travels to or from every BlackBerry in North America.

RIM said that the defective software was intended to improve the operation of the BlackBerry handling system’s cache. RIM did not believe the upgrade would turn into the mess that unfolded. Instead of improving operations, the upgrade “triggered a compounding series of interaction errors” between the cache and the database that is used to track operations. I would hate to be the guy who flipped the switch that made this all happen. (Note: if anybody at RIM is out of a job over this, be sure to send us an email - we'd gladly hire you to report/write for!).

It seems things then went from bad to worse. When the company determined that the database and the cache were not functioning properly, it attempted to switch to a backup system, by activating a "failover process" which of course then failed. "The failover process did not fully perform to R.I.M.’s expectations and therefore caused further delay in restoring service."
While RIM now says the system "continues to operate normally," it also noted that the company had not yet completed its analysis of the failure or fully developed a way to avoid it in the future.

"Proper analysis can take several days or longer and RIM’s commitment is to provide the most accurate and complete information possible in such situations." The company did confirm that the fault was not related to "core software infrastructure," hardware, a security breach or a lack of capacity.

I've already forgiven the Men with BlackBerries for the little hiccup in service. I'm just glad to have my BlackBerry up and running. But comes on Boys, lets make sure it doesn't happen again!