Our friends at BoxTone hit us up with an email earlier today with some information of how things played out yesterday from both their and their customers' perspective. We've talked about BoxTone lots of times here before on CrackBerry - they focus on enterprise companies that have a LOT of BlackBerry Smartphones running and provide monitoring and managements services. An afternoon like yesterday is where their services really make an impact. Here's what they told us (all times EST):
- Between 3:00 and 4:00 PM - Problems with BBM and BIS internet browsing reported around the web (I personally experienced around 3:30 PM).
- Between 6:30 and 7:00 PM - The problem extended to BES email, preventing the delivery of BES emails to and from BlackBerry smartphones. At each of our customers, BoxTone detected a greater than normal quantity of users with messages pending, based on our learned baseline of what is normal for each server and carrier, and immediately generated a warning alert our customers before the flood of user calls (Sample email alert below). BoxTone also placed all affected BES and Carriers in a Critical state on our customers' Operations Dashboards (depicted by the red dots next to each BES and carrier). The steady growth in Pending Messages beginning around 6:45 is annotated in the attached screenshot and continued until the issue was resolved early this morning. From our monitoring data, it appears that BES were able to communicate with the RIM NOC throughout the outage; however, the NOC was unable to deliver messages.
- At approximately 12:09 AM, BoxTone detected a brief disconnect in the SRP connection of each BES to the NOC; it appears RIM reset the NOC SRP connection to complete their fixes. Following this reset, delivery of BES mail resumed.
- By 2:45 AM or earlier, BoxTone detected that most of our customers had returned to their normal (baselined) service levels, and that the backlog of pending mail had been delivered. BoxTone generated notifications informing our users that their service levels had returned to normal and updated the status of the BES and carriers to Normal.
Pretty interesting stuff. Big thanks to Mitch and the BoxTone team for sharing it with us. You can learn more about their services at BoxTone.com.
