November « 2009

8:30AM PST – Happy Thanksgiving.  Unfortunately, there is an issue with logging into the control panel and some functions for the API.  Sorry for the inconvenience.  We are checking on it.

More updates to come shortly…

Update: 9:12AM PST- Engineers are still checking

Update: 9:34AM PST – this is resolved.  The root cause is human error.  We are adding some safeguards to make sure it doesn’t happen again.  Happy Thanksgiving.

Mzima did an emergency roll back of the code revisions done on Sunday starting at 8:40PM PST.

For the most part, services were uninterrupted.

It seems like there was a bug in the Force10 Core upgrade they did on the switches.  Customers may have noticed brief outages in Los Angeles from 11:41 to 11:45PM PST for Los Angeles routed traffic.  For the most part, the Anycast network routed things very well but we are investigating how to prevent any outage from this type of upgrade.

As of 12:03PM PST, we expect everything to be back to normal and all Cores are rolled back.  If you received any reports of issues, please get in touch with us.

Thanks to the Mzima team for their hard work in restoring service.

Mzima is  doing a Core Router Software Upgrade.  It is planned for no downtime – as anycast will route as each router is taken down for an upgrade.  This type of software update happens every 6 months or so to the core routers to keep them up to the latest code base.