We use Volusion Ecommerce for a number of client website projects and we rely on our third party vendors to provide 24×7 reliable web hosting and ecommerce solutions.  Active ecommerce websites can lose thousands of dollars if outages last just a few hours.

It seems that some customers or industry professionals were ticked off at the lack of transparency from Volusion initially.   Read this post from Patrick:  Volusion PR Hell

I commend Clay Olivier, CTO for Volusion on his openness about the Volusion hardware failure that was caused by a defect in an HP Blade Server.  Here is the letter that was sent to clients and the technical explanation about the Volusion Ecommerce outage from Clay Olivier.

As you likely noticed, we experienced a critical hardware failure yesterday that resulted in your storefront not functioning properly. We take full responsibility for this incident; our mission is to help you succeed online, and yesterday we didn’t live up to that promise. We apologize for the issues this caused you and your customers.

In addition to the technical issues, we also did not communicate with you throughout the incident as we should have. In our efforts to get your store back up and running, we neglected to keep you informed.

While I cannot promise that problems will never arise, I can promise that we are committed to mitigating future issues in a way that minimizes the impact on your business.

I’ve attached a technical explanation of what occurred yesterday to the bottom of this message.

Sincerely,
Clay Olivier
Chief Operating Officer
Volusion
TalkToClay@volusion.com

Volusion Technical Root Cause Analysis

Early yesterday, one of our blade server chassis lost connectivity to our Storage Area Network. This resulted in many of our customers experiencing downtime or errors on their storefronts.

Upon identification of the issue, we immediately brought HP on-site. After extensive troubleshooting, HP engineers diagnosed the issue as a failed mid-plane within the chassis. The engineers replaced the mid-plane, and all connections were restored.

This is the second HP midplanethat has failed in the past 30 days. While the chances of two failures of a similar nature are very remote, HP was able to confirm this to be the case. Based on the serial numbers of the two failed components, we have reason to believe that both came from a bad batch. We are researching this further with the help of HP’s labs, as well as performing a full audit on all of our HP equipment.

We recognize the impact that yesterday’s outage may have had on your business. We are working diligently with HP to prevent a recurrence of this issue in the future.