On April 8th 2009 at noon EST, after nearly six years without a serious network outage, DedicatedIT experienced a complete failure of our NetApp storage system. Our storage system is less than six months old, and is considered to be one of the best brands of storage systems in the industry. Because of the redundancy built into these systems, it is very unlikely to have a catastrophic failure like we experienced today. No data was lost.
Our techs were dispatched to our data center immediately, and NetApp priority technical support worked with them all day to restore service.
Replacement parts were delivered within two hours of the outage, but they did not fix the problem. A replacement storage system was delivered late in the evening and all data was migrated to the new system. That restored all services.
As an added precaution, a third storage system will be delivered Thursday.
I understand your frustration and know the impact this type of outage has on your business. This event brought down not only your business, but ours as well.
Our team will be reviewing our system over the next week and will determine how to mitigate this risk in the future. I will report back to you, as soon as a solid design is in place. I’ll be happy to explain, in as much detail as you would like, what happened, what we did to fix the problem and what we are going to do to keep it from happening again.
While this type of event can happen to any business using any man-made system, it is humbling and embarrassing to have it happen to us. I apologize.
Please contact me to discuss further.
Adam Steinhoff
President/CEO
DedicatedIT – a division of Steinhoff Consulting, Inc.
Direct: (561) 491-5725
Email: asteinhoff@DedicatedIT.com
Twitter: @adamsteinhoff
I neglected to mention that a client (Wicker Smith) came through with the replacement SAN at the last minute. Had we waited on NetApp’s replacement, we would have been down another 12 hours.
Thank you very much, Mike!