The number 1 cause of emergency callouts for data centres that BACS encounters is water entering sub-floors. Water and electricity do not mix. Should they come in contact, downtime is a major risk. The most common source of flood water is leakage from CRAC and other air conditioning unit pipework. Less frequently, water can inundate a data centre from other areas of the building and even from outside. BACS has recently remediated a sub-floor contaminated with raw sewage resulting from a valve failure at the street level. We have also cleaned up storm water which collected on a roof, leaking past several floors to pool in a data centre. Unlike this dramatic vision of a flood at a data centre in Turkey or this account of an outage at a Melbourne data centre caused by a storm, most events are fortunately less destructive and rarely the result of natural disasters.
No matter what the cause, what would you do if you received a call at 2AM saying water was entering your sub-floor? Rapid response is essential. You must have a disaster recovery plan already in place to address these unforeseen events before they occur. Here are some actions to take as part of your plan.
Post in a prominent location the out-of-hours telephone numbers for:
- Emergency first-responders
- Person who can isolate data centre equipment
- Facility manager
- Vendor who has water removal equipment suitable for use in data centres
Keep a dedicated disaster recovery kit readily available onsite which includes:
The cost of a vacuum may mean you save many hours of valuable response time.
Data Centre Risk Management Series Part 2: Security Coming in March 2015