"Key backup and disaster recovery strategies for flood disaster management"
By Alan Earls
June 7 2010 – Jim Lippie, vice president, Thrive Networks, an IT services company that targets the SMB market, said companies are sometimes their own worst enemies when it comes to preparing for disasters. "More often than we'd like, we see equipment placement minefields in SMB environments -- whether it's improper ventilation and lack of an adequate cooling strategy, equipment littering the floor, or water damage waiting to happen," he said. For instance, Lippie said, at one company his team made an alarming discovery -- the server room also housed the company's hot water heater. "On the spot, our engineer insisted on setting up a tape backup," which later proved crucial to recovery when a leak did develop, said Lippie.
On a smaller scale, a countless number of organizations across the Northeast had to deal with unexpected flooding and flood disaster management this spring. Richard Booth, of the North Kingston, Rhode Island school department, was dealing with widespread flooding across his whole community. Booth is in charge of IT operations for the community's nine schools and manages 1.6 TB of "mostly mission-critical data." According to Booth, one of the buildings where he manages operations had a significant roof leak two years ago. That incident didn't affect his operations or cause any loss of data, but it was a wakeup call for his disaster recovery (DR) strategy. So, this spring, when the water began to rise across town, shutting down roads, flooding basements, and cutting off power, Booth made sure he was prepared for this type of disaster. He equipped his community with a pair of Exagrid disk backup appliances, an EX5000 and an EX3000. Booth installed the units in two locations, at opposite ends of the town. The investment made operations more secure and also eliminated the need for a manual data backup each weekend.
However, warned Lippie, counting on luck or last-minute fixes is a high-risk strategy. Thus, he recommends having a third-party assessment of your data backup environment "at least every two years" to look for and fix any potential hazards. Another option -- popular for its time and cost savings -- is looking into a collocation data center to house data center equipment. Lippie said Thrive Networks itself has taken this option. From 2000 to 2007, Thrive Networks experienced at least one power outage per year ranging from 10 minutes to several hours. So in early 2007, the company moved its infrastructure to a collocated facility. Now, Lippie said he "sleeps better at night." Lippie also said that Thrive Networks is also helping a financial services firm that processes thousands of financial transaction every day and can't afford downtime, to evaluate on premise infrastructure vs. "colo". According to Lippie, determining which strategy you want to go with comes down to space, redundancy, power, cooling and associated costs.
In terms of a data backup procedure, companies that use tape storage should have a daily tape rotation. Another data backup strategy to consider is offsite data backup through online data vaulting. "Without a proven data protection plan, they're simply taking an unacceptable risk," said Lippie.
Furthermore, he noted, one-size-fits-all approaches can leave you vulnerable. He cited the example of a client located on the top floor of a multi-level office building where the building roof was getting tarred. Unbeknownst to the company, there was a small hole in the roof above the server room. "Imagine their consternation when they discovered that tar had dripped down into the server room, coated the server and rendered it completely useless," he said. Fortunately, the company had implemented a tape rotation backup and they were able to get a replacement server up and running within a day.
"While companies don't necessarily need to have servers on standby, it's important to always have an accurate and up-to-date asset inventory, including equipment specs, so you can do an overnight order if necessary to preserve business continuity," said Lippie. Again, in terms of a comprehensive disaster recovery plan, Lippie said companies should have flexibility to adjust to not-so-typical disasters, and include measures for communicating to employees and distinguishing between short- and long-term emergencies.
Less exotic and far too common is the "sin" of poor data backup practices. Lippie said he has encountered many companies that perform daily backups but without testing. Instead, they simply change tapes from day to day without ever determining if the tapes are operating correctly. "We've gone into some prospects and discovered that they haven't had a successful backup in more than six months," he said. In fact, he said, studies have shown that more than a third of companies never test their backups, and of those that do, more than three quarters have found tapes that fail to restore properly. "Needless to say, doing test restores is absolutely a must," said Lippie.