As an enterprise admin for over 20 years now, I have seen many technology successes, challenges, and even failures throughout my career. For all enterprise admins, learning is a critical element of job success, but so is sharing what you have learned with others.
Today I am embarking on a series of posts where I will be sharing many technology challenges that could have been avoided with a proper backup strategy. From there we will touch on how a modern backup solution can protect your organization from the preventable issues I’ll be writing about.
It’s Budget Time
Let’s face it, not all enterprises have large budgets for IT and may need to spread dollar resources thin. In working with organizations over the years, I have seen this over and over. Often, the backup strategy is the first to get scrutinized as a way to cut costs. Saving server backups can consume large amounts of storage, and a lack of proper planning or budget can leave companies in a crunch.
Development and Testing Environments
Another area I have commonly seen cut is the backups of development and testing environments. The name DevTest can make these workloads sound disposable, and easily rebuildable. However, I have seen firsthand how the decision to cut DevTest backups can severely impact an enterprise.
Case in point: I was doing some infrastructure work for an enterprise, and they called me about a major outage they were experiencing. Their storage array that primarily hosted all their development and testing environments had lost over 100 disks due to a bug in the firmware applied to the storage. Yes, a 100-disk failure in a SAN! Their backup strategy for development and testing was to not do backups. In this environment, the impact spanned about 80 servers, and all that data was just simply lost. This loss also impacted productivity.
At some point in the past, the team had decided these environments were disposable. The problem with this strategy was that many of the test systems were used to prove out code for internal application development before it went into production. Development of these applications was ongoing, and as a result, most of this enterprise’s custom applications were now stalled, and much work was lost.
Teams scrambled and a complete rebuild process was started once system priorities were captured. Without any backups, it took more than a month to recover from this outage.
Lessons Learned
Time is money. With the extent of this outage, there were no savings gained by passing on the backups of development and testing environments.
Do not work in an IT silo. Make your backup decisions with application owners and do not base your decisions from assumptions.
Backup Development and Testing environments with business-driven SLAs (service level agreements) in mind.
Know your backup RPOs (recovery point objective) and RTOs (recovery time objective). Check your current development and testing backup SLAs to determine if any changes need to be made immediately to properly support your business.
Stay tuned for next time when I talk about unexpected issues when automating desktop deployments and how backups can ensure a quick recovery.
