We have tackled many tough situations during my series of posts about why enterprises need a great backup strategy. We have looked at many unexpected infrastructure outages in many different capacities: On-premises, Office 365 and this time ransomware. None of these scenarios were fun to live through, and this one will not be any different.
So, what happens when ransomware impacts your enterprise servers? During this installment, I will dive into some of the unexpected issues that can occur when virtualized environments cannot be completely locked down due to business requirement, and how proper data protection is necessary to ensure minimal downtime.
Virtualization and Internet Access
Many enterprises rely on virtualized environments as a form of remote desktop or remote application for employees to do their job. For example, when I am remote I may not use a software VPN on my laptop, but may use a web-based secure gateway to access my corporate applications. The server on the other side where the application is essentially providing me access to a remote desktop or application housed in the corporate data center. Yes, this could also be configured in the cloud, but in this particular real-world scenario everything was on-premises.
In these configurations it is not unusual to allow internet access as well. In today’s modern work-world we would want internet access to do our jobs. In some configurations I have restricted access to certain internet sites within these virtual deployments, but let’s face it – restricted or not there is still some access from a server to the internet.
So what happened, and what actions were taken to resolve?
As one of the administrators of this environment we had things as secure as they could be for the work that the enterprise users needed to be doing when remote. On an ordinary day, a remote user connected to their web-based application to complete some work. Next, they switched focus to the internet to do some research. During their research process they clicked on an ad that turned out to have ransomware buried in it. Thankfully the enterprise was well educated around what to do if they click on something that didn’t seem right, so after the click the user called in an urgent ticket to the service desk.
After being quickly engaged, it was clear that the server they were connected to had been compromised by ransomware. First, the virtual NIC was disconnected to ensure that anyone else using this server couldn’t spread the ransomware into their backups or to any other systems. Yes, this disrupted their connections, but there was so much more at risk. Each user was contacted, so that they understood why their sessions were no longer functioning.
Then investigation needed to occur around whether any other servers were being held by ransomware. By leveraging some of the security tools in the enterprise, we quickly learned this was the only compromise.
Finally, what to do with the compromised server? Due to a great backup strategy for production servers, we were able to recover that server from backup to a state before the ransomware within 10 minutes, and without any data loss. Which was a great success!
Lessons Learned
- Educate Users – Great security education in the environment ensured that if something didn’t seem right to call it in right away. Reporting quickly ensured this incident was contained.
- Timely backups – Having backups – especially backups that are up to date and current – will ensure that important business data doesn’t become lost when a recovery is required.
- Great backup solutions allow for quick recoveries – A solution that can meet enterprise RTOs and RPOs will ensure system uptime even during the unexpected.
Key takeaways: If you have been following this series of post it’s becomes clear that across all of these real-world problems that your enterprise should always be ready for the unexpected. Corporate data is a business asset, and being able to get your systems back online quickly reduces the enterprise impact of major outages and ensures employee productivity.
You can find my many previous posts from the real world here:
- Why Enterprises Need a Great Backup Strategy
- Why Enterprises Need Great Backups: Server Success Story
- Why Enterprises Need Great Backups: Office 365 Exchange Online Edition
