Veeam and Azure help government agency avoid a third system crash

The business challenges

There are two words that strike terror in the heart of any IT professional: system crash. If the next two words are data loss, the situation escalates from bad to worse.

That’s what happened when an IT system containing private identifiable information (PII) including employees’ names, addresses and social security numbers crashed at a federal government agency.

“We were backing up and replicating our PII system three times each day to four locations on-premises,” explained the IT Project Manager who experienced the situation firsthand. “We thought there was no way this system could ever crash, but we were dead wrong.”

The first crash occurred two days before Thanksgiving. Several members of the IT team were taking time off, so they immediately returned to the office and spent the next 36 hours trying to resolve the problem. Even though they had hardware redundant storage for the virtual server platform, the aged storage just crashed without warning; it corrupted most of the VM data.

“We tried restoring the 30 virtual machines (VMs) supporting the PII system from backups, but our backup software (Syncsort) failed us miserably,” he said. “At one point we feared the system was unfixable and data was lost, which meant 3,000 federal employees wouldn’t be getting their next paychecks.”

The IT team refused to accept defeat and continued troubleshooting. Their only recourse was to rebuild the 30 VMs from scratch. It took several hours, but it worked.

“We breathed a big sigh of relief, but we knew the system would crash again, and it did—four weeks later, just before Christmas. Again, some of us were home celebrating with our families, so we returned to the office and found that our backup software had failed again. I told my boss I wanted to replace it with Veeam, but he said we didn’t have the funds. That’s when I put my career on the line.”

The Project Manager worked with Veeam Software’s federal team to create a proof-of-concept test that overwhelmingly convinced his boss to replace legacy backup.

“We bet on Veeam, we won, and we’ve never looked back,” he said.

The Veeam solution

The government agency replaced legacy backup with Veeam Backup & Replication to protect all IT systems. Veeam extends data protection to the cloud, verifies the recoverability of backups and recovers from them minutes, not hours like legacy backup. Veeam also supports regulatory compliance with the Privacy Act and the Federal Information Security Management Act (FISMA).

“Veeam and Azure are core components of our DR strategy,” said the Project Manager. “We know we can restore our systems fast in any emergency, giving us unparalleled peace of mind.”

Veeam backs up and replicates 32 TB across 400 physical servers and VMs from NetApp storage snapshots on-premises in a Cisco UCS FlexPod infrastructure. Veeam also sends backup copies to Azure. If there’s an emergency Veeam Direct Restore to Microsoft Azure lets the IT team restore on-prem physical and virtual workloads to Azure in minutes.

“Sending backup copies to the cloud wasn’t possible before we deployed Veeam,” the Project Manager said. “Being able to restore our entire onprem infrastructure to Azure from backups gives us a new level of disaster preparedness.”

Veeam enables additional “firsts” for the agency, and they’re all because of Veeam DataLabs. This feature provides safe, isolated areas outside of the production environment to verify the recoverability of backups (SureBackup), recover in minutes (Instant VM Recovery and Instant File- Level Recovery) and test, develop and troubleshoot (On-Demand Sandbox for Storage Snapshots).

“Veeam provides many more features than we expected in a backup solution,” he said. “We’re really happy with its performance and trust it to protect our IT systems.”

More than 1,200 federal agencies trust Veeam to protect their critical systems. Veeam Software’s federal team understands and supports their needs, requirements and procurement processes. To simplify and streamline processes, Veeam partners with industry-leading hardware and software companies as well as system integrators. Veeam is fully certified and compliant to work with all agencies.

Compliance is a key requirement in government. Veeam supports agency compliance with regulations including the Privacy Act and FISMA. Agencies know their data is recoverable and available around the clock.

“We’re a success story because of Veeam,” he said. “I just wish we had deployed it sooner.”

The results

• Protects critical systems by extending data protection to the cloud
  Veeam backs up and replicates physical and virtual workloads on-premises and sends backup copies to Azure. The agency’s IT team can restore any workload to Azure, providing a new level of disaster preparedness.
• Verifies recoverability of backups and recovers in minutes, not hours
  SureBackup is a gamechanger for government agencies because it automatically verifies the integrity of a backup and recoverability. A similar feature called SureReplica verifies the recoverability of replicas.
• Supports compliance with regulations such as the Privacy Act and FISMA
  Federal, state and local governments are held accountable for properly handling large volumes of sensitive data. Veeam ensures the protection of data and provides seamless audit capabilities including reporting and documentation.