← Back to case studies

Private Client · Europe

The CrowdStrike Outage Wasn't a Cyberattack. It Was a Quality Problem.

On July 19, 2024, a faulty CrowdStrike update crashed 8.5 million Windows machines. One of our clients — a hospital network — lost their booking system for 14 hours. Here's what we did.

DevOpsIncident ResponseProcess Design

Measured results

  • Restored 340 hospital workstations in 14 hours
  • Implemented staged update deployment with automated rollback
  • Reduced update-related incidents by 90%
  • Built vendor risk assessment process for critical systems

Challenge

The blocker

When the CrowdStrike update crashed 8.5 million Windows machines globally, one of our clients lost their entire hospital booking system for 14 hours. Here's how we got them back online and redesigned their update process so it can't happen again.

Solution

The approach

We recovered their systems using Microsoft's USB boot tool, redesigned their update deployment pipeline with staged rollouts and automated rollback, and built a vendor risk assessment process for critical infrastructure.

Execution story

On July 19, 2024, at 04:09 UTC, CrowdStrike released a configuration update for its Falcon sensor software. The update contained a logic error in Channel File 291 that caused Windows machines to crash with a blue screen of death (BSOD) on reboot.

Within hours, 8.5 million Windows devices were offline. Airlines grounded flights. Banks couldn't process transactions. Hospitals couldn't access patient records. 911 call centers went down. It was, as the BBC put it, possibly the worst cyber event in history — and it wasn't a cyberattack. It was a quality problem.

One of our clients — a hospital network with 340 workstations across three sites — lost their entire patient booking and records system. Surgeries were postponed. Emergency departments reverted to paper. We got the call at 6 AM.

The first 14 hours

The recovery was manual and slow. The fix required booting each affected machine from a USB drive, navigating to the CrowdStrike directory, and deleting the problematic channel file. For 340 machines, that's a lot of USB drives.

"We currently estimate that CrowdStrike's update affected 8.5 million Windows devices, or less than one percent of all Windows machines. While the percentage was small, the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services." — Microsoft, July 20, 2024

We triaged by criticality:

PrioritySystemsCountRestored by
P0Emergency department workstations45Hour 4
P1Surgery scheduling and patient records80Hour 8
P2Pharmacy and lab systems60Hour 11
P3Administrative and non-clinical155Hour 14

Microsoft released a recovery tool that automated the USB boot process, which sped up the later phases. But the first 4 hours were pure manual work, with IT staff running between machines with USB drives.

Why it happened

CrowdStrike's Falcon sensor uses channel files — configuration updates that define what the sensor monitors. Channel File 291 monitors named pipe execution on Windows. The faulty update contained a logic error that caused the sensor to access invalid memory, triggering a BSOD.

The update passed CrowdStrike's internal testing but failed on certain Windows configurations that weren't represented in their test environment. Specifically, the error occurred on systems running a particular combination of Windows version, driver configuration, and sensor version.

The deeper issue was deployment velocity. CrowdStrike pushed the update to all customers simultaneously, with no staged rollout. A configuration update that affects kernel-level behavior went to 8.5 million machines at once.

"This is a function of the very homogenous technology that goes into the backbone of all of our IT infrastructure. What really causes this mess is that we rely on very few companies." — Gregory Falco, Cornell University

What we changed

After the recovery, we sat down with the hospital's IT leadership and redesigned their approach to third-party updates on critical systems.

Staged update deployment

We built a staged deployment pipeline for all third-party updates on critical infrastructure:

StageEnvironmentWait TimeAction on Failure
1Test lab (10 machines)24 hoursAuto-rollback, alert
2Pilot group (50 machines)48 hoursAuto-rollback, alert
3Production (remaining)ContinuousAuto-rollback, alert

Updates that pass Stage 1 automatically promote to Stage 2 after 24 hours of clean operation. Updates that pass Stage 2 promote to Stage 3 after 48 hours. Any failure at any stage triggers automatic rollback and alerts the team.

This adds 3 days to the update cycle for critical systems. For a hospital, that's an acceptable trade-off.

Automated rollback

We configured their endpoint management system to automatically revert to the previous configuration if any machine in the current stage crashes within 2 hours of the update. This catches the exact pattern that caused the CrowdStrike outage: an update that passes initial validation but crashes machines on reboot.

Vendor risk assessment

We built a vendor risk assessment process for any software that runs at the kernel level or can push updates without human approval:

  • Deployment model: Staged rollout or simultaneous? (Staged is required.)
  • Rollback capability: Can the vendor automatically revert a bad update? (Required.)
  • Test coverage: Does the vendor test against our specific OS and driver configurations? (Required for kernel-level software.)
  • Incident history: Has the vendor had a similar incident in the past 3 years? (Reviewable.)

Vendors that don't meet these criteria for critical systems get flagged for review. If they can't meet the criteria, we look for alternatives.

Business continuity

The hospital had a paper-based fallback for patient records, but it hadn't been tested in years. Staff didn't know where the forms were. The backup printer didn't work. We ran a quarterly business continuity drill where the EHR system was taken offline for 2 hours and staff had to operate on paper. The first drill was chaotic. The third was smooth.

The numbers

MetricBeforeAfter
Update deploymentSimultaneous, all machinesStaged, 3-day cycle
RollbackManual (hours)Automatic (minutes)
Update-related incidents4-6 per year0-1 per year
Business continuity drillsNeverQuarterly
Recovery time from major incident14 hours< 2 hours (projected)

What we learned

The CrowdStrike outage wasn't a security failure. It was a quality and deployment failure. The lesson isn't "don't use CrowdStrike" — it's "don't let any vendor push updates to your critical infrastructure without staged deployment and automated rollback."

The teams that weathered the outage best were the ones that had:

  1. Tested their business continuity plan recently (not 5 years ago)
  2. A staged update deployment process (even if it was manual)
  3. The ability to roll back updates quickly
  4. A clear inventory of what was running where

The teams that struggled had none of these. They trusted their vendors to ship good updates, and when a vendor shipped a bad one, they had no safety net.

Trust your vendors, but verify their updates. That's not paranoia. That's operations.

Want a result like this for your product?

Start a conversation →