Private Client · Europe
The CrowdStrike Outage Wasn't a Cyberattack. It Was a Quality Problem.
On July 19, 2024, a faulty CrowdStrike update crashed 8.5 million Windows machines. One of our clients — a hospital network — lost their booking system for 14 hours. Here's what we did.
Measured results
- Restored 340 hospital workstations in 14 hours
- Implemented staged update deployment with automated rollback
- Reduced update-related incidents by 90%
- Built vendor risk assessment process for critical systems
Challenge
The blocker
When the CrowdStrike update crashed 8.5 million Windows machines globally, one of our clients lost their entire hospital booking system for 14 hours. Here's how we got them back online and redesigned their update process so it can't happen again.
Solution
The approach
We recovered their systems using Microsoft's USB boot tool, redesigned their update deployment pipeline with staged rollouts and automated rollback, and built a vendor risk assessment process for critical infrastructure.
Execution story
On July 19, 2024, at 04:09 UTC, CrowdStrike released a configuration update for its Falcon sensor software. The update contained a logic error in Channel File 291 that caused Windows machines to crash with a blue screen of death (BSOD) on reboot.
Within hours, 8.5 million Windows devices were offline. Airlines grounded flights. Banks couldn't process transactions. Hospitals couldn't access patient records. 911 call centers went down. It was, as the BBC put it, possibly the worst cyber event in history — and it wasn't a cyberattack. It was a quality problem.
One of our clients — a hospital network with 340 workstations across three sites — lost their entire patient booking and records system. Surgeries were postponed. Emergency departments reverted to paper. We got the call at 6 AM.
The first 14 hours
The recovery was manual and slow. The fix required booting each affected machine from a USB drive, navigating to the CrowdStrike directory, and deleting the problematic channel file. For 340 machines, that's a lot of USB drives.
"We currently estimate that CrowdStrike's update affected 8.5 million Windows devices, or less than one percent of all Windows machines. While the percentage was small, the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services." — Microsoft, July 20, 2024
We triaged by criticality:
| Priority | Systems | Count | Restored by |
|---|---|---|---|
| P0 | Emergency department workstations | 45 | Hour 4 |
| P1 | Surgery scheduling and patient records | 80 | Hour 8 |
| P2 | Pharmacy and lab systems | 60 | Hour 11 |
| P3 | Administrative and non-clinical | 155 | Hour 14 |
Microsoft released a recovery tool that automated the USB boot process, which sped up the later phases. But the first 4 hours were pure manual work, with IT staff running between machines with USB drives.
Why it happened
CrowdStrike's Falcon sensor uses channel files — configuration updates that define what the sensor monitors. Channel File 291 monitors named pipe execution on Windows. The faulty update contained a logic error that caused the sensor to access invalid memory, triggering a BSOD.
The update passed CrowdStrike's internal testing but failed on certain Windows configurations that weren't represented in their test environment. Specifically, the error occurred on systems running a particular combination of Windows version, driver configuration, and sensor version.
The deeper issue was deployment velocity. CrowdStrike pushed the update to all customers simultaneously, with no staged rollout. A configuration update that affects kernel-level behavior went to 8.5 million machines at once.
"This is a function of the very homogenous technology that goes into the backbone of all of our IT infrastructure. What really causes this mess is that we rely on very few companies." — Gregory Falco, Cornell University
What we changed
After the recovery, we sat down with the hospital's IT leadership and redesigned their approach to third-party updates on critical systems.
Staged update deployment
We built a staged deployment pipeline for all third-party updates on critical infrastructure:
| Stage | Environment | Wait Time | Action on Failure |
|---|---|---|---|
| 1 | Test lab (10 machines) | 24 hours | Auto-rollback, alert |
| 2 | Pilot group (50 machines) | 48 hours | Auto-rollback, alert |
| 3 | Production (remaining) | Continuous | Auto-rollback, alert |
Updates that pass Stage 1 automatically promote to Stage 2 after 24 hours of clean operation. Updates that pass Stage 2 promote to Stage 3 after 48 hours. Any failure at any stage triggers automatic rollback and alerts the team.
This adds 3 days to the update cycle for critical systems. For a hospital, that's an acceptable trade-off.
Automated rollback
We configured their endpoint management system to automatically revert to the previous configuration if any machine in the current stage crashes within 2 hours of the update. This catches the exact pattern that caused the CrowdStrike outage: an update that passes initial validation but crashes machines on reboot.
Vendor risk assessment
We built a vendor risk assessment process for any software that runs at the kernel level or can push updates without human approval:
- Deployment model: Staged rollout or simultaneous? (Staged is required.)
- Rollback capability: Can the vendor automatically revert a bad update? (Required.)
- Test coverage: Does the vendor test against our specific OS and driver configurations? (Required for kernel-level software.)
- Incident history: Has the vendor had a similar incident in the past 3 years? (Reviewable.)
Vendors that don't meet these criteria for critical systems get flagged for review. If they can't meet the criteria, we look for alternatives.
Business continuity
The hospital had a paper-based fallback for patient records, but it hadn't been tested in years. Staff didn't know where the forms were. The backup printer didn't work. We ran a quarterly business continuity drill where the EHR system was taken offline for 2 hours and staff had to operate on paper. The first drill was chaotic. The third was smooth.
The numbers
| Metric | Before | After |
|---|---|---|
| Update deployment | Simultaneous, all machines | Staged, 3-day cycle |
| Rollback | Manual (hours) | Automatic (minutes) |
| Update-related incidents | 4-6 per year | 0-1 per year |
| Business continuity drills | Never | Quarterly |
| Recovery time from major incident | 14 hours | < 2 hours (projected) |
What we learned
The CrowdStrike outage wasn't a security failure. It was a quality and deployment failure. The lesson isn't "don't use CrowdStrike" — it's "don't let any vendor push updates to your critical infrastructure without staged deployment and automated rollback."
The teams that weathered the outage best were the ones that had:
- Tested their business continuity plan recently (not 5 years ago)
- A staged update deployment process (even if it was manual)
- The ability to roll back updates quickly
- A clear inventory of what was running where
The teams that struggled had none of these. They trusted their vendors to ship good updates, and when a vendor shipped a bad one, they had no safety net.
Trust your vendors, but verify their updates. That's not paranoia. That's operations.