Microsoft has released a custom WinPE recovery tool to find and remove the faulty CrowdStrike update that crashed an estimated 8.5 million Windows devices on Friday.
On Friday, CrowdStrike pushed out a faulty update that caused millions of Windows devices worldwide to suddenly crash with a Blue Screen of Death (BSOD) and enter reboot loops.
This glitch caused massive IT outages, as companies suddenly found that all of their Windows devices no longer worked. These IT outages affected airports, hospitals, banks, companies, and government agencies worldwide.
To resolve the fix, admins needed to reboot impacted Windows devices into Safe Mode or the Recovery Environment and manually remove the buggy kernel driver from the C:\Windows\System32\drivers\CrowdStrike folder.
However, as organizations face hundreds, if not thousands, of impacted Windows devices, manually performing these fixes can be problematic, time consuming, and difficult.
To help IT admins and support staff, Microsoft has released a custom recovery tool that automates the removal of the buggy CrowdStrike update from Windows devices so that they can once again boot normally.
"As a follow-up to the CrowdStrike Falcon agent issue impacting Windows clients and servers, we have released a USB tool to help IT Admins expedite the repair process," reads a Microsoft support bulletin.
"The signed Microsoft Recovery Tool can be found in the Microsoft Download Center: https://go.microsoft.com/fwlink/?linkid=2280386."
To use Microsoft's recovery tool, IT staff need a Windows 64-bit client with at least 8 GB of space, administrative privileges on this device, a USB drive with at least 1 GB of storage, and a Bitlocker recovery key if required.
It should be noted that you will need a USB flash drive that has a 32GB partition or smaller, as otherwise you will not be able to format it with FAT32, which is required to boot the drive.
The recovery tool is created through a PowerShell script downloaded from Microsoft, which needs to run with Administrative privileges. When run, it will format a USB drive and then create a custom WinPE image, which is copied to the drive and made bootable.
Source: BleepingComputer
You can then boot your impacted Windows device with the USB key, and it will automatically run a batch file named CSRemediationScript.bat.
Source: BleepingComputer
This batch file will prompt you to enter any necessary Bitlocker recovery keys, which can be retrieved using these steps.
The script will then search for the buggy CrowdStrike kernel driver in the C:\Windows\system32\drivers\CrowdStrike folder, and if it's detected, automatically delete it.
BleepingComputer's tests and review of the batch file show that it will not create any logs or a backup of the CrowdStrike driver.
When completed, the script will prompt you to press any key, and your device will reboot.
Now that the CrowdStrike driver has been deleted, the device should boot back into Windows and be available again.
Unfortunately, Windows admins' biggest obstacle is retrieving any necessary Bitlocker recovery keys.
Therefore, determining if one is needed and recovering it should be the first steps taken before attempting to recover devices.
Update 7/22/24: Clarified that your USB drive must have a 32GB partition or smaller. Thx joshuayoder.
Comments
ZeroYourHero - 3 months ago
OK, thanks, but for all us who had to recover hundreds of machines where WinRE couldn't even access the C: drive to delete the bad driver file maybe you could do something to keep the WinRE environment up to date with the signed drivers necessary to access to C: drive? Am I asking too much here?? Maybe on ALL supported versions of Windows?? Yeah, I know, nothing but crickets from the billionaires in charge.
GT500 - 3 months ago
Could always make a Win10XPE ISO to use on a USB flash drive. It's been a while since I've tried something like that, but you used to be able to add your own drivers (as well as customize the software that will be included) before building the ISO.
There's also a decent list of third-party Windows PE bundles at the following link that may have better driver support than the Windows Recovery Environment, although I'm not certain about the legality of downloading a Windows PE ISO from a source other than Microsoft (if any of them allow you to build the disk yourself from a Windows ISO then it should be fine)
GT500 - 3 months ago
Looks like the link was removed. It's better to build your own PE disk than try to download one anyway. Less chance of violating the terms of the Windows EULA that way.
pseymour - 3 months ago
Add your own drivers to WinRE. Takes almost no time. I doubt MSFT could add all necessary drivers for all hardware to a recovery partition that is usually tiny.
GT500 - 3 months ago
Typo: "Safe More"
jkr4m3r - 3 months ago
"Typo: "Safe More""
The entire post is a mess of unformatted HTML from my POV.
GT500 - 3 months ago
The typo was in the article, and has since been fixed.
As for my post, someone appears to have edited it to remove a link, and it broke the formatting. It was a link to an article about third-party Windows PE versions (such as Hiren's BootCD PE) that you can download, so I can understand why it was removed.
joshuayoder - 3 months ago
"It should be noted that you will need a USB flash drive that is 32GB or smaller, as otherwise you will not be able to format it with FAT32, which is required to boot the drive."
Advanced Tip:
Technically the drive itself can be bigger than 32GB, which such drives are extremely common now. In the event the flash drive is LARGER than 32GB, deleting the partition(s) from the drive, then partition it to have a 32GB Partition and then that partition can be formatted to FAT32.
Later the drive can be re-partitioned to support the full capacity.
Lawrence Abrams - 3 months ago
Thanks and good tip. Updated the article and clarified that point.
jmwoods - 3 months ago
I would consider creating a Linux Live USB, boot from that, and delete the Windows\system32\drivers\CrowdStrike folder from there.