Update 1/17/21: Microsoft has released OOB updates to fix the Windows Server bugs.
The latest Windows Server updates are causing severe issues for administrators, with domain controllers having spontaneous reboots, Hyper-V not starting, and inaccessible ReFS volumes until the updates are rolled back
Yesterday, Microsoft released the Windows Server 2012 R2 KB5009624 update, the Windows Server 2019 KB5009557 update, and the Windows Server 2022 KB5009555 update as part of the January 2022 Patch Tuesday.
After installing these updates, administrators have been battling multiple issues that are only resolved after removing the updates.
Windows domain controller boot loops
The most serious issue introduced by these updates is that Windows domain controllers enter a boot loop, with servers getting into an endless cycle of Windows starting and then rebooting after a few minutes.
As first reported by BornCity, this issue affects all supported Windows Server versions.
"Looks KB5009557 (2019) and KB5009555 (2022) are causing something to fail on domain controllers, which then keep rebooting every few minutes," a user posted to Reddit.
A Windows Server administrator told BleepingComputer that they see the LSASS.exe process use all of the CPU on a server and then ultimately terminate.
As LSASS is a critical process required for Windows to operate correctly, the operating system will automatically restart when the process is terminated.
The following error will be logged to the event viewer when restarting due to a crashed LSASS process, as another user on Reddit shared.
"The process wininit.exe has initiated the restart of computer [computer_name] on behalf of user for the following reason: No title for this reason could be found Reason Code: 0x50006 Shutdown Type: restart Comment: The system process 'C:\WINDOWS\system32\lsass.exe' terminated unexpectedly with status code -1073741819. The system will now shut down and restart."
Hyper-V no longer starts
In addition to the boot loops, BleepingComputer has been told by Windows administrators that after installing the patches, Hyper-V no longer starts on the server.
This bug primarily affects Windows Server 2012 R2 server, but other unverified reports say it affects newer versions of Windows Server.
As Hyper-V is not started, when attempting to launch a virtual machine, users will receive an error stating the following:
"Virtual machine xxx could not be started because the hypervisor is not running."
Microsoft released security updates to fix four different Hyper-V vulnerabilities yesterday (CVE-2022-21901, CVE-2022-21900, CVE-2022-21905, and CVE-2022-21847), which are likely causing this issue.
ReFS file systems are no longer accessible
Finally, numerous admins are reporting that Windows Resilient File System (ReFS) volumes are no longer accessible or are seen as RAW (unformatted) after installing the updates.
The Resilient File System (ReFS) is a Microsoft proprietary file system that has been designed for high availability, data recovery, and high performance for very large storage volumes.
"Installed these updates tonight, in a two server Exchange 2016 CU22 DAG, running on Server 2012 R2. After a really long reboot, the server came back up with all the ReFS volumes as RAW," explained a Microsoft Exchange administrator on Reddit.
"NTFS volumes attached were fine. I realize this is not exclusively an exchange question but it is impacting my ability to bring services for Exchange back online."
Uninstalling the Windows Server updates made the ReFS volumes accessible again.
Yesterday, Microsoft fixed seven remote code execution vulnerabilities in ReFS, with one or more likely behind the inaccessible ReFS volumes.
These vulnerabilities are tracked as CVE-2022-21961, CVE-2022-21959, CVE-2022-21958, CVE-2022-21960, CVE-2022-21963, CVE-2022-21892, CVE-2022-21962, CVE-2022-21928.
How to fix?
Unfortunately, the only way to fix these issues is to uninstall the corresponding cumulative update for your Windows version.
Admins can do this by using one of the following commands:
Windows Server 2012 R2: wusa /uninstall /kb:KB5009624
Windows Server 2019: wusa /uninstall /kb:KB5009557
Windows Server 2022: wusa /uninstall /kb:KB5009555
As Microsoft bundles all security fixes into the single update, removing the cumulative update may fix the bugs, but will also remove all fixes for recently patched vulnerabilities.
Therefore, uninstalling these updates should only be done if absolutely necessary.
Not to be outdone by Windows Server, Windows 10 and Windows 11's updates are also breaking L2TP VPN connections.
On January 17th, Microsoft released out-of-band updates to fix the following issues:
- Windows L2TP VPN connection issues
- Domain controller reboots
- Hyper-V not starting
- ReFS volumes becoming inaccessible
More information about these updates can be found in our dedicated "Microsoft releases emergency fixes for Windows Server, VPN bugs" article.
Update 1/17/21: Added information about the OOB updates released to fix these issues.
Comments
sonic111 - 2 years ago
I can also confirm that Windows Server 2012 KB5009586 also causes the same issue and that uninstalling the update fixes it
gleep52 - 2 years ago
Hyper-V working fine on my 2016 Cluster.
DCs on 2019 haven't crashed. LSASS using maybe 0-1% CPU.
VPN is dead on all systems. :(
Microsoft - are you trying to one-up Log4J work?
dparmentier - 2 years ago
Can we also send Microsoft an invoice for additional hourly services ?
It is definitely no longer reliable... :-(
is149au - 2 years ago
Also wrecks Exchange 2013. Uninstalling KB5009624 fixed the issue though.
gleep52 - 2 years ago
What was the bug with Exchange? I have an exchange 2013 CU23 and it doesn't seem to be having issues? Would the services not start, or something specific?
Zurv - 2 years ago
Are these core Hyper-V installs or windows installs with Hyper-V?
We are running about 40 hyper-v core servers (2016 and 2019) - haven't patched them yet :P
troyv808 - 2 years ago
This drove me crazy last night, I thought we had hardware failure, check the BIOS and everything else and sure enough after I removed the patches, everything works - can start the Servers on VM again.
BTW - you're the first to report this - I googled last night and this morning and couldn't anything about this!
NoneRain - 2 years ago
Some places where you can see earlier reports from admins:
Reddit /r/sysadmin
Spiceworks: https://community.spiceworks.com/windows/microsoft-windows-server
Trevez - 2 years ago
seem to boot every 15 mins. So you uninstall as per this advice and it boots during the uninstall (** completely borked DC!) Suggest you boot into safe mode to uninstall. Ours doesn't seem to boot in safe mode. Either that or stop the net logon service as I just read on reddit. I think safe mode "safer" :-)
Also not affecting our azure 2019 DC's with this patch installed.
Update: Oh great now the update wont uninstall! :-( tried many different methods, after reboot get the error "we couldnt complete the updates, undoing changes" Ticket for MS on the way..
Michiel1981 - 2 years ago
Hey man,
The issue of reboots only happens if 2 or more DC's have the update installed. Just turn off 1 dc or boot it into safemode without networking and the other DC stops rebooting. Then you have atleast 1 DC up for people to continue work and you have time to uninstall the patches.
I had the same thing happen on 2012R2 and when i was in safe mode with 1 dc trying to uninstall the patch the other DC stopped rebooting. When i fired up the rolled back DC the other DC with the update kept running fine. Offcourse i did uninstall the update anyway on it.
rekingus - 2 years ago
Yes!
Sprid - 2 years ago
I also read unplugging the network cable will stop the random reboots.
Lawrence Abrams - 2 years ago
Seriously? Unplugging the network cable stops the reboots?
Tjackson - 2 years ago
yup unplugging (or disabling) works. stops rebooting with enough time to uninstall.
mbeckwith - 2 years ago
I have instructions in the comments below about how to remove it when it won't uninstall. Had the same problem and had to call Microsoft. Use command line >Dism
unclebloodyfester - 2 years ago
Just had it here - takes a good while to uninstall ( 20 + minutes) and the issue is not fixed until you reboot after removing the patch which is where it sits on 'working on updates' but as soon as it has rebooted and got to that screen, the server is to all intents and purposes operational.
Thankfully nothing running HyperV is a DC so that seems to have limited the damage.
KB5009557 for 2019
KB5009624 for 2012
KB5009546 for 2016
Although it seems that 2019 and 2012 were the ones that hated it the most. Was wondering what the occasional increase in server whine was whilst i was drinking coffee - 2019 and 2012 servers doing alternate reboots.
Removed all three from all affected servers just in case .
QuangoUK - 2 years ago
One of our HyperV server 2012 hosts stopped running HyperV yesterday - a day lost trying to figure out why, then moving the VMs to a different machine. Thanks MS
rebirth13 - 2 years ago
Thanks for this. Good info.
chromeskull - 2 years ago
Thanks to bleeping computer being displayed on Google News & Interests I was able to uninstall KB5009557 which sat pending restart.. still waiting on a restart but at least now it's to remove the update.
Typical I finally get to replace my old SBS20011 server and updated to Server 2019 with new hardware to then get this sort of worry. Maybe the slow system wasn't so bad.
Matt64 - 2 years ago
Thanks for this info
Dorwin6 - 2 years ago
The moment you first logon.
1. Immediately run command prompt, right click, run as administrator (just incase)
2. type in "net stop netlogon" (shows The Netlogon service was stopped successfully.)
3. Program and Features
4. Select Uninstall (click on Security Update for Microsoft Windows (KB5009557)
5. You must restart your computer to apply these changes (click Restart Later)
6. Select Uninstall (click Update for Microsoft Windows (KB5008873)
7. You must restart your computer to apply these changes (click Restart Now)
Reboots and Walla.
wwonka72 - 2 years ago
I uninstalled the update KB5009586 on my Windows 2012 servers - but they continue to reboot - any ideas?
mbeckwith - 2 years ago
Are you sure it actually uninstalled? I uninstalled it multiple times and it kept reverting. After a very long day, I was able to uninstall the package the following way:
Using Admin powershell (not in safe mode, so you must be quick)
Look up all packages using >dism /online /get-packages
Find the correct package and copy the entire name of the package name by highlighting the name and right-clicking on the window bar >edit>copy
Type > dism /online /remove-package /packagename:PASTE FROM TITLE BAR FULL NAME OF PACKAGE THAT YOU COPIED
This will remove the update even when add/remove and the wusa command will not.
jbruns2022 - 2 years ago
This is a sample for 2012 but just change the fix number on line 4.
1. Logon to server with RDP (assumes admin rights)
2. Open powershell (run these commands copy/paste)
stop-service netlogon -force
$package = dism /online /get-packages | findstr 5009619 | foreach {$_.split(":")[1]} | foreach {$_.Trim()}
dism /online /remove-package /packagename:$package
3. It will prompt you if you want to restart computer now, provide Y
Michiel1981 - 2 years ago
Uninstall KB5009624
abdielhiram - 2 years ago
Were having a hard time rebooting in Safe Mode (2K19 DC's) unplugged the NIC allowed to uninstall the update.
lucasnooker - 2 years ago
Hi,
I'm wondering if anyone can help/advise me please? I have had issues with this since the update and now have been unable to remote in to any of the vm's on the hosts. I am unable to uninstall the update as I can't access the OS's. What is the best course of action from here in order to solve this issue? I have rebooted the hosts but they seem to be inaccessible whatever I do. I can ping some of them but not others but can't manage to establish a secure connection anymore...
mbeckwith - 2 years ago
have you tried going to the hyper v console and using connect rather than using remote desktop?
DarkLog - 2 years ago
I was just experimenting with this, and installed it on a RODC in a lab, so far on the Server 2022 RODC it's not causing a boot loop.
I manually downloaded KB500955 from the update catalog, and installed it.
After the reboot I checked update history and it shows as "Failed to install - 0xc1900401", but if I try to install again it shows already installed, and it is in the list of updates in the uninstall updates menu.
So I'm wondering does the issue maybe not impact RODC's? Or has MS done something to the update without notice?
Tjackson - 2 years ago
For those of you fighting with having enough time to get the update uninstalled. I was able to remove the NIC from HyperV settings and then i had all the time i needed to remove the update. just re add the nic once youre done.
bitanalyst - 2 years ago
I have a Server 2016 RODC that is still experiencing this reboot loop even after removing all of the January patches. It's running on Hyper-V, disabling networking keeps the machine running but as soon as networking is turned on it reboots again due to lsass.exe.
CazualFilth - 2 years ago
Server 2016 can be included in the list - KB5009546
I work for a gov agency that has a number of large domains and DCs all running 2016 OS. While not all our DCs were impacted by the reboot loop our busiest DCs in regards to lsass were impacted severely after patching. Removal of patch has stopped the reboots. Not ideal but there it is.
petrolej - 2 years ago
Hi folks, also having troubles with this. I was able to disconnect LAN from a VM with DC. This prevents the DC to reboot but then I am unable to login. The server says incorrect password or username. Does anyone have experience with this? Thanks.
CompKing - 2 years ago
What a panic that was! Ugh! Did the updates over the weekend and they didn't get noticed until the business was open..
Normally don't take updates until they are matured, but this time I was closing some loops due to a recent security focus with a breach.
Simple fix, but too a while to implement.. Ugh.
Anyone have any good sources of info for critical things such as this to be sent to you so you don't make this mistake again? Good source of Server concerns?
gleep52 - 2 years ago
Yes - you're on THEE site for it :) Bleeping computer is awesome - I follow them on telegram for updates so I get push notifications.
rinatin - 2 years ago
Microsoft lists KB5009619 AND KB5009586 as both causing reboots on 2012 Hyper-V domain controllers.
Lawrence Abrams - 2 years ago
Microsoft has released OOB updates that claim to fix these bugs.
https://www.bleepingcomputer.com/news/microsoft/microsoft-releases-emergency-fixes-for-windows-server-vpn-bugs/
If you can let us know in the above article if the updates fix the issues, it would be appreciated.
sc2111 - 2 years ago
Does anyone, as we are, have had the same issue with windows server 2016 ?
HARIOMSAI7 - 2 years ago
wmic qfe | find "5009624"
:: Windows Server 2012 R2:
wusa /uninstall /kb:5009624
:: Windows Server 2019:
wmic qfe | find "5009557"
wusa /uninstall /kb:5009557
:: Windows Server 2022:
wmic qfe | find "5009555"
wusa /uninstall /kb:5009555