After last patching (2018-03) “system writer” is missing and backup tools like wbadmin, Hiback, NBU or TSM failed to perform system state backup on several servers across multiple environment (physical, virtual).
Affected OS: Windows Server 2008 SP2 only
Installed KBs:
KB4090450
KB4089229
KB4089344
KB4087398
KB4089175
KB4056564
For non of this KB´s is this bug documented.
Error description per tool:
wbadmin error: “System writer is not found in the backup.” EventID 517, error code 2155348226
Hiback error: “finerr 213″
TSM (Tivoli Storage Manager) error:“Backups have ended with status 12 err on server …”
NetBackup: “The VSS System Writer is not found on this system. The backup is stopping because the System Writer is required for a successful system state backup. Have the system administrator check the VSS infrastructure and fix the issue. 86024:save: Error occured while saving disaster recovery save sets.”
Performed actions (no success):
Takeown /f %windir%\winsxs\temp\PendingRenames /a
icacls %windir%\winsxs\temp\PendingRenames /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\temp\PendingRenames /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\temp\PendingRenames /grant BUILTIN\Users:(RX)
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant BUILTIN\Users:(RX)
net stop cryptsvc
net start cryptsvc
- Re-register DLL´s, SFC check
net stop "System Event Notification Service"
net stop "Background Intelligent Transfer Service"
net stop "COM+ Event System"
net stop "Microsoft Software Shadow Copy Provider"
net stop "Volume Shadow Copy"
cd /d %windir%\system32
net stop vss
net stop swprv
regsvr32 /s ATL.DLL
regsvr32 /s comsvcs.DLL
regsvr32 /s credui.DLL
regsvr32 /s CRYPTNET.DLL
regsvr32 /s CRYPTUI.DLL
regsvr32 /s dhcpqec.DLL
regsvr32 /s dssenh.DLL
regsvr32 /s eapqec.DLL
regsvr32 /s esscli.DLL
regsvr32 /s FastProx.DLL
regsvr32 /s FirewallAPI.DLL
regsvr32 /s kmsvc.DLL
regsvr32 /s lsmproxy.DLL
regsvr32 /s MSCTF.DLL
regsvr32 /s msi.DLL
regsvr32 /s msxml3.DLL
regsvr32 /s ncprov.DLL
regsvr32 /s ole32.DLL
regsvr32 /s OLEACC.DLL
regsvr32 /s OLEAUT32.DLL
regsvr32 /s PROPSYS.DLL
regsvr32 /s QAgent.DLL
regsvr32 /s qagentrt.DLL
regsvr32 /s QUtil.DLL
regsvr32 /s raschap.DLL
regsvr32 /s RASQEC.DLL
regsvr32 /s rastls.DLL
regsvr32 /s repdrvfs.DLL
regsvr32 /s RPCRT4.DLL
regsvr32 /s rsaenh.DLL
regsvr32 /s SHELL32.DLL
regsvr32 /s shsvcs.DLL
regsvr32 /s /i swprv.DLL
regsvr32 /s tschannel.DLL
regsvr32 /s USERENV.DLL
regsvr32 /s vss_ps.DLL
regsvr32 /s wbemcons.DLL
regsvr32 /s wbemcore.DLL
regsvr32 /s wbemess.DLL
regsvr32 /s wbemsvc.DLL
regsvr32 /s WINHTTP.DLL
regsvr32 /s WINTRUST.DLL
regsvr32 /s wmiprvsd.DLL
regsvr32 /s wmisvc.DLL
regsvr32 /s wmiutils.DLL
regsvr32 /s wuaueng.DLL
sfc /SCANFILE=%windir%\system32\catsrv.DLL
sfc /SCANFILE=%windir%\system32\catsrvut.DLL
sfc /SCANFILE=%windir%\system32\CLBCatQ.DLL
net start "COM+ Event System"
-
- Delete .NET Framework cache + restart Cryptographic service
ECHO Deleting Cache
Del /F /Q /S %LOCALAPPDATA%\Microsoft\WebsiteCache\*.*
Del /F /Q /S %LOCALAPPDATA%\Temp\VWDWebCache\*.*
Del /F /Q /S "%LOCALAPPDATA%\Microsoft\Team Foundation\3.0\Cache\*.*"
Del /F /Q /S "<C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\*.*>"
Del /F /Q /S "<C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\*.*>"
Del /F /Q /S "<C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files\*.*>"
Del /F /Q /S "<C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\*.*>"
ECHO Complete
- Server reboot was performed, still same issue, no event ID related to VSS was logged.
- Windows server backup feature reinstalled
- Running “Process Monitor” and logging restart of Cryptographic service and then “vssadmin list writers” did not log anything interesting.
Performed actions (success):
-
- Running system state backup via wbadmin logged interesting data in “Process Monitor”.
wbadmin start systemstatebackup -backupTarget:D:
It shown, that svchost.exe is accessing setupapi.ev* files and then stops.
So after little of googling i´ve found that those files may be corrupted and needs to be replaced. So i took same files from different Win2K8SP2 server, replaced those files, restarted “Cryptographic service” and when running “vssadmin list writers”, voilà System writer was present.
In the meantime i have opened a MS case and consulted those findings with their specialist, who has confirmed that this is known issue and this steps should be reproduced on all affected servers.
Than this fix was successfully applied to more than 20 servers after March 2018 patching and issue was solved.
If you do not have access to another 2008 SP2 server, you can download those files here: setupapi_ev123
They has been extracted from clean 2008 SP2 installation for this purpose. winver -> Version 6.0 (Build 6002: Service Pack 2)
Thanks Microsoft 🙂
