{"id":29,"date":"2018-04-04T13:28:24","date_gmt":"2018-04-04T13:28:24","guid":{"rendered":"http:\/\/blog.mada.cz\/?p=29"},"modified":"2018-04-06T10:08:12","modified_gmt":"2018-04-06T10:08:12","slug":"vss-system-writer-is-missing-after-last-patching-2018-03","status":"publish","type":"post","link":"https:\/\/blog.mada.cz\/index.php\/2018\/04\/04\/vss-system-writer-is-missing-after-last-patching-2018-03\/","title":{"rendered":"VSS: System writer is missing after last patching (2018-03)"},"content":{"rendered":"<p>After last patching (2018-03) &#8220;system writer&#8221; is missing and backup tools like wbadmin, Hiback, NBU or TSM failed to perform system state backup on several servers across multiple environment (physical, virtual).<\/p>\n<p><strong>Affected OS:<\/strong>\u00a0Windows Server\u00a0<strong>2008 SP2 <\/strong>only<\/p>\n<p><strong>Installed KBs:\u00a0<\/strong><br \/>\nKB4090450<br \/>\nKB4089229<br \/>\nKB4089344<br \/>\nKB4087398<br \/>\nKB4089175<br \/>\nKB4056564<\/p>\n<p>For non of this KB\u00b4s is this bug documented.<\/p>\n<p><b>Error description per tool:<\/b><\/p>\n<p style=\"padding-left: 30px;\"><b>wbadmin error:\u00a0<\/b>&#8220;System writer is not found in the backup.&#8221; EventID 517, error code 2155348226<\/p>\n<p style=\"padding-left: 30px;\"><b>Hiback error: &#8220;<\/b>finerr 213&#8243;<\/p>\n<p style=\"padding-left: 30px;\"><b>TSM (Tivoli Storage Manager) error:<\/b>&#8220;Backups have ended with status 12 err on server &#8230;&#8221;<\/p>\n<p style=\"padding-left: 30px;\"><b>NetBackup:<\/b>\u00a0&#8220;The VSS System Writer is not found on this system. The backup is stopping because the System Writer is required for a successful system state backup. Have the system administrator check the VSS infrastructure and fix the issue. 86024:save: Error occured while saving disaster recovery save sets.&#8221;<\/p>\n<p><strong>Performed actions (no success):<\/strong><\/p>\n<ul>\n<li>Take ownership<\/li>\n<\/ul>\n<pre class=\"height-set:true height:300 lang:batch decode:true \">Takeown \/f %windir%\\winsxs\\temp\\PendingRenames \/a \r\nicacls %windir%\\winsxs\\temp\\PendingRenames \/grant \"NT AUTHORITY\\SYSTEM:(RX)\"\r\nicacls %windir%\\winsxs\\temp\\PendingRenames \/grant \"NT Service\\trustedinstaller:(F)\"\r\nicacls %windir%\\winsxs\\temp\\PendingRenames \/grant BUILTIN\\Users:(RX)\r\nTakeown \/f %windir%\\winsxs\\filemaps\\* \/a \r\nicacls %windir%\\winsxs\\filemaps\\*.* \/grant \"NT AUTHORITY\\SYSTEM:(RX)\"\r\nicacls %windir%\\winsxs\\filemaps\\*.* \/grant \"NT Service\\trustedinstaller:(F)\"\r\nicacls %windir%\\winsxs\\filemaps\\*.* \/grant BUILTIN\\Users:(RX)\r\nnet stop cryptsvc\r\nnet start cryptsvc<\/pre>\n<ul>\n<li>Re-register DLL\u00b4s, SFC check<\/li>\n<\/ul>\n<pre class=\"height-set:true height:300 lang:batch decode:true\" title=\"Re-registering DLL\u00b4s, SFC check\">net stop \"System Event Notification Service\" \r\nnet stop \"Background Intelligent Transfer Service\" \r\nnet stop \"COM+ Event System\" \r\nnet stop \"Microsoft Software Shadow Copy Provider\" \r\nnet stop \"Volume Shadow Copy\" \r\ncd \/d %windir%\\system32 \r\nnet stop vss \r\nnet stop swprv \r\nregsvr32 \/s ATL.DLL \r\nregsvr32 \/s comsvcs.DLL \r\nregsvr32 \/s credui.DLL \r\nregsvr32 \/s CRYPTNET.DLL \r\nregsvr32 \/s CRYPTUI.DLL \r\nregsvr32 \/s dhcpqec.DLL \r\nregsvr32 \/s dssenh.DLL \r\nregsvr32 \/s eapqec.DLL \r\nregsvr32 \/s esscli.DLL \r\nregsvr32 \/s FastProx.DLL \r\nregsvr32 \/s FirewallAPI.DLL \r\nregsvr32 \/s kmsvc.DLL \r\nregsvr32 \/s lsmproxy.DLL \r\nregsvr32 \/s MSCTF.DLL \r\nregsvr32 \/s msi.DLL \r\nregsvr32 \/s msxml3.DLL \r\nregsvr32 \/s ncprov.DLL \r\nregsvr32 \/s ole32.DLL \r\nregsvr32 \/s OLEACC.DLL \r\nregsvr32 \/s OLEAUT32.DLL \r\nregsvr32 \/s PROPSYS.DLL \r\nregsvr32 \/s QAgent.DLL \r\nregsvr32 \/s qagentrt.DLL \r\nregsvr32 \/s QUtil.DLL \r\nregsvr32 \/s raschap.DLL \r\nregsvr32 \/s RASQEC.DLL \r\nregsvr32 \/s rastls.DLL \r\nregsvr32 \/s repdrvfs.DLL \r\nregsvr32 \/s RPCRT4.DLL \r\nregsvr32 \/s rsaenh.DLL \r\nregsvr32 \/s SHELL32.DLL \r\nregsvr32 \/s shsvcs.DLL \r\nregsvr32 \/s \/i swprv.DLL \r\nregsvr32 \/s tschannel.DLL \r\nregsvr32 \/s USERENV.DLL \r\nregsvr32 \/s vss_ps.DLL \r\nregsvr32 \/s wbemcons.DLL \r\nregsvr32 \/s wbemcore.DLL \r\nregsvr32 \/s wbemess.DLL \r\nregsvr32 \/s wbemsvc.DLL \r\nregsvr32 \/s WINHTTP.DLL \r\nregsvr32 \/s WINTRUST.DLL \r\nregsvr32 \/s wmiprvsd.DLL \r\nregsvr32 \/s wmisvc.DLL \r\nregsvr32 \/s wmiutils.DLL \r\nregsvr32 \/s wuaueng.DLL \r\nsfc \/SCANFILE=%windir%\\system32\\catsrv.DLL \r\nsfc \/SCANFILE=%windir%\\system32\\catsrvut.DLL \r\nsfc \/SCANFILE=%windir%\\system32\\CLBCatQ.DLL \r\nnet start \"COM+ Event System\"\r\n<\/pre>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Delete .NET Framework cache + restart Cryptographic service<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<pre class=\"lang:default decode:true \">ECHO Deleting Cache \r\nDel \/F \/Q \/S %LOCALAPPDATA%\\Microsoft\\WebsiteCache\\*.* \r\nDel \/F \/Q \/S %LOCALAPPDATA%\\Temp\\VWDWebCache\\*.* \r\nDel \/F \/Q \/S \"%LOCALAPPDATA%\\Microsoft\\Team Foundation\\3.0\\Cache\\*.*\" \r\nDel \/F \/Q \/S \"&lt;C:\\Windows\\Microsoft.NET\\Framework\\v2.0.50727\\Temporary ASP.NET Files\\*.*&gt;\" \r\nDel \/F \/Q \/S \"&lt;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319\\Temporary ASP.NET Files\\*.*&gt;\" \r\nDel \/F \/Q \/S \"&lt;C:\\Windows\\Microsoft.NET\\Framework64\\v2.0.50727\\Temporary ASP.NET Files\\*.*&gt;\" \r\nDel \/F \/Q \/S \"&lt;C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319\\Temporary ASP.NET Files\\*.*&gt;\" \r\nECHO Complete<\/pre>\n<p>&nbsp;<\/p>\n<ul>\n<li>Server reboot was performed, still same issue, no event ID related to VSS was logged.<\/li>\n<li>Windows server backup feature reinstalled<\/li>\n<li>Running &#8220;Process Monitor&#8221; and logging restart of Cryptographic service and then &#8220;vssadmin list writers&#8221; did not log anything interesting.<\/li>\n<\/ul>\n<p><strong>Performed actions (success):<\/strong><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Running system state backup via wbadmin\u00a0logged interesting data in &#8220;Process Monitor&#8221;.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<pre class=\"lang:batch decode:true \">wbadmin start systemstatebackup -backupTarget:D:<\/pre>\n<p>It shown, that <em>svchost.exe<\/em> is accessing<em> setupapi.ev*<\/em> files and then stops.<\/p>\n<p><a href=\"http:\/\/blog.mada.cz\/wp-content\/uploads\/2018\/04\/procmon_system_writer.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-57\" src=\"http:\/\/blog.mada.cz\/wp-content\/uploads\/2018\/04\/procmon_system_writer-1024x258.png\" alt=\"\" width=\"840\" height=\"212\" srcset=\"https:\/\/blog.mada.cz\/wp-content\/uploads\/2018\/04\/procmon_system_writer-1024x258.png 1024w, https:\/\/blog.mada.cz\/wp-content\/uploads\/2018\/04\/procmon_system_writer-300x75.png 300w, https:\/\/blog.mada.cz\/wp-content\/uploads\/2018\/04\/procmon_system_writer-768x193.png 768w, https:\/\/blog.mada.cz\/wp-content\/uploads\/2018\/04\/procmon_system_writer-1200x302.png 1200w, https:\/\/blog.mada.cz\/wp-content\/uploads\/2018\/04\/procmon_system_writer.png 1395w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>So after little of googling i\u00b4ve found that those files may be corrupted and needs to be replaced. So i took same files from different Win2K8SP2 server, replaced those files, restarted &#8220;Cryptographic service&#8221; and when running &#8220;vssadmin list writers&#8221;,\u00a0 voil\u00e0 System writer was present.<\/p>\n<p>In the meantime i have opened a MS case and consulted those findings with their specialist, who has confirmed that this is known issue and this steps should be reproduced on all affected servers.<\/p>\n<p>Than this fix was successfully applied to more than 20 servers after March 2018 patching and issue was solved.<\/p>\n<p>If you do not have access to another 2008 SP2 server, you can download those files here:\u00a0<a href=\"http:\/\/blog.mada.cz\/wp-content\/uploads\/2018\/04\/setupapi_ev123.7z\">setupapi_ev123<\/a><\/p>\n<p>They has been extracted from clean 2008 SP2 installation for this purpose. winver -&gt; Version 6.0 (Build 6002: Service Pack 2)<\/p>\n<p>Thanks Microsoft \ud83d\ude42<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>After last patching (2018-03) &#8220;system writer&#8221; is missing and backup tools like wbadmin, Hiback, NBU or TSM failed to perform system state backup on several servers across multiple environment (physical, virtual). Affected OS:\u00a0Windows Server\u00a02008 SP2 only Installed KBs:\u00a0 KB4090450 KB4089229 KB4089344 KB4087398 KB4089175 KB4056564 For non of this KB\u00b4s is this bug documented. Error description &hellip; <a href=\"https:\/\/blog.mada.cz\/index.php\/2018\/04\/04\/vss-system-writer-is-missing-after-last-patching-2018-03\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;VSS: System writer is missing after last patching (2018-03)&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,14],"tags":[12,13,15,11,10],"class_list":["post-29","post","type-post","status-publish","format-standard","hentry","category-fun-with-microsoft-patches","category-process-monitor-cases","tag-12","tag-backup","tag-process-monitor","tag-system-writer","tag-vss"],"_links":{"self":[{"href":"https:\/\/blog.mada.cz\/index.php\/wp-json\/wp\/v2\/posts\/29"}],"collection":[{"href":"https:\/\/blog.mada.cz\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mada.cz\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mada.cz\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mada.cz\/index.php\/wp-json\/wp\/v2\/comments?post=29"}],"version-history":[{"count":28,"href":"https:\/\/blog.mada.cz\/index.php\/wp-json\/wp\/v2\/posts\/29\/revisions"}],"predecessor-version":[{"id":85,"href":"https:\/\/blog.mada.cz\/index.php\/wp-json\/wp\/v2\/posts\/29\/revisions\/85"}],"wp:attachment":[{"href":"https:\/\/blog.mada.cz\/index.php\/wp-json\/wp\/v2\/media?parent=29"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mada.cz\/index.php\/wp-json\/wp\/v2\/categories?post=29"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mada.cz\/index.php\/wp-json\/wp\/v2\/tags?post=29"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}