2010-01-22: Windows Update woes

Windows Update is a great technology. It ensures that most Windows users are automatically kept up to date with the latest security patches for their OS. Especially now that is no longer tied to Microsoft's Internet Explorer browser, I'm a big fan (with one exception). Unfortunately, on rare occasions, the Windows Update client gets in a corrupted state and things get messed up in a big way. This can be difficult to diagnose or troubleshoot.

I recently performed some simple maintenance on a set of company PCs. They were fairly recent, albeit somewhat underpowered Dell desktops with 2GHz Celerons and 1GB of RAM running Windows Vista Home Basic 32-bit. For the relatively light office work with people browsing, and using Office 2007 for Word, Excel and e-mail, this should be sufficient however. All employees using those PCs, one in particular, had been complaining of generally sluggish performance. Replacing the bloated anti-virus software with something a little more lightweight, the crappy Adobe Reader with Foxit's (free) alternative and upgrading the RAM to 2GB made all the employees a darn sight happier. All, except one.

This one PC responded slowly to everything. Even sitting idle at the desktop, with no open applications, there was constant disk thrashing. Task Manager also revealed a continuous 100% CPU usage. Task manager set to show processes from all users revealed that most of this was the Windows Search Indexer. One would have to wonder what it was indexing continuously. Shutting down indexing didn't have much effect, as the CPU cycles were now simply absorbed by a few other processes and the disk thrashing continued. They were TrustedInstaller.exe and wermgr.exe. The former is the "Windows Modules Installer" service, the latter is the "Windows Error Reporting" service.

Fortunately, all editions of Windows Vista and Windows 7 come with a nice tool called the Performance Monitor. Particularly the resource monitor was of interest. This can be accessed from the "Performance" tab in Task manager or by running perfmon.exe directly. Resource monitor showed that the disk I/O was a reasonably constant 5MB/sec. It also shows which files are being read/written. Some files, such as C:\$Mft and C:\$LogFile are part of the normal operation of NTFS file systems and frequent access to those, while an indication that something else may be amiss, does not point directly to the problem. Other accesses were more suspicious. There was an inordinate number of accesses to C:\Windows\Logs\CBS\CBS.log and C:\Windows\Logs\CBS\CBS.persist.log. These are Vista's patch logs, containing information that was stored in WindowsUpdate.log on Windows XP. There was also a very large number of writes to ...\Microsoft\Windows\WER\ReportQueue and C:\Windows\SoftwareDistribution\Download.

A quick check at the logs files and in the ReportQueue showed that the system was silently logging and reporting an error roughly every 5 seconds, writing mini core dumps every time as well as continually attempting to locally store updates and failing. A few tests showed that this machine did indeed fail to download updates, getting stuck at 0 or sometime during the download process or when it did install them, it would get stuck on Configuring updates Stage 3 of 3 at 0, 92 or 100 percent, depending on how "lucky" you were.

Getting past the Configuring updates Stage 3 of 3 problem required booting into safe mode and either using System Restore to go back to an earlier point of time or, if that no suitable restore point is available or this is undesired, by deleting C:\Windows\Winsxs\pending.xml, which is what I had to do. Still, that only allows booting back into Windows, by ignoring the pending update, and doesn't solve the underlying problem. Of course, this left Windows Update in an inconsistent state and it ended up throwing the dreaded catch-all 0x8000FFFF error code. Fortunately, Microsoft have an excellent knowledge base article that describes what registry settings to delete to get it working again.

After that, I could take the time to figure out why it was getting stuck. The update function had gone haywire, having stored close to a hundred thousand files, making the C:\Windows\SoftwareDistribution\Download folder a ridiculous 22GB in size, accounting for some 40% of all allocated disk space. After stopping the Windows Update service, it is possible to just delete the folder. You can even delete the entire C:\Windows\SoftwareDistribution folder, thus resetting all of Windows Update, but Windows Update will require a lot more time on its next run. The sheer number of files caused that delete operation to take more than an hour. After that, I manually installed what is basically the Windows Update Update.

It took a few hours of forensics and thumb twiddling, but finally all updates had gone through, including Vista SP2 and the system was running normally again. The disk thrashing had stopped, CPU usage was down to single digits and everything responded smoothly. Having a smoothly running system turned one complaining employee into a happy camper again.