SharePoint A process serving application pool terminated unexpectedly. The process id was ''. The process exit code was '0xffffff

Oh joys of working in IT...

Some days are really bl**dy awful.

This morning my MOSS 2007 staging farm (thank god it was not the live one) was down, not able to go to ANY of the SharePoint sites.

Symptoms:

1. Getting the cryptic "Service Unavailable" in IE when browsing to the SharePoint site (including the Central Admin)
2. Checking out in IIS the SP App Pools have crashed, tried to start them again, IIS reset but whenever I browse to the Central Admin again, the app pool crashes again.
3. The following events are in the System Logs

A process serving application pool terminated unexpectedly. The process id was ''. The process exit code was '0xffffff'

What's that? Never seen it before!?

And it only appeared at 3am this morning... hang on a minute... 3am?
WINDOWS AUTOMATIC "KILLER" UPDATES!!!

Nightmare...

After checking the almighty logs I worked out what KB was installed:

Restart Required: To complete the installation of the following updates, the computer will be restarted within 15 minutes:
- Security Update for Windows Server 2003 (KB974318)
- Windows Malicious Software Removal Tool - December 2009 (KB890830)
- Security Update for Windows Server 2003 (KB973904)
- Update for Windows Server 2003 (KB971737)
- Update for Windows Server 2003 (KB973917)
- Security Update for Windows Server 2003 (KB974392)
- Cumulative Security Update for Internet Explorer 8 for Windows Server 2003 (KB976325)

Doing a quick search in Google harvested the following:

http://gavin.mclelland.ca/2009/12/09/iis-service-unavailable-aka-windows-security-update-kills-application-pool/
and
http://thecrmgrid.wordpress.com/2009/12/09/windows-update-killed-my-app-pool/
and
http://blogs.msdn.com/jaskis/archive/2009/12/09/after-installing-kb-973917-the-iis-6-0-application-pools-cannot-start-up.aspx

(THANK YOU GOOGLE, THANK YOU GOOGLE, THANK YOU GOOGLE...)

Anyway, tried what was said there and installed Windows Server 2003 SP2 again.
That solved the crashing app pool problems but created a new one:

1. Now getting an error HTTP 403 when browsing any SharePoint site (including Central Admin)
2. In the IIS logs I get

2009-12-10 12:27:31 W3SVC1611466368 10.4.4.39 GET / - 1000 DOMAIN\username 10.4.4.39 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.2;+Trident/4.0;+.NET+CLR+1.1.4322;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.4506.2152;+.NET+CLR+3.5.30729) 403 14 5

Did some research and that came up
(THANK YOU GOOGLE, THANK YOU GOOGLE, THANK YOU GOOGLE...)

https://www.hensongroup.com/blogs/archive/2009/05/11/moss-2007-403-forbidden-error-after-net-framework-updates-fail-to-install.aspx

UPDATE (06/09/2012): Someone mentioned that the link above does not work anymore so let's go straight to the horse's mouth: http://support.microsoft.com/Default.aspx?kbid=2009746

Extract from that page (i.e. solution):

To resolve this problem, reinstall Service Pack 2 for Windows Server 2003 on the web server.  This will bring all IIS 6.0 components up to the correct file versions, and will maintain the installation of the KB973917 update.  Reinstalling the KB98266 and/or KB973917 update should not be necessary.
NOTE: Re-installing SP2 will enable Scalable Networking Pack (SNP) Feature that is on by default with Windows 2003 Service Pack 2. When service pack 2 was originally released this caused the issues documented in the following KB: http://support.microsoft.com/kb/948496

It may be necessary to disable the Scalable Networking Pack after re-applying Service Pack 2 to address the issues described in KB 948496

That was my problem...

Central Admin is back.

Will try later to re-install the faulty KB973917 just for fun and because I really have time to waste in my day like Microsoft seems to think!!!.... NOT

Thank you very much Microsoft Automatic Updates for wrecking my day once more!