Exchange Server Backup, Restore, and Disaster Recovery


A few years ago several colleagues and I decided to write an Exchange Server 2016 book, but scheduling and bandwidth issues caused us to delay it until Exchange Server 2019. However, we soon realized we were simply too busy to commit the amount of time needed to properly see the book to its completion. As a result, I was left with a few chapters destined to die a slow death in a folder on my desktop. As an alternative to that grim scenario, I’ve decided to publish them on my personal website (with zero warranty or proper editing). As the topic relates to Exchange Server troubleshooting, I thought I would cross-post it here.

Also, if you enjoy the topic on Exchange Backup, Restore, & DR, checkout the Exchange Server Troubleshooting Companion now available for free on TechNet

AD Certificate Services not starting due to database in Dirty Shutdown


Background

I had a customer running SBS (Small Business Server) 2011, which runs Exchange 2010, who needed to renew their SSL Certificate as it had recently expired. I have quite a bit of experience with SBS since we have a large Support customer base running it & while it can be a pain to troubleshoot because of so many moving pieces (AD/Exchange/SharePoint/WSUS/SQL/RD Gateway all on one box) there are a few cool features. One of these features is the “Setup your Internet Address” wizard. Because SBS is also its own Certificate Authority the wizard will generate a certificate for you & assign it to Exchange/IIS/RD Gateway. It will also configure all the Exchange virtual directories for you as well as create a certificate install package for you to deploy to non-domain joined systems so your Outlook Anywhere clients will trust your CA.

 

Issue

However, when going to re-run the wizard to renew the certificate I received an error regarding the Active Directory Certificate Service not running. The System event logs had a 7024 event from “Service Control Manager” stating “The Active Directory Certificate Services service terminated with the service-specific error %%939523546”.

So we were unable to request a new certificate & the customer was hoping to avoid purchasing a third-party certificate since they had been working fine (for an extremely small shop) like this for several years.

 

Solution

After researching the error, I found that the error code given pointed to the Certificate Authority database being corrupted. So I navigated to C:\Windows\System32\Certlog & I found an old friend; an ESE (Extensible Storage Engine) database file.  If you didn’t know already, ESE isn’t just used for Exchange.

AD Certificate Services, DHCP (C:\Windows\System32\dhcp\dhcp.mdb), & Active Directory itself (C:\Windows\NTDS\ntds.dit) all use ESE databases.

The caveat however is that instead of ESEUTIL, you should use ESENTUTL to work with them.

(Additional references 1 & 2)

So I ran esentutl /mh <CA Name>.edb to view the header of the database file & found that it was in a Dirty Shutdown. I then tried to run a Recovery against the database by running Esentutl /r edb but this failed.

If this were an Exchange database then this would be where I would try to restore from a backup. Unfortunately this customer did not have a backup of their CA database file (I think a lot of customers would fall into this category) so I had to move onto running a Repair which is the dreaded “/P”.

Microsoft Support offers strict guidance around running a “/p” on Exchange (like performing a Defrag or a Mailbox Move followed by an Integrity Check/Mailbox Repair immediately after having to run a /p; Also, it should be considered a LAST resort) but no such guidance exists for Certificate Services since it is a much MUCH simpler database structure. But a ‘/P” is almost always a destructive action, with associated data loss, so if you have a backup you should always pursue that option first

I ran esentutl /p <CA Name>.edb & after it completed I was then able to start the Active Directory Certificate Services Service. All the proper data (including Issued Certificates & Templates) were still there & after re-running the SBS “Setup your Internet Address” wizard the customer now had a renewed certificate.