Quick method to diagnose Exchange Active Directory Access & Service Startup Issues


Background:

My colleague Jedidiah Hammond wrote a great post awhile back on troubleshooting Exchange Service start-up issues. One of the main areas of focus of the post were issues with Active Directory Global Catalog servers. This can be considered an ad-on to that post as I’ll describe a useful method to troubleshoot Exchange permissions in Active Directory; more specifically, verifying Exchange has the proper access to the Global Catalog servers in and out of it’s respective Active Directory site.

Scenario:

Suppose you find that the Microsoft Exchange Active Directory Topology Service isn’t starting; or the System Attendant, or the Information Store service. Or perhaps the Exchange Management Console or Exchange Management Shell will not connect and is complaining of Active Directory/Global Catalog issues.
Often times this is a result of a port being blocked by Anti-V/Firewall between the Exchange Server and your Global Catalog. Or possibly a configuration issue on the network stack (IP/DNS/etc); maybe someone even powered your GC off much to your dismay. Assuming you have already worked through the above scenarios, one useful tool to verify Exchange/AD functionality is actually a very commonly used one; Event Viewer.

When you first deploy Exchange and run “setup /PrepareAD” (or you let the GUI setup do it for you) it is actually setting many of these permissions in AD. (For a list of all of these changes see this Technet article).

Steps:

Below is an excerpt from MSExchange ADAccess Informational Event ID 2080. You’ll find it occurring roughly every 15min on your Exchange Servers.
Description:
Process STORE.EXE (PID=3376). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
Austin.ASH.ORG    CDG 1 7 7 1 0 1 1 7 1
 Out-of-site:
Houston.ASH.ORG    CDG 1 7 7 1 0 1 1 7 1

This is an example of what the output should look like. You might be asking what those series of numbers represent. Well buried deep within the land of Exchange 2000 there lies a KB article explaining just that.

After reading the article you’ll find that these numbers are basically describing Exchange’s understanding of the Global Catalog servers made available to it; along with whether or not it has the proper ACLs set to be able to utilize them. If you find yourself pulling your hair out as to why Exchange is showing the symptoms I listed earlier, then look for this event on your Exchange server and you just might see something like the following:

Description:
Process STORE.EXE (PID=3376). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
Austin.ASH.ORG    CDG 1 7 7 1 0 0 1 7 1
 Out-of-site:
Houston.ASH.ORG    CDG 1 7 7 1 0 0 1 7 1

Notice it ends with “0171” instead of “1171”. If we reference the above KB article then this tells us Exchange lacks the proper ACL’s in AD.

I’ve seen this many times with customers who have modified the Default Domain Controllers Group Policy or somehow blocked it’s use. I’ve also seen similar issues arise from unchecking “Include Inheritable Permissions from this Object’s Parent” in AD for various objects. If this is the case then please see the post I referenced earlier on how to resolve that. In addition, I’ve found re-running “setup.com /PrepareAD” to be a very useful troubleshooting step in situations such as these where you feel AD permissions may be at fault. Some customers have been weary of running this but honestly their fears stem from ignorance because “it just sounds scary” ; a quick read over the article I referenced earlier will tell you that running it again will only re-add the permissions Exchange has needed all along.
However, be aware that re-running PrepareAD may only resolve the issue temporarily as any bad Group Policies may find themselves being re-applied in about 15min so fixing the actual source of the issue should be the ultimate goal.

An additional note here is if you’re utilizing AD Split permissions with Exchange, there may be additional precautions to be taken before running PrepareAD  again.

MSExchange ADAccess Event ID’s 2601, 2604, 2501


communication brain_blogMSExchange ADAccess Event ID’s 2601, 2604, 2501

After a reboot of of Exchange 2010 server that resides on a Windows 2008 R2 server, the following events are logged in the Application Log

Log Name: Application
Source: MSExchange ADAccess
Level: Warning
Event ID: 2601

Log Name: Application
Source: MSExchange ADAccess
Event ID: 2604
Level: Error

Log Name: Application
Source: MSExchange ADAccess
Event ID: 2501
Level: Error

A NetLogon error of 5719 might also be seen in the Application Log.

 

While this article points out that this can be a normal occurrence it doesn’t explain why this is:

Today’s switches and NICs have advanced protocols that enable allot of really great functionality as well as stability, unfortunately many times that comes at the cost of negotiation time.

Here are some things you can do to remedy the issue

  1. Enable functions like “port Fast” on your switch
  2. Disable advanced functions on the switch (such as spanning tree)
  3. Disable advanced functions on the NICs.
  4. Delay the service startup (properties of the service –> startup type)
  5. Configure Recovery options on the properties of the service to force it to restart the service.
  6. In extreme cases you can make a service dependant on another service.http://support.microsoft.com/kb/193888

NOTE: disabling some services on a switch can put you at risk for things like network loops, so document your changes and weigh the pros and cons.

Error Installing Exchange 2010 Mailbox Role


Problem

Couldn’t mount the database that you specified. Specified database: Mailbox Database XYZ; Error code: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionNotFound: Unable to mount database. (hr=0x8004010f, ec=-2147221233)
[Database: Mailbox Database XYZ, Server: server.domain.local].

An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionNotFound: Unable to mount database. (hr=0x8004010f, ec=-2147221233)
[Database: Mailbox Database XYZ, Server: server.domain.local]

An Active Manager operation failed. Error: Operation failed with message: MapiExceptionNotFound: Unable to mount database. (hr=0x8004010f, ec=-2147221233)
[Server: server.domain.local]

MapiExceptionNotFound: Unable to mount database. (hr=0x8004010f, ec=-2147221233)

Solution

    1. Uninstall Mailbox role
    2. Run Setup /prepareAD
    3. Reinstall mailbox role
      • If you have problem uninstalling the role you may have to remove the Mailbox and Public Folder database using ADSI (not recommended if there is any data in your databases!!)image

Exchange System Attendant or Exchange AD Topology service will not start


When these services will not start its usually for 1 of 2 reasons

  1. Cant get to Active Directory
  2. Not allowed to access Active Directory

Most Common Solutions

  1. Make sure you can ping the Global Catalog (GC) server
  2. DNS Entries are not correct, you can either manual fix or rebuild Dynamic DNS
  3. Test Replication between domain controllers, if it fails troubleshoot AD.
  4. Disable 3rd party firewall products on GC
  5. make sure there is a GC in the AD site your exchange server is in
  6. Sites and service not configured – Need to configure AD Sites and services subnets to match your site topology.
  7. Be sure the Exchange Domain Servers and Exchange Enterprise Servers groups have not been removed from the Users OU in AD   Capture
  8. Be sure the server that the service will not start on is in the following groups
    1. 2003 = Exchange Servers
    2. 2007 = Exchange Servers, Exchange Install Domain Servers
    3. 2010 = Exchange Servers, Exchange Install Domain Servers, Exchange Trusted Subsystem -> (Needs to be in Local Administrators as well)
  9. Make sure the Manage auditing and security log has Exchange Servers and Exchange Enterprise Servers assigned to it on the Default Domain Controller Policy KB316709
  10. Permission on the NTFS directory for the Exchange binaries has been modified.
    System and Administrators should have full access to the Directory.
    1. 2003 Default Location = C:\program Files\Exchsrvr
    2. 2007 Default Location = C:\Program Files\Microsoft\Exchange Server
    3. 2010 Default Location = C:\Program Files\Microsoft\Exchange Server\V14

Glossary

  • AD=Active Directory
  • DC=Domain Controller
  • GC=Global Catalog DC
  • OU= Organizational unit