Remember the basics when working with Dynamic Distribution Groups (I didn’t)


Overview:

I recently had a customer come to me with a simple issue of mail not being received in his Exchange 2013 environment when sending to a Dynamic Distribution Group he had just created. Well it certainly seemed like an easy issue to track down (which it technically was) but unfortunately I was a little too confident in my abilities & made the age-old mistake of overlooking the basics. Hopefully others can avoid that mistake after giving this a read.

Scenario:

Create a Dynamic Distribution Group named TestDL#1 whose membership is defined by a Universal Security Group named TestSecurityGroup using the following command in shell:

New-DynamicDistributionGroup -Name “TestDL#1” -RecipientFilter {MemberOfGroup -eq “CN=TestSecurityGroup,OU=End_Users,OU=Company_Users,DC=ASH,DC=NET”}

Note: This command places the Dynamic DL object into the default Users OU & also sets the msExchDynamicDLBaseDN to the Users OU’s Distibguished Name (CN=Users,DC=ASH,DC=NET). This will become important later.

I can verify the membership of this group by running:

$var = Get-DynamicDistributionGroup “TestDL#1”

Get-Recipient -RecipientPreviewFilter $var.RecipientFilter

In my case, the members show up correctly as John, Bob, Sam, & Dave. However, if I send emails to this group nobody gets them. When looking at messagetracking, the recipients show as {} (see below screenshot)

1

Now here’s the really interesting part. My security group, as well as my users are in the OU=End_Users,OU=Company_Users,DC=ASH,DC=NET Organizational Unit. However (as mentioned before in my Note), my Dynamic DL is in the CN=Users,DC=ASH,DC=NET Organizational Unit. Now if I move my users into the Users OU, then they receive the email & show up as valid recipients.

2

Now no matter which OU I move my Dynamic Distribution Group (TestDL#1) to, this behavior is the same.

For instance, if I had run the below command instead, I never would have noticed an issue because the Dynamic DL would’ve been created in the same OU as the users & the Security Group.

New-DynamicDistributionGroup -Name “TestDL#1” -OrganizationalUnit “ash.net/Company_Users/End_Users” -RecipientFilter {MemberOfGroup -eq “CN=TestSecurityGroup,OU=End_Users,OU=Company_Users,DC=ASH,DC=NET”}

The last head scratcher is if I move the actual AD Security Group (TestSecurityGroup) that I’m using to filter against to a different OU, I get the same behavior (no emails).

So it would seem that the solution is to ensure you always place the Dynamic Distribution Group into the same OU where ALL of your Security Group members are as well as the security group itself is.

This seemed crazy so I had to assume I wasn’t creating the filter correctly. It was at this point I pinged some colleagues of mine to see where I was going wrong.

Tip: Always get your buddies to peer review your work. A second set of eyes on an issue usually goes a long way to figuring things out.

Solution:

As it turned out, there were two things I failed to understand about this issue.

  1. When you create a Dynamic Distribution Group, by default, the RecipientContainer setting for that group is set to the OU where the DDG is placed. This means that because I initially did not specify the OU for the DDG to be placed in, it was placed in the Users OU (CN=Users,DC=ASH,DC=NET). So when Exchange was performing its query to determine membership, it could only see members that were in the Users OU. So the solution in my scenario would be to use the –RecipientContainer parameter when creating the OU & specify the entire domain.

EX: New-DynamicDistributionGroup -Name “TestDL#1” -RecipientFilter {MemberOfGroup -eq “CN=TestSecurityGroup,OU=End_Users,OU=Company_Users,DC=ASH,DC=NET”} –RecipientContainer “ASH.NET”

This one was particularly embarrassing because the answer was clearly in the TechNet article for the New-DynamicDistributionGroup cmdlet.

  1. The other thing I didn’t realize was the reason my DDG broke when moving the Security Group I was filtering against. It was breaking because I specified the Security Group using its Distinguished Name, which included the OU it resided in (CN=TestSecurityGroup,OU=End_Users,OU=Company_Users,DC=ASH,DC=NET). So by moving the group I was making my query come up empty. Now the first thing I thought of was if I could specify the group using the common name or the GUID instead. Unfortunately, you cannot because of an AD limitation:

“MemberOfGroup filtering requires that you supply the full AD distinguished name of the group you’re trying to filter against. This is an AD limitation, and it happens because you’re really filtering this calculated back-link property from AD, not the simple concept of “memberOf” that we expose in Exchange.”

So the important thing to remember here is to either not move the Security Group you’re filtering against, or if you move it, to update your filter.

Thanks go to MVPs Tony Redmond & Tony Murray for pointing these two important facts out to me.

Conclusion:

As I found out, a strong foundational knowledge of Active Directory is key to being a strong Exchange Admin/Consultant/Support Engineer. But even when you feel confident in your abilities for a given topic, don’t be afraid to ask people you trust. You might find out you’re either a bit rusty or not as knowledgeable as you thought you were J

Quick method to diagnose Exchange Active Directory Access & Service Startup Issues


Background:

My colleague Jedidiah Hammond wrote a great post awhile back on troubleshooting Exchange Service start-up issues. One of the main areas of focus of the post were issues with Active Directory Global Catalog servers. This can be considered an ad-on to that post as I’ll describe a useful method to troubleshoot Exchange permissions in Active Directory; more specifically, verifying Exchange has the proper access to the Global Catalog servers in and out of it’s respective Active Directory site.

Scenario:

Suppose you find that the Microsoft Exchange Active Directory Topology Service isn’t starting; or the System Attendant, or the Information Store service. Or perhaps the Exchange Management Console or Exchange Management Shell will not connect and is complaining of Active Directory/Global Catalog issues.
Often times this is a result of a port being blocked by Anti-V/Firewall between the Exchange Server and your Global Catalog. Or possibly a configuration issue on the network stack (IP/DNS/etc); maybe someone even powered your GC off much to your dismay. Assuming you have already worked through the above scenarios, one useful tool to verify Exchange/AD functionality is actually a very commonly used one; Event Viewer.

When you first deploy Exchange and run “setup /PrepareAD” (or you let the GUI setup do it for you) it is actually setting many of these permissions in AD. (For a list of all of these changes see this Technet article).

Steps:

Below is an excerpt from MSExchange ADAccess Informational Event ID 2080. You’ll find it occurring roughly every 15min on your Exchange Servers.
Description:
Process STORE.EXE (PID=3376). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
Austin.ASH.ORG    CDG 1 7 7 1 0 1 1 7 1
 Out-of-site:
Houston.ASH.ORG    CDG 1 7 7 1 0 1 1 7 1

This is an example of what the output should look like. You might be asking what those series of numbers represent. Well buried deep within the land of Exchange 2000 there lies a KB article explaining just that.

After reading the article you’ll find that these numbers are basically describing Exchange’s understanding of the Global Catalog servers made available to it; along with whether or not it has the proper ACLs set to be able to utilize them. If you find yourself pulling your hair out as to why Exchange is showing the symptoms I listed earlier, then look for this event on your Exchange server and you just might see something like the following:

Description:
Process STORE.EXE (PID=3376). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
Austin.ASH.ORG    CDG 1 7 7 1 0 0 1 7 1
 Out-of-site:
Houston.ASH.ORG    CDG 1 7 7 1 0 0 1 7 1

Notice it ends with “0171” instead of “1171”. If we reference the above KB article then this tells us Exchange lacks the proper ACL’s in AD.

I’ve seen this many times with customers who have modified the Default Domain Controllers Group Policy or somehow blocked it’s use. I’ve also seen similar issues arise from unchecking “Include Inheritable Permissions from this Object’s Parent” in AD for various objects. If this is the case then please see the post I referenced earlier on how to resolve that. In addition, I’ve found re-running “setup.com /PrepareAD” to be a very useful troubleshooting step in situations such as these where you feel AD permissions may be at fault. Some customers have been weary of running this but honestly their fears stem from ignorance because “it just sounds scary” ; a quick read over the article I referenced earlier will tell you that running it again will only re-add the permissions Exchange has needed all along.
However, be aware that re-running PrepareAD may only resolve the issue temporarily as any bad Group Policies may find themselves being re-applied in about 15min so fixing the actual source of the issue should be the ultimate goal.

An additional note here is if you’re utilizing AD Split permissions with Exchange, there may be additional precautions to be taken before running PrepareAD  again.

Exchange 2010 EMC cannot access AD configuration data after you demote a DC


Environment:

Exchange 2010/Domain Controller combo server running on Windows 2008 R2.

Problem:

Demote Domain Controller role, causes Exchange Management Console fails to retrieve any Exchange information with error message “Active directory response: The LDAP server is unavailable.”  It’s still looking for the demoted DC although it’s been cleaned out of AD/DNS. All Exchange services start fine, and Exchange Shell works fine.

Reason:

     The obsolete information is cached in an Exchange Management Console file in the Windows profile for the user. EMC is trying to connect to orginal DC that is stored in the file.

Solution:

   Go to the following folder and delete the Exchange Management Console file.

   C:\users\<specific user>\AppData\Roaming\Microsoft\MMC\Exchange Management Console

   Close EMC and reopen it.

Reference: http://support.microsoft.com/kb/2019500

/Preparedomain error when security customizations have been done to Active Directory


Error:

Configuring Microsoft Exchange Server

    Organization Preparation                                  FAILED
     The following error was generated when "$error.Clear();
          if ($RolePrepareAllDomains)
          {
              initialize-DomainPermissions -AllDomains:$true -CreateTenantRoot:(
$RoleIsDatacenter -or $RoleIsPartnerHosted);
          }
          elseif ($RoleDomain -ne $null)
          {
              initialize-DomainPermissions -Domain $RoleDomain -CreateTenantRoot
:($RoleIsDatacenter -or $RoleIsPartnerHosted);
          }
          else
          {
              initialize-DomainPermissions -CreateTenantRoot:($RoleIsDatacenter
-or $RoleIsPartnerHosted);
          }
        " was run: "PrepareDomain for domain Domain was unable to add the group CN=Exchange Install Domain Servers,CN=Microsoft Exchange System Objects,DC=domain,DC=local to the group CN=Exchange Servers,OU=Microsoft Exchange Security Groups,DC=domain,DC=local on domain controller server.domain.local, because the current user does not have permissions to modify Exchange Servers. Please ensure that the current user can modify the membership of Exchange Servers and run PrepareDomain again.".

Problem:
The user doesn’t have permission to modify the AD groups it needs to modify.
“Exchange Server” group that was created by /preparedomain is member of “Windows Authorization Access Group” group. 
If the permission on that group are changed, /preparedomain may not be able to modify the membership of it. 
Of course, exchange setup gives you some bogus error, which does not make any sense. Winking smile

Solution:

  1. Verify that you are running the /preparedomain as a domain admin
  2. Once we reset it’s permission by checking “inherted” option on the “Windows Authorization Access Group”,  we can manually add Exchange Server group as a member of “Windows Authorization Access Group” Group, and re run /preparedomain and it should run without error.

clip_image002

MSExchange ADAccess Event ID’s 2601, 2604, 2501


communication brain_blogMSExchange ADAccess Event ID’s 2601, 2604, 2501

After a reboot of of Exchange 2010 server that resides on a Windows 2008 R2 server, the following events are logged in the Application Log

Log Name: Application
Source: MSExchange ADAccess
Level: Warning
Event ID: 2601

Log Name: Application
Source: MSExchange ADAccess
Event ID: 2604
Level: Error

Log Name: Application
Source: MSExchange ADAccess
Event ID: 2501
Level: Error

A NetLogon error of 5719 might also be seen in the Application Log.

 

While this article points out that this can be a normal occurrence it doesn’t explain why this is:

Today’s switches and NICs have advanced protocols that enable allot of really great functionality as well as stability, unfortunately many times that comes at the cost of negotiation time.

Here are some things you can do to remedy the issue

  1. Enable functions like “port Fast” on your switch
  2. Disable advanced functions on the switch (such as spanning tree)
  3. Disable advanced functions on the NICs.
  4. Delay the service startup (properties of the service –> startup type)
  5. Configure Recovery options on the properties of the service to force it to restart the service.
  6. In extreme cases you can make a service dependant on another service.http://support.microsoft.com/kb/193888

NOTE: disabling some services on a switch can put you at risk for things like network loops, so document your changes and weigh the pros and cons.

Exchange 2007 installed on 2008 DC cant communicate with 2008 domain controller (itself)


Situation
Exchange services will not start if the 2008 server that exchange is on is the only Domain Controller, it MAY start if there is another DC in the environment
 
Root Cause:
Windows Server 2008 has made TCP/IPv6 the default communication protocol stack over which connections are made by clients connecting to the server that is running Microsoft Exchange. (Exchange is a client of Active Directory)
If you disable or do not configure IPV6 you will have problems communicating with itself.
 
There are 2 possible solutions to this issue
 
1. Enable and Configure ipv6 to have a the same address as the IPV4 address Ex  “::FFFF:192.168.x.x”

2. Disable IPV6 and modify local “host” file 

Note:   
In this step, %SystemRoot% refers to the local hard disk where the Windows system files are located.   

b. Search for the line that contains the term “localhost” by using the CTR+F key combination. 
c. Select the whole line and make it a comment by putting a number sign (#) at the beginning and end of the line. 
d. Press ENTER and, on the next line, type the following lines to provide the TCP/IPv4 address, hostname, and FQDN name for the Exchange server that is running both the Client Access and Mailbox server roles: 
  <TCP/IPv4 address>   <host name of the computer> 
  <TCP/IPv4 address>   <FQDN of the computer> 
e. Click Save, and then close the file. 
 
f. Reboot the server

This Document explainse the cause root cause but is not targeted to this issue.
http://technet.microsoft.com/en-us/library/cc671176.aspx

   

Doing a Disaster Recovery on a Exchange Server that is also a DC


email_exchange_iconHave you every worked on a failed exchange server that also happens to be a DC (not recommended, but it happens)

Well if you do and you find yourself trying to recover it here is how you can.

  1. Note critical information
    1. What are the drive letters
    2. Where is the logs and database located
    3. What is the service pack level
  2. Remove data from server
  3. Format and re-install the OS – using the same drive letters
  4. Seize Roles if they were on the failed server
  5. Run through a metadata cleanup to remove the failed server from AD
  6. Replicate changes to all DCs
  7. Join rebuilt server to the domain  – Using the Same name
  8. Add the Server object to the correct exchange groups
    1. Exchange 2007 – “Exchange Servers”, “Exchange Install Domain Servers”
    2. Exchange 2010 – “Exchange Servers”, “Exchange Install Domain Servers”, “Exchange Trusted Subsystem”
    3. Exchange 2003 – “Exchange Domain Servers”
  9. Windows Update the Server
  10. Do a disaster recovery install of exchange
    1. Exchange 2003 = setup /disasterrecovery
    2. Exchange 2007\2010 = Setup.com /m:recoverserver
  11. Restore data using backup application or recovered databases from failure
  12. and away you go!