Unable to Recreate Exchange Virtual Directory


Issue

A customer of mine recently had an issue where their Exchange 2013 OWA Virtual Directory was missing in IIS. When attempting to recreate the vDir we encountered the below error message:

“An error occurred while creating the IIS virtual directory `IIS://ServerName/W3SVC/1/ROOT/OWA’

1

To resolve this error I needed to resort to using a long lost tool from the days of old, the IIS 6 Resource Kit.

Note: This blog post could also be relevant if the OWA (or any other) vDir needed to be recreated and you encountered the same error upon recreation.

Resolution

Back in the days of Exchange 2003, the IIS Resource Kit, or more specifically the Metabase Explorer, could be used when recreating a Virtual Directory. Fortunately, the Metabase Explorer tool still works with IIS 8.

Download Link for the IIS 6 Resource Kit

The error encountered above was a result of the IIS Metabase still holding remnants of a past instance of the OWA Virtual Directory, which was preventing the New-OwaVirtualDirectory Cmdlet from successfully completing. It’s important to understand that an Exchange Virtual Directory is really located in two places; Active Directory and IIS. When running the Get-OwaVirtualDirectory Cmdlet (or similar commands for other Virtual Directories), you’re really querying Active Directory. For example, the OWA Virtual Directories for both the Default Web Site and Exchange Back End website in my lab are located in the following location in AD (via ADSIEDIT):

2

So if a vDir is missing in IIS but present in AD, you’ll likely need to first remove it using the Remove-*VirtualDirectory Cmdlet otherwise it will generate an error stating it already exists. In my customer’s scenario, I had to do this beforehand as the OWA vDir was present in AD but missing in IIS.

This brought us to the state we were in at the beginning of this post; receiving the above error message. The OWA vDir was no longer present in AD nor in the Default Web Site, but when trying to recreate it using New-OwaVirtualDirectory we received the above error message.

Tip: Use Get-*VirtualDirectory with the –ShowMailboxVirtualDirectories parameter to view the Virtual Directories on both web sites. For example:

3

The solution was to install the IIS 6 Resource Kit and use Metabase Explorer to delete the ghosted vDir. When installing the Resource Kit, select Custom Install and then uncheck all features except for Metabase Explorer 1.6 and proceed with the installation. Once it finishes, it may require you add the .NET Framework 3.5 Feature.

When you open the tool on the Exchange Server in question, navigate to the below tree structure and delete the old OWA Virtual Directory by right-clicking it and selecting Delete. When completed, the OWA vDir should no longer be present (as seen below).

4

You should now be able to successfully execute the New-OwaVirtualDirectory Cmdlet. It’s always a bit nostalgic seeing a tool of days gone by still able to save the day. I’d like to thank my co-worker John Dixon for help with this post. When I can’t figure something out in Exchange/IIS (or anything really) he’s who I lean on for help.

Failures when proxying HTTP requests from Exchange 2013 to a previous Exchange version


Overview

I’ve seen this issue a few times over the past months & most recently this past week with a customer. Luckily there’s a fairly simple fix to the issue published by Microsoft, but realizing not everyone remembers every Microsoft KB that gets released I thought I’d shine a spotlight on this one.

Scenario

As part of the migration process, when customers move their namespace from either Exchange 2007 or 2010 to 2013, HTTP connections start proxying through 2013 to the legacy Exchange Servers and some users will experience failures. The potential affected workloads are:
AutoDiscover
Exchange Web Services (Free/Busy)
ActiveSync
OWA
Outlook

Test or new mailboxes may not be affected.

Resolution

The cause of this is the age old problem of Token Bloat. Users being members of too many groups or having large tokens.

The fix is to implement the changes in the below Microsoft KB article

“HTTP 400 Bad Request” error when proxying HTTP requests from Exchange Server 2013 to a previous version of Exchange Server
https://support.microsoft.com/en-us/kb/2988444

The interesting thing in this scenario is that the issue was not experienced in the legacy version of Exchange & even if you look at the tokens themselves, they may not seem overly large. It seems that the process of proxying Exchange traffic is much more sensitive to this issue. Also, in a recent case that went to Microsoft, even if you increase the recommended values to a value higher than your current headers it may not have the desired effect. In our case we had to set the MaxRequestBytes & MaxFieldLength values to exactly match the values in the Microsoft KB (65536 (Decimal)).

For further reading, please see the below articles.

Complimentary Articles

“HTTP 400 – Bad Request (Request Header too long)” error in Internet Information Services (IIS)
https://support.microsoft.com/en-us/kb/2020943

How to use Group Policy to add the MaxTokenSize registry entry to multiple computers
https://support.microsoft.com/en-us/kb/938118

 

Additional Note

As an FYI, another issue I commonly see when namespaces get transitioned to 2013 is authentication popups when connections proxy to the legacy Exchange Servers. Please see the below KB for that issue

Outlook Anywhere users prompted for credentials when they try to connect to Exchange Server 2013
https://support.microsoft.com/en-us/kb/2990117

I also blogged about it here
https://exchangemaster.wordpress.com/2014/10/30/exchange-2010-outlook-anywhere-users-receiving-prompts-when-proxied-through-exchange-2013/

The Importance of Updated Domain Controllers When Deploying Exchange


Overview
Much is made about a healthy Active Directory environment being a prerequisite for a healthy Exchange deployment. This can be especially challenging when there are separate teams managing AD & Exchange; meaning sometimes things can slip through the cracks.

Issue
A colleague of mine recently ran into an issue when preparing to deploy Exchange 2013 into an existing Exchange Organization. While running Setup /PrepareAD, the process would fail at about 14%, stating the domain controller is not available. It was determined that the DC holding all of the FSMO roles was in the process of a reboot. At first the assumption was that this was coincidental; possibly the work of the AD team. After the server came back up, /PrepareAD was run again & had the exact same result! So it appeared something that the /PrepareAd process was doing was the culprit. The event logs on the DC gave the below output:

EVENTID: 1000
Faulting application name: lsass.exe, version: 6.3.9600.16384, time stamp: 0x5215e25f

Faulting module name: ntdsai.dll, version: 6.3.9600.16421, time stamp: 0x524fcaed

Exception code: 0xc0000005

Fault offset: 0x000000000019e45d

Faulting process id: 0x1ec

Faulting application start time: 0x01d0553575d64eb5

Faulting application path: C:\Windows\system32\lsass.exe

Faulting module path: C:\Windows\system32\ntdsai.dll

Report Id: 53c0474e-c12d-11e4-9406-005056890b81

Faulting package full name:

Faulting package-relative application ID:

EVENTID: 1015
A critical system process, C:\Windows\system32\lsass.exe, failed with status code c0000005.  The machine must now be restarted.

The logs were saying that the Lsass.exe process was crashing, leading to the Domain Controller restarting (see image below).

1

The easiest path of troubleshooting lead towards moving the FSMO roles to another server & seeing if the issue followed it. Setup /PrepareAD was run again & the issue did in fact follow the FSMO roles.

Resolution
It was at this point that I was engaged & I had a feeling this was either a performance issue on the domain controllers or something buggy at play. Before too long I was able to find the below MS KB for an issue that seemed to match our symptoms:

“Lsass.exe process and Windows Server 2012 R2-based domain controller crashes when the server runs under low memory”
http://support.microsoft.com/en-us/kb/3025087

The customer was more than willing to install the hotfix, but we soon realized that we also had to install the prerequisite update package below (which was sizeable):

Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 update: April 2014
http://support.microsoft.com/en-us/kb/2919355

During this time, the domain controller was also updated to .NET 4.5.2. After all of this was done, Setup /PrepareAD completed successfully. My colleague was 90% certain the hotfix was the fix, but also noted that before the patch the DC’s CPU utilization was consistently running at 60%. After the updates, it now sits in the 20-30% range. So regardless, we saw much better performance & stability after updating the Domain Controllers.

Conclusions
While I understand we can’t all be up to date on our patching 100% of the time, there is some health checking we can do to the environments we manage.

For all Windows servers, I strongly recommend getting a performance baseline of the big 3: Disk, Memory, & CPU. I like to say that you can’t truly say what bad performance is defined by if you don’t have a definition of good performance in the first place. Staying up to date with Windows Updates can greatly help with this. Even though a system may have performed to a certain level at one point in time, doesn’t mean any number of other variable couldn’t have changed since then to result in poor performance today; often times, vendor updates can remedy this.

As for Domain Controllers, they’re one of the easiest workloads to test with, since a new DC can be created with relative ease. You can use a test environment (recommended) or simply deploy Windows updates to a select number of domain controllers & then compare the current behavior with your baseline.

In this customer’s case, these performance/stability issues could have resulted in any number of applications to fail that relied on AD. Some failures may have been silent, while others could’ve been show stoppers like this one.

 

Remember the basics when working with Dynamic Distribution Groups (I didn’t)


Overview:

I recently had a customer come to me with a simple issue of mail not being received in his Exchange 2013 environment when sending to a Dynamic Distribution Group he had just created. Well it certainly seemed like an easy issue to track down (which it technically was) but unfortunately I was a little too confident in my abilities & made the age-old mistake of overlooking the basics. Hopefully others can avoid that mistake after giving this a read.

Scenario:

Create a Dynamic Distribution Group named TestDL#1 whose membership is defined by a Universal Security Group named TestSecurityGroup using the following command in shell:

New-DynamicDistributionGroup -Name “TestDL#1” -RecipientFilter {MemberOfGroup -eq “CN=TestSecurityGroup,OU=End_Users,OU=Company_Users,DC=ASH,DC=NET”}

Note: This command places the Dynamic DL object into the default Users OU & also sets the msExchDynamicDLBaseDN to the Users OU’s Distibguished Name (CN=Users,DC=ASH,DC=NET). This will become important later.

I can verify the membership of this group by running:

$var = Get-DynamicDistributionGroup “TestDL#1”

Get-Recipient -RecipientPreviewFilter $var.RecipientFilter

In my case, the members show up correctly as John, Bob, Sam, & Dave. However, if I send emails to this group nobody gets them. When looking at messagetracking, the recipients show as {} (see below screenshot)

1

Now here’s the really interesting part. My security group, as well as my users are in the OU=End_Users,OU=Company_Users,DC=ASH,DC=NET Organizational Unit. However (as mentioned before in my Note), my Dynamic DL is in the CN=Users,DC=ASH,DC=NET Organizational Unit. Now if I move my users into the Users OU, then they receive the email & show up as valid recipients.

2

Now no matter which OU I move my Dynamic Distribution Group (TestDL#1) to, this behavior is the same.

For instance, if I had run the below command instead, I never would have noticed an issue because the Dynamic DL would’ve been created in the same OU as the users & the Security Group.

New-DynamicDistributionGroup -Name “TestDL#1” -OrganizationalUnit “ash.net/Company_Users/End_Users” -RecipientFilter {MemberOfGroup -eq “CN=TestSecurityGroup,OU=End_Users,OU=Company_Users,DC=ASH,DC=NET”}

The last head scratcher is if I move the actual AD Security Group (TestSecurityGroup) that I’m using to filter against to a different OU, I get the same behavior (no emails).

So it would seem that the solution is to ensure you always place the Dynamic Distribution Group into the same OU where ALL of your Security Group members are as well as the security group itself is.

This seemed crazy so I had to assume I wasn’t creating the filter correctly. It was at this point I pinged some colleagues of mine to see where I was going wrong.

Tip: Always get your buddies to peer review your work. A second set of eyes on an issue usually goes a long way to figuring things out.

Solution:

As it turned out, there were two things I failed to understand about this issue.

  1. When you create a Dynamic Distribution Group, by default, the RecipientContainer setting for that group is set to the OU where the DDG is placed. This means that because I initially did not specify the OU for the DDG to be placed in, it was placed in the Users OU (CN=Users,DC=ASH,DC=NET). So when Exchange was performing its query to determine membership, it could only see members that were in the Users OU. So the solution in my scenario would be to use the –RecipientContainer parameter when creating the OU & specify the entire domain.

EX: New-DynamicDistributionGroup -Name “TestDL#1” -RecipientFilter {MemberOfGroup -eq “CN=TestSecurityGroup,OU=End_Users,OU=Company_Users,DC=ASH,DC=NET”} –RecipientContainer “ASH.NET”

This one was particularly embarrassing because the answer was clearly in the TechNet article for the New-DynamicDistributionGroup cmdlet.

  1. The other thing I didn’t realize was the reason my DDG broke when moving the Security Group I was filtering against. It was breaking because I specified the Security Group using its Distinguished Name, which included the OU it resided in (CN=TestSecurityGroup,OU=End_Users,OU=Company_Users,DC=ASH,DC=NET). So by moving the group I was making my query come up empty. Now the first thing I thought of was if I could specify the group using the common name or the GUID instead. Unfortunately, you cannot because of an AD limitation:

“MemberOfGroup filtering requires that you supply the full AD distinguished name of the group you’re trying to filter against. This is an AD limitation, and it happens because you’re really filtering this calculated back-link property from AD, not the simple concept of “memberOf” that we expose in Exchange.”

So the important thing to remember here is to either not move the Security Group you’re filtering against, or if you move it, to update your filter.

Thanks go to MVPs Tony Redmond & Tony Murray for pointing these two important facts out to me.

Conclusion:

As I found out, a strong foundational knowledge of Active Directory is key to being a strong Exchange Admin/Consultant/Support Engineer. But even when you feel confident in your abilities for a given topic, don’t be afraid to ask people you trust. You might find out you’re either a bit rusty or not as knowledgeable as you thought you were J

Beware effects to Exchange of setting Primary Group in AD


Overview:
Here’s a quick one regarding an issue I came across not too long ago with a customer. The issue was that members of distribution lists were not getting emails addressed to it.

Issue:
Consider this scenario:

Exchange 2013 CU7 (thought it would also have the same effect on Exchange 2010; have not tested on 2007)
Users:John, Bill, Sam, & Ron

You create a Mail-Enabled Security Group in EAC called TestDL#1 & add John/Bill/Sam/Ron to it. In EAC as well as when using the Get-DistributionGroupMember; John, Bill, Sam, & Ron all show up as members. They can all receive emails sent to this group. You then go to Ron’s user account in AD Users & Computers & on the “Member Of” tab you select the TestDL#1 group & then click the “Set Primary Group” option. Obviously, in ADUC it still shows Ron as being a member of this group. However, in EAC or in shell, Ron is no longer listed as a member of the group.

The biggest problem is that when emailing the group, Ron no longer gets the emails. However, as soon as I change his Primary Group to something else he then shows up & can get the emails. This creates a situation where a user is supposed to be getting emails but isn’t. This issue is easily reproducible in a lab.

Solution:
Nothing advanced or fancy here, just don’t change the Primary Group value in AD to be a Mail-Enabled Security Group. Exchange is unable to query the membership of a user for a group that’s also been set as their Primary Group. This is because modifying this property changes the way the object appears in AD & therefore changes the results of Exchange’s query (when it routes mail to it as well as how it lists membership within its management tools).

This also brings up another suggested practice which can help you avoid this scenario all together; use Mail-Enabled Distribution Groups instead of Mail-Enabled Security Groups when possible.

References:
Distribution groups in Exchange 2010 are not showing some members
http://www.run-corp.com/distribution-groups-in-exchange-2010-are-not-showing-some-members/

Members in Exchange 2010 Distribution Group and Active Directory differ
https://social.technet.microsoft.com/Forums/exchange/en-US/7ff27688-31e1-43c1-ba0f-fee95299f31f/members-in-exchange-2010-distribution-group-and-active-directory-differ?forum=exchange2010

Setting Primary Group Excludes the User from the Group Membership in Active Directory
http://support.microsoft.com/kb/275523

OOF messages not being sent in Exchange 2013 CU5 Environment


Scenario

Customer had a single server Exchange 2013 SP1 environment that had been migrated from Exchange 2003 over time (double hop migration). The customer installed CU5 & about a month later they noticed Out of Office messages were not being sent for users who had it enabled. The environment had only been running a short while before updating to CU5 so at the time the customer didn’t correlate the issue happening with updating to CU5 (more on this later).

Issue

In this scenario, Mailbox-A would enable their Out of Office, Mailbox-B would send Mailbox-A an email, Mailbox-A would receive the email message but Mailbox-B would never receive an OOF message.

In addition to this, if you looked in the queues you would see the original email message (not the OOF message) queued even though it had already been successfully delivered.

1

 

The LastError on this message stated “432 4.2.0 STOREDRV.Deliver; Agent transient failure during message resubmission [Agent: Mailbox Rules Agent]” To make matters worse, this message would eventually timeout & the original sender (Mailbox-B in my above scenario) would receive an NDR containing “#< #4.4.7 smtp;550 4.4.7 QUEUE.Expired; message expired> #SMTP#” which would lead them to the false belief that the original message was never delivered. If Mailbox-A turned off their OOF then the message in the queue would eventually be removed without an NDR.

This situation not only prevented the customer from utilizing Out Of Office as it was intended to be used, but it also caused NDRs to be generated to anyone sending them emails while OOF was enabled.

Troubleshooting

This was a really perplexing case as I threw just about everything I had in my bag of tricks at it. Before relenting & eventually escalating to Microsoft I performed the following:

  • Using Get-TransportAgent to verify no 3rd party agents were causing issues
  • Tested with new mailboxes on new mailbox databases
  • Set all Connector & Diagnostic logging to Verbose
  • Recreated the default receive connectors
  • Enabled Pipeline Tracing for the Transport as well as the Mailbox Transport services to see the messages at each stage of Transport
  • Installed a second Exchange server in the environment & tested with new database/users
  • Recreated all of the System Mailboxes
  • Re-ran setup /PrepareAD

All tests resulted in the same behavior. In fact, I found that I could recreate the issue without even enabling OOF. If I created a mailbox rule to have the system send an email to the original sender (effectively functioning like an OOF) I would experience the same behavior. However, an OOF is really just a mailbox rule, which would explain the error we’re seeing in the queue regarding the “Mailbox Rules Agent”. So obviously something was not allowing the rules to fire & my money was on either permissions or some goofy object in Active Directory (like an invalid character in a LegacyDN or the Organization name) that was causing the Rules Agent to choke when it tried to generate the email message.

I was able to get to Microsoft Tier 3 Support (a benefit of being a Premier Partner) & they were equally perplexed. As was expected, they had me run both an ExTrace as well as an IDNA Trace. In short, each of these tools gives the Microsoft debuggers a deep look at what’s going on within the individual processes. I’ve looked at the output of these traces a few times myself but it tends to be a little too deep for a non-debugger to make use of. Needless to say, if you’ve reached this point with Microsoft Support then you’ve certainly got an interesting case.

Resolution

So what was the eventual resolution? Well it was permissions but it still left me scratching my head. First off, here’s the fix:

We opened ADSIEDIT & navigated to the Default Receive Connector of the Exchange Server, went to the Security tab, selected the ExchangeLegacyInterop security group. By default, there is a Deny entry (seen below) for the “Accept Organization Headers” permission.

2

In our case, we unchecked Deny for this permission & after restarting the MSExchange Transport service the issue went away; we then started receiving OOF messages as expected. Microsoft Support explained that this permission is used when the system needs to issue the “MAIL FROM” SMTP verb; which is required when generating OOF messages or similar Rules.

I was excited to figure out the cause of the issue but was also curious why this would be affecting us since as far as I understand, only Exchange 2000/2003 servers were ever members of the ExchangeLegacyInterop security group. This group was used to enable mail flow between the legacy servers & Exchange 07/10.

Upon inspection, this customer’s Exchange 2013 servers had been made members of this group. This security group had zero members in my Greenfield Exchange 2013 environment so I suspected the customer added them there for some odd reason in the past.

Oddly enough, the Microsoft Engineer told me they had a lab environment that was also migrated from Exchange 2003 & it also had this group populated with the 2013 servers. So after asking around to several colleagues, every answer I got back was that their ExchangeLegacyInterop security group was empty. So what does this tell us? Either both environments had someone modify this group manually at some time (for whatever strange reason), or there is a particular set of circumstances that could lead to a 2013 environment having the 2013 servers as members of this group. So if it happens to you then hopefully this article will be of some use to you.

Lastly, the thing I find most bizarre about this situation is that I was unable to reproduce this issue in my SP1 lab. After adding my three 2013 SP1 servers to the ExchangeLegacyInterop security group (forcing replication & rebooting each server to be sure) I could not reproduce the issue. However, after installing CU5 on one of the servers (and thereby updating the Active Directory Schema) the issue occurred on all 3 servers. So apparently something about the CU5 schema change triggers this to be an issue. However, everything I have seen has proven that no 2013 servers should be members of this group to begin with so I’d say it’s not a bug but more a series of unfortunate events 🙂

Note: MS Support told me that they had another case like this but in that situation, the Accept Organization Headers permission had been denied to the Authenticated Users group on the receive connectors. Not sure how that happened either but it just proves this permission is important for the system to have.