Windows Updates on Domain Controllers cause Exchange login issues

Overview
Over the span of the last 3 weeks, I’ve encountered five different customers experiencing this issue. Through my own lab testing and working with Microsoft Premier Support, we were able to diagnose the issue as being related to a recent Windows Update that was installed on the customers’ Windows Server 2012 Domain Controllers that introduced authentication issues.

Symptoms
Outlook users in Exchange environments experience repeated authentication prompts when attempting to access their mailbox. OWA and ActiveSync were not affected by this “login prompt loop.” The issue began presenting itself after Windows Updates were applied to the Domain Controllers in the environment. Another symptom experienced in some customer environments were authentication issues when accessing Terminal Services/Remote Desktop Services.

Troubleshooting
In testing, I actually found the authentication loop began when the Outlook client attempted to authenticate to the AutoDiscover service. From the client machine, I decided to test the authentication process by using Internet Explorer to browse to https://AutoDiscover.Contoso.com/AutoDiscover/AutoDiscover.xml. After initially authenticating, I was presented with repeated login prompts; the same symptoms seen in the Outlook clients. At one point I was curious to see if the issue was related to Windows/Kerberos authentication, so I decided to disable all authentication methods on the AutoDiscover virtual directory except for Basic. After doing this, I was able to successfully authenticate to AutoDiscover. Since this issue was affecting both the AutoDiscover and RPC virtual directories, it made me think it wasn’t necessarily Exchange that was broken, but AD authentication itself.

We first looked at recent Windows Updates on the Exchange servers themselves, but saw nothing unusual. Even as a precautionary measure we removed the Windows Updates that were recently installed on the Exchange Servers the previous day, but the issue remained. We then noticed some recently installed updates on the Domain Controllers which were related to authentication. After removing all updates installed the previous day, the issue was resolved and the authentication loops were gone. We then spent time (via process of elimination) trying to determine which update was the culprit.

Solution
Steps to reproduce:
-Have a 2012 non-R2 DC
-Have KB3175024 installed but DO NOT have KB3177108 (released back in August) installed.

Obviously one possible resolution in my case is to install KB3177108 (which immediately resolved my RDP and Exchange login issues). However, I was curious as to why KB3177108 was not installed on multiple customers’ environments. After working with Microsoft, it appears that it was either because there was an issue initially installing KB3177108, or some customers chose to not install it for possible incompatibility reasons.

Ultimately, the reason we encountered these issues was because the KB3175024 update builds on dependencies of KB3177108, but will install anyways (in error) if it is not present; resulting in the issue above. In short, KB3175024 makes changes that assumes KB3177108 is present. It installs even if KB3177108 is not present and causes authentication issues.

Microsoft has released the below KB to address this issue.

NTLM authentication fails with 0xC0000022 error for Windows Server 2012, Windows 8.1, and Windows Server 2012 R2 after update is applied
https://support.microsoft.com/en-us/kb/3195799

Cause
This issue occurs because a recent update rollup missed a dependency in updating Netlogon.dll.

Resolution
To fix this problem, install the security updates that are described in Microsoft Security Bulletin MS16-101

Disabling Outlook Anywhere & Avoiding Unnecessary Authentication Prompts for Certain Mailboxes

So this is a complicated scenario but only because this particular customer made it that way; in fact the solution ended up being very simple.

Scenario:

One of my Consultant co-workers pinged me on an issue he was sorting through at a customer site. They were using UAG for their Outlook Anywhere endpoint, both internally & externally. They had a policy to only allow Outlook Anywhere for roughly 30% of their user base. They were enforcing this using AD group membership in UAG to block access to the Outlook Anywhere rule for all users except for those on the allowed list.

Not only was this a nightmare to manage but it also caused Outlook Authentication prompts in certain scenarios. I’ll explain:

When internal Outlook users moved between wired & wireless networks (or vice versa), Outlook would be disconnected just long enough for it to attempt an Outlook Anywhere connection over HTTPS (since the RPC/MAPI connection didn’t reconnect quite fast enough for Outlook’s liking). Well since they were using NTLM for Outlook Anywhere this didn’t really pose a problem for the users who had been allowed to use the OA rule in UAG. However, the users who had been blocked (the majority of their users) would get Outlook auth prompts.

This raised another question from the Consultant & the client; why does enabling Outlook Anywhere on your Client Access Server result in all Outlook clients being enabled for Outlook Anywhere? Shouldn’t there be a method to disable it by default & only enable it via AutoDiscover in Outlook on the mailboxes we choose? Well I’m not Microsoft so I couldn’t answer that but what I was able to do was give them a much better solution going forward which wouldn’t require the hassle of managing group membership for the UAG rule.

Background:

When you enable Outlook Anywhere on your Client Access Server (Exchange 2007/2010), AutoDiscover will then start handing out information to all Outlook Clients on how to connect via OA if a direct RPC/MAPI/TCPIP isn’t available. This allows external Outlook clients to connect to their Mailbox without the use of a VPN.

Exchange AutoDiscover hands these out using what’s called Outlook Providers. These allow Administrators & Exchange itself to differentiate between the various settings used with Outlook Anywhere VS direct RPC/MAPI/TCPIP connections.

The EXCH Outlook Provider is used to hand out settings used when connecting via RPC/MAPI/TCPIP while the EXPR Outlook Provider is used to hand out settings when connecting via Outlook anywhere (RPC over HTTPS). You can view the settings of each by running Get-OutlookProvider | Format-List.

This is the response received using the Test E-mail AutoConfiguration utility in Outlook for a mailbox after Outlook Anywhere has been enabled in the environment. This image shows the EXCH settings.

This image shows the EXPR settings received in the same AutoDiscover response. These are the settings Outlook will use to connect to Outlook Anywhere if it needs to. Notice here it says “Exchange HTTP” for the Protocol opposed to “Exchange RPC” in the previous image.

Below you’ll find the XML response from the “XML” tab of the Test E-mail AutoConfiguration utility. You can see the settings for both the EXCH & EXPR Outlook Providers.

<Protocol>

        <Type>EXCH</Type>

        <Server>CASArrayAustin.contoso.local</Server>

        <ServerDN>/o=Contoso/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=CASArrayAustin.contoso.local</ServerDN>

        <ServerVersion>7383807B</ServerVersion>

        <MdbDN>/o=Contoso/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=CASArrayAustin.contoso.local/cn=Microsoft Private MDB</MdbDN>

        <PublicFolderServer>EX10A.contoso.local</PublicFolderServer>

        <AD>ausdc.contoso.local</AD>

        <ASUrl>https://mail.ash.org/ews/exchange.asmx</ASUrl&gt;

        <EwsUrl>https://mail.ash.org/ews/exchange.asmx</EwsUrl&gt;

        <EcpUrl>https://mail.ash.org/ecp/</EcpUrl&gt;

        <EcpUrl-um>?p=customize/voicemail.aspx&amp;exsvurl=1</EcpUrl-um>

        <EcpUrl-aggr>?p=personalsettings/EmailSubscriptions.slab&amp;exsvurl=1</EcpUrl-aggr>

        <EcpUrl-mt>PersonalSettings/DeliveryReport.aspx?exsvurl=1&amp;IsOWA=&lt;IsOWA&gt;&amp;MsgID=&lt;MsgID&gt;&amp;Mbx=&lt;Mbx&gt;</EcpUrl-mt>

        <EcpUrl-ret>?p=organize/retentionpolicytags.slab&amp;exsvurl=1</EcpUrl-ret>

        <EcpUrl-sms>?p=sms/textmessaging.slab&amp;exsvurl=1</EcpUrl-sms>

        <OOFUrl>https://mail.ash.org/ews/exchange.asmx</OOFUrl&gt;

        <UMUrl>https://mail.ash.org/ews/UM2007Legacy.asmx</UMUrl&gt;

        <OABUrl>https://mail.ash.org/oab/69ed661e-c685-4ae2-a284-da308d7bd480/</OABUrl&gt;

      </Protocol>

<Protocol>

        <Type>EXPR</Type>

        <Server>oa.ash.org</Server>

        <SSL>On</SSL>

        <AuthPackage>Basic</AuthPackage>

        <ASUrl>https://mail.ash.org/ews/exchange.asmx</ASUrl&gt;

        <EwsUrl>https://mail.ash.org/ews/exchange.asmx</EwsUrl&gt;

        <EcpUrl>https://mail.ash.org/ecp/</EcpUrl&gt;

        <EcpUrl-um>?p=customize/voicemail.aspx&amp;exsvurl=1</EcpUrl-um>

        <EcpUrl-aggr>?p=personalsettings/EmailSubscriptions.slab&amp;exsvurl=1</EcpUrl-aggr>

        <EcpUrl-mt>PersonalSettings/DeliveryReport.aspx?exsvurl=1&amp;IsOWA=&lt;IsOWA&gt;&amp;MsgID=&lt;MsgID&gt;&amp;Mbx=&lt;Mbx&gt;</EcpUrl-mt>

        <EcpUrl-ret>?p=organize/retentionpolicytags.slab&amp;exsvurl=1</EcpUrl-ret>

        <EcpUrl-sms>?p=sms/textmessaging.slab&amp;exsvurl=1</EcpUrl-sms>

        <OOFUrl>https://mail.ash.org/ews/exchange.asmx</OOFUrl&gt;

        <UMUrl>https://mail.ash.org/ews/UM2007Legacy.asmx</UMUrl&gt;

        <OABUrl>https://mail.ash.org/oab/69ed661e-c685-4ae2-a284-da308d7bd480/</OABUrl&gt;

      </Protocol>

      <Protocol>

This image shows the actual Outlook Anywhere settings being configured on the client as a result of the AutoDiscover EXPR response. (File>Account Settings>Change>More Settings>Connection)

Resolution:

So the solutions here is actually fairly easy & oddly enough, not well known. The Set-CASMailbox command can be used to block a particular mailbox from accessing various Client Access features. In this case we can use it to block Outlook Anywhere for John’s Mailbox. (Note: This command can also be scripted or piped to take effect on any number of mailboxes in the environment).

Set-CASMailbox –Identity John –MAPIBlockOutlookRpcHttp $True

After running this command you may need to wait about 15min for AD replication to take effect as well as 15min for AutoDiscover, Outlook Anywhere, & ultimately Outlook to take the change. To speed this process up you can recycle the MSExchangeAutodiscoverAppPool in IIS as well as restart the Microsoft Exchange Service Host service on each CAS.

Now, if you run Test E-mail AutoConfiguraton you’ll see that the Outlook client doesn’t even get the EXPR response because they’ve had that feature blocked.

Now if you look at the Outlook Anywhere settings (below) in Outlook, they are no longer even configured/enabled. Now in my lab using Outlook 2013 I had to actually perform a profile repair to get this change to take effect immediately. You will likely either have to wait longer for it to take effect or manually repair the profile.

So in this customer’s case, users who were not allowed to use Outlook Anywhere would not get the Outlook Authentication prompt when moving from internal wired to wireless or vice versa because their Outlook client never attempted the Outlook Anywhere connection; they just remained in a disconnected state until the new connection came fully online.

Also, after showing the customer this command they no longer had to rely on UAG to control who could or couldn’t access Outlook Anywhere; they could now just script the Set-CASMailbox command.

SBS 2008 and RU3 and Outlook Repeat Login Prompt

UPDATE Exchange RU9 is supposed to fix this

On Small Business Server 2008 we have see that if you install RU3 you may start getting repeatedly prompted for log in credentials when opening outlook.

I believe that is because of some additional NTLM security that is put in place which I believe is included with RU3 based on this blurb in the update documentation

“After you install this update, the authentication mode that is set for the Web site at http://companyweb changes from the NTLM authentication to the Kerberos authentication. The reason this change occurs is because we recommend that you use the Kerberos authentication instead of the NTLM authentication.”

The solution I found so far was

1. Remove RU3 for now.
2. re-apply the Windows Authentication on the EWS, OAB, and Autodiscover virtual directories
   1. Open Internet information Services (IIS) Manager
   2. Expand the Server -> Sites -> SBS Web Applications
   3. Click on autodiscover -> authentication ->
      
   4. windows Authentication -> Disable -> enable
      
   5. Repeat for OAB and EWS virtual directories
3. restart the server

Note: It may be necessary to add the root exchange URL to the “intranet sites” for on the clients

Tools -> Internet options -> security tab -> Local Intranet -> Sites -> advanced -> add -> https://remote.domain.com