DatabaseCopyAutoActivationPolicy Setting Breaks Client Access in Exchange 2013 CU2

This issue comes fresh from a Microsoft Crit-Sit case I was just on for one of our customers.

Issue:

All client access was broken (specifically OWA) on a standalone Multi-Role Exchange 2013 CU2 Server. User’s would receive “The website cannot display the page” after authenticating to OWA. This started after the customer installed CU2.

Also, if you look in the HTTPProxy logs (C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Owa) you would see the following error:

“ServerLocatorError,POST,,,,, The database with ID 5105a9bc-cfcd-4842-baaf-451561550e08 couldn’t be found. —> Microsoft.Exchange.Data.ApplicationLogic.Cafe.MailboxServerLocatorException: The database with ID 5105a9bc-cfcd-4842-baaf-451561550e08 couldn’t be found”

Full Error:

 

Resolution:

After a night’s worth of troubleshooting we escalated to Tier 3 in Microsoft Support & our resolution came from a setting I would not at all have expected. Before installing CU2 the customer had read a blog stating some maintenance steps he should perform on his Exchange Server beforehand. One of them was running “Set-MailboxServer -Identity EXServerName -DatabaseCopyAutoActivationPolicy Blocked”. This customer did not have a DAG so this command was not needed but nonetheless this command should have absolutely no ill effect on the ability of CAS to proxy requests to the mailbox server components. All this command should do is tell the DAG that no mailbox database copies can be automatically activated on this server. It would take an admin action to override this & activate the database. But again, no DAG so it should not matter.

However in this case it was causing CAS to break as it could not find the mailbox database. I was able to replicate this issue in my own lab by setting my DatabaseCopyAutoActivationPolicy to Blocked on my two Exchange 2013 CU2 Mailbox Servers (also not in a DAG so the setting “should” not matter). After making the change & restarting some services I was greeted with the very same errors when trying to login to OWA. I also received the very same “ServerLocatorError” “The database with ID <GUID> couldn’t be found”.

So the resolution in this case is to just run “Set-MailboxServer -Identity EXServerName -DatabaseCopyAutoActivationPolicy Unrestricted”

I was told Microsoft Support would escalate this internally but I am currently unsure if this affects only CU2 or all Exchange 2013 builds as my lab is only 2013 CU2. I’m also unsure if this only affects multi-role servers or only servers not in a DAG but I hope to test & report the findings.

Update: I’ve been told by others that this setting has this same impact on CU1 systems.

Update#2: We tried asking MS Support to classify this as a bug so it would be fixed (also so our support bill would be compensated as is the case with all bugs). However, they would not agree to classify this as a bug. The answer from Support was “the fact that is was not easy to find is simply due to the complexity/functionality of our product”. We were told that if we wanted to push harder to classify it as a bug then we would first have to write-up a business impact statement & then it could be tested/researched internally. However, if the Product Team did not deem it a bug then we would be charged for the hours spent testing. I’m pretty disappointed in this response.

Update#3 I’ve tested & this still affects CU3 systems.

Update#4 I’ve tested & this still affects SP1 systems.

External Connections to Internal Web Servers Over 443 Fail

Probably one of the strangest issues I’ve seen, at least it seemed that way at the time.

Scenario:

2 internal web servers experiencing the symptom, one running 2010 OWA and the other a custom web application on 443. All internal users can hit each page just fine. External users cannot hit the pages and they just receive a timeout. However, if the admin logs into either of the two servers locally or via RDP and then you try again externally, it works and they can hit the web pages. This behavior only happened on 443. Customer was just using a Cisco ASA for their firewall with no web publishing.

Resolution:

Customer was a school district and I was reminded of a former life where I worked for a school district where web filtering was common. We found out that external users could only hit the pages when an Admin was logged into either of the servers; not a regular user. Combined with the below Cisco thread I found when trying to potentially pin this on the ASA it seemed a web filter or Intrusion Detection System was killing our connections.

https://supportforums.cisco.com/thread/2043090

According to the thread a Filter/IDS on the inside could potentially be issuing resets for web traffic that it did not like. In our case it was the customers “iBoss” content filter that started blocking access after a firmware update. It worked when an Administrator was logged into the web servers because it could filter based on the currently logged on AD account and there were exclusions for the Admins.

OWA 2007 Search only displays the first 100 results.

The default limit is 100 items in an Exchange 2007 OWA search, if you try to search for something that has more than 100 results it will only display the first 100.

 

Change the default search limit in OWA 2007

You can change this by modifying the the web.config

The default location is: C:\Program Files\Microsoft\Exchange Server\ClientAccess\Owa\ web.config

Look for the following entry and adjust.

      <add key="MaximumIdentityArraySize" value="100" />

And adjust the values to whatever you want, but keep in mind that could have a performance impact on the CAS if there is high use of large queries

Walkthrough Series: Threat Management Gateway \ Exchange publishing

Since it seems to be a popular series I wanted to consolidate links to all my TMG Publishing articles.

As you may know they are designed to be a very simple walkthrough to get you started, the in no way cover every scenario but it should be enough to get you started.Then once you have it working, backup the config and tweak to your hearts content :), Have fun securing exchange.

1. OWA
2. EWS\Outlook anywhere
   Alternate method without pre-auth: using the same listener as OWA
3. Active sync
4. SMTP 
5. Front-End \ Back-end TMG 
6. Ehlo article and Detailed white Paper From Greg Taylor.

Publishing Exchange through TMG Back-end\Front-end configuration

Can you believe it I finally got this done! This process can be used for Exchange 2007 or 2010.
This is a basic walkthrough on getting OWA published through a TMG Front-end\Back-end scenario.

Well lets get started!
First we have to establish the basic configuration

The lab will be configured as shown

First Obviously we need physical connectivity as defined.

• 2 TMG servers with 2 NICs each
• Each with a NIC on the DMZ network.
• The Frontend connected to the ISP
• The Backend connected to the LAN

Backend server

1. Configure NICs
   1. DMZ NIC =  IP: 192.168.1.2/24, Gateway: 192.168.1.1, , DNS: Null
   2. Inter NIC = IP: 192.168.2.1/24 Gateway: Null, DNS: 192.168.2.10 (Internal Domain DNS)
2. Join to domain
3. Install TMG
4. Configuration
   1. Getting Started Wizard
      
   2. Configure Network Settings
      1. Next
         
      2. Next
         
      3. Important: Choose Private at the Bottom so the BE can route.
      4. 
      5. Finish
   3. Configure System Settings
      1. I make sure mine is connected to the domain (just makes permissions easier)
         
   4. Define Deployment options
      1. This is a preference but for this Lab I disable all updates or NIS updates
   5. Remote Access Wizard (again preference But I limit config as this is a publishing lab not client access)
      1. 
      2. 
      3. 
      4. 
      5. This one can make troubleshooting difficult if configured any other way
         
      6. 
      7. 
   6. Network Rule Creation
      1. Edit the Internal to Perimeter Rule
         
      2. 
      3. 
   7. Firewall Rule Creation
      1. 
      2. 
      3. 
      4. 
         
         
      5. 
      6. 
      7. 
      8. 
      9. 
      10. 
      11. 
      12. 

Front-end server

1. Configure NICs
   1. DMZ NIC =  IP: 192.168.1.1/24, Gateway: 192.168.1.1, , DNS: 192.168.2.10 (Internal Domain DNS)
   2. Inter NIC = IP: ISP assigned Gateway: ISP assigned, DNS: null
2. Install TMG
3. Configuration
   1. Getting Started Wizard
      
   2. Configure Network Settings
      1. Next
      2. 
      3. Be sure to add the additional route for the LAN network behind the back-end server.
         This also adds the internal LAN network to the Internal Network object(networking\networks), and adds a static route for the Internal network as well (Networking\routing tab) 
         
      4. In my case I have a dynamic IP in my lab, but this would be your ISP provided IP
         
      5. 
      6. At this point you should have routing connectivity to the domain.
   3. Configure System Settings
      1. I make sure mine is connected to the domain (just makes permissions easier)
         
         You can join the domain here
   4. Define Deployment options
      1. This is a preference but for this Lab I disable all updates or NIS updates
   5. Remote Access Wizard (again preference But I limit config as this is a publishing lab not client access)
      1. 
      2. 
      3. 
      4. 
      5. This one can make troubleshooting difficult if configured any other way
         
      6. 
      7. 
   6. Publishing Rules (Same as previous Posts, sample here see other posts for more details)
      1. 
      2. This is a basic auth listener that will work for OWA\EAS\OLA but doesn’t include forms
      3. 
      4. 
      5. 
      6. 
      7. Make sure this Name is accessible from the FE server (the name also needs to be on the trusted certificate on the exchange server)
         
      8. 
      9. 
      10. 
      11. 
      12. 
      13. 
      14. 
      15. 
      16. 
      17. This may change based on your scenario
          
      18. 
      19. Finish
   7. Apply Changes and Test!!!

 

Note: I also like to create a test Rule that I leave disabled unless I need to determine if I have a Firewall rule issue or Network issue.

Here is another reference for the same process in a slightly different scenario
http://araihan.wordpress.com/2010/06/17/how-to-configure-back-to-back-firewall-with-perimeter-dmz-topology-step-by-step-guide/

Publish Exchange 2010 with TMG (cont)

Walkthrough on publishing all roles through TMG with AD pre-auth on TMG. (Part 3/4 active sync)

Configure Active sync rule on TMG

1. Open Forefront TMG
2. Click on
3. In the Action Pane under Task click 
4. Give the rule a Name ill name mine “2010 Activesync”
5. 
6. Next –> Next
7. 
8. Internal Site Name should be your CAS server FQDN (needs to be on the cert)
9. 
10. The external name is what you use to access active sync(Also needs to be on the cert)
11. 
12. Select the Listener OA listener created on Part 2.
13. 
14. 
15. 
16. Finish
17. Now Outlook anywhere is published!

Go Back To OWA

Go Back to Outlook anywhere

Move on to SMTP

Publish Exchange 2010 with TMG (Forefront Threat Management Gateway) Series:

1. OWA
2. EWS\Outlook anywhere
3. Active sync
4. SMTP

Issues sending to 2003 mail store

If your having problems sending mail to a particular store user but OWA still works.

You receive an immediate NDR with the following text:
There’s a problem with the recipient’s mailbox. Please try resending this message. If the problem continues, please contact your helpdesk.

You may have a store corruption issue.

• To resolve either move users to a new mail store
• or
• run isinteg –fix –test alltests on the database (this will require down time)

Another possibility (same solution is that there are too many named properties (look for 9667 events) – in this case you have to move users to another store\database.

 

Re-create Exchange 2007 OWA Virtual Directories..

Have you had issues with OWA where you needed to re-create the directories? or maybe you made customizations and now it doesn’t work and you want to just get back to the default config?

Here are some scripts to get re-install the OWA virtual directories, they have been tested in lab environments

As with everything use at your own risk and always make a backup first!
YOU are responsible for your server\data.

ONLY do this if you have 1 CAS server and have not customized the OWA directories.

Save this as rebuildOWA.ps1 and run from powershell

************************Start of script**********************************************

$server = hostname

Get-OwaVirtualDirectory -server $server | Remove-OwaVirtualDirectory

New-OwaVirtualDirectory -name “owa” -OwaVersion Exchange2007 -WebSiteName “Default Web Site”

New-OwaVirtualDirectory “exchange” -OwaVersion Exchange2003or2000 -VirtualDirectoryType Mailboxes -WebSiteName “Default Web Site”

New-OwaVirtualDirectory “public” -OwaVersion Exchange2003or2000 -VirtualDirectoryType PublicFolders -WebSiteName “Default Web Site”

New-OwaVirtualDirectory “exchweb” -OwaVersion Exchange2003or2000 -VirtualDirectoryType Exchweb -WebSiteName “Default Web Site”

new-owavirtualdirectory “Exadmin” -owaversion:Exchange2003or2000 -virtualDirectoryType Exadmin -WebSiteName “Default Web Site”

Get-AutodiscoverVirtualDirectory | Remove-AutodiscoverVirtualDirectory

 New-AutodiscoverVirtualDirectory -WebsiteName “Default Web Site” -BasicAuthentication $true -WindowsAuthentication $true

IISRESET

************************End of script********************************************

Here is one for SBS 2008

Save this as rebuildSBSOWA.ps1 and run from powershell

************************Start of script**********************************************

$LocalServerName = hostname

$ActiveSyncMailboxName = “Windows SBS Mobile Mailbox Policy” + ” ” + $LocalServerName

$OABVDir = $LocalServerName + “\OAB (SBS Web Applications)”

$OAB = Get-OfflineAddressBook | Select-Object -Property Name

$strDomainDNS = [System.DirectoryServices.ActiveDirectory.Domain]::GetCurrentDomain().Name

$DomainAdmins = $strDomainDNS + “\Domain Admins”

$OrgName = Get-OrganizationConfig | Select-Object -Property DistinguishedName

$DefaultExchangeCertificate = “CN=” + $LocalServerName

Get-ExchangeCertificate | Where { $_.Subject -eq “$DefaultExchangeCertificate” } | ForEach { Remove-ExchangeCertificate -Thumbprint $_.Thumbprint }

Get-OwaVirtualDirectory -server $LocalServerName | Remove-OwaVirtualDirectory

New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2007” -ExternalAuthenticationMethods Fba

Set-OWAVirtualDirectory -InternalUrl “https://sites/owa/&#8221; -ClientAuthCleanupLevel “Low” -LogonFormat “UserName” -DefaultDomain $strDomainDNS -Identity “Owa (SBS Web Applications)”

New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000” -VirtualDirectoryType “Exadmin” -ExternalAuthenticationMethods Fba

New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000” -VirtualDirectoryType “Mailboxes” -ExternalAuthenticationMethods Fba

New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000” -VirtualDirectoryType “Exchweb” -ExternalAuthenticationMethods Fba

New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000” -VirtualDirectoryType “PublicFolders” -ExternalAuthenticationMethods Fba

iisreset /noforce

cd $env:windir\system32\inetsrv

.\appcmd.exe unlock config “-section:system.webserver/security/authentication/windowsauthentication”

.\appcmd.exe set site “Default Web Site” /Bindings:http/*:80:

.\appcmd.exe start site “Default Web Site”

.\appcmd.exe start site “SBS Web Applications”

************************End of script********************************************

Problem logging into OWA after installing Exchange 2010

(UPDATE: This is resolved by Exchange RU9)

Do you still have have Exchange 2007 and 2010  coexisting in the same environment and after you installed 2010 come users couldn’t access OWA anymore.?

Your getting this error:

“The mailbox you’re trying to access isn’t currently available. If the problem continues, contact your helpdesk”

AND in the application event log you have and Event ID 46 source MSExchange OWA

The issue may be you are trying to us CAS 2010 to CAS 2007 proxy and you don’t have the correct files on the 2010 server.

To resolve

1. From the 2007 Exchange server copy the following folder to 2010:

      “\Program Files\Microsoft\Exchange Server\Client Access\Owa\8.x.xxx.x“

Copy the highest numbered 8.x.xxx.x folder to the Exchange 2010 Client Access server:

      “\Program Files\Microsoft\Exchange Server\V14\Client Access\Owa”

the 2010 OWA directory should look some thing like this

2. Then run IISRESET from and elevated command prompt

Problems with Autodiscover, Out of Office, Free Busy, OWA and Outlook Anywhere

One of the most common issues I see has to do with certificates, so to start out we need to understand some things about certificates.

Certificates are used to encrypt traffic between exchange servers and clients.

There are 3 things that need to be true for a certificate to be valid.

1. The name used to access the resource needs match the certificate exactly.
   Example: If I connect to say owa with mail.mydomain.com then the certificate needs to also have mail.mydomain.com on it in either the subject or the subject alternate name field.
2. The Certificate time must be valid
3. The issuing Certificate Authority must be trusted by the client. (It needs to exist in the “Trusted Root Certificate Authorities)

Now that we have some VERY basic info about certificates.

The issues I see constantly are: Autodiscover, Out of Office, Free Busy and Outlook Anywhere miss-configuration.

Reasons:

1. Not using a trusted certificate
   • Solution: use a 3^rd party cert provider
2. The certificate name does not match the DNS name\s
   • Solution: create a new cert request containing all the names used to access the server. Minimum of
     1. Autodiscover.domain.com
     2. <ExternalName>.domain.com
     3. <InternalName>.domain.local (if using for internal systems also)

Example of a correct cert request:

•  
  • New-ExchangeCertificate -GenerateRequest -SubjectName “C=US, O=Org Name, CN=mail.domain.com” -domainname mail.domain.com, autodiscover.domain.com, servername, servername.domain.local -FriendlyName mail.domain.com -privatekeyexportable:$true -path c:\cert_myserver.txt

Example of Cert import

•  
  • Import-ExchangeCertificate –Path “C:\CertificateFile.cer” | Enable-ExchangeCertificate -Services pop, smtp, iis, imap  (2007 Example)
  • Import-ExchangeCertificate -FileData ([Byte[]]$(Get-Content -Path c:\certificates\newcert.cer -Encoding Byte -ReadCount 0)) | Enable-ExchangeCertificate -Services SMTP   (2010 Example)

1. External URLs not defined correctly
   • Solution: Configure the External urls
   • Set-WebServicesVirtualDirectoy -ID server.domain.com\* -externalurl https://mail.domain.com/ews/exchange.asmx
   • Set-OABVirtualDirectoy -ID server.domain.com\* -externalurl http://mail.com.com/OAB
   • Set-UMVirtualDirectoy -ID server.domain.com\*-externalurl https://server.com.com/UnifiedMessaging/services.asmx
2. Can’t resolve Fully qualified domain names (FQDN)
   • Solution: make sure that the FQDNs for your external URLs as well as autodiscover have A records registered in DNS
   • Verify you can access the autodiscover XML file https://autodiscover.domain.com/autodiscover/autodiscover.xml

           Should look like this

        

1. SCP Record does not contain the correct value
   1. Test from outlook:
      1. Hold CTRL and Click the outlook Icon in the system tray and select “Test Email Auto Configuration”
      2. Uncheck guess smart and click Test
   2. check SCP value returned
      1. If you get info on the results tab then autodiscover is working
      2. If not look at the Log tab and look at the URL that is returned

             

1.  
   1. Test the URL (Type it into Internet explorer) if its not change SCP to a valid URL
      1. Run ADSIEDIT and view the “Service Binding Information” to verify the correct value

                    

2. Set the SCP allong with the internal URL: Set-ClientAccessServer CASServerName -AutoDiscoverServiceInternalUri https://mail.domain.local/Autodiscover/Autodiscover.xml