DatabaseCopyAutoActivationPolicy Setting Breaks Client Access in Exchange 2013 CU2


This issue comes fresh from a Microsoft Crit-Sit case I was just on for one of our customers.

Issue:

All client access was broken (specifically OWA) on a standalone Multi-Role Exchange 2013 CU2 Server. User’s would receive “The website cannot display the page” after authenticating to OWA. This started after the customer installed CU2.

Also, if you look in the HTTPProxy logs (C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Owa) you would see the following error:

“ServerLocatorError,POST,,,,, The database with ID 5105a9bc-cfcd-4842-baaf-451561550e08 couldn’t be found. —> Microsoft.Exchange.Data.ApplicationLogic.Cafe.MailboxServerLocatorException: The database with ID 5105a9bc-cfcd-4842-baaf-451561550e08 couldn’t be found

Full Error:
Capture

 

Resolution:

After a night’s worth of troubleshooting we escalated to Tier 3 in Microsoft Support & our resolution came from a setting I would not at all have expected. Before installing CU2 the customer had read a blog stating some maintenance steps he should perform on his Exchange Server beforehand. One of them was running “Set-MailboxServer -Identity EXServerName -DatabaseCopyAutoActivationPolicy Blocked”. This customer did not have a DAG so this command was not needed but nonetheless this command should have absolutely no ill effect on the ability of CAS to proxy requests to the mailbox server components. All this command should do is tell the DAG that no mailbox database copies can be automatically activated on this server. It would take an admin action to override this & activate the database. But again, no DAG so it should not matter.

However in this case it was causing CAS to break as it could not find the mailbox database. I was able to replicate this issue in my own lab by setting my DatabaseCopyAutoActivationPolicy to Blocked on my two Exchange 2013 CU2 Mailbox Servers (also not in a DAG so the setting “should” not matter). After making the change & restarting some services I was greeted with the very same errors when trying to login to OWA. I also received the very same “ServerLocatorError” “The database with ID <GUID> couldn’t be found”.

So the resolution in this case is to just run “Set-MailboxServer -Identity EXServerName -DatabaseCopyAutoActivationPolicy Unrestricted

I was told Microsoft Support would escalate this internally but I am currently unsure if this affects only CU2 or all Exchange 2013 builds as my lab is only 2013 CU2. I’m also unsure if this only affects multi-role servers or only servers not in a DAG but I hope to test & report the findings.

Update: I’ve been told by others that this setting has this same impact on CU1 systems.

Update#2: We tried asking MS Support to classify this as a bug so it would be fixed (also so our support bill would be compensated as is the case with all bugs). However, they would not agree to classify this as a bug. The answer from Support was “the fact that is was not easy to find is simply due to the complexity/functionality of our product”. We were told that if we wanted to push harder to classify it as a bug then we would first have to write-up a business impact statement & then it could be tested/researched internally. However, if the Product Team did not deem it a bug then we would be charged for the hours spent testing. I’m pretty disappointed in this response.

Update#3 I’ve tested & this still affects CU3 systems.

Update#4 I’ve tested & this still affects SP1 systems.

Advertisements

External Connections to Internal Web Servers Over 443 Fail


Probably one of the strangest issues I’ve seen, at least it seemed that way at the time.

Scenario:

2 internal web servers experiencing the symptom, one running 2010 OWA and the other a custom web application on 443. All internal users can hit each page just fine. External users cannot hit the pages and they just receive a timeout. However, if the admin logs into either of the two servers locally or via RDP and then you try again externally, it works and they can hit the web pages. This behavior only happened on 443. Customer was just using a Cisco ASA for their firewall with no web publishing.

Resolution:

Customer was a school district and I was reminded of a former life where I worked for a school district where web filtering was common. We found out that external users could only hit the pages when an Admin was logged into either of the servers; not a regular user. Combined with the below Cisco thread I found when trying to potentially pin this on the ASA it seemed a web filter or Intrusion Detection System was killing our connections.

https://supportforums.cisco.com/thread/2043090

According to the thread a Filter/IDS on the inside could potentially be issuing resets for web traffic that it did not like. In our case it was the customers “iBoss” content filter that started blocking access after a firmware update. It worked when an Administrator was logged into the web servers because it could filter based on the currently logged on AD account and there were exclusions for the Admins.

OWA 2007 Search only displays the first 100 results.


The default limit is 100 items in an Exchange 2007 OWA search, if you try to search for something that has more than 100 results it will only display the first 100.

image

 

Change the default search limit in OWA 2007

You can change this by modifying the the web.config

The default location is: C:\Program Files\Microsoft\Exchange Server\ClientAccess\Owa\ web.config

Look for the following entry and adjust.

      <add key="MaximumIdentityArraySize" value="100" />

image

And adjust the values to whatever you want, but keep in mind that could have a performance impact on the CAS if there is high use of large queries

Walkthrough Series: Threat Management Gateway \ Exchange publishing


Since it seems to be a popular series I wanted to consolidate links to all my TMG Publishing articles.

As you may know they are designed to be a very simple walkthrough to get you started, the in no way cover every scenario but it should be enough to get you started.Then once you have it working, backup the config and tweak to your hearts content :), Have fun securing exchange.

  1. OWA
  2. EWS\Outlook anywhere
    Alternate method without pre-auth: using the same listener as OWA
  3. Active sync
  4. SMTP 
  5. Front-End \ Back-end TMG 
  6. Ehlo article and Detailed white Paper From Greg Taylor.

Publishing Exchange through TMG Back-end\Front-end configuration


Can you believe it I finally got this done! This process can be used for Exchange 2007 or 2010.
This is a basic walkthrough on getting OWA published through a TMG Front-end\Back-end scenario.

Well lets get started!
First we have to establish the basic configuration

The lab will be configured as shown

image

First Obviously we need physical connectivity as defined.

  • 2 TMG servers with 2 NICs each
  • Each with a NIC on the DMZ network.
  • The Frontend connected to the ISP
  • The Backend connected to the LAN

Backend server

  1. Configure NICs
    1. DMZ NIC =  IP: 192.168.1.2/24, Gateway: 192.168.1.1, , DNS: Null
    2. Inter NIC = IP: 192.168.2.1/24 Gateway: Null, DNS: 192.168.2.10 (Internal Domain DNS)
  2. Join to domain
  3. Install TMG
  4. Configuration
    1. Getting Started Wizard
      image
    2. Configure Network Settings
      1. Next
        image
      2. Next
        image
      3. Important: Choose Private at the Bottom so the BE can route.
      4. image
      5. Finish
    3. Configure System Settings
      1. I make sure mine is connected to the domain (just makes permissions easier)
        image[65]
    4. Define Deployment options
      1. This is a preference but for this Lab I disable all updates or NIS updates
    5. Remote Access Wizard (again preference But I limit config as this is a publishing lab not client access)
      1. image
      2. image
      3. image
      4. image
      5. This one can make troubleshooting difficult if configured any other way
        image
      6. image
      7. image
    6. Network Rule Creation
      1. Edit the Internal to Perimeter Rule
        image
      2. image
      3. image
    7. Firewall Rule Creation
      1. image
      2. image
      3. image
      4. image
        image
        image
      5. image
      6. image
      7. imageimage
      8. imageimage
      9. image
      10. image
      11. image
      12. image

Front-end server

  1. Configure NICs
    1. DMZ NIC =  IP: 192.168.1.1/24, Gateway: 192.168.1.1, , DNS: 192.168.2.10 (Internal Domain DNS)
    2. Inter NIC = IP: ISP assigned Gateway: ISP assigned, DNS: null
  2. Install TMG
  3. Configuration
    1. Getting Started Wizard
      image
    2. Configure Network Settings
      1. Next
      2. image
      3. Be sure to add the additional route for the LAN network behind the back-end server.
        This also adds the internal LAN network to the Internal Network object(networking\networks), and adds a static route for the Internal network as well (Networking\routing tab) 
        imageimage
      4. In my case I have a dynamic IP in my lab, but this would be your ISP provided IP
        image
      5. image
      6. At this point you should have routing connectivity to the domain.
    3. Configure System Settings
      1. I make sure mine is connected to the domain (just makes permissions easier)
        image
        You can join the domain here
    4. Define Deployment options
      1. This is a preference but for this Lab I disable all updates or NIS updates
    5. Remote Access Wizard (again preference But I limit config as this is a publishing lab not client access)
      1. image
      2. image
      3. image
      4. image
      5. This one can make troubleshooting difficult if configured any other way
        image
      6. image
      7. image
    6. Publishing Rules (Same as previous Posts, sample here see other posts for more details)
      1. image
      2. This is a basic auth listener that will work for OWA\EAS\OLA but doesn’t include forms
      3. image
      4. image
      5. image
      6. image
      7. Make sure this Name is accessible from the FE server (the name also needs to be on the trusted certificate on the exchange server)
        image
      8. image
      9. image
      10. image
      11. image
      12. image
      13. image
      14. image
      15. image
      16. image
      17. This may change based on your scenario
        image
      18. image
      19. Finish
    7. Apply Changes and Test!!!

 

Note: I also like to create a test Rule that I leave disabled unless I need to determine if I have a Firewall rule issue or Network issue.
image

Here is another reference for the same process in a slightly different scenario
http://araihan.wordpress.com/2010/06/17/how-to-configure-back-to-back-firewall-with-perimeter-dmz-topology-step-by-step-guide/

Publish Exchange 2010 with TMG (cont)


Walkthrough on publishing all roles through TMG with AD pre-auth on TMG. (Part 3/4 active sync)

Configure Active sync rule on TMG

  1. Open Forefront TMG
  2. Click on image_thumb5[1]
  3. In the Action Pane under Task click  image_thumb6[2]
  4. Give the rule a Name ill name mine “2010 Activesync”
  5. image
  6. Next –> Next
  7. image
  8. Internal Site Name should be your CAS server FQDN (needs to be on the cert)
  9. image
  10. The external name is what you use to access active sync(Also needs to be on the cert)
  11. image
  12. Select the Listener OA listener created on Part 2.
  13. image_thumb24[1]
  14. image
  15. image
  16. Finish
  17. Now Outlook anywhere is published!
  • Go Back To OWA
  • Go Back to Outlook anywhere

  • Move on to SMTP

    Publish Exchange 2010 with TMG (Forefront Threat Management Gateway) Series:

    1. OWA
    2. EWS\Outlook anywhere
    3. Active sync
    4. SMTP

  • Issues sending to 2003 mail store


    If your having problems sending mail to a particular store user but OWA still works.

    You receive an immediate NDR with the following text:
    There’s a problem with the recipient’s mailbox. Please try resending this message. If the problem continues, please contact your helpdesk.

    You may have a store corruption issue.

    • To resolve either move users to a new mail store
    • or
    • run isinteg –fix –test alltests on the database (this will require down time)

    Another possibility (same solution is that there are too many named properties (look for 9667 events) – in this case you have to move users to another store\database.