External Connections to Internal Web Servers Over 443 Fail

Probably one of the strangest issues I’ve seen, at least it seemed that way at the time.

Scenario:

2 internal web servers experiencing the symptom, one running 2010 OWA and the other a custom web application on 443. All internal users can hit each page just fine. External users cannot hit the pages and they just receive a timeout. However, if the admin logs into either of the two servers locally or via RDP and then you try again externally, it works and they can hit the web pages. This behavior only happened on 443. Customer was just using a Cisco ASA for their firewall with no web publishing.

Resolution:

Customer was a school district and I was reminded of a former life where I worked for a school district where web filtering was common. We found out that external users could only hit the pages when an Admin was logged into either of the servers; not a regular user. Combined with the below Cisco thread I found when trying to potentially pin this on the ASA it seemed a web filter or Intrusion Detection System was killing our connections.

https://supportforums.cisco.com/thread/2043090

According to the thread a Filter/IDS on the inside could potentially be issuing resets for web traffic that it did not like. In our case it was the customers “iBoss” content filter that started blocking access after a firmware update. It worked when an Administrator was logged into the web servers because it could filter based on the currently logged on AD account and there were exclusions for the Admins.

Disable TOE and RSS

These technologies are great if your environment support them end to end, but if not you may see some of the following issues.

Symptoms include

• Sporadic Network issues.
• Service failing (Because of network login issues)
• Delay in service start (Because of network login issues)
• Unexplained issues that Seem to be network related but other areas have already been investigated

 

Resolution: To keep it as simple and reliable as possible

1. Update to latest drivers
2. Disable Everything that says offload or scaling in the NIC properties
3. Disable it for the OS as well

2008      

• netsh interface tcp set global rss=disabled
• netsh interface tcp set global chimney=disabled
• netsh interface tcp set global autotuninglevel=disabled

2003

• Netsh int ip set chimney disabled

DAG Cross Site\Subnet networking – Additional Configuration

When you add servers to a DAG it will create a network for every subnet\NIC that server is connected to, this is nice because as soon as you add the server it can replicate with the other nodes.
However there are some post configuration steps you need to take otherwise replication will occur over the MAPI\Client network and never use the replication network.

1. You should NEVER have multiple gateways, if you have a private\heartbeat network that is routed you need to remove the gateway and add a static route.
   Example: You configure the gateway on the public NICs  
   and configure the following static routes:
   Site A 
   Site B 
    
2. Next you will notice your “DAG Networks” may look something like this
   The issue with this configuration is that there is no clearly defined replication or mapi networks, so what we need to do is collapse them into 2 dag networks.
3. Modify the networks to include both subnets, (I named mine for easy identification.
   i.e. Combine 10.0.1.x with 10.0.2.x and 192.168.2.x and 192.168.1.x
   
4. I would also recommend disabling replication on the MAPI or client network, (it will be used anyway if the replication network is not available.

You should now be replicating over the replication network, you can verify with the following:

• Get-MailboxDatabaseCopyStatus <DatabaseName> -ConnectionStatus | fl name, outgoingconnections,incomminglogcopyingnetwork