Incorrectly Adding New Receive Connector Breaks Exchange 2013 Transport


I feel the concepts surrounding this issue have been mentioned already via other sources (1 2) but I’ve seen at least 5 recent cases where our customers were being adversely impacted by this issue; so it’s worth describing in detail.

Summary:

After creating new Receive Connectors on Multi-Role Exchange 2013 Servers, customers may encounter mail flow/transport issues within a few hours/days. Symptoms such as:

  • Sporadic inability to connect to the server over port 25
  • Mail stuck in the Transport Queue both on the 2013 servers in question but also on other SMTP servers trying to send to/through it
  • NDR’s being generated due to delayed or failed messages

This happens because the Receive Connector was incorrectly created (which is very easy to do), resulting in two services both trying to listen on port 25 (the Microsoft Exchange FrontEnd Transport Service & the Microsoft Exchange Transport Service). The resolution to this issue is to ensure that you specify the proper “TransportRole” value when creating the Receive Connector either via EAC or Shell. You can also edit the Receive Connector after the fact using Set-ReceiveConnector.

Detailed Description:

Historically, Exchange Servers listen on & send via port 25 for SMTP traffic as it’s the industry standard. However, you can listen/send on any port you choose as long as the parties on each end of the transmission agree upon it.

Exchange 2013 brought a new Transport Architecture & without going into a deep dive, the Client Access Server (CAS) role runs the Microsoft Exchange FrontEnd Transport Service which listens/sends on port 25 for SMTP traffic. The Mailbox Server role has the Microsoft Exchange Transport Service which is similar to the Transport Service in previous versions of Exchange & also listens on port 25. There are two other Transport Services (MSExchange Mailbox Delivery & Mailbox Submission) but they aren’t relevant to this discussion.

So what happens when both of these services reside on the same server (like when deploying Multi-Role; which is my recommendation)? In this scenario, the Microsoft Exchange FrontEnd Transport Service listens on port 25, since it is meant to handle inbound/outbound connections with public SMTP servers (which expect to use port 25). Meanwhile, the Microsoft Exchange Transport Service listens on port 2525. Because this service is used for intra-org communications, all other Exchange 2013 servers in the Organization know to send using 2525 (however, 07/10 servers still use port 25 to send to multi-role 2013 servers, which is why Exchange Server Authentication is enabled by default on your default FrontEndTransport Receive Connectors on a Multi-Role box; in case you were wondering).

So when you create a new Receive Connector on a Multi-Role Server, how do you specify which service will handle it? You do so by using the -TransportRole switch via the Shell or by selecting either “Hub Transport” or “FrontEnd Transport” under “Role” when creating the Receive Connector in the EAC.

The problem is there’s nothing keeping you from creating a Receive Connector of Role “Hub Transport” (which it defaults to) that listens on port 25 on a Multi-Role box. What you then have is two different services trying to listen on port 25. This actually works temporarily, due to some .NET magic that I’m not savvy enough to understand, but regardless, eventually it will cause issues. Let’s go through a demo.

Demo:

Here’s the output of Netstat on a 2013 Multi-Role box with default settings. You’ll see MSExchangeFrontEndTransport.exe is listening on port 25 & EdgeTransport.exe is listening on 2525. These processes correspond to the Microsoft Exchange FrontEnd Transport & Microsoft Exchange Transport Services respectively.

1new

Now let’s create a custom Receive Connector, as if we needed it to allow a network device to Anonymously Relay through Exchange (the most common scenario where I’ve seen this issue arise). Notice in the first screenshot, you’ll see the option to specify which Role should handle this Receive Connector. Also notice how Hub Transport is selected by default, as is port 25.

3

4

5

After adding this Receive Connector, see how the output of Netstat differs. We now have two different processes listening on the same port (25).

6

So there’s a simple fix to this. Just use Shell (there’s no GUI option to edit the setting after it’s been created) to modify the existing Receive Connector to be handled by the MSExchange FrontEndTransport Service instead of the MSExchange Transport Service. Use the following command:

Set-ReceiveConnector Test-Relay –TransportRole FrontEndTransport

7

I recommend you restart both Transport Services afterwards.

 

 

Update: In recent releases of Exchange 2013 (unsure which CU this fix was implemented in), the EAC will no longer let you mis configure a receive connector in this way. So hopefully we should see less of this issue.

 

Advertisements

Checking for Open Relay in Exchange 2007/2010


Scenario:

So this is a fairly common scenario & I figured I’d post an easy method to diagnose the issue. Customers will often suspect that they’re an open relay due to being placed on a blacklist or having issues sending email to certain domains. There’s some general confusion as to what constitutes as an Open Relay & even the difference between a Relay & a Submit action in SMTP terminology. Hopefully this can clear some of the confusion.

Background:

Submit = Submitting an email message to an SMTP server that is destined for a domain that exists on that server (or in that server’s environment). You’re sending it to an address that the server is authoritative for.

Relay = Submitting an email message to an SMTP server that is destined for a domain that exists in another messaging environment. You’re sending to an address that the server is not authoritative for.

So there’s nothing inherently wrong with relaying. It’s what happens if you use your Hotmail account to send an email to someone’s Gmail account. It happens every time you email someone outside of your own messaging system. The key detail is whether or not you have authenticated to the SMTP server beforehand. So when you’re using Hotmail or Exchange via Outlook/OWA then you have obviously authenticated either via an Authentication Prompt, OWA Form, or using NTLM.

So this typically comes up when a customer needs to have an application, network printer, or other device be able to send emails through Exchange (or any internal SMTP server).

So the important thing to point out here is that as long as the application/device only needs to be able to send to addresses that your SMTP server is authoritative for then it is a Submit action & not a Relay action. This just means you only need it to be able to hit a Receive Connector that allows Anonymous Submit; which is how most of the world’s SMTP servers are configured to accept email from the Internet.

However, if your application/device needs to be able to send to an address not under the authority of the local SMTP server then it will be performing an SMTP Relay action & will require additional configuration.

The recommended approach is to have the Application/Device authenticate to your SMTP server if it supports it. Alternatively, you can configure the Receive Connector (Exchange) to allow Anonymous Relaying from that Application/Device’s IP address.

For instructions please see this Microsoft Post.

http://blogs.technet.com/b/exchange/archive/2006/12/28/3397620.aspx

This is a very common issue amongst customers because they may not be familiar with how to configure this. However, unfortunately I will often see customers make an even worse mistake; allowing Anonymous Relaying from an entire range of IP Addresses or possibly the entire Internet. It won’t take long for Internet folks with malicious intent to figure this out & start using your server to SPAM whoever they wish. This typically results in your Exchange Server’s sending IP being placed on various Blacklists which can prevent you from sending to certain email domains.

Resolution:

It is ALWAYS recommended to create a separate Receive Connector for this purpose. In fact I tell customers to never mess with the Default Receive Connectors if they can get away with it. But what will ultimately happen is the customer will use the steps in the Microsoft article above to enable Anonymous Relaying on their Default Receive Connector instead, which they’re probably also using as their Internet ingress point. The problem with this is that the Remote IP range of that connector is 0.0.0.0-255.255.255.255 out of the box; meaning the entire Internet.

Another thing the customer might do is create a new Receive Connector for Relaying but instead of just having 1 IP address in there (the IP of their Application Server or Network Device) they’ll add an entire range or more IPs than are needed. This can get pretty complicated to troubleshoot if you have many different Receive Connectors on many different Exchange Servers in the environment.

So I’m hoping people can use my explanation to help them configure this properly as well as troubleshoot any issues they may have. In addition to that, here’s a very useful command to use in Exchange Management Shell to list out all Receive Connectors in the environment that have the Anonymous Relay permission enabled. Use this to track these connectors down & then verify the RemoteIP Ranges are properly scoped/configured to be as secure as possible.

Get-ReceiveConnector | Get-ADPermission -User “NT Authority\Anonymous Logon” | Where-Object {$_.ExtendedRights -like “ms-Exch-SMTP-Accept-Any-Recipient”} | Format-List Identity,ExtendedRights

Testing SMTP


So this is fairly basic but I realized I don’t have anything laying out the basic tools for testing SMTP posted.

Test with Telnet

  1. Determine the address of your SMTP server (unless you already know it or are testing local to the server)
    1. Open a command line (each line is a line return)
    2. type nslookup
      1. set type=mx
      2. domain.com
      3. Record the results
  2. Test with Telnet
    1. telnet <mail.domain.com> 25
      1. (this could be localhost or the output from the nslookup step)
      2. If you mistype past this point hit enter and try again don’t use backspace)
    2. ehlo mail.mydomain.com (you can replace ehlo with helo if your not using an exchange server)
    3. mail from: test@domain.com
    4. rcpt to: user@domain.com
    5. data
    6. Subject: This is a test
    7. Here you type your test message
    8. . (this is a period)
    9. quit

Web Based tools

Local Test tools

  • Pop3\SMTP mail client (outlook\express, thunderbird)
  • powershell
  • CMD\Telnet

Exchange 2007\2010 Powershell tools

  • Test-EdgeSynchronization (if using an edge subscription)
  • Test-Mailflow
  • Test-SmtpConnectivity

Supporting URLs

Publishing Exchange through TMG Back-end\Front-end configuration


Can you believe it I finally got this done! This process can be used for Exchange 2007 or 2010.
This is a basic walkthrough on getting OWA published through a TMG Front-end\Back-end scenario.

Well lets get started!
First we have to establish the basic configuration

The lab will be configured as shown

image

First Obviously we need physical connectivity as defined.

  • 2 TMG servers with 2 NICs each
  • Each with a NIC on the DMZ network.
  • The Frontend connected to the ISP
  • The Backend connected to the LAN

Backend server

  1. Configure NICs
    1. DMZ NIC =  IP: 192.168.1.2/24, Gateway: 192.168.1.1, , DNS: Null
    2. Inter NIC = IP: 192.168.2.1/24 Gateway: Null, DNS: 192.168.2.10 (Internal Domain DNS)
  2. Join to domain
  3. Install TMG
  4. Configuration
    1. Getting Started Wizard
      image
    2. Configure Network Settings
      1. Next
        image
      2. Next
        image
      3. Important: Choose Private at the Bottom so the BE can route.
      4. image
      5. Finish
    3. Configure System Settings
      1. I make sure mine is connected to the domain (just makes permissions easier)
        image[65]
    4. Define Deployment options
      1. This is a preference but for this Lab I disable all updates or NIS updates
    5. Remote Access Wizard (again preference But I limit config as this is a publishing lab not client access)
      1. image
      2. image
      3. image
      4. image
      5. This one can make troubleshooting difficult if configured any other way
        image
      6. image
      7. image
    6. Network Rule Creation
      1. Edit the Internal to Perimeter Rule
        image
      2. image
      3. image
    7. Firewall Rule Creation
      1. image
      2. image
      3. image
      4. image
        image
        image
      5. image
      6. image
      7. imageimage
      8. imageimage
      9. image
      10. image
      11. image
      12. image

Front-end server

  1. Configure NICs
    1. DMZ NIC =  IP: 192.168.1.1/24, Gateway: 192.168.1.1, , DNS: 192.168.2.10 (Internal Domain DNS)
    2. Inter NIC = IP: ISP assigned Gateway: ISP assigned, DNS: null
  2. Install TMG
  3. Configuration
    1. Getting Started Wizard
      image
    2. Configure Network Settings
      1. Next
      2. image
      3. Be sure to add the additional route for the LAN network behind the back-end server.
        This also adds the internal LAN network to the Internal Network object(networking\networks), and adds a static route for the Internal network as well (Networking\routing tab) 
        imageimage
      4. In my case I have a dynamic IP in my lab, but this would be your ISP provided IP
        image
      5. image
      6. At this point you should have routing connectivity to the domain.
    3. Configure System Settings
      1. I make sure mine is connected to the domain (just makes permissions easier)
        image
        You can join the domain here
    4. Define Deployment options
      1. This is a preference but for this Lab I disable all updates or NIS updates
    5. Remote Access Wizard (again preference But I limit config as this is a publishing lab not client access)
      1. image
      2. image
      3. image
      4. image
      5. This one can make troubleshooting difficult if configured any other way
        image
      6. image
      7. image
    6. Publishing Rules (Same as previous Posts, sample here see other posts for more details)
      1. image
      2. This is a basic auth listener that will work for OWA\EAS\OLA but doesn’t include forms
      3. image
      4. image
      5. image
      6. image
      7. Make sure this Name is accessible from the FE server (the name also needs to be on the trusted certificate on the exchange server)
        image
      8. image
      9. image
      10. image
      11. image
      12. image
      13. image
      14. image
      15. image
      16. image
      17. This may change based on your scenario
        image
      18. image
      19. Finish
    7. Apply Changes and Test!!!

 

Note: I also like to create a test Rule that I leave disabled unless I need to determine if I have a Firewall rule issue or Network issue.
image

Here is another reference for the same process in a slightly different scenario
http://araihan.wordpress.com/2010/06/17/how-to-configure-back-to-back-firewall-with-perimeter-dmz-topology-step-by-step-guide/

Publish Exchange 2010 with TMG (cont)


Walkthrough on publishing all roles through TMG. (Part 4/4 SMTP)

This assumes:

  1. you have a MX record pointed to a name that points to the external IP of your TMG
  2. You have already configure your exchange to allow anonymous access on the receive connector

Configure SMTP rule on TMG

  1. Open Forefront TMG
  2. Click on  image_thumb51_thumb1[1]
  3. In the Action Pane under Task click image
  4. image
  5. image
  6. Enter your Transport Server internal IP
  7. image
  8. image
  9. image
  10. You should now be able to send mail through your TMG to your Exchange.

Publish Exchange 2010 with TMG (Forefront Threat Management Gateway) Series:

1. OWA
2. EWS\Outlook anywhere
3. Active sync
4. SMTP

Mail Flow Troubleshooting


Mail flow issues should be troubleshot by first determining what direction the issue

  • Internal to Internal
  • Internal to External
  • External to Internal

MX Record – You need to make sure that the MX record for you domain points to a host record that points to your IP

Example(purely theory): If my IP was 65.55.88.22 and my domain is Domain.com then I my MX should look something like this

Domain Pref Hostname IP Address TTL
Domain.com 10 mail.domain.com 254.63.63.63 60

Services

Make sure the services are running (in order 2003, 2007, 2010)

Capture

image

 image

Firewall

Email uses port 25(SMTP) primarily for mailflow, you need to be sure that port 25 is forwarded through your firewall to the exchange server
Also if you have (you can test with MXtoolbox.com or Manual SMTP test (below) to verify that you can get to the mail server.

Accepted domains

In order to receive mail to a domain name you need to have that name listed in the  accepted domains (2007\2010)

image 

Email Address Policy (2007\2010)

In addition to the accepted domains you need your users to have the domain address stamped on the users, to do that you need to add it to the Address Policy

(Left is 2010\2007, Right is 2003)image

image image

image

Next -> Next -> Finish

Active directory lookups

Make sure you can access a Global catalog Domain controller in your site

Also make sure that replication is ok between servers
(Active Directory Sites and Services -> Site Name -> Servers -> ServerName -> NDTS Settings -> Rt Click the connection in the right window and choose “Replicate Now”)

Manual SMTP Testing (Red TXT is typed commands)

Click Start -> run -> CMD (enter)

telnet mail.domain.com 25220 mail.domain.com Microsoft ESMTP MAIL Service ready at Sat, 19 Dec 20
09 15:57:35 -0600
ehlo domain.com
250-mail.domain.com Hello [192.168.3.10]
250-SIZE
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-X-ANONYMOUSTLS
250-AUTH NTLM
250-X-EXPS GSSAPI NTLM
250-8BITMIME
250-BINARYMIME
250-CHUNKING
250-XEXCH50
250 XRDST
mail from: me@domain.com
250 2.1.0 Sender OK
rcpt to: administrator@domain.com250 2.1.5 Recipient OK
data
354 Start mail input; end with <CRLF>.<CRLF>
this is a test
.
250 2.6.0 <f23d11e3-fcac-4033-8450-9f1b5451430c@mail.domain.com> Queued mail for delivery