Incorrectly Adding New Receive Connector Breaks Exchange 2013 Transport


I feel the concepts surrounding this issue have been mentioned already via other sources (1 2) but I’ve seen at least 5 recent cases where our customers were being adversely impacted by this issue; so it’s worth describing in detail.

Summary:

After creating new Receive Connectors on Multi-Role Exchange 2013 Servers, customers may encounter mail flow/transport issues within a few hours/days. Symptoms such as:

  • Sporadic inability to connect to the server over port 25
  • Mail stuck in the Transport Queue both on the 2013 servers in question but also on other SMTP servers trying to send to/through it
  • NDR’s being generated due to delayed or failed messages

This happens because the Receive Connector was incorrectly created (which is very easy to do), resulting in two services both trying to listen on port 25 (the Microsoft Exchange FrontEnd Transport Service & the Microsoft Exchange Transport Service). The resolution to this issue is to ensure that you specify the proper “TransportRole” value when creating the Receive Connector either via EAC or Shell. You can also edit the Receive Connector after the fact using Set-ReceiveConnector.

Detailed Description:

Historically, Exchange Servers listen on & send via port 25 for SMTP traffic as it’s the industry standard. However, you can listen/send on any port you choose as long as the parties on each end of the transmission agree upon it.

Exchange 2013 brought a new Transport Architecture & without going into a deep dive, the Client Access Server (CAS) role runs the Microsoft Exchange FrontEnd Transport Service which listens/sends on port 25 for SMTP traffic. The Mailbox Server role has the Microsoft Exchange Transport Service which is similar to the Transport Service in previous versions of Exchange & also listens on port 25. There are two other Transport Services (MSExchange Mailbox Delivery & Mailbox Submission) but they aren’t relevant to this discussion.

So what happens when both of these services reside on the same server (like when deploying Multi-Role; which is my recommendation)? In this scenario, the Microsoft Exchange FrontEnd Transport Service listens on port 25, since it is meant to handle inbound/outbound connections with public SMTP servers (which expect to use port 25). Meanwhile, the Microsoft Exchange Transport Service listens on port 2525. Because this service is used for intra-org communications, all other Exchange 2013 servers in the Organization know to send using 2525 (however, 07/10 servers still use port 25 to send to multi-role 2013 servers, which is why Exchange Server Authentication is enabled by default on your default FrontEndTransport Receive Connectors on a Multi-Role box; in case you were wondering).

So when you create a new Receive Connector on a Multi-Role Server, how do you specify which service will handle it? You do so by using the -TransportRole switch via the Shell or by selecting either “Hub Transport” or “FrontEnd Transport” under “Role” when creating the Receive Connector in the EAC.

The problem is there’s nothing keeping you from creating a Receive Connector of Role “Hub Transport” (which it defaults to) that listens on port 25 on a Multi-Role box. What you then have is two different services trying to listen on port 25. This actually works temporarily, due to some .NET magic that I’m not savvy enough to understand, but regardless, eventually it will cause issues. Let’s go through a demo.

Demo:

Here’s the output of Netstat on a 2013 Multi-Role box with default settings. You’ll see MSExchangeFrontEndTransport.exe is listening on port 25 & EdgeTransport.exe is listening on 2525. These processes correspond to the Microsoft Exchange FrontEnd Transport & Microsoft Exchange Transport Services respectively.

1new

Now let’s create a custom Receive Connector, as if we needed it to allow a network device to Anonymously Relay through Exchange (the most common scenario where I’ve seen this issue arise). Notice in the first screenshot, you’ll see the option to specify which Role should handle this Receive Connector. Also notice how Hub Transport is selected by default, as is port 25.

3

4

5

After adding this Receive Connector, see how the output of Netstat differs. We now have two different processes listening on the same port (25).

6

So there’s a simple fix to this. Just use Shell (there’s no GUI option to edit the setting after it’s been created) to modify the existing Receive Connector to be handled by the MSExchange FrontEndTransport Service instead of the MSExchange Transport Service. Use the following command:

Set-ReceiveConnector Test-Relay –TransportRole FrontEndTransport

7

I recommend you restart both Transport Services afterwards.

 

 

Update: In recent releases of Exchange 2013 (unsure which CU this fix was implemented in), the EAC will no longer let you mis configure a receive connector in this way. So hopefully we should see less of this issue.

 

26 thoughts on “Incorrectly Adding New Receive Connector Breaks Exchange 2013 Transport

  1. Pingback: NeWay Technologies – Weekly Newsletter #79 – January 23, 2014 | NeWay

  2. Pingback: NeWay Technologies – Weekly Newsletter #79 – January 24, 2014 | NeWay

  3. Pingback: interesting things i see on the internet – 27/01/2014 | 503 5.0.0 polite people say HELO

  4. Pingback: Incorrectly Adding New Receive Connector Breaks Exchange 2013 Transport | Troubleshooting Exchange | JC's Blog-O-Gibberish

  5. Pingback: Smtp25 » Exchange 2013 Transport. Connectors.

  6. Pingback: NeWay Technologies – Weekly Newsletter #80 – January 31, 2014 | NeWay

  7. Thank you so much for this post!! This is exactly the issue that I was experiencing with a new Exchange 2013 install and this resolution came in handy after suffering for hours. I was having to restart the transport service every few minutes and eventually figured out that an internal receive conector I had created for scanners was causing an issue. After disabling the internal receive connector I had created for the scanners, external connections on port 25 would eventually fail anyway. Making this change so that the connector for the scanners of type “FrontendTransport” stabilized email flow.

  8. Pingback: Exchange 2013 SP1 Breaks Hub Transport service | Troubleshooting Exchange

  9. Pingback: Exchange 2013 Upgrade Fails Due to Connector Conflict

  10. Pingback: Upgrade to CU8 Fails on Receive Connector misconfiguration | Jaap Wesselius

  11. Migrating from Exchange 2007 to 2013, I’m not too familiar with the receive connector configuration! When using a “Mail Relay” connector for internal mail alerts, the emails always go into my Outlook’s Junk folder, which it doesn’t routing through E2k7! The relay connectors are setup the same and all normal internal mail works fine. Any advice?

  12. I appear to have the symptoms of this problem, but I am not running a custom receive connector. Yet, every 8 seconds I get error ID messages 1036, 1019, and 7031. The description in error ID 1019 says “Failed to start listening (Error: 10048). Binding: 0.0.0.0:25.” Netsat -ano | find 25 results in just one line: TCP 0.0.0.0:25 0.0.0.0 LISTENING 1916. Any ideas as to what this issue is here?

    • I would try to use the PID column in task manager to try and see which process that corresponds to. If that’s not the issue, I’d recommend using a working 2013 lab server to compare the settings of all connectors with your own. If it’s not a port conflict then it may be a misconfig. Or possibly anti-v.

  13. Hi Andrew,
    we have 2 CAS & 2 Mailbox servers. The default frontend and the connector used for internal relay(let’s say “interconn1”) & both use port 25. The interconn1 uses frontend transport service. The interconn1 has everything check-boxed except “partners”. I’ve allowed only some internal IPs in scoping for interconn1. There’s spam firewall infront of CAS. The problem is that external users can send email by “myserverpublicIP:25” via telnet, with the use of internal doamin user name in “mail from:” field.

  14. Hi Andrew,

    We have 2 CAS & 2 mailbox servers. CAS1 has 2 receive connectors both using port 25. The first one is “default frontend” with anonymous option. The second one is for internal relay with anonymous option, all check-box checked except partners in permission group. And this connector is created with frontend transport service allowing only some internal IPs. The problem is external users can use without password if they use the domain account in the “mail from:” field via telnet.

    • This is by design. Anyone can anonymously submit using the internet receive connector. That’s how internet SMTP works. Unless I’m misunderstanding your scenario.

  15. Pingback: My most commonly used blog posts for troubleshooting Exchange | A bit of Exchange & Office 365

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s