Quick Exchange 2013 DAG Setup Guide


Background:

Had a co-worker ask for some basic DAG setup instructions in Exchange 2013 so I wrote a quick little guide. This covers the high points around creating the DAG as well as configuring the DAG member NICs & networks.

Step 1 – Pre-Stage DAG Computer Account
Reference. When deploying a DAG on Exchange Servers running Server 2012 you need to pre-stage the DAG computer account. The above link points to the official TechNet article for doing this but here are the basics of it:

  • Create a Computer Account in AD with the name of the DAG. For example, DAG-A.
  • Disable the Computer Account.
  • In Active Directory Users & Computers click View>Advanced Features. Go to the Computer Account & select Properties>Security tab.
  • From here you have two options; either Grant the Exchange Trusted Subsystem Full Control permissions to the DAG Computer Account or give the Computer Account of the first node you plan to join to the DAG Full Control permissions over the DAG Computer Account Object.
  • Reference2

Step 2 – Configure DAG NIC’s
Reference. Exchange 2013 performs automatic DAG network configuration depending on how the NIC’s are configured. This means if the NIC’s are configured correctly then you should not have to manually collapse the DAG Networks post DAG Setup. Upon adding the nodes to the DAG, it looks for the following properties on the NICs & makes a decision based on them:

  • NIC Binding Order
  • Default Gateway Present
  • Register DNS Checked

The DAG needs to separate MAPI/Public networks from Replication networks. This enables the DAG to properly utilize a network that the administrator has provisioned for Replication traffic & to only use the MAPI/Public networks for Replication if the Replication networks are down.

You want your MAPI/Public NICs to be top of the binding order in the OS & any Replication, Management, Backup, or iSCSI networks at the bottom of the binding order. This is a Core Windows Networking best practice as well as what the DAG looks for when trying to determine which NIC’s will be associated with the MAPI/Public DAG Networks.

The DAG also looks for the presence of a Default Gateway on the MAPI/DAG network NIC. Going along with another Windows Networking best practice, you should only have 1 Default Gateway configured in a Windows OS. If you have additional networks with different subnets on the DAG nodes then you would need to add static routes on each of the nodes using NETSH. More on this later.

Finally, NIC Properties>IPv4 Properties>Advanced>DNS>Register this connection’s addresses in DNS should be unchecked on all adapters except for the MAPI/Public NICs. This means all Replication, iSCSI, dedicated backup or management NICs should have this option unchecked. Again, this is a Windows Networking best practice but is vital for proper Automatic DAG Network Configuration in Exchange 2013.

Step 3 – Configure Routing if Needed (optional depending on DAG design)
If your DAG stretches subnets & you’re using dedicated Replication networks then they should be on their own subnet isolated from the MAPI/Public network. A common setup for a network such as this might be:

Site-Austin:
MAPI Network 192.168.1.0/24; Default Gateway 192.168.1.254
Replication Network 10.0.1.0/24; Default Gateway $Null

Site-Houston:
MAPI Network 192.168.2.0/24; Default Gateway 192.168.2.254
Replication Network 10.0.2.0/24; Default Gateway $Null

Now with the above configuration you would have some form of routing taking place between the two MAPI subnets. You would also have routing between the two Replication subnets. However, because you should only have 1 Default Gateway configured per server, DAG nodes in each site would be unable to communicate with each other over the Replication networks. This is where static routes come into play. You would run the following commands on the nodes to allow them to ping across to each other between the 10.0.1.x & 10.0.2.x networks (in the below example, REPL is the name of each node’s Replication NIC):

On Nodes in Site-Austin: “netsh interface ip add route 10.0.2.0/24 “REPL” 0.0.0.0”

On Nodes in Site-Houston: “netsh interface ip add route 10.0.1.0/24 “REPL” 0.0.0.0”

This is the preferred format for this command. There are some references to using the local interface IP instead of 0.0.0.0 but the format I use above is what is recommended by the Windows Networking Team. Reference.

According to our Networking Development Groups, the recommendation actually is that on-link routes should be added with a 0.0.0.0 entry for the next hop, not with the local address (particularly because the local address might be deleted) and with the interface specified.”

This all assumes there is physical routing in place between the two subnets, like a Router, layer 3 Switch, or a shared virtual network in Hyper-V/ESX.

Verify connectivity between nodes over these 10.0.x.x networks using Tracert or Pathping. Note that these steps are only required if your DAG spans subnets & has replication networks in different subnets. While it technically should work, it is not recommended to stretch subnets for DAG Networks across the WAN.

It should also be noted that there should be no routing between the MAPI Networks & the Replication Networks. They should be on isolated networks that have no contact with each other. Also, Microsoft wants no greater than 500ms round trip latency between DAG nodes when you have DAG members across latent network connections. It’s important for customers to realize that you should not set your expectations around this number alone. You could easily have a connection over 500ms & not experience copy queues if you have only 20 mailboxes with low usage profiles. Alternatively, you could have a connection with only 50ms of round-trip latency but see high copy queues if you have thousands of high-usage mailboxes & a small bandwidth pipe. Just know that this number is not an end all be all.

Step 4 – Create DAG & Add Nodes
This part is pretty straightforward & you can use the EAC to do it. Just remember to give the DAG an IP address in every MAPI subnet where you have DAG nodes. So in our scenario above you would give the DAG 2 IP addresses; one in the 192.168.1.0 subnet & another in the 192.168.2.0 subnet.

Step 5 – Manually configure DAG Networks if needed
Reference. If you have dedicated management networks, dedicated backup networks, or iSCSI NIC’s then you would actually have to perform some manual steps after your DAG is setup. These networks should be ignored by the DAG & for cluster use. In order to do this we must first enable Manual DAG Network Configuration, which is disabled by default. We would then need to configure the iSCSI or similar network to be ignored by the cluster. Perform the following steps:

  • Get-DatabaseAvailabilityGroup
  • Set-DatabaseAvailabilityGroup <DAGName> -ManualDagNetworkConfiguration:$True
  • Get-DatabaseAvailabilityGroupNetwork
  • Set-DatabaseAvailabilityGroupNetwork <iSCSI/Backup/Mgmt NetworkName> -IgnoreNetwork:$True

Finally, let’s validate everything. Run the below command:

Get-DatabaseAvailabilityGroupNetwork | Format-List Identity,ReplicationEnabled,IgnoreNetwork

Verify that the iSCSI/Backup/Mgmt networks have IgnoreNetwork set to True (the MAPI & Replication networks should have this set to False). Also verify that the Replication Networks have ReplicationEnabled set to True. Finally, verify that the MAPI network has ReplicationEnabled set to False. This prevents the MAPI network from being used for Replication by default. It can still be used for Replication if all other possible replication paths go down.

References:
http://technet.microsoft.com/en-us/library/ff367878.aspx

http://technet.microsoft.com/en-us/library/dd298065(v=exchg.150).aspx

http://blogs.technet.com/b/scottschnoll/archive/2012/10/01/storage-high-availability-and-site-resilience-in-exchange-server-2013-part-2.aspx

http://blogs.technet.com/b/askcore/archive/2009/05/26/active-route-gets-removed-on-windows-2008-failover-cluster-ip-address-offline.aspx

http://technet.microsoft.com/en-us/library/dd298008(v=exchg.141).aspx

New behavior in Outlook 2013 causing certificate errors in some environments


Background:

I originally discovered this issue back in early Feb & let a couple people on the Exchange Product Team know about it via the TAP but it seems to be affecting more customers than initially thought so I thought I’d share.

In Outlook 2007 through Outlook 2010 all domain-joined Outlook clients would initially query Active Directory for AutoDiscover information & ultimately find a Service Connection Point (SCP) value that would point them to their nearest Client Access Server’s AutoDiscover virtual directory. If that failed then they would revert to using DNS like any non-domain-joined Outlook client. The order of this non-domain-joined lookup is as follows:

https://company.com/autodiscover/autodiscover.xml

https://autodiscover.company.com/autodiscover/autodiscover.xml

Local XML File

http://company.com/autodiscover/autodiscover.xml (looking for a redirect website)

SCP AutoDiscover Record

Why it ever looked to https://company.com/autodiscover/autodiscover.xml I’ll never really know because honestly I’ve never come across a customer who had it deployed that way; most have https://autodiscover.company.com/autodiscover/autodiscover.xml but I imagine when Exchange 2007 was first being developed they weren’t exactly sure how customers would be implementing AutoDiscover.

Issue:

The above methods have served us well since Exchange 2007 timeframe but for some reason the Outlook team decided to try & implement some giddyup into Outlook & try to speed up the process. They decided to have domain-joined Outlook 2013 clients query both the SCP values in AD as well as the DNS records at the same time. If an SCP record was found it would still be used but in the event it failed then it would already have the DNS response ready to go. Great idea, however there’s one problem in the implementation.

If Outlook 2013 encounters any kind of Certificate error while doing the simultaneous DNS query then you will receive a pop-up in Outlook about the cert.

I actually stumbled upon this while in the middle of the scenario below:

error

That’s right, I actually get a certificate pop-up for my lab’s domain name (ash15.com) & not autodiscover.ash15.com like one would expect if I were to have a certificate issue on Exchange.

When Outlook 2013 does it’s simultaneous DNS AutoDiscover query the first URL it tries is https://company.com/autodiscover/autodiscover.xml, which in my lab environment resolved to my Domain Controller, which was also serving DNS, as well as a Certificate Authority. Ash15.com resolved to this server because it’s my internal Active Directory domain name & the name server entry resolves to my DC (just ping internaldomainname.local in your AD lab environment & you’ll see the same thing).

Now because I have web enrollment enabled & am listening on 443 in IIS the server responded. Also, because I did not have a cert installed on the server with ash15.com in the Subject or Subject Alternative Name then it gave the certificate error we see above.

Resolution:

The error is easy enough to get through & it only occurred on initial profile creation but this can definitely prove painful for some customers. Obviously my lab environment is a corner case but there have been several other customers report this issue with Outlook 2013 as well.

Here’s an example scenario.

Imagine you have a public website for andrewswidgets.com hosted by a third-party hosting site & you did not pay for HTTPS/443 services. However if you were to query the website using https then it could respond & obviously not return a certificate with andrewswidgets.com on it (because you haven’t paid for it you cheapskate…). Now imagine you begin deploying users using Outlook 2013 in your internal environment. In the past, they would have found the SCP record that would have pointed them to your internal Exchange 07/10/13 server for AutoDiscover & would have been happy as a clam (one Exchange Product Manager’s favorite way to describe Exchange bliss). However, now they may get a certificate pop-up for andrewswidgets.com when creating a new profile.

There are a couple ways around this. Make sure andrewswidgets.com doesn’t listen on 443, or possibly get a proper cert on your website that is listening on 443. Simply put, just make sure whatever andrewswidgets.com resolves to is something that’s not going to throw a certificate error.

I’ve heard nothing concrete or public but the Outlook team is aware of the issue & listening to customer feedback. I suggest contacting Microsoft Support if your organization is running into this issue.

 

Also, this KB offers methods to control which AutoDiscover methods are used by your Outlook clients

 

Beware Full OAB Downloads After Installing 1st Exchange 2013 Server in Existing 07/10 Environment


Members of the Exchange 2013 Technology Adoption Program (TAP) have known about this issue for a while & the general public had the potential to figure it out once Exchange 2010 SP3 came out last month which allowed co-existence with 2013 in a lab environment; now the Exchange Team has been very clear about it with this recent blog post today. Actually, a Microsoft Support-led session at MEC was when I first heard about it in detail. So what’s the issue? Basically, you have the potential to experience an organization-wide full Offline Address Book download just as a result of installing the first Exchange 2013 server into your existing Exchange environment.

Background:

The Offline Address Book is used by Outlook Cached mode clients to be able to have offline access to Address Lists as well as some Group Metric data when they aren’t connected to the Exchange Server. For a very detailed explanation from Neil Hobson see the following article from him.

Issues can occur when the OAB for an organization grows to a large size, sometimes in the hundreds of MB. Things that contribute to this size are things like number of recipients in AD, number of distribution groups, populated user attributes, & certificate usage (reference ). It’s important to note that GAL Photos are NOT stored in the OAB. The OAB just includes a pointer to AD where the photo is actually stored (reference ). Fortunately, Exchange/Outlook is smart enough to only download the changes to the OAB instead of the entire thing every day. There are still some circumstances where the entire OAB will be downloaded again, which makes it very important to understand the size of your OAB so you know just how much of your networks bandwidth will be used when all Outlook Cached Mode clients perform a full download (reference ).

So as you might imagine, whether or not clients will perform a full OAB download becomes a topic of concern during an Exchange migration.

Offline Address Books in Exchange 2003/2007/2010 are associated with Mailbox Databases, specifically on the Properties>Client Settings Tab of the Mailbox Database in the associated Management Console:

OAB1

However, in the screenshot above you’ll notice that the “Offline Address Book” field is blank on this Mailbox Database. This is the case by default with all Mailbox Databases. This is not an issue because any Mailbox Database that has its “OfflineAddressBook” attribute set to $Null by default will use the Default Offline Address Book in the Exchange Organization. This OAB can be seen below:

OAB2

This means if you have configured multiple Offline Address Books in your environment then you would need to manually specify the additional OAB’s on the Databases you would want to use them; otherwise the default OAB would be used. Simply put, if the value for OAB is blank on a Mailbox Database then it will use the Default OAB. If it is hard-set then it will use whichever OAB you hard set it to. Some customers will hard set this value if they want Mailboxes on specific Mailbox Databases to use a specific OAB. Maybe an OAB that only contains a specific Address List instead of the entire GAL like the example below:

OAB3

Many organizations just have 1 OAB & as a result have never populated the Properties>Client Settings>Offline Address Book value of their Mailbox Databases. This is where a big issue can come into play during an Exchange 2013 migration, or even if you just want a single Exchange 2013 server in your environment for a test group of users.

Issue:

As the recent Exchange Team Blog post announcing 2013 CU1 states, you need to make sure all of your Exchange 2007/2010 Mailbox Databases have an actual value populated for their Offline Address Book. If they are currently blank, then populate them with your current Default OAB. Nothing will change in the environment as a result of this because they will continue to use their current OAB & continue to only download the OAB changes.

Failure to do this will result in each of these Mailbox Databases switching to use the Exchange 2013 Offline Address Book that gets created during installation of your first Exchange 2013 Mailbox Server. This will result in a Full OAB Download for all of your Outlook Cached Mode clients on these Mailbox Databases; a potentially nasty situation which could bring your network to its knees.

You can see the Exchange Team Blog post in question for steps on how to repoint these databases or just use the following commands which I have taken from the post:

“Get-MailboxDatabase | Where {$_.OfflineAddressBook -eq $Null} | FT Name,OfflineAddressBook –AutoSize”

This lists all Mailbox Databases in your environment with their OfflineAddressBook attribute set to $null.

Then run:

“Get-MailboxDatabase | Where {$_.OfflineAddressBook -eq $Null} | Set-MailboxDatabase -OfflineAddressBook (Get-OfflineAddressBook | Where {$_.IsDefault -eq $True})”

This command will grab each of these Mailbox Databases & populate the OfflineAddressBook attribute with the value of your Organizations current Default OAB. Effectively changing nothing in terms of client behavior but ensuring that when you install Exchange 2013, each of these Mailbox Databases do not switch over to using the 2013 OAB; at least not until you are ready & can stage this process, maybe one MDB at a time.

Summary:

These steps should be mandatory for any organization considering implementing an Exchange 2013 Server into their existing Exchange 2007/2010 environment.

Checking for Open Relay in Exchange 2007/2010


Scenario:

So this is a fairly common scenario & I figured I’d post an easy method to diagnose the issue. Customers will often suspect that they’re an open relay due to being placed on a blacklist or having issues sending email to certain domains. There’s some general confusion as to what constitutes as an Open Relay & even the difference between a Relay & a Submit action in SMTP terminology. Hopefully this can clear some of the confusion.

Background:

Submit = Submitting an email message to an SMTP server that is destined for a domain that exists on that server (or in that server’s environment). You’re sending it to an address that the server is authoritative for.

Relay = Submitting an email message to an SMTP server that is destined for a domain that exists in another messaging environment. You’re sending to an address that the server is not authoritative for.

So there’s nothing inherently wrong with relaying. It’s what happens if you use your Hotmail account to send an email to someone’s Gmail account. It happens every time you email someone outside of your own messaging system. The key detail is whether or not you have authenticated to the SMTP server beforehand. So when you’re using Hotmail or Exchange via Outlook/OWA then you have obviously authenticated either via an Authentication Prompt, OWA Form, or using NTLM.

So this typically comes up when a customer needs to have an application, network printer, or other device be able to send emails through Exchange (or any internal SMTP server).

So the important thing to point out here is that as long as the application/device only needs to be able to send to addresses that your SMTP server is authoritative for then it is a Submit action & not a Relay action. This just means you only need it to be able to hit a Receive Connector that allows Anonymous Submit; which is how most of the world’s SMTP servers are configured to accept email from the Internet.

However, if your application/device needs to be able to send to an address not under the authority of the local SMTP server then it will be performing an SMTP Relay action & will require additional configuration.

The recommended approach is to have the Application/Device authenticate to your SMTP server if it supports it. Alternatively, you can configure the Receive Connector (Exchange) to allow Anonymous Relaying from that Application/Device’s IP address.

For instructions please see this Microsoft Post.

http://blogs.technet.com/b/exchange/archive/2006/12/28/3397620.aspx

This is a very common issue amongst customers because they may not be familiar with how to configure this. However, unfortunately I will often see customers make an even worse mistake; allowing Anonymous Relaying from an entire range of IP Addresses or possibly the entire Internet. It won’t take long for Internet folks with malicious intent to figure this out & start using your server to SPAM whoever they wish. This typically results in your Exchange Server’s sending IP being placed on various Blacklists which can prevent you from sending to certain email domains.

Resolution:

It is ALWAYS recommended to create a separate Receive Connector for this purpose. In fact I tell customers to never mess with the Default Receive Connectors if they can get away with it. But what will ultimately happen is the customer will use the steps in the Microsoft article above to enable Anonymous Relaying on their Default Receive Connector instead, which they’re probably also using as their Internet ingress point. The problem with this is that the Remote IP range of that connector is 0.0.0.0-255.255.255.255 out of the box; meaning the entire Internet.

Another thing the customer might do is create a new Receive Connector for Relaying but instead of just having 1 IP address in there (the IP of their Application Server or Network Device) they’ll add an entire range or more IPs than are needed. This can get pretty complicated to troubleshoot if you have many different Receive Connectors on many different Exchange Servers in the environment.

So I’m hoping people can use my explanation to help them configure this properly as well as troubleshoot any issues they may have. In addition to that, here’s a very useful command to use in Exchange Management Shell to list out all Receive Connectors in the environment that have the Anonymous Relay permission enabled. Use this to track these connectors down & then verify the RemoteIP Ranges are properly scoped/configured to be as secure as possible.

Get-ReceiveConnector | Get-ADPermission -User “NT Authority\Anonymous Logon” | Where-Object {$_.ExtendedRights -like “ms-Exch-SMTP-Accept-Any-Recipient”} | Format-List Identity,ExtendedRights

Quick method to diagnose Exchange Active Directory Access & Service Startup Issues


Background:

My colleague Jedidiah Hammond wrote a great post awhile back on troubleshooting Exchange Service start-up issues. One of the main areas of focus of the post were issues with Active Directory Global Catalog servers. This can be considered an ad-on to that post as I’ll describe a useful method to troubleshoot Exchange permissions in Active Directory; more specifically, verifying Exchange has the proper access to the Global Catalog servers in and out of it’s respective Active Directory site.

Scenario:

Suppose you find that the Microsoft Exchange Active Directory Topology Service isn’t starting; or the System Attendant, or the Information Store service. Or perhaps the Exchange Management Console or Exchange Management Shell will not connect and is complaining of Active Directory/Global Catalog issues.
Often times this is a result of a port being blocked by Anti-V/Firewall between the Exchange Server and your Global Catalog. Or possibly a configuration issue on the network stack (IP/DNS/etc); maybe someone even powered your GC off much to your dismay. Assuming you have already worked through the above scenarios, one useful tool to verify Exchange/AD functionality is actually a very commonly used one; Event Viewer.

When you first deploy Exchange and run “setup /PrepareAD” (or you let the GUI setup do it for you) it is actually setting many of these permissions in AD. (For a list of all of these changes see this Technet article).

Steps:

Below is an excerpt from MSExchange ADAccess Informational Event ID 2080. You’ll find it occurring roughly every 15min on your Exchange Servers.
Description:
Process STORE.EXE (PID=3376). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
Austin.ASH.ORG    CDG 1 7 7 1 0 1 1 7 1
 Out-of-site:
Houston.ASH.ORG    CDG 1 7 7 1 0 1 1 7 1

This is an example of what the output should look like. You might be asking what those series of numbers represent. Well buried deep within the land of Exchange 2000 there lies a KB article explaining just that.

After reading the article you’ll find that these numbers are basically describing Exchange’s understanding of the Global Catalog servers made available to it; along with whether or not it has the proper ACLs set to be able to utilize them. If you find yourself pulling your hair out as to why Exchange is showing the symptoms I listed earlier, then look for this event on your Exchange server and you just might see something like the following:

Description:
Process STORE.EXE (PID=3376). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
Austin.ASH.ORG    CDG 1 7 7 1 0 0 1 7 1
 Out-of-site:
Houston.ASH.ORG    CDG 1 7 7 1 0 0 1 7 1

Notice it ends with “0171″ instead of “1171″. If we reference the above KB article then this tells us Exchange lacks the proper ACL’s in AD.

I’ve seen this many times with customers who have modified the Default Domain Controllers Group Policy or somehow blocked it’s use. I’ve also seen similar issues arise from unchecking “Include Inheritable Permissions from this Object’s Parent” in AD for various objects. If this is the case then please see the post I referenced earlier on how to resolve that. In addition, I’ve found re-running “setup.com /PrepareAD” to be a very useful troubleshooting step in situations such as these where you feel AD permissions may be at fault. Some customers have been weary of running this but honestly their fears stem from ignorance because “it just sounds scary” ; a quick read over the article I referenced earlier will tell you that running it again will only re-add the permissions Exchange has needed all along.
However, be aware that re-running PrepareAD may only resolve the issue temporarily as any bad Group Policies may find themselves being re-applied in about 15min so fixing the actual source of the issue should be the ultimate goal.

An additional note here is if you’re utilizing AD Split permissions with Exchange, there may be additional precautions to be taken before running PrepareAD  again.

Exchange 2010 Powershell Lab Tools


Recently a good friend of mine pointed out some great tools that Mike Pfeiffer posted and I have to say they are pretty awesome and I would recommend checking them out.

Populating Exchange Labs with Mailboxes using PowerShell

Provision Exchange Mailboxes from CSV using PowerShell Advanced Functions

Generating Test Email Data for Exchange Labs with PowerShell

External Connections to Internal Web Servers Over 443 Fail


Probably one of the strangest issues I’ve seen, at least it seemed that way at the time.

Scenario:

2 internal web servers experiencing the symptom, one running 2010 OWA and the other a custom web application on 443. All internal users can hit each page just fine. External users cannot hit the pages and they just receive a timeout. However, if the admin logs into either of the two servers locally or via RDP and then you try again externally, it works and they can hit the web pages. This behavior only happened on 443. Customer was just using a Cisco ASA for their firewall with no web publishing.

Resolution:

Customer was a school district and I was reminded of a former life where I worked for a school district where web filtering was common. We found out that external users could only hit the pages when an Admin was logged into either of the servers; not a regular user. Combined with the below Cisco thread I found when trying to potentially pin this on the ASA it seemed a web filter or Intrusion Detection System was killing our connections.

https://supportforums.cisco.com/thread/2043090

According to the thread a Filter/IDS on the inside could potentially be issuing resets for web traffic that it did not like. In our case it was the customers “iBoss” content filter that started blocking access after a firmware update. It worked when an Administrator was logged into the web servers because it could filter based on the currently logged on AD account and there were exclusions for the Admins.

Doing a Disaster Recovery on a Exchange Server that is also a DC


email_exchange_iconHave you every worked on a failed exchange server that also happens to be a DC (not recommended, but it happens)

Well if you do and you find yourself trying to recover it here is how you can.

  1. Note critical information
    1. What are the drive letters
    2. Where is the logs and database located
    3. What is the service pack level
  2. Remove data from server
  3. Format and re-install the OS – using the same drive letters
  4. Seize Roles if they were on the failed server
  5. Run through a metadata cleanup to remove the failed server from AD
  6. Replicate changes to all DCs
  7. Join rebuilt server to the domain  - Using the Same name
  8. Add the Server object to the correct exchange groups
    1. Exchange 2007 – “Exchange Servers”, “Exchange Install Domain Servers”
    2. Exchange 2010 - “Exchange Servers”, “Exchange Install Domain Servers”, “Exchange Trusted Subsystem”
    3. Exchange 2003 – “Exchange Domain Servers”
  9. Windows Update the Server
  10. Do a disaster recovery install of exchange
    1. Exchange 2003 = setup /disasterrecovery
    2. Exchange 2007\2010 = Setup.com /m:recoverserver
  11. Restore data using backup application or recovered databases from failure
  12. and away you go!