Quick Exchange 2013 DAG Setup Guide

Background:

Had a co-worker ask for some basic DAG setup instructions in Exchange 2013 so I wrote a quick little guide. This covers the high points around creating the DAG as well as configuring the DAG member NICs & networks.

Step 1 – Pre-Stage DAG Computer Account
Reference. When deploying a DAG on Exchange Servers running Server 2012 you need to pre-stage the DAG computer account. The above link points to the official TechNet article for doing this but here are the basics of it:

• Create a Computer Account in AD with the name of the DAG. For example, DAG-A.
• Disable the Computer Account.
• In Active Directory Users & Computers click View>Advanced Features. Go to the Computer Account & select Properties>Security tab.
• From here you have two options; either Grant the Exchange Trusted Subsystem Full Control permissions to the DAG Computer Account or give the Computer Account of the first node you plan to join to the DAG Full Control permissions over the DAG Computer Account Object.
• Reference2

Step 2 – Configure DAG NIC’s
Reference. Exchange 2013 performs automatic DAG network configuration depending on how the NIC’s are configured. This means if the NIC’s are configured correctly then you should not have to manually collapse the DAG Networks post DAG Setup. Upon adding the nodes to the DAG, it looks for the following properties on the NICs & makes a decision based on them:

• NIC Binding Order
• Default Gateway Present
• Register DNS Checked

The DAG needs to separate MAPI/Public networks from Replication networks. This enables the DAG to properly utilize a network that the administrator has provisioned for Replication traffic & to only use the MAPI/Public networks for Replication if the Replication networks are down.

You want your MAPI/Public NICs to be top of the binding order in the OS & any Replication, Management, Backup, or iSCSI networks at the bottom of the binding order. This is a Core Windows Networking best practice as well as what the DAG looks for when trying to determine which NIC’s will be associated with the MAPI/Public DAG Networks.

The DAG also looks for the presence of a Default Gateway on the MAPI/DAG network NIC. Going along with another Windows Networking best practice, you should only have 1 Default Gateway configured in a Windows OS. If you have additional networks with different subnets on the DAG nodes then you would need to add static routes on each of the nodes using NETSH. More on this later.

Finally, NIC Properties>IPv4 Properties>Advanced>DNS>Register this connection’s addresses in DNS should be unchecked on all adapters except for the MAPI/Public NICs. This means all Replication, iSCSI, dedicated backup or management NICs should have this option unchecked. Again, this is a Windows Networking best practice but is vital for proper Automatic DAG Network Configuration in Exchange 2013.

Step 3 – Configure Routing if Needed (optional depending on DAG design)
If your DAG stretches subnets & you’re using dedicated Replication networks then they should be on their own subnet isolated from the MAPI/Public network. A common setup for a network such as this might be:

Site-Austin:
MAPI Network 192.168.1.0/24; Default Gateway 192.168.1.254
Replication Network 10.0.1.0/24; Default Gateway $Null

Site-Houston:
MAPI Network 192.168.2.0/24; Default Gateway 192.168.2.254
Replication Network 10.0.2.0/24; Default Gateway $Null

Now with the above configuration you would have some form of routing taking place between the two MAPI subnets. You would also have routing between the two Replication subnets. However, because you should only have 1 Default Gateway configured per server, DAG nodes in each site would be unable to communicate with each other over the Replication networks. This is where static routes come into play. You would run the following commands on the nodes to allow them to ping across to each other between the 10.0.1.x & 10.0.2.x networks (in the below example, REPL is the name of each node’s Replication NIC):

On Nodes in Site-Austin: “netsh interface ip add route 10.0.2.0/24 “REPL” 0.0.0.0”

On Nodes in Site-Houston: “netsh interface ip add route 10.0.1.0/24 “REPL” 0.0.0.0”

This is the preferred format for this command. There are some references to using the local interface IP instead of 0.0.0.0 but the format I use above is what is recommended by the Windows Networking Team. Reference.

“According to our Networking Development Groups, the recommendation actually is that on-link routes should be added with a 0.0.0.0 entry for the next hop, not with the local address (particularly because the local address might be deleted) and with the interface specified.”

This all assumes there is physical routing in place between the two subnets, like a Router, layer 3 Switch, or a shared virtual network in Hyper-V/ESX.

Verify connectivity between nodes over these 10.0.x.x networks using Tracert or Pathping. Note that these steps are only required if your DAG spans subnets & has replication networks in different subnets. While it technically should work, it is not recommended to stretch subnets for DAG Networks across the WAN.

It should also be noted that there should be no routing between the MAPI Networks & the Replication Networks. They should be on isolated networks that have no contact with each other. Also, Microsoft wants no greater than 500ms round trip latency between DAG nodes when you have DAG members across latent network connections. It’s important for customers to realize that you should not set your expectations around this number alone. You could easily have a connection over 500ms & not experience copy queues if you have only 20 mailboxes with low usage profiles. Alternatively, you could have a connection with only 50ms of round-trip latency but see high copy queues if you have thousands of high-usage mailboxes & a small bandwidth pipe. Just know that this number is not an end all be all.

Step 4 – Create DAG & Add Nodes
This part is pretty straightforward & you can use the EAC to do it. Just remember to give the DAG an IP address in every MAPI subnet where you have DAG nodes. So in our scenario above you would give the DAG 2 IP addresses; one in the 192.168.1.0 subnet & another in the 192.168.2.0 subnet.

Step 5 – Manually configure DAG Networks if needed
Reference. If you have dedicated management networks, dedicated backup networks, or iSCSI NIC’s then you would actually have to perform some manual steps after your DAG is setup. These networks should be ignored by the DAG & for cluster use. In order to do this we must first enable Manual DAG Network Configuration, which is disabled by default. We would then need to configure the iSCSI or similar network to be ignored by the cluster. Perform the following steps:

• Get-DatabaseAvailabilityGroup
• Set-DatabaseAvailabilityGroup <DAGName> -ManualDagNetworkConfiguration:$True
• Get-DatabaseAvailabilityGroupNetwork
• Set-DatabaseAvailabilityGroupNetwork <iSCSI/Backup/Mgmt NetworkName> -IgnoreNetwork:$True

Finally, let’s validate everything. Run the below command:

Get-DatabaseAvailabilityGroupNetwork | Format-List Identity,ReplicationEnabled,IgnoreNetwork

Verify that the iSCSI/Backup/Mgmt networks have IgnoreNetwork set to True (the MAPI & Replication networks should have this set to False). Also verify that the Replication Networks have ReplicationEnabled set to True. Finally, verify that the MAPI network has ReplicationEnabled set to False. This prevents the MAPI network from being used for Replication by default. It can still be used for Replication if all other possible replication paths go down.

References:

http://technet.microsoft.com/en-us/library/ff367878.aspx

http://technet.microsoft.com/en-us/library/dd298065(v=exchg.150).aspx

http://blogs.technet.com/b/scottschnoll/archive/2012/10/01/storage-high-availability-and-site-resilience-in-exchange-server-2013-part-2.aspx

http://blogs.technet.com/b/askcore/archive/2009/05/26/active-route-gets-removed-on-windows-2008-failover-cluster-ip-address-offline.aspx

http://technet.microsoft.com/en-us/library/dd298008(v=exchg.141).aspx

New behavior in Outlook 2013 causing certificate errors in some environments

Background:

I originally discovered this issue back in early Feb & let a couple people on the Exchange Product Team know about it via the TAP but it seems to be affecting more customers than initially thought so I thought I’d share.

In Outlook 2007 through Outlook 2010 all domain-joined Outlook clients would initially query Active Directory for AutoDiscover information & ultimately find a Service Connection Point (SCP) value that would point them to their nearest Client Access Server’s AutoDiscover virtual directory. If that failed then they would revert to using DNS like any non-domain-joined Outlook client. The order of this non-domain-joined lookup is as follows:

https://company.com/autodiscover/autodiscover.xml

https://autodiscover.company.com/autodiscover/autodiscover.xml

Local XML File

http://company.com/autodiscover/autodiscover.xml
(looking for a redirect website)

SCP AutoDiscover Record

Why it ever looked to
https://company.com/autodiscover/autodiscover.xml
I’ll never really know because honestly I’ve never come across a customer who had it deployed that way; most have
https://autodiscover.company.com/autodiscover/autodiscover.xml
but I imagine when Exchange 2007 was first being developed they weren’t exactly sure how customers would be implementing AutoDiscover.

Issue:

The above methods have served us well since Exchange 2007 timeframe but for some reason the Outlook team decided to try & implement some giddyup into Outlook & try to speed up the process. They decided to have domain-joined Outlook 2013 clients query both the SCP values in AD as well as the DNS records at the same time. If an SCP record was found it would still be used but in the event it failed then it would already have the DNS response ready to go. Great idea, however there’s one problem in the implementation.

If Outlook 2013 encounters any kind of Certificate error while doing the simultaneous DNS query then you will receive a pop-up in Outlook about the cert.

I actually stumbled upon this while in the middle of the scenario below:

That’s right, I actually get a certificate pop-up for my lab’s domain name (ash15.com) & not autodiscover.ash15.com like one would expect if I were to have a certificate issue on Exchange.

When Outlook 2013 does it’s simultaneous DNS AutoDiscover query the first URL it tries is
https://company.com/autodiscover/autodiscover.xml
, which in my lab environment resolved to my Domain Controller, which was also serving DNS, as well as a Certificate Authority. Ash15.com resolved to this server because it’s my internal Active Directory domain name & the name server entry resolves to my DC (just ping internaldomainname.local in your AD lab environment & you’ll see the same thing).

Now because I have web enrollment enabled & am listening on 443 in IIS the server responded. Also, because I did not have a cert installed on the server with ash15.com in the Subject or Subject Alternative Name then it gave the certificate error we see above.

Resolution:

The error is easy enough to get through & it only occurred on initial profile creation but this can definitely prove painful for some customers. Obviously my lab environment is a corner case but there have been several other customers report this issue with Outlook 2013 as well.

Here’s an example scenario.

Imagine you have a public website for andrewswidgets.com hosted by a third-party hosting site & you did not pay for HTTPS/443 services. However if you were to query the website using https then it could respond & obviously not return a certificate with andrewswidgets.com on it (because you haven’t paid for it you cheapskate…). Now imagine you begin deploying users using Outlook 2013 in your internal environment. In the past, they would have found the SCP record that would have pointed them to your internal Exchange 07/10/13 server for AutoDiscover & would have been happy as a clam (one Exchange Product Manager’s favorite way to describe Exchange bliss). However, now they may get a certificate pop-up for andrewswidgets.com when creating a new profile.

There are a couple ways around this. Make sure andrewswidgets.com doesn’t listen on 443, or possibly get a proper cert on your website that is listening on 443. Simply put, just make sure whatever andrewswidgets.com resolves to is something that’s not going to throw a certificate error.

I’ve heard nothing concrete or public but the Outlook team is aware of the issue & listening to customer feedback. I suggest contacting Microsoft Support if your organization is running into this issue.

 

Also, this KB offers methods to control which AutoDiscover methods are used by your Outlook clients

 

Disabling Outlook Anywhere & Avoiding Unnecessary Authentication Prompts for Certain Mailboxes

So this is a complicated scenario but only because this particular customer made it that way; in fact the solution ended up being very simple.

Scenario:

One of my Consultant co-workers pinged me on an issue he was sorting through at a customer site. They were using UAG for their Outlook Anywhere endpoint, both internally & externally. They had a policy to only allow Outlook Anywhere for roughly 30% of their user base. They were enforcing this using AD group membership in UAG to block access to the Outlook Anywhere rule for all users except for those on the allowed list.

Not only was this a nightmare to manage but it also caused Outlook Authentication prompts in certain scenarios. I’ll explain:

When internal Outlook users moved between wired & wireless networks (or vice versa), Outlook would be disconnected just long enough for it to attempt an Outlook Anywhere connection over HTTPS (since the RPC/MAPI connection didn’t reconnect quite fast enough for Outlook’s liking). Well since they were using NTLM for Outlook Anywhere this didn’t really pose a problem for the users who had been allowed to use the OA rule in UAG. However, the users who had been blocked (the majority of their users) would get Outlook auth prompts.

This raised another question from the Consultant & the client; why does enabling Outlook Anywhere on your Client Access Server result in all Outlook clients being enabled for Outlook Anywhere? Shouldn’t there be a method to disable it by default & only enable it via AutoDiscover in Outlook on the mailboxes we choose? Well I’m not Microsoft so I couldn’t answer that but what I was able to do was give them a much better solution going forward which wouldn’t require the hassle of managing group membership for the UAG rule.

Background:

When you enable Outlook Anywhere on your Client Access Server (Exchange 2007/2010), AutoDiscover will then start handing out information to all Outlook Clients on how to connect via OA if a direct RPC/MAPI/TCPIP isn’t available. This allows external Outlook clients to connect to their Mailbox without the use of a VPN.

Exchange AutoDiscover hands these out using what’s called Outlook Providers. These allow Administrators & Exchange itself to differentiate between the various settings used with Outlook Anywhere VS direct RPC/MAPI/TCPIP connections.

The EXCH Outlook Provider is used to hand out settings used when connecting via RPC/MAPI/TCPIP while the EXPR Outlook Provider is used to hand out settings when connecting via Outlook anywhere (RPC over HTTPS). You can view the settings of each by running Get-OutlookProvider | Format-List.

This is the response received using the Test E-mail AutoConfiguration utility in Outlook for a mailbox after Outlook Anywhere has been enabled in the environment. This image shows the EXCH settings.

This image shows the EXPR settings received in the same AutoDiscover response. These are the settings Outlook will use to connect to Outlook Anywhere if it needs to. Notice here it says “Exchange HTTP” for the Protocol opposed to “Exchange RPC” in the previous image.

Below you’ll find the XML response from the “XML” tab of the Test E-mail AutoConfiguration utility. You can see the settings for both the EXCH & EXPR Outlook Providers.

<Protocol>

        <Type>EXCH</Type>

        <Server>CASArrayAustin.contoso.local</Server>

        <ServerDN>/o=Contoso/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=CASArrayAustin.contoso.local</ServerDN>

        <ServerVersion>7383807B</ServerVersion>

        <MdbDN>/o=Contoso/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=CASArrayAustin.contoso.local/cn=Microsoft Private MDB</MdbDN>

        <PublicFolderServer>EX10A.contoso.local</PublicFolderServer>

        <AD>ausdc.contoso.local</AD>

        <ASUrl>https://mail.ash.org/ews/exchange.asmx</ASUrl&gt;

        <EwsUrl>https://mail.ash.org/ews/exchange.asmx</EwsUrl&gt;

        <EcpUrl>https://mail.ash.org/ecp/</EcpUrl&gt;

        <EcpUrl-um>?p=customize/voicemail.aspx&amp;exsvurl=1</EcpUrl-um>

        <EcpUrl-aggr>?p=personalsettings/EmailSubscriptions.slab&amp;exsvurl=1</EcpUrl-aggr>

        <EcpUrl-mt>PersonalSettings/DeliveryReport.aspx?exsvurl=1&amp;IsOWA=&lt;IsOWA&gt;&amp;MsgID=&lt;MsgID&gt;&amp;Mbx=&lt;Mbx&gt;</EcpUrl-mt>

        <EcpUrl-ret>?p=organize/retentionpolicytags.slab&amp;exsvurl=1</EcpUrl-ret>

        <EcpUrl-sms>?p=sms/textmessaging.slab&amp;exsvurl=1</EcpUrl-sms>

        <OOFUrl>https://mail.ash.org/ews/exchange.asmx</OOFUrl&gt;

        <UMUrl>https://mail.ash.org/ews/UM2007Legacy.asmx</UMUrl&gt;

        <OABUrl>https://mail.ash.org/oab/69ed661e-c685-4ae2-a284-da308d7bd480/</OABUrl&gt;

      </Protocol>

<Protocol>

        <Type>EXPR</Type>

        <Server>oa.ash.org</Server>

        <SSL>On</SSL>

        <AuthPackage>Basic</AuthPackage>

        <ASUrl>https://mail.ash.org/ews/exchange.asmx</ASUrl&gt;

        <EwsUrl>https://mail.ash.org/ews/exchange.asmx</EwsUrl&gt;

        <EcpUrl>https://mail.ash.org/ecp/</EcpUrl&gt;

        <EcpUrl-um>?p=customize/voicemail.aspx&amp;exsvurl=1</EcpUrl-um>

        <EcpUrl-aggr>?p=personalsettings/EmailSubscriptions.slab&amp;exsvurl=1</EcpUrl-aggr>

        <EcpUrl-mt>PersonalSettings/DeliveryReport.aspx?exsvurl=1&amp;IsOWA=&lt;IsOWA&gt;&amp;MsgID=&lt;MsgID&gt;&amp;Mbx=&lt;Mbx&gt;</EcpUrl-mt>

        <EcpUrl-ret>?p=organize/retentionpolicytags.slab&amp;exsvurl=1</EcpUrl-ret>

        <EcpUrl-sms>?p=sms/textmessaging.slab&amp;exsvurl=1</EcpUrl-sms>

        <OOFUrl>https://mail.ash.org/ews/exchange.asmx</OOFUrl&gt;

        <UMUrl>https://mail.ash.org/ews/UM2007Legacy.asmx</UMUrl&gt;

        <OABUrl>https://mail.ash.org/oab/69ed661e-c685-4ae2-a284-da308d7bd480/</OABUrl&gt;

      </Protocol>

      <Protocol>

This image shows the actual Outlook Anywhere settings being configured on the client as a result of the AutoDiscover EXPR response. (File>Account Settings>Change>More Settings>Connection)

Resolution:

So the solutions here is actually fairly easy & oddly enough, not well known. The Set-CASMailbox command can be used to block a particular mailbox from accessing various Client Access features. In this case we can use it to block Outlook Anywhere for John’s Mailbox. (Note: This command can also be scripted or piped to take effect on any number of mailboxes in the environment).

Set-CASMailbox –Identity John –MAPIBlockOutlookRpcHttp $True

After running this command you may need to wait about 15min for AD replication to take effect as well as 15min for AutoDiscover, Outlook Anywhere, & ultimately Outlook to take the change. To speed this process up you can recycle the MSExchangeAutodiscoverAppPool in IIS as well as restart the Microsoft Exchange Service Host service on each CAS.

Now, if you run Test E-mail AutoConfiguraton you’ll see that the Outlook client doesn’t even get the EXPR response because they’ve had that feature blocked.

Now if you look at the Outlook Anywhere settings (below) in Outlook, they are no longer even configured/enabled. Now in my lab using Outlook 2013 I had to actually perform a profile repair to get this change to take effect immediately. You will likely either have to wait longer for it to take effect or manually repair the profile.

So in this customer’s case, users who were not allowed to use Outlook Anywhere would not get the Outlook Authentication prompt when moving from internal wired to wireless or vice versa because their Outlook client never attempted the Outlook Anywhere connection; they just remained in a disconnected state until the new connection came fully online.

Also, after showing the customer this command they no longer had to rely on UAG to control who could or couldn’t access Outlook Anywhere; they could now just script the Set-CASMailbox command.

Quick method to diagnose Exchange Active Directory Access & Service Startup Issues

Background:

My colleague Jedidiah Hammond wrote a great post awhile back on troubleshooting Exchange Service start-up issues. One of the main areas of focus of the post were issues with Active Directory Global Catalog servers. This can be considered an ad-on to that post as I’ll describe a useful method to troubleshoot Exchange permissions in Active Directory; more specifically, verifying Exchange has the proper access to the Global Catalog servers in and out of it’s respective Active Directory site.

Scenario:

Suppose you find that the Microsoft Exchange Active Directory Topology Service isn’t starting; or the System Attendant, or the Information Store service. Or perhaps the Exchange Management Console or Exchange Management Shell will not connect and is complaining of Active Directory/Global Catalog issues.
Often times this is a result of a port being blocked by Anti-V/Firewall between the Exchange Server and your Global Catalog. Or possibly a configuration issue on the network stack (IP/DNS/etc); maybe someone even powered your GC off much to your dismay. Assuming you have already worked through the above scenarios, one useful tool to verify Exchange/AD functionality is actually a very commonly used one; Event Viewer.

When you first deploy Exchange and run “setup /PrepareAD” (or you let the GUI setup do it for you) it is actually setting many of these permissions in AD. (For a list of all of these changes see this Technet article).

Steps:

Below is an excerpt from MSExchange ADAccess Informational Event ID 2080. You’ll find it occurring roughly every 15min on your Exchange Servers.
Description:
Process STORE.EXE (PID=3376). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
Austin.ASH.ORG    CDG 1 7 7 1 0 1 1 7 1
 Out-of-site:
Houston.ASH.ORG    CDG 1 7 7 1 0 1 1 7 1

This is an example of what the output should look like. You might be asking what those series of numbers represent. Well buried deep within the land of Exchange 2000 there lies a KB article explaining just that.

After reading the article you’ll find that these numbers are basically describing Exchange’s understanding of the Global Catalog servers made available to it; along with whether or not it has the proper ACLs set to be able to utilize them. If you find yourself pulling your hair out as to why Exchange is showing the symptoms I listed earlier, then look for this event on your Exchange server and you just might see something like the following:

Description:
Process STORE.EXE (PID=3376). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
Austin.ASH.ORG    CDG 1 7 7 1 0 0 1 7 1
 Out-of-site:
Houston.ASH.ORG    CDG 1 7 7 1 0 0 1 7 1

Notice it ends with “0171″ instead of “1171″. If we reference the above KB article then this tells us Exchange lacks the proper ACL’s in AD.

I’ve seen this many times with customers who have modified the Default Domain Controllers Group Policy or somehow blocked it’s use. I’ve also seen similar issues arise from unchecking “Include Inheritable Permissions from this Object’s Parent” in AD for various objects. If this is the case then please see the post I referenced earlier on how to resolve that. In addition, I’ve found re-running “setup.com /PrepareAD” to be a very useful troubleshooting step in situations such as these where you feel AD permissions may be at fault. Some customers have been weary of running this but honestly their fears stem from ignorance because “it just sounds scary” ; a quick read over the article I referenced earlier will tell you that running it again will only re-add the permissions Exchange has needed all along.
However, be aware that re-running PrepareAD may only resolve the issue temporarily as any bad Group Policies may find themselves being re-applied in about 15min so fixing the actual source of the issue should be the ultimate goal.

An additional note here is if you’re utilizing AD Split permissions with Exchange, there may be additional precautions to be taken before running PrepareAD  again.

Requesting an Exchange Certificate from an Enterprise Certificate Authority using Command-Line (WebServer Template Missing in 08 /CertSRV)

Scenario:

Customer had hired a Consultant to originally setup their Exchange 2007 environment and now their Certificate had expired. Was originally setup to use their 2008 Enterprise CA so customer not only did not know how to generate the request from within Exchange but also did not know how to submit it to their own CA (I know).

Now with a 2003 CA I would just generate the certificate request from within Exchange Management Shell then Browse to
http://CA-Name/CertSRV
> Click Request a Certificate>Advanced Certificate Request>Submit a Certificate Request by Using a Base-64…..>Then select “Web Server” from the Certificate Template drop-down (Figure 1).

However, on a 2008 CA you do not have the option for Web Server (Figure 2)

This obviously makes it difficult to use the old familiar web-based interface to request your certificate. I believe these additional templates were removed from /CertSRV by default due to security reasons but I have yet to confirm.

Resolution:

So in this case I just needed to generate the certificate request on 2007, copy the .req file to my CA, and use the certreq.exe utility on the CA to process the request. The commands for the request are as follows:

Certreq.exe –submit –attrib “CertificateTemplate:webserver” C:\RequestFile.req NewCertName.cer

Depending on the settings of your CA this request may be auto approved (in which case the .cer file will be located in your current working directory in Command-Prompt; or just specify a path in the command) or you may need to approve it. You can do this either by launching the Certificate Authority MMC snap-in and going to “Pending Requests” or using the following command:

Certreq.exe –accept NewCertName.cer

Once you get the cert file just import it using Exchange Management Shell (if 2007; I usually recommend the GUI Wizard in 2010).

References:

Exchange 2007 Certificate Request Command

Exchange 2007 Certificate Import Command

Exchange Enable Certificate for Services

Exchange 2010 Certificate Request/Import/Assignment Process

If you choose to use the command line method on a 2003 CA then you may have to go through the following article

In searching to see if anyone else had published these steps I ran across the blog of Jeff Schertz. I’ve been to his blog before and always find great content. Here’s the referenced post but check out some of his other great articles; specifically for Lync.

Edit: Check this post if you receive a “Certificate Not Issued (Incomplete)” message via command prompt.

Windows Backup Error 2155348010 – The request could not be performed because of an I/O device error

Technically a Windows Backup/External USB Drive issue but it came about when trying to perform an Exchange backup.

Scenario:

Smaller customer running Exchange 2010 on a new deployment. They’d been having trouble setting up their backup and as a result had 30GB of transaction logs and were running low on space. Customer was trying to use a Western Digital USB 2TB HD as the backup destination. Backups would take forever on “Running Consistency Check for Application Exchange” and would ultimately fail with the following error message within Windows Server Backup:

“One of the backup files could not be created. Detailed Error: The request could not be performed because of an I/O device error”

Event Viewer would also give:

Source: Microsoft-Windows-Backup
“failed with following error code ’2155348010′ (One of the backup files could not be created.). Please rerun backup once issue is resolved.“

Resolution:

Ran vssadmin list writers and also enabled expert level logging (Set-EventLogLevel “MSExchangeIS\9002 System\Exchange Writer” -Level Expert) but the writers all showed a state of Stable.

The backup was hanging at “Running Consistency Check for Application Exchange…” because of all the transaction logs. Dismounted database, removed logs, checkpoint file, and re-mounted. The backup would now just take much less time to fail with the same error

Found the following articles that seemed to address this issue:

http://social.technet.microsoft.com/Forums/en-US/windowsbackup/thread/5d9e2f23-ee70-4d41-8bfc-c9c4068ee4e2

http://www.cmoullas.net/windows-backup-error-0x8078002a-2-5tb-3tb-wd-drives/

The issue was being caused by the fact that Windows Server Backup does not support drives with large sector size (4096 bytes) unless they support 512 byte emulation.

Information about Microsoft Support policy for large-sector drives in Windows

http://support.microsoft.com/kb/2510009

Drive type	Support/functionality
4K native (4K logical sector size): Most of the drives larger than 2 terabytes and with a USB connection are this kind of drive.	Currently not supported on any released operating system version. Contact your drive manufacturer for support.

The fix was ultimately supplied from the blog post in the second link. We downloaded and used the WD Quick Formatter Tool (
http://wdc.custhelp.com/app/answers/detail/search/1/a_id/3868
) on the USB drive to format it with the non-XP compatible setting. Hopefully other manufacturers have similar tools or properly support 512 byte emulation.

Afterwards the Windows Backup completed successfully and the Exchange logs were being properly truncated.

Disable TOE and RSS

These technologies are great if your environment support them end to end, but if not you may see some of the following issues.

Symptoms include

• Sporadic Network issues.
• Service failing (Because of network login issues)
• Delay in service start (Because of network login issues)
• Unexplained issues that Seem to be network related but other areas have already been investigated

 

Resolution: To keep it as simple and reliable as possible

1. Update to latest drivers
2. Disable Everything that says offload or scaling in the NIC properties
3. Disable it for the OS as well

2008      

• netsh interface tcp set global rss=disabled
• netsh interface tcp set global chimney=disabled
• netsh interface tcp set global autotuninglevel=disabled

2003

• Netsh int ip set chimney disabled

How to Fully Re-Install Exchange On SBS 2008

(Note: this will remove from your environment so you will have to re-create\attach mailboxes)

Recently I had a situation where I needed to fully remove and re-install Exchange on SBS2008, here is the process I was able to develop using technet, a couple blogs and trial an error

Prep For uninstall

Run Exchange Management Shell as administrator

Answer A or Y to all PS Prompts

1. Remove all mailboxes
   • Get-mailbox | disable-mailbox
2. Remove-receive Connectors
   • Get-recieveconnector | remove-recieveconnector
   • Get-sendconnector | remove-sendconnector
3. Remove Public Folder and oab
   • Get-offlineaddressbook | remove-offlineaddressbook
   • Adsiedit.msc – > Connect to : Configuration ->
   • Configuration -> Configuration -> Services -> Microsoft Exchange -> First Org.. -> administrative Groups -> Exchange Admin Group.. -> Servers -> Servername -> Information Store -> Second Information Store -> Right Click and delete “Public Folder Databas
     

Uninstall

1. Start -> control -> programs and features -> Microsoft Exchange -> uninstall -> uncheck all roles ->
   
2. Rename Databases Folders
   1. From Powershell > move c:\program files\Microsoft\Exchange Server\mailbox” c:\program files\Microsoft\Exchange Server\mailbox.old”

Re-Install Exchange

1. Download Exchange2007 SP1 (E2K7SP1EN64.exe)  
   http://www.microsoft.com/downloads/details.aspx?FamilyID=44c66ad6-f185-4a1d-a9ab-473c1188954c&displaylang=en
   
2. Create a registry value to use with Windows SBS 2008 Disc 2 (Component Technologies for Server Repair) and Microsoft Exchange Server 2007 by doing the following:
   1. Open Registry Editor.
   2. On the User Account Control page, click Continue.
   3. Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Products\461C2B4266EDEF444B864AD6D9E5B613\SourceList\Media.
   4. Create a new string value named MediaPackage: Right-click Media, click New, click String Value, in Name type MediaPackage, and then press ENTER.
   5. Double-click MediaPackage, in Edit String, in Value data, type \CMPNENTS\Exchange12\, and then click OK.
   6. Close Registry Editor.
3. Run the Setup from the extracted Directory, Rt click Setup and Run As Administrator
4. Click Step 4: Install Microsoft Exchange Server 2007 Sp1
5. Next -> Select Mailbox\CAS\HUB
    
6. Enter Org Name (Default is “First Organization”)
    
7. Next
   

SBS Roles Reinstall Script

1. Run Windows PowerShell commands by doing the following:
   1. Copy and paste the following cmdlets into a text file: (Run Notepad as administrator)

      $LocalServerName = hostname$ActiveSyncMailboxName = “Windows SBS Mobile Mailbox Policy” + ” ” + $LocalServerName $OABVDir = $LocalServerName + “\OAB (SBS Web Applications)” $OAB = Get-OfflineAddressBook | Select-Object -Property Name $strDomainDNS = [System.DirectoryServices.ActiveDirectory.Domain]::GetCurrentDomain().Name $DomainAdmins = $strDomainDNS + “\Domain Admins” $OrgName = Get-OrganizationConfig | Select-Object -Property DistinguishedName $DefaultExchangeCertificate = “CN=” + $LocalServerName Get-ExchangeCertificate | Where { $_.Subject -eq “$DefaultExchangeCertificate” } | ForEach { Remove-ExchangeCertificate -Thumbprint $_.Thumbprint } Set-ForeignConnector “Windows SBS Company Web Connector $LocalServerName” -SourceTransportServers $LocalServerName REG ADD HKLM\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ParametersSystem /v “Disable MAPI Clients” /t REG_SZ /d 0.0.0-5.3164.0 Get-MailboxDatabase | Set-MailboxDatabase -OfflineAddressBook $OAB.Name -DeletedItemRetention “30.00:00:00″ Get-StorageGroup | Set-StorageGroup -CircularLoggingEnabled 1 Remove-OWAVirtualDirectory -Identity “Owa (Default Web Site)” -Confirm:$false Remove-OWAVirtualDirectory -Identity “Exadmin (Default Web Site)” -Confirm:$false Remove-OWAVirtualDirectory -Identity “Exchange (Default Web Site)” -Confirm:$false Remove-OWAVirtualDirectory -Identity “Exchweb (Default Web Site)” -Confirm:$false Remove-OWAVirtualDirectory -Identity “Public (Default Web Site)” -Confirm:$false Remove-WebServicesVirtualDirectory -Identity “EWS (Default Web Site)” -Confirm:$false Remove-ActiveSyncVirtualDirectory -Identity “Microsoft-Server-ActiveSync (Default Web Site)” -Confirm:$false Remove-OabVirtualDirectory -Identity “OAB (Default Web Site)” -Force:$true -Confirm:$false Remove-UMVirtualDirectory -Identity “UnifiedMessaging (Default Web Site)” -Confirm:$false Remove-AutodiscoverVirtualDirectory -Identity “Autodiscover (Default Web Site)” -Confirm:$false Remove-OWAVirtualDirectory -Identity “Exadmin (SBS Web Applications)” -Confirm:$false Remove-OWAVirtualDirectory -Identity “Exchange (SBS Web Applications)” -Confirm:$false Remove-OWAVirtualDirectory -Identity “Public (SBS Web Applications)” -Confirm:$false New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2007″ -ExternalAuthenticationMethods Fba Set-OWAVirtualDirectory -InternalUrl “https://sites/owa/” -ClientAuthCleanupLevel “Low” -LogonFormat “UserName” -DefaultDomain $strDomainDNS -Identity “Owa (SBS Web Applications)” New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000″ -VirtualDirectoryType “Exadmin” -ExternalAuthenticationMethods Fba New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000″ -VirtualDirectoryType “Mailboxes” -ExternalAuthenticationMethods Fba New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000″ -VirtualDirectoryType “Exchweb” -ExternalAuthenticationMethods Fba New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000″ -VirtualDirectoryType “PublicFolders” -ExternalAuthenticationMethods Fba New-WebServicesVirtualDirectory -WebsiteName “SBS Web Applications” -InternalUrl “https://Sites/EWS/Exchange.asmx” -basicauthentication 1 -windowsauthentication 1 New-ActiveSyncVirtualDirectory -WebsiteName “SBS Web Applications” -InternalUrl “https://Sites/Microsoft-Server-ActiveSync” -ExternalAuthenticationMethods Basic -InternalAuthenticationMethods Basic New-OabVirtualDirectory -WebsiteName “SBS Web Applications” -InternalUrl “https://Sites/OAB” Set-OabVirtualDirectory -PollInterval “30″ -Identity “oab (sbs web applications)” New-UMVirtualDirectory -WebsiteName “SBS Web Applications” -InternalUrl “https://Sites/UnifiedMessaging/Service.asmx” New-AutodiscoverVirtualDirectory -WebsiteName “SBS Web Applications” -InternalUrl “https://Sites/Autodiscover/Autodiscover.xml” -BasicAuthentication 1 -WindowsAuthentication 1 Set-ClientAccessServer -Identity $LocalServerName -AutoDiscoverServiceInternalUri “https://sites/Autodiscover/Autodiscover.xml” Set-OfflineAddressBook $OAB.Name -VirtualDirectories $OABVDir -Versions Version2,Version3,Version4 -PublicFolderDistributionEnabled:$True iisreset /noforce cd $env:windir\system32\inetsrv .\appcmd.exe unlock config “-section:system.webserver/security/authentication/windowsauthentication” .\appcmd.exe set config “SBS Web Applications/ews” “-section:windowsAuthentication” “-useKernelMode:False” /commit:apphost .\appcmd.exe set config “SBS Web Applications/AutoDiscover” “-section:windowsAuthentication” “-useKernelMode:False” /commit:apphost .\appcmd.exe set config “SBS Web Applications/oab” “-section:windowsAuthentication” “-useKernelMode:False” /commit:apphost .\appcmd.exe set site “Default Web Site” /Bindings:http/*:80: .\appcmd.exe start site “Default Web Site” .\appcmd.exe start site “SBS Web Applications”

   2. Name the text file sbsAllrolereinstall.ps1, and then save it in C:\windows\system32.
   3. Open Exchange Powershell: Click Start, click All Programs, click Microsoft Exchange Server 2007, right-click Exchange Management Shell, and then click Run as administrator.
   4. On the User Account Control page, click Continue.
   5. In Exchange Management Shell, type .\sbsAllrolereinstall.ps1, and then press ENTER.
   6. Type Y to confirm the action, and then press ENTER.
   7. Close Exchange Management Shell.
2. Update the settings in the Internet Information Services (IIS) Manager by doing the following:
   1. Click Start, click Administrative Tools, and then click Internet Information Services (IIS) Manager.
   2. On the User Account Control page, click Continue.
   3. Expand the name of your Windows SBS 2008 server, and then expand Sites.
   4. Click Default Web Site, and then click Start.
   5. Expand SBS Web Applications, and then click OAB.
   6. In OAB Home, double-click SSL Settings.
   7. Select Require SSL and Require 128-bit SSL, and then click Apply.
3. To send e-mail messages to the Internet, run the Fix My Network Wizard by doing the following:
   1. Open the Windows SBS Console.
   2. On the navigation bar, click Network, and then click Connectivity.
   3. In the task pane, click Fix my network.
   4. Follow the instructions in the wizard. You can click each potential problem that the wizard lists to get more information about the problem. To recreate the default Send and Receive connectors to re-establish mail flow, select Exchange SMTP connectors are invalid.
4. Rerun the Internet Address Management Wizard to associate the new virtual directories with the correct external URL.
5. If your server uses a trusted certificate, you must also run the Add Trusted Certificate Wizard.
6. You may end up with the exchange saying its not license you can register Exchange on SBS with this tool.
   Activate Exchange
   
   http://rahman.eu/ActivateExchange.exe
   

Info gathered from

Rahman® Reviews

http://duitwithsbs.wordpress.com/2009/04/28/sbs-2008-how-to-repair-exchange-2007/


http://technet.microsoft.com/en-us/library/dd728006(WS.10).aspx

List of SBS Recovery options.

http://technet.microsoft.com/en-us/library/dd430085%28WS.10%29.aspx

1.  

Exchange 2007\2010 Database Replication issues – Re-seed

Database replication problems can be caused by many things

• Hardware Failure
• Network Outage
• Administrative Change
• Log Corruption

If you have replication issues this WILL cause problems in moving the over the active node of your cluster, that is one reason it is very important you do not use cluster administrator to perform the move but rather use either EMC or EMS.

EMS Process = Move-ClusteredMailboxServer -id ClusterName -TargetMachine Node2 (2007)
                      Move-ActiveMailboxDatabase “Mailbox Database” -ActivateOnServer Node2 (2010)
EMC Process =
Server Configuration -> Mailbox -> Right Click ServerName -> Manage Clustered Mail Box Server -> browse for server -> next -> move (2007)
Organization Configuration -> Mailbox -> Database management -> Rt Click Database -> Move Active mailbox Database -> browse for server -> ok -> move (2010)

If you suspect you have a replication issue or you did but have now resolve the problem and are ready to “Re-Seed” your databases here is the process

EMS Process = Open Exchange management Shell ON the PASSIVE NODE

1. First verify that the database is not replicating correctly
   “get-storagegroupcopystatus or (Get-MailboxDatabaseCopyStatus for 2010)”
   If the state is other than healthy or the numbers for the queues are increasing then you have a replication issue.
   CopyQueue length (Logs queuing up to be sent to the passive node)
   ReplayQueue length (Logs queuing up to be replayed to the passive node database)
    
2. To re-seed a database (remove all the passive contents and re-copy)
   • On the PASIVE node first suspend the Storage group in question
     “Suspend-StorageGroupCopy -id clustername\storagegroup” (2007)
     “Suspend-MailboxDatabaseCopy -id mailboxname\servername” (2010)
     then force an update removing the old data on the passive.
     “Update-storagegroucopy -id clustername\storagegroup -DeleteExistingFiles” (2007)
     “Update-MailboxDatabasecopy -id mailboxname\servername -DeleteExistingFiles” (2010)
     Or if you just want to reseed the entire Server (2007)
     • Get-StorageGroup -Server clustername | suspend-StorageGroupCopy
     • Get-StorageGroup -Server clustername| Update-StorageGroupCopy -DeleteExistingFiles

EMC Process = Open Exchange Management Console on the PASSIVE NODE

1. Server Configuration -> Mailbox -> Select Cluster- > rt click Storage Group -> Suspend Storage group Copy (2007)
   Organization Configuration -> Mailbox -> Database management -> Rt Click Database copy -> suspend database copy (2010)
2. Server Configuration -> Mailbox -> Select Cluster- > rt click Storage Group -> update storage group copy -> -> check “Delete Existing”->  Next -> Update (2007)

                                     

Organization Configuration -> Mailbox -> Database management -> Rt Click Database copy -> update Database copy -> update (2010)

                                                  

Glossary
EMS = Exchange Management Shell
EMC = Exchange Management Console
Variable
Powershell command

How To: Exchange database Recovery 2003\2007 or Become an Exchange database Recovery expert!!!!!

SUMMARY

This article describes how to use the Microsoft Exchange Server 2003 Eseutil tool (Eseutil.exe)  

How to Recover a Corrupt Exchange Database

1. Be sure there are no hardware issues and that the File System is intact (run chkdsk /r and repeat until it doesn’t return errors), restore from backup is preferable if possible.
2. Add the Exchange bin to the “path” statement
   1. Start -> Right click my computer -> properties -> advanced Tab -> environment variables (scroll down if you have to)
   2. In the bottom pane double click “path”
   3. Go to the end of the line enter a semi-colon and enter the path to the exchange bin directory
3. Verify that there is twice as much free space as the size of the database,
   i.e if the database is 10 gb you need to have at least 20 gb free
4. Open command prompt
5. Change to the directory where the database is (usually C:\program files\exchsrv\mdbdata, if you cant find the database search for *.edb)
6. Type eseutil /mh <database name> (usually priv1.edb)
   1. Look for the shutdown state (may have to scroll up) , if its in clean shutdown then you should be able to mount the database, but chances are if your reading this its in a dirty shutdown and you need to continue on
7. Type eseutil /k <database name> (usually priv1.edb) look for “Bad Checksums” if you have any, your repair possibility will be slim (just make a note of it)
8. Type eseutil /ml e00 we are looking for any missing logs (if none missing go on to step 9)
   1. if there are any we have to move all subsequent logs. and the e00 log to another location, rename the last log to e00.log
   2. Example we have logs E00021-E00035 and E00033 is missing, remove E00, E00034 and E00035
9. Re-name the .chk file to .oldchk
10. Type eseutil /r E00
    1. Wait for that to complete, if it completes successfully move on to 11 otherwise run “P”
       1. P. eseutil /p <database name> this will force the database into a clean shutdown and will rip out any incomplete pages, this should only be done as a last resort
11. Start information store service, verify that the store is mounted in Exchange System Manager (ESM), if its not mounted right click and mount
    1. Start -> All Programs -> Microsoft Exchange -> System Manager
    2. Expand Administrative Group -> Administrative group -> Servers -> {Server name} -> {Storage Group}
    3. Right Click Mailbox store -> mount
12. Dismount the store in ESM
13. Go back to the command window
    1. type isinteg –s <servername> -fix –test alltests
    2. When prompted select the store that you repaired
    3. If any fixes are reported re-run until none are reported (just like chkdsk)
14. Type eseutil /d <database name> (usually priv1.edb)
15. Remount the database and should now be able to send receive mail

Command Description
Eseutil is a Jet database repair utility
Switch Action
/D Defragments the database
/R Soft recovery of the database
/G Integrity Check
/K Checksum
/P Hard Repair
/ML Log File dump
/MH Database file dump
/Y Copy
/C Restore

Isinteg is a integrity checker much like the windows chkdsk for NTFS

ESM Exchange System Manager is the GUI Management tool for exchange
Exchange 2007 changed the name to EMC Exchange Management Console

REFERENCES

For more information about Exchange 2003, visit the following Microsoft Web site:

http://technet.microsoft.com/en-us/library/bb123872.aspx

Use the Eseutil Utility to Detect File Header Damage in Exchange 2003

http://support.microsoft.com/kb/825088

How to defragment with the Eseutil utility (Eseutil.exe)

http://support.microsoft.com/kb/192185

Description of the Isinteg utility

http://support.microsoft.com/kb/182081

Technorati Tags: Eseutil,isinteg,exchange,priv1.edb,Exchange Recovery