Unable to Recreate Exchange Virtual Directory


Issue

A customer of mine recently had an issue where their Exchange 2013 OWA Virtual Directory was missing in IIS. When attempting to recreate the vDir we encountered the below error message:

“An error occurred while creating the IIS virtual directory `IIS://ServerName/W3SVC/1/ROOT/OWA’

1

To resolve this error I needed to resort to using a long lost tool from the days of old, the IIS 6 Resource Kit.

Note: This blog post could also be relevant if the OWA (or any other) vDir needed to be recreated and you encountered the same error upon recreation.

Resolution

Back in the days of Exchange 2003, the IIS Resource Kit, or more specifically the Metabase Explorer, could be used when recreating a Virtual Directory. Fortunately, the Metabase Explorer tool still works with IIS 8.

Download Link for the IIS 6 Resource Kit

The error encountered above was a result of the IIS Metabase still holding remnants of a past instance of the OWA Virtual Directory, which was preventing the New-OwaVirtualDirectory Cmdlet from successfully completing. It’s important to understand that an Exchange Virtual Directory is really located in two places; Active Directory and IIS. When running the Get-OwaVirtualDirectory Cmdlet (or similar commands for other Virtual Directories), you’re really querying Active Directory. For example, the OWA Virtual Directories for both the Default Web Site and Exchange Back End website in my lab are located in the following location in AD (via ADSIEDIT):

2

So if a vDir is missing in IIS but present in AD, you’ll likely need to first remove it using the Remove-*VirtualDirectory Cmdlet otherwise it will generate an error stating it already exists. In my customer’s scenario, I had to do this beforehand as the OWA vDir was present in AD but missing in IIS.

This brought us to the state we were in at the beginning of this post; receiving the above error message. The OWA vDir was no longer present in AD nor in the Default Web Site, but when trying to recreate it using New-OwaVirtualDirectory we received the above error message.

Tip: Use Get-*VirtualDirectory with the –ShowMailboxVirtualDirectories parameter to view the Virtual Directories on both web sites. For example:

3

The solution was to install the IIS 6 Resource Kit and use Metabase Explorer to delete the ghosted vDir. When installing the Resource Kit, select Custom Install and then uncheck all features except for Metabase Explorer 1.6 and proceed with the installation. Once it finishes, it may require you add the .NET Framework 3.5 Feature.

When you open the tool on the Exchange Server in question, navigate to the below tree structure and delete the old OWA Virtual Directory by right-clicking it and selecting Delete. When completed, the OWA vDir should no longer be present (as seen below).

4

You should now be able to successfully execute the New-OwaVirtualDirectory Cmdlet. It’s always a bit nostalgic seeing a tool of days gone by still able to save the day. I’d like to thank my co-worker John Dixon for help with this post. When I can’t figure something out in Exchange/IIS (or anything really) he’s who I lean on for help.

NIC DNS Registration and Exchange Servers


Symptom

I recently worked with a customer who had introduced an Exchange 2013 Server into an existing Exchange 2007 environment. The issue was the 2013 Server was unable to send email anywhere; neither externally or to other Exchange Servers. If you executed the below command to view the status of the transport queues you received the below output:

Get-Queue <Queue Identity> | FL

NIC

Specifically, the error message you would receive is “4.4.0 DNS query failed. The error was: DNS query failed with error ErrorRetry”

This is a fairly common error indicating there is an issue contacting the DNS Server or Servers that Exchange is configured to use. ReferenceA ReferenceB

Resolution

However, in this case the issue was not obvious, unless you had already seen this issue before or knew a little bit about the health checks Exchange uses to ensure it’s healthy.

I remembered seeing a similar issue on a Reddit thread awhile back, which caused me to search and find this Microsoft KB article titled “DNS query failed” error when an email message is stuck in the Draft folder in an Exchange Server 2013 environment”.

This was the resolution in my scenario as well. To resolve the issue, I simply had to re-check the “Register this connection’s addresses in DNS” option on the IPv4> Properties>Advanced>DNS tab on the primary NIC used for Active Directory communications. While you can uncheck this box on secondary NICs (such as for iSCSI, Replication, Backup, etc.), it should always remain checked on the MAPI/Primary NIC. I’ve also seen issues where having this unchecked on a 2013/2016 DAG node will result in Managed Availability-triggered database failovers.

Quick method to determine installed version of .NET Framework


Due to recent issues with unsupported versions of .NET being installed on Exchange servers, as well as the fact that Exchange Server requires specific versions of .NET to be installed (Exchange Server 2013 System Requirements & Exchange Server 2016 System Requirements), there is a need to quickly query the installed version of .NET on Exchange servers. I have also been involved in several Exchange support escalations where updating the Exchange servers from .NET 4.5.1 to 4.5.2 resolved CPU performance issues.

Fortunately, my coworker and fellow Exchange MCM Mark Henderson wrote this quick and easy way to query the currently installed version of .NET.

PowerShell Query Method

To query the local Registry using PowerShell, execute the below command in an elevated PowerShell session.

(Get-ItemProperty ‘HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full’  -Name Release).Release

You can then use the table below to reference the installed version of .NET. For instance, if the returned value is 379893, then .NET 4.5.2 is installed.

 

Version Value of the Release DWORD
.NET Framework 4.5 378389
.NET Framework 4.5.1 installed with Windows 8.1 378675
.NET Framework 4.5.1 installed on Windows 8, Windows 7 SP1, or Windows Vista SP2 378758
.NET Framework 4.5.2 379893
.NET Framework 4.6 installed with Windows 10 393295
.NET Framework 4.6 installed on all other Windows OS versions 393297
.NET Framework 4.6.1 installed on Windows 10 394254
.NET Framework 4.6.1 installed on all other Windows OS versions 394271
.NET Framework 4.6.2 installed on Windows 10 Anniversary Update 394802
.NET Framework 4.6.2 installed on all other Windows OS versions 394806

Script method

Copy the below text into a text file and rename the extension to .ps1. You can then execute this script and have it automatically tell you the installed version of .NET.

# Determine the version of .net 4 framework by querying Registry HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full for Value of Release
#
# Based on https://msdn.microsoft.com/en-us/library/hh925568(v=vs.110).aspx
#
#
#

$Netver = (Get-ItemProperty ‘HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full’ -Name Release).Release

If ($Netver -lt 378389)
{
Write-Host “.NET Framework version OLDER than 4.5” -foregroundcolor yellow
}
ElseIf ($Netver -eq 378389)
{
Write-Host “.NET Framework 4.5” -foregroundcolor red
}
ElseIf ($Netver -le 378675)
{
Write-Host “.NET Framework 4.5.1 installed with Windows 8.1” -foregroundcolor red
}
ElseIf ($Netver -le 378758)
{
Write-Host “.NET Framework 4.5.1 installed on Windows 8, Windows 7 SP1, or Windows Vista SP2” -foregroundcolor red
}
ElseIf ($Netver -le 379893)
{
Write-Host “.NET Framework 4.5.2” -foregroundcolor red
}
ElseIf ($Netver -le 393295)
{
Write-Host “.NET Framework 4.6 installed with Windows 10” -foregroundcolor red
}
ElseIf ($Netver -le 393297)
{
Write-Host “.NET Framework 4.6 installed on all other Windows OS versions” -foregroundcolor red
}
ElseIf ($Netver -le 394254)
{
Write-Host “.NET Framework 4.6.1 installed on Windows 10” -foregroundcolor red
}
ElseIf ($Netver -le 394271)
{
Write-Host “.NET Framework 4.6.1 installed on all other Windows OS versions” -foregroundcolor red
}
ElseIf ($Netver -le 394802)
{
Write-Host “.NET Framework 4.6.2 installed on Windows 10 Anniversary Update” -foregroundcolor red
}
ElseIf ($Netver -le 394806)
{
Write-Host “.NET Framework 4.6.2 installed on all other Windows OS versions” -foregroundcolor red
}

 

References:

How to: Determine Which .NET Framework Versions Are Installed

 

Mailbox Anchoring affecting new deployments & upgrades


Update2 (March 1st 2016): Microsoft has released the following blog post which states this behavior will be reverted/absent in 2013 CU12 and RTM/CU1 versionf of Exchange 2016 Remote PowerShell Proxying Behavior in Exchange 2013 CU12 and Exchange 2016

Update: Microsoft has released the following KB article to address this issue: “Cannot process argument transformation” error for cmdlets in Exchange Server 2013 with CU11

Note: This article should also apply when Exchange 2016 CU1 releases and includes Mailbox Anchoring (unless Microsoft makes a change to behavior before it’s release). So the scenario of installing the first Exchange 2016 server using CU1 bits into an existing environment would also apply.

Summary

It was announced in Microsoft’s recent blog post about Exchange Management Shell and Mailbox Anchoring that the way Exchange is managed will change going forward. Starting with Exchange 2013 CU11 (released 12/10/2015) and Exchange 2016 CU1 (soon to be released), an Exchange Management Shell session will be directed to the Exchange Server where the user who is attempting the connection’s mailbox is located. If the connecting user does not have a mailbox, an arbitration mailbox (specifically SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c) will be used instead. In either case, if the mailbox is unavailable (because it’s on a database that’s dismounted or is on a legacy version of Exchange) then Exchange Management Shell will be inoperable.

Issue

While it has always been recommended to move system and Arbitration mailboxes to the newest version of Exchange as soon as possible, there is a scenario involving Exchange 2013 CU11 which have led to customer issues:

  • Existing Exchange 2010 Environment
  • The first version of Exchange 2013 installed into the environment is CU11
  • Upon installation, the Exchange Admin is unable to use Exchange Management Shell on Exchange 2013. Thus preventing the management of Exchange 2013 objects
  • The Exchange Admin may also be unable to access the Exchange Admin Center using traditional means

This is due to the new Mailbox Anchoring changes. If the Exchange Admin’s mailbox (or the Arbitration mailbox, if the Exchange Admin did not have a mailbox) was on Exchange 2013 then this issue would not exist. However, because this was the first Exchange 2013 server installed into the environment, and it was CU11, there was no way to prevent this behavior.

This issue was first reported by Exchange MVP Ed Crowley, and yesterday a customer of mine also encountered the issue. The symptoms were mostly the same but the ultimate resolution was fairly straightforward.

Possible Resolutions

Resolution#1:

Attempt to connect to Exchange Admin Center on 2013 using the “Ecp/?ExchClientVer=15” string at the end of the URL (Reference). For Example:

I’ve heard mixed results using this method. When Ed Crowley encountered this issue, this URL worked, yet when I worked with my customer I was still unable to access EAC by using this method. However, it is worth an attempt. Once you’re connected to EAC, you can use it to move your Exchange Admin mailbox to 2013. However, should you not have a mailbox for your Exchange Admin account, this method may fail because there’s currently no way to move Arbitration Mailboxes via the EAC. So it’s recommended to create a mailbox for your Exchange Admin account using the EAC and then you’ll be able to connect via EMS.

Resolution#2:

Note: Using this method has a low probability of success as Microsoft recommends using the newer version of Exchange to “pull” a mailbox from the older version. Based on feedback I’ve received from Microsoft Support, you may consider just skipping this step and going to Step 3.

Use Exchange 2010 to attempt to move the Exchange Admin mailbox to a database on Exchange 2013. Historically, it’s been recommended to always use the newest version of Exchange to perform a mailbox move. In my experience this is hit or miss depending on the version you’re moving from and the version you’re moving to. However, it’s worth attempting:

Issue the below command using Exchange 2010 Management Shell to move the Exchange Admin’s mailbox to the Exchange 2013 server:

New-MoveRequest <AdminMailbox> -TargetDatabase <2013Database>

If the Exchange Administrator does not have a mailbox, then move the Arbitration mailboxes to Exchange 2013:

Get-Mailbox –Arbitration | New-MoveRequest -TargetDatabase <2013Database>

Resolution#3:

Connect to Exchange 2013 CU11 using Local PowerShell and manually load the Exchange modules:

  • On the Exchange 2013 CU11 Server, open a Windows PowerShell window as Administrator
  • Run the following command:
    • Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn

At this point the local PowerShell module can be used to move the Exchange Admin’s mailbox to the Exchange 2013 server:

New-MoveRequest <AdminMailbox> -TargetDatabase <2013Database>

If the Exchange Administrator does not have a mailbox, then move the Arbitration mailboxes to Exchange 2013:

Get-Mailbox –Arbitration | New-MoveRequest -TargetDatabase <2013Database>

In addition, there have been reported issues with 2013 EMS still having connectivity issues even after the relevant mailboxes have been moved. A different Windows user with appropriate Exchange permissions (using a different Windows profile) will work fine however. It seems there are PowerShell cookies for the initial profile used which could still be causing problems. In this scenario, you may have to remove all listed cookies in the following registry key (Warning, edit the registry at your own risk. A backup of the registry is recommended before making modifications):

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\WSMAN\Client\ConnectionCookies

Summary

It should be noted that while this scenario involved Exchange 2013 CU11 being installed into an existing Exchange 2010 environment, it can affect other scenarios as well:

  • Exchange 2013 CU11 or newer being installed into an existing Exchange 2010 environment
  • Exchange 2013 CU11 or newer being installed into an existing Exchange 2007 environment
  • Exchange 2016 CU1 (when released) or newer being installed into an existing Exchange 2010 environment

So unless Microsoft changes the behavior of Mailbox Anchoring, this is a precaution that should be taken when installing the first Exchange 2013 CU11/2016 CU1 (when released) server into an existing environment.

 

Edit: This forum post also describes the issue. In it, the user experiences odd behavior with the 2013 servers not being displayed if you run a Get-ExchangeServer & other odd behavior. This is similar to what I experienced in some lab testing. Ultimately, the same resolution applies.

https://social.technet.microsoft.com/Forums/en-US/05897b40-0717-437d-90ca-d550e3226c2a/exchange-2013-cu-11-breaks-some-admin-accounts-?forum=exchangesvrdeploy

 

Unable to Delete Exchange 2016 Database


Edit: Microsoft has released the below KB to acknowledge this issue:

https://support.microsoft.com/en-us/kb/3093175

Edit#2: I’ve been told that this should be remedied in a future Exchange 2016 CU by having the error dialog include the -AuditLog parameter.

Symptom

After installing Exchange 2016 you may wish to delete the default Mailbox Database that is installed. Or during regular operations, you decide to remove any 2016 Mailbox Database in the environment. Of course before you do this you’ll need to move all mailboxes from the soon to be deleted database to another Mailbox Database. You could then run the following command to verify no mailboxes still exist on the database:

Get-Mailbox –Database MailboxDatabase

This should return no results

F1

However, when attempting to remove the database you receive the below message:

This mailbox database contains one or more mailboxes, mailbox plans, archive mailboxes, public folder mailboxes or arbitration mailboxes. To get a list of all mailboxes in this database, run the command Get-Mailbox -Database <Database ID>. To get a list of all mailbox plans in this database, run the command Get-MailboxPlan. To get a list of archive mailboxes in this database, run the command Get-Mailbox -Database <Database ID> -Archive. To get a list of all public folder mailboxes in this database, run the command Get-Mailbox -Database <Database ID> -PublicFolder. To get a list of all arbitration mailboxes in this database, run the command Get-Mailbox -Database <Database ID> -Arbitration. To disable a non-arbitration mailbox so that you can delete the mailbox database, run the command Disable-Mailbox <Mailbox ID>. To disable an archive mailbox so you can delete the mailbox database, run the command Disable-Mailbox <Mailbox ID> -Archive. To disable a public folder mailbox so that you can delete the mailbox database, run the command Disable-Mailbox <Mailbox ID> -PublicFolder. Arbitration mailboxes should be moved to another server; to do this, run the command New-MoveRequest <parameters>. If this is the last server in the organization, run the command Disable-Mailbox <Mailbox ID> -Arbitration -DisableLastArbitrationMailboxAllowed to disable the arbitration

F2

A common issue in Exchange 2010 and 2013 was being unfamiliar with how to view hidden Arbitration mailboxes, which are used for various system functions, and could prevent a database from being removed. To view these mailboxes you could run the following command:

Get-Mailbox –Database MailboxDatabase –Arbitration

This should return the Arbitration mailboxes installed on the first Mailbox Database in the Exchange environment. When performing a migration from a previous version of Exchange, these mailboxes should be moved to the newest version of Exchange.

F3

However, even after these mailboxes have been vacated from this database, the original error is still displayed. In Exchange 2016 RTM, the Get-Mailbox parameters which the error dialog box presents for you to try are:

  • -Archive
  • -Arbitration
  • -PublicFolder

Unfortunately, these may still display no results. There is one additional parameter, which is new to Exchange 2016, you will likely need to use:

Get-Mailbox –Database MailboxDatabase –AuditLog

This should display a system mailbox new to Exchange 2016.

F4

Once this mailbox is moved, you should be able to successfully delete the Mailbox Database.

This –AuditLog parameter seems effectively undocumented, as the TechNet Article for the Get-Mailbox command states it is reserved for internal Microsoft use. Unfortunately, we found that is not entirely the case.

 

 

Web Management Service will not start and causes Exchange update to fail


Today I had an Exchange update issue that I’d previously never encountered before. Exchange 2013 CU10 update failed saying the Web Management Service could not be started. Attempts to manually start the service failed. Application logs pointed to IIS-IISManager 1007 event saying the following:

“Unable to read the certificate with thumbprint ‘{thumbprint}’. Please make sure the SSL certificate exists and that is correctly configured in the Management Service page.”

The thumbprint it was listing was not found on the server, either using Get-ExchangeCertificate or the MMC certificate snap-in. A web search led me to the below article which resolved the issue. Normally, an Exchange server will have a certificate called “WMSvc-servername” (Friendly Name of WMSvc) and it will be bound in IIS to the Web Management Service, but in this case the certificate was missing. By binding another certificate to the service we were able to get the service to start and continue the Exchange Update. An alternative would be to request a new certificate for the purposes of this service.

https://technet.microsoft.com/en-us/library/cc735088(v=ws.10).aspx

Find the SSL certificate that the Web Management Service is using

To find the SSL certificate that the Web Management Service is using:

  1. Click Start, click Control Panel, and then click Administrative Tools.
  2. Right-click Internet Information Services (IIS) Manager and select Run as administrator.
  3. In the Connections pane, select the server that you want to manage.
  4. In Features View, double-click Management Service.
  5. Under SSL certificate ensure that a certificate is selected.
  6. Note the name of the certificate. By default, the name starts with “WMSvc”.

Additional Reference:
http://exctech2013.blogspot.com/2013/10/the-web-management-service-could-not-be.html